feat: Enhance Classification Prompt for Stricter DevRel Query Handling #104
Conversation
- Implement strict activation rules in DEVREL_TRIAGE_PROMPT - Only trigger on explicit bot mentions or direct project questions - Ignore general chatter, greetings, and off-topic conversations - Review and improve all system prompts for clarity and efficiency - Add precise guidelines and examples for better classification - Reduce false positives and improve user experience Resolves AOSSIE-Org#93
WalkthroughThe changes revise multiple prompt templates used by the DevRel AI assistant, focusing on stricter classification logic for agent activation and improving clarity, precision, and formatting across all prompts. The updates introduce explicit rules, enhanced guidelines, and illustrative examples, ensuring the agent responds only to relevant queries and maintains consistent, actionable communication. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant DevRelClassifier
participant DevRelAgent
User->>DevRelClassifier: Send message
DevRelClassifier->>DevRelClassifier: Apply strict rules:\n- Explicit bot mention?\n- Project-specific question?
alt Needs DevRel
DevRelClassifier->>DevRelAgent: Forward relevant query
DevRelAgent->>User: Respond per improved prompts
else No DevRel needed
DevRelClassifier-->>User: No action / ignore
end
Assessment against linked issues
Possibly related PRs
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (5)
🚧 Files skipped from review as they are similar to previous changes (2)
🧰 Additional context used🧠 Learnings (3)📓 Common learningsbackend/app/classification/classification_router.py (1)backend/app/classification/prompt.py (1)🧬 Code Graph Analysis (1)backend/integrations/discord/bot.py (1)
🔇 Additional comments (6)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
Hey @Aditya30ag, Thanks a lot for contributing to this. |
|
Could you please test a bit on asking the model to output properly formatted markdown? |
| 4. github_toolkit - Handle GitHub operations (issues, PRs, repos, docs) | ||
| 5. complete - Task is finished, format final response | ||
| 1. web_search - Search for current/external information not in knowledge base | ||
| 2. faq_handler - Answer common DevRel/project questions from knowledge base |
There was a problem hiding this comment.
this has to be around the Organization to help answer specific queries related to the organization, its social, platform support, repository info; moreover aligned in a general way, but yeah organization/product specific only.
There was a problem hiding this comment.
implemented this also
| - Simple numbered lists (1. 2. 3.) - no markdown bullets | ||
| - Plain text with clear line breaks - avoid **bold** or *italic* | ||
| - Plain URLs: https://example.com | ||
| - Simple emojis for visual appeal: 🔗 📚 ⚡ |
There was a problem hiding this comment.
could you please remove these emojis and mention it to use simple emojis?
i observed that the responses after this prompt will be focused on using only these ones.
There was a problem hiding this comment.
- Plain text with clear line breaks - avoid bold or italic
- Plain URLs: https://example.com
- Simple emojis for visual appeal (use sparingly and appropriately)
- Short, scannable paragraphs
| 1. EXPLICIT BOT MENTION/TAG: | ||
| - User explicitly mentions the DevRel AI bot (e.g., "@Devr.AI", "hey bot", | ||
| "can you help", "@assistant") |
There was a problem hiding this comment.
this needs to be tested. might respond to user queries not requiring DevRel
| IGNORE COMPLETELY (always return false): | ||
| - General chatter between users: "Hey, how's it going?", "What's for lunch?" | ||
| - Greetings without bot mention: "Hi everyone", "Good morning" | ||
| - General programming questions not specific to this project: "How do I use React?" |
There was a problem hiding this comment.
please check the result on this query in both cases, with tagging the bot and this general one too. I don't think @ DevR will result in passing of DevR in the message, as internally Discord uses user IDs in interactions.
DevRel Classification ImprovementsOverviewThis document summarizes the improvements made to the DevRel classification system to address concerns raised by contributors regarding the bot's responsiveness and accuracy. Issues Addressed1. Discord Mentions Not WorkingProblem: @smokeyScraper pointed out that Discord mentions like "@devr" wouldn't work because Discord uses user IDs internally, not display names. Solution:
2. Responding to Inappropriate QueriesProblem: @smokeyScraper expressed concern that the bot might respond to user queries that don't require DevRel assistance. Solution:
Technical Changes1. Discord Bot (
|
|
A few interactions with the agent on these interactions, please @Aditya30ag. Also, did you try to make the model output markdown? The prompts are still aligned to enforce "no markdown" in outputs. |
|
closing this PR @Aditya30ag as commits are not well aligned with the requirements and this requires heavy changes. It would be better for you to start from scratch after a brief discussion with me and with contributors who prefer to work on the issue. |
2 participants
🎯 Overview
This PR implements the feature request from #93 to enhance the classification prompt with much stricter DevRel query handling, significantly reducing false positives and improving user experience.
✨ Key Changes
🔧 Main Enhancement: Classification Prompt
DEVREL_TRIAGE_PROMPTwith strict activation rules@Devr.AI, "hey bot", "can you help")🔧 Secondary Enhancement: System Prompt Review
Improved all other system prompts for clarity and efficiency:
react_prompt.py: More specific action selection criteria and clearer reasoningresponse_prompt.py: Reduced redundancy, better organized formattingsearch_prompt.py: Enhanced guidelines with examples for better search extractionintent_analysis.py: Added precise classification guidelines with specific examplesgeneral_github_help.py: Streamlined formatting requirementssummarization_prompt.py: Clearer focus areas and better organization📊 Expected Impact
Before vs After:
Test Results:
🎯 Benefits
✅ Reduced False Positives: Agent won't respond to irrelevant conversations
✅ Improved User Experience: Less intrusive AI that only engages when needed
✅ Resource Efficiency: Significant reduction in unnecessary processing
✅ Enhanced Prompt Suite: All system prompts reviewed and improved for clarity
🔍 Testing
The enhanced classification prompt has been tested with various message types:
📝 Files Changed
backend/app/classification/prompt.py- Main classification enhancementbackend/app/agents/devrel/prompts/react_prompt.py- Improved action selectionbackend/app/agents/devrel/prompts/response_prompt.py- Streamlined formattingbackend/app/agents/devrel/prompts/search_prompt.py- Enhanced search extractionbackend/app/agents/devrel/prompts/summarization_prompt.py- Better organizationbackend/app/agents/devrel/github/prompts/intent_analysis.py- Precise guidelinesbackend/app/agents/devrel/github/prompts/general_github_help.py- Reduced redundancy�� Related
Closes #93 - "FEATURE REQUEST: Enhance Classification Prompt for Stricter DevRel Query Handling"
Summary by CodeRabbit