Refine prompt and fix badcases (#13079)

Author: JiaqiZhang-Dev

tools/sdk-ai-bots/azure-sdk-qa-bot-backend/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Release History
 
+## 0.3.0 (2025-11-27)
+
+### Bugs Fixed
+
+* Ehhance intent recognition for non-technical and technical message
+  
+### Features Added
+
+* Add TypeSpec Validation category for intention prompt
+
+### Other Changes
+
+* Restructured prompt templates to enhance clarity and maintainability
+
 ## 0.2.0 (2025-11-13)
 
 ### Features Added

tools/sdk-ai-bots/azure-sdk-qa-bot-backend/config/tenant.go

@@ -26,6 +26,7 @@ var typespecSources = []model.Source{
 
 var SourceTopK = map[model.Source]int{
 	model.Source_TypeSpecMigration: 3,
+	model.Source_TypeSpecQA:        3,
 }
 
 var tenantConfigMap = map[model.TenantID]TenantConfig{
@@ -34,30 +35,30 @@ var tenantConfigMap = map[model.TenantID]TenantConfig{
 		SourceFilter: map[model.Source]string{
 			model.Source_AzureSDKGuidelines: "search.ismatch('python_*', 'title')",
 		},
-		PromptTemplate:          "qa_prompt_python.md",
-		IntentionPromptTemplate: "intention_prompt_python.md",
-		AgenticSearchPrompt:     "You are an Azure SDK for Python expert query analyzer. Your task is to decompose complex user questions into specific, searchable sub-queries that can be processed in parallel to retrieve comprehensive information from Azure SDK documentation, Python-specific guides, and TypeSpec resources.\n\n## Query Analysis Guidelines:\n1. **Identify Core Concepts**: Extract the main technical topics, SDK services, Python patterns, and TypeSpec elements\n2. **Separate Concerns**: Split questions about installation, configuration, implementation, troubleshooting, and best practices\n3. **Consider Context**: Include both Python-specific and general Azure SDK aspects\n4. **Target Sources**: Create queries that align with different knowledge sources (Python SDK docs, wikis, TypeSpec specs)\n\n## Sub-Query Generation Strategy:\n- **Installation/Setup**: How to install, configure, or initialize specific components\n- **API Usage**: Method signatures, parameters, return types, and usage patterns\n- **Authentication**: Authentication methods, credential management, and security patterns\n- **Error Handling**: Common errors, troubleshooting steps, and exception handling\n- **Best Practices**: Recommended patterns, performance optimization, and code examples\n- **Integration**: How components work together, dependencies, and compatibility\n\n## Search Optimization:\n- Use specific technical terms and SDK method names\n- Include both conceptual and implementation-focused queries\n- Add context about Python version compatibility when relevant\n- Consider async/sync patterns for Python-specific searches\n\nGenerate 3-7 focused sub-queries that together will provide comprehensive coverage of the user's question, ensuring parallel search efficiency and minimal overlap.",
+		PromptTemplate:          "language_python/qa.md",
+		IntentionPromptTemplate: "language_python/intention.md",
+		AgenticSearchPrompt:     "language_python/agentic_search.md",
 	},
 	model.TenantID_GolangChannelQaBot: {
 		Sources: append([]model.Source{model.Source_AzureSDKForGo, model.Source_AzureSDKGuidelines}, typespecSources...),
 		SourceFilter: map[model.Source]string{
 			model.Source_AzureSDKGuidelines: "search.ismatch('golang_*', 'title')",
 		},
-		PromptTemplate:          "language_channel.md",
-		IntentionPromptTemplate: "intention.md",
-		AgenticSearchPrompt:     "You are an Azure SDK for Golang expert query analyzer. Your task is to decompose complex user questions into specific, searchable sub-queries that can be processed in parallel to retrieve comprehensive information from Azure SDK documentation, Golang-specific guides, and TypeSpec resources.\n\n## Query Analysis Guidelines:\n1. **Identify Core Concepts**: Extract the main technical topics, SDK services, Golang patterns, and TypeSpec elements\n2. **Separate Concerns**: Split questions about installation, configuration, implementation, troubleshooting, and best practices\n3. **Consider Context**: Include both Golang-specific and general Azure SDK aspects\n4. **Target Sources**: Create queries that align with different knowledge sources (Golang SDK docs, wikis, TypeSpec specs)\n\n## Sub-Query Generation Strategy:\n- **Installation/Setup**: How to install, configure, or initialize specific components\n- **API Usage**: Method signatures, parameters, return types, and usage patterns\n- **Authentication**: Authentication methods, credential management, and security patterns\n- **Error Handling**: Common errors, troubleshooting steps, and exception handling\n- **Best Practices**: Recommended patterns, performance optimization, and code examples\n- **Integration**: How components work together, dependencies, and compatibility\n\n## Search Optimization:\n- Use specific technical terms and SDK method names\n- Include both conceptual and implementation-focused queries\n- Add context about Golang version compatibility when relevant\n- Consider async/sync patterns for Golang-specific searches\n\nGenerate 3-7 focused sub-queries that together will provide comprehensive coverage of the user's question, ensuring parallel search efficiency and minimal overlap.",
+		PromptTemplate:          "common/language_channel.md",
+		IntentionPromptTemplate: "prompt_template/intention.md",
+		AgenticSearchPrompt:     "common/agentic_search.md",
 	},
 	model.TenantID_AzureSDKQaBot: {
-		PromptTemplate:          "typespec.md",
+		PromptTemplate:          "typespec/qa.md",
 		Sources:                 typespecSources,
-		IntentionPromptTemplate: "intention.md",
-		AgenticSearchPrompt:     "You are a TypeSpec and Azure API expert query analyzer. Your role is to intelligently decompose complex user questions into targeted sub-queries that can be searched in parallel across TypeSpec documentation, Azure REST API specifications, and Azure API guidelines.\n\n## Query Decomposition Strategy:\n1. **Syntax & Language**: TypeSpec syntax, decorators, built-in types, and language constructs\n2. **Azure Integration**: Azure-specific decorators, patterns, and service integration\n3. **API Design**: REST API modeling, resource definitions, and operation patterns\n4. **Code Generation**: Emitter behavior, target language specifics, and generated artifacts\n5. **Migration & Adoption**: Moving from OpenAPI/Swagger to TypeSpec, conversion patterns\n6. **Compliance & Guidelines**: Azure API guidelines adherence, ARM requirements, naming conventions\n\n## Sub-Query Optimization:\n- **Specific Technical Terms**: Use exact decorator names (@route, @doc, @example), built-in types (string, int32, etc.)\n- **Context-Aware**: Include Azure service context (ARM, data-plane, management operations)\n- **Progressive Complexity**: Start with basic concepts, then dive into advanced patterns\n- **Cross-Reference**: Link TypeSpec syntax with resulting OpenAPI/ARM template patterns\n- **Practical Examples**: Focus on real-world usage scenarios and common implementation patterns\n\n## Search Targeting:\n- **TypeSpec Core**: Language fundamentals, syntax, and basic patterns\n- **Azure TypeSpec**: Azure-specific decorators, templates, and service patterns\n- **API Guidelines**: Compliance requirements, naming conventions, and best practices\n- **ARM/RPC**: Resource provider contracts, management plane requirements\n- **Migration Guidance**: Conversion patterns, tooling, and adoption strategies\n\nGenerate 4-8 precise sub-queries that comprehensively cover the user's question while enabling efficient parallel search across different knowledge domains.",
+		IntentionPromptTemplate: "typespec/intention.md",
+		AgenticSearchPrompt:     "typespec/agentic_search.md",
 	},
 	model.TenantID_AzureSDKOnboarding: {
-		PromptTemplate:          "azure_sdk_onboarding.md",
+		PromptTemplate:          "azure_sdk_onboarding/qa.md",
 		Sources:                 []model.Source{model.Source_AzureSDKDocsEng},
-		AgenticSearchPrompt:     "You are an Azure SDK onboarding query analyzer specializing in identifying basic onboarding phase documentation. Your goal is to decompose user questions into targeted searches that efficiently locate foundational guidance, prerequisites, and phase-specific documentation.\n\nNote: The user query already contains category information that should be leveraged for targeted search.\n\n## Core Onboarding Phases (Search Priority):\n1. **Service Onboarding**: Service registration, prerequisites, readiness criteria, initial setup\n2. **API Design**: REST API specifications, TypeSpec vs OpenAPI workflows, design guidelines\n3. **SDK Development**: Multi-language SDK generation, client library patterns, coding standards\n4. **SDK Release**: Release planning, versioning, GA criteria, publication workflows\n\n## Required Sub-Query Types (MUST include at least one from each):\n1. **Category Guidelines**: ALWAYS generate at least one sub-query specifically targeting category-specific guidelines, standards, and best practices based on the category information in the user query\n2. **Phase-Specific Documentation**: Target the specific onboarding phase mentioned or implied in the query\n3. **Process Documentation**: Search for workflows, procedures, and step-by-step guidance\n\n## Basic Documentation Search Strategy:\n- **Category Guidelines**: Extract the category from user query and search for \"[category] guidelines\", \"[category] standards\", \"[category] best practices\", \"[category] requirements\"\n- **Prerequisites & Getting Started**: Focus on 'what do I need before', 'requirements', 'setup'\n- **Phase-Specific Guides**: Target 'how to start', 'basic steps', 'introduction to'\n- **Common Workflows**: Search for 'typical process', 'standard workflow', 'basic procedure'\n- **Foundational Concepts**: Look for 'overview', 'fundamentals', 'key concepts'\n- **Phase Transitions**: Find 'next steps', 'moving from X to Y', 'when to proceed'\n\n## Query Optimization for Basic Guidance:\n- **Category-First Approach**: Always include the category information from the user query in at least one sub-query\n- **Beginner-Friendly Terms**: Use simple, clear terminology over advanced jargon\n- **Process-Oriented**: Focus on 'how to', 'steps to', 'process for'\n- **Phase Identification**: Include specific phase names in queries\n- **Sequential Learning**: Structure searches from basic to intermediate concepts\n- **Common Questions**: Target frequent onboarding pain points and basic uncertainties\n\n## Search Query Structure:\n- Start with a category-specific guideline query based on the user's query category\n- Include both conceptual ('what is') and procedural ('how to') searches\n- Target documentation that explains fundamentals before diving into details\n- Focus on getting started guides, overviews, and introductory materials\n- Search for phase-specific checklists, requirements, and basic workflows\n\nGenerate 4-6 focused sub-queries that efficiently locate basic phase documentation and foundational guidance relevant to the user's onboarding question. ENSURE that at least one sub-query specifically targets category guidelines based on the category information present in the user query.",
-		IntentionPromptTemplate: "azure_sdk_onboarding_intention.md",
+		AgenticSearchPrompt:     "azure_sdk_onboarding/agentic_search.md",
+		IntentionPromptTemplate: "azure_sdk_onboarding/intention.md",
 	},
 }
 

tools/sdk-ai-bots/azure-sdk-qa-bot-backend/service/agent/service.go

@@ -92,7 +92,7 @@ func (s *CompletionService) ChatCompletion(ctx context.Context, req *model.Compl
 		// Skip RAG workflow for non-technical messages
 		log.Printf("Skipping RAG workflow - non-technical message detected")
 		chunks = []string{}
-		promptTemplate = "non_technical_question_prompt.md"
+		promptTemplate = "common/non_technical_question.md"
 	} else {
 		// Run agentic search and knowledge search in parallel, then merge results
 		chunks, err = s.runParallelSearchAndMergeResults(ctx, req, query)
@@ -110,7 +110,9 @@ func (s *CompletionService) ChatCompletion(ctx context.Context, req *model.Compl
 	}
 
 	// 5. Get answer from LLM
-	llmMessages = append(llmMessages, &azopenai.ChatRequestSystemMessage{Content: azopenai.NewChatRequestSystemMessageContent(prompt)})
+	llmMessages = append([]azopenai.ChatRequestMessageClassification{
+		&azopenai.ChatRequestSystemMessage{Content: azopenai.NewChatRequestSystemMessageContent(prompt)},
+	}, llmMessages...)
 	result, err := s.getLLMResult(llmMessages, tenantConfig.PromptTemplate)
 	if err != nil {
 		log.Printf("LLM request failed: %v", err)
@@ -136,7 +138,9 @@ func (s *CompletionService) RecognizeIntention(promptTemplate string, messages [
 		return nil, err
 	}
 
-	messages = append(messages, &azopenai.ChatRequestSystemMessage{Content: azopenai.NewChatRequestSystemMessageContent(promptStr)})
+	messages = append([]azopenai.ChatRequestMessageClassification{
+		&azopenai.ChatRequestSystemMessage{Content: azopenai.NewChatRequestSystemMessageContent(promptStr)},
+	}, messages...)
 
 	resp, err := config.OpenAIClient.GetChatCompletions(context.TODO(), azopenai.ChatCompletionsOptions{
 		Messages:       messages,
@@ -474,7 +478,11 @@ func (s *CompletionService) agenticSearch(ctx context.Context, query string, req
 		log.Printf("ERROR: %s", err)
 		return nil, err
 	}
-	log.Printf("Agentic search sub queries: %+v", resp.Activity)
+	for _, activity := range resp.Activity {
+		if activity.Type == "AzureSearchQuery" {
+			log.Printf("Agentic search sub query: %s", activity.Query.Search)
+		}
+	}
 	if resp.Response == nil {
 		return nil, nil
 	}

tools/sdk-ai-bots/azure-sdk-qa-bot-backend/service/prompt/azure_sdk_onboarding/agentic_search.md

@@ -0,0 +1,45 @@
+# SYSTEM ROLE
+You are an Azure SDK onboarding query analyzer. Break down the user's question into 4-6 specific search queries to find onboarding guidance.
+
+Note: The user query contains category information (e.g., "category: sdk-onboard") that indicates the onboarding phase.
+
+# Your Task
+Analyze the question and generate focused sub-queries based on the category:
+
+## Onboarding Categories (from intention recognition)
+- **sdk-onboard**: Service registration, prerequisites, initial setup
+- **api-design**: REST API specs, TypeSpec vs OpenAPI, design guidelines
+- **sdk-develop**: SDK generation, client library patterns, coding standards
+- **sdk-release**: Release planning, versioning, GA criteria, publication
+
+## Sub-Query Generation Strategy
+1. **Category guidelines** - ALWAYS search for "[category] guidelines" or "[category] best practices" based on the category in the query
+2. **Prerequisites** - What's needed before starting (requirements, setup)
+3. **Step-by-step process** - How to complete the phase (workflow, procedures)
+4. **Key concepts** - Core concepts and fundamentals for this phase
+5. **Common issues** - Frequent questions or problems in this phase
+6. **Next steps** - What comes after completing this phase
+
+# Sub-Query Guidelines
+- Extract the category from the user query and use it in at least one sub-query
+- Use simple, beginner-friendly terminology
+- Focus on "how to" and "what is" questions
+- Include phase-specific terms (onboarding, design, development, release)
+- Make queries searchable (use terms from documentation)
+
+# Examples
+Question: "category: sdk-onboard - What are the prerequisites for onboarding a new Azure service?"
+Sub-queries:
+- "Azure SDK service onboarding guidelines"
+- "Prerequisites for Azure service onboarding"
+- "How to register new Azure service for SDK"
+- "Service onboarding checklist"
+
+Question: "category: api-design - Should I use TypeSpec or OpenAPI for my REST API?"
+Sub-queries:
+- "API design guidelines Azure SDK"
+- "TypeSpec vs OpenAPI for Azure services"
+- "When to use TypeSpec for Azure API"
+- "Azure REST API specification best practices"
+
+Generate 4-6 sub-queries that will help answer the onboarding question. Include the category information in at least one sub-query.

tools/sdk-ai-bots/azure-sdk-qa-bot-backend/service/prompt/prompt_template/azure_sdk_onboarding_intention.md renamed to tools/sdk-ai-bots/azure-sdk-qa-bot-backend/service/prompt/azure_sdk_onboarding/intention.md

@@ -1,14 +1,15 @@
-## Role Description
+# Role Description
 You are an intent recognition assistant specialized in analyzing Azure SDK onboarding questions and determining their context, scope, and categorization.
 
-## Task Description
+# Task Description
 Your task is to:
 1. Rewrite any follow-up questions as standalone questions, maintaining the original context and language
 2. Categorize the question's intent based on its content, scope, and the onboarding phase it relates to
 3. Identify whether the question involves differences between TypeSpec and OpenAPI/Swagger workflows
 4. Check if the question aligns with proper onboarding sequence, if not, fallback question to onboading process
+5. Analyze if the latest user message needs RAG processing
 
-## Intent Categories
+# Intent Categories
 The question must be classified into one of these categories:
 
 - **sdk-onboard**: Questions about Azure service prerequisites and onboarding requirements, such as:
@@ -44,16 +45,23 @@ The question must be classified into one of these categories:
     - Are unclear or ambiguous about the Azure SDK context
     - Don't relate directly to Azure SDK onboarding processes
 
-## Response Format
+## Need RAG Processing
+  - Greetings/Thanks message, should be false
+  - Suggestions/Questions about Azure SDK Q&A bot, should be false
+  - Announcements or system message, should be false
+  - Technical questions, should be true
+  - For all other cases not covered above, should be true
+
+# Response Format
 Respond with a JSON object using this structure (no markdown formatting needed):
 {
   "question": string,    // The rewritten standalone question
   "category": string,    // Must be one of the categories: sdk-onboard, api-design, sdk-develop, and sdk-release
   "spec_type": string,   // user's service specification language: TypeSpec or OpenAPI or unknown
-  "needs_rag_processing": boolean    // Whether to invoke RAG workflow (true for technical questions, false for greetings/announcements)
+  "needs_rag_processing": boolean    // Whether to invoke RAG workflow, default is true
 }
 
-## Examples
+# Examples
 
 Original: "How do I get my service ready for SDK onboarding?"
 Response: