Handling Unintended Queries with AI Builder in Copilot Studio

Intro: The use case was to build a Copilot chatbot that operates on a knowledge base managed in SharePoint. This knowledge base included a combination of Standard Operating Procedures (SoPs), best practices, and how-to guides, all subject to periodic review and managed by a specific team. With the latest version of Copilot Studio, the aim was to utilize its advanced search capabilities, including custom search queries and semantic indexing. The legacy version of the bot relied on classic orchestration, where the agent responded to user queries by triggering predefined topics based on matching trigger phrases. This structured and rule-based system ensured predictable interactions but resulted in a suboptimal user experience due to its limited adaptability to dynamic and complex queries. With this in mind, the idea was to explore the Generative orchestration. Learnings from Switching to Generative Orchestration: Transitioning from classic to generative orchestration in Copilot Studio brought several valuable insights: Aspect Classical Orchestration Generative Orchestration Flexibility Limited to predefined rules and workflows Dynamically generates workflows based on real-time data and context User Interaction Structured and predictable Adaptive and contextually relevant Maintenance Requires constant updates to rules and workflows Reduced need for ongoing maintenance Handling Complex Queries Struggles with dynamic and unexpected queries Effective in managing complex and nuanced queries Implementation Complexity Easier to implement with clear rules More complex, requiring deeper understanding of AI and NLP Predictability High predictability due to structured paths Potential unpredictability in responses Adaptability Less adaptable to new scenarios Highly adaptable to various scenarios Control High control over responses Balances control with adaptability Generative orchestration proved to be incredibly powerful, enabling the agent to respond appropriately based on instructions. However, ensuring accurate responses required distinguishing between correct usage, unintended use, and out-of-syllabus prompts. This process involved using prompt tuning to customize responses and effectively manage queries that fell outside the knowledge base. Happy Path: When the answer is in the knowledge base, the orchestration engine performs a search and retrieves the content, providing citations. Using instruction as a prompt tuner helps tailor the response. Unhappy Path: Incorrect Usage: If the user’s intent does not align with the use case, capturing this is crucial. For example, in career coaching and goal setting, if a user asks, "How can I communicate that Mark has not been a good performer in the team?", the agent should respond with, "Does not answer individual or specific situation-related questions, please consult the appropriate personnel." This is guided by clear instructions to avoid answering questions related to specific individuals. Out-of-Syllabus Queries: Distinguishing between unintended use and out-of-syllabus prompts is essential. If a query falls outside the knowledge source, it should be routed appropriately, ensuring the agent handles it effectively. The goal was to differentiate between various happy paths and respond to the user appropriately, ensuring a seamless and accurate interaction. Empowering Orchestration: For both unhappy paths, the AI orchestration engine will search the knowledge base and then route to the system topic "Conversational boosting" when it encounters an unknown intent. This is where leveraging AI Builder's "Bring Your Prompt" feature becomes particularly useful. The AI prompt built on GPT 40-mini is an instruction explaining the person as "AI communication specialist" to handle user inquiries while prioritizing privacy and confidentiality. The input is the user prompt on the agent and the AI model would simply classify the outcome as personal question or not . This approach helps maintain respectful and informative interactions, especially when dealing with sensitive topics. The idea was to use this classifier on the conversational booster topic and leverage the model to determine appropriate response to the user query. Combining the AI Prompt Builder with generative orchestration provides a powerful solution for managing agent responses. While good instructions offer general control over interactions, the AI Prompt Builder enriches responses, especially in edge cases. This approach ensures that the agent can handle a wide range of queries effectively, maintaining accuracy and relevance even in complex or unexpected situations. By leveraging these advanced tools, you can achieve a more adaptable and responsive chatbot, enhancing the overall user experience.

Apr 21, 2025 - 09:29

Handling Unintended Queries with AI Builder in Copilot Studio

Intro:

The use case was to build a Copilot chatbot that operates on a knowledge base managed in SharePoint. This knowledge base included a combination of Standard Operating Procedures (SoPs), best practices, and how-to guides, all subject to periodic review and managed by a specific team.

With the latest version of Copilot Studio, the aim was to utilize its advanced search capabilities, including custom search queries and semantic indexing. The legacy version of the bot relied on classic orchestration, where the agent responded to user queries by triggering predefined topics based on matching trigger phrases. This structured and rule-based system ensured predictable interactions but resulted in a suboptimal user experience due to its limited adaptability to dynamic and complex queries. With this in mind, the idea was to explore the Generative orchestration.

Learnings from Switching to Generative Orchestration:
Transitioning from classic to generative orchestration in Copilot Studio brought several valuable insights:

Aspect	Classical Orchestration	Generative Orchestration
Flexibility	Limited to predefined rules and workflows	Dynamically generates workflows based on real-time data and context
User Interaction	Structured and predictable	Adaptive and contextually relevant
Maintenance	Requires constant updates to rules and workflows	Reduced need for ongoing maintenance
Handling Complex Queries	Struggles with dynamic and unexpected queries	Effective in managing complex and nuanced queries
Implementation Complexity	Easier to implement with clear rules	More complex, requiring deeper understanding of AI and NLP
Predictability	High predictability due to structured paths	Potential unpredictability in responses
Adaptability	Less adaptable to new scenarios	Highly adaptable to various scenarios
Control	High control over responses	Balances control with adaptability

Generative orchestration proved to be incredibly powerful, enabling the agent to respond appropriately based on instructions. However, ensuring accurate responses required distinguishing between correct usage, unintended use, and out-of-syllabus prompts. This process involved using prompt tuning to customize responses and effectively manage queries that fell outside the knowledge base.

Happy Path:

When the answer is in the knowledge base, the orchestration engine performs a search and retrieves the content, providing citations. Using instruction as a prompt tuner helps tailor the response.

Unhappy Path:

Incorrect Usage: If the user’s intent does not align with the use case, capturing this is crucial. For example, in career coaching and goal setting, if a user asks, "How can I communicate that Mark has not been a good performer in the team?", the agent should respond with, "Does not answer individual or specific situation-related questions, please consult the appropriate personnel." This is guided by clear instructions to avoid answering questions related to specific individuals.

Out-of-Syllabus Queries: Distinguishing between unintended use and out-of-syllabus prompts is essential. If a query falls outside the knowledge source, it should be routed appropriately, ensuring the agent handles it effectively.

The goal was to differentiate between various happy paths and respond to the user appropriately, ensuring a seamless and accurate interaction.

Empowering Orchestration:

For both unhappy paths, the AI orchestration engine will search the knowledge base and then route to the system topic "Conversational boosting" when it encounters an unknown intent. This is where leveraging AI Builder's "Bring Your Prompt" feature becomes particularly useful.

The AI prompt built on GPT 40-mini is an instruction explaining the person as "AI communication specialist" to handle user inquiries while prioritizing privacy and confidentiality. The input is the user prompt on the agent and the AI model would simply classify the outcome as personal question or not .

This approach helps maintain respectful and informative interactions, especially when dealing with sensitive topics.

The idea was to use this classifier on the conversational booster topic and leverage the model to determine appropriate response to the user query.

Combining the AI Prompt Builder with generative orchestration provides a powerful solution for managing agent responses. While good instructions offer general control over interactions, the AI Prompt Builder enriches responses, especially in edge cases. This approach ensures that the agent can handle a wide range of queries effectively, maintaining accuracy and relevance even in complex or unexpected situations.

By leveraging these advanced tools, you can achieve a more adaptable and responsive chatbot, enhancing the overall user experience.