Bug Report – Test Cases Failing in Copilot Studio Kit

[gondor-netizen]

Test Cases Failing with "AI-Generated Assessment to Review" in Copilot Studio Kit
Description:
We are encountering issues with test cases failing during Test Runs in Copilot Studio Kit. The following observations have been made:

1. Test Failure Behavior:
   • Test cases with 'Generative Answers' selected and 'Answered' State often fail, resulting in a "2 - Failure" status.
   • The Result Reason field displays static text: "AI-generated Assessment of the Response. Please Review," but no detailed assessment is accessible.
2. Generative Analysis Flow:
   • The Generative Analysis Flow executed during Test Runs provides only a cursory instruction to determine if the expected response is related to the question, assigning a simple status (Success, Failure, Unknown).
   • There is no mechanism to access or review the AI-generated assessment mentioned in the Result Reason field.
3. Dataverse Table Limitations:
   • The Agent Test Result Table in Dataverse does not contain additional columns or data to provide insight into the AI-generated assessment or the reason for test failures.
     Request:
     We need assistance with the following:

• Investigating why test cases are failing and whether detailed AI-generated assessments can be accessed or generated.
• Identifying whether the Generative Analysis Flow or Dataverse Table configurations are contributing to the issue.
• Providing guidance on resolving the "AI-generated Assessment to Review" issue and improving test result transparency.
  Priority: High

[SiddharthPoweCat]

Step-by-Step Solution

1. Verify Flow Status:

   • Check if the "Test Automation | Analyze Generated Answers with AI Builder" flow runs successfully. This process typically takes 7–10 minutes.

2. AI Builder Requirements:

   • Ensure sufficient AI Builder credits in your tenant.
   • Verify AI Builder is enabled in environment settings.
   • Confirm your environment supports AI Builder scenarios (not all regions do).

3. Agent Configuration:

   • Ensure the agent is published in Copilot Studio.
   • Enable Orchestration under agent settings to allow generative AI responses.

4. App Insights Permissions:

   • Follow the ENABLE-APPINSIGHTS.md documentation.
   • Verify the user has appropriate roles for App Insights access.
   • Check connection authentication in Power Automate flows.

5. Environment Support:

   • Confirm your environment type supports AI Builder. Some environments may show "This scenario is not supported in this environment."

6. Manual AI Builder Analysis Trigger:

   • Navigate to the Test Run record in Copilot Studio Kit.
   • Click "Rerun" → "Analyze with AI Builder" and confirm the dialog to execute AI Builder model analysis.

Additional Context

• Generative Answers test case is not designed to validate exact matches; responses can vary each time. This test type is best for evaluating whether a question is answered or not, along with a few other predefined outcome types.
• Ensure your environment supports transcript generation and data population for KPIs. Only Sandbox and Production environments support automatic transcript generation.
• Proper agent setup is required for data population; cross-environment data sync is not sufficient.
• If you encounter access errors, check user roles and permissions for App Insights and Dataverse.

Reference Links

• Test type -> Generative Answers -> Result is 'pending' and Result Reason is 'Pending analysis with AI Builder' #264
• Testing with generative AI not working #86
• Generative Answers Test Failing all the time #215

[SiddharthPoweCat]

Hi @gondor-netizen , could you please provide the update on the issue, whether the above steps help you identify and resolve the issue? Thanks

[gondor-netizen]

Hello, Please see below for responses. Verify Flow Status: Check if the "Test Automation | Analyze Generated Answers with AI Builder" flow runs successfully. This process typically takes 7–10 minutes. Response: We are running Test Runs with a successful completion status on the AI Builder Power Automate Flow AI Builder Requirements: Ensure sufficient AI Builder credits in your tenant. Verify AI Builder is enabled in environment settings. Confirm your environment supports AI Builder scenarios (not all regions do). Response: We are currently consuming AI Builder credits with our Test runs and getting responses from the Agent Agent Configuration: Ensure the agent is published in Copilot Studio. Enable Orchestration under agent settings to allow generative AI responses. Response: The Agent has been published recently and the Agent Configuration is available to be selected for our test runs in testing. App Insights Permissions: Follow the ENABLE-APPINSIGHTS.md documentation. Verify the user has appropriate roles for App Insights access. Check connection authentication in Power Automate flows. Response: We are not using App Insights Analysis and is currently disabled for our Configurations. This shouldn't be a requirement for the PA Flow and Prompt that is doing generative analysis as there are no references to the App Insights PA Flow. Environment Support: Confirm your environment type supports AI Builder. Some environments may show "This scenario is not supported in this environment." Response: We are currently using AI Builder for our tests successfully. Manual AI Builder Analysis Trigger: Navigate to the Test Run record in Copilot Studio Kit. Click "Rerun" $B"*(J "Analyze with AI Builder" and confirm the dialog to execute AI Builder model analysis. Response: We are successfully executing the Analyze with AI Builder the Generative Answers Analysis Prompt is returning a failure code based on its Analysis What other steps do you recommend?

…

________________________________ From: SiddharthPoweCat ***@***.***> Sent: Friday, October 31, 2025 12:54:09 AM To: microsoft/Power-CAT-Copilot-Studio-Kit ***@***.***> Cc: gondor-netizen ***@***.***>; Author ***@***.***> Subject: Re: [microsoft/Power-CAT-Copilot-Studio-Kit] Bug Report – Test Cases Failing in Copilot Studio Kit (Issue #433) [https://avatars.githubusercontent.com/u/199201783?s=20&v=4]SiddharthPoweCat left a comment (microsoft/Power-CAT-Copilot-Studio-Kit#433)<#433 (comment)> Step-by-Step Solution 1. Verify Flow Status: * Check if the "Test Automation | Analyze Generated Answers with AI Builder" flow runs successfully. This process typically takes 7–10 minutes. 2. AI Builder Requirements: * Ensure sufficient AI Builder credits in your tenant. * Verify AI Builder is enabled in environment settings. * Confirm your environment supports AI Builder scenarios (not all regions do). 3. Agent Configuration: * Ensure the agent is published in Copilot Studio. * Enable Orchestration under agent settings to allow generative AI responses. 4. App Insights Permissions: * Follow the ENABLE-APPINSIGHTS.md documentation. * Verify the user has appropriate roles for App Insights access. * Check connection authentication in Power Automate flows. 5. Environment Support: * Confirm your environment type supports AI Builder. Some environments may show "This scenario is not supported in this environment." 6. Manual AI Builder Analysis Trigger: * Navigate to the Test Run record in Copilot Studio Kit. * Click "Rerun" → "Analyze with AI Builder" and confirm the dialog to execute AI Builder model analysis. Additional Context * Generative Answers test case is not designed to validate exact matches; responses can vary each time. This test type is best for evaluating whether a question is answered or not, along with a few other predefined outcome types. * Ensure your environment supports transcript generation and data population for KPIs. Only Sandbox and Production environments support automatic transcript generation. * Proper agent setup is required for data population; cross-environment data sync is not sufficient. * If you encounter access errors, check user roles and permissions for App Insights and Dataverse. Reference Links * #264<#264> * #86<#86> * #215<#215> — Reply to this email directly, view it on GitHub<#433 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BZPXWQEZ2RFKFZIIXWQTWLD32LTPDAVCNFSM6AAAAACKVQM7ZWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTINZRGMZTKMZXGY>. You are receiving this because you authored the thread.