Skip to content

Commit d1648bf

Browse files
Copilotmitchdenny
Copilot
and
mitchdenny
authored
Update test-scenario workflow to create issues instead of agent tasks (#12150)
* Initial plan * Update test-scenario workflow to create issues instead of agent tasks Co-authored-by: mitchdenny <[email protected]> * Fix YAML syntax issues in test-scenario workflow Co-authored-by: mitchdenny <[email protected]> * Fix format string vulnerability in issue body generation Co-authored-by: mitchdenny <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: mitchdenny <[email protected]>
1 parent 7f30741 commit d1648bf

File tree

2 files changed

+124
-25
lines changed

2 files changed

+124
-25
lines changed

‎.github/workflows/test-scenario.yml‎

Lines changed: 120 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -85,52 +85,149 @@ jobs:
8585
gh --version
8686
fi
8787
88-
- name: Create agent task
89-
id: create_agent_task
88+
- name: Create issue and assign to copilot
89+
id: create_issue
9090
run: |
91-
echo "Creating agent task..."
91+
echo "Creating issue in aspire-playground repository..."
9292
PROMPT_FILE="${{ steps.check_prompt.outputs.prompt_file }}"
93+
SCENARIO_NAME="${{ steps.parse_scenario.outputs.scenario_name }}"
94+
SOURCE_PR_URL="${{ github.event.issue.html_url }}"
95+
SOURCE_PR_NUMBER="${{ github.event.issue.number }}"
96+
SOURCE_REPO="${{ github.repository }}"
9397
9498
# Auth using the token
9599
gh auth login --with-token <<< "$GH_PLAYGROUND_TOKEN"
96100
97-
# Create the agent task using stdin and capture the output
98-
OUTPUT=$(cat "$PROMPT_FILE" | gh agent-task create \
101+
# Build the issue body with context from the source PR and the prompt
102+
ISSUE_TITLE="Test Scenario: ${SCENARIO_NAME}"
103+
104+
# Read prompt content first
105+
PROMPT_CONTENT=$(cat "$PROMPT_FILE")
106+
107+
# Build issue body using printf with proper format strings to avoid injection
108+
printf -v ISSUE_BODY '%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s' \
109+
"## Test Scenario Request" \
110+
"" \
111+
"**Scenario:** ${SCENARIO_NAME}" \
112+
"**Source PR:** ${SOURCE_PR_URL}" \
113+
"**Source Repository:** ${SOURCE_REPO}" \
114+
"" \
115+
"---" \
116+
"" \
117+
"$PROMPT_CONTENT" \
118+
"" \
119+
"---" \
120+
"" \
121+
"This issue was created automatically from PR #${SOURCE_PR_NUMBER} in ${SOURCE_REPO}."
122+
123+
# Create the issue and assign to copilot
124+
echo "Creating issue with title: $ISSUE_TITLE"
125+
ISSUE_OUTPUT=$(gh issue create \
99126
--repo "${REPO_OWNER}/${REPO_NAME}" \
100-
-F - \
127+
--title "$ISSUE_TITLE" \
128+
--body "$ISSUE_BODY" \
129+
--assignee "copilot" \
101130
2>&1)
102131
103-
echo "Agent task output:"
104-
echo "$OUTPUT"
132+
echo "Issue creation output:"
133+
echo "$ISSUE_OUTPUT"
105134
106-
# Extract the PR URL from the output
107-
PR_URL=$(echo "$OUTPUT" | \
108-
grep -oP 'https://github.com/[^/]+/[^/]+/pull/\d+' | head -1)
135+
# Extract the issue URL from the output
136+
ISSUE_URL=$(echo "$ISSUE_OUTPUT" | \
137+
grep -oP 'https://github.com/[^/]+/[^/]+/issues/\d+' | head -1)
109138
110-
if [ -z "$PR_URL" ]; then
111-
echo "Warning: Could not extract PR URL from output"
112-
echo "pr_url=" >> $GITHUB_OUTPUT
139+
if [ -z "$ISSUE_URL" ]; then
140+
echo "Error: Could not extract issue URL from output"
141+
exit 1
142+
fi
143+
144+
echo "Successfully created issue: $ISSUE_URL"
145+
echo "issue_url=$ISSUE_URL" >> $GITHUB_OUTPUT
146+
147+
# Extract issue number for later use
148+
ISSUE_NUMBER=$(echo "$ISSUE_URL" | grep -oP '/issues/\K\d+')
149+
echo "issue_number=$ISSUE_NUMBER" >> $GITHUB_OUTPUT
150+
151+
- name: Wait for agent PR to be created
152+
id: wait_for_pr
153+
run: |
154+
echo "Waiting for GitHub Copilot agent to create a PR..."
155+
ISSUE_NUMBER="${{ steps.create_issue.outputs.issue_number }}"
156+
MAX_ATTEMPTS=30
157+
SLEEP_SECONDS=10
158+
ATTEMPT=0
159+
FOUND_PR=""
160+
161+
# Auth using the token
162+
gh auth login --with-token <<< "$GH_PLAYGROUND_TOKEN"
163+
164+
while [ $ATTEMPT -lt $MAX_ATTEMPTS ]; do
165+
ATTEMPT=$((ATTEMPT + 1))
166+
echo "Attempt $ATTEMPT of $MAX_ATTEMPTS..."
167+
168+
# Try to find linked PRs by searching for PRs that reference this issue
169+
LINKED_PRS=$(gh pr list \
170+
--repo "${REPO_OWNER}/${REPO_NAME}" \
171+
--search "linked:issue-${ISSUE_NUMBER}" \
172+
--json number,url,title \
173+
--jq '.[0].url' 2>/dev/null || echo "")
174+
175+
if [ -n "$LINKED_PRS" ] && [ "$LINKED_PRS" != "null" ] && [ "$LINKED_PRS" != "" ]; then
176+
echo "Found linked PR: $LINKED_PRS"
177+
FOUND_PR="$LINKED_PRS"
178+
break
179+
fi
180+
181+
# Also check for PRs created by the copilot bot recently
182+
COPILOT_PRS=$(gh pr list \
183+
--repo "${REPO_OWNER}/${REPO_NAME}" \
184+
--author "copilot" \
185+
--limit 5 \
186+
--json number,url,title,createdAt \
187+
--jq 'sort_by(.createdAt) | reverse | .[0].url' 2>/dev/null || echo "")
188+
189+
if [ -n "$COPILOT_PRS" ] && [ "$COPILOT_PRS" != "null" ] && [ "$COPILOT_PRS" != "" ]; then
190+
echo "Found recent copilot PR: $COPILOT_PRS"
191+
FOUND_PR="$COPILOT_PRS"
192+
break
193+
fi
194+
195+
if [ $ATTEMPT -lt $MAX_ATTEMPTS ]; then
196+
echo "No PR found yet, waiting ${SLEEP_SECONDS} seconds..."
197+
sleep $SLEEP_SECONDS
198+
fi
199+
done
200+
201+
if [ -n "$FOUND_PR" ]; then
202+
echo "pr_url=$FOUND_PR" >> $GITHUB_OUTPUT
113203
else
114-
echo "Successfully created agent task with PR: $PR_URL"
115-
echo "pr_url=$PR_URL" >> $GITHUB_OUTPUT
204+
echo "Warning: No PR found after $MAX_ATTEMPTS attempts"
205+
echo "The agent may still be working on it."
206+
echo "pr_url=" >> $GITHUB_OUTPUT
116207
fi
117208
118-
- name: Comment on PR with agent task link
119-
if: steps.create_agent_task.outputs.pr_url != ''
209+
- name: Comment on PR with issue and agent PR links
120210
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
121211
with:
122212
github-token: ${{ secrets.GITHUB_TOKEN }}
123213
script: |
124-
const prUrl = '${{ steps.create_agent_task.outputs.pr_url }}';
214+
const issueUrl = '${{ steps.create_issue.outputs.issue_url }}';
215+
const prUrl = '${{ steps.wait_for_pr.outputs.pr_url }}';
125216
const scenarioName = '${{ steps.parse_scenario.outputs.scenario_name }}';
126-
const comment = `🤖 **AI Agent Task Created**
217+
218+
let comment = `🤖 **AI Agent Task Created**
127219
128220
Scenario: **${scenarioName}**
129221
130-
An AI agent has been triggered to execute this scenario.
131-
You can track the progress here:
222+
An AI agent has been assigned to execute this scenario.
223+
224+
📝 **Issue:** ${issueUrl}`;
132225
133-
${prUrl}`;
226+
if (prUrl) {
227+
comment += `\n🔀 **Agent PR:** ${prUrl}`;
228+
} else {
229+
comment += `\n\n⏳ The agent is working on this task. The PR will be linked in the issue once created.`;
230+
}
134231
135232
await github.rest.issues.createComment({
136233
issue_number: context.issue.number,

‎tests/agent-scenarios/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,8 +53,10 @@ The workflow will:
5353
1. Validate the scenario name format
5454
2. Look for `tests/agent-scenarios/scenario-name/prompt.md`
5555
3. Read the prompt from the file
56-
4. Create an agent task using the GitHub CLI
57-
5. Post a comment with the scenario name and link to the agent's PR
56+
4. Create an issue in the `aspire-playground` repository with the prompt and PR context
57+
5. Assign the issue to the GitHub Copilot agent
58+
6. Wait for the agent to create a PR (up to 5 minutes)
59+
7. Post a comment with links to both the issue and the agent's PR (if available)
5860

5961
## Creating a New Scenario
6062

0 commit comments

Comments
 (0)