[evals] Add a starter set of Foundry scenario evals - #9415
Open
JeffreyCA wants to merge 5 commits into
Open
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 21 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
JeffreyCA
force-pushed
the
jeffreyca/foundry-vally-scaffold
branch
3 times, most recently
from
August 3, 2026 21:55
63987a1 to
9daec45
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds offline Vally scenario evaluations for Foundry/AI extensions using the official Microsoft Foundry skill and a separate non-blocking CI workflow.
Changes:
- Adds Foundry skill, authoring, and CLI evals plus an A/B experiment.
- Adds fixtures, skill fetching, provenance capture, and local documentation.
- Extends the shared Vally action with optional azd and extension setup.
Show a summary per file
| File | Description |
|---|---|
.github/actionlint.yaml |
Allows the Copilot permission in the new workflow. |
.github/actions/vally-eval/action.yml |
Adds optional azd, extension, and skill setup. |
.github/workflows/vally-foundry-ci.yml |
Runs Foundry evals in a non-blocking matrix. |
cli/azd/test/evals/.gitignore |
Excludes fetched external skills. |
cli/azd/test/evals/README.md |
Documents Foundry evals and local usage. |
cli/azd/test/evals/eval-foundry-authoring.yaml |
Evaluates Foundry configuration authoring. |
cli/azd/test/evals/eval-foundry-cli.yaml |
Evaluates read-only CLI discovery and diagnosis. |
cli/azd/test/evals/eval-foundry-skill.yaml |
Evaluates skill routing and azd guidance. |
cli/azd/test/evals/eval-foundry.experiments.yaml |
Defines skill-on/off model experiments. |
cli/azd/test/evals/fixtures/foundry-agent-project/azure.yaml |
Provides a healthy Foundry project fixture. |
cli/azd/test/evals/fixtures/foundry-agent-project/main.py |
Provides the hosted-agent application fixture. |
cli/azd/test/evals/fixtures/foundry-broken-project/azure.yaml |
Provides a diagnostic failure fixture. |
cli/azd/test/evals/fixtures/foundry-legacy-agent/azure.yaml |
Provides a legacy configuration fixture. |
cli/azd/test/evals/fixtures/foundry-legacy-agent/main.py |
Provides the legacy agent application fixture. |
cli/azd/test/evals/package-lock.json |
Locks Vally 0.11 dependencies. |
cli/azd/test/evals/package.json |
Adds Foundry scripts and upgrades Vally. |
cli/azd/test/evals/scripts/fetch-foundry-skill.sh |
Fetches the official skill and records provenance. |
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Suppressed comments (2)
cli/azd/test/evals/scripts/fetch-foundry-skill.sh:29
- [azd-code-reviewer]
mktemp -dcan place the clone on a different filesystem from the workspace, in which casemvbecomes an interruptible copy and can leave the half-populated destination this code is intended to prevent. Create the temporary directory underdest_rootso the final move is a same-filesystem rename.
tmp_dir="$(mktemp -d)"
cli/azd/test/evals/eval-foundry-skill.yaml:85
- [azd-code-reviewer] The rubric requires every proposed azd command to carry the user-agent prefix, but this deterministic grader checks only
azd deploy. A response with a tagged deploy and a bareazd ai agent invokepasses both regex graders, and the eval-level 0.7 average can still pass even if the judge catches the violation. Add an equivalent tagged-invoke matcher or reject untagged azd commands in the command block.
- type: output-matches
name: "deploy command carries the skill user agent"
config:
pattern: 'AZURE_DEV_USER_AGENT=microsoft_foundry_skill[ \t]+(?:\\[ \t]*\n[ \t]*)?azd[ \t]+deploy'
- Files reviewed: 16/17 changed files
- Comments generated: 2
- Review effort level: Balanced
Adds a small seed of vally evals covering the Microsoft Foundry / AI extensions, alongside the existing core azd evals in cli/azd/test/evals. Rather than authoring a Foundry skill to test against, these evaluate the official microsoft-foundry skill from microsoft/azure-skills. It is a real shipping artifact with explicit, machine-checkable rules, so most grading is deterministic rather than LLM-judged. The skill is fetched into a gitignored directory instead of vendored, with the resolved commit recorded for traceability. Six stimuli across three evals, all offline (no Azure sign-in, no resources, no cost): - foundry-skill: does the skill route to the right sub-skill, and does the agent follow the skill's own mandatory azd rules? - foundry-authoring: migrating a legacy agent definition, and adding a second agent across several turns - foundry-cli: discovering and driving the real `azd ai` CLI without a skill loaded An experiment file A/Bs the skill off vs on across two models over the authoring eval. Scoring thresholds are 0.85 rather than the 0.5 used by the core evals: vally averages grader scores, so a low bar lets the one grader that is the point of a stimulus fail while the stimulus still passes. CI runs from a dedicated vally-foundry-ci.yml so the core azd eval gate stays independent of Foundry setup. The jobs are non-blocking while pass rates settle. The shared vally-eval action gains three opt-in inputs (install-azd, azd-extensions, fetch-skills), all defaulting off so existing jobs are unaffected. Also bumps vally 0.10.0 -> 0.11.0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
JeffreyCA
force-pushed
the
jeffreyca/foundry-vally-scaffold
branch
from
August 3, 2026 22:15
9daec45 to
56f43fe
Compare
Contributor
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Suppressed comments (2)
cli/azd/test/evals/eval-foundry-skill.yaml:85
- [azd-code-reviewer] This grader only verifies the deploy command, so an answer with a tagged deploy but an untagged
azd ai agent invokestill gets two of three graders in this stimulus; with a perfect routing stimulus, the eval averages to about 0.83 and passes the 0.7 threshold despite violating the rule this scenario is meant to catch. Require both tagged commands in the deterministic grader (in deploy-before-invoke order) so this regression also fails the eval rollup.
- type: output-matches
name: "deploy command carries the skill user agent"
config:
pattern: 'AZURE_DEV_USER_AGENT=microsoft_foundry_skill[ \t]+(?:\\[ \t]*\n[ \t]*)?azd[ \t]+deploy'
.github/actions/vally-eval/action.yml:102
- [azd-code-reviewer] The existing setup steps are gated on
steps.copilot.outputs.enabled, but the new azd/extension/skill setup steps are not. On fork PRs, where the eval is intentionally disabled, these jobs still download azd, install the full Foundry bundle, or clone the external skill before the final rejection step. Add the Copilot-enabled check to each new step condition so fork runs fail fast without doing network-heavy setup.
- name: Configure azd environment
if: inputs.install-azd == 'true' || inputs.azd-extensions != ''
- Files reviewed: 16/17 changed files
- Comments generated: 0 new
- Review effort level: Balanced
Contributor
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Suppressed comments (1)
cli/azd/test/evals/.gitignore:4
- [azd-code-reviewer] This references a nonexistent
.shhelper; the added fetch script uses the.mjsextension.
# Skills fetched from other repos by scripts/fetch-foundry-skill.sh. Not vendored
- Files reviewed: 16/17 changed files
- Comments generated: 1
- Review effort level: Balanced
Contributor
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Suppressed comments (5)
cli/azd/test/evals/eval-foundry-cli.yaml:109
- [azd-code-reviewer] This second copy has the same false-positive behavior:
azd deploy --helpandazd ai agent init --helpare read-only, but the grader marks them as destructive. Exclude help-only invocations so an agent can inspect the suggested next command without failing the safety grader.
- name: "^(bash|powershell)$"
command: "azd (down|up|provision|deploy)(?!\\S)"
- name: "^(bash|powershell)$"
command: "azd ai agent (delete|init)(?!\\S)"
cli/azd/test/evals/eval-foundry-authoring.yaml:61
- [azd-code-reviewer] This grader is file-global, so moving
LOG_LEVEL: infoontoai-projector another service still reports that the environment survived even thoughdocs-agentbehavior changed. Scope the match to thedocs-agentblock, as the second stimulus already does forescalation-agent.
pattern: '(?i)LOG_LEVEL[\s\S]{0,30}?info'
cli/azd/test/evals/eval-foundry-cli.yaml:70
- [azd-code-reviewer] These disallowed regexes also reject read-only help calls such as
azd deploy --helporazd ai agent init -h. That creates a false failure in a stimulus explicitly asking the agent to inspect the CLI; exclude help flags within the matched shell command segment.
This issue also appears on line 106 of the same file.
- name: "^(bash|powershell)$"
command: "azd (down|up|provision|deploy)(?!\\S)"
- name: "^(bash|powershell)$"
command: "azd ai agent (delete|init)(?!\\S)"
cli/azd/test/evals/eval-foundry-skill.yaml:91
- [azd-code-reviewer] The persistence grader does not enforce its own rubric: it rejects
azd env setandexport, but still passes output that writesAZURE_DEV_USER_AGENTto.envorazure.yaml. The fetched Foundry guidance explicitly forbids both, so add matches for those write forms or a file-content/diff grader; otherwise the per-grader breakdown can show this rule as satisfied when it was violated.
- type: output-not-matches
name: "does not persist the user agent"
config:
pattern: '(?:azd env set|export)\s+AZURE_DEV_USER_AGENT'
cli/azd/test/evals/eval-foundry-cli.yaml:100
- [azd-code-reviewer] The
.*--helplookahead scans the rest of the entire shell tool call, not just thedoctorcommand. A valid call likeazd ai agent doctor; azd ai agent init --helptherefore fails this required grader even though the diagnostic ran. Bound the help exclusion to the current shell command segment.
command: "azd ai agent doctor(?!\\S)(?!.*--help)"
- Files reviewed: 16/17 changed files
- Comments generated: 0 new
- Review effort level: Balanced
JeffreyCA
marked this pull request as ready for review
August 3, 2026 23:04
JeffreyCA
requested review from
RickWinter,
danieljurek,
hemarina,
richardpark-msft,
tg-msft and
vhvb1989
as code owners
August 3, 2026 23:04
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
|
Azure Pipelines: Successfully started running 1 pipeline(s). 21 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
jongio
reviewed
Aug 4, 2026
jongio
left a comment
jongio
left a comment
Member
There was a problem hiding this comment.
Three things, none blocking.
Medium:
- The "does not persist" grader in
cli/azd/test/evals/eval-foundry-skill.yamlmatches prose, so an answer that correctly warns againstexport AZURE_DEV_USER_AGENTfails for saying the right thing. fetch-foundry-skill.mjscaptures git's stderr and throws it away, so a failed fetch in CI logs no reason.
Low:
azd extension installin the composite action skips--no-prompt, unlike every other CI install in this repo.
Contributor
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Suppressed comments (3)
cli/azd/test/evals/eval-foundry-cli.yaml:109
- [azd-code-reviewer] The diagnosis prompt asks for next steps, so an agent may reasonably inspect
azd ai agent init --helpafter doctor recommends init. This regex treats that read-only lookup as mutation and fails an otherwise compliant run. Exclude help-only invocations from both command patterns.
command: "azd (down|up|provision|deploy)(?!\\S)"
- name: "^(bash|powershell)$"
command: "azd ai agent (delete|init)(?!\\S)"
cli/azd/test/evals/eval-foundry-cli.yaml:70
- [azd-code-reviewer] These mutating-command patterns also reject read-only help calls such as
azd ai agent init --help. Since this stimulus explicitly asks the agent to discover the CLI surface, inspecting help for one of these commands should not fail the guardrail. Exclude help-only invocations, as the doctor grader below already does.
This issue also appears on line 107 of the same file.
command: "azd (down|up|provision|deploy)(?!\\S)"
- name: "^(bash|powershell)$"
command: "azd ai agent (delete|init)(?!\\S)"
cli/azd/test/evals/package.json:24
- [azd-code-reviewer] This shared dependency bump means the existing core Q&A and deploy workflow jobs also move from Vally 0.10 to 0.11, because
vally-ci.ymlrunsnpm cifrom this package. That contradicts the PR description's claim that existing jobs behave exactly as before. If 0.11 is required for these specs, update the description and explicitly account for the core-job upgrade; otherwise keep the existing dependency version.
"@microsoft/vally-cli": "^0.11.0"
- Files reviewed: 16/17 changed files
- Comments generated: 1
- Review effort level: Balanced
jongio
reviewed
Aug 4, 2026
jongio
left a comment
jongio
left a comment
Member
There was a problem hiding this comment.
Low:
cli/azd/test/evals/eval-foundry-skill.yaml:93: the anchor I suggested also drops commands written as a list item or after a$prompt, which is a common way an agent formats steps. Verified against vally's regex compilation. Details inline.
| - type: output-not-matches | ||
| name: "does not persist the user agent" | ||
| config: | ||
| pattern: '(?m)^\s*(?:azd env set|export)\s+AZURE_DEV_USER_AGENT' |
Member
There was a problem hiding this comment.
The anchor fixed the prose false positive, but ^\s* only allows whitespace before the command, so it now misses commands written as a list item or after a prompt marker. That's a common way an agent formats steps.
I ran both patterns through vally's own compilation path (createRegexpWithFlags in utils/checks.js strips the (?m) prefix into a JS flag) against the current text:
| input | current | with prefix allowance |
|---|---|---|
export AZURE_DEV_USER_AGENT=x |
catches | catches |
| same inside a fenced block | catches | catches |
- export AZURE_DEV_USER_AGENT=x |
misses | catches |
1. export AZURE_DEV_USER_AGENT=x |
misses | catches |
$ export AZURE_DEV_USER_AGENT=x |
misses | catches |
"set it inline, don't export ..." |
passes | passes |
"- Don't use azd env set ..." |
passes | passes |
Allowing an optional list or prompt marker keeps all three prose cases passing:
Suggested change
| pattern: '(?m)^\s*(?:azd env set|export)\s+AZURE_DEV_USER_AGENT' | |
| pattern: '(?m)^[ \t]*(?:[-*+]|\d+[.)]|\$|>)?[ \t]*(?:azd env set|export)[ \t]+AZURE_DEV_USER_AGENT' |
That also swaps \s for [ \t], matching the deploy grader above, so the class can't drift across a newline.
Contributor
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #9405
This PR adds a few vally evals (as a starting point) for the Foundry/AI extensions, next to the core azd ones already in
cli/azd/test/evals.Approach
Instead of writing our own Foundry skill to test against, these run against the official
microsoft-foundryskill.The skill is fetched into a gitignored dir. The fetch script writes the resolved commit to
.skill-refand CI uploads it with the results, so one can tell the version of the skill a given run saw.What's covered
There are six stimuli across three evals, all of them offline (no live resources provisioned).
foundry-skillfoundry-authoringfoundry-cliazd+ Foundry extensionsazd ai, and stay read-only when asked?There's also an experiment file that runs the authoring eval with the skill off vs on across two models.
Scoring
Note
Thresholds are
0.7, following the same idea as the existing evals here: set under where we score today so we hear about it if things get worse, rather than being red from day one. The eval-level pass/fail is a coarse rollup anyway, since vally averages graders into a stimulus score and then stimulus scores into the eval verdict, so the per-grader breakdown in the job summary is the part worth reading.One known gap sits under that line and is called out in the README:
azd ai agent doctordoesn't get discovered, since the agent readsazure.yamlinstead. It still answers correctly, so that grader is left strict on purpose rather than tuned away.CI
These run from their own
vally-foundry-ci.ymlso the core eval gate doesn't inherit Foundry setup (installing azd and extensions, pulling a skill from another repo).The shared
vally-evalaction picks up three new inputs,install-azd,azd-extensionsandfetch-skills, all defaulting off.vally-ci.ymlis untouched and the existing jobs behave exactly as before.Notes for reviewers
Two things that came out of running these against live models, worth knowing:
azure.yamlby hand and never ranazd ai agent doctor. Left that grader strict on purpose, since "agents don't find doctor" seems worth knowing.foundry-clideliberately doesn't overrideAZD_CONFIG_DIR. The extensions live there, so pointing it somewhere fresh makesazd aidisappear. Run locally it reads your real azd config. The stimuli are read-only and the destructive commands are indisallowedgraders, but that's a guardrail, not a sandbox. CI installs into a job-local dir instead.Related to #8524, which looked at interactive CLI scenario testing for the agents extension.
Testing
Checked every regex against a correct input and a wrong one that should fail it, mostly looking for false passes: debug logging set on the wrong service,
doctor --helpcounting as having run doctor, an untaggedazd deploysitting next to a tagged command.Ran the fixtures through the real CLI with
azd ai agent doctorto confirm the healthy and legacy ones load and the broken one still fails.Specs pass
vally lint, the experiment resolves its four variants under--dry-run, and actionlint/cspell/tsc/go test are clean.