test(azure.ai.agents): validate azure.yaml examples in docs

[glharper]

Fixes #9330

Problem

azure.yaml examples in this extension's docs aren't validated by anything, so they drift out of sync with the code and ship in a state where copying them fails. Two instances were found recently — both by hand, neither catchable by CI:

1. README.md — the "Migrating Legacy Agent Configuration" After: example omitted the required agent name. Copying it failed with name cannot be empty (fixed in fix: name the azure.yaml key in the RAI policy validation error #9328).
2. A Learn article documented rai_config.rai_policy_name on the azure.ai.agent service, which azd doesn't support at all. Because azd ignores unrecognized service properties, the agent deployed with no guardrail and no error (fixed in MicrosoftDocs/azure-ai-docs-pr#13567).

Approach

Implements approach 1 from the issue — validate doc snippets in a test — with one addition, because the two defects above are genuinely different classes.

TestDocExamplesAreValid walks the extension's markdown, extracts every fenced YAML block declaring an azure.ai.agent service, and applies two checks:

1. Resolver. The snippet must survive AgentDefinitionFromService, the same entry point azd uses at deploy time. This catches defect class 1.

2. Vocabulary. Every property must be declared in schemas/azure.ai.agent.json or parsed by azd core. This catches defect class 2, which the resolver cannot — UnmarshalStruct is non-strict, so an unsupported key passes cleanly. That leniency is correct at runtime (forward compatibility for other extensions), but our own docs shouldn't rely on it. The check is docs-only; no runtime behavior changes.

The vocabulary list is read from the schema that already ships with the extension, so it stays current without a second place to maintain.

The partial-snippet caveat

The issue flags that a naive validator false-positives on deliberately incomplete snippets: the three azure.ai.agent entries in docs/private-networking.md omit kind on purpose (the snippet is about the project's network: block; azd falls back to the on-disk agent.yaml).

Rather than inferring intent from the absence of kind — which would silently excuse a complete example that accidentally dropped it — snippets opt out explicitly:

<!-- azd:doc-example partial -->

Default is strict (must fully resolve); the marker only relaxes check 1. The failure message names the marker, so the escape hatch is discoverable at the moment it's needed.

Verification

Beyond the tests passing, I confirmed the test actually fails on each defect it claims to catch, by temporarily reintroducing them:

Injected defect	Result
Drop name from a README example (recreates #9328)	❌ README.md:43: ... - template.name not in valid format: name cannot be empty
Add rai_config: {rai_policy_name: ...} (recreates the Learn defect)	❌ README.md:43: service "my-agent" documents property "rai_config", which azd does not support.
Remove a partial marker	❌ docs/private-networking.md:11: ... did not resolve (is kind missing?)

Failures are reported per snippet as file:line/service, pointing straight at the fence to fix. All docs were restored afterwards — the diff touches no example content.

TestExtractYAMLExamples covers the extractor itself (language filtering, marker scoping, dedenting of indented fences, and CommonMark fence-length nesting), since every other assertion depends on it finding the right blocks.

Validation

From cli/azd/extensions/azure.ai.agents:

• go build ./... — clean
• go test ./internal/project/ -count=1 — ok (full package)
• gofmt -s -l ./internal — no output
• golangci-lint run ./internal/project/... — 0 issues
• cspell lint (Go, repo config) — 0 issues

Currently validates 6 snippets: 3 in README.md, 3 in docs/private-networking.md. New docs are picked up automatically.

Notes

• Test-only plus doc markers; no production code and no core cli/azd changes.
• No new dependencies (gopkg.in/yaml.v3 was already direct).
• No go.mod / go.sum changes.
• AGENTS.md documents the convention so future doc authors know the rule and the escape hatch.
• Approach 2 from the issue (make schemas/examples/*.azure.yaml the single source of truth) is deliberately not taken here — it's a larger docs restructure, and this is complementary to it.

[github-actions]

📋 Prioritization Note

Thanks for the contribution! The linked issue isn't in the current milestone yet.
Thank you for logging this issue; our team is reviewing it. If you need urgent prioritization, tag @RickWinter and @kristenwomack to let us know.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
21 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds automated validation to prevent azure.ai.agent documentation examples from drifting out of sync.

Changes:

• Extracts and validates fenced YAML examples.
• Adds explicit markers for intentionally partial snippets.
• Documents the validation convention.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
internal/project/doc_examples_test.go	Adds extraction and validation tests.
docs/private-networking.md	Marks three examples as partial.
AGENTS.md	Documents validation requirements and opt-out usage.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:278

• [azd-code-reviewer] These map fields do not apply core's actual shape rules as the helper claims. For example, core decodes docker into project.DockerProjectOptions, where path is a string, but this mirror accepts docker: {path: [Dockerfile]}; checkVocabulary then skips docker as core-owned, so the broken snippet passes. Mirror the nested core types faithfully (also for K8s/Infra/Hooks), or use core's decoder.

	Docker               map[string]any    `yaml:"docker"`
	K8s                  map[string]any    `yaml:"k8s"`
	Module               string            `yaml:"module"`
	Infra                map[string]any    `yaml:"infra"`
	Hooks                map[string]any    `yaml:"hooks"`

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:407

• [azd-code-reviewer] This assertion only proves that one snippet was discovered. If any of the six existing examples changes its fence label, loses host, or misspells it as azure.ai.agents, that example silently drops out while the test remains green. Assert the current baseline count (and update it for intentional additions/removals), or track the expected snippets individually so losing coverage is visible.

	require.NotZero(t, checked, "no azure.ai.agent doc examples were found — is the extractor still working?")

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:130

• [azd-code-reviewer] Treating absent/true additionalProperties as runtime support leaves the same silent-no-op gap for typed objects. The shipped schema marks agentEndpoint and agentCard with additionalProperties: true, while runtime unmarshals them into fixed Go structs; therefore agentEndpoint: {rai_config: ...} is ignored by json.Unmarshal and still passes this vocabulary check. Tighten those schema nodes (while retaining genuinely free-form maps such as metadata/credentials) or validate them against the runtime types.

This issue also appears on line 274 of the same file.

			if allowed, ok := additional.(bool); !declared || (ok && allowed) {
				continue

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:402

• [azd-code-reviewer] The extractor has dedicated tests, but neither advertised validation rule has a committed negative regression test. With only valid live docs as fixtures, a change that makes checkVocabulary permissive or stops enforcing resolver failures can leave this suite green; the temporary defect injections described in the PR will not protect future changes. Add table-driven invalid snippets for missing name, unknown top-level/nested properties, and the partial-marker exception.

				checkVocabulary(t, e, name, svc, schema)

[jongio]

One nit inline. Everything else checks out.

I ran the new test on the branch and mutated the docs to confirm each guard actually fires: dropping name from the README migration example reproduces the name cannot be empty failure from #9330, an undeclared key at the top level and one nested under policies[0] both fail with the file:line message, and project: [src] is a core parse error now instead of a silent coercion to "".

The extractor's nested-fence handling holds up too, so the example block inside AGENTS.md isn't picked up as its own snippet.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:349

• [azd-code-reviewer] Add negative regression fixtures for the validation rules. The current docs are already valid, so this test still passes if the missing-name check or the top-level/recursive vocabulary rejection is accidentally weakened; the manually injected defects listed in the PR description are not preserved in CI. Table-driven snippets covering missing name, an unknown service sibling, and an unknown nested property would lock down both defect classes this test is meant to catch.

func TestDocExamplesAreValid(t *testing.T) {

[jongio]

One blocking item and one note.

Blocking: the new test fails CI on all three BuildExtension legs. It passes on this branch in isolation, which is why a local run looks clean. It only breaks against the merge with main. #9079 added a partial env: snippet to the extension README after this branch was cut, and strict-by-default rejects it for missing kind. Detail and the one-line fix are inline on the require.True assertion.

Note: the reply on my structpb thread posted as the literal text @C:\Users\glharper\AppData\Local\Temp\reply9368.md instead of the file contents, so whatever you wrote there isn't visible on the PR. Worth reposting.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:278

• [azd-code-reviewer] These untyped mirrors still do not apply core's nested shape rules. For example, a valid agent snippet containing hooks: {predeploy: "echo hi"} passes this map[string]any decode and the resolver, but core's HooksConfig.UnmarshalYAML rejects that scalar because each hook must be a mapping or sequence (pkg/ext/hooks_config.go:33-72). The test would therefore approve an example that cannot be copied into azure.yaml. Mirror the concrete nested field shapes (including their custom unmarshalling) or decode through core's actual ServiceConfig; the same gap applies to docker, k8s, and infra.

	Docker               map[string]any    `yaml:"docker"`
	K8s                  map[string]any    `yaml:"k8s"`
	Module               string            `yaml:"module"`
	Infra                map[string]any    `yaml:"infra"`
	Hooks                map[string]any    `yaml:"hooks"`

[jongio]

The blocking item from my last review is cleared. I verified against the merge with main rather than the branch, since that's where it failed.

On 3ca6440 with main merged in, the suite validates 7 snippets and passes. Dropping the marker back out reproduces the exact CI signature I flagged:

README.md:63: service "my-agent" did not resolve to an agent definition (is `kind` missing?).

I also checked whether the marker buys more silence than intended, since the escape hatch is the part of this with the most room to go wrong later. It doesn't. With the marker still in place, an undeclared key on that same snippet fails:

README.md:64: service "my-agent" documents property "rai_config", which azd does not support.

and project: [src] fails with cannot be parsed by azd core. So partial drops only the "must resolve to an agent definition" assertion and leaves the vocabulary and core-parse checks running, which is what AGENTS.md describes.

The reply on the structpb thread that posted as a literal file path is showing real content now.

No new findings. My earlier approval went stale when these commits landed, so this still needs a current approval to merge.