⚡ Copilot Token Optimization2026-09-01 — Smoke Copilot BYOK AOAI (Entra)

[github-actions]

Target Workflow: smoke-copilot-byok-aoai-entra

Source report: #7881
Estimated cost per run: $0.00 (cost tracking not populated in report; token volume is the driver)
Total tokens per run: ~223K avg (207,860 and 237,752 across the 2 runs in the last 7 days)
Cache hit rate: Very low — working_set.rebuild_factor is 7.9×–8.8× (peak input ~25-26K tokens but cumulative input 202K-230K), meaning nearly the entire context is being re-sent and re-processed on almost every turn instead of being served from cache
LLM turns (invocations): 8-9 per run

Current Configuration

Setting	Value
Tools loaded	bash: ["*"] (full bash), github MCP restricted to toolsets: [pull_requests], safeoutputs MCP (required)
Tools actually used	github-list_pull_requests (1 call), bash (cat file, verify), add_comment/add_labels/noop (safe-outputs)
Network groups	defaults, github, login.microsoftonline.com — all in active use (Entra token exchange requires the last one)
Pre-agent steps	Yes — PR fetch, connectivity check, file write/read test, context file assembly
Prompt size	~10.4 KB markdown body (frontmatter + prompt)

Recommendations

1. Eliminate the GitHub MCP server call entirely for this smoke test

Estimated savings: ~60-90K tokens/run (~30-40%)

The github MCP toolset (even restricted to pull_requests) still costs ~5-10K tokens just to load its schema into context on every turn, and the github-list_pull_requests tool call result gets folded into subsequent turn context (contributing to the 8-9x rebuild factor). Since the pre-agent steps: block already fetches the same PR data via gh pr list and writes it to /tmp/gh-aw/agent/smoke-pr-data.txt, the live MCP call is redundant — it exists only to "additionally verify" connectivity, which the pre-agent step's gh call already implicitly does.

Implementation:

• Remove tools: { github: { toolsets: [pull_requests] } } from smoke-copilot-byok-aoai-entra.md.
• Rewrite the "GitHub MCP Testing" section of the prompt to validate solely against the pre-fetched /tmp/gh-aw/agent/smoke-pr-data.txt (already required as a fallback path in the current prompt), dropping the live MCP call path entirely.
• If MCP connectivity truly needs testing, move it to the pre-agent steps: block using gh api instead of relying on the agent's live tool call.

2. Break the low-cache-hit-rate cycle by avoiding per-run variable content early in context

Estimated savings: ~40-70K tokens/run (~20-30%), by pushing rebuild_factor from ~8x toward ~2-3x

The pre-agent step embeds GITHUB_RUN_ID directly into the test file path (smoke-test-copilot-byok-aoai-entra-${GITHUB_RUN_ID}.txt), and that path is written into /tmp/gh-aw/agent/smoke-context.txt, which the agent reads verbatim early in its turn. Because this file's content changes every run (unique run ID, timestamp from date), the model's prompt-prefix cache cannot be reused across runs, and combined with the multi-file cat pattern, the working set gets rebuilt on nearly every turn (7.9-8.8x factor, 8-9 invocations for what is fundamentally a 4-check smoke test).

Implementation:

• Use a fixed, predictable file name (e.g., smoke-test-copilot-byok-aoai-entra.txt, no run ID suffix) since the pre-agent step already runs fresh in an isolated container per run — the run ID adds no verification value.
• Move all variable/run-specific values (timestamps, run ID, HTTP code) to the end of smoke-context.txt rather than interleaved, so the static instructional prefix of the prompt caches consistently.
• Consolidate the 4 separate cat/read operations the agent currently performs (smoke-context.txt, then implicitly re-reading file content/path/http-code files referenced in the prompt) into a single pre-agent-assembled JSON or single flat file so the agent needs only one bash read instead of several.

3. Trim prompt verbosity in the workflow body

Estimated savings: ~3-5K tokens/run (~2%)

The prompt's "Purpose" section (~600 words) re-explains the BYOK/Entra/OIDC architecture in detail on every single run — this is static documentation, not something the model needs re-derived each time to complete a 4-item checklist.

Implementation:

• Shorten the "Purpose" paragraph in smoke-copilot-byok-aoai-entra.md to 2-3 sentences; move the detailed architecture explanation to a code comment or docs/ reference link instead of the live agent prompt.
• Remove the sibling-workflow cross-reference sentences (smoke-copilot-byok-aoai-apikey, smoke-copilot-byok) from the prompt — these don't affect test verification and can live in file/PR-level documentation instead.

4. Confirm bash: ["*"] is intentionally broad

Estimated savings: minimal direct token savings, but reduces attack surface and avoids future schema growth

bash: ["*"] grants unrestricted bash rather than scoping to the specific commands actually used (cat, curl is already in pre-agent steps only, not needed in the agent prompt). Since the agent prompt only needs to cat a handful of files, consider scoping bash to an explicit allowlist (e.g., bash: ["cat *"]) if the gh-aw engine supports command-level restriction — this doesn't reduce per-turn token cost materially but is good hygiene and prevents future scope creep.

Expected Impact

Metric	Current	Projected	Savings
Total tokens/run	~223K avg	~90-120K	-45-55%
Cost/run	not tracked	not tracked	—
LLM turns	8-9	3-4	-5
Rebuild factor	7.9-8.8x	~2-3x	-65%

Implementation Checklist

• Remove github MCP tool block from smoke-copilot-byok-aoai-entra.md; rely solely on pre-fetched smoke-pr-data.txt
• Drop the GITHUB_RUN_ID suffix from the pre-agent test file name; use a fixed file name instead
• Reorder smoke-context.txt so static content comes first and variable/run-specific values (timestamps, HTTP code, run ID) come last
• Consolidate the multiple pre-agent output files (smoke-http-code.txt, smoke-file-content.txt, smoke-file-path.txt, smoke-pr-data.txt) into the single smoke-context.txt the agent already reads, removing the need for extra cat calls
• Shorten the "Purpose" section of the prompt to 2-3 sentences
• Recompile: gh aw compile .github/workflows/smoke-copilot-byok-aoai-entra.md
• Post-process: npx tsx scripts/ci/postprocess-smoke-workflows.ts
• Verify CI passes on PR
• Compare token usage on new run vs baseline (target: <120K tokens/run, ≤4 turns)

Generated by Daily Copilot Token Optimization Advisor · copilot · auto · 42.1 AIC · ⊞ 10.7K · ◷

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #8003

View newer issue

---

This action was performed automatically by the Daily Copilot Token Optimization Advisor workflow.