Extensibility requests - evals by AleksandricMarko · Pull Request #490 · microsoft/BC-Bench

AleksandricMarko · 2026-01-25T20:00:42Z

Please go to the Preview tab and select the appropriate template for your changes:

Experiment - For running benchmarks and experiments
Standard PR - Just delete this text and fill in description

haoranpb

Looks good overall.

The only piece missing is reading the agent's output during the evaluate step.

Will need some refactoring if you want it merged, but good if you just need it to run locally.

- Enhanced parse_metrics_ext to search session logs for JSON output (agent writes to stdout, not stderr where metrics are parsed) - Updated step7 instructions to require comprehensive JSON output with all workflow step results in specified structure - Updated prompt template to explicitly request JSON output - Agent now consistently produces structured JSON output for analysis Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Implements comparison between agent output and expected results. Validates outcome is FEASIBLE and labels match expected values. Sets resolved based on validation result. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

…ensibility_evals

haoranpb · 2026-02-27T09:28:49Z

+  extensibility-request-template: |
+    You are working with a Business Central (AL) code repository at {{repo_path}}.
+
+    Task: Analyze and process the extensibility request using the custom agent - "Argus extensibility agent"


We probably don't need to specify this in the prompt

…ensibility_evals

Copilot

Pull request overview

Adds extensibility-request evaluations using the Argus agent, a YAML benchmark dataset, structured output parsing, and LLM-based comment scoring.

Changes:

Adds extensibility dataset, pipeline, result, CLI, and Copilot support.
Adds Argus instructions, rules, templates, and team mappings.
Updates dependencies, tests, repository cleanup, and analysis notebooks.

Reviewed changes

Copilot reviewed 59 out of 60 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
`uv.lock`	Locks autoevals dependencies.
`pyproject.toml`	Adds autoevals dependency.
`src/bcbench/types.py`	Adds extensibility category and metrics.
`src/bcbench/config.py`	Adds extensibility dataset paths.
`src/bcbench/cli.py`	Adjusts stream typing.
`src/bcbench/commands/run.py`	Supports running extensibility requests.
`src/bcbench/commands/evaluate.py`	Supports extensibility evaluation.
`src/bcbench/dataset/__init__.py`	Exports extensibility dataset APIs.
`src/bcbench/dataset/dataset_entry.py`	Defines extensibility entries.
`src/bcbench/dataset/dataset_loader.py`	Loads YAML extensibility datasets.
`src/bcbench/evaluate/__init__.py`	Exports extensibility comparison.
`src/bcbench/evaluate/base.py`	Registers the new pipeline.
`src/bcbench/evaluate/bugfix.py`	Adds environment-version fallback.
`src/bcbench/evaluate/testgeneration.py`	Adds environment-version fallback.
`src/bcbench/evaluate/extensibility.py`	Implements extensibility scoring.
`src/bcbench/results/base.py`	Deserializes extensibility results.
`src/bcbench/results/bceval_export.py`	Exports extensibility results.
`src/bcbench/results/extensibility.py`	Defines extensibility result type.
`src/bcbench/operations/git_operations.py`	Changes repository cleanup behavior.
`src/bcbench/operations/instruction_operations.py`	Handles read-only deletion.
`src/bcbench/agent/copilot/agent.py`	Runs Argus through Copilot CLI.
`src/bcbench/agent/copilot/metrics.py`	Extracts structured agent output.
`src/bcbench/agent/shared/__init__.py`	Exports extensibility prompts.
`src/bcbench/agent/shared/prompt.py`	Builds extensibility prompts.
`src/bcbench/agent/shared/config.yaml`	Configures Argus and its prompt.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/Argus.agent.md`	Defines the Argus workflow.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/step0-getting-started.md`	Defines startup checks.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/step1-collect-data.md`	Defines issue collection.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/step2-eligibility-check.md`	Defines eligibility checks.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/step3-request-types.md`	Defines request classification.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/step4-requirements-check.md`	Defines requirement validation.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/step5-codebase-analysis.md`	Defines code analysis.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/step6-team-assignment.md`	Defines team assignment.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/step7-labels-comments.md`	Defines final output.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/team-configuration/team_namespace_mapping.yaml`	Maps namespaces to teams.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/comment-templates/comment_templates.yaml`	Adds issue comment templates.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/input-requirements/general_requirements.yaml`	Adds general requirements.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/input-requirements/event_request_requirements.yaml`	Adds event requirements.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/input-requirements/enum_request_requirements.yaml`	Adds enum requirements.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/input-requirements/extensibility_enhancement_requirements.yaml`	Adds enhancement requirements.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/input-requirements/request_for_external_requirements.yaml`	Adds external-access requirements.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/codebase-rules/general_blockers.yaml`	Adds general blockers.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/codebase-rules/event_request_blockers.yaml`	Adds event blockers.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/codebase-rules/event_request_warnings.yaml`	Adds event warnings.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/codebase-rules/event_request_implementation.yaml`	Adds event implementation rules.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/codebase-rules/event_request_alternative_suggestions.yaml`	Adds event alternatives.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/codebase-rules/event_request_ishandled_blockers.yaml`	Adds IsHandled blockers.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/codebase-rules/event_request_ishandled_warnings.yaml`	Adds IsHandled warnings.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/codebase-rules/event_request_ishandled_implementation.yaml`	Adds IsHandled implementation rules.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/codebase-rules/event_request_ishandled_alternative_suggestions.yaml`	Adds IsHandled alternatives.
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/Argus/codebase-rules/request_for_external_implementation.yaml`	Adds external-access rules.
`dataset/extensibility_requests/extensibility_dataset.yaml`	Adds benchmark entries.
`dataset/extensibility_requests/default_setup.json`	Adds dataset defaults.
`docs/_data/extensibility-request.json`	Initializes documentation data.
`tests/test_cli_commands.py`	Tests container-free run command.
`tests/test_custom_instructions.py`	Uses explicit UTF-8 reads.
`tests/test_type_exhaustiveness.py`	Excludes the specialized category.
`notebooks/test-generation/altest-comparison.ipynb`	Updates test-generation analysis.
`notebooks/bug-fix/altool-comparison.ipynb`	Updates bug-fix analysis.
`notebooks/archive/multi-run.ipynb`	Changes multi-run pass@k analysis.

+    for entry in entries:
+        if entry_id:
+            if entry.instance_id == entry_id:
+                return [entry]
+            continue
+        entries.append(entry)
+
+    if entry_id:
+        raise EntryNotFoundError(entry_id)
+
+    return entries


+class ExtAgentMetrics(BaseModel):
+    """Metrics collected during extensibility agent execution.
+
+    Separates runtime execution data from experiment configuration.
+    """
+
+    model_config = ConfigDict(frozen=True)
+
+    # Total execution time in seconds
+    execution_time: float | None = None
+    llm_duration: float | None = None
+
+    turn_count: int | None = None
+
+    # Token usage from LLM calls
+    prompt_tokens: int | None = None
+    completion_tokens: int | None = None
+
+    # Tool usage statistics from agent logs
+    tool_usage: dict[str, int] | None = None
+
+    # JSON output produced by the extensibility agent
+    json_output: str | None = None


+  enabled: true
+  name: Argus


+    result = subprocess.run(["git", "clean", "-ffd"], check=False, cwd=repo_path, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
+    if result.returncode != 0:
+        logger.warning(f"git clean -ffd exited with {result.returncode}: {result.stderr.decode(errors='replace')}")


    "for k in k_values:\n",
    "    first_k_runs = run_ids_sorted[:k]\n",
    "    pivot_k = pivot[first_k_runs]\n",
-    "    total = sum(pass_at_k(k, int(row.sum()), k) for _, row in pivot_k.iterrows())\n",
+    "    total = sum(pass_at_k(n_runs, int(row.sum()), k) for _, row in pivot_k.iterrows())\n",


+          mandatory: true
+          validation_hints: ["usage", "context", "table", "field", "page"]
+
+    extend_enum:


-    "table_df = ci_df.copy()\n",
-    "table_df[\"Mean \u00b1 95% CI\"] = table_df.apply(lambda r: f\"{r['Mean %']:.1f}% \u00b1 {r['CI \u00b1']:.1f}%\", axis=1)\n",
-    "table_df[[\"Setup\", \"N Runs\", \"Mean \u00b1 95% CI\"]]"
+    "ci_df = pd.DataFrame(ci_data).sort_values(\"Mean %\", ascending=True)"


    for category in EvaluationCategory:
+        if category == EvaluationCategory.EXTENSIBILITY_REQUEST:
+            continue  # Uses get_info_from_dataset_entry_ext with ExtensibilityDatasetEntry


+    if category == EvaluationCategory.EXTENSIBILITY_REQUEST:
+        if dataset_path is None:
+            dataset_path = _config.paths.ext_dataset_path
+        entries: list[ExtensibilityDatasetEntry] = load_ext_dataset_entries(dataset_path, entry_id=entry_id)


+_JSON_FENCE_REAL_NEWLINES = re.compile(r"```json\s*(\{[\s\S]*?\})\s*```")
+
+# Pattern to find JSON code fences with literal \n (escaped newlines in JSON string values
+# found in session log files where assistant content is stored as JSON strings)
+_JSON_FENCE_ESCAPED_NEWLINES = re.compile(r"```json\\n(\{.*?\})\\n```")


AleksandricMarko added 2 commits January 25, 2026 15:01

Extensibility evaluation init

b79f31f

Extensibility evaluation fix

5ed86bd

AleksandricMarko temporarily deployed to ado-read January 25, 2026 20:01 — with GitHub Actions Inactive

github-code-quality Bot found potential problems Jan 25, 2026

View reviewed changes

Comment thread src/bcbench/results/extensibility.py Fixed

agent generated to make it work

ee0d944

AleksandricMarko temporarily deployed to ado-read January 26, 2026 10:17 — with GitHub Actions Inactive

github-code-quality Bot found potential problems Jan 26, 2026

View reviewed changes

Comment thread src/bcbench/agent/copilot/agent.py Fixed

haoranpb reviewed Jan 29, 2026

View reviewed changes

Comment thread dataset/extensibility_requests/default_setup.json

Comment thread src/bcbench/agent/copilot/agent.py Fixed

Comment thread src/bcbench/commands/run.py

Comment thread src/bcbench/dataset/dataset_entry.py

Comment thread src/bcbench/evaluate/extensibility.py Outdated

AleksandricMarko and others added 5 commits February 4, 2026 15:51

Refactor metrics parsing and fix type errors

11cef06

Add validation logic for extensibility evaluation

20f23ce

Implements comparison between agent output and expected results. Validates outcome is FEASIBLE and labels match expected values. Sets resolved based on validation result. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

agent update

f2b1430

fix

57023d3

AleksandricMarko temporarily deployed to ado-read February 4, 2026 18:23 — with GitHub Actions Inactive

instruciton fix

5807f81

AleksandricMarko temporarily deployed to ado-read February 4, 2026 21:17 — with GitHub Actions Inactive

AleksandricMarko added 8 commits February 12, 2026 15:47

metric created

5cfbdc4

fix

b81330e

fix

2520f3b

instruction fix

7d943b9

dataset

1e5ffd2

Merge branch 'main' of https://github.com/microsoft/BC-Bench into ext…

32cf6fb

…ensibility_evals

instr fix

de4d8c8

fix

64a89b6

AleksandricMarko temporarily deployed to ado-read February 26, 2026 11:37 — with GitHub Actions Inactive

pipeline issue fix

29afa7f

AleksandricMarko temporarily deployed to ado-read February 26, 2026 11:41 — with GitHub Actions Inactive

haoranpb reviewed Feb 27, 2026

View reviewed changes

AleksandricMarko added 2 commits March 23, 2026 14:07

additional datasets

fb9293c

Merge branch 'main' of https://github.com/microsoft/BC-Bench into ext…

780d5ff

…ensibility_evals

AleksandricMarko temporarily deployed to ado-read March 23, 2026 13:29 — with GitHub Actions Inactive

not sure what are these changes

a486cee

Copilot AI review requested due to automatic review settings July 29, 2026 05:47

Copilot started reviewing on behalf of AleksandricMarko July 29, 2026 05:47 View session

Copilot AI reviewed Jul 29, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Extensibility requests - evals - #490

Extensibility requests - evals#490
AleksandricMarko wants to merge 21 commits into
mainfrom
extensibility_evals

AleksandricMarko commented Jan 25, 2026

Uh oh!

Uh oh!

Uh oh!

haoranpb left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

haoranpb Feb 27, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

AleksandricMarko commented Jan 25, 2026

Uh oh!

Uh oh!

Uh oh!

haoranpb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

haoranpb Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants