Add data-query category: AL query-generation benchmark

[onbuyuka]

What

Adds a new data-query evaluation category: an offline benchmark for AL query generation. Given a natural-language data question, the agent authors a single AL query object (query.al) and it's scored deterministically — compile + run the generated query and a gold reference query against the container's Contoso demo data, pass if the result sets match. No MCP server, no LLM judge.

This complements the AI Test Toolkit evals in the platform repo: those test the MCP server end-to-end; this benchmarks models/agents on query generation.

How it works

• Dataset (dataset/dataquery.jsonl) — 11 entries, each with nl_prompt + gold_query (the reference AL query whose result set defines "correct") + environment_setup_version + ordered.

• Pipeline (evaluate/dataquery.py) — the agent writes query.al; the harness compiles + runs the generated and gold queries and compares result sets.

  • build = generated query compiled and ran; resolved (ResolutionRate) = result set matches gold.

  • Rows compared by value; numeric columns are normalized scale-insensitively (500 == 500.0) while string Code/No. columns are compared verbatim ("001" != "1"). Column names/order ignored; order-insensitive unless the entry marks the question ordered.

• Run mechanism (operations/bc_operations.py) — wrap_query_as_api turns each query into an API query; execute_al_query publishes a throwaway app to the container and reads its OData endpoint (following @odata.nextLink so large result sets aren't truncated).

• Execution-based category (requires_container = True, runner GitHub-BCBench) — a stock BC sandbox artifact (Cronus/Contoso data) suffices; no special build needed. Setup-ContainerAndRepository.ps1 skips the git clone for data-query (there's no repo) and just provisions the container.

• Skill — an agent-facing al-query-authoring SKILL.md (instructions/dataquery-bc/skills/) documenting AL query authoring, including the canonical column(Name) { Method = Count; } idiom (a field-less Count; a source field trips AL0353).

Scoring integrity & robustness

Review-driven hardening so scores can't be gamed or silently corrupted:

• Gold withheld from the agent — the agent runs with unrestricted tools in a workspace inside the checkout, so it could read the reference answers. During generation we strip gold_query from the on-disk dataset and relocate the checkout's .git out of the agent-reachable tree (defeating git show HEAD:… recovery), restoring both afterward. The harness scores from the in-memory entry, loaded before the agent starts.

• Gold validated independently first — evaluation runs the gold query before the agent's; a broken gold is recorded as unscorable rather than blamed on the agent, and unscorable results are excluded from ResolutionRate and the bceval export.

• Deterministic comparison fixes — numeric-only decimal canonicalization (Code strings preserved), OData paging, timeout → BuildTimeoutExpired, malformed agent output → BuildError, case-insensitive query/QueryType wrapping, OrderBy treated as a property, and throwaway apps uninstalled between runs to avoid object-ID conflicts.

How to run

Actions → Evaluation with GitHub Copilot (or Claude Code) → Run workflow → category = data-query. The self-hosted runner provisions the container; no local setup.

New --skills dispatch input (mirrors --al-lsp / --al-mcp) toggles the agent skill per run — default off, so the with/without-skills ablation is preserved.

Tested

• ✅ Unit tests: result_sets_match (incl. digit-only Code non-collapsing), wrap_query_as_api, run-template wiring, gold-withholding round-trip, and --skills override (tests/test_dataquery_evaluation.py, tests/test_agent_skills.py); full suite 696 pass; ruff + ty clean.

• ✅ End-to-end on the self-hosted runner: the execute_al_query container round-trip (compile → publish → OData → result-set compare) is validated — generated and gold queries build, publish, and compare, producing real build/resolved scores.

Results

Run 30405864654 (claude-sonnet-4.6, skills on + AL LSP): 9/11 resolved, 11/11 build. Every generated query compiles and runs, and the two misses (avg-invoice-amount-by-country, total-purchase-amount-by-vendor) both build but diverge from gold on metric-definition ambiguity (which amount / net vs incl-VAT, per-invoice vs per-line) — left as documented misses for a future gold/prompt-reconciliation pass rather than harness defects.

Known non-blocking CI caveat: on a data-query run the matrix + summarize + bceval-upload jobs succeed, but the summarize job's Update leaderboard step fails because it checks out main, whose EvaluationCategory enum has no data-query yet. This self-resolves once this PR merges.

Review feedback has been addressed in follow-up commits and every review thread is resolved — see the review-response comments.

[Groenbech96]

Looks good. Good stuff.

[Copilot]

Pull request overview

Adds the data-query benchmark for deterministic AL query generation and execution against Business Central demo data.

Changes:

• Adds 11 query-generation dataset entries and agent guidance.
• Implements query wrapping, execution, comparison, and result reporting.
• Integrates container setup, workflows, tests, and documentation.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
dataset/dataquery.jsonl	Adds benchmark entries and gold queries.
src/bcbench/evaluate/dataquery.py	Implements evaluation and result comparison.
src/bcbench/operations/bc_operations.py	Adds query wrapping and OData execution.
src/bcbench/dataset/dataset_entry.py	Defines data-query entries.
src/bcbench/dataset/__init__.py	Exports the new entry type.
src/bcbench/types.py	Registers category runtime behavior.
src/bcbench/results/base.py	Adds a general execution-result factory.
src/bcbench/evaluate/__init__.py	Exports the pipeline.
src/bcbench/operations/__init__.py	Exports query operations.
src/bcbench/commands/evaluate.py	Supports mock data-query evaluation.
src/bcbench/agent/shared/config.yaml	Adds the agent prompt template.
src/bcbench/agent/shared/instructions/dataquery-bc/skills/al-query-authoring/SKILL.md	Adds AL query authoring guidance.
scripts/Setup-ContainerAndRepository.ps1	Creates clone-free workspaces.
scripts/BCBenchUtils.psm1	Resolves the new dataset category.
.github/workflows/copilot-evaluation.yml	Enables Copilot runs.
.github/workflows/claude-evaluation.yml	Enables Claude runs.
tests/test_dataquery_evaluation.py	Tests comparison and wrapping logic.
tests/conftest.py	Adds data-query fixtures.
tests/test_type_exhaustiveness.py	Covers category type dispatch.
docs/data-query.md	Documents the benchmark.
docs/index.md	Links the new category.

Comments suppressed due to low confidence (1)

src/bcbench/evaluate/dataquery.py:115

• execute_al_query also raises BuildTimeoutExpired on a gold-query timeout, and it is not a BuildError. This exception escapes instead of taking the intended harness/container failure path.

        except BuildError as e:
            logger.exception(f"Gold query failed to compile/run for {context.entry.instance_id}")
            self.save_result(
                context,

[onbuyuka]

Review feedback addressed (commit 129c5ef)

Thanks for the thorough review. Summary of what changed:

Scoring integrity

• Numeric rounding false-positives — result_sets_match now canonicalizes numbers with Decimal.normalize() (full precision, scale-insensitive) instead of rounding through float to 4 decimals. 1.00001 ≠ 1.00002; 500 == 500.0.
• OData pagination — the fetch now follows @odata.nextLink until exhausted, so large result sets aren't silently truncated to the first page.
• Gold-failure biasing scores — a gold/harness failure is now recorded as unscorable (new scorable flag) and excluded from resolved/total/build/instance_results, so it no longer counts against the agent's ResolutionRate.
• Timeout escaping — BuildTimeoutExpired is now caught around both generated and gold execution (it isn't a BuildError), so a timeout is recorded instead of breaking summarization.
• ValueError on malformed output — wrap_query_as_api now raises BuildError (handled downstream) when there's no query declaration or no object body.

Robustness

• Case sensitivity — the query keyword reassignment and QueryType removal are now case-insensitive and don't require a leading newline, so Query 50123 / { QueryType = Normal; ... } no longer break ID reassignment or create a duplicate QueryType.
• Object-ID conflict on re-run — execute_al_query now uninstalls/unpublishes the throwaway app before and after each run, so re-running locally against the same container doesn't hit a 50100/50101 conflict.

Docs/cleanup

• SKILL.md: OrderBy corrected to a property (OrderBy = descending(Col);), not a block.
• types.py: stale MCP/seed comments removed (folded DATA_QUERY into the existing match arms).
• PR description: validation scope updated to the current 11 gold queries; the 5 new ones are in runner shakeout now.

Added unit tests for the precision fix, the case-insensitive/malformed wrap_query_as_api paths, and the paging/cleanup wiring. Full suite: 635 pass, ruff + ty clean.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/bcbench/operations/bc_operations.py:293

• AL escapes a quote inside a quoted identifier by doubling it ("A ""quoted"" query"), not with a backslash. This regex stops at the first doubled quote, leaves the rest of the original name behind, and turns an otherwise valid query into invalid AL. Match doubled quotes in the quoted-name branch.

    text, replaced = re.subn(
        r'(\bquery\s+)\d+\s+("(?:[^"\\]|\\.)*"|\w+)',
        rf"\g<1>{object_id} {safe_name}",

src/bcbench/evaluate/dataquery.py:31

• This converts every numeric-looking string to a number, so distinct AL text/code values such as "001" and "1" compare equal (and the earlier None conversion similarly equates null with ""). That can award resolution to a query returning the wrong identifier. Preserve string/null identity and normalize only values known to be numeric, or carry type information into comparison.

    try:
        # Canonical decimal form: scale/trailing-zero-insensitive (500 == 500.0) but full precision
        # preserved, so distinct values like 1.00001 and 1.00002 are NOT collapsed. No float rounding.
        return str(Decimal(text).normalize())
    except (InvalidOperation, ValueError):

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

src/bcbench/results/base.py:116

• scorable=False is not propagated to the bc-eval records. category_metrics exports only resolved=False and build=True, and ResolutionRate/BuildRate score those values directly, so the externally reported headline metrics still count a gold-query failure as a resolution failure (and a build success), contrary to the new unscorable semantics. Export scorable and make the downstream evaluators skip these records, or omit unscorable records from the bc-eval export.

    def create_unscorable(cls, context: "EvaluationContext", output: str, error_message: str) -> Self:
        """A harness/dataset failure (not the agent's fault) that must not count toward the resolution rate."""
        return cls(**cls._base_fields(context), output=output, build=True, resolved=False, scorable=False, error_message=error_message)

src/bcbench/operations/bc_operations.py:426

• The OData JSON is parsed through Python float before _normalize_value sees it, so high-magnitude BC Decimal values can lose precision and distinct results can compare equal (for example, adjacent cent values near BC Decimal's upper range). Parse JSON decimal literals directly as Decimal to preserve the deterministic comparison promised by the matcher.

    rows = json.loads(result_file.read_text(encoding="utf-8-sig") or "[]")

src/bcbench/evaluate/dataquery.py:88

• The new pipeline's outcome logic is not covered by the added tests: there are no tests that mock execute_al_query and verify match, mismatch, generated build failure, and gold-query unscorable results. These branches define the benchmark's scores, and the current ordering/export issues are examples that helper-only tests do not catch. Add focused pipeline tests like those used for the existing evaluation pipelines.

    def evaluate(self, context: EvaluationContext[DataQueryEntry]) -> None:

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

src/bcbench/operations/bc_operations.py:305

• The transform is not comment-aware. In a valid query with a preceding comment such as // QueryType = Normal;, this substitution removes the comment occurrence because count=1, leaves the real property, and then injects a second QueryType, causing compilation to fail. Likewise, text.find("{") can select a brace in a leading comment. Locate the declaration/body with comment-aware parsing and remove the actual object-level property rather than the first textual match.

    text = re.sub(r"\bQueryType\s*=\s*\w+\s*;", "", text, count=1, flags=re.IGNORECASE)

    brace_index = text.find("{")
    if brace_index == -1:
        raise BuildError("query-wrap", f"Generated query has no object body ('{{' not found):\n{query_text}")

src/bcbench/agent/shared/instructions/dataquery-bc/skills/al-query-authoring/SKILL.md:27

• This example is effectively the gold solution for dataquery__outstanding-sales-value-by-customer-1: it uses the same Customer → Sales Line join, Order filter, and Outstanding Amount sum. Any run with this skill enabled receives the answer to a benchmark entry (including the first test-run entry), inflating that experiment's score. Replace it with a valid query pattern that is not represented in the dataset.

            dataitem(SalesLine; "Sales Line")
            {
                DataItemLink = "Sell-to Customer No." = Customer."No.";
                DataItemTableFilter = "Document Type" = const(Order);
                column(OutstandingAmount; "Outstanding Amount") { Method = Sum; }

src/bcbench/results/bceval_export.py:53

• This scoring-critical skip path has no regression coverage: tests/test_result_writer.py comprehensively exercises write_bceval_results, but no test creates an execution result with scorable=False. Add mixed and all-unscorable cases to verify these records never reach the bc-eval JSONL output.

            # Unscorable results (harness/dataset failures, e.g. a gold query that didn't compile) must
            # not reach the uploaded/core score, or they'd count against the agent's ResolutionRate.
            if isinstance(result, ExecutionBasedEvaluationResult) and not result.scorable:
                logger.info(f"Skipping unscorable result from bceval export: {result.instance_id}")
                continue

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

src/bcbench/results/base.py:103

• Add regression coverage for the new scorable contract. The current tests never create an unscorable execution result, so they do not verify that summaries exclude it from total/rates/instance_results and that the bc-eval writer omits it. These are core scoring-integrity paths and can regress independently.

    # False marks a harness/dataset failure (e.g. the gold query didn't compile) that must be
    # excluded from the resolution rate so it is not counted against the agent.
    scorable: bool = True

src/bcbench/evaluate/dataquery.py:30

• This normalizes every string as a decimal when possible, so distinct Business Central text/code values collapse ("00100" == "100", and a text "500" equals numeric 500). Since the benchmark frequently scores No./code columns, an incorrect result set can therefore pass. Preserve the type of textual OData values and normalize only actual numeric JSON values (or otherwise carry schema/type information through comparison).

        # Canonical decimal form: scale/trailing-zero-insensitive (500 == 500.0) but full precision
        # preserved, so distinct values like 1.00001 and 1.00002 are NOT collapsed. No float rounding.
        return str(Decimal(text).normalize())

src/bcbench/evaluate/dataquery.py:79

• The advertised --al-mcp/--al-lsp levers cannot provide project-aware feedback for this category: this setup leaves the workspace empty, DataQueryEntry.project_paths remains [], and both builders pass only entry.project_paths as AL projects (mcp.py:63, lsp.py:85). They therefore launch with no AL project or app.json. Create a minimal project scaffold before agent execution and point these integrations at it, or disable the unsupported levers for data-query.

    def setup_workspace(self, entry: DataQueryEntry, repo_path: Path) -> None:
        _prepare_repo_path(repo_path)

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (4)

src/bcbench/results/base.py:116

• build is defined as whether the generated query compiled and ran, but this factory is called when the gold query fails before the generated query is executed. Persisting build=True therefore records a build that never happened in the per-instance result; set it to False.

        return cls(**cls._base_fields(context), output=output, build=True, resolved=False, scorable=False, error_message=error_message)

src/bcbench/results/bceval_export.py:52

• Add regression coverage proving that an unscorable result is omitted while adjacent scorable results are still exported. tests/test_result_writer.py exercises this writer extensively, but no test reaches this new scoring-integrity branch, so a regression would again charge a gold/harness failure to the agent.

            # Unscorable results (harness/dataset failures, e.g. a gold query that didn't compile) must
            # not reach the uploaded/core score, or they'd count against the agent's ResolutionRate.
            if isinstance(result, ExecutionBasedEvaluationResult) and not result.scorable:
                logger.info(f"Skipping unscorable result from bceval export: {result.instance_id}")

src/bcbench/results/summary.py:166

• Add a mixed scorable/unscorable regression test that verifies total, resolved, failed, build, percentage, and instance_results. tests/test_evaluation_summary.py covers from_results calculations but does not exercise this new exclusion path, which directly controls the reported benchmark rate.

        # Exclude unscorable results (harness/dataset failures) from every rate so they are not
        # counted against the agent.
        scorable = [r for r in results if not (isinstance(r, ExecutionBasedEvaluationResult) and not r.scorable)]
        total = len(scorable)

docs/data-query.md:32

• test-run currently samples four entries (src/bcbench/commands/dataset.py:54 uses random=4), not two. Update this instruction so the documented workflow behavior matches the run.

2. Set **category** = `data-query`, pick a **model**, leave **test-run** = `true` for a quick 2-entry run.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (8)

src/bcbench/evaluate/dataquery.py:56

• This does not move the gold history outside the agent's reach. Both agents can run unrestricted filesystem commands, so they can enumerate repo_root.parent and pass this sibling directory to git --git-dir; if the rename fails, the fallback deliberately leaves .git in place. Once this dataset is public, the same committed blob is also fetchable from the remote. The gold must be supplied to an isolated evaluator only after generation (or stored behind permissions unavailable to the agent), otherwise benchmark scores remain contaminable.

    hideaway = repo_root.parent / f".bcbench-withheld-git-{uuid.uuid4().hex}"
    try:
        git_dir.rename(hideaway)
    except OSError as exc:
        logger.warning("Could not relocate .git to withhold gold history (%s); working-tree strip only", exc)

src/bcbench/results/base.py:116

• build=True contradicts the category's definition that build means the generated query compiled and ran. This factory is called immediately after the gold fails, before the generated query is executed, so raw artifacts falsely report a successful agent build even though that outcome is unknown.

        return cls(**cls._base_fields(context), output=output, build=True, resolved=False, scorable=False, error_message=error_message)

src/bcbench/operations/bc_operations.py:426

• The default JSON decoder converts OData decimal literals to binary float before comparison, so sufficiently precise distinct values can collapse to the same value despite _normalize_value claiming full precision. Parse JSON floats as Decimal (and add the import) so the execution path receives the exact decimal text; the current precision tests bypass this path by passing strings directly.

    rows = json.loads(result_file.read_text(encoding="utf-8-sig") or "[]")

src/bcbench/evaluate/dataquery.py:63

• Suppressing a restore failure can leave the checkout permanently without .git and leak a hidden directory into the self-hosted runner, while the evaluation appears successful. Restoration is a required invariant here; let the error fail the job (and log the hideaway path) rather than silently continuing with corrupted runner state.

        with contextlib.suppress(OSError):
            hideaway.rename(git_dir)

src/bcbench/results/bceval_export.py:52

• The new export skip is scoring-critical, but the existing write_bceval_results test suite has no case with scorable=False. Add a regression test proving an unscorable result emits no JSONL row while neighboring scorable results are still exported; otherwise this previously identified score-contamination path can regress unnoticed.

            if isinstance(result, ExecutionBasedEvaluationResult) and not result.scorable:
                logger.info(f"Skipping unscorable result from bceval export: {result.instance_id}")

src/bcbench/operations/bc_operations.py:292

• The quoted-name branch uses backslash escaping, but AL escapes a quote inside a quoted identifier by doubling it. A valid declaration such as query 50100 "A""B" is therefore matched only through "A" and rewritten into invalid AL. Match doubled quotes as part of the identifier before replacing the declaration.

        r'(\bquery\s+)\d+\s+("(?:[^"\\]|\\.)*"|\w+)',

docs/data-query.md:8

• This says there is no live server in the loop, but the next paragraph and execution section state that both queries are published to and run against a live BC container/OData endpoint. Clarify that there is no external MCP/production service; the local container is still part of scoring.

This category benchmarks an agent's ability to **generate Business Central AL queries** from a natural-language data question — an offline query-generation benchmark. There is **no MCP server and no live server in the loop**: the agent writes an AL query, and the query is evaluated deterministically.

src/bcbench/results/summary.py:169

• The new denominator filtering is not covered by the extensive ExecutionBasedEvaluationResultSummary.from_results tests. Add a mixed scorable/unscorable case asserting total, failed, build, percentage, and instance_results; otherwise an unscorable gold failure can silently re-enter local leaderboard rates.

        scorable = [r for r in results if not (isinstance(r, ExecutionBasedEvaluationResult) and not r.scorable)]
        total = len(scorable)

        resolved = sum(1 for r in scorable if isinstance(r, ExecutionBasedEvaluationResult) and r.resolved)
        build = sum(1 for r in scorable if isinstance(r, ExecutionBasedEvaluationResult) and r.build)

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

src/bcbench/evaluate/dataquery.py:63

• Restoration failures are silently discarded here. If the destination is recreated or the rename otherwise fails, evaluation continues with the original checkout's .git stranded in the hideaway, despite the context manager's restoration guarantee. Surface the failure so CI cannot proceed from a corrupted checkout.

    finally:
        with contextlib.suppress(OSError):
            hideaway.rename(git_dir)

src/bcbench/operations/bc_operations.py:301

• This removes the first QueryType = ... text even when it occurs in an AL comment. A valid query containing a comment such as // QueryType = Normal; before its actual property retains the real property, then receives a second injected QueryType, causing a harness-induced compile failure. Match an object property rather than arbitrary source text (and apply the same comment-aware approach to declaration parsing).

    text = re.sub(r"\bQueryType\s*=\s*\w+\s*;", "", text, count=1, flags=re.IGNORECASE)

src/bcbench/evaluate/dataquery.py:99

• Decimal.normalize() applies the active decimal context (precision 28), so values with more than 28 significant digits are rounded during comparison. Two distinct AL Decimal values can therefore normalize to the same string and incorrectly mark a result as resolved. Canonicalize without a context-sensitive operation (for example, exact fixed-point formatting plus fractional-zero trimming).

        # Canonical decimal form: scale/trailing-zero-insensitive (500 == 500.0) but full precision
        # preserved, so distinct values like 1.00001 and 1.00002 are NOT collapsed. No float rounding.
        return str(Decimal(text).normalize())
    except (InvalidOperation, ValueError):

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

src/bcbench/evaluate/dataquery.py:52

• Relocating .git to a randomly named sibling does not make the gold history inaccessible. Both agent processes run with unrestricted filesystem permissions from <checkout>/testbed, so they can enumerate the checkout's parent, find .bcbench-withheld-git-*, and run git --git-dir <path> show HEAD:dataset/dataquery.jsonl. The object database must be placed behind an actual filesystem/process isolation boundary (or omitted from the agent-visible checkout), otherwise benchmark answers remain recoverable.

    hideaway = repo_root.parent / f".bcbench-withheld-git-{uuid.uuid4().hex}"

src/bcbench/evaluate/dataquery.py:63

• Suppressing restoration failures can leave the checkout permanently missing its .git directory while the evaluation continues as if cleanup succeeded. Propagate the rename failure (or retry and fail explicitly) so a held handle or other filesystem error cannot silently corrupt the runner/local checkout.

        with contextlib.suppress(OSError):
            hideaway.rename(git_dir)

src/bcbench/operations/bc_operations.py:427

• json.loads converts JSON decimals to binary float before _normalize_value sees them, so distinct high-precision BC Decimal values can already have collapsed to the same value (for example, amounts beyond 53 bits of precision). Parse JSON fractional numbers as Decimal to preserve the full precision that the comparator promises.

    rows = json.loads(result_file.read_text(encoding="utf-8-sig") or "[]")