From 1acc07f953510b12eeb7e3b4b7e4059f23aecba3 Mon Sep 17 00:00:00 2001 From: Xiwei Pan Date: Sun, 19 Jul 2026 01:09:04 +0800 Subject: [PATCH 1/2] Add benchmark result submission skill --- .agents/skills/run-api-benchmark/SKILL.md | 16 +-- .agents/skills/run-cli-benchmark/SKILL.md | 14 +-- .../skills/submit-benchmark-result/SKILL.md | 101 ++++++++++++++++++ .../agents/openai.yaml | 4 + CONTRIBUTING.md | 8 +- README.md | 4 +- benchmark/tests/test_docs.py | 25 +++++ intake/cloudflare-worker/README.md | 15 +++ 8 files changed, 164 insertions(+), 23 deletions(-) create mode 100644 .agents/skills/submit-benchmark-result/SKILL.md create mode 100644 .agents/skills/submit-benchmark-result/agents/openai.yaml diff --git a/.agents/skills/run-api-benchmark/SKILL.md b/.agents/skills/run-api-benchmark/SKILL.md index 7670245..d0113c9 100644 --- a/.agents/skills/run-api-benchmark/SKILL.md +++ b/.agents/skills/run-api-benchmark/SKILL.md @@ -41,8 +41,8 @@ Do not dump every configuration question into one message. > 2. Upload an intake test that is scored privately but excluded from the leaderboard. > 3. Upload an official submission. - Default to local-only only when the caller explicitly delegates the choice. Do not ask - for intake credentials unless option 2 or 3 is selected. + Default to local-only only when the caller explicitly delegates the choice. The + `$submit-benchmark-result` skill owns intake authentication and upload. 4. Resolve `PR_REF` (default `v0.6.0`), `SUBMIT_LIMIT` (default 100), and `STAMP` (default: the Makefile timestamp). Show the derived authoritative path @@ -102,15 +102,9 @@ python -m benchmark.submit --predictions --dry-run Report `bugs_found`, `total_tokens_k`, submit attempts, any `run_error`, and absolute output and log paths. A `run_error` means partial salvage, not a clean zero-bug completion. -Upload only when the caller explicitly selected an intake upload and provided -`PRB_SUBMIT_URL` plus `PRB_API_KEY` locally: - -```bash -python -m benchmark.submit --predictions # official -python -m benchmark.submit --predictions --test # intake test -``` - -Use `--test` only for option 2. Never upload merely because the run completed. +For option 2 or 3, invoke `$submit-benchmark-result` with the authoritative path and the +already-selected test/official mode. That skill owns authentication, final confirmation, +upload, and submission-ID reporting. Never upload merely because the run completed. An exit code 137 means the engine needs more memory. Preserve partial outputs and read actual command errors before recommending changes. diff --git a/.agents/skills/run-cli-benchmark/SKILL.md b/.agents/skills/run-cli-benchmark/SKILL.md index a726c30..ecd00b6 100644 --- a/.agents/skills/run-cli-benchmark/SKILL.md +++ b/.agents/skills/run-cli-benchmark/SKILL.md @@ -55,7 +55,7 @@ numbered stage at a time when its answer controls the next branch. > 2. Upload an intake test that is scored privately but excluded from the leaderboard. > 3. Upload an official submission. - Do not ask for intake credentials unless option 2 or 3 is selected. + The `$submit-benchmark-result` skill owns intake authentication and upload. 5. Resolve `PR_REF` (default `v0.6.0`), `SUBMIT_LIMIT` (default 100), and three separate, explicit paths: clone destination, authoritative submission JSON, and log directory. @@ -123,12 +123,6 @@ python -m benchmark.submit --predictions --dry-run Report `bugs_found`, `total_tokens_k`, submit attempts, any `run_error`, CLI warnings, and absolute output/log paths. Preserve partial results and logs on failure. -Upload only when the caller explicitly selected an intake upload and configured -`PRB_SUBMIT_URL` plus `PRB_API_KEY` locally: - -```bash -python -m benchmark.submit --predictions # official -python -m benchmark.submit --predictions --test # intake test -``` - -Use `--test` only for option 2. Never upload merely because the run completed. +For option 2 or 3, invoke `$submit-benchmark-result` with the authoritative path and the +already-selected test/official mode. That skill owns authentication, final confirmation, +upload, and submission-ID reporting. Never upload merely because the run completed. diff --git a/.agents/skills/submit-benchmark-result/SKILL.md b/.agents/skills/submit-benchmark-result/SKILL.md new file mode 100644 index 0000000..70e966a --- /dev/null +++ b/.agents/skills/submit-benchmark-result/SKILL.md @@ -0,0 +1,101 @@ +--- +name: submit-benchmark-result +description: Validate and upload an existing problem-reductions benchmark submission through the private intake. Use when a user wants to submit, upload, publish, dry-run, or test a submission.json they already produced, especially when the user is not a repository maintainer. Handles local validation, test-versus-official intent, intake authentication, upload confirmation, and submission ID reporting; routes missing results to run-benchmark instead of running a model itself. +--- + +# Submit a benchmark result + +Submit an existing `submission.json` without requiring GitHub repository, R2, Worker, or +Actions access. Treat the file as confidential because it contains certificates and the +submit ledger. + +## 1. Locate the authoritative result + +Ask for the submission path only when it was not supplied. Accept the authoritative output +from either benchmark backend; do not reconstruct it from logs or edit its metrics, +certificates, `library_commit`, or ledger. + +If no result exists and the user wants to run the benchmark, invoke `$run-benchmark`. Do not +choose a model/backend or start a paid run inside this skill. + +Read the current round from `README.md` and the current structure from +`benchmark/submission.schema.json`. Inspect only summary fields; never print certificates, +source instances, submit-log contents, trajectories, or credentials. Report: + +- model and `library_commit`; +- claimed bugs, token total, and submit-attempt count; +- `run_error`, when present; +- absolute submission path. + +Run the repository client as a local courtesy check: + +```bash +python3 -m benchmark.submit --predictions --dry-run +``` + +Stop on failure. Do not repair an invalid result by hand; send it back to the producing run. + +## 2. Choose the outcome + +Ask only when the user has not already chosen: + +> What should happen to this result? +> +> 1. Keep it local after validation. +> 2. Upload an intake test that is scored privately and never reaches the leaderboard. +> 3. Upload an official submission. + +For option 1, report validation and stop. For option 2, use `--test`. For option 3, never +use `--test`. A non-test result carrying `run_error` is not a clean official submission; +offer local-only or test upload instead. + +## 3. Authenticate without exposing credentials + +Require `PRB_SUBMIT_URL` from the repository documentation or maintainer. Never ask the user +to paste any token into chat, print an environment variable, or commit credentials. + +Prefer GitHub-backed Cloudflare Access when the deployed client documents +`PRB_ACCESS_TOKEN` or `--access-token`: + +1. Require `cloudflared` locally. +2. Run `cloudflared access login `; the user completes the configured + GitHub login in a browser. +3. Obtain the application-scoped Access token with `cloudflared access token + -app=` and pass it only through the client-supported environment + variable or flag. Do not display or persist it. + +Do not substitute `gh auth token`, a GitHub personal access token, or `GITHUB_TOKEN`; the +intake must never receive the user's general GitHub credential. + +If the deployed client does not yet support Access, use the legacy `PRB_API_KEY` path only +when a credential is already configured locally and the maintainer confirms that endpoint +mode is enabled. If it is absent, stop and report that self-service authentication is not +deployed. Ask the maintainer to enable GitHub Access or issue an out-of-band per-user or +one-time intake credential; never request a shared long-lived key in chat. + +## 4. Confirm and upload once + +Before the external write, show the endpoint hostname, absolute file path, model, claimed +bug count, and test/official mode. State that the private certificate payload will leave the +machine, then obtain explicit confirmation. + +Use the repository client so validation and test marking stay consistent: + +```bash +python3 -m benchmark.submit --predictions --test # private test +python3 -m benchmark.submit --predictions # official +``` + +Do not retry an ambiguous timeout automatically: the first request may already have reached +R2 and a retry can create a duplicate. On HTTP 401/403, re-authenticate or report the missing +Access deployment. On HTTP 413, do not trim the evidence; report the size limit. On HTTP 429, +stop and report the rate limit. + +## 5. Report the handoff + +On success, report the returned `submission_id`, model, mode, and endpoint hostname. Make it +clear that `accepted` means privately queued, not scored or published. The submitter does not +trigger Actions, inspect R2, or open the leaderboard PR; those are maintainer responsibilities. + +Never claim a score until the maintainer confirms the private scorer finished. Preserve the +local submission and logs until that confirmation. diff --git a/.agents/skills/submit-benchmark-result/agents/openai.yaml b/.agents/skills/submit-benchmark-result/agents/openai.yaml new file mode 100644 index 0000000..46895c9 --- /dev/null +++ b/.agents/skills/submit-benchmark-result/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Submit Benchmark Result" + short_description: "Validate and upload a benchmark result securely" + default_prompt: "Use $submit-benchmark-result to validate and submit my benchmark result." diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b6216ba..1170303 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -182,7 +182,13 @@ Submission is a **CLI upload** — no web form, and the file never enters git. C - upload an intake test with `--test` (privately scored, excluded from the leaderboard); - upload an official submission with neither flag. -For either upload, get the endpoint URL and token from the maintainer, then: +Use `$submit-benchmark-result` when the result already exists. It validates the authoritative +file, preserves test/official intent, handles the authentication mode actually deployed by +the intake, asks before the external write, and reports the opaque submission ID. Submitters +do not need repository write access, R2 access, or permission to run the scoring workflow. + +The current legacy deployment uses an endpoint URL plus an intake credential configured +locally. Do not paste the credential into chat or commit it: ```bash export PRB_SUBMIT_URL= # from the maintainer diff --git a/README.md b/README.md index c52e719..2842f46 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,9 @@ Add its direct dispatch case to `_run_backend()` in `benchmark/run_submission.py The backend is supported only after its adapter tests pass and `harness-evaluation.json` reports `verdict: reliable`; command success alone is not enough. -A run is packaged as a `submission.json` (see `benchmark/submission.schema.json`) and uploaded with `python -m benchmark.submit`. See [CONTRIBUTING.md](CONTRIBUTING.md). +A run is packaged as a `submission.json` (see `benchmark/submission.schema.json`). Use +`$submit-benchmark-result` to validate and upload an existing result without repository or +R2 access. See [CONTRIBUTING.md](CONTRIBUTING.md). During evaluation, counterexamples use a different, agent-only command: diff --git a/benchmark/tests/test_docs.py b/benchmark/tests/test_docs.py index c02ff56..7e290ef 100644 --- a/benchmark/tests/test_docs.py +++ b/benchmark/tests/test_docs.py @@ -14,6 +14,7 @@ ENV_EXAMPLE = REPO_ROOT / "submission.env.example" API_SKILL = REPO_ROOT / ".agents/skills/run-api-benchmark/SKILL.md" CLI_SKILL = REPO_ROOT / ".agents/skills/run-cli-benchmark/SKILL.md" +SUBMIT_SKILL = REPO_ROOT / ".agents/skills/submit-benchmark-result/SKILL.md" def _text(path: Path) -> str: @@ -78,3 +79,27 @@ def test_each_skill_exposes_three_submission_goals(self, skill): assert "keep and validate" in t assert "intake test" in t assert "official submission" in t + + @pytest.mark.parametrize("skill", [API_SKILL, CLI_SKILL]) + def test_run_skills_delegate_upload(self, skill): + assert "$submit-benchmark-result" in _text(skill) + + +class TestSubmitSkill: + def test_submit_skill_exists(self): + assert SUBMIT_SKILL.exists(), "submit-benchmark-result skill missing" + + def test_submit_skill_validates_before_upload(self): + t = _text(SUBMIT_SKILL) + assert "python3 -m benchmark.submit" in t + assert "--dry-run" in t + assert "--test" in t + assert "explicit confirmation" in t + + def test_submit_skill_keeps_github_credentials_out_of_intake(self): + t = _text(SUBMIT_SKILL) + prose = " ".join(t.split()) + assert "github-backed cloudflare access" in t + assert "gh auth token" in t + assert "personal access token" in t + assert "self-service authentication is not deployed" in prose diff --git a/intake/cloudflare-worker/README.md b/intake/cloudflare-worker/README.md index 30ca648..e7ed286 100644 --- a/intake/cloudflare-worker/README.md +++ b/intake/cloudflare-worker/README.md @@ -50,6 +50,21 @@ python3 -m benchmark.submit --predictions out/submission.json --test Remove `--test` only when the run is ready to become an official leaderboard submission. +### Authentication direction + +`PRB_API_KEY` is the legacy bootstrap mode. For multiple submitters, prefer placing this +Worker behind [Cloudflare Access](https://developers.cloudflare.com/cloudflare-one/access-controls/applications/choose-application-type/), +using [GitHub as the identity provider](https://developers.cloudflare.com/cloudflare-one/integrations/identity-providers/github/), +and authorizing the intended GitHub organization, team, or accounts. Cloudflare supports +protecting a Worker directly, including its `workers.dev` route, and end users can obtain an +application-scoped CLI session with `cloudflared access login`. + +Do not accept a user's `gh auth token` or GitHub personal access token. Before removing the +legacy key, update the submit client to send the application-scoped Access token and make the +Worker validate the signed `Cf-Access-Jwt-Assertion` issuer and audience as documented by +Cloudflare. Until both pieces are deployed, the repository skill reports GitHub self-service +authentication as unavailable rather than silently weakening authentication. + ## R2 credentials for the scoring worker The GitHub Actions scorer reads the bucket via the S3 API. In the Cloudflare dashboard → From a55139978dbbd44606174c7285c42345f7f452c5 Mon Sep 17 00:00:00 2001 From: Xiwei Pan Date: Sun, 19 Jul 2026 14:22:22 +0800 Subject: [PATCH 2/2] Avoid printing Access tokens during submission --- .agents/skills/submit-benchmark-result/SKILL.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/.agents/skills/submit-benchmark-result/SKILL.md b/.agents/skills/submit-benchmark-result/SKILL.md index 70e966a..86a9ccf 100644 --- a/.agents/skills/submit-benchmark-result/SKILL.md +++ b/.agents/skills/submit-benchmark-result/SKILL.md @@ -55,14 +55,15 @@ Require `PRB_SUBMIT_URL` from the repository documentation or maintainer. Never to paste any token into chat, print an environment variable, or commit credentials. Prefer GitHub-backed Cloudflare Access when the deployed client documents -`PRB_ACCESS_TOKEN` or `--access-token`: +`PRB_ACCESS_TOKEN`: 1. Require `cloudflared` locally. -2. Run `cloudflared access login `; the user completes the configured - GitHub login in a browser. -3. Obtain the application-scoped Access token with `cloudflared access token - -app=` and pass it only through the client-supported environment - variable or flag. Do not display or persist it. +2. Obtain the application-scoped token only inside the confirmed upload command, for example + `PRB_ACCESS_TOKEN="$(cloudflared access token -app=)" `. + `cloudflared` opens the configured GitHub login in a browser when needed. +3. Never run `cloudflared access login` or `cloudflared access token` standalone because + some versions print the JWT. Pass it through the client-supported environment variable + for that one process only; do not display, persist, or put it in a command-line flag. Do not substitute `gh auth token`, a GitHub personal access token, or `GITHUB_TOKEN`; the intake must never receive the user's general GitHub credential.