Skip to content

test: add offline ComBEE prompt contracts and reproduction guide - #43

Open
Frankie-Xu wants to merge 1 commit into
ace-agent:mainfrom
Frankie-Xu:test/combee-prompt-contract
Open

Frankie-Xu wants to merge 1 commit into
ace-agent:mainfrom
Frankie-Xu:test/combee-prompt-contract

Conversation

@Frankie-Xu

Copy link
Copy Markdown

Addresses #41 by documenting and testing the current native prompt/operation contract. This does not resolve the missing official Terminal-Bench prompts, task split or experimental configuration, and does not claim to reproduce benchmark scores or fix a functional bug.

This adds offline tests of the real Reflector and Curator, plus a reproduction guide covering template sources/hashes, terminal evidence mapping and grouping/augmentation settings. Deterministic replacements at the model-call boundary exercise GT/no-GT formatting, missing-GT fallback, proposal/reducer JSON parsing, ADD-only filtering, playbook application and invalid responses. Ordinary curate's returned unsupported operations are explicitly distinguished from operations actually applied.

No production behavior changes, CaseGraph adapter code, prompt loader, or provider integration are included. Generic prompt loading remains the scope of #38. This draft leaves room to discuss which reproduction documentation belongs upstream.

Validation:

  • python -m unittest discover -s tests -v: 8 passed, zero skips, including an independent management rerun.
  • Same suite passed from a clean git archive; compileall and diff-check passed.
  • Temporary in-memory mutation checks detected a wrong no-GT template and disabled ADD filtering; restored tests passed.

No real LLM, embedding model, terminal harness, full ACEBatch scheduling run or upstream CI validation was performed. Tests use the small import dependency environment documented in the guide; optional embedding dependencies are not exercised.

@Frankie-Xu
Frankie-Xu marked this pull request as ready for review September 18, 2026 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant