test: add offline ComBEE prompt contracts and reproduction guide - #43
Open
Frankie-Xu wants to merge 1 commit into
Open
Frankie-Xu wants to merge 1 commit into
Frankie-Xu wants to merge 1 commit into
Conversation
Frankie-Xu
marked this pull request as ready for review
September 18, 2026 07:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #41 by documenting and testing the current native prompt/operation contract. This does not resolve the missing official Terminal-Bench prompts, task split or experimental configuration, and does not claim to reproduce benchmark scores or fix a functional bug.
This adds offline tests of the real Reflector and Curator, plus a reproduction guide covering template sources/hashes, terminal evidence mapping and grouping/augmentation settings. Deterministic replacements at the model-call boundary exercise GT/no-GT formatting, missing-GT fallback, proposal/reducer JSON parsing, ADD-only filtering, playbook application and invalid responses. Ordinary curate's returned unsupported operations are explicitly distinguished from operations actually applied.
No production behavior changes, CaseGraph adapter code, prompt loader, or provider integration are included. Generic prompt loading remains the scope of #38. This draft leaves room to discuss which reproduction documentation belongs upstream.
Validation:
python -m unittest discover -s tests -v: 8 passed, zero skips, including an independent management rerun.No real LLM, embedding model, terminal harness, full ACEBatch scheduling run or upstream CI validation was performed. Tests use the small import dependency environment documented in the guide; optional embedding dependencies are not exercised.