Add frozen, reproducible KEGG reaction benchmark (Phase 1) - #13
Open
janisshin wants to merge 5 commits into
Open
Add frozen, reproducible KEGG reaction benchmark (Phase 1)#13janisshin wants to merge 5 commits into
janisshin wants to merge 5 commits into
Conversation
…ifact The evaluation work existed only on an unmerged research branch, with no reproducible corpus behind its reported 68 models / 4,379 reactions. This rebuilds it as a frozen, checksum-verified dataset and records what the historical numbers actually were. Fix KEGG reaction URI matching. main matched only `kegg.reaction:R00024`, while BioModels overwhelmingly uses the slash form, so extraction returned zero ground truth on real files. Parametrised tests now cover slash, colon, and URN forms. Preserve every valid ground-truth ID rather than the first encountered. 91 reactions carry multiple IDs, one of them ten; scoring against a single arbitrary ID under-credits correct predictions. Separate pipeline failures from scientific exclusions. A missing download or unparseable file is a defect in our tooling and is reported apart from exclusions that are properties of the data. SSX reactions are retained as records and reduce reaction counts only, never model counts. Surface parser problems instead of absorbing them: per model, raw `kegg.reaction` mentions are compared against extracted identifiers. This showed BIOMD0000000579 has no reaction-level ground truth at all, only 13 species misannotated with reaction IDs, so it is a genuine data exclusion. Cluster near-duplicate models for leakage-safe splitting, using Jaccard plus containment gated on comparable size and minimum shared identifiers. Without containment the three Smallbone2013 yeast variants split apart; without the gates, clustering collapsed the corpus. Observed: 74 models, 5,838 ground-truth reactions, 5,816 evaluable. All 75 files verified against upstream SHA-256, all ten dataset invariants pass, and repeated builds are byte-identical. The 68 / 4,379 figures are not reproducible from this manifest under any defensible rule and were not engineered into; benchmark/data/RECONCILIATION.md documents the evidence, including that upstream files have not changed since the snapshot.
… digests verify after checkout on any platform
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR establishes Phase 1 of the AAAIM reaction-annotation benchmark: a frozen, checksum-verified dataset for evaluating KEGG reaction annotation under imperfect metabolite identifiers.
What this adds
This branch also integrates the reaction-evaluation changes from
upstream/test/no-rule-evals, including support for slash-, colon-, and URN-form KEGG identifiers, KEGG compound annotations, exchange/SSX detection, and ontology/cofactor ablation controls.Frozen benchmark snapshot
All 10 dataset invariants pass. Two consecutive builds produced byte-identical output across all frozen artifacts.
Historical-count reconciliation
The previously reported benchmark size of 68 models and 4,379 reactions could not be reproduced from the canonical 75-model manifest under a defensible exclusion rule. This PR documents and investigates the discrepancy rather than adjusting exclusions to force agreement.
The evidence indicates that the historical evaluation likely used a different or truncated model list. The current benchmark therefore freezes the fully reconciled observed dataset:
75 models = 74 included + 1 scientific exclusion + 0 pipeline failures5,838 reactions = 5,816 evaluable + 22 reaction-level exclusionsValidation
The Phase 1 regression suite contains 21 tests, and all tests pass.
The frozen
reactions.csvSHA-256 is: