Skip to content

Add frozen, reproducible KEGG reaction benchmark (Phase 1) - #13

Open
janisshin wants to merge 5 commits into
sys-bio:mainfrom
janisshin:benchmark/phase-1
Open

Add frozen, reproducible KEGG reaction benchmark (Phase 1)#13
janisshin wants to merge 5 commits into
sys-bio:mainfrom
janisshin:benchmark/phase-1

Conversation

@janisshin

Copy link
Copy Markdown
Member

Summary

This PR establishes Phase 1 of the AAAIM reaction-annotation benchmark: a frozen, checksum-verified dataset for evaluating KEGG reaction annotation under imperfect metabolite identifiers.

What this adds

  • A canonical 75-model BioModels manifest and reproducible downloader
  • SHA-256 verification of every downloaded SBML model against BioModels
  • Deterministic benchmark-generation scripts and versioned output tables
  • Reaction ground truth that preserves every valid KEGG identifier per reaction
  • Explicit separation of scientific exclusions from pipeline failures
  • Diagnostics for malformed, misplaced, and unparsed KEGG annotations
  • Near-duplicate model clustering to prevent train/test leakage
  • Regression tests covering extraction, exclusions, clustering, and determinism
  • Documentation reconciling the observed benchmark with the historical reported counts

This branch also integrates the reaction-evaluation changes from upstream/test/no-rule-evals, including support for slash-, colon-, and URN-form KEGG identifiers, KEGG compound annotations, exchange/SSX detection, and ontology/cofactor ablation controls.

Frozen benchmark snapshot

  • 75 manifest accessions downloaded, parsed, and checksum-verified
  • 74 models included
  • 1 model scientifically excluded
  • 0 pipeline failures
  • 5,838 ground-truth reaction records
  • 5,816 evaluable reactions
  • 19 exchange/SSX exclusions
  • 3 malformed-identifier exclusions
  • 91 reactions with multiple valid KEGG identifiers
  • 1,906 distinct KEGG reaction identifiers
  • 54 model clusters, including 9 multi-member clusters

All 10 dataset invariants pass. Two consecutive builds produced byte-identical output across all frozen artifacts.

Historical-count reconciliation

The previously reported benchmark size of 68 models and 4,379 reactions could not be reproduced from the canonical 75-model manifest under a defensible exclusion rule. This PR documents and investigates the discrepancy rather than adjusting exclusions to force agreement.

The evidence indicates that the historical evaluation likely used a different or truncated model list. The current benchmark therefore freezes the fully reconciled observed dataset:

  • 75 models = 74 included + 1 scientific exclusion + 0 pipeline failures
  • 5,838 reactions = 5,816 evaluable + 22 reaction-level exclusions

Validation

python benchmark/scripts/download_biomodels.py
python benchmark/scripts/build_benchmark.py
python -m pytest tests/test_benchmark_build.py

The Phase 1 regression suite contains 21 tests, and all tests pass.

The frozen reactions.csv SHA-256 is:

f11ebc6f4d0734deb14c054caf277b81695b22668017d70af9c43cd6146004f1

janisshin and others added 5 commits September 1, 2026 12:26
…ifact

The evaluation work existed only on an unmerged research branch, with no
reproducible corpus behind its reported 68 models / 4,379 reactions. This
rebuilds it as a frozen, checksum-verified dataset and records what the
historical numbers actually were.

Fix KEGG reaction URI matching. main matched only `kegg.reaction:R00024`,
while BioModels overwhelmingly uses the slash form, so extraction returned
zero ground truth on real files. Parametrised tests now cover slash, colon,
and URN forms.

Preserve every valid ground-truth ID rather than the first encountered. 91
reactions carry multiple IDs, one of them ten; scoring against a single
arbitrary ID under-credits correct predictions.

Separate pipeline failures from scientific exclusions. A missing download or
unparseable file is a defect in our tooling and is reported apart from
exclusions that are properties of the data. SSX reactions are retained as
records and reduce reaction counts only, never model counts.

Surface parser problems instead of absorbing them: per model, raw
`kegg.reaction` mentions are compared against extracted identifiers. This
showed BIOMD0000000579 has no reaction-level ground truth at all, only 13
species misannotated with reaction IDs, so it is a genuine data exclusion.

Cluster near-duplicate models for leakage-safe splitting, using Jaccard plus
containment gated on comparable size and minimum shared identifiers. Without
containment the three Smallbone2013 yeast variants split apart; without the
gates, clustering collapsed the corpus.

Observed: 74 models, 5,838 ground-truth reactions, 5,816 evaluable. All 75
files verified against upstream SHA-256, all ten dataset invariants pass, and
repeated builds are byte-identical. The 68 / 4,379 figures are not
reproducible from this manifest under any defensible rule and were not
engineered into; benchmark/data/RECONCILIATION.md documents the evidence,
including that upstream files have not changed since the snapshot.
… digests verify after checkout on any platform
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant