Skip to content

README.md

Latest commit

 

History

History
158 lines (118 loc) · 7.76 KB

README.md

File metadata and controls

158 lines (118 loc) · 7.76 KB
Raw

diffbisect

You left a big uncommitted diff and a test now fails. Somewhere in those 400 lines are the few that broke it.

git bisect can't help — it needs commits, and it can't see inside one, let alone into a dirty working tree. diffbisect works below commit granularity: it delta-debugs the diff between a base rev and your working tree, splits it into change regions, and tells you which hunks to revert.

npx @precisionutilityguild/diffbisect -- npm test

That's it. No config, no daemon, no API key, no LLM, no Python. diffbisect only reads the exit code of the command after --, so vitest, jest, node --test, pytest, and cargo test all work unmodified. (mocha needs a one-line wrapper — see pkg/README.md.) Deterministic given a deterministic test command.

A region is a contiguous run of changed lines — the unit diffbisect turns on and off. Regions are what the output calls hunks; renames, mode changes, binary files, and untracked additions are each one atomic unit.

What a run looks like

Captured output, with H2–H7 elided for length. The demo repo has nine working-tree edits: seven benign refactors and one seeded two-region breakage — primary() and backup() were each flipped to false, and route() === "ok" only fails when both are present. That's the hard case (interference, not a lone bad line), and the default mode isolates it exactly.

$ diffbisect -- node check.mjs
diffbisect · "node check.mjs" · base HEAD
  9 regions · 26 probes (59 logged incl. memoized)

regions
  H1  math.js:1-4  [region]
  ...                              (H2–H7 elided; all nine print in the full run)
  H8  math.js:35-39  [region]
  H9  math.js:40-45  [region]

culprits (revert these): H5 H8

causes
  #1  H5 H8  (1-minimal witness)

complete: reverting the culprits from the full diff passes (confirmed by a passing probe).

Revert H5 and H8, rerun, done. That last line is not a claim: diffbisect ran the command on a worktree with H5 and H8 reverted and observed exit 0. The full transcript with all nine hunk diffs, and the printed patch for each culprit, is in pkg/README.md.

Reading the output

Three words carry the result:

  • culprits — the revert set. Remove exactly these hunks and your tree passes.
  • causes — the independent breakages. One 400-line diff can carry more than one unrelated regression, so each is reported separately with a witnessMinimal flag.
  • cluster — hunks that interfere too much to carve apart within the budget. They land here rather than being force-fit into a false "minimal" answer. Treat it as a suspect list.

If your tests need node_modules (read this one)

This is the trap that catches everyone. Each probe runs in a fresh git worktree, which shares the object database but not untracked files — node_modules/, build output, and anything .gitignored is absent. A bare npm test fails to start, so the base-precondition probe FAILs and diffbisect exits 2 with the base tree does not PASS, before any bisection begins.

The fix is the wrapper git bisect run users already write: symlink the untracked artifacts into each probe's worktree first.

diffbisect -- sh -c '[ -e node_modules ] || ln -sfn /path/to/repo/node_modules node_modules; exec npm test'

/path/to/repo is your real checkout; the [ -e ... ] || guard makes it idempotent across probes. Same applies to any generated artifact the tests need — a dist/, a build cache, a binary.

Flags

Flag Default Meaning
--json off Emit the machine-readable contract.
--base <rev> HEAD Base rev to diff the working tree against.
--target <rev> — Bisect the committed range <base>..<rev> instead of the working tree.
--max-probes <n> 500 Probe budget. Exhaustion is a clean exit 3, not an error.
--confirm <n> 1 Confirmation reruns for the first FAIL and revert-confirming probes (0 disables).
--minimal off Classic delta debugging end-to-end: a 1-minimal witness per cause, guaranteed but slower.
--fast off A single fast pass only; may return a residue cluster (explicitly labelled).
--jobs <n> 1 Speculative parallel probes; wall-clock only, byte-identical log to --jobs 1.
--keep-worktrees off Leave probe worktrees on disk (debug).
-- <cmd> required Everything after -- is the test command; its exit code is the oracle.

Exit codes

Your test command is the oracle, and diffbisect reads it using git bisect run's convention:

Command exit Meaning to diffbisect
0 PASS
1–127 (except 125) FAIL
125 UNRESOLVED (skip / untestable)
≥128 signal death — abort the whole run

diffbisect's own exit codes are distinct — branch on these, not on output text:

Code Meaning
0 Culprit set isolated, cluster empty.
1 Runtime/internal failure: not a git repo, worktree setup died, or the command was killed by a signal.
2 Usage/precondition error: bad flags, no command, or the base tree does not PASS / the full diff does not FAIL.
3 Residue: the probe budget was exhausted or an interdependency cluster remains. The partial result still prints.

What this is NOT

Everything nearby works on commits. diffbisect works on your dirty tree.

  • git bisect bisects commits, and cannot look inside one.
  • git-bifurcate has a --strategy hunk, but stays commit-anchored and is Python.
  • lockbisect / depbisect bisect lockfile entries or dependency versions across commits.
  • Codecov / diff-test-coverage gate lines changed between commits; they don't isolate which hunk broke a test.

Limitations

  • Interdependent hunks can exhaust the budget into a cluster. Non-monotone interference is the normal case for real diffs; violations degrade minimality, never correctness.
  • Flaky tests degrade to UNRESOLVED. --confirm protects the search from an inconsistent probe; it is not a flake detector. A truly flaky suite will not isolate cleanly.
  • Binary, rename, and mode units are atomic, as are untracked new files. Windows is unverified.
  • Your tree and index are never touched. Every probe runs in a fresh git worktree.

More

  • Full contract — the --json fields, the probe exit-code convention, the mocha wrapper caution, and the exit-2 / exit-3 transcripts: pkg/README.md.
  • Release: v0.1.0.

Lineage: Zeller & Hildebrandt, Simplifying and Isolating Failure-Inducing Input, IEEE TSE 2002 (https://doi.org/10.1109/32.988498).

Part of a family

Four tools, one idea: answers grounded in what your tests actually execute — evidence an agent can't hallucinate. Pick by the question you're holding:

Your question Tool
"A test is failing — which line is the bug?" culprits — spectrum fault localization (npm)
"One test is failing — which function returns the wrong value?" apd — algorithmic debugging, LLM-as-oracle (npm)
"Where is feature X implemented?" recon — feature location by coverage diff (npm)
"My uncommitted diff broke the tests — which hunks?" diffbisect — delta debugging below commit granularity (npm)

MIT © Precision Utility Guild