Skip to content

Add archive extraction benchmarks and adversarial-input tests - #2117

Draft
NickJosevski wants to merge 2 commits into
mainfrom
nj/archive-extraction-benchmarks-and-robustness
Draft

Add archive extraction benchmarks and adversarial-input tests#2117
NickJosevski wants to merge 2 commits into
mainfrom
nj/archive-extraction-benchmarks-and-robustness

Conversation

@NickJosevski

@NickJosevski NickJosevski commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Follow-up to the two items flagged as "worth a ticket" in this review comment on #2087. Neither blocked that PR. Both were properties of the extraction code that were unmeasured before the SharpCompress bump and would have stayed unmeasured after it.

Rebased onto the merged #2087, so this measures the extractors as they now ship on SharpCompress 0.49.1. No production code changes.

1. Benchmarks (source/Calamari.Benchmarks, BenchmarkDotNet)

New project. Compiled by the solution build, but the benchmarks are not executed in CI — see "Why not run them in CI" below.

ConsolidatedPackageBenchmarks — the first measurement of ExtractCalamariPackage, so the ~6s-vs-~500ms observation from the original thread can finally be confirmed or dismissed. Point it at a real package with CALAMARI_BENCHMARK_CONSOLIDATED_PACKAGE=/path/to/Calamari.<hash>.zip; otherwise it generates a synthetic one of the same shape.

PackageExtractionBenchmarks — the SharpCompress-backed extractors, so the next version bump can be measured rather than reasoned about.

The two suites are not interchangeable. ExtractCalamariPackage uses System.IO.Compression.ZipArchive from the BCL, not SharpCompress. A SharpCompress bump cannot move those numbers — that suite is the control, not the experiment. That means the ~6s observation and the SharpCompress upgrade were never actually related, which wasn't obvious to me before reading the code.

Indicative numbers, Apple M-series, synthetic package, 4KB payloads:

Benchmark 50 files/platform 200 files/platform
LoadIndex 8.0 ms 32.1 ms
ExtractSingleFlavourPlatform 1.8 ms 7.4 ms
ExtractAllFlavourPlatforms (54 combos) 98 ms / 54 MB alloc 317 ms / 218 MB alloc

Two things that fall out, neither addressed here:

  • ExtractCalamariPackage re-opens the archive and rebuilds a ToDictionary over every entry on each call, so per-call cost scales with total archive size rather than with the files that call wants. Across 54 flavour/platform combinations that is 54 full scans.
  • Real payloads are far larger than 4KB, so production numbers scale up from here. Nothing yet contradicts the ~6s report — this gives us the means to check it.

I've deliberately left the production code alone; optimising it is a separate change with its own risk.

2. ArchiveRobustnessFixture

Covers the adversarial vectors the suite didn't reach: high entry counts (1,000 vs the samples' single digits), 40-level nesting, 150-character entry names, and archives that are truncated or corrupt rather than merely "not an archive at all". 20 tests, ~3s.

These are characterisation tests. Where current behaviour is arguably wrong, they assert what happens today and say so, rather than asserting an aspiration and failing:

Format Truncated Corrupt entry payload
.zip throws throws
.tar.bz2 throws throws
.tar.gz silent partial extract throws
.tar silent partial extract written through undetected

A truncated .tar or .tar.gz extracts what it can, returns normally, and reports a file count lower than the archive held. Neither format has a trailing index or whole-archive checksum, so the reader can't distinguish a truncated archive from one that simply ended. A package that arrives incomplete deploys as a partial package with no error raised — Calamari has no independent record of how many entries it should have seen.

An uncompressed .tar writes corrupted payloads through undetected. Inherent to the format — there's nothing to check the bytes against. Recorded so the exposure is explicit: of the formats Calamari accepts, plain .tar is the one where silent content corruption is possible.

Both are worth fixing, but fixing them is a behaviour change well beyond characterising them. The tests exist so that when someone does, or when an archive-library change alters it, it's visible.

I ran this table against both 0.37.2 and 0.49.1 and it is identical, with one improvement: bzip2 payload corruption surfaces InvalidFormatException on 0.49.1 where 0.37.2 leaked an IndexOutOfRangeException from inside the decoder. The tests assert only that it fails, not the type — that's a library implementation detail.

Why not run the benchmarks in CI

Worth stating explicitly, since "manual tool" is usually a smell:

  • These are wall-clock, disk-touching measurements on shared agents across Windows, Ubuntu and ARM64. A regression threshold would be either too loose to catch anything or flaky enough that people mute it, and a muted check is worse than none.
  • There's no stored baseline to compare against, so a CI run would print numbers into a build log nobody diffs.
  • Cost: tar.bz2 at 2,000 files runs ~2.3s per iteration; the full suite is minutes, per build, for numbers nobody reads.

The real risk that leaves is rot — the solution build catches compile breakage, but nothing proved the harness still runs. So the fragile part, the fixture builders in Support/, is now covered by ordinary tests in the existing suite (SyntheticArchiveFixture, SyntheticConsolidatedPackageTests): every generated format is extractable by the matching extractor, and the generated consolidated package is readable by the real ConsolidatedPackageFactory.

Those test projects source-link the Support/ files rather than taking a ProjectReference, so BenchmarkDotNet doesn't become a dependency of packages that ship to test agents. Verified: no BenchmarkDotNet assemblies in either test output.

Verification

  • Full solution builds clean (0 errors) on 0.49.1.
  • 90/90 in ArchiveRobustnessFixture + SyntheticArchiveFixture + PackageExtractorFixture; 4/4 in SyntheticConsolidatedPackageTests.
  • All 28 benchmarks execute end to end.
  • Pre-existing failures confirmed against pristine main, not caused by this branch: Calamari.ConsolidateCalamariPackages.Tests 18/20 fail identically (needs a full Nuke build to have produced the packages), and 5 Docker/GitHub downloader tests fail identically (need credentials).

.sln is merge=union; dotnet sln add reindented two unrelated project blocks as a side effect and I reverted that, so the diff there is 6 purely additive lines.

Follow-up to the two gaps identified while reviewing the SharpCompress
0.49.1 upgrade (SF-1864), neither of which was a blocker for that change
but both of which were left unmeasured.

Calamari.Benchmarks (new, BenchmarkDotNet, not run in CI):

- ConsolidatedPackageBenchmarks covers ExtractCalamariPackage, giving the
  first measurement of the ~6s-vs-~500ms observation that was never
  confirmed or dismissed. Can be pointed at a real consolidated package
  via CALAMARI_BENCHMARK_CONSOLIDATED_PACKAGE, otherwise generates a
  synthetic one of the same shape.
- PackageExtractionBenchmarks covers the SharpCompress-backed extractors,
  so a future version bump can be measured rather than reasoned about.

The project deliberately takes no direct SharpCompress reference; it
tracks whatever Calamari.Common resolves.

ArchiveRobustnessFixture characterises the adversarial vectors the suite
did not reach: high entry counts, deep nesting, long entry names, and
truncated or corrupt archives rather than "not an archive at all".

Two of those tests document behaviour that is arguably wrong, and say so
rather than asserting an aspiration:

- A truncated .tar or .tar.gz extracts what it can, returns normally, and
  reports a file count lower than the archive held. Neither format has a
  trailing index, so an incomplete package deploys with no error raised.
- An uncompressed .tar carries no checksum over entry data, so corrupted
  payloads are written through undetected. Inherent to the format.

Both are recorded so a change in behaviour is visible; fixing them is a
behaviour change beyond characterising them.
@NickJosevski
NickJosevski force-pushed the nj/archive-extraction-benchmarks-and-robustness branch from 9cc9614 to daeaea6 Compare August 6, 2026 05:37
The benchmark project is compiled by the solution build but never
executed in CI, so nothing proved the harness still worked. Compile rot
was covered; runtime rot was not. A broken builder would have surfaced
during the next upgrade, when nobody wants to be debugging a benchmark.

The fragile part is Support/, which builds the fixtures the benchmarks
measure, so that is what is now tested:

- SyntheticArchiveFixture (Calamari.Tests) — every format is extractable
  by the matching extractor, entries are nested rather than flat, and
  payloads do not compress away to nothing.
- SyntheticConsolidatedPackageTests (Calamari.ConsolidateCalamariPackages
  .Tests) — the package is readable by the real ConsolidatedPackageFactory,
  every flavour and platform resolves its files, and shared files are
  stored once and referenced many times.

Both test projects source-link the Support files with <Compile Include>
rather than taking a ProjectReference, so BenchmarkDotNet does not become
a dependency of packages that ship to test agents. Verified: no
BenchmarkDotNet assemblies in either test output.

Running the benchmarks in CI as a timed check was considered and
rejected — shared agents make thresholds either useless or flaky, and
there is no stored baseline to compare against. README says so now,
instead of just asserting "not wired into CI".

Also rebased onto the merged #2087, which required two fixes:
WriterFactory.Open is WriterFactory.OpenWriter on 0.49.1, and bzip2
payload corruption now surfaces InvalidFormatException where 0.37.2
leaked IndexOutOfRangeException. The rest of the damaged-archive
characterisation is unchanged across both versions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant