Add archive extraction benchmarks and adversarial-input tests - #2117
Draft
NickJosevski wants to merge 2 commits into
Draft
Add archive extraction benchmarks and adversarial-input tests#2117NickJosevski wants to merge 2 commits into
NickJosevski wants to merge 2 commits into
Conversation
Follow-up to the two gaps identified while reviewing the SharpCompress 0.49.1 upgrade (SF-1864), neither of which was a blocker for that change but both of which were left unmeasured. Calamari.Benchmarks (new, BenchmarkDotNet, not run in CI): - ConsolidatedPackageBenchmarks covers ExtractCalamariPackage, giving the first measurement of the ~6s-vs-~500ms observation that was never confirmed or dismissed. Can be pointed at a real consolidated package via CALAMARI_BENCHMARK_CONSOLIDATED_PACKAGE, otherwise generates a synthetic one of the same shape. - PackageExtractionBenchmarks covers the SharpCompress-backed extractors, so a future version bump can be measured rather than reasoned about. The project deliberately takes no direct SharpCompress reference; it tracks whatever Calamari.Common resolves. ArchiveRobustnessFixture characterises the adversarial vectors the suite did not reach: high entry counts, deep nesting, long entry names, and truncated or corrupt archives rather than "not an archive at all". Two of those tests document behaviour that is arguably wrong, and say so rather than asserting an aspiration: - A truncated .tar or .tar.gz extracts what it can, returns normally, and reports a file count lower than the archive held. Neither format has a trailing index, so an incomplete package deploys with no error raised. - An uncompressed .tar carries no checksum over entry data, so corrupted payloads are written through undetected. Inherent to the format. Both are recorded so a change in behaviour is visible; fixing them is a behaviour change beyond characterising them.
NickJosevski
force-pushed
the
nj/archive-extraction-benchmarks-and-robustness
branch
from
August 6, 2026 05:37
9cc9614 to
daeaea6
Compare
The benchmark project is compiled by the solution build but never executed in CI, so nothing proved the harness still worked. Compile rot was covered; runtime rot was not. A broken builder would have surfaced during the next upgrade, when nobody wants to be debugging a benchmark. The fragile part is Support/, which builds the fixtures the benchmarks measure, so that is what is now tested: - SyntheticArchiveFixture (Calamari.Tests) — every format is extractable by the matching extractor, entries are nested rather than flat, and payloads do not compress away to nothing. - SyntheticConsolidatedPackageTests (Calamari.ConsolidateCalamariPackages .Tests) — the package is readable by the real ConsolidatedPackageFactory, every flavour and platform resolves its files, and shared files are stored once and referenced many times. Both test projects source-link the Support files with <Compile Include> rather than taking a ProjectReference, so BenchmarkDotNet does not become a dependency of packages that ship to test agents. Verified: no BenchmarkDotNet assemblies in either test output. Running the benchmarks in CI as a timed check was considered and rejected — shared agents make thresholds either useless or flaky, and there is no stored baseline to compare against. README says so now, instead of just asserting "not wired into CI". Also rebased onto the merged #2087, which required two fixes: WriterFactory.Open is WriterFactory.OpenWriter on 0.49.1, and bzip2 payload corruption now surfaces InvalidFormatException where 0.37.2 leaked IndexOutOfRangeException. The rest of the damaged-archive characterisation is unchanged across both versions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the two items flagged as "worth a ticket" in this review comment on #2087. Neither blocked that PR. Both were properties of the extraction code that were unmeasured before the SharpCompress bump and would have stayed unmeasured after it.
Rebased onto the merged #2087, so this measures the extractors as they now ship on SharpCompress 0.49.1. No production code changes.
1. Benchmarks (
source/Calamari.Benchmarks, BenchmarkDotNet)New project. Compiled by the solution build, but the benchmarks are not executed in CI — see "Why not run them in CI" below.
ConsolidatedPackageBenchmarks— the first measurement ofExtractCalamariPackage, so the ~6s-vs-~500ms observation from the original thread can finally be confirmed or dismissed. Point it at a real package withCALAMARI_BENCHMARK_CONSOLIDATED_PACKAGE=/path/to/Calamari.<hash>.zip; otherwise it generates a synthetic one of the same shape.PackageExtractionBenchmarks— the SharpCompress-backed extractors, so the next version bump can be measured rather than reasoned about.The two suites are not interchangeable.
ExtractCalamariPackageusesSystem.IO.Compression.ZipArchivefrom the BCL, not SharpCompress. A SharpCompress bump cannot move those numbers — that suite is the control, not the experiment. That means the ~6s observation and the SharpCompress upgrade were never actually related, which wasn't obvious to me before reading the code.Indicative numbers, Apple M-series, synthetic package, 4KB payloads:
LoadIndexExtractSingleFlavourPlatformExtractAllFlavourPlatforms(54 combos)Two things that fall out, neither addressed here:
ExtractCalamariPackagere-opens the archive and rebuilds aToDictionaryover every entry on each call, so per-call cost scales with total archive size rather than with the files that call wants. Across 54 flavour/platform combinations that is 54 full scans.I've deliberately left the production code alone; optimising it is a separate change with its own risk.
2.
ArchiveRobustnessFixtureCovers the adversarial vectors the suite didn't reach: high entry counts (1,000 vs the samples' single digits), 40-level nesting, 150-character entry names, and archives that are truncated or corrupt rather than merely "not an archive at all". 20 tests, ~3s.
These are characterisation tests. Where current behaviour is arguably wrong, they assert what happens today and say so, rather than asserting an aspiration and failing:
.zip.tar.bz2.tar.gz.tarA truncated
.taror.tar.gzextracts what it can, returns normally, and reports a file count lower than the archive held. Neither format has a trailing index or whole-archive checksum, so the reader can't distinguish a truncated archive from one that simply ended. A package that arrives incomplete deploys as a partial package with no error raised — Calamari has no independent record of how many entries it should have seen.An uncompressed
.tarwrites corrupted payloads through undetected. Inherent to the format — there's nothing to check the bytes against. Recorded so the exposure is explicit: of the formats Calamari accepts, plain.taris the one where silent content corruption is possible.Both are worth fixing, but fixing them is a behaviour change well beyond characterising them. The tests exist so that when someone does, or when an archive-library change alters it, it's visible.
I ran this table against both 0.37.2 and 0.49.1 and it is identical, with one improvement: bzip2 payload corruption surfaces
InvalidFormatExceptionon 0.49.1 where 0.37.2 leaked anIndexOutOfRangeExceptionfrom inside the decoder. The tests assert only that it fails, not the type — that's a library implementation detail.Why not run the benchmarks in CI
Worth stating explicitly, since "manual tool" is usually a smell:
tar.bz2at 2,000 files runs ~2.3s per iteration; the full suite is minutes, per build, for numbers nobody reads.The real risk that leaves is rot — the solution build catches compile breakage, but nothing proved the harness still runs. So the fragile part, the fixture builders in
Support/, is now covered by ordinary tests in the existing suite (SyntheticArchiveFixture,SyntheticConsolidatedPackageTests): every generated format is extractable by the matching extractor, and the generated consolidated package is readable by the realConsolidatedPackageFactory.Those test projects source-link the
Support/files rather than taking a ProjectReference, so BenchmarkDotNet doesn't become a dependency of packages that ship to test agents. Verified: no BenchmarkDotNet assemblies in either test output.Verification
ArchiveRobustnessFixture+SyntheticArchiveFixture+PackageExtractorFixture; 4/4 inSyntheticConsolidatedPackageTests.main, not caused by this branch:Calamari.ConsolidateCalamariPackages.Tests18/20 fail identically (needs a full Nuke build to have produced the packages), and 5 Docker/GitHub downloader tests fail identically (need credentials)..slnismerge=union;dotnet sln addreindented two unrelated project blocks as a side effect and I reverted that, so the diff there is 6 purely additive lines.