Release: develop -> main#4364
Merged
Merged
Conversation
…er (#4341) * feat(storage): fail-closed Azure/S3 content-verify gate for the cutover Add scripts/storage/verify-content.ts: the byte-content companion to the (key, size) reconciler. It proves Azure and MinIO/S3 hold byte-identical content for a list of containers, closing the same-size content-divergence gap the reconciler deliberately leaves open. Every shared key is classified into exactly one proof class -- metadata-match (Azure Content-MD5 == single-part S3 ETag), backfill-covered (both sides older than a proven backfill cutoff), size-mismatch, or needs-hash -- and any key that cannot be proven is gate-red (fail-closed; a known MD5 mismatch is never treated as covered). REPORT mode is read-only; --hash-delta streams and SHA-256-compares the needs-hash set (rate-limited, capped, with a fixpoint re-list) and reports same-size content divergence as the most severe finding. Runs during the dual-write phase; the final proof run belongs in a write freeze before Azure teardown. backfill-covered requires an explicit operator assertion that the backfill was content-proven (rclone reported zero un-hashable blobs), never assumed. * fix(storage): bind the content-gate hash proof to object version, harden guards Address the review of the content-verify gate: - The hash proof was cached per key with no version binding, so an object overwritten (same size, different bytes) after being hashed was skipped by both the fixpoint loop and the final gate -- a false-green. The proof is now keyed by an object signature (Azure/S3 lastModified, size, S3 ETag); a re-list with a changed signature discards the stale proof and re-hashes. - Reject a BACKFILL_PROOF_CUTOFF set in the future, which would otherwise let virtually every current object pass as backfill-covered without a hash. - streamSha256 now verifies the hashed byte count against the known object size, so a silently truncated stream can no longer count as a full hash. - Soften the ETag=MD5 comments to state the (MinIO, single-part, no SSE) assumption; a non-MD5 ETag fails the match and falls through to hashing. - prefer-const and an explicit arrow return type. * fix(storage): key the content-gate proof on object ETags, not timestamps The hash-proof signature used second-granularity lastModified plus a silent s3.etag fallback, so an Azure-only overwrite (same size, same second) could keep the signature identical and let a stale proof survive -- a false-green. Both stores expose an ETag in their listings that advances on every write; the signature is now azureEtag|azureSize|s3Etag|s3Size, and a missing ETag on either side fails loud instead of defaulting. This invalidates a stale proof on any write to either side. Also: require BACKFILL_PROOF_CUTOFF to be strict ISO-8601; count stream bytes with Buffer.byteLength for string chunks; and export isStillProvenByHash so the version-binding (stale proof discarded on a changed ETag) is unit-tested. * fix(storage): bind the content-gate hash proof to the downloaded version Round-3 review hardening of the content-verify gate: - The proof signature was taken from the listing, but the hash was computed from a fresh download that could be a different version. Because an S3 ETag is a content hash (it can return to a prior value on a byte-revert), a stale signature could match a later divergent state -- a narrow false-green. The hash functions now return the download-response ETag, and a proof is stored only when both downloaded ETags equal the listed ones (assertHashVersionUnchanged); a version change during hashing is skipped and re-listed, never proven. - BACKFILL_PROOF_CUTOFF must now be UTC ISO-8601 ending in Z and calendar valid; a timezone-less value or a rolled-over invalid day (e.g. 2026-04-31) fails loud instead of silently shifting the cutoff. * docs(storage): state the content-gate's write-freeze requirement at the exit A green "CONTENT VERIFIED" result is only authoritative under a write freeze, because Azure and S3 are listed non-atomically -- an overwrite in the window between the two listings is not detectable. This was documented in the header but not at the point of the success output; both success messages now say so, so a green result from a live (unfrozen) dual-write run is not mistaken for an absolute proof. * fix(storage): normalize ETags in the content gate version guard A live rehearsal against an S3-compatible store and an Azure Blob emulator showed the --hash-delta path never converging: every object was treated as changed between listing and hashing, re-listed, and the fixpoint loop aborted with "Fixpoint loop exceeded MAX_FIXPOINT_ITERATIONS". The gate could therefore never go green, not even inside a write freeze. The cause was the version guard comparing raw ETags: the listing returns an unquoted value while the download returns the same value in double quotes. ETags are now normalized before comparison (optional weak-validator prefix, surrounding quotes, outer whitespace); a genuinely different ETag still fails closed, and the error message keeps reporting both raw values for diagnosis. * fix(storage): reject weak ETag validators instead of normalizing them away Review of the previous commit raised four points, all addressed here. The weak-validator prefix is no longer stripped. RFC 7232 section 2.1 states that a weak validator does not guarantee byte identity, which is exactly what this guard exists to prove, so equating a weak ETag with a strong one silently widened the set of values considered equal beyond what the reported bug required. A weak validator on either side now fails closed with an explicit message instead. Quotes are only removed as a matched pair, so two malformed values such as an unterminated leading quote and an unterminated trailing quote no longer collapse onto the same normalized value. The three separate quote-stripping snippets that had accumulated in this file are consolidated onto a single helper, leaving the single-part detection and the MD5 comparison semantically unchanged. Tests cover the edge cases that were previously untested: empty string, a lone quote, asymmetric quoting, weak validators on either or both sides, and multipart suffixes remaining distinguishable. * fix(storage): close content gate scope and log gaps
github-actions
Bot
requested review from
TaprootFreak and
davidleomay
as code owners
July 24, 2026 12:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automatic Release PR
This PR was automatically created after changes were pushed to develop.
Commits: 1 new commit(s)
Checklist