feat(media): add sharded media object-key paths - #4533
Draft
bradseiler wants to merge 10 commits into
Draft
Conversation
added 6 commits
August 3, 2026 10:13
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Classify sharded payload keys in storage sweeps, preserve physical totals, and deduplicate logical legacy/sharded copies. Export read resolution, fallback, and duplicate-layout metrics. Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Keep self-hosted upgrades on legacy writes by default, expose the write-layout gate through Helm and Compose, and document the explicit legacy-to-dual-to-sharded migration sequence. Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Consolidate read and write policy behind one upgrade-safe phase and ship guarded maintenance binaries for backfill and legacy cleanup. Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
This was referenced Aug 3, 2026
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: Brad Seiler <seiler@squareup.com>
…igration * origin/main: (95 commits) fix(desktop): virtualize channel member lists (#4991) fix(desktop): enforce owner-only access in internal builds (#4053) test(desktop): match attachment button label (#4993) fix(acp): pace observer telemetry at 1/s with per-channel batch envelopes (#4917) fix(desktop): enable the content security policy (#4614) fix(mobile): merge relay recounts with locally seen thread replies (#4633) fix(desktop): enable message editing in Inbox (#2198) relay: fuzz WebSocket 1012 restart-close timing on graceful drain (BUZZ_DRAIN_JITTER_MS) (#4542) fix(desktop): outline the selected community (#4969) fix(desktop): clamp thread panel to channel surface (#4965) style(messages): increase username contrast (#4948) fix(desktop): rename generic attachment action from 'Attach image' to 'Attach file' (#2381) (#4304) fix(reactions): support max-length custom emoji (#3833) feat(desktop): allow leaving your final community (#3621) fix(buzz-agent): recover from context-window 400s instead of sticking (#4946) docs(persona-pack): fix stale desktop import instructions (#4500) fix(desktop): route macos notification clicks (#4799) feat(mobile): sync themes per community (#3767) feat(desktop): sync themes per community (#3653) feat(desktop): cap OpenClaw agent parallelism at 5 (#4019) ... Signed-off-by: Brad Seiler <seiler@squareup.com>
bradseiler
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Media payload keys are currently flat, so high-volume traffic lacks hash-leading S3 prefix entropy. Existing deployments also need a safe, explicit migration that preserves old reads and rollback behavior without changing storage behavior merely by upgrading.
What
media/<sha[0:2]>/<sha[2:4]>/<community>/<filename>while leaving Git/CAS namespaces unchangedBUZZ_MEDIA_MIGRATION_PHASEwith four complete read/write policies:legacy-only(default): read and write only legacy paths, preserving existing deployment behaviordual-read-legacy-writedual-read-dual-writesharded-onlysharded-onlybefore their first upload, avoiding any future backfill or legacy cleanupbuzz-media-layout-backfillandbuzz-media-layout-delete-legacyin both relay image targets, with bounded paging, checkpoints, request-rate limiting, idempotency, destination verification, dry-run, and destructive confirmationValidation
cargo test -p buzz-media --all-targets: 119 passed; live-MinIO test ignoredcargo test -p buzz-relay --lib config::tests::: 29 passedcargo test -p buzz-relay --lib storage_sweep: 15 passedcargo clippy -p buzz-media -p buzz-relay --all-targets -- -D warnings: cleancargo fmt --all -- --check, desktop Tauri fmt, andgit diff --check: cleanorigin/main: Rust, desktop, Tauri, and mobile suites passedGenerated with Goose