Model downloads: a rate limit is said out loud, a shared env build survives a cancel, and sizes stop disagreeing

[iamsdas]

Summary

A user reported downloading two models and seeing only one move — the other sat at 0. Downloads already run in parallel (a probe proved it), so this fixes the things that made a working parallel download look stuck, and makes the numbers on the page honest.

• A Hugging Face rate limit is now waited out and said on the row. A 429 used to be indistinguishable from a dead link: it spent the segment's whole fault budget in about seven seconds and handed the repo to hf's own snapshot_download, which is slower — so a throttled user watched a download crawl, having been throttled and never told. Throttles now get their own allowance, separate from the fault budget, and the row reads "Hugging Face is limiting this download", with a sign-in prompt when no token is stored.
• The wait comes from the header the Hub actually sends. HF signals a limit with RateLimit: "resolvers";r=0;t=87 — the IETF scheme — not Retry-After. Parsing only Retry-After meant guessing the wait on every real Hub throttle. Both are read now, RateLimit first, capped per-wait at THROTTLE_WAIT_MAX_S and in aggregate at THROTTLE_TOTAL_MAX_S (600s) so a host that 429s forever still gives up rather than hanging a job indefinitely.
• A 429 on the resolve call is now recognised too. _hub_file_meta raises a requests-shaped HfHubHTTPError, not urllib.error.HTTPError, so a throttle on the metadata hop bypassed the new branch entirely — and that hop is the one the Hub actually meters (quota is counted per URL carrying a /resolve/ segment; our ranged GETs go to the presigned CDN and cost nothing). Both call sites funnel through _resolved_meta, which reads either exception shape without importing requests into this stdlib-only module.
• Cancelling one download no longer kills an environment build the others are waiting on. Worker.install_key names a shared single-flight install, so cancelling on the key alone tore down the build every other download of that runner had joined — and before an interpreter is pinned that key is machine-global, so the blast radius was every runner. Installs are refcounted now: the last waiter out is the one that cancels, and the count is written under _lock so a cancel and a _terminate can't interleave.
• A download parked behind someone else's env build says so, instead of reading "Preparing Fake — sync…" identically to the one doing the work — which was indistinguishable from a row that had died.
• A live download's own total replaces the catalog's advertised size. This is the "68 GB out of 64 GB" report: the two numbers were never in conflict, they were two different numbers shown side by side — a hand-written catalog.py estimate next to a row counting real listing bytes. One rule now decides, in one unit, at every site: never understate. A live total larger than the advertised figure means a stale constant and the row wins; a smaller one is either one phase of a multi-part fetch or a conservative constant, and quoting it would promise a cheaper download than the user will get.
• A single file is now capped at 500 chunks. CHUNK_BYTES becomes a floor: chunk = max(CHUNK_BYTES, ceil(size / 500)). Comfy-Org/MiniMax-H3 has single files up to 66.3GB, which planned 1,976 segments in one file and 14,057 across the repo — and every segment's cursor lives in a sidecar rewritten whole every second, so that download serialised ~2,000 dicts a second per file in flight, for hours. This is a bookkeeping measure and the comment says so out loud: chunk count consumes no Hub quota. Files below ~16.8GB are chunked exactly as before; SIDECAR_VERSION goes to 3 because a v2 sidecar for a larger file is internally consistent and wrong, which only a version number can reject.

Not changed: no download queue and no concurrency cap were added. Downloads are already concurrent; the perceived serialisation was the shared env build (fixed here), the per-process connection split, and throttling (also fixed here).

Visuals

What a segment does with an HTTP error — the throttle branch is the delta.

flowchart TD
    A[Segment gets an HTTP error] --> B{429, or 503<br/>with a wait header?}
    B -->|no| C[Count a fault<br/>backoff, retry]
    C --> D{Fault budget spent?}
    D -->|yes| E[Fall back to<br/>snapshot_download]
    B -->|yes| F[Say it on the job row]
    F --> G[Sleep in slices,<br/>separate allowance]
    G --> A

Loading

Before this PR the yes branch did not exist: a 429 took the no path, exhausted the fault budget, and reached the fallback silently.

Usage

Nothing new is user-invocable — these are behaviour changes on paths already in use.

• Throttling: start a model download while over a Hub quota. The row reads Hugging Face is limiting this download — waiting 42s, or …— sign in to Hugging Face in Preferences → AI for a higher limit when no token is stored. The sign-in it points at is the existing device-code flow (D402); a 429 from a FUSED_MODEL_MIRROR host says only This download is being rate-limited, since neither the Hub nor a Hub sign-in is involved.
• Shared env build: start two downloads for the same runner before its venv exists, then cancel either one. The other keeps building. Previously the cancelled one took it down with the install was cancelled.
• Joined install: the same two downloads — the joiner's row reads Waiting for the <runner> environment — another download is building it…. No percentage, because nothing here knows how far another worker's uv sync has got.
• Sizes: any AI Models or Playground surface showing a size. While a download runs, its measured total shows if it exceeds the catalog figure; otherwise the catalog estimate — hedged with ~ on the Local tab's recommended card, which is the one surface that has always carried the hedge.
• Chunking: transparent, except that an in-flight .fusedpart for a file over ~16.8GB will be re-planned rather than resumed (sidecar v2 → v3).

Test Plan

• tests/test_ai_hub_fetch.py — throttle recognition (_is_throttled's 503-needs-a-wait-header distinction), RateLimit parsing (t=, malformed, absent → fall back to Retry-After), all four Retry-After forms (delta-seconds, HTTP-date, naive-as-GMT, past-as-zero), per-wait and aggregate caps, cancel-during-a-throttle, mirror-vs-Hub wording, sign-in text only without a token, a 429 on the resolve hop, and the regression: SEGMENT_ATTEMPTS * 4 consecutive 429s now complete the download (on stashed code it failed with exactly gave up at byte 0 — HTTP 429). Plus chunk-plan cases driving _chunks with sizes as parameters — nothing allocated — covering the floor, the ceiling, the 16.8GB meeting point, and the SEGMENT_MIN_BYTES short-circuit.
• tests/test_env_install.py — start() reports claimed on the spawn path only.
• tests/test_ai_runtime.py — new shared_install fixture (real _ensure_venv, faked envinstall whose cancel marks the record done-with-error like the real one, claim-once/join-after). Either side's cancel leaves the other's install alive; the joiner's row carries the waiting detail; a re-hold landing before the cancel call stops it. All were red before the fix (assert ['shared-key'] == [], and 'Preparing Fake — sync…').
• frontend/src/apps/ai_models/shared/modelSize.test.ts — cases over liveModelTotal / modelSizeLabel / modelSizeHint / catalogSizeBytes, mutation-checked.
• 40 consecutive stress runs of the install-cancel mechanism, zero related failures — the original flake reproduced at roughly 1 in 11.
• Full local suite: failure-ID set identical to main in both directions (this repo's macOS baseline is not zero — see the caveats below).
• bun run typecheck clean and bun test src/apps/ai_models/shared/modelSize.test.ts green after merging origin/main (D428 reshaped this exact file).
• CI green.
• Manual: reviewer reproduces the throttle row against a rate-limited Hub account, and the two-download cancel above.

Known residuals, stated rather than hidden

• The cancel-then-instantly-reload window is narrowed, not closed. Between dropping the last refcount and calling envinstall.cancel, a new waiter can re-hold the key; a second lock hold re-checks and bails, which shrinks the window to a few bytecodes but does not remove it. Closing it properly needs a generation token at the envinstall layer — deliberately out of scope. Worst case is one spurious the install was cancelled on a fast cancel-then-retry.
• A card now reads 4.3 GB where catalog.py writes 4.6. The catalog's decimal GB is converted to bytes and formatted base-1024 like every other size on the page, including the progress row it sits beside. The card and its own progress bar agreeing was chosen over agreeing with the constant.
• The Playground polls job rows again. D428's sidebar rewrite removed this page's jobs polling along with the progress bar it fed; the size rule needs a live pull's total, so the poll is back — gated on isBusy(runtime), so it runs only while something is actually live and clears its state otherwise. The progress bar was not resurrected.
• /api/env/install gains a "claimed" field on the record. Additive; no client reads it.
• test_a_refused_body_this_cannot_frame_ends_the_connection is a pre-existing ConnectionResetError flake (~17 of 25 runs, a different parametrisation each time) that reproduces on main. Not from this branch, not fixed here.

🤖 Generated with Claude Code

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
fused_render
envinstall.py					1629, 1656, 1745-1749
fused_render/ai
supervisor.py					930
fused_render/ai/runners
worker_base.py					1164, 1202-1203, 1205, 1209, 1797
Project Total

This report was generated by python-coverage-comment-action

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.