Skip to content

fix: post-merge follow-ups — demux hardening, edit lists, paced timing, load identity#5

Merged
dooly123 merged 6 commits into
BasisVR:mainfrom
towneh:fix/mp4-demux-hardening
Jul 8, 2026
Merged

fix: post-merge follow-ups — demux hardening, edit lists, paced timing, load identity#5
dooly123 merged 6 commits into
BasisVR:mainfrom
towneh:fix/mp4-demux-hardening

Conversation

@towneh

@towneh towneh commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up batch from review feedback on #1, #2 and #4 that reached me outside the PR threads after they merged. One branch as agreed, a commit per concern — five source commits, smallest first:

Malformed-input hardening. Remote MP4 reaches this parser (HLS/CMAF segments, VOD URLs), so stream-supplied fields now get the full treatment: tfhd/tfdt/trun reads are bounded to their box, a trun's declared per-sample table has to fit inside the box (with a sane per-fragment sample cap), the box walkers' size checks are rewritten in subtraction form so a size near INT_MAX can't overflow the comparison, run positions and sample-fit checks are 64-bit, and tick→µs conversion goes through a quotient/remainder helper instead of v * 1000000 / timescale (which overflows at crafted timescales). Also: ctts version 0 offsets are now read unsigned per spec, stz2 surfaces a clear unsupported-format error instead of starting playback with that track silently missing, and the worst-case Annex B output size moved into a shared basis_avcc_annexb_cap used by both MP4 paths and RTMP — RTMP previously allocated dlen + 64, which undersizes 1–3-byte NAL length prefixes and silently dropped the access unit.

Cleanups. Dead next_dts field, a stale fMP4-only header comment, a nested /* that warned on every TU.

Track edit lists (progressive). edts/elst were ignored, so a track delayed by an initial empty edit — common muxer output for A/V alignment — played that much early, permanently desynced. The classic walk now maps media time onto the movie timeline: empty edits become a presentation delay, and the first normal edit's media_time becomes the media-time origin (this also picks up ffmpeg's AAC priming edit, so audio/video line up sample-accurately). Fragmented MP4 is untouched — its timeline comes from tfdt.

Load identity scoped to every source replacement (C#, from #2's feedback). LoadGeneration advanced only in LoadUrl, so the resolver staleness guard couldn't see a newer load that arrived through LoadLocalPath, a direct LoadSource or a CPU Source assignment — it now advances on every source replacement. The LoadUrl metadata seed is keyed to the generation that armed it (a seed abandoned by a resolver-less or failed load no longer attaches to the next unrelated load), LoadSource no longer writes the metadata origin back into the caller's BasisMediaSource (it's retained player-side, trusted only while the same instance reloads with an unchanged Uri), Metadata returns a snapshot so external code can't mutate the live instance past OnMetadataChanged, and assigning a CPU Source clears the previous media's metadata instead of leaving it reported as current. The resolver contract is unchanged — capture LoadGeneration, resolve, compare, LoadSource — so the existing yt-dlp integration works as-is.

Pace delivery on decode time, not presentation time. The substantive one. Delivery pacing gated on the pts of whatever the demux thread submitted last, so a video AU whose composition offset exceeds the ~400 ms pacing lead slept the thread past its own decode turn — and the other track's earlier samples, queued behind it on the same thread, arrived late enough for the audio queue to trim them as stale. The video sink now carries the decode timestamp alongside pts; the gate holds on dts while the decoder and caption store keep receiving pts. Demuxers without decode timestamps (TS, RTSP) pass pts for both, which is exactly the old behaviour; RTMP forwards the FLV tag timestamp it already had. The progressive walk had the file-order variant of the same stall — a coarsely interleaved file (a chunk of video ahead of the matching audio) blocked the forward reader on paced video before it could reach the audio bytes — so samples whose file order runs ahead of their delivery turn now park in bounded per-track queues (32 MB; past the bound they deliver anyway, degrading to the old behaviour rather than buffering without limit).

Verification

Closed-loop demux harness (the same one used for #1/#4), old vs new:

  • fMP4 CMAF concat regression: 360 video AUs / 564 audio frames, worst delivery-order A/V gap ~187 ms — unchanged.
  • Progressive fixtures (a generated 12 s file and a real VRCDN recording): full demux, same counts; first audio pts is now −21.3 ms, which is the AAC priming edit being honoured rather than a regression.
  • New edit-list fixture (audio delayed 2 s via an empty edit, ffmpeg -itsoffset 2 … -c copy): audio now delivers at 1.978 s (2 s minus priming) — previously it played at 0 and sat 2 s early forever. The verbose dump shows early audio parking in the queue and releasing in decode order while video streams.
  • Trailing-moov error and video-only fixtures behave as before.

Windows DLL (MSVC 19.51) and Android arm64 .so (NDK clang 18) rebuilt from this branch.

Editor-validated (Windows desktop, Unity 6000.5.3f1, this branch consumed as the project's media player package):

  • Live RTSPT regression — plays as before.
  • CMAF HLS fixture — video with audio at real-time pacing (the path most affected by the decode-time gate).
  • Progressive VOD (a real ~100 MB recording) — paced, end-to-end, A/V in sync.
  • The edit-list fixture — audio joins at ~2 s as the elst dictates and plays clean to the end; previously it started at 0.
  • Load-identity race — a direct URL loaded while a YouTube resolve was in flight wins and keeps its title (the stale resolve stands down); Reload keeps the title.

Console clean throughout.

Notes and limitations

  • One residual load-identity gap is documented rather than closed: a direct LoadSource racing a resolver that already failed silently, with no other load in between, can still consume that load's seed. Closing it needs the continuation to carry an explicit token through LoadSource's signature, which didn't seem worth the API churn while the plain call has to stay resolver-compatible — the wrong origin lasts only until anything real loads.

  • Gating on dts can release an AU up to its composition offset earlier than the old pts gate; typical offsets are a small fraction of the decode ring's span.

  • Explicit tfhd.base_data_offset is still ignored (pre-existing — fragment placement derives from relative trun offsets). Kept out to keep this reviewable; it needs the moof's absolute stream offset threaded through.

  • The harness and fixtures still live out of tree; porting them into the repo as a proper test target is a sensible follow-up if you want the repo to carry tests.

Heads-up: this conflicts with #3's rebuilt .so by construction; I'll rebase #3 and rebuild it once this lands.

towneh added 6 commits July 8, 2026 20:46
Remote MP4 reaches this parser (HLS/CMAF segments, VOD URLs), so treat
every stream-supplied field as hostile:

- parse_traf: validate tfhd/tfdt/trun field reads against the box body,
  and reject a trun whose declared per-sample table does not fit in the
  box (or exceeds a sane per-fragment sample cap). Unchecked counts
  could previously drive reads past the buffered moof.
- Box walkers: rewrite child-size checks in subtraction form
  (sz > len - off) so a crafted size near INT_MAX cannot overflow the
  signed addition and bypass the bound.
- consume_mdat / consume_progressive: widen run positions to int64 and
  make the sample fit checks overflow-safe; cap progressive sample
  sizes at the box cap so a crafted stsz entry cannot drive a
  multi-gigabyte allocation.
- Timestamps: convert ticks to microseconds via quotient/remainder
  (ticks_to_us) instead of value * 1000000 / timescale, which overflows
  at stream-controlled timescales.
- ctts: honour the FullBox version - version 0 offsets are unsigned,
  version 1 signed. Previously all offsets were cast signed.
- stz2: detect the compact sample-size table and surface a clear
  unsupported-format error instead of starting playback with the
  affected track silently missing.
- stsd: bound the mp4a fixed-header reads to the sample entry size.
- Annex B sizing: move the worst-case AVCC-to-Annex-B output size into
  a shared helper (basis_avcc_annexb_cap) and use it from both MP4
  paths and RTMP. RTMP previously allocated dlen + 64, which undersizes
  streams using 1-3 byte NAL length prefixes and silently dropped the
  access unit.
Remove the never-read next_dts track field, update the fMP4-only header
comment (the demuxer has handled progressive MP4 since the classic-table
work), refresh the top-of-file assumptions to match, and break the
nested '/*' in basis_bitstream.h that warned on every translation unit.
Progressive playback previously ignored edts/elst, so every track started
at presentation time zero regardless of its edit list. A track delayed by
an initial empty edit (common muxer output for A/V alignment) played that
much early, permanently desynced from the other track.

Parse mvhd for the movie timescale and each track's elst, then map media
time onto the movie timeline at emission: initial empty edits become a
presentation-time delay, and the first normal edit's media_time becomes
the media-time origin (encoder priming / initial trim). Samples ahead of
the origin keep a negative presentation time rather than being dropped,
since video there can still carry reference frames; multi-segment edits
beyond the first normal entry are out of scope for a linear walk.

Fragmented MP4 is unaffected - its timeline comes from tfdt.
Delivery pacing gated on the presentation timestamp of whatever the demux
thread submitted last. A video AU whose composition offset puts its pts
past the pacing lead therefore slept the demux thread out beyond its own
decode turn - and any earlier samples of the other track queued behind it
on the same thread arrived late enough for the audio queue to trim them
as stale. Streams reordering less than the ~400 ms lead masked this.

Carry the decode timestamp through the video sink alongside pts and gate
on it, while the decoder and caption store keep receiving pts. Demuxers
without decode timestamps (TS, RTSP) pass pts for both, which is the old
behaviour; RTMP forwards the FLV tag timestamp it already had.

The progressive MP4 walk had the file-order variant of the same stall: a
muxer that interleaves chunks coarsely (a chunk of video ahead of the
matching audio) blocked the forward reader on paced video before it could
reach the audio bytes at all. Samples whose file order runs ahead of
their delivery turn now park in bounded per-track queues and deliver in
decode-time order across tracks; past the 32 MB bound (or once the mdat
has nothing more to read) the earliest parked sample delivers regardless,
degrading to the previous behaviour instead of buffering without limit.

The fragmented path keeps its decode-time merge and hands the merge key
to the sink as the decode timestamp.
basis_media_native.dll (win-x64, MSVC 19.51) and libbasis_media_native.so
(android arm64-v8a, NDK clang 18) built from this branch.
The load-identity plumbing around metadata had three gaps, all variants
of 'which load does this async continuation belong to':

- LoadGeneration advanced only in LoadUrl, so a stale resolver
  continuation could pass its staleness guard after a LoadLocalPath,
  a direct LoadSource or a CPU Source assignment replaced the source.
  It now advances on every source replacement, whichever entry point.
- The LoadUrl metadata seed was a bool that stayed armed when routing
  ended without a load (no resolver installed, or a resolver that
  errored before LoadSource), so the next unrelated load inherited the
  abandoned URL as its metadata origin. The seed is now keyed to the
  generation that armed it and is retired on the no-resolver path and
  by every non-continuation entry point.
- LoadSource wrote the resolved metadata origin back into the caller's
  BasisMediaSource, where it went stale if the caller mutated the
  descriptor's Uri and loaded it again. The origin is now retained
  player-side and only trusted while the same instance reloads with an
  unchanged Uri; the caller's descriptor is no longer touched.

Also: Metadata now returns a snapshot (and OnMetadataChanged delivers
one), so external code can't mutate the player's live instance past the
change event - ApplyMetadata is the mutation path. Assigning a CPU
Source clears the previous media's metadata (event fires with null)
instead of leaving it reported as current.
@towneh towneh changed the title fix: harden MP4/RTMP demux and pace delivery on decode time fix: post-merge follow-ups — demux hardening, edit lists, paced timing, load identity Jul 8, 2026
@dooly123 dooly123 merged commit 883f2b5 into BasisVR:main Jul 8, 2026
@towneh towneh deleted the fix/mp4-demux-hardening branch July 9, 2026 00:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants