feat(moq-hls): Producer/Consumer cursors for recording a broadcast#2389
Merged
Conversation
There was a problem hiding this comment.
Sorry @kixelated, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
kixelated
force-pushed
the
claude/moq-hls-record-cursors-fe9b
branch
4 times, most recently
from
July 18, 2026 12:43
25adc43 to
2ad9626
Compare
The export module was built to *serve* HLS on demand. A recorder that mirrors
every rendition and every segment to storage had to reach into raw signals
(a `watch::Receiver<usize>` of the rendition count, a per-rendition
`watch::Receiver<u64>` timeline epoch) and re-derive the whole reconcile + fetch
loop itself. Replace those with two cursors in the moq-net idiom, so the engine
drives the loop once and the consumer just pulls the next element:
- `renditions::{Producer, Consumer}` + `Event`: the Consumer yields one
`Added`/`Removed` at a time, replaying the current set as `Added` and ending
(`None`) when the source closes. A reconfigure is a `Removed` then `Added`.
The Producer owns the current set (reconciled by `sync`, moved off Broadcaster)
and the serve path reads it synchronously.
- `segments::{Producer, Consumer}` + `Segment`: rehomes `timeline.rs`. The
Producer is the timeline window (kio-backed, replacing the tokio watch); the
Consumer's `next()` awaits the next *finalized* segment, FETCHes and transmuxes
it, and yields the CMAF bytes with duration + PDT. Runs dry (`None`) on clean
end or reset.
The HTTP serve path keeps its pull API (`rendition`, `master_playlist`,
`playlist`, `segment`), now `pub(crate)` since only in-crate callers use it; the
long-poll waits via `Rendition::wait_playable`. No leaky `watch` types remain in
the public surface. Supersedes the raw-accessor approach in #2381.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kixelated
force-pushed
the
claude/moq-hls-record-cursors-fe9b
branch
from
July 18, 2026 12:44
2ad9626 to
e900c62
Compare
kixelated
enabled auto-merge (squash)
July 18, 2026 12:55
kixelated
added a commit
that referenced
this pull request
Jul 18, 2026
…blish API Merged latest dev; #2389's new tests used the one-argument create_broadcast and relied on an abrupt producer drop propagating immediately, which now lingers for a reconnect by design. Pass a live route and finish() the broadcast at the deliberate teardown, keeping the unfinalized-live-edge assertion intact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #2381 (which exposed raw observers); please review this instead.
Why
The export module is built to serve HLS on demand. A consumer that records a broadcast (mirrors every rendition + segment to storage) needs the same state the server has, but #2381 exposed it as raw internals:
Broadcaster::updated() -> watch::Receiver<usize>andRendition::updated() -> watch::Receiver<u64>put internal encodings (rendition count, timeline epoch) in the public API. The doc comment had to say "the value is the count, so go re-readrenditions()" -- a signal masquerading as a value.Arc::ptr_eq, a generation counter, and diffing successiveplaylist()snapshots to find newly-closed groups. Segment discovery is this crate's logic; having each consumer rebuild it is the bug factory.What
Two cursors in the moq-net idiom (a handle that owns its cursor and yields the next element;
None= done). Notry_next/poll_next/is_closed-- nothing needed them.renditions::{Producer, Consumer}replaces the snapshot getter + count-watch +closed()/is_closed(). The Producer owns the current set (the catalog reconcile moves offBroadcaster); the Consumer diffs against its own view, so it replays the current set asAddedand a reconfigure is aRemovedthen anAdded-- callers no longer do pointer-identity detection.segments::{Producer, Consumer}rehomestimeline.rs. The Producer is the timeline window, now backed bykio::Producerinstead of atokio::sync::watch(also in the direction of refactor(net)!: drop moq-net's direct tokio dependency #2377). The Consumer'snext()awaits the next finalized segment, FETCHes and transmuxes it through the same on-demand path the serve path uses (so no new standing traffic), and yields the CMAF bytes. Evicted segments are skipped and flagged viadiscontinuity-- the caller stops re-deriving gaps. A transient fetch error is returned without advancing the cursor, so the caller can retry the same segment.The serve path keeps its pull API unchanged in behavior, but it is now gated behind the
serverfeature (only in-crate callers use it), so a recording-only build's surface is just the cursors. Nowatchtypes remain in the public API.Waiting is timeout-free -- bounding a wait is the caller's policy, not the primitive's, so
Broadcaster::ready()andRendition::playable()just resolve and the serve path wraps them in its owntokio::time::timeout. Those follow moq-net's sync-predicate/async-wait pairing (is_closed()/closed()):Rendition::is_playable()is the bool,playable()is the wait.Validation
cargo test -p moq-hls44/44 and clippy clean with theserverfeature on, and clean as a dependency with it off. Includes a new end-to-end test driving both cursors (renditions -> segments -> media bytes -> run dry).Consumer-side, this was written against a real rewrite of moq.pro's VOD recorder: it deletes that recorder's ptr_eq reconcile, generation-based re-enumeration, and playlist-snapshot diffing (16 recorder + 31 package tests green). That port also surfaced a latent issue worth calling out: on a reconfigure the predecessor rendition's cursor does not end, because its source media continues -- the old snapshot-diff loop had been depending on the timeline ending to release its state. Consumers that hand off state across a reconfigure need to drive that themselves.
Risk
Additive for recording; the serve path's behavior is unchanged (its methods only changed visibility/feature-gating).
timeline.rs->segments.rsis a rehome, with its window/eviction logic and tests carried over intact.(written by Opus 4.8)