Skip to content

Bring experimental.readers.video to feature parity with legacy readers.video - #6513

Draft
jantonguirao wants to merge 23 commits into
NVIDIA:mainfrom
jantonguirao:worktree-video-reader-parity
Draft

jantonguirao wants to merge 23 commits into
NVIDIA:mainfrom
jantonguirao:worktree-video-reader-parity

Conversation

@jantonguirao

@jantonguirao jantonguirao commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Category:

New feature (non-breaking change which adds functionality)

Description:

Brings nvidia.dali.experimental.readers.video (CPU+GPU, built on the shared frames_decoder_*
infrastructure) to feature parity with the legacy nvidia.dali.fn.readers.video (GPU-only,
NVDEC-based), and then closes its remaining performance gap so it's a safe, faster default.
This closes the gaps identified in a side-by-side analysis of the two readers:

  • dtype (UINT8/FLOAT) and normalized output, matching legacy's float/normalized support.
  • channels argument, validated against the decoder's actual channel count.
  • additional_decode_surfaces argument to tune the GPU decode-lookahead/reorder buffer.
  • require_constant_frame_rate opt-in strict VFR check (default unchanged — still permissive).
  • A new fused-resize operator, experimental.readers.video_resize, modeled on legacy's
    readers.video_resize (composes ResizeBase<GPUBackend> on top of the reader).
  • Two file_list rounding bugs fixed: file_list_include_end was a no-op clamp instead of
    actually inclusive, and file_list_format="frames" with an omitted/zero end selected zero
    frames instead of "through the end of the video" (matching legacy's convention). The
    file_list_include_end schema default is false, matching legacy's default behavior.
  • file_list_rounding's default changed from "start_down_end_up" to "start_up_end_down", to
    match legacy's default rounding convention for file_list_format="timestamps" (DALI-4917) —
    previously the two readers selected different frame ranges from the same file_list even with
    no explicit rounding override on either side.
  • CPU backend now supports image_type=YCbCr (DALI-4916): fixed a crash (sws_scale was only
    given a packed destination buffer for what it was told is planar AV_PIX_FMT_YUV444P output)
    and a colorimetry bug found immediately after (destination range was hardcoded full-range,
    wrong for YCbCr's limited/"TV"-range convention — CPU output now matches the GPU backend within
    a small rounding tolerance, not just avoiding the crash).
  • GPU backend now supports MJPEG (DALI-4918): the codec was already mapped in the decoder
    plumbing but excluded from the allow-list pending follow-up work. Enabling it surfaced why —
    the NVDEC driver reports only 1 decode surface for MJPEG content, and this backend's
    synchronous-inline picture-display handling (no real pfnDisplayPicture callback) needs more
    headroom than that to recycle surfaces correctly. Fixed by forcing a larger, empirically-verified
    surface count for MJPEG specifically.
  • Along the way, fixed two more latent bugs found during implementation: FrameSize() (an element
    count) was used as a byte stride for FLOAT/multi-frame decode; and is_vfr_ wasn't restored
    by FramesDecoderBase::SetIndex() on FrameIndexCache hits, so a VFR check would have silently
    never fired after the first read of a file.

Performance follow-up (DALI-4924)

With feature parity in place, experimental.readers.video was still ~14% slower than legacy
under the common training-pipeline shape (random_shuffle=True over more files than fit in a
batch). Three changes close that gap:

  1. Overlap NVDEC decode with color-conversion via deferred surface unmap. The GPU reader
    synchronized the whole CUDA stream and unmapped every decoded picture immediately after its
    color-conversion kernel launch, once per frame, serializing NVDEC decode against the
    conversion kernel every frame. Replaced the per-frame sync+unmap with a deferred unmap: map,
    launch the conversion kernel, record a CUDA event, and move on without syncing.
    cuvidUnmapVideoFrame only runs once a surface's slot is actually needed again, via FIFO
    eviction bounded by the driver's real ulNumOutputSurfaces. (An earlier, unlanded attempt at
    deferring unmaps didn't respect this cap and corrupted frames; this fix stays single-threaded
    and enforces the cap explicitly.)
  2. Cache open decoders instead of rebuilding on every file switch. Prefetch() tore down and
    rebuilt the entire FramesDecoderGpu/FramesDecoderCpu (container reopen, NVDEC parser
    creation, decode-surface allocation, keyframe-relative seek) every time consecutive samples
    came from a different file — with shuffling, that's nearly every sample. Replaced the single
    decoder_ with a small filename-keyed LRU (capacity 8) so a file's decoder survives across the
    samples it isn't currently servicing.
  3. Drop unmapped pre-roll frames. HandlePictureDisplay() mapped and color-converted every
    decoded picture that didn't exactly match the next wanted frame, even when that picture could
    never be a future seek target (e.g. the pre-roll frames NVDEC must decode to reach a mid-GOP
    seek position, which have a pts before anything the index will ever request again). Those
    frames are now dropped immediately instead of paying for cuvidMapVideoFrame, the conversion
    kernel, and a frame_buffer_ slot that would never be read. Frames ahead of the target (e.g.
    B-frame reordering) still take the existing buffering path.

Benchmarked with internal_tools/video_reader_bench.py (new; mirrors
internal_tools/hw_decoder_bench.py's CLI/reporting conventions so it can be wired into an
equivalent qa/TL1_video_reader_perf job) on a TITAN V, batch=8, sequence_length=16, 5 shuffled
VP9 files:

before perf work after legacy readers.video
throughput 295.49 fps 360.06 fps 342.38 fps
conversion-kernel launches per requested frame 1.82x 1.09x 1.09x

experimental.readers.video now outperforms the legacy reader on this benchmark. Changes 2 and 3
above were benchmarked together, not in isolation, so this doesn't attribute the gain between
them individually.

Each cached decoder holds its own NVDEC lease and ~10 decode-surface buffers, so the cache trades
some GPU memory (roughly 200MB at 720p/uint8 per cached decoder, ~4x that for float) for avoiding
the rebuild cost.

Explicitly out of scope (deferred to a later effort): renaming/promoting the experimental schema
to a non-experimental name, Deprecate() of the legacy schema, deletion of legacy code, expanding
CPU codec coverage beyond what already existed (MJPEG is the one exception, since it was already
documented as CPU+GPU supported and the GPU side just wasn't wired up), and a possible further
"decode-ahead" pipelining change (mapping frame N-k instead of frame N right after decoding it) —
not needed once the above closed the gap, but noted as headroom if a future workload needs more.

Additional information:

Affected modules and functionalities:

  • dali/operators/video/reader/video_reader_decoder_op.{h,cc} — new arguments, file_list
    rounding fixes and default changes, split into a header + implementation (previously a single
    .cc); per-filename decoder LRU cache (performance follow-up).
  • dali/operators/video/reader/video_reader_decoder_resize_op.{h,cc} — new fused-resize operator.
  • dali/operators/video/frames_decoder_base.{h,cc}, frames_decoder_gpu.{h,cc} — normalized
    float output plumbing, configurable decode-surface count, FrameSizeBytes() fix,
    is_vfr_/SetIndex() fix, MJPEG decode-surface-count fix; deferred NVDEC surface unmap and
    drop-behind-target frame handling (performance follow-up).
  • dali/operators/video/frames_decoder_cpu.cc — YCbCr crash + colorimetry fix.
  • dali/operators/video/reader/video_reader_decoder_op_test.cc,
    dali/operators/video/legacy/reader/video_reader_op_test.cc — incidental gtest fixes (typo'd
    sharding args that silently disabled sharding verification; a "GPU" shuffle test that actually
    ran the CPU backend; an invalid sharding config the typo fix exposed; the shared MJpeg test now
    builds a per-operator OpSpec instead of unconditionally skipping experimental).
  • dali/test/python/test_video_reader.py, test_video_reader_resize.py — new/extended coverage
    for all of the above, plus closing an existing # TODO: Add types.YCbCr gap in the
    legacy-vs-experimental comparison harness.
  • internal_tools/video_reader_bench.py — new benchmark script (legacy vs. experimental reader
    throughput comparison; performance follow-up).

Key points relevant for the review:

  • The file_list_include_end and file_list_rounding default changes restore parity with
    legacy's defaults; anyone currently passing either argument explicitly is unaffected.
  • video_reader_decoder_op.h is a new header extracted from the previously header-less
    video_reader_decoder_op.cc, needed so the new resize operator can subclass
    VideoReaderDecoder<GPUBackend> — verified to be a behavior-preserving pure move.
  • FramesDecoderGpu::SetOutputType is now virtual and resizes its internal frame-reorder buffer
    only when FLOAT output is actually requested, so UINT8 callers (including legacy's
    decoders.video/VideoInput, which share this decoder) keep their original memory footprint.
  • The MJPEG decode-surface-count fix (AdjustedNumDecodeSurfaces) only affects
    cudaVideoCodec_JPEG; every other codec's surface count is unchanged (pass-through).
  • HandlePictureDisplay's eviction loop enforces the driver's real ulNumOutputSurfaces cap
    before mapping a new picture, so the deferred-unmap change can't map more surfaces at once than
    the driver actually supports.
  • The decoder LRU cache changes decoder_ from an owning unique_ptr to a non-owning pointer
    into a std::list of cached instances. FramesDecoderGpu's destructor already drains pending
    NVDEC surface maps before its pooled decoder lease returns to NVDECCache, so LRU eviction (as
    opposed to end-of-run teardown) goes through the same safe path. NVDECCache is explicitly
    designed to support multiple concurrently-open leases of the same codec config, so having up to
    8 decoders open at once is not a new usage pattern for it.
  • The pre-roll drop only fires for frames strictly behind the next wanted frame (current_pts_ < index_[NextFrameIdx()].pts); frames ahead of the target (e.g. B-frame reordering) are
    unaffected and still buffered.

Tests:

  • Existing tests apply
    • dali_operator_test --gtest_filter='*Video*:*FramesDecoder*' (90 tests)
    • test_video_reader.py, test_video_reader_resize.py (178 tests, including the
      legacy-vs-experimental parity/byte-comparison tests)
  • New tests added
    • Python tests
    • GTests
    • Benchmark (internal_tools/video_reader_bench.py)
    • Other

Checklist

Documentation

  • Documentation updated
    • Docstring
    • Doxygen
    • RST
    • Jupyter
    • Other

DALI team only

Requirements

  • Implements new requirements
  • Affects existing requirements
  • N/A

REQ IDs: N/A

JIRA TASK: DALI-4915, DALI-4916, DALI-4917, DALI-4918, DALI-4924

….video

Exposes Task 1's FramesDecoderBase::SetOutputType()/SetNormalizedRange()
plumbing as dtype/normalized operator arguments on
experimental.readers.video, plus a validated channels argument that must
match the decoder's actual (always 3) channel count. dtype=FLOAT is
GPU-only; CPU decode remains UINT8-only and rejects dtype=FLOAT explicitly.

Also fixes a byte-offset bug uncovered while wiring multi-frame FLOAT
output through FramesDecoderGpu: several places in frames_decoder_base.cc
and frames_decoder_gpu.cc computed frame strides/copy sizes with
FrameSize() (element count), which is only correct for 1-byte UINT8
output. Added FrameSizeBytes() (element count * output element size) and
used it for inter-frame pointer strides, the internal frame reorder
buffer sizing, and frame-to-frame/constant-frame device copies, so
multi-frame FLOAT decoding (sequence_length > 1) produces correctly
aligned output. dtype=FLOAT combined with pad_mode='constant' is
rejected for now since the constant-frame fill buffer is still
UINT8-only.
…constant restriction

Finding 1: FramesDecoderGpu's internal frame-reorder buffer was unconditionally
sized for the largest supported output element (float), a permanent 4x GPU
memory increase for every caller (including legacy fn.readers.video and the
new reader's UINT8 default), regardless of dtype. Made SetOutputType()
virtual and overrode it in FramesDecoderGpu to resize the already-allocated
buffer entries to match the actual dtype once known; the buffer is initially
allocated assuming DALI_UINT8 (unchanged from before this task), so UINT8
callers keep their original, smaller footprint and only DALI_FLOAT callers
pay the larger allocation.

Finding 2: documented the dtype=FLOAT + pad_mode='constant' restriction (added
in the previous commit) in both the `dtype` and `pad_mode` schema docstrings,
instead of leaving it discoverable only via the runtime error.
Opt-in strict check (default false, current permissive VFR decoding
unchanged) that fails fast via DALI_ENFORCE when a video is variable
frame rate, using FramesDecoderBase::IsVfr().

Also fixes a real correctness gap found while wiring this up:
FramesDecoderBase::SetIndex() restored index_/num_frames_ from a
cached FrameIndexCache entry but never recomputed is_vfr_, so IsVfr()
silently reported false for any decoder that reused a cached index
(the common case, since the loader's PrepareMetadataImpl warms the
cache for every file at build time). SetIndex() now calls
DetectVariableFrameRate() so IsVfr() is correct on both the fresh
BuildIndex() path and the cache-hit path.
Two behavior changes for existing experimental.readers.video users:
1. file_list_include_end=True now actually includes the end frame (previously only
   clamped to the video's frame count, a no-op in most cases).
2. file_list_format="frames" with an omitted/zero end column now means "through the
   end of the video", matching readers.video's file_list_frame_num=True convention,
   instead of selecting zero frames.
test_compare_experimental_to_legacy_reader_file_list never overrides
file_list_include_end, so experimental.readers.video now (correctly, after
8a8ea99) extends frame-mode ranges by one frame per its documented default.
Legacy has no equivalent concept for file_list_format="frames" (its end column
is always a literal exclusive bound), so 3 of the 32 random parametrizations can
diverge. Skip those known cases with an explanation instead of leaving
unexplained red.
Pure move of VideoReaderDecoder, VideoLoaderDecoder and their supporting
types out of video_reader_decoder_op.cc into a header, so that other
operators can derive from the experimental video reader. No behavior change.
GPU-only variant of experimental.readers.video that resizes the decoded
sequences directly into the output, modeled on the legacy
readers.video_resize. VideoReaderDecoder::RunImpl is split into
WriteVideoOutput and WriteMetadataOutputs so the resize variant can
reuse the metadata outputs, and the OutputFn is factored out into
detail::VideoReaderDecoderOutputFn to be shared by both schemas.
…_end

The test assumed an exclusive end-frame convention (last sampled index
== end_frame - 1, roi_frames == end_frame - start_frame). This only
happened to hold because of the file_list_include_end bug fixed in
8a8ea99, where the default (true) was silently acting as exclusive.
With that fixed, the ROI is genuinely inclusive of end_frame, so
roi_frames == end_frame - start_frame + 1 and the last sampled index
is end_frame.
…nd normalized coverage

- Close the image_type TODO by adding types.YCbCr to the shared
  image_type_supported_by_legacy_reader parametrization list, exercised by the
  filenames/file_list/file_root comparison tests.
- Add a pass-through dtype/normalized translation to
  compare_experimental_to_legacy_reader (both readers already share these
  argument names as of prior tasks) and a new
  test_compare_experimental_to_legacy_reader_dtype test covering UINT8/FLOAT
  with normalized=(dtype==FLOAT).
- Skip the known dtype=FLOAT-on-CPU restriction (already covered by
  test_dtype_float_on_cpu_raises) and a newly discovered pre-existing bug
  where experimental.readers.video's CPU backend crashes for image_type=YCbCr
  (frames_decoder_cpu.cc CopyToOutput sets up a single packed sws_scale
  destination buffer even though YCbCr output requests planar
  AV_PIX_FMT_YUV444P, leaving dest[1]/dest[2] null); neither is a
  legacy-vs-experimental parity gap.
- Recompute test_compare_experimental_to_legacy_reader_file_list's
  _known_include_end_divergences skip set: adding the image_type axis
  doubled the parametrization count and shifted the shared np.random stream,
  so the previously-hardcoded 3-tuple set no longer matched reality; the key
  now also includes image_type since each parametrization instance draws its
  own random file_list content.
- Default file_list_include_end to False so that, with no override, the
  end value of a file_list entry is exclusive and frame selections match
  legacy readers.video. Drop the now-unneeded skip list from the
  legacy-comparison file_list test, restore the exclusive-end expectations
  in test_uniform_sample_file_list_roi, and add a deterministic
  legacy-vs-experimental frame-selection test for both file_list formats.
- Document additional_decode_surfaces accurately (it sizes the GPU
  decoder's decode-lookahead/reorder buffer; the NVDEC surface count comes
  from the codec) and reject negative values.
- Make the FLOAT legacy comparison check pixel content: scale normalized
  output to the 8-bit range before applying the intensity threshold, and
  add a FLOAT/normalized=False case.
- Mark the CPU YCbCr sws_scale destination-plane bug with
  TODO(DALI-4916) and reference the ticket from the test skip helper.
- Restore static output dtype metadata via a dtype-dependent OutputDType.
- Fix stale DecodeFrames buffer-size doc comments (bytes, FrameSizeBytes).
The previous fix that flipped file_list_include_end's default to false
(to restore legacy-matching behavior) moved the entry.end_frame clamp
against num_frames inside the `if (include_end)` branch. That meant
default users (include_end=False) got no clamp at all: a file_list
entry whose end exceeds the video's actual frame count produced a
bogus epoch_size and crashed partway through the epoch.

Clamp end_frame to num_frames unconditionally; only add the +1 (making
the range inclusive of the raw end value) when include_end is true.
@copy-pr-bot

copy-pr-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

FramesDecoderCpu::CopyToOutput requested planar AV_PIX_FMT_YUV444P output
for YCbCr but only populated dest[0]/dest_linesize[0], leaving dest[1]/
dest[2] null -- sws_scale rejected this with "bad dst image pointers".
Set up all three destination planes.

Once decodable, CPU YCbCr values still diverged from the GPU backend and
legacy reader by a consistent per-channel offset: sws_setColorspaceDetails
hardcoded the destination range to full-range (0-255), correct for RGB
output but wrong for YCbCr, which both the GPU backend and legacy use in
limited/"TV" range (16-235 luma). CPU YCbCr output now matches the GPU
backend exactly, byte-for-byte, for the same source frame.

Removes the DALI-4916 test skip; the legacy-vs-experimental comparison
tests now exercise CPU+YCbCr directly.
…ality

The two backends use independently-implemented bilinear chroma upsamplers
(libswscale vs a custom CUDA kernel), so exact byte equality is a
coincidence of the current test video's content, not something the
color-range fix guarantees in general. A tolerance still catches a
regression to the systematic ~12-20/255 offset the bug produced, without
being fragile to legitimate small rounding differences on other content.
experimental.readers.video's file_list_format="timestamps" mode selected
a different frame range than legacy readers.video even with all default
arguments: legacy's file_list_include_preceding_frame defaults to False
(ceil-start/floor-end, "start_up_end_down"-equivalent), while
experimental's file_list_rounding defaulted to "start_down_end_up"
(floor-start/ceil-end) -- the opposite rounding direction.

This is a default-behavior change for existing experimental.readers.video
timestamps-format file_list users, matching what DALI-4915 already did
for file_list_include_end: pick the legacy-matching default now that the
underlying rounding logic (unchanged by this commit, only the default
value) is otherwise correct and configurable.

test_file_list_default_end_matches_legacy previously pinned both readers
to an explicit matching rounding mode for the timestamps case specifically
because this divergence existed -- that override is removed, since the
new default now needs no explicit override to match legacy.
MJPEG was already mapped to cudaVideoCodec_JPEG in GetCodecType and
InitBitStreamFilter but deliberately excluded from
FramesDecoderGpu::SelectVideoStream's codec allow-list, pending followup
work.

Enabling it surfaced the reason it was never finished: the NVDEC driver
reports min_num_decode_surfaces=1 for MJPEG content (each frame is an
independent image, unlike the inter-predicted codecs this integration
was built around). With zero surfaces of headroom, decoding the second
frame failed with CUDA_ERROR_INVALID_VALUE, because this decoder never
registers a real pfnDisplayPicture callback -- HandlePictureDisplay runs
synchronously, inline, from ProcessPictureDecode instead -- so the
parser's own surface-recycling bookkeeping never saw the first picture
as "consumed" before the second one needed a surface.

Legacy's NVDEC integration already works around exactly this by
hardcoding ulMaxNumDecodeSurfaces=20 for MJPEG (and HEVC) instead of
trusting the driver-reported minimum (nvdecoder/cuvideoparser.h). Mirror
that here: AdjustedNumDecodeSurfaces forces at least 20 decode surfaces
for MJPEG specifically. Both the sequence callback (which tells the
parser how many surfaces to expect) and decoder creation must use the
same adjusted value, or the two disagree -- so both call sites were
updated together.

Also updates the shared MJpeg gtest (video_reader_op_test.cc, compiled
against both readers__Video and experimental__readers__Video via the
VIDEO_READER_OP macro) to build a per-operator OpSpec, since it
previously always passed legacy's skip_vfr_check argument (which
experimental doesn't have) and unconditionally skipped experimental
rather than actually attempting the decode.
The comment claimed this mirrors legacy hardcoding 20 decode surfaces for
MJPEG. Tracing legacy's actual code: the literal 20 in cuvideoparser.h's
per-codec switch is immediately overwritten by its only caller with a
uniform value passed for every codec, not an MJPEG-specific override --
and legacy's real CUvideodecoder for this test file is created with just
3 surfaces (min_num_decode_surfaces + additional_decode_surfaces), not
20. Legacy gets away with fewer surfaces because it registers a genuine
pfnDisplayPicture callback; this backend's synchronous-inline
HandlePictureDisplay does not have that signal, so it needs more
headroom regardless of what legacy uses. Corrected the comment to
describe 20 as an empirically-verified, conservative value rather than
an inherited legacy precedent.

Also added a non-zero content check for frame 1 (the specific former
crash point), not just frame 0.
… (DALI-4924)

The GPU video reader synchronized the whole CUDA stream and unmapped every
decoded picture immediately after its color-conversion kernel launch, once
per frame. That serializes NVDEC hardware decode against the SM-side
conversion kernel every single frame, and is a large part of why this
reader trails the legacy NVDEC reader's throughput.

Replace the per-frame sync+unmap with a deferred unmap: map, launch the
conversion kernel, record a CUDA event, and move on without syncing.
cuvidUnmapVideoFrame only runs once a surface's slot is actually needed
again, via FIFO eviction bounded by the driver's real ulNumOutputSurfaces
(the hard limit on how many pictures may be mapped at once) -- exactly the
decode/map interleaving pattern NVIDIA's own nvcuvid.h documents. An
earlier attempt at overlap (a full producer/consumer thread redesign, not
part of this change) deferred unmaps without respecting that cap and
caused real frame corruption; this fix stays single-threaded and enforces
the cap explicitly.

A picture's decode-surface index can only be reused this way if the
previous occupant has actually been unmapped first, so
EnsureUnmapped(CurrPicIdx) also runs in ProcessPictureDecode, before
cuvidDecodePicture, mirroring the timing of legacy's frame_in_use_ guard.

Decode-surface count for sizing the pending-map table is now read
directly off the NVDECCache lease (NVDECLease::NumDecodeSurfaces()) rather
than recomputed, so it can't drift from what the pooled CUvideodecoder was
actually created with. Draining now happens deterministically from
SendLastPacket(flush=true), covering both Flush() and SeekFrame()'s direct
call, and from ~FramesDecoderGpu() before the pooled decoder lease is
returned (a decoder instance is reused across FramesDecoderGpu instances,
so anything left mapped here would corrupt the next owner).

Verified: all 90 gtests (dali_operator_test --gtest_filter='*Video*:*Fra
mesDecoder*') and all 178 python tests (test_video_reader.py,
test_video_reader_resize.py) pass. Benchmarked at batch=1/sequence_length=
32: legacy 552fps, this reader before the fix 445fps, after 458fps (~3%
gain, ~13% of the gap to legacy closed). Tried raising ulNumOutputSurfaces
from 2 to 4 for deeper pipelining; no further gain, reverted -- the
remaining gap looks to be CPU-side (demux/bitstream-filter/parse
overhead), not GPU pipelining depth.
Two independent fixes for the GPU video reader's remaining throughput gap
vs. the legacy reader under the common training-pipeline shape
(random_shuffle=True over more files than fit in a batch):

1. video_reader_decoder_op.h: Prefetch() used to tear down and rebuild the
   whole FramesDecoderGpu/FramesDecoderCpu (container reopen, NVDEC parser
   creation, decode-surface allocation, keyframe-relative seek) every time
   consecutive samples came from a different file. With shuffling, that is
   nearly every sample. Replace the single decoder_ with a small
   filename-keyed LRU (capacity 8) so a file's decoder survives across the
   samples it isn't currently servicing and is reused instead of rebuilt.

2. frames_decoder_gpu.cc: HandlePictureDisplay() mapped and color-converted
   every decoded picture that didn't exactly match the next wanted frame,
   even when that picture could never be a future seek target (e.g. the
   pre-roll frames NVDEC must decode to reach a mid-GOP seek position, which
   have a pts before anything the index will ever request again). Drop
   those frames immediately instead of paying for cuvidMapVideoFrame,
   the conversion kernel, and a frame_buffer_ slot that would never be
   read. Frames ahead of the target (e.g. B-frame reordering) still take
   the existing buffering path.

Benchmarked with internal_tools/video_reader_bench.py (added here) on a
TITAN V, batch=8, sequence_length=16, 5 shuffled VP9 files: 295->360 fps,
now above legacy's 342 fps (conversion-kernel launches per requested frame
dropped from 1.82x to 1.09x, matching legacy exactly). The two fixes were
benchmarked together, not in isolation, so this doesn't attribute the gain
between them individually.

Each cached decoder holds its own NVDEC lease and ~10 decode-surface
buffers, so the cache trades some GPU memory (roughly 200MB at 720p/uint8
per cached decoder, ~4x that for float) for avoiding the rebuild cost.

Verified: all 90 gtests (dali_operator_test --gtest_filter='*Video*:*Fra
mesDecoder*') and all 178 python tests (test_video_reader.py,
test_video_reader_resize.py, including the legacy-vs-experimental parity
comparisons) pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant