Skip to content

feat(mediaplayer): HLS and LPCM (m2ts) playback on Android#927

Draft
towneh wants to merge 6 commits into
BasisVR:developerfrom
towneh:feat/mediaplayer-android-hls
Draft

feat(mediaplayer): HLS and LPCM (m2ts) playback on Android#927
towneh wants to merge 6 commits into
BasisVR:developerfrom
towneh:feat/mediaplayer-android-hls

Conversation

@towneh

@towneh towneh commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #926 — review the last four commits (one per concern). Re-raised from BasisVR/BasisMediaPlayer#3 as part of the monorepo collapse, and now also carries the Android LPCM lane (folded in from #928 — both features gate on the same Quest pass from a single build, so one PR keeps the tested artifact and the merged artifact identical).

Summary

Two Android audio/streaming gaps closed, one .so:

HLS playback. The HLS source (protocol/basis_hls.c) is portable C and already ships in the Android build — the only gate was run_hls constructing its HTTP provider under #if defined(_WIN32). This wires the existing JNI HttpsURLConnection byte source in as the Android provider (same code path the push-model https fallback already uses: TLS, redirects and JVM thread-attach handled there), with a 60 s read timeout to sit out LL-HLS blocking playlist reloads. .m3u8 URLs are handed to the HLS source ahead of the AMediaExtractor attempt — the extractor can't stitch segments, so trying it first only added a rejected network round-trip. Delivery semantics carry over unchanged from the portable core: live playlists present at the live edge with AU-level delivery pacing, ENDLIST playlists auto-detect as VOD (TS-segment VOD additionally picks up #926's seek/duration support wherever the portable layer provides it).

LPCM (m2ts). Brings the full-7.1 audio carriage to Android the way Windows has it (AAC decoders cap at 5.1). The demux side was already portable — the TS demuxer recognises Blu-ray HDMV LPCM (stream_type 0x80) and delivers header-stripped big-endian frames through the shared sink; Android received them and had nowhere to put them. On the LPCM codec no MediaCodec is created: submit_audio converts big-endian 16/24-bit Blu-ray-order PCM to interleaved WAVE-order float (the same channel-assignment remap tables as the Windows lane, which match ffmpeg's pcm_bluray remap and were verified by ear against a 7.1 channel-marker stream) straight into the PCM ring, whole frames only so the ring keeps its channel-phase alignment. 48 kHz / 16- or 24-bit, matching what the demuxer announces. One steering fix rides along: AMediaExtractor accepts .m2ts URLs but doesn't surface HDMV LPCM — it would play the video with the audio silently missing — so m2ts stays on the portable TS demuxer + bypass path.

No behaviour change on Windows for any of it (the extractor attempt is a no-op there and run_hls keeps its WinHTTP provider). Rebuilt libbasis_media_native.so (arm64-v8a); the Windows DLL is untouched.

Status — draft pending device validation

Code-complete, not yet Quest-tested — one headset pass covers both features:

  1. HLS live: TS-segment live HLS from the test origin (mediamtx, mpegts variant).
  2. HLS VOD + TLS: a public TS-segment HLS VOD stream (EXT-X-ENDLIST) — also exercises redirects through the JNI fetch.
  3. LPCM 7.1: yuma_lpcm71.m2ts (16-bit) — channel-isolation listening pass, same method as the 5.1 AAC validation: all 8 lanes audible, WAVE order correct.
  4. LPCM 5.1: blits_lpcm51.m2ts (24-bit) — exercises the 24-bit conversion path.
  5. Regression: stereo/5.1 AAC intact on the same build (the audio configure path gained a branch; the TS→AAC push path was already validated over RTSP on Quest).

Note for testers: http:// origins need a build with cleartext traffic allowed (Android policy); https:// and rtspt:// origins need nothing special. The m2ts assets can be served over rtspt:// (no extractor involvement) or http:// (exercises the new m2ts steer).

Required checks

All boxes below must be ticked before this PR can merge. If a check is genuinely N/A, tick it anyway and explain under Notes.

  • Tested — I built and ran this locally. The change works in the editor and (where relevant) in a built player.
  • Transform access is combined and limited — In hot paths, transform reads/writes go through TransformAccessArray or are otherwise batched. I have not added per-frame transform.position / transform.rotation / transform.localPosition calls inside loops. Whenever I need both position and rotation, I use the combined APIs — SetPositionAndRotation / SetLocalPositionAndRotation for writes, GetPositionAndRotation / GetLocalPositionAndRotation for reads — instead of two separate property accesses; the combined call does one local-to-world matrix traversal instead of two.
  • Addressables used for asset/memory loading — Any new asset loads go through Addressables. No new Resources.Load, no direct asset references that pull large content into memory on scene load.
  • No new GetComponent / AddComponent where avoidable — Where unavoidable, the result is cached on a field, and any GetComponent<T> is replaced with TryGetComponent<T>(out var x) — bare GetComponent will be denied. TryGetComponent is the modern API (Unity 2019.2+) and skips the Editor-only GC allocation GetComponent causes when a component is missing: Unity wraps the null return in a managed "fake null" object so its overloaded == operator can still detect destroyed C++ objects, and constructing that wrapper allocates; TryGetComponent returns a bool plus out parameter and never builds the wrapper. None of these calls run inside Update, LateUpdate, FixedUpdate, jobs, or other per-frame code paths.
  • Per-frame work is scheduled through BasisEventDriver — Any new per-frame work hooks into BasisEventDriver rather than adding standalone Update / LateUpdate / FixedUpdate callbacks on a MonoBehaviour.
  • Anything added to BasisEventDriver is bulletproof, or guarded by try/catchBasisEventDriver runs the single per-frame tick that drives the whole framework (network apply, local player sim, blendshapes, JigglePhysics, nameplates, and more) as one sequential chain. An unhandled exception anywhere in that chain aborts the rest of the tick, so every step after the throwing one is silently skipped for that frame. New work added to the driver must either be guaranteed not to throw, or be wrapped in a try/catch that contains the failure and surfaces it through BasisDebug — logged once / rate-limited, never every frame (see the existing HVRBasisBuiltInAddresses.Simulate() guard for the pattern). Expect this to be scrutinized closely in review.
  • Considered jobification — I asked whether this work can be moved to a Unity Job (Burst-compiled where possible). If it can, it is. If it cannot, the reason is in Notes.
  • No needless { get; set; } properties or access lockdowns — Public fields are fine; Basis is meant to be read and modified freely, so don't wall things off private/internal without a real reason. Don't wrap a field in { get; set; } when the accessors do nothing — property accessors have a real performance cost vs direct field access, and the lead maintainer prefers plain fields (or a method / setter-only property when only the setter needs logic) over a noop-getter pair. For .Instance singletons, callers reassigning Type.Instance is allowed; if that would break your code, log a warning or throw — don't block the assignment. Locking down access is not your call.
  • Camera access goes through BasisLocalCameraDriver — Code that needs the local camera (transform, projection, rig data, etc.) pulls it from BasisLocalCameraDriver rather than looking one up itself. Don't roll a separate camera discovery path.
  • Logging uses BasisDebug — All new logging calls go through BasisDebug.Log / BasisDebug.LogWarning / BasisDebug.LogError (with an appropriate LogTag) instead of UnityEngine.Debug.Log / Debug.LogWarning / Debug.LogError. BasisDebug routes through Basis's tagged, color-coded logger and respects the project-wide LoggingDisabled toggle so logging can be killed at runtime; bare Debug.Log calls bypass that and will be denied.
  • No scene-wide discovery for dependencies — New code is architected so it does not need FindObjectOfType / FindObjectsOfType / GameObject.Find / FindGameObjectsWithTag to locate what it depends on. References are wired in — registered through an existing manager/driver, injected at init, or passed in by the caller — rather than discovered by scanning the scene at runtime. If a scene scan is genuinely unavoidable, justify it under Notes.
  • No allocations in hot paths — Per-frame code (Update / LateUpdate / FixedUpdate, simulation loops, jobs, anything called once per frame or more) does not allocate. No new on reference types, no LINQ, no string concatenation/interpolation, no boxing, no foreach over interface-typed collections. Allocate once at init and reuse the buffer.
  • No debugging in hot paths — No log calls of any kind on per-frame paths, including BasisDebug. Hot-path logging floods the console and incurs cost on every frame regardless of whether the message is filtered out downstream. If a hot-path log is needed while iterating, gate it behind #if UNITY_EDITOR and remove (or leave gated) before merge.
  • Hot-path collection access is optimized — Cache .Count (lists) / .Length (arrays) into a local int before the loop instead of re-reading the property each iteration. Prefer T[] (with a separate length int when the array is over-sized) over List<T> where the data is hot — Unity's mono BCL doesn't expose CollectionsMarshal.AsSpan(List<T>), so a list can't be fed into Span<T> / unsafe paths cleanly. Where the perf justifies it, drop into Span<T> / ref locals / Unsafe.As / unsafe pointer code to skip bounds checks and copies, and call out the invariants you're relying on under Notes so reviewers can sanity-check them.

Testing details

Tick the platforms you actually tested on. Leave the rest unticked — these are informational and do not block merge.

  • Windows
  • Linux
  • Android
  • iOS
  • macOS

Input / control mode coverage:

  • Tested in VR (note headset under Notes)
  • Tested in desktop / non-VR mode
  • Tested with phone controls (mobile touch input)
  • N/A — change does not touch player/XR/input code

Where applicable, confirm these flows still work after your changes:

  • Hot-switching (desktop ↔ VR mode swap at runtime)
  • Avatar swapping
  • Server swapping (joining / leaving / changing servers)
  • N/A — change does not touch any of the above

Notes

Native C only, Android-guarded — no managed/per-frame code, so the driver/allocation/jobification items are N/A (the HLS work is network I/O on the existing threads; the LPCM conversion runs on the demux thread with a reused scratch buffer whose only allocation is one-time growth). "Tested" reflects the Windows no-behaviour-change leg (the guarded code compiles for both targets; Windows runs exactly as before — validated in-editor on the stacked branch, and the Windows LPCM lane validated this exact conversion and remap in its 7.1 listening pass when it shipped) — the Android device pass is precisely what the draft status above holds for. Kept as draft until the Quest pass listed under Status.

@towneh towneh changed the title feat(mediaplayer): enable HLS playback on Android feat(mediaplayer): HLS and LPCM (m2ts) playback on Android Jul 9, 2026
towneh added 4 commits July 9, 2026 18:29
The HLS source (protocol/basis_hls.c) is portable C and already ships in the
Android build; only the HTTP provider selection in run_hls was Windows-only.
Wire the JNI HttpsURLConnection byte source in as the Android provider, so
playlist and segment fetches ride the same code path the push-model https
fallback already uses (TLS, redirects and thread attach handled there). The
read timeout is 60s to sit out LL-HLS blocking playlist reloads.

Also hand .m3u8 URLs to the HLS source ahead of the AMediaExtractor attempt:
the extractor cannot stitch segments, so trying it first only added a
rejected network round-trip before the fallback. No behaviour change on
Windows, where basis_decoder_try_open_url always returns 0.

Delivery semantics carry over unchanged from the portable core: live
playlists present at the live edge with AU-level delivery pacing, ENDLIST
playlists auto-detect as VOD and play once. fMP4/CMAF-segment HLS remains
unsupported on every platform; TS-segment HLS is the working case.

Rebuilt libbasis_media_native.so (arm64-v8a).
Ports the Windows LPCM lane to the Android backend so m2ts LPCM - the
only full-7.1 audio carriage (AAC decoders cap at 5.1) - plays there
too. The demux side was already portable: the TS demuxer recognises
Blu-ray HDMV LPCM (stream_type 0x80) and delivers header-stripped
big-endian frames through the shared sink on every platform; Android
received them and had nowhere to put them.

On the LPCM codec no MediaCodec is created: submit_audio converts
big-endian 16/24-bit Blu-ray-order PCM to interleaved WAVE-order float
(same channel-assignment remap tables as the Windows lane) and writes
whole frames straight into the PCM ring, preserving the ring's
channel-phase alignment contract. 48 kHz 16/24-bit only, matching what
the demuxer announces; the config blob carries the channel assignment
and bits code.
The extractor doesn't surface HDMV LPCM, so handing it an .m2ts URL
plays the video with the audio silently missing. Keep m2ts on the
portable TS demuxer + LPCM bypass path, which plays both. No change on
Windows, where the extractor attempt is already a no-op.
Built from this branch (NDK clang 18); the Windows DLL is untouched.
@towneh towneh force-pushed the feat/mediaplayer-android-hls branch from 342a71c to a907118 Compare July 9, 2026 17:29
towneh added 2 commits July 9, 2026 22:24
Point find_library past the NDK sysroot so the vendored librist static links on Android (no-op on desktop toolchains), and rebuild libbasis_media_native.so with -DBASIS_WITH_RIST=ON. RIST plain + AES-128 are device-validated on Quest (UDP, unaffected by cleartext). Also documents in the CMake header that the shipped win-x64/android-arm64 binaries are RIST-enabled and that omitting the flag silently drops rist:// playback.
…frames

5.1 AAC frames exceed the codec's default input buffer, so submit_audio truncated them and MediaCodec (C2SoftAacDec) rejected every oversized frame with a decode error (0x4004), substituting silence — audible glitching on 5.1 while small stereo frames were unaffected. Set max-input-size to 32 KB (above the ~8 KB ADTS frame ceiling) so whole multichannel frames fit, and drop rather than truncate any frame that still exceeds the buffer. Device-validated on Quest: the 0x4004 flood is gone (0 vs ~11,900) and 5.1 AAC decodes cleanly.
@towneh towneh added the enhancement New feature or request label Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant