feat(mediaplayer): HLS and LPCM (m2ts) playback on Android#927
Draft
towneh wants to merge 6 commits into
Draft
Conversation
This was referenced Jul 9, 2026
The HLS source (protocol/basis_hls.c) is portable C and already ships in the Android build; only the HTTP provider selection in run_hls was Windows-only. Wire the JNI HttpsURLConnection byte source in as the Android provider, so playlist and segment fetches ride the same code path the push-model https fallback already uses (TLS, redirects and thread attach handled there). The read timeout is 60s to sit out LL-HLS blocking playlist reloads. Also hand .m3u8 URLs to the HLS source ahead of the AMediaExtractor attempt: the extractor cannot stitch segments, so trying it first only added a rejected network round-trip before the fallback. No behaviour change on Windows, where basis_decoder_try_open_url always returns 0. Delivery semantics carry over unchanged from the portable core: live playlists present at the live edge with AU-level delivery pacing, ENDLIST playlists auto-detect as VOD and play once. fMP4/CMAF-segment HLS remains unsupported on every platform; TS-segment HLS is the working case. Rebuilt libbasis_media_native.so (arm64-v8a).
Ports the Windows LPCM lane to the Android backend so m2ts LPCM - the only full-7.1 audio carriage (AAC decoders cap at 5.1) - plays there too. The demux side was already portable: the TS demuxer recognises Blu-ray HDMV LPCM (stream_type 0x80) and delivers header-stripped big-endian frames through the shared sink on every platform; Android received them and had nowhere to put them. On the LPCM codec no MediaCodec is created: submit_audio converts big-endian 16/24-bit Blu-ray-order PCM to interleaved WAVE-order float (same channel-assignment remap tables as the Windows lane) and writes whole frames straight into the PCM ring, preserving the ring's channel-phase alignment contract. 48 kHz 16/24-bit only, matching what the demuxer announces; the config blob carries the channel assignment and bits code.
The extractor doesn't surface HDMV LPCM, so handing it an .m2ts URL plays the video with the audio silently missing. Keep m2ts on the portable TS demuxer + LPCM bypass path, which plays both. No change on Windows, where the extractor attempt is already a no-op.
Built from this branch (NDK clang 18); the Windows DLL is untouched.
342a71c to
a907118
Compare
Point find_library past the NDK sysroot so the vendored librist static links on Android (no-op on desktop toolchains), and rebuild libbasis_media_native.so with -DBASIS_WITH_RIST=ON. RIST plain + AES-128 are device-validated on Quest (UDP, unaffected by cleartext). Also documents in the CMake header that the shipped win-x64/android-arm64 binaries are RIST-enabled and that omitting the flag silently drops rist:// playback.
…frames 5.1 AAC frames exceed the codec's default input buffer, so submit_audio truncated them and MediaCodec (C2SoftAacDec) rejected every oversized frame with a decode error (0x4004), substituting silence — audible glitching on 5.1 while small stereo frames were unaffected. Set max-input-size to 32 KB (above the ~8 KB ADTS frame ceiling) so whole multichannel frames fit, and drop rather than truncate any frame that still exceeds the buffer. Device-validated on Quest: the 0x4004 flood is gone (0 vs ~11,900) and 5.1 AAC decodes cleanly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two Android audio/streaming gaps closed, one
.so:HLS playback. The HLS source (
protocol/basis_hls.c) is portable C and already ships in the Android build — the only gate wasrun_hlsconstructing its HTTP provider under#if defined(_WIN32). This wires the existing JNIHttpsURLConnectionbyte source in as the Android provider (same code path the push-model https fallback already uses: TLS, redirects and JVM thread-attach handled there), with a 60 s read timeout to sit out LL-HLS blocking playlist reloads..m3u8URLs are handed to the HLS source ahead of theAMediaExtractorattempt — the extractor can't stitch segments, so trying it first only added a rejected network round-trip. Delivery semantics carry over unchanged from the portable core: live playlists present at the live edge with AU-level delivery pacing, ENDLIST playlists auto-detect as VOD (TS-segment VOD additionally picks up #926's seek/duration support wherever the portable layer provides it).LPCM (m2ts). Brings the full-7.1 audio carriage to Android the way Windows has it (AAC decoders cap at 5.1). The demux side was already portable — the TS demuxer recognises Blu-ray HDMV LPCM (
stream_type 0x80) and delivers header-stripped big-endian frames through the shared sink; Android received them and had nowhere to put them. On the LPCM codec no MediaCodec is created:submit_audioconverts big-endian 16/24-bit Blu-ray-order PCM to interleaved WAVE-order float (the same channel-assignment remap tables as the Windows lane, which match ffmpeg's pcm_bluray remap and were verified by ear against a 7.1 channel-marker stream) straight into the PCM ring, whole frames only so the ring keeps its channel-phase alignment. 48 kHz / 16- or 24-bit, matching what the demuxer announces. One steering fix rides along:AMediaExtractoraccepts.m2tsURLs but doesn't surface HDMV LPCM — it would play the video with the audio silently missing — so m2ts stays on the portable TS demuxer + bypass path.No behaviour change on Windows for any of it (the extractor attempt is a no-op there and
run_hlskeeps its WinHTTP provider). Rebuiltlibbasis_media_native.so(arm64-v8a); the Windows DLL is untouched.Status — draft pending device validation
Code-complete, not yet Quest-tested — one headset pass covers both features:
mpegtsvariant).EXT-X-ENDLIST) — also exercises redirects through the JNI fetch.yuma_lpcm71.m2ts(16-bit) — channel-isolation listening pass, same method as the 5.1 AAC validation: all 8 lanes audible, WAVE order correct.blits_lpcm51.m2ts(24-bit) — exercises the 24-bit conversion path.Note for testers:
http://origins need a build with cleartext traffic allowed (Android policy);https://andrtspt://origins need nothing special. The m2ts assets can be served overrtspt://(no extractor involvement) orhttp://(exercises the new m2ts steer).Required checks
All boxes below must be ticked before this PR can merge. If a check is genuinely N/A, tick it anyway and explain under Notes.
TransformAccessArrayor are otherwise batched. I have not added per-frametransform.position/transform.rotation/transform.localPositioncalls inside loops. Whenever I need both position and rotation, I use the combined APIs —SetPositionAndRotation/SetLocalPositionAndRotationfor writes,GetPositionAndRotation/GetLocalPositionAndRotationfor reads — instead of two separate property accesses; the combined call does one local-to-world matrix traversal instead of two.Resources.Load, no direct asset references that pull large content into memory on scene load.GetComponent/AddComponentwhere avoidable — Where unavoidable, the result is cached on a field, and anyGetComponent<T>is replaced withTryGetComponent<T>(out var x)— bareGetComponentwill be denied.TryGetComponentis the modern API (Unity 2019.2+) and skips the Editor-only GC allocationGetComponentcauses when a component is missing: Unity wraps thenullreturn in a managed "fake null" object so its overloaded==operator can still detect destroyed C++ objects, and constructing that wrapper allocates;TryGetComponentreturns aboolplusoutparameter and never builds the wrapper. None of these calls run insideUpdate,LateUpdate,FixedUpdate, jobs, or other per-frame code paths.BasisEventDriver— Any new per-frame work hooks intoBasisEventDriverrather than adding standaloneUpdate/LateUpdate/FixedUpdatecallbacks on a MonoBehaviour.BasisEventDriveris bulletproof, or guarded bytry/catch—BasisEventDriverruns the single per-frame tick that drives the whole framework (network apply, local player sim, blendshapes, JigglePhysics, nameplates, and more) as one sequential chain. An unhandled exception anywhere in that chain aborts the rest of the tick, so every step after the throwing one is silently skipped for that frame. New work added to the driver must either be guaranteed not to throw, or be wrapped in atry/catchthat contains the failure and surfaces it throughBasisDebug— logged once / rate-limited, never every frame (see the existingHVRBasisBuiltInAddresses.Simulate()guard for the pattern). Expect this to be scrutinized closely in review.{ get; set; }properties or access lockdowns — Public fields are fine; Basis is meant to be read and modified freely, so don't wall things offprivate/internalwithout a real reason. Don't wrap a field in{ get; set; }when the accessors do nothing — property accessors have a real performance cost vs direct field access, and the lead maintainer prefers plain fields (or a method / setter-only property when only the setter needs logic) over a noop-getter pair. For.Instancesingletons, callers reassigningType.Instanceis allowed; if that would break your code, log a warning or throw — don't block the assignment. Locking down access is not your call.BasisLocalCameraDriver— Code that needs the local camera (transform, projection, rig data, etc.) pulls it fromBasisLocalCameraDriverrather than looking one up itself. Don't roll a separate camera discovery path.BasisDebug— All new logging calls go throughBasisDebug.Log/BasisDebug.LogWarning/BasisDebug.LogError(with an appropriateLogTag) instead ofUnityEngine.Debug.Log/Debug.LogWarning/Debug.LogError.BasisDebugroutes through Basis's tagged, color-coded logger and respects the project-wideLoggingDisabledtoggle so logging can be killed at runtime; bareDebug.Logcalls bypass that and will be denied.FindObjectOfType/FindObjectsOfType/GameObject.Find/FindGameObjectsWithTagto locate what it depends on. References are wired in — registered through an existing manager/driver, injected at init, or passed in by the caller — rather than discovered by scanning the scene at runtime. If a scene scan is genuinely unavoidable, justify it under Notes.newon reference types, no LINQ, nostringconcatenation/interpolation, no boxing, noforeachover interface-typed collections. Allocate once at init and reuse the buffer.BasisDebug. Hot-path logging floods the console and incurs cost on every frame regardless of whether the message is filtered out downstream. If a hot-path log is needed while iterating, gate it behind#if UNITY_EDITORand remove (or leave gated) before merge..Count(lists) /.Length(arrays) into a localintbefore the loop instead of re-reading the property each iteration. PreferT[](with a separate length int when the array is over-sized) overList<T>where the data is hot — Unity's mono BCL doesn't exposeCollectionsMarshal.AsSpan(List<T>), so a list can't be fed intoSpan<T>/ unsafe paths cleanly. Where the perf justifies it, drop intoSpan<T>/reflocals /Unsafe.As/unsafepointer code to skip bounds checks and copies, and call out the invariants you're relying on under Notes so reviewers can sanity-check them.Testing details
Tick the platforms you actually tested on. Leave the rest unticked — these are informational and do not block merge.
Input / control mode coverage:
Where applicable, confirm these flows still work after your changes:
Notes
Native C only, Android-guarded — no managed/per-frame code, so the driver/allocation/jobification items are N/A (the HLS work is network I/O on the existing threads; the LPCM conversion runs on the demux thread with a reused scratch buffer whose only allocation is one-time growth). "Tested" reflects the Windows no-behaviour-change leg (the guarded code compiles for both targets; Windows runs exactly as before — validated in-editor on the stacked branch, and the Windows LPCM lane validated this exact conversion and remap in its 7.1 listening pass when it shipped) — the Android device pass is precisely what the draft status above holds for. Kept as draft until the Quest pass listed under Status.