From 5d92a6c7856a053304e61423e69825076235be1f Mon Sep 17 00:00:00 2001 From: Dale Curtis Date: Wed, 30 Sep 2026 23:38:31 +0000 Subject: [PATCH] Clarify color space handling across decoding, encoding, and rendering Fixes #940. - Define an internal color space on the underlying media resource (which may represent H.273 code points beyond the WebIDL enums), exposing unspecified or unrecognized components as null on VideoFrame.colorSpace. - In Output VideoFrames: use the decoder-emitted color space if decoding transforms the color space; otherwise select the most specified color space between [[active decoder config]].colorSpace and the bitstream color space (breaking ties in favor of [[active decoder config]].colorSpace), falling back to a codec-specific default or sRGB (RGB formats) / REC709 (other formats) only when not fully specified and the decoder requires a fully specified color space. - In Output EncodedVideoChunks: allow encoders to modify the input frame's internal color space matrix and range to satisfy codec requirements (assuming full-range rgb for RGB formats and limited-range bt709 otherwise when unspecified), and populate outputConfig.colorSpace from the encoder-emitted color space if specified or else the frame's internal color space after any modifications. - Remove the heuristic guessing allowance from Pick Color Space and specify fallback to sRGB (RGB formats) or REC709 (other formats) during rendering when the internal color space is not fully specified. --- index.src.html | 103 ++++++++++++++++++++++++++++++++++++------------- 1 file changed, 76 insertions(+), 27 deletions(-) diff --git a/index.src.html b/index.src.html index befaa064..afa1e902 100644 --- a/index.src.html +++ b/index.src.html @@ -940,23 +940,38 @@ {{VideoDecoderConfig/displayAspectHeight}} [=map/exist=] in the {{VideoDecoder/[[active decoder config]]}}, assign their values to |displayAspectWidth| and |displayAspectHeight| respectively. - 4. Let |colorSpace| be the {{VideoColorSpace}} for |output| as detected - by the codec implementation. If no {{VideoColorSpace}} is detected, - let |colorSpace| be `undefined`. - - NOTE: The codec implementation can detect a {{VideoColorSpace}} by - analyzing the bitstream. Detection is made on a best-effort - basis. The exact method of detection is implementer defined and - codec-specific. Authors can override the detected - {{VideoColorSpace}} by providing a - {{VideoDecoderConfig/colorSpace}} in the {{VideoDecoderConfig}}. - - 5. If {{VideoDecoderConfig/colorSpace}} [=map/exists=] in the - {{VideoDecoder/[[active decoder config]]}}, assign its value to - |colorSpace|. In that case, User Agents MAY replace `null` members - of |colorSpace| with the corresponding values detected by the codec - implementation. - FIXME: Properly specify the case of `null` members. + 4. Determine the [=internal color space=] of |output|'s + [=media resource=] as follows: + 1. If {{VideoDecoder/[[codec implementation]]}} transforms the color + space during decoding (for example, converting to an + [=RGB format=] or modifying matrix or range values), use the + transformed color space emitted by + {{VideoDecoder/[[codec implementation]]}}. + 2. Otherwise, between + {{VideoDecoder/[[active decoder config]]}}'s + {{VideoDecoderConfig/colorSpace}} (if it [=map/exists=]) and the + color space detected in the bitstream by + {{VideoDecoder/[[codec implementation]]}}, select the color space + with the most specified (non-`null`) components, breaking ties in + favor of {{VideoDecoderConfig/colorSpace}}: + 1. If the selected color space is fully specified, use it. + 2. Otherwise, if {{VideoDecoder/[[codec implementation]]}} + requires a fully specified color space, use the + codec-specific default color space, or if none is defined, + the [=sRGB Color Space=] when |output| uses an [=RGB format=] + and the [=REC709 Color Space=] otherwise. + 3. Otherwise, use the selected color space as-is. + + NOTE: The codec implementation can detect an [=internal color space=] + by analyzing the bitstream on a best-effort, codec-specific + basis. Bitstream components that are present and not + `unspecified` count as specified components when comparing with + {{VideoDecoderConfig/colorSpace}}, even if they do not map to a + WebIDL enum value. + + 5. Let |colorSpace| be a new {{VideoColorSpace}} populated from the + [=internal color space=] of |output|'s [=media resource=], with any + unspecified or unrecognized components set to `null`. 6. Assign the values of {{VideoDecoderConfig/rotation}} and {{VideoDecoderConfig/flip}} to |rotation| and |flip| respectively. 7. Let |frame| be the result of running the [=Create a VideoFrame=] @@ -1697,7 +1712,24 @@ associated with |output| to `outputConfig.rotation`. 7. Assign {{VideoFrame/[[flip]]}} from the {{VideoFrame}} associated with |output| to `outputConfig.flip`. - 8. Assign the remaining keys of `outputConfig` as determined by + 8. Initialize `outputConfig.colorSpace` from the {{VideoFrame}} + associated with |output| as follows: + 1. Encoding MAY modify the + [=internal color space=] {{VideoColorSpace/matrix}} and + {{VideoColorSpace/fullRange}} to satisfy codec requirements, + assuming full-range {{VideoMatrixCoefficients/rgb}} for + [=RGB formats=] and limited-range + {{VideoMatrixCoefficients/bt709}} otherwise when + unspecified. + 2. Let |encodedColorSpace| be the color space emitted by + {{VideoEncoder/[[codec implementation]]}} if specified, or + otherwise the [=internal color space=] after any + modifications from step 1. + 3. Set `outputConfig.colorSpace` to a new + {{VideoColorSpaceInit}} populated from |encodedColorSpace|, + with any unspecified or unrecognized components set to + `null`. + 9. Assign the remaining keys of `outputConfig` as determined by {{VideoEncoder/[[codec implementation]]}}. The User Agent MUST ensure that the configuration is completely described such that |outputConfig| could be used to @@ -2096,7 +2128,9 @@ Configures the {{VideoFrame}}.{{VideoFrame/colorSpace}} for {{VideoFrame}}s associated with this {{VideoDecoderConfig}}. If {{VideoDecoderConfig/colorSpace}} [=map/exists=], the provided values will - override any in-band values from the bitsream. + override any in-band values from the bitstream unless the decoder transforms + the color space or a more specified color space is present in the bitstream, + as described in [=Output VideoFrames=].
hardwareAcceleration
@@ -2819,7 +2853,10 @@ ### Reference Counting ### {#raw-media-memory-model-reference-counting} A media resource is storage for the actual pixel data or the audio -sample data described by a {{VideoFrame}} or {{AudioData}}. +sample data described by a {{VideoFrame}} or {{AudioData}}. For video, a +[=media resource=] has an associated underlying pixel format and an +internal color space (e.g., [[H.273]] code points) which may be more +expressive than {{VideoPixelFormat}} and {{VideoColorSpace}}. The {{AudioData}} {{AudioData/[[resource reference]]}} and {{VideoFrame}} {{VideoFrame/[[resource reference]]}} internal slots hold a reference to a @@ -3536,7 +3573,12 @@ from the {{EncodedVideoChunk}} corresponding to this {{VideoFrame}}. : [[color space]] -:: The {{VideoColorSpace}} associated with this frame. +:: A {{VideoColorSpace}} describing the color space of the {{VideoFrame}}. + Individual members of {{VideoFrame/[[color space]]}} will be `null` whenever + the corresponding component of the underlying [=media resource=]'s + [=internal color space=] is unspecified or does not map to a value in + {{VideoColorPrimaries}}, {{VideoTransferCharacteristics}}, or + {{VideoMatrixCoefficients}}. : \[[metadata]] :: The {{VideoFrameMetadata}} associated with this frame. @@ -3836,7 +3878,11 @@ {{VideoFrame/[[duration]]}}. : colorSpace -:: The {{VideoColorSpace}} associated with this frame. +:: The {{VideoColorSpace}} associated with this frame. Individual members will + be `null` whenever the corresponding component of the underlying + [=media resource=]'s [=internal color space=] is unspecified or does not map + to a value in {{VideoColorPrimaries}}, {{VideoTransferCharacteristics}}, or + {{VideoMatrixCoefficients}}. The {{VideoFrame/colorSpace}} getter steps are to return {{VideoFrame/[[color space]]}}. @@ -4012,11 +4058,6 @@ : Pick Color Space (with |overrideColorSpace| and |format|) :: 1. If |overrideColorSpace| is provided, return a new {{VideoColorSpace}} constructed with |overrideColorSpace|. - - User Agents MAY replace `null` members of the - provided |overrideColorSpace| with guessed values as determined by implementer - defined heuristics. - 2. Otherwise, if {{VideoFrame/[[format]]}} is an [=RGB format=] return a new instance of the [=sRGB Color Space=] 3. Otherwise, return a new instance of the [=REC709 Color Space=]. @@ -4467,6 +4508,14 @@ be converted to a color space compatible with the rendering target, unless color conversion is explicitly disabled. +When applying color space conversion (for rendering or in +[=VideoFrame/Convert to RGB frame=]), User Agents MUST +use the [=media resource=]'s [=internal color space=] if it is fully specified. +If the [=internal color space=] is missing or only partially specified, User +Agents MUST treat the source color space as the +[=sRGB Color Space=] when the [=media resource=] uses an [=RGB format=], and as +the [=REC709 Color Space=] otherwise. + Color space conversion during {{ImageBitmap}} construction is controlled by {{ImageBitmapOptions}} {{ImageBitmapOptions/colorSpaceConversion}}. Setting this value to {{ColorSpaceConversion/"none"}} disables color space conversion.