Skip to content

Clarify color space handling across decoding, encoding, and rendering #954

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dalecurtis wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Clarify color space handling across decoding, encoding, and rendering #954

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 76 additions & 27 deletions index.src.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -940,23 +940,38 @@
{{VideoDecoderConfig/displayAspectHeight}} [=map/exist=] in the
{{VideoDecoder/[[active decoder config]]}}, assign their values to
|displayAspectWidth| and |displayAspectHeight| respectively.
4. Let |colorSpace| be the {{VideoColorSpace}} for |output| as detected
by the codec implementation. If no {{VideoColorSpace}} is detected,
let |colorSpace| be `undefined`.

NOTE: The codec implementation can detect a {{VideoColorSpace}} by
analyzing the bitstream. Detection is made on a best-effort
basis. The exact method of detection is implementer defined and
codec-specific. Authors can override the detected
{{VideoColorSpace}} by providing a
{{VideoDecoderConfig/colorSpace}} in the {{VideoDecoderConfig}}.

5. If {{VideoDecoderConfig/colorSpace}} [=map/exists=] in the
{{VideoDecoder/[[active decoder config]]}}, assign its value to
|colorSpace|. In that case, User Agents MAY replace `null` members
of |colorSpace| with the corresponding values detected by the codec
implementation.
FIXME: Properly specify the case of `null` members.
4. Determine the [=internal color space=] of |output|'s
[=media resource=] as follows:
1. If {{VideoDecoder/[[codec implementation]]}} transforms the color
space during decoding (for example, converting to an
[=RGB format=] or modifying matrix or range values), use the
transformed color space emitted by
{{VideoDecoder/[[codec implementation]]}}.
2. Otherwise, between
{{VideoDecoder/[[active decoder config]]}}'s
{{VideoDecoderConfig/colorSpace}} (if it [=map/exists=]) and the
color space detected in the bitstream by
{{VideoDecoder/[[codec implementation]]}}, select the color space
with the most specified (non-`null`) components, breaking ties in
favor of {{VideoDecoderConfig/colorSpace}}:
1. If the selected color space is fully specified, use it.
2. Otherwise, if {{VideoDecoder/[[codec implementation]]}}
requires a fully specified color space, use the
codec-specific default color space, or if none is defined,
the [=sRGB Color Space=] when |output| uses an [=RGB format=]
and the [=REC709 Color Space=] otherwise.
3. Otherwise, use the selected color space as-is.

NOTE: The codec implementation can detect an [=internal color space=]
by analyzing the bitstream on a best-effort, codec-specific
basis. Bitstream components that are present and not
`unspecified` count as specified components when comparing with
{{VideoDecoderConfig/colorSpace}}, even if they do not map to a
WebIDL enum value.

5. Let |colorSpace| be a new {{VideoColorSpace}} populated from the
[=internal color space=] of |output|'s [=media resource=], with any
unspecified or unrecognized components set to `null`.
6. Assign the values of {{VideoDecoderConfig/rotation}} and
{{VideoDecoderConfig/flip}} to |rotation| and |flip| respectively.
7. Let |frame| be the result of running the [=Create a VideoFrame=]
Expand Down Expand Up @@ -1697,7 +1712,24 @@
associated with |output| to `outputConfig.rotation`.
7. Assign {{VideoFrame/[[flip]]}} from the {{VideoFrame}}
associated with |output| to `outputConfig.flip`.
8. Assign the remaining keys of `outputConfig` as determined by
8. Initialize `outputConfig.colorSpace` from the {{VideoFrame}}
associated with |output| as follows:
1. Encoding <em class="rfc2119">MAY</em> modify the
[=internal color space=] {{VideoColorSpace/matrix}} and
{{VideoColorSpace/fullRange}} to satisfy codec requirements,
assuming full-range {{VideoMatrixCoefficients/rgb}} for
[=RGB formats=] and limited-range
{{VideoMatrixCoefficients/bt709}} otherwise when
unspecified.
2. Let |encodedColorSpace| be the color space emitted by
{{VideoEncoder/[[codec implementation]]}} if specified, or
otherwise the [=internal color space=] after any
modifications from step 1.
3. Set `outputConfig.colorSpace` to a new
{{VideoColorSpaceInit}} populated from |encodedColorSpace|,
with any unspecified or unrecognized components set to
`null`.
9. Assign the remaining keys of `outputConfig` as determined by
{{VideoEncoder/[[codec implementation]]}}. The User Agent
<em class="rfc2119">MUST</em> ensure that the configuration is
completely described such that |outputConfig| could be used to
Expand Down Expand Up @@ -2096,7 +2128,9 @@
Configures the {{VideoFrame}}.{{VideoFrame/colorSpace}} for {{VideoFrame}}s
associated with this {{VideoDecoderConfig}}. If
{{VideoDecoderConfig/colorSpace}} [=map/exists=], the provided values will
override any in-band values from the bitsream.
override any in-band values from the bitstream unless the decoder transforms
the color space or a more specified color space is present in the bitstream,
as described in [=Output VideoFrames=].
</dd>

<dt><dfn dict-member for=VideoDecoderConfig>hardwareAcceleration</dfn></dt>
Expand Down Expand Up @@ -2819,7 +2853,10 @@
### Reference Counting ### {#raw-media-memory-model-reference-counting}

A <dfn>media resource</dfn> is storage for the actual pixel data or the audio
sample data described by a {{VideoFrame}} or {{AudioData}}.
sample data described by a {{VideoFrame}} or {{AudioData}}. For video, a
[=media resource=] has an associated underlying pixel format and an
<dfn>internal color space</dfn> (e.g., [[H.273]] code points) which may be more
expressive than {{VideoPixelFormat}} and {{VideoColorSpace}}.

The {{AudioData}} {{AudioData/[[resource reference]]}} and {{VideoFrame}}
{{VideoFrame/[[resource reference]]}} internal slots hold a reference to a
Expand Down Expand Up @@ -3536,7 +3573,12 @@
from the {{EncodedVideoChunk}} corresponding to this {{VideoFrame}}.

: <dfn attribute for=VideoFrame>[[color space]]</dfn>
:: The {{VideoColorSpace}} associated with this frame.
:: A {{VideoColorSpace}} describing the color space of the {{VideoFrame}}.
Individual members of {{VideoFrame/[[color space]]}} will be `null` whenever
the corresponding component of the underlying [=media resource=]'s
[=internal color space=] is unspecified or does not map to a value in
{{VideoColorPrimaries}}, {{VideoTransferCharacteristics}}, or
{{VideoMatrixCoefficients}}.

: <dfn attribute for=VideoFrame>\[[metadata]]</dfn>
:: The {{VideoFrameMetadata}} associated with this frame.
Expand Down Expand Up @@ -3836,7 +3878,11 @@
{{VideoFrame/[[duration]]}}.

: <dfn attribute for=VideoFrame>colorSpace</dfn>
:: The {{VideoColorSpace}} associated with this frame.
:: The {{VideoColorSpace}} associated with this frame. Individual members will
be `null` whenever the corresponding component of the underlying
[=media resource=]'s [=internal color space=] is unspecified or does not map
to a value in {{VideoColorPrimaries}}, {{VideoTransferCharacteristics}}, or
{{VideoMatrixCoefficients}}.

The {{VideoFrame/colorSpace}} getter steps are to return
{{VideoFrame/[[color space]]}}.
Expand Down Expand Up @@ -4012,11 +4058,6 @@
: <dfn for=VideoFrame>Pick Color Space</dfn> (with |overrideColorSpace| and |format|)
:: 1. If |overrideColorSpace| is provided, return a new {{VideoColorSpace}}
constructed with |overrideColorSpace|.

User Agents <em class="rfc2119">MAY</em> replace `null` members of the
provided |overrideColorSpace| with guessed values as determined by implementer
defined heuristics.

2. Otherwise, if {{VideoFrame/[[format]]}} is an [=RGB format=] return a new
instance of the [=sRGB Color Space=]
3. Otherwise, return a new instance of the [=REC709 Color Space=].
Expand Down Expand Up @@ -4467,6 +4508,14 @@
be converted to a color space compatible with the rendering target, unless color
conversion is explicitly disabled.

When applying color space conversion (for rendering or in
[=VideoFrame/Convert to RGB frame=]), User Agents <em class="rfc2119">MUST</em>
use the [=media resource=]'s [=internal color space=] if it is fully specified.
If the [=internal color space=] is missing or only partially specified, User
Agents <em class="rfc2119">MUST</em> treat the source color space as the
[=sRGB Color Space=] when the [=media resource=] uses an [=RGB format=], and as
the [=REC709 Color Space=] otherwise.

Color space conversion during {{ImageBitmap}} construction is controlled by
{{ImageBitmapOptions}} {{ImageBitmapOptions/colorSpaceConversion}}. Setting this
value to {{ColorSpaceConversion/"none"}} disables color space conversion.
Expand Down
Loading