<?xml version="1.0" encoding="UTF-8"?> version='1.0' encoding='UTF-8'?>

<!DOCTYPE rfc >
<?rfc compact="yes"?>
<?rfc subcompact="yes"?>
<?rfc iprnotified="no" ?>
<?rfc strict="yes"?>
<?rfc symrefs="yes"?>
<?rfc toc="yes"?>
<?rfc tocdepth="4"?> [
  <!ENTITY nbsp    "&#160;">
  <!ENTITY zwsp   "&#8203;">
  <!ENTITY nbhy   "&#8209;">
  <!ENTITY wj     "&#8288;">
]>

<rfc xmlns:xi="http://www.w3.org/2001/XInclude" category="exp" docName="draft-ietf-avtext-framemarking-16" number="9626" consensus="true" updates="" obsoletes="" ipr="trust200902" submissionType="IETF"> submissionType="IETF" symRefs="true" sortRefs="true" tocInclude="true" tocDepth="4" version="3" xml:lang="en">

  <front>
    <title abbrev="Video Frame Marking">Video Frame Marking RTP Header Extension</title>
    <seriesInfo name="RFC" value="9626"/>
    <author fullname="Mo Zanaty" initials="M" surname="Zanaty">
      <organization>Cisco Systems</organization>
      <address>
        <postal>
          <street>170 West Tasman Drive</street>
          <city>San Jose</city>
          <region>CA</region>
          <code>95134</code>
          <country>US</country>
          <country>United States of America</country>
        </postal>
        <email>mzanaty@cisco.com</email>
      </address>
    </author>
    <author initials="E." surname="Berger" fullname="Espen Berger">
      <organization>Cisco Systems</organization>
      <address>
        <email>espeberg@cisco.com</email>
      </address>
    </author>
    <author fullname="Suhas Nandakumar" initials="S" surname="Nandakumar">
      <organization>Cisco Systems</organization>
      <address>
        <postal>
          <street>170 West Tasman Drive</street>
          <city>San Jose</city>
          <region>CA</region>
          <code>95134</code>
          <country>US</country>
          <country>United States of America</country>
        </postal>
        <email>snandaku@cisco.com</email>
      </address>
    </author>
    <date day="04" month="March" year="2024"/>

    <area>Applications</area>
    <keyword>Internet-Draft</keyword> month="February" year="2025"/>

    <area>WIT</area>
    <workgroup>avtcore</workgroup>

    <abstract>
      <t>This document describes a Video Frame Marking RTP header extension used to
      convey information about video frames that is critical for error recovery
      and packet forwarding in RTP middleboxes or network nodes. It is most
      useful when media is encrypted, encrypted and essential when the middlebox or node
      has no access to the media decryption keys. It is also useful for
      codec-agnostic processing of encrypted or unencrypted media, while it also
      supports extensions for codec-specific information.</t>
    </abstract>
  </front>
  <middle>
    <section title="Introduction" anchor="intro">
      <name>Introduction</name>
      <t>Many widely deployed RTP <xref target="RFC3550" /> target="RFC3550"/> topologies
        <xref target="RFC7667" /> target="RFC7667"/> used in modern voice and video
      conferencing systems include a centralized component that acts as an RTP switch.
      It receives voice and video streams from each participant, which may be encrypted using
      SRTP
      Secure Real-time Transport Protocol (SRTP) <xref target="RFC3711" />, target="RFC3711"/> or extensions that provide participants with
      private media <xref target="RFC8871" /> target="RFC8871"/>
      via end-to-end encryption where the switch has no access to media decryption keys.
      The goal is to provide a set of streams back to
      the participants participants, which enable them to render the right media content. In For example, in a
      simple video configuration, for example, the goal will be that each participant
      sees and hears just the active speaker. In that case, the goal of the switch is to
      receive the voice and video streams from each participant, determine the active
      speaker based on energy in the voice packets, possibly using the client-to-mixer
      audio level RTP header extension <xref target="RFC6464" />, target="RFC6464"/>, and select the corresponding video
        stream for transmission to participants; see <xref target="rtpswitch" />.</t> target="rtpswitch"/>.</t>
      <t>In this document, an "RTP switch" is used as a common short term shorthand for the terms
      "switching RTP mixer", "source projecting middlebox",
      "source forwarding unit/middlebox" and "video switching MCU" Multipoint Control Unit (MCU)", as
      discussed in <xref target="RFC7667" />.</t> target="RFC7667"/>.</t>
      <figure title="RTP switch" anchor="rtpswitch"><artwork><![CDATA[ anchor="rtpswitch">
        <name>RTP Switch</name>
        <artwork><![CDATA[
         +---+      +------------+      +---+
         | A |<---->|            |<---->| B |
         +---+      |            |      +---+
                    |    RTP     |
         +---+      |   Switch   |      +---+
         | C |<---->|            |<---->| D |
         +---+      +------------+      +---+
]]>
    </artwork></figure>

    ]]></artwork>
      </figure>
      <t>In order to properly support the switching of video streams, the RTP switch typically needs
    some critical information about video frames in order to start and stop forwarding streams.
      <list style="symbols">
      </t>
      <ul>
        <li>

          <t>Because of inter-frame dependencies, it should ideally switch video streams at a point
        where the first frame from the new speaker can be decoded by recipients without prior
        frames, e.g e.g., switch on an intra-frame.</t>
        </li>
        <li>
          <t>In many cases, the switch may need to drop frames in order to realize congestion control
        techniques, and it needs to know which frames can be dropped with minimal impact to video quality.</t>
        </li>
        <li>
          <t>For scalable streams with dependent layers, the switch may need to selectively forward
        specific layers to specific recipients due to recipient bandwidth or decoder limits.</t>
      </list>
     </t>
        </li>
      </ul>
      <t>Furthermore, it is highly desirable to do this in a payload format-agnostic way which that is not
        specific to each different video codec.
        Most modern video codecs share common concepts around frame types and other critical information
          to make this codec-agnostic handling possible.</t>
      <t>It is also desirable to be able to do this for SRTP without requiring the video switch to
        decrypt the packets. SRTP will encrypt the RTP payload format contents and consequently contents; consequently, this
        data is not usable for the switching function without decryption, which may not even
        be possible in the case of end-to-end encryption of private media
        <xref target="RFC8871" />.</t> target="RFC8871"/>.</t>
      <t>By providing meta-information about the RTP streams outside the encrypted media payload, an
      RTP switch can do codec-agnostic selective forwarding without decrypting the payload.
      This document specifies the necessary meta-information in an RTP header extension.
      </t>
    </section>
    <section title="Key Words for Normative Requirements">
    <section>
      <name>Requirements Language</name>

        <t>
    The key words &quot;MUST&quot;, &quot;MUST NOT&quot;, &quot;REQUIRED&quot;,
        &quot;SHALL&quot;, &quot;SHALL NOT&quot;, &quot;SHOULD&quot;, &quot;SHOULD NOT&quot;, &quot;RECOMMENDED&quot;, &quot;NOT RECOMMENDED&quot;,  &quot;MAY&quot;, "<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>",
    "<bcp14>REQUIRED</bcp14>", "<bcp14>SHALL</bcp14>", "<bcp14>SHALL NOT</bcp14>",
    "<bcp14>SHOULD</bcp14>", "<bcp14>SHOULD NOT</bcp14>",
    "<bcp14>RECOMMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>",
    "<bcp14>MAY</bcp14>", and
        &quot;OPTIONAL&quot; "<bcp14>OPTIONAL</bcp14>" in this document are to be
    interpreted as described in
        BCP 14 BCP&nbsp;14 <xref target="RFC2119" /> target="RFC2119"/> <xref target="RFC8174" />
    target="RFC8174"/> when, and only when, they appear in all capitals, as
    shown here.
        </t>

    </section>

    <section title="Frame
    <section>
      <name>Video Frame Marking RTP Header Extension"> Extension</name>
      <t>This specification uses RTP header extensions as defined in <xref target="RFC8285" />. target="RFC8285"/>. A subset of
      meta-information from the video stream is provided as an RTP header extension to allow an RTP switch
      to do generic selective forwarding of video streams encoded with potentially different video codecs.</t>
      <t>The Video Frame Marking RTP header extension is encoded
      	using the one-byte header or two-byte header as described in <xref target="RFC8285" />. target="RFC8285"/>.
        The one-byte header format is used for examples in this memo. document.
        The two-byte header format is used when other two-byte header extensions
        are present in the same RTP packet, packet since mixing one-byte and two-byte extensions
        is not possible in the same RTP packet.</t>
      <t>This extension is only specified for Source (not Redundancy) RTP Streams
            <xref target="RFC7656" /> target="RFC7656"/> that carry video payloads.
            It is not specified for audio payloads, nor is it specified for Redundancy RTP Streams.
            The (separate) specifications for Redundancy RTP Streams often include
            provisions for recovering any header extensions that were part of the original source packet.
            Such provisions can be followed to recover the Video Frame Marking RTP header extension of the
            original source packet.
            Source packet frame markings may be useful when generating Redundancy RTP Streams;
            for example, the I (Independent Frame) and D (Discardable Frame) bits,
            defined in <xref target="mandatory-scalable" />, target="mandatory-scalable"/>,
            can be used to generate extra or no redundancy, respectively,
            and redundancy schemes with source blocks can align source block boundaries with
            independent frame boundaries as marked by the I bit.
      </t>
      <t>A frame, in the context of this specification, is the set of RTP packets
          with the same RTP timestamp from a specific RTP synchronization source Synchronization Source (SSRC).
          A frame within a layer is the set of RTP packets with the same RTP timestamp, SSRC,
          Temporal
          Temporal-layer ID (TID), and Layer ID (LID).</t>
      <section title="Long anchor="mandatory-scalable">
        <name>Long Extension for Scalable Streams" anchor="mandatory-scalable"> Streams</name>
        <t>The following RTP header extension is RECOMMENDED <bcp14>RECOMMENDED</bcp14> for scalable streams.
           It MAY <bcp14>MAY</bcp14> also be used for non-scalable streams, in which case the TID, LID LID, and TL0PICIDX MUST <bcp14>MUST</bcp14> be 0 or omitted.
           The ID is assigned per <xref target="RFC8285" />,
           and the target="RFC8285"/>.
           The length is encoded as L=2 which indicates follows:</t>
	   <ul>
	     <li>L=2 to indicate 3 octets of data when nothing is omitted,
           or L=1 omitted,</li>
	     <li>L=1 for 2 octets when TL0PICIDX is omitted, or L=0 or</li>
	     <li>L=0 for 1 octet when both the LID and TL0PICIDX are omitted.</t>
      	<figure> omitted.</li></ul>
        <artwork><![CDATA[
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  ID=? |  L=2  |S|E|I|D|B| TID |   LID         |    TL0PICIDX  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
           or
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  ID=? |  L=1  |S|E|I|D|B| TID |   LID         | (TL0PICIDX omitted)
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
           or
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  ID=? |  L=0  |S|E|I|D|B| TID | (LID and TL0PICIDX omitted)
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
]]></artwork></figure>
]]></artwork>
        <t>The following information are is extracted from the media payload and sent in the Video Frame Marking RTP header extension.
         	<list style='symbols'>
            <t>S:
        </t>
        <dl newline="true">

            <dt>S: Start of Frame (1 bit) - MUST bit)</dt><dd><bcp14>MUST</bcp14> be 1 in the first packet in a frame
                within a layer; otherwise MUST otherwise, <bcp14>MUST</bcp14> be 0.</t>
            <t>E: 0.</dd>

            <dt>E: End of Frame (1 bit) - MUST bit)</dt><dd><bcp14>MUST</bcp14> be 1 in the last packet in a frame
                within a layer; otherwise MUST otherwise, <bcp14>MUST</bcp14> be 0.
                Note that the RTP header marker bit MAY <bcp14>MAY</bcp14> be used to infer the last packet of the highest enhancement layer, layer in payload formats with such semantics.</t>
            <t>I: semantics.</dd>

            <dt>I: Independent Frame  (1 bit) - MUST bit)</dt><dd><bcp14>MUST</bcp14> be 1 for a frame within a layer that can be
               decoded independent of temporally prior frames, e.g. e.g., intra-frame, VPX keyframe,
               H.264 IDR Instantaneous Decoding Refresh (IDR) <xref target="RFC6184" />, target="RFC6184"/>, or
               H.265 IDR/CRA/BLA/RAP IDR / Clean Random Access (CRA) / Broken Link Access (BLA) / Random Access Point (RAP) <xref target="RFC7798" />;
               otherwise MUST target="RFC7798"/>;
               otherwise, <bcp14>MUST</bcp14> be 0.
               Note that this bit only signals temporal independence, so it can be
               1 in spatial or quality enhancement layers that depend on temporally
               co-located layers but not temporally prior frames.</t>
            <t>D: frames.</dd>

            <dt>D: Discardable Frame (1 bit) - MUST bit)</dt><dd><bcp14>MUST</bcp14> be 1 for a frame within a layer the sender knows can be discarded, discarded
                and still provide a decodable media stream; otherwise MUST otherwise, <bcp14>MUST</bcp14> be 0. </t>
            <t>B: </dd>

            <dt>B: Base Layer Sync (1 bit) - When bit)</dt><dd>When the TID is not 0, this MUST <bcp14>MUST</bcp14> be 1 if the sender knows this frame within a layer only depends
                on the base temporal layer; otherwise MUST otherwise, <bcp14>MUST</bcp14> be 0. When the TID is 0 or if no scalability is used, this MUST <bcp14>MUST</bcp14> be 0.</t>
            <t>TID: Temporal 0.
          </dd>

            <dt>TID: Temporal-layer ID (3 bits) - Identifies bits)</dt><dd>Identifies the temporal layer/sub-layer encoded,
               starting with 0 for the base layer, layer and increasing with higher temporal fidelity.
               If no scalability is used, this MUST <bcp14>MUST</bcp14> be 0. It is implicitly 0 in the short extension format.</t>
            <t>LID: format.
          </dd>

            <dt>LID: Layer ID (8 bits) - Identifies bits)</dt><dd>Identifies the spatial and quality layer encoded,
               starting with 0 for the base layer, layer and increasing with higher fidelity.
               If no scalability is used, this MUST <bcp14>MUST</bcp14> be 0 or omitted to reduce length.
               When the LID is omitted, TL0PICIDX MUST <bcp14>MUST</bcp14> also be omitted. It is implicitly 0 in the short extension format
               or when omitted in the long extension format.</t>
            <t>TL0PICIDX: format.</dd>

            <dt>TL0PICIDX: Temporal Layer 0 Picture Index (8 bits) - When bits)</dt><dd>When the TID is 0 and the LID is 0, this is a cyclic counter labeling
               base layer frames. When the TID is not 0 or the LID is not 0,
               this indicates
               the indication is that a dependency on the given index, such that this frame within this layer
               depends on the frame with this label in the layer with a TID 0 and LID 0.
               If no scalability is used, or the cyclic counter is unknown, this MUST TL0PICIDX <bcp14>MUST</bcp14> be omitted to reduce length.
               Note that 0 is a valid index value for TL0PICIDX.</t>
         	</list>
      	</t> TL0PICIDX.</dd>

        </dl>
        <t>The layer information contained in the TID and LID convey useful aspects of the layer structure that
          can be utilized in selective forwarding.</t>
        <t>Without further information about the layer structure,
          these TID/LID identifiers can only be used for relative priority of layers
          and implicit dependencies between layers.
          They convey a layer hierarchy with TID=0 TID = 0 and LID=0 LID = 0 identifying the base layer.
          Higher values of TID identify higher temporal layers with higher frame rates.
          Higher values of LID identify higher spatial and/or quality layers with higher resolutions and/or bitrates.
          Implicit dependencies between layers assume that a layer with a given TID/LID MAY <bcp14>MAY</bcp14> depend
          on layer(s) a layer or layers with the same or lower TID/LID, but MUST NOT they <bcp14>MUST NOT</bcp14> depend on layer(s) a layer or layers with higher TID/LID.
          </t><t>
        </t>
        <t>
          With further information,
          for example, possible future RTCP SDES source description (SDES) items that convey full layer structure information, it may
          be possible to map these TIDs and LIDs to specific absolute frame rates, resolutions and resolutions, bitrates,
          as well as and explicit dependencies between layers.
          Such additional layer information may be useful for forwarding decisions in the RTP switch, switch
          but is beyond the scope of this memo. document. The relative layer information is still useful
          for many selective forwarding decisions decisions, even without such additional layer information.
        </t>
      </section>
      <section title="Short anchor="mandatory-non-scalable">
        <name>Short Extension for Non-Scalable Streams" anchor="mandatory-non-scalable"> Streams</name>
        <t>The following RTP header extension is RECOMMENDED <bcp14>RECOMMENDED</bcp14> for non-scalable streams.
           It is identical to the shortest form of the extension for scalable streams,
           except the last four bits (B and TID) are replaced with zeros.
           It MAY <bcp14>MAY</bcp14> also be used for scalable streams if the sender has limited or no
           information about stream scalability.
           The ID is assigned per <xref target="RFC8285" />,
           and target="RFC8285"/>;
           the length is encoded as L=0 L=0, which indicates 1 octet of data.</t>

      	<figure>
        <artwork><![CDATA[
 0                   1
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  ID=? |  L=0  |S|E|I|D|0 0 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

]]></artwork></figure>

]]></artwork>
        <t>The following information are is extracted from the media payload and sent in the Video Frame Marking RTP header extension.
          <list style='symbols'>
            <t>S:
        </t>

	          <dl newline="true">
            <dt>S: Start of Frame (1 bit) - MUST bit)</dt><dd><bcp14>MUST</bcp14> be 1 in the first packet in a frame; otherwise MUST otherwise, <bcp14>MUST</bcp14> be 0.</t>
            <t>E: 0.</dd>

            <dt>E: End of Frame (1 bit) - MUST bit)</dt><dd><bcp14>MUST</bcp14> be 1 in the last packet in a frame; otherwise MUST otherwise, <bcp14>MUST</bcp14> be 0.
            SHOULD
            <bcp14>SHOULD</bcp14> match the RTP header marker bit in payload formats with such semantics for marking end of frame.</t>
            <t>I: frame.</dd>

            <dt>I: Independent Frame  (1 bit) - MUST bit)</dt><dd><bcp14>MUST</bcp14> be 1 for frames that can be
               decoded independent of temporally prior frames, e.g. e.g., intra-frame, VPX keyframe,
               H.264 IDR <xref target="RFC6184" />, target="RFC6184"/>, or
               H.265 IDR/CRA/BLA/IRAP <xref target="RFC7798" />;
               otherwise MUST target="RFC7798"/>;
               otherwise, <bcp14>MUST</bcp14> be 0. </t>
            <t>D: </dd>
            <dt>D: Discardable Frame (1 bit) - MUST bit)</dt><dd><bcp14>MUST</bcp14> be 1 for frames the sender knows can be discarded, discarded
                and still provide a decodable media stream; otherwise MUST otherwise, <bcp14>MUST</bcp14> be 0. </t>
            <t>The </dd>

            <dt>The remaining (4 bits) - bits)</dt><dd>These are reserved/fixed values and not used for non-scalable streams;
              they MUST <bcp14>MUST</bcp14> be set to 0 zero upon transmission and ignored upon reception.</t>
          </list>
        </t> reception.</dd>
          </dl>

      </section>

      <section title="Layer ID
      <section>
        <name>LID Mappings for Scalable Streams"> Streams</name>
        <t> This section maps the specific Layer ID (LID) information contained in specific scalable codecs to the generic LID and TID fields. </t>
        <t> Note that non-scalable streams have no Layer ID information and thus LID information; thus, they have no mappings. </t>

      <section title="VP9
        <section>
          <name>VP9 LID Mapping"> Mapping</name>
          <t> The VP9 <xref target="I-D.ietf-payload-vp9" />
           Spatial Layer target="RFC9628"/>
           Spatial-layer ID (SID, 3 bits) and Temporal Layer Temporal-layer ID (TID, 3 bits)
           in the VP9 payload descriptor are mapped to the generic LID and TID fields
           in the header extension as shown in the following figure.</t>

        <figure>
          <artwork><![CDATA[
   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |  ID=? |  L=2  |S|E|I|D|B| TID |0|0|0|0|0| SID |    TL0PICIDX  |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  ]]></artwork></figure>
  ]]></artwork>
          <t> The S bit MUST <bcp14>MUST</bcp14> match the B bit in the VP9 payload descriptor.</t>
          <t> The E bit MUST <bcp14>MUST</bcp14> match the E bit in the VP9 payload descriptor.</t>
          <t> The I bit MUST <bcp14>MUST</bcp14> match the inverse of the P bit in the VP9 payload descriptor.</t>

          <t> The D bit MUST <bcp14>MUST</bcp14> be 1 if the refresh_frame_flags bits in the VP9 payload uncompressed header are all 0, otherwise 0; otherwise, it MUST <bcp14>MUST</bcp14> be 0.</t>
          <t> The B bit MUST <bcp14>MUST</bcp14> be 0 if the TID is 0; otherwise, if the TID is not 0, it MUST <bcp14>MUST</bcp14> match the U bit in the VP9 payload descriptor. Note: When descriptor.</t> <aside><t>Note: when using temporally nested scalability structures as recommended in <xref target="scalable-structures" />, target="scalable-structures"/>, the B bit and VP9 U bit will always be 1 if the TID is not 0, 0 since it is always
           possible to switch up to a higher temporal layer in such nested structures.</t>
       <t> structures.</t></aside>
          <t>The TID, SID SID, and TL0PICIDX MUST <bcp14>MUST</bcp14> match the correspondingly named fields in the VP9 payload descriptor,
           with SID aligned in the least significant 3 bits of the 8-bit LID field and zeros
           in the most significant 5 bits.</t>
        </section>

    <section title="H265
        <section>
          <name>H265 LID Mapping"> Mapping</name>

          <t> The H265 <xref target="RFC7798" /> LayerID target="RFC7798"/> layer ID (6 bits) bits), and TID (3 bits)
             from the NAL Network Abstraction Layer (NAL) unit header are mapped to the generic LID and TID fields
             in the header extension as shown in the following figure.</t>

      <figure>
          <artwork><![CDATA[
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  ID=? |  L=2  |S|E|I|D|B| TID |0|0|  LayerID layer ID  |    TL0PICIDX  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
]]></artwork></figure>
]]></artwork>

          <t>The S and E bits MUST <bcp14>MUST</bcp14> match the correspondingly named bits in PACI:PHES:TSCI payload structures.</t>
          <t>The I bit MUST <bcp14>MUST</bcp14> be 1 when the NAL unit type is 16-23 (inclusive) or 32-34 (inclusive), or an aggregation packet or fragmentation unit encapsulating any of these types, otherwise types; otherwise, it MUST <bcp14>MUST</bcp14> be 0. These ranges cover intra (IRAP) frames as well as
          critical parameter sets (VPS, SPS, PPS).</t> (Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS)).</t>

          <t>The D bit MUST <bcp14>MUST</bcp14> be 1 when the if either:</t>
	  <ul>
	    <li>the payload's NAL unit type header's NRI field is 0, 2, 4, 6, 8, 10, 12, 14, or 38, or or</li>
	    <li>the payload is an aggregation packet or fragmentation unit encapsulating only these types, otherwise NAL units with NRI = 0.</li>
	  </ul>
	  <t>Otherwise, it MUST <bcp14>MUST</bcp14> be 0. These ranges cover 0.</t>
	  <t>The NRI = 0 condition signals non-reference frames as well as filler data.</t> frames.</t>
          <t>The B bit can not cannot be determined reliably from simple inspection of payload headers, and therefore headers; therefore, it is determined by implementation-specific means. For example, internal codec interfaces may provide information to set this reliably.</t>
       <t>
          <t>The TID and LayerID MUST layer ID <bcp14>MUST</bcp14> match the correspondingly named fields in the H265 NAL unit header,
           with LayerID layer ID aligned in the least significant 6 bits of the 8-bit LID field and zeros
           in the most significant 2 bits.</t>
        </section>

    <section title="H264-SVC
        <section>
          <name>H264 Scalable Video Coding (SVC) LID Mapping"> Mapping</name>
          <t> The following shows H264-SVC <xref target="RFC6190" /> target="RFC6190"/> Layer encoding information (3 bits for
   spatial/dependency layer, layer (DID), 4 bits for quality layer (QID), and 3 bits for temporal layer) mapped to the generic LID and TID fields.</t>
          <t>The S, E, I I, and D bits MUST <bcp14>MUST</bcp14> match the correspondingly named bits in PACSI Payload Content Scalability Information (PACSI) payload structures.</t>
          <t>The I bit MUST <bcp14>MUST</bcp14> be 1 when the NAL unit type is 5, 7, 8, 13, or 15,
     or an aggregation packet or fragmentation unit encapsulating any of these types, otherwise types; otherwise, it MUST <bcp14>MUST</bcp14> be 0. These ranges cover intra (IDR) frames as well as
       critical parameter sets (SPS/PPS variants).</t>
       <t>The D bit MUST <bcp14>MUST</bcp14> be 1 when the if either:</t>
       <ul>
	 <li>the payload's NAL unit header header's NRI field is 0, or or</li>
	 <li>the payload is an aggregation packet or fragmentation unit encapsulating only NAL units with NRI=0, otherwise NRI = 0.</li>
       </ul>
	 <t>Otherwise, it MUST <bcp14>MUST</bcp14> be 0.
       The NRI=0 0.</t>
       <t>The NRI = 0 condition signals non-reference frames.</t>
          <t>The B bit can not cannot be determined reliably from simple inspection of payload headers, and therefore headers; therefore, it is determined by implementation-specific means. For example, internal codec interfaces may provide information to set this reliably.</t>

      <figure>
          <artwork><![CDATA[
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  ID=? |  L=2  |S|E|I|D|B| TID |0| DID |  QID  |    TL0PICIDX  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
]]></artwork></figure>
]]></artwork>
        </section>

    <section title="H264
        <section>
          <name>H264 Advanced Video Coding (AVC) LID Mapping"> Mapping</name>
          <t> The following  shows the header extension for H264 (AVC) <xref target="RFC6184" /> target="RFC6184"/> that contains
    only temporal layer information.</t>
          <t> The S bit MUST <bcp14>MUST</bcp14> be 1 when the timestamp in the RTP header differs from the timestamp
     in the prior RTP sequence number from the same SSRC, otherwise SSRC; otherwise, it MUST <bcp14>MUST</bcp14> be 0.</t>
          <t> The E bit MUST <bcp14>MUST</bcp14> match the M bit in the RTP header.</t>
          <t>The I bit MUST <bcp14>MUST</bcp14> be 1 when the NAL unit type is 5, 7, or 8,
     or an aggregation packet or fragmentation unit encapsulating any of these types,
     otherwise types;
     otherwise, it MUST <bcp14>MUST</bcp14> be 0. These ranges cover intra (IDR) frames as well as
     critical parameter sets (SPS/PPS).</t>
     <t>The D bit MUST <bcp14>MUST</bcp14> be 1 when the if either:</t>
     <ul>
       <li>the payload's NAL unit header header's NRI field is 0,
      or
       or</li>
       <li>the payload is an aggregation packet or fragmentation unit encapsulating only
       NAL units with NRI=0, otherwise NRI = 0.</li>
     </ul>
     <t>Otherwise, it MUST <bcp14>MUST</bcp14> be 0.
      The NRI=0 0.</t>
      <t>The NRI = 0 condition signals non-reference frames.</t>
          <t>The B bit can not cannot be determined reliably from simple inspection of payload headers, and therefore headers; therefore, it is determined by implementation-specific means. For example, internal codec interfaces may provide information to set this reliably.</t>
      <figure>
          <artwork><![CDATA[
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  ID=? |  L=2  |S|E|I|D|B| TID |0|0|0|0|0|0|0|0|    TL0PICIDX  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
]]></artwork></figure>
]]></artwork>
        </section>

    <section title="VP8
        <section>
          <name>VP8 LID Mapping"> Mapping</name>
          <t> The following  shows the header extension for VP8 <xref target="RFC7741" /> target="RFC7741"/> that contains
    only temporal layer information.</t>
          <t> The S bit MUST <bcp14>MUST</bcp14> match the correspondingly named bit in the VP8 payload descriptor when PID=0, otherwise PID=0; otherwise, it MUST <bcp14>MUST</bcp14> be 0.</t>
          <t> The E bit MUST <bcp14>MUST</bcp14> match the M bit in the RTP header. </t>
          <t> The I bit MUST <bcp14>MUST</bcp14> match the inverse of the P bit in the VP8 payload header.</t>
          <t> The D bit MUST <bcp14>MUST</bcp14> match the N bit in the VP8 payload descriptor.</t>

<t> The B bit MUST <bcp14>MUST</bcp14> match the Y bit in the VP8 payload descriptor. Note: When descriptor.</t>
<aside><t>Note: when using temporally nested scalability structures as recommended in <xref target="scalable-structures" />, target="scalable-structures"/>, the B bit and VP8 Y bit will always be 1 if the TID is not 0, 0 since it is always
       possible to switch up to a higher temporal layer in such nested structures.</t>
   <t> structures.</t></aside>
          <t>The TID and TL0PICIDX MUST <bcp14>MUST</bcp14> match the correspondingly named fields in the VP8 payload descriptor. </t>
      <figure>
          <artwork><![CDATA[
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  ID=? |  L=2  |S|E|I|D|B| TID |0|0|0|0|0|0|0|0|    TL0PICIDX  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
]]></artwork></figure>
]]></artwork>
        </section>

    <section title="Future
        <section>
          <name>Future Codec LID Mapping"> Mapping</name>
          <t>The RTP payload format specification for future video codecs SHOULD <bcp14>SHOULD</bcp14> include a section describing
        the LID mapping and TID mapping for the codec.</t>
        </section>
      </section>

    <section title="Signaling Information">
      <section>
        <name>Signaling Information</name>
        <t>The URI for declaring this header extension in an extmap attribute is
      "urn:ietf:params:rtp-hdrext:framemarking". It does not contain any
      extension attributes. </t>
        <t>An example attribute line in SDP:</t>
      <figure>
        <artwork><![CDATA[
   a=extmap:3 urn:ietf:params:rtp-hdrext:framemarking
]]></artwork></figure>
]]></artwork>
      </section>

    <section title="Usage Considerations">
      <section>
        <name>Usage Considerations</name>

        <t>The header extension values MUST <bcp14>MUST</bcp14> represent what is already in the RTP payload.</t>
      <t> When
        <t>When an RTP switch needs to discard a received video frame frames due to congestion control considerations,
	it is RECOMMENDED <bcp14>RECOMMENDED</bcp14> that it preferably drop frames drop:</t>
	<ul>
	  <li>frames marked with the D (Discardable) bit set,
        or
          or</li>
	  <li>frames with the highest values of TID and LID, which LID (which indicate the highest temporal and spatial/quality enhancement layers, layers) since those typically have fewer dependenices dependencies on them than lower layers.</t> layers.</li></ul>

        <t> When an RTP switch wants to forward a new video stream to a receiver, it is RECOMMENDED <bcp14>RECOMMENDED</bcp14> to
	select the new video stream from the first switching point with the I (Independent) bit set in all spatial layers and forward the same. video stream from that point on. An RTP switch can request that a media source to generate a switching point by sending an RTCP
      Full Intra Request (RTCP FIR) (FIR) as defined in <xref target="RFC5104" />, target="RFC5104"/>, for example. </t>

      <section title="Relation
        <section>
          <name>Relation to Layer Refresh Request (LRR)"> (LRR)</name>
          <t>Receivers can use the Layer Refresh Request (LRR) <xref target="I-D.ietf-avtext-lrr" /> target="RFC9627"/>
          RTCP feedback message
          to upgrade to a higher layer in scalable encodings. The TID/LID values
          and formats used in LRR messages MUST <bcp14>MUST</bcp14> correspond to the same values and formats
          specified in <xref target="mandatory-scalable" />. target="mandatory-scalable"/>.
          </t>

          <t>Because frame marking can only be used with temporally-nested temporally nested streams,
            temporal-layer LRR refreshes requested with an LRR message are unnecessary for frame-marked streams.
          Other refreshes can be detected based on the I bit being set for the specific spatial layers.
          </t>
        </section>
        <section title="Scalability Structures" anchor="scalable-structures">
          <name>Scalability Structures</name>
          <t>The LID and TID information is most useful for fixed scalability structures,
            such as nested hierarchical temporal layering structures, where each temporal
            layer only references lower temporal layers or the base temporal layer.
            The LID and TID information is less useful, or even not useful at all,
            for complex, irregular scalability structures that do not conform to common,
            fixed patterns of inter-layer dependencies and referencing structures.
            Therefore
            Therefore, it is RECOMMENDED <bcp14>RECOMMENDED</bcp14> to use LID and TID information for
            RTP switch forwarding decisions only in the case of temporally nested
            scalability structures, and it is NOT RECOMMENDED <bcp14>NOT RECOMMENDED</bcp14> for other
            (more complex or irregular) scalability structures.</t>
        </section>
      </section>
    </section>

    <section title="Security Considerations
    <section>
      <name>Security and Privacy Considerations"  > Considerations</name>
      <t>In the Secure Real-Time Transport Protocol (SRTP) <xref "<xref target="RFC3711" />, format="title"/>" <xref target="RFC3711"/>, RTP header extensions are
      authenticated and optionally encrypted <xref target="RFC9335" />. target="RFC9335"/>.
      When unencrypted header extensions are used, some metadata is
      exposed and visible to middle boxes middleboxes on the network path,
      while encrypted media data and metadata in encrypted header extensions are not exposed.</t>
      <t>The primary utility of this specification is for RTP switches to make proper media forwarding decisions.
      RTP switches are the SRTP peers of endpoints, so they can access encrypted header extensions,
      but not end-to-end encrypted private media payloads. Other middle boxes middleboxes on the network path can only access
      unencrypted header extensions, extensions since they are not SRTP peers.</t>
      <t>RTP endpoints which that negotiate this extension should consider whether this whether:
      </t>
      <ul><li>this video frame marking metadata
      needs to be exposed to the SRTP peer only, in which case the header extension can be encrypted; or whether
      other middle boxes or</li>
      <li>other middleboxes on the network path also need this metadata, for example, to optimize packet drop decisions
      that minimize media quality impacts, in which case the header extension can be unencrypted, if the endpoint
      accepts the potential privacy leakage of this metadata. metadata.</li>
      </ul>
<t>
      For example, it would be possible to determine
      keyframes and their frequency in unencrypted header extensions. This information can often be obtained via
      statistical analysis of encrypted data. For example, keyframes are usually much larger than other frames,
      so frame size alone can leak this in the absence of any unencrypted metadata. However, unencrypted metadata
      provides a reliable signal rather than a statistical probability; so endpoints should take that into consideration
      to balance the privacy leakage risk against the potential benefit of optimized media delivery when deciding
      whether to negotiate and encrypt this header extension.</t>
    </section>

	<section title="Acknowledgements">
      <t>Many thanks

    <!--[rfced] Regarding IANA:

a) Please note that we will communicate the following AUTH48 update to Bernard Aboba, Jonathan Lennox, Stephan Wenger, Dale Worley, and Magnus Westerlund
IANA for their inputs.</t>
    </section>

    <section title="IANA Considerations">
      <t>This an update at
https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml
once all authors have approved the document:

Old:
urn:ietf:params:rtp-hdrext:framemarkinginfo

New:
urn:ietf:params:rtp-hdrext:framemarking

b) We will update to use "registry" and "registry group" instead of
"subregistry" and "registry" if IANA so wishes in this text:

Current:
   This document defines a new extension URI listed in the "RTP Compact
   Header Extensions" subregistry of the "Real-Time Transport Protocol
   (RTP) Parameters" registry, according to the RTP following data:

Perhaps:
   This document defines a new extension URI listed in the "RTP Compact HeaderExtensions sub-registry
   Header Extensions" registry of the
      	Real-Time "Real-Time Transport Protocol
   (RTP) Parameters Parameters" registry group, according to the following data:

-->
    <section>
      <name>IANA Considerations</name>
      <t>This document defines a new extension URI listed in the "RTP Compact Header Extensions" subregistry of the
      	"Real-Time Transport Protocol (RTP) Parameters" registry, according to the following data:</t>
      <t>Extension URI:  urn:ietf:params:rtp-hdrext:framemarkinginfo  urn:ietf:params:rtp-hdrext:framemarking </t>
      <t>Description: Frame marking information for video streams </t>
      <t>Contact: mzanaty@cisco.com </t>
      <t>Reference: RFC XXXX</t>

      <t>Note to RFC Editor: please replace RFC XXXX with the number of this RFC.</t> 9626</t>

    </section>
  </middle>
  <back>

    <references title="Normative References">
    <?rfc include="reference.RFC.2119"?>
    <?rfc include="reference.RFC.8174"?>
    <?rfc include="reference.RFC.8285"?>
    <?rfc include="reference.RFC.6184"?>
    <?rfc include="reference.RFC.6190"?>
    <?rfc include="reference.RFC.7741"?>
    <?rfc include="reference.RFC.7798"?>

    <references>

      <name>References</name>
      <references>
        <name>Normative References</name>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2119.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8174.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8285.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.6184.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.6190.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.7741.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.7798.xml"/>
      </references>
    <references title="Informative References">
    <?rfc include="reference.RFC.7656"?>
    <?rfc include="reference.RFC.7667"?>
    <?rfc include="reference.RFC.6464"?>
    <?rfc include="reference.RFC.3550"?>
    <?rfc include="reference.RFC.3711"?>
    <?rfc include="reference.RFC.5104"?>
    <?rfc include="reference.RFC.8871"?>
    <?rfc include="reference.RFC.9335"?>
    <?rfc include="reference.I-D.ietf-avtext-lrr"?>
    <?rfc include="reference.I-D.ietf-payload-vp9"?>
      <references>
        <name>Informative References</name>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.7656.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.7667.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.6464.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.3550.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.3711.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.5104.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8871.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9335.xml"/>

<!-- [I-D.ietf-avtext-lrr]; Companion document -->

<reference anchor="RFC9627" target="https://www.rfc-editor.org/info/rfc9627">
   <front>
      <title>The Layer Refresh Request (LRR) RTCP Feedback Message</title>
      <author initials="J." surname="Lennox" fullname="Jonathan Lennox">
         <organization>Vidyo, Inc.</organization>
      </author>
      <author initials="D." surname="Hong" fullname="Danny Hong">
         <organization>Vidyo, Inc.</organization>
      </author>
      <author initials="J." surname="Uberti" fullname="Justin Uberti">
         <organization>Google, Inc.</organization>
      </author>
      <author initials="S." surname="Holmer" fullname="Stefan Holmer">
         <organization>Google, Inc.</organization>
      </author>
      <author initials="M." surname="Flodman" fullname="Magnus Flodman">
         <organization>Google, Inc.</organization>
      </author>
      <date month="February" year="2025" />
   </front>
   <seriesInfo name="RFC" value="9627" />
   <seriesInfo name="DOI" value="10.17487/RFC9627"/>

</reference>

<!-- [I-D.ietf-payload-vp9]; Companion document -->

<reference anchor="RFC9628" target="https://www.rfc-editor.org/info/rfc9628">
   <front>
      <title>RTP Payload Format for VP9 Video</title>
      <author initials="J." surname="Uberti" fullname="Justin Uberti">
         <organization>Google, Inc.</organization>
      </author>
      <author initials="S." surname="Holmer" fullname="Stefan Holmer">
         <organization>Google, Inc.</organization>
      </author>
      <author initials="M." surname="Flodman" fullname="Magnus Flodman">
         <organization>Google, Inc.</organization>
      </author>
      <author initials="D." surname="Hong" fullname="Danny Hong">
         <organization>Google, Inc.</organization>
      </author>
      <author initials="J." surname="Lennox" fullname="Jonathan Lennox">
         <organization>8x8, Inc. / Jitsi</organization>
      </author>
      <date month="February" year="2025" />

   </front>

     <seriesInfo name="RFC" value="9628"/>
  <seriesInfo name="DOI" value="10.17487/RFC9628"/>

</reference>

      </references>
    </references>

        <section numbered="false">
      <name>Acknowledgements</name>
      <t>Many thanks to  <contact fullname="Bernard Aboba"/>,  <contact fullname="Jonathan Lennox"/>,  <contact fullname="Stephan Wenger"/>,  <contact fullname="Dale Worley"/>, and  <contact fullname="Magnus Westerlund"/> for their inputs.</t>
	</section>
  </back>
</rfc>