Redundant RTP coding

There are different mechanisms to make media transmission more robust to packet loss.   Some of those techniques are negative acknowledgments and retransmissions, sending redundant information for forward error correction and signal processing algorithms to reduce the impact of packet looses.

In case of audio codecs with the advent of OPUS most of packet lost issues disappear due to the robustness of the codec.  You can test yourself the quality with 30% of packet loss!!! [1]

Where those techniques makes more sense are with video codecs because they are typically more fragile to packet loss and this is specially critical when sending key frames that can impact communication for a long period of time (until next keyframe).

This post is an overview on how redundant encoding works at RTP level and what are the implications of it.

The core idea is to modify RTP packets format so that instead of including only the primary content payload (video encoded content) it can also include extra blocks with other codifications of the same content that can be used to recover lost packets.    This redundant codification could be a lower quality encoding of the previous frame or a forward error correction bitstream containing information of the last N packets for example.

The basic format from RFC 2198 is this one:
    0                   1                    2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3  4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |V=2|P|X| CC=0  |M|      PT     |   sequence number of primary  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |              timestamp  of primary encoding                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           synchronization source (SSRC) identifier            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |1| block PT=117|  timestamp offset         |   block length    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| block PT=100|                                               |
   +-+-+-+-+-+-+-+-+                                               +
   +               fec encoded redundant data (PT=117)             +
   +                                               +---------------+
   |                                               |               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+               +
   |               video encoded primary data (PT=100)             |
   +                                               +---------------+
   |                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

It is worth noting that:




    

  1. The RTP header format (first 12 bytes) doesn't change
    2. 

  2. The payload type of the RTP packet (byte 1) won't be the payload type of the primary codec (f.e. VP8) but the payload identifying the usage of packets with this multi-block structure
    3. 

  3. The primary data is always at the end if present
    4. 

  4. Is it possible to have packets only with redundant data or only with primary data
    5. 






There are different solutions to encode the redundant data, one of them is ulpfec [3] where the redundant data is an XOR of different packets.   ulpfec includes the concept of layers so that we can send more redudand data for the initial part of the packet and less redundant data for the latest parts of the packets.



Support for red (for packetization) and ulpfec (for actual protection) is usually negotiated in SDP:

    m=video 1 RTP/SAVPF 100 116 117

    a=rtpmap:100 VP8/90000

    a=rtpmap:116 red/90000

    a=rtpmap:117 ulpfec/90000



Due to the overhead introduced by adding any redundant data it is usually a good idea to activate it only when the network conditions require it.   That is what is done in WebRTC implementations where all the video packets are sent in redundant format but they only start including redundant data after detecting packet losses in the connection.



The good news is that all these solutions are being integrated in next generation of browsers supporting WebRTC and will make our live as customers or service providers much easier.



[1] opus-codec.org/examples/samples/plc30_opus.wav

[2] http://tools.ietf.org/html/rfc2198

[3] http://tools.ietf.org/html/rfc5109































Comments











Post a Comment



















Popular posts from this blog









Bandwidth Estimation in WebRTC (and the new Sender Side BWE)




















Image







Bandwidth estimation is probably the most critical component in the video engine of WebRTC.  The bandwidth estimation (BWE) module is responsible for deciding how much video* traffic you can send without congesting the network to prevent degradation of the video quality.     In the past bandwidth estimation algorithms used to be very rudimentary and based mostly on packet loss.  Basically we used to start increasing slowly the video bitrate until we detected packets being lost.  To detect the packet loss you use the standard RTCP feedback mechanisms where the receiver side reports packet loss periodically using RTCP Receiver Report (RR) messages.   Modern bandwidth estimation algorithms are more advanced and try to detect congestion before it is bad enough to make routers discard packets.   These algorithms predicts congestion analyzing the delay between packets.  The idea is that when you start having some congestion, the buffers in the routers will start filling and the delay will be








Read more










Improving Real Time Communications with Machine Learning




















Image







 When we talk about the applications of Artificial Intelligence / Machine Learning (AI/ML) for Real Time Communications (RTC) we can group them in two different planes:   Service Level : There are many features that can be added to a videoconference service, for example identification of the participants, augmented reality, emotion detection, speech transcription or audio translation .  These features are usually based on image and speech recognition and language processing.  Infrastructure Level : There are many ways to apply ML that do not provide new features but improve the quality and/or reliability of the audio/video transmission.      Service level applications are fun, but they are more for Product Managers and I like technology more, so in the next sections I will try to describe possible applications of AI/ML for Real Time Communications at Infrastructure Level organizing those ideas in five different categories.    Optimizing video quality   Some of the ML algorithms used fo








Read more










Controlling bandwidth usage in WebRTC (and how googSuspendBelowMinBitrate works)




















Image







There are cases when we would like to limit the maximum bitrate being transmitted by WebRTC to avoid wasting resources in the user endpoints or save money reducing the bandwidth usage in our servers.   This is because the maximum bitrate by default in Chrome is around 2Mbps and for many use cases a much lower bitrate provides still pretty good quality.   BTW using a lower bitrate can also help with stability of quality in multiparty scenarios by reducing the amount of competition among different streams.   There is no simple API to configure the maximum bitrate in WebRTC (although there is one in ORTC) but there are 3 ways to do this by mangling the SDP.   1. U se the standard  b=AS:BITRATE (Chrome) or b=TIAS:BITRATE  (Firefox) attributes in the SDP for the audio or video channel[1]  2. Use codec specific attributes (this work at least for opus audio codec with maxaveragebitrate property) [2]  3. Use the proprietary x-max-bitrate attribute in the video channel of the SDP answer.  For e








Read more