How are retransmissions implemented in WebRTC

Executive Summary: This ended up being more complex than expected, if you are building an SFU just take the nack_module.cc & co files and use them.  Or even better, just use OpenTok or any other existing platform.   Anyway it is fun stuff to play with.

Introduction

When your network is congested and some of the packets are lost you need a mechanism to recover them.   If we ignore signal processing tricks (like stretching the previous and next audio packets in the jitter buffer) there are two alternatives:
  • Forward error correction: in each packet you add some extra information about the previous ones in case they are lost and you need to reconstruct them (flexfec is the new format to do this in WebRTC [1]).
  • Retransmissions: the sender will resend the packets usually after the receiver requests that with a negative acknowledgement message (NACK) indicating which packets were lost.
These mechanisms can be combined depending on the network conditions and can also be tuned for specific cases like scalable video as described in [2]. In Chrome the retransmissions are implemented for both audio and video but by default it is only enabled for video.

This post is about the retransmissions approach, specifically about when to ask for a retransmission and when not to do it.

Disclaimer: Please report any mistake/inaccuracy in this post and I will fix it asap for future reference. BTW this post is not about rtx "encapsulation" vs the legacy way of sending retransmissions but I'm happy to talk about that in twitter :)

Implementation in the RTP Receiver Side


The RTP receiver side is the one requesting the retransmission sending a NACK RTCP packet when it detects a missing packet.

But... should I request for retransmissions immediately after detecting a gap in the sequence numbers?  Google implementation does that plus also have a periodic timer to keep requesting them.

And... for how long should I keep requesting the retransmission? Chrome keeps requesting the retransmission of a specific packet unless the sequence number is "more than 10000 old", the number of missing packets in the list is larger than 1000, you asked for the same packet 10 times already or you have a new decodable full frame (no packet is missing for any other frame it depends on).

This is the Google implementation of WebRTC (and the same code is copied into Firefox and Safari) transcribed as pseudocode:


Implementation in the RTP Sender Side


The RTP sender side is the one receiving the retransmission requests (NACKs) and resending the lost packets if possible.

The main question here is if I should obey all the retransmission requests or not.  The way this is implemented in Google's WebRTC implementation right now is this one:
  1. Keep a copy of the packets sent in the last 1000 msecs (the "history").
  2. When a NACK is received try to send the packets requests if we still have them in the history.
  3. But... (rtp_sender.cc)
    • Ignore the request if the packet has been resent in the last RTT msecs.
    • Ignore the request if we are sending too many retransmissions.  There is a rate limiter (retransmission_rate_limiter) that is apparently configured to the whole channel bandwidth estimation.
    • If pacing is enabled insert this packet in the queue with normal priority.

Comments

  1. great post man!
    Have a question. I'm searching the google code for the audio jitter buffer and not the video jitter buffer .
    I want to make some tests and wrap this thing in my app.
    do you know where i can start from?

    ReplyDelete
    Replies
    1. It is here, but don't expect it to be easy to reuse in your app, it probably has many dependencies with rest of webrtc codebase: https://chromium.googlesource.com/external/webrtc/+/master/webrtc/modules/audio_coding/neteq/

      Delete
  2. Thank you for the post.

    A typical video stream has about 100 packets per second. Packet age of 10,000 is about 100 seconds old and 1,000 nack queue length is equal to at least 10 seconds. Is there a use case where 100 or 10 seconds old packets are still useful? Why don't they use 1 second limit as in the receiver side?

    Why do they keep a list of keyframes and not just the most recent one?

    I think they also check for the first packet in a keyframe and not if we got all the frame parts:
    bool is_keyframe =
    packet.is_first_packet_in_frame && packet.frameType == kVideoFrameKey;

    Seems like the webrtc source has moved:
    https://chromium.googlesource.com/external/webrtc/+/master/modules/video_coding/nack_module.cc

    ReplyDelete

  3. Hi, i think that i saw you visited my weblog thus i came to “return the favor”.I'm trying to find things to enhance my site!I suppose its ok to use some of your ideas!! facebook sign in

    ReplyDelete
  4. Thanks very much for this post! I'm currently implementing retransmissons for aiortc, my Python implementation of WebRTC and it answers a number of questions I had.

    ReplyDelete
  5. Hi,we are using OPUS audio codec in our application for compressing audio.But we can't properly handle our packet losses like webRTC. Is there any packet loss handling part in webRTC without OPUS?Any suggestion?

    [We also enabled Forward Error Correction]

    ReplyDelete
  6. Don’t follow your role model. Be the Role model person for others. But it's so simple by getting Hadoop training in Chennai. Because it is an assurance course to bounce back from a double salary. For joining call 7502633633.

    ReplyDelete

Post a Comment

Popular posts from this blog

Bandwidth Estimation in WebRTC (and the new Sender Side BWE)

Improving Real Time Communications with Machine Learning

Controlling bandwidth usage in WebRTC (and how googSuspendBelowMinBitrate works)