Sending Packet Loss Feedback in WebRTC SFUs

One of the responsibilities of WebRTC SFUs is to receive and send RTCP packets.  RTCP packets include different types of feedback about audio and video streams and one of the most important RTCP packets is the Receiver Report (RR).

RR packets are sent from the receiver of the media stream towards the sender of that media stream.  In case of an SFU the RR are generated and sent from the SFU to the media stream Sender and also from every stream Receiver to the SFU (Figure 1).

The feedback sent inside RR packets include fields to calculate the round-trip-time delay, the jitter and the packet loss introduced by the network.


The packet loss reported in these RR packets is important because the audio and video being sent will be adjusted based on that parameter:
  • In case of audio streams the packet loss in the network modifies the level of robustness of the OPUS codec.   In presence of high packet loss the sender increases the level of redundancy of the forward error correction (FEC) included in the audio packets.
  • In case of video streams the packet loss modifies the video bitrate encoded and sent.   In presence of high packet loss the sender decreases the bitrate sent to reduce the possible congestion in the network.
Based on that behavior the question is... How should a SFU calculate the packet loss reported in the RR packets sent from the SFU to the Sender to get the best possible user experience?

In the next sections you can find a suggestion for how to handle this in three different types of streams: Audio, Video with simulcast and Video without simulcast.

Audio

Opus FEC is sent inband and because of that FEC is end to end (cannot be added/updated in an SFU) and everybody in the room will receive the same level of FEC.

If we want audio FEC to work well for the participant with the weakest link then we have to make sure we report the worse packet loss to the sender.

Because of that the packet loss reported from the SFU to the sender should be the worse packet of the receivers (max(PL1, PL2)).

For example the packet loss reported to the sender will be 20% if one of the receiving participants is experiencing a 20% packet loss in the downlink even if the sender and rest of receivers are in perfect network conditions.


Video with Simulcast

When using Simulcast the SFU has the ability to send a different video quality to each participant so there is no need to adapt the sender stream to any specific participant.

Because of that the packet loss reported from the SFU to the sender should be the packet loss in that link (Sender-SFU) no matter what the packet loss received the Receiver 1 and 2 is.

For example the packet loss reported to the sender will be 0 if the sender uplink is good even if the receivers are experiencing big packet loss in the downlink.   That will be corrected with retransmissions and bitrate adaptation done in the SFU by selection a lower resolution/layer to be forwarded to those participants.


Video with No-Simulcast  

When there is no Simulcast the SFU has to send the same video quality to every participant.  Because of that it is important that the sender adapts the sending bitrate to the participant with the weakest link (and/or disable the video for some participants).

This bitrate adaptation is mostly controlled by REMB RTCP packets where the SFU includes the bandwith available in the worse receiver.   But packet loss has also an impact on top of the bitrate reported in the REMB packets so we need to decide what packet loss to include in the RR packets.

In this case I think both approaches described in the previous sections can work reasonably well.   Either a) the SFU reports back the worse packet loss of the receivers or b) use the packet loss of each receiver to estimate its bandwidth and report only the sender side packet loss in the RR packets.

Note: If you use P2P connections when there are 2 participants in the room and switch to the SFU with Simulcast when there are 3 or more participants in the room you should be almost never in this Video with No-Simulcast and multiple participants case.


You can follow me in Twitter if you are interested in Real Time Communications.



Comments

  1. sebaDecember 12, 2018 at 4:50 AM

    The correct thing to do is highly dependant n the context. If your SFU has 20 clients with great connections and only 1 with a very bad connection. Do you really want to reduce image quality for everybody?

    ReplyDelete
  2. amsa lekaFebruary 9, 2019 at 3:18 AM


    Nice blog..! I really loved reading through this article. Thanks for sharing such a
    amazing post with us and keep blogging... iot training in chennai | iot training in chennai quora | iot training and placement in chennai | iot training center in chennai | best iot training centre in chennai

    ReplyDelete
  3. amsa lekaMarch 5, 2019 at 12:43 AM

    Thanks for your great and helpful presentation I like your good service. I always appreciate your post. That is very interesting I love reading and I am always searching for informative information like this.angular 4 training in chennai | angularjs training in chennai | .angularjs best training center in chennai | angularjs training in velachery |

    ReplyDelete

Post a Comment

Popular posts from this blog

Bandwidth Estimation in WebRTC (and the new Sender Side BWE)

Image
Bandwidth estimation is probably the most critical component in the video engine of WebRTC. The bandwidth estimation (BWE) module is responsible for deciding how much video* traffic you can send without congesting the network to prevent degradation of the video quality.

In the past bandwidth estimation algorithms used to be very rudimentary and based mostly on packet loss. Basically we used to start increasing slowly the video bitrate until we detected packets being lost.  To detect the packet loss you use the standard RTCP feedback mechanisms where the receiver side reports packet loss periodically using RTCP Receiver Report (RR) messages.

Modern bandwidth estimation algorithms are more advanced and try to detect congestion before it is bad enough to make routers discard packets. These algorithms predicts congestion analyzing the delay between packets. The idea is that when you start having some congestion, the buffers in the routers will start filling and the delay will be more va…
Read more

Controlling bandwidth usage in WebRTC (and how googSuspendBelowMinBitrate works)

Image
There are cases when we would like to limit the maximum bitrate being transmitted by WebRTC to avoid wasting resources in the user endpoints or save money reducing the bandwidth usage in our servers.   This is because the maximum bitrate by default in Chrome is around 2Mbps and for many use cases a much lower bitrate provides still pretty good quality.   BTW using a lower bitrate can also help with stability of quality in multiparty scenarios by reducing the amount of competition among different streams.

There is no simple API to configure the maximum bitrate in WebRTC (although there is one in ORTC) but there are 3 ways to do this by mangling the SDP.

1. Use the standard b=AS:BITRATE (Chrome) or b=TIAS:BITRATE (Firefox) attributes in the SDP for the audio or video channel[1]
2. Use codec specific attributes (this work at least for opus audio codec with maxaveragebitrate property) [2]
3. Use the proprietary x-max-bitrate attribute in the video channel of the SDP answer.  For example w…
Read more

Using DSCP for WebRTC packet marking and prioritization

Image
It is a common request from WebRTC developers and customers to know how they can differentiate WebRTC traffic from other type in their networks.  Usually the goal is to be able to prioritize RTC traffic over other types of less important traffic.

By prioritizing the WebRTC traffic in the edge routers it is possible to improve the quality in some network scenarios.  The most common cases where this marking may help are:

Congested broadband uplink where the router can discard other type of traffic instead of WebRTC traffic when queues get full.Congested local wireless network One obvious way to do this is forcing all the traffic to be relayed through a TURN or SFU server and se the priority based on IP addresses.   That's easy and always work, but the problem is that it is very difficult to maintain when your infrastructure changes often (scaling elastically, new datacenters...).

Other way to accomplish this is to use Differentiated Services Code Point (DSCP) marking. With DSCP you c…
Read more