Sending Packet Loss Feedback in WebRTC SFUs

One of the responsibilities of WebRTC SFUs is to receive and send RTCP packets.  RTCP packets include different types of feedback about audio and video streams and one of the most important RTCP packets is the Receiver Report (RR).

RR packets are sent from the receiver of the media stream towards the sender of that media stream.  In case of an SFU the RR are generated and sent from the SFU to the media stream Sender and also from every stream Receiver to the SFU (Figure 1).

The feedback sent inside RR packets include fields to calculate the round-trip-time delay, the jitter and the packet loss introduced by the network.


The packet loss reported in these RR packets is important because the audio and video being sent will be adjusted based on that parameter:
  • In case of audio streams the packet loss in the network modifies the level of robustness of the OPUS codec.   In presence of high packet loss the sender increases the level of redundancy of the forward error correction (FEC) included in the audio packets.
  • In case of video streams the packet loss modifies the video bitrate encoded and sent.   In presence of high packet loss the sender decreases the bitrate sent to reduce the possible congestion in the network.
Based on that behavior the question is... How should a SFU calculate the packet loss reported in the RR packets sent from the SFU to the Sender to get the best possible user experience?

In the next sections you can find a suggestion for how to handle this in three different types of streams: Audio, Video with simulcast and Video without simulcast.

Audio

Opus FEC is sent inband and because of that FEC is end to end (cannot be added/updated in an SFU) and everybody in the room will receive the same level of FEC.

If we want audio FEC to work well for the participant with the weakest link then we have to make sure we report the worse packet loss to the sender.

Because of that the packet loss reported from the SFU to the sender should be the worse packet of the receivers (max(PL1, PL2)).

For example the packet loss reported to the sender will be 20% if one of the receiving participants is experiencing a 20% packet loss in the downlink even if the sender and rest of receivers are in perfect network conditions.


Video with Simulcast

When using Simulcast the SFU has the ability to send a different video quality to each participant so there is no need to adapt the sender stream to any specific participant.

Because of that the packet loss reported from the SFU to the sender should be the packet loss in that link (Sender-SFU) no matter what the packet loss received the Receiver 1 and 2 is.

For example the packet loss reported to the sender will be 0 if the sender uplink is good even if the receivers are experiencing big packet loss in the downlink.   That will be corrected with retransmissions and bitrate adaptation done in the SFU by selection a lower resolution/layer to be forwarded to those participants.


Video with No-Simulcast  

When there is no Simulcast the SFU has to send the same video quality to every participant.  Because of that it is important that the sender adapts the sending bitrate to the participant with the weakest link (and/or disable the video for some participants).

This bitrate adaptation is mostly controlled by REMB RTCP packets where the SFU includes the bandwith available in the worse receiver.   But packet loss has also an impact on top of the bitrate reported in the REMB packets so we need to decide what packet loss to include in the RR packets.

In this case I think both approaches described in the previous sections can work reasonably well.   Either a) the SFU reports back the worse packet loss of the receivers or b) use the packet loss of each receiver to estimate its bandwidth and report only the sender side packet loss in the RR packets.

Note: If you use P2P connections when there are 2 participants in the room and switch to the SFU with Simulcast when there are 3 or more participants in the room you should be almost never in this Video with No-Simulcast and multiple participants case.


You can follow me in Twitter if you are interested in Real Time Communications.



Comments

  1. sebaDecember 12, 2018 at 4:50 AM

    The correct thing to do is highly dependant n the context. If your SFU has 20 clients with great connections and only 1 with a very bad connection. Do you really want to reduce image quality for everybody?

    ReplyDelete
    Replies
    1. mary BrownJuly 20, 2020 at 8:42 PM

      The development of artificial intelligence (AI) has propelled more programming architects, information scientists, and different experts to investigate the plausibility of a vocation in machine learning. Notwithstanding, a few newcomers will in general spotlight a lot on hypothesis and insufficient on commonsense application. machine learning projects for final year In case you will succeed, you have to begin building machine learning projects in the near future.

      Projects assist you with improving your applied ML skills rapidly while allowing you to investigate an intriguing point. Furthermore, you can include projects into your portfolio, making it simpler to get a vocation, discover cool profession openings, and Final Year Project Centers in Chennai even arrange a more significant compensation.


      Data analytics is the study of dissecting crude data so as to make decisions about that data. Data analytics advances and procedures are generally utilized in business ventures to empower associations to settle on progressively Python Training in Chennai educated business choices. In the present worldwide commercial center, it isn't sufficient to assemble data and do the math; you should realize how to apply that data to genuine situations such that will affect conduct. In the program you will initially gain proficiency with the specialized skills, including R and Python dialects most usually utilized in data analytics programming and usage; Python Training in Chennai at that point center around the commonsense application, in view of genuine business issues in a scope of industry segments, for example, wellbeing, promoting and account.

      Delete
  2. AnonymousFebruary 9, 2019 at 3:18 AM


    Nice blog..! I really loved reading through this article. Thanks for sharing such a
    amazing post with us and keep blogging... iot training in chennai | iot training in chennai quora | iot training and placement in chennai | iot training center in chennai | best iot training centre in chennai

    ReplyDelete
  3. AnonymousMarch 5, 2019 at 12:43 AM

    Thanks for your great and helpful presentation I like your good service. I always appreciate your post. That is very interesting I love reading and I am always searching for informative information like this.angular 4 training in chennai | angularjs training in chennai | .angularjs best training center in chennai | angularjs training in velachery |

    ReplyDelete
  4. jenifer ireneMay 6, 2019 at 1:43 AM

    Very good information provided, Thanks a lot for sharing such useful information.
    best aviation academy in Chennai
    Air Hostess Training Institute in Chennai
    Airline Courses in Chennai
    airport ground staff training in Chennai
    Aviation Academy in Chennai
    air hostess training in Chennai
    airport management courses in Chennai
    ground staff training in Chennai

    ReplyDelete
  5. DedicatedHosting4uJuly 1, 2019 at 7:06 AM

    This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to article very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.
    DedicatedHosting4u.com

    ReplyDelete
  6. MindtechAffiliatesJuly 12, 2019 at 11:58 PM

    Consistently learning the new skills and using them to improve myself in the form of performance for betterment.

    Thanks
    Cpa offers

    ReplyDelete
  7. digital marketing company IndoreJuly 15, 2019 at 5:45 AM



    This is a fantastic idea! I like it a lot because it's super easy for the audience to see the value of opting in. wonderful and amazing post very use full your post thanks for sharing your article
    Android Application development
    Web application

    ReplyDelete
  8. digital marketing company IndoreJuly 15, 2019 at 5:46 AM


    Your post is very good. I got to learn a lot from your post. Thank you for sharing your article for us. it is amazing post
    what is seo
    types of seo


    ReplyDelete

Post a Comment

Popular posts from this blog

Bandwidth Estimation in WebRTC (and the new Sender Side BWE)

Image
Bandwidth estimation is probably the most critical component in the video engine of WebRTC. The bandwidth estimation (BWE) module is responsible for deciding how much video* traffic you can send without congesting the network to prevent degradation of the video quality. In the past bandwidth estimation algorithms used to be very rudimentary and based mostly on packet loss. Basically we used to start increasing slowly the video bitrate until we detected packets being lost.  To detect the packet loss you use the standard RTCP feedback mechanisms where the receiver side reports packet loss periodically using RTCP Receiver Report (RR) messages. Modern bandwidth estimation algorithms are more advanced and try to detect congestion before it is bad enough to make routers discard packets. These algorithms predicts congestion analyzing the delay between packets. The idea is that when you start having some congestion, the buffers in the routers will start filling and the delay will be
Read more

Improving Real Time Communications with Machine Learning

Image
When we talk about the applications of Artificial Intelligence / Machine Learning (AI/ML) for Real Time Communications (RTC) we can group them in two different planes: Service Level : There are many features that can be added to a videoconference service, for example identification of the participants, augmented reality, emotion detection, speech transcription or audio translation .  These features are usually based on image and speech recognition and language processing. Infrastructure Level : There are many ways to apply ML that do not provide new features but improve the quality and/or reliability of the audio/video transmission. Service level applications are fun, but they are more for Product Managers and I like technology more, so in the next sections I will try to describe possible applications of AI/ML for Real Time Communications at Infrastructure Level organizing those ideas in five different categories. Optimizing video quality Some of the ML algorithms used fo
Read more

Controlling bandwidth usage in WebRTC (and how googSuspendBelowMinBitrate works)

Image
There are cases when we would like to limit the maximum bitrate being transmitted by WebRTC to avoid wasting resources in the user endpoints or save money reducing the bandwidth usage in our servers.   This is because the maximum bitrate by default in Chrome is around 2Mbps and for many use cases a much lower bitrate provides still pretty good quality.   BTW using a lower bitrate can also help with stability of quality in multiparty scenarios by reducing the amount of competition among different streams. There is no simple API to configure the maximum bitrate in WebRTC (although there is one in ORTC) but there are 3 ways to do this by mangling the SDP. 1. U se the standard  b=AS:BITRATE (Chrome) or b=TIAS:BITRATE  (Firefox) attributes in the SDP for the audio or video channel[1] 2. Use codec specific attributes (this work at least for opus audio codec with maxaveragebitrate property) [2] 3. Use the proprietary x-max-bitrate attribute in the video channel of the SDP answer.  For e
Read more