Audio Mixing or Forwarding

How many audio streams should your WebRTC server forward to the participants in a room? There are various options, ranging from the simplest approach of forwarding everything, to the most extreme option of mixing all audio and sending just a single stream.

A few weeks ago, we engaged in a Twitter conversation about this very topic. Following that discussion, bloggeek also wrote a post on the subject. For me it is always interesting to see what different types of applications are doing because at least in some of those cases they have the ability to do A/B testing and compare the results with millions of users before making a decision.

The simplest way to determine the best approach is to enter a room with different applications and inspect the SDP (Session Description Protocol) in chrome://webrtc-internals. Within this tool, you can examine how many channels are being forwarded when you're in a room and look for potential clues within the SDP (some people use the "mixed" identifier for audio tracks, among other things). For example in case of Google Meet you can see in the SDP that joining a room you get 3 hardcoded audio streams created:

m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126, mid=3
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126, mid=4
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126, mid=5

All the tests below were done with the web clients and Zoom was not included because it doens't make use of standard WebRTC and everything is binary&compiled so it would take more effort to reverse engineer it (for example analysing the audio bitrates when multiple participants speak).

What's the optimal approach for this task? There isn't a one-size-fits-all solution; it hinges on your particular use case and specific requirements. The most straightforward implementation is plain forwarding, which is what the majority of open-source projects and platforms employ. Interestingly, enterprise solutions tend to lean toward mixing. This inclination might be attributed to the fact that these services have evolved from earlier audio mixing solutions, the necessity for PSTN interconnection, or the need to accommodate unusual scenarios, such as devices with extremely limited available bandwidth.

Comments

Post a Comment

Popular posts from this blog

Bandwidth Estimation in WebRTC (and the new Sender Side BWE)

Image
Bandwidth estimation is probably the most critical component in the video engine of WebRTC. The bandwidth estimation (BWE) module is responsible for deciding how much video* traffic you can send without congesting the network to prevent degradation of the video quality. In the past bandwidth estimation algorithms used to be very rudimentary and based mostly on packet loss. Basically we used to start increasing slowly the video bitrate until we detected packets being lost.  To detect the packet loss you use the standard RTCP feedback mechanisms where the receiver side reports packet loss periodically using RTCP Receiver Report (RR) messages. Modern bandwidth estimation algorithms are more advanced and try to detect congestion before it is bad enough to make routers discard packets. These algorithms predicts congestion analyzing the delay between packets. The idea is that when you start having some congestion, the buffers in the routers will start filling and the delay will be
Read more

Improving Real Time Communications with Machine Learning

Image
When we talk about the applications of Artificial Intelligence / Machine Learning (AI/ML) for Real Time Communications (RTC) we can group them in two different planes: Service Level : There are many features that can be added to a videoconference service, for example identification of the participants, augmented reality, emotion detection, speech transcription or audio translation .  These features are usually based on image and speech recognition and language processing. Infrastructure Level : There are many ways to apply ML that do not provide new features but improve the quality and/or reliability of the audio/video transmission. Service level applications are fun, but they are more for Product Managers and I like technology more, so in the next sections I will try to describe possible applications of AI/ML for Real Time Communications at Infrastructure Level organizing those ideas in five different categories. Optimizing video quality Some of the ML algorithms used fo
Read more

Controlling bandwidth usage in WebRTC (and how googSuspendBelowMinBitrate works)

Image
There are cases when we would like to limit the maximum bitrate being transmitted by WebRTC to avoid wasting resources in the user endpoints or save money reducing the bandwidth usage in our servers.   This is because the maximum bitrate by default in Chrome is around 2Mbps and for many use cases a much lower bitrate provides still pretty good quality.   BTW using a lower bitrate can also help with stability of quality in multiparty scenarios by reducing the amount of competition among different streams. There is no simple API to configure the maximum bitrate in WebRTC (although there is one in ORTC) but there are 3 ways to do this by mangling the SDP. 1. U se the standard  b=AS:BITRATE (Chrome) or b=TIAS:BITRATE  (Firefox) attributes in the SDP for the audio or video channel[1] 2. Use codec specific attributes (this work at least for opus audio codec with maxaveragebitrate property) [2] 3. Use the proprietary x-max-bitrate attribute in the video channel of the SDP answer.  For e
Read more