WebRTC Architecture that keeps working when us-east-1 goes down

This week AWS had an outage in the northern Virginia region, probably the most commonly used region and many of the most popular services in the world were affected.

It was at least the third time in five years that AWS’s northern Virginia cluster, known as US-EAST-1, contributed to a major internet meltdown.

Every service is different, and nowadays, with so many dependencies and coupling, it is hard not to be affected by an incident like this. However, specifically for WebRTC platforms, it should be relatively straightforward to handle cases where one specific AWS region goes down.

Basic WebRTC Architecture

The basic architecture of most of WebRTC platforms comprises four components:

  • Load balancing service: The client application’s initial point of contact, selecting the appropriate media server for a session.
  • Database: Stores server information (status, load, location) and session-to-server mappings, often using a fast database like Redis.
  • Signaling service: Manages session state and forwards control messages. For this discussion, it’s assumed to be part of the media server and ignored in the following diagrams.
  • Media forwarding service: The core component, routing audio and video among participants. It’s typically stateless and reports its status.
Press enter or click to view image in full size

The whole solution usually includes other components to manage things like recording or firewall traversal but those are ignored for the purpose of this discussion and anyway usually they are stateless and easy to make them robust in case of a network issue.

Distributed WebRTC Architecture

When you want to provide the best possible quality, you deploy media servers in multiple regions, and your architecture typically looks like this:

Press enter or click to view image in full size

This is the most common deployment in my experience, but it is obvious that there is a central point of failure that can be the database or the Load Balancing service + Database depending on the way you deploy it.

Robust distributed architecture

If the Load Balancing service and the Database are the problem in case of outage in the region where they are located the solution looks obvious, let’s just duplicate them.

Replicating the Load Balancing service is easy, deploy it to as many regions as needed (ideally all of them) and use one of these approaches to failover to a different Load Balancer in case there is an outage in your closest region:

  • Rely on health checks and short DNS timeouts. That way if there is an outage the DNS servers will mark some region as unavailable and route client requests to a different one.. This can be in problem in case of an outage affecting the DNS service (like Route53).
  • Have clients configured to fallback to different url (for example api.domain.com and api-backup.domain.com)

Replicating the database is a little bit harder, the most common approaches are these ones:

  • The easiest way is to do it in an active/passive way, where multiple instances of the database are available, but only one is used at a given time, with the others just replicating the information. In case of an outage, we move the queries manually or automatically with some monitoring process to use one of those replicas instead.
  • A more evolved approach is active/active using sharding. It is more complex but possible and gives the additional advantage of distributing the load.
  • You can use an already distributed database like Azure Spanner. This can be tricky in case this distributed database has some central component, as apparently happened with DynamoDB in the case of the last AWS outage.

This is how the final architecture looks like:

Press enter or click to view image in full size

With the solutions proposed above your WebRTC infrastructure should be resilient to all the problems affecting a single region in your cloud provider.

It is possible to go further and for example deploy on multiple providers (AWS, GCP, Azure…) or using your own infrastructure but that’s not something I would recommend unless it is a really critical piece of software or you need it for other reasons like cost or compliance.

Comments

Post a Comment

Popular posts from this blog

Bandwidth Estimation in WebRTC (and the new Sender Side BWE)

Image
Bandwidth estimation is probably the most critical component in the video engine of WebRTC. The bandwidth estimation (BWE) module is responsible for deciding how much video* traffic you can send without congesting the network to prevent degradation of the video quality. In the past bandwidth estimation algorithms used to be very rudimentary and based mostly on packet loss. Basically we used to start increasing slowly the video bitrate until we detected packets being lost.  To detect the packet loss you use the standard RTCP feedback mechanisms where the receiver side reports packet loss periodically using RTCP Receiver Report (RR) messages. Modern bandwidth estimation algorithms are more advanced and try to detect congestion before it is bad enough to make routers discard packets. These algorithms predicts congestion analyzing the delay between packets. The idea is that when you start having some congestion, the buffers in the routers will start filling and the delay wil...
Read more

Improving Real Time Communications with Machine Learning

Image
When we talk about the applications of Artificial Intelligence / Machine Learning (AI/ML) for Real Time Communications (RTC) we can group them in two different planes: Service Level : There are many features that can be added to a videoconference service, for example identification of the participants, augmented reality, emotion detection, speech transcription or audio translation .  These features are usually based on image and speech recognition and language processing. Infrastructure Level : There are many ways to apply ML that do not provide new features but improve the quality and/or reliability of the audio/video transmission. Service level applications are fun, but they are more for Product Managers and I like technology more, so in the next sections I will try to describe possible applications of AI/ML for Real Time Communications at Infrastructure Level organizing those ideas in five different categories. Optimizing video quality Some of the ML algorithms use...
Read more

OpenAI WebRTC API Review

Image
There is a new interface added to OpenAI RealTime models. Now it supports WebRTC! Given the people working on it I'm sure it has to be great so as usual let’s take a look and see what is under the hood in terms of audio transmission. Signalling or Establishment of the connection There are two options for the establishment of a RealTime session with the OpenAI servers: WebSocket signalling : much nicer API without ugly SDPs involved but less suited for public networks. HTTP/WebRTC signalling : has an uglier API including SDP offer/answer negotiations but can work well in real networks that is critical for most of the use cases. In the rest of the post we will focus only in the later (HTTP/WebRTC) that is the most interesting one. Authentication The first step to use these RealTime APIs sending audio data directly from clients to OpenAI servers is to obtain an ephemeral key using you OpenAI API Secret. This is a simple HTTP request that for testing you can do from the command line: `...
Read more