Media Ingestion protocols Review

This post summaries the most relevant protocols using for media ingestion.    In these media ingestion flows a client that is broadcasting some content sends it to an endpoint (typically a server) that will re-format it for distribution to a potentially large audience, using a different protocol optimised for media playback.

The following sections show the main ideas behind each protocol as well as some of their advantages and disadvantages.    The protocols are organised into three buckets (tradicional protocols, modern protocols and next generation protocols) that correspond to three waves of protocols with the first one based on TCP, the second one on UDP or HTTP and the third one with QUIC.  You can see a summary of those characteristics as well as the timeline and adoption in the following diagram.

Traditional protocols

RTMP

RTMP (Real-Time Messaging Protocol) is the most widely used protocol for media ingestion nowadays.  It was developed more than 20y ago and popularised by Adobe/Flash initially as a playback protocol.   It is supported by the biggest Internet video platforms like Youtube, Facebook or Twitch.

The protocol transports media over TCP, includes an initial handshake and support for arbitrary content (metadata, RPC and media) using a proprietary binary format.   The feedback mechanisms to control the media transmission is limited as well as the codecs supported.   There are some variations of RTMP developed for proxy traversal or P2P but they are not that popular.

It is a protocol used for ingestion and not playback so in most of the cases it requieres other protocol and some repacketization for distribution (typically using HLS protocol).

RTSP

RTSP (Real-Time Streaming Protocol) is another protocol developed in the 90s but in this case it was fully standard from the beginning and reuses other internet standard protocols like RTP (Real-Time Protocol), RTCP (Real-Time Control Protocol) and SDP (Session Description Protocol).   The main motivation was its standardisation and the lower latency compared with RTMP.   It is not used that much today and mostly limited to specific use cases like IP cameras.

It is a protocol that separates the control and the media plane.  The establishment part of RTSP is similar to HTTP and uses SDP to describe the media content and the media part uses RTP over UDP.  It could support most of the feedback and capabilities of RTP extensions but usually the implementations are limited to the basic RTP profile.

Modern protocols

SRT

SRT (Secure Reliable Protocol) is a protocol invented in 2012 and released in 2017 as a replacement for the tradicional protocols to be better suited for low latency transmission over noisy networks.  It is used by many professional broadcasters and some internet services like Caffeine.tv.

(source)

The protocol makes use of UDP for the transmission of control and data packets and includes feedback for retransmissions under packet loss.  The protocol syntax is based on a previously existing protocol for file transfer called ULP and has different modes to enable use cases from reliable file transfer to low latency streaming.

RIST

RIST (Reliable Internet Stream Transport) is a protocol similar to SRT in terms of features and motivations.    It was developed to increase the reliability against packet losses and the security included in SRT as well as to support higher bitrate streams.

This protocol is also transported over UDP although in this case it uses RTP instead of inventing a new protocol for that. It includes feedback for retransmissions under packet loss.

It is a slightly better protocol that reuses some existing protocols but so far it had lower adoption than SRT.

WebRTC

WebRTC is a project that was designed for videoconferencing use cases but that has seen also some adoption for streaming applications.   Its real time nature makes it suitable for use cases where ultra low  latency is more important than getting the highest possible quality and also the native support in browsers makes it very simple to integrate for applications.    Different streaming services use it in different degrees like for example Youtube Studio or StreamYard.

The protocol doesn't define the signalling/control plane and that's left to each application although recently a new specification (WHIP) has defined a very simple HTTP signalling protocol for basic ingestion use cases.    The protocol does define the media plane that is composed of different layers for connectivity (ICE protocol), encryption (DTLS&SRTP protocols) and media itself (RTP&RTP procols).   That makes it a complex protocol even if it is mostly a composition of protocols existing for many years.    It includes feedback mechanisms for retransmissions and bandwidth estimation as well as many other extensions for FEC or buffers control.

The WebRTC protocol stack can be used also for playback if needed although in many cases the applications still rely on traditional protocols for playback to prioritise reachability and quality over the lowest possible latency.

FLT

FLT (Faster than Light Protocol) is a protocol designed to provide a very low latency replacement for RTMP.   The protocol was designed for Microsoft Mixer service and many applications and services implemented it although it is mostly unused with the shutdown of Mixer.

The protocol was making use of UDP and RTP to provide ultra low latency in a way similar to WebRTC but without the ICE and DTLS parts and it includes a control protocol over TCP to establish and negotiate the media connection.

HTTP ingestion (HLS, DASH...)

HTTP ingestion consist on reusing the protocols that are mostly used for media delivery (HLS, DASH...) but can also be used for media ingestion.   The most popular of these protocols is probably HLS (HTTP Live Streaming) and it is supported by Akamai or Youtube Live in addition to RTMP for ingestion.

These protocols use HTTP as transport to send the media segments encoding pieces of audio+video of a small number of seconds duration and also to send the playlists with the metadata and the references to those segments.  They make use of the delivery and encryption mechanisms of HTTP and doesn't include any specific feedback mechanism to adapt the ingestion to the network conditions beyond controlling latencies and buffers.

Next generation protocols

RUSH

RUSH (Reliable (unreliable) streaming protocol) is a bidirectional media protocol designed by Facebook as a replacement for RTMP with the goal to provide support for new audio and video codecs, extensibility ini the form of new message types, and multi-track support.   This protocol is only used by Facebook at this point both for ingestion and for media distribution inside their network.

The RUSH protocol is based on two core ideas:

  • Use QUIC as the transport protocol.  The QUIC protocol, built on UDP, serves as the foundation for HTTP/3 and offers robust security measures in addition to affording granular control over packet priorities and reliability.
  • Use a different QUIC stream for each video frame so that they can be prioritized independently in case of congestion based on the latency and the importance of each frame. 

The specification includes a control protocol and the media protocol both running on top of a single QUIC connection.   The messages have a binary encoding and include fields similar to the ones in RTP.

WARP

WARP (Live media transport over QUIC) protocol is also a bidirectional protocol but in this case designed by Twitch as a replacement for RTMP for the ingestion but also for existing playback protocols like HLS.    Twitch is already using it for the distribution of media chunks in their backbone network.

The ideas behind this protocol are similar to the ones in RUSH with a configurable latency vs quality trade-off.  The main difference is that the unit of transmission is not a frame but a segment.  A segment is a group of encoded frames packages in a MP4 container.   Another difference is that WARP is defined on top of WebTransport instead of just QUIC (WebTransport is a thin layer over QUIC intended for web browsers support)

The specification includes a control protocol and the media protocol both running on top of a single QUIC connection.   All the messages (control and media) includes a type-length encoded binary header that should allow any intermediaries in the distribution pipeline to prioritise and filter messages based on their content. 

The protocol can also be used for playback and the use of segments makes it very simple to do the conversion to other segment-based protocol like HLS if needed.

Conclussions

After many years where most of the ingestion has been happening with RTMP there is finally some alternatives getting enough traction.     WebRTC and SRT where the first ones and have already been adopted for many use cases while the new protocols designed over QUIC (RUSH, WARP or QUICR) are likely to be widely deployed in the next years.

The use of one protocol over the others depends mostly on two factors:

  • Support and interoperability.   The most widely supported protocol is still RTMP and some streaming equipment only support that protocol or the newer SRT/RIST protocols.     However, if we ignore the new QUIC protocols, most of these protocols are already supported in the widely used opensource OBS streaming client.
  • Latency vs quality trade-off.   Depending on the use case it is possible to select a protocol that is better suited for ultra low latency (like WebRTC) or when that is not strictly required other protocol can be selected that can provide the highest possible quality (like HLS).

As a final note this is the result of the wowza report on protocols being used by for ingestion:




Comments

Post a Comment

Popular posts from this blog

Bandwidth Estimation in WebRTC (and the new Sender Side BWE)

Image
Bandwidth estimation is probably the most critical component in the video engine of WebRTC. The bandwidth estimation (BWE) module is responsible for deciding how much video* traffic you can send without congesting the network to prevent degradation of the video quality. In the past bandwidth estimation algorithms used to be very rudimentary and based mostly on packet loss. Basically we used to start increasing slowly the video bitrate until we detected packets being lost.  To detect the packet loss you use the standard RTCP feedback mechanisms where the receiver side reports packet loss periodically using RTCP Receiver Report (RR) messages. Modern bandwidth estimation algorithms are more advanced and try to detect congestion before it is bad enough to make routers discard packets. These algorithms predicts congestion analyzing the delay between packets. The idea is that when you start having some congestion, the buffers in the routers will start filling and the delay will be
Read more

Improving Real Time Communications with Machine Learning

Image
When we talk about the applications of Artificial Intelligence / Machine Learning (AI/ML) for Real Time Communications (RTC) we can group them in two different planes: Service Level : There are many features that can be added to a videoconference service, for example identification of the participants, augmented reality, emotion detection, speech transcription or audio translation .  These features are usually based on image and speech recognition and language processing. Infrastructure Level : There are many ways to apply ML that do not provide new features but improve the quality and/or reliability of the audio/video transmission. Service level applications are fun, but they are more for Product Managers and I like technology more, so in the next sections I will try to describe possible applications of AI/ML for Real Time Communications at Infrastructure Level organizing those ideas in five different categories. Optimizing video quality Some of the ML algorithms used fo
Read more

Controlling bandwidth usage in WebRTC (and how googSuspendBelowMinBitrate works)

Image
There are cases when we would like to limit the maximum bitrate being transmitted by WebRTC to avoid wasting resources in the user endpoints or save money reducing the bandwidth usage in our servers.   This is because the maximum bitrate by default in Chrome is around 2Mbps and for many use cases a much lower bitrate provides still pretty good quality.   BTW using a lower bitrate can also help with stability of quality in multiparty scenarios by reducing the amount of competition among different streams. There is no simple API to configure the maximum bitrate in WebRTC (although there is one in ORTC) but there are 3 ways to do this by mangling the SDP. 1. U se the standard  b=AS:BITRATE (Chrome) or b=TIAS:BITRATE  (Firefox) attributes in the SDP for the audio or video channel[1] 2. Use codec specific attributes (this work at least for opus audio codec with maxaveragebitrate property) [2] 3. Use the proprietary x-max-bitrate attribute in the video channel of the SDP answer.  For e
Read more