Improving Real Time Communications with Machine Learning


When we talk about the applications of Artificial Intelligence / Machine Learning (AI/ML) for Real Time Communications (RTC) we can group them in two different planes:
  • Service Level: There are many features that can be added to a videoconference service, for example identification of the participants, augmented reality, emotion detection, speech transcription or audio translation.  These features are usually based on image and speech recognition and language processing.
  • Infrastructure Level: There are many ways to apply ML that do not provide new features but improve the quality and/or reliability of the audio/video transmission.


Service level applications are fun, but they are more for Product Managers and I like technology more, so in the next sections I will try to describe possible applications of AI/ML for Real Time Communications at Infrastructure Level organizing those ideas in five different categories.

Optimizing video quality

Some of the ML algorithms used for image recognition can be used to optimize the video transmission in RTC services.  I can imagine at least three ways in which these algorithms can help to improve the video quality.

The first way would be to select the best possible encoding parameters for a specific video or a specific part of the frame.  For example if we can detect the most important parts of the scene (maybe the talking head) and use better encoding quality (lower quantization level) for those areas.  Or another example, we can detect the type of information and give preference to framerate vs quality depending if it is a high motion video or a typical conversation.

The second way could be to reduce the amount of information being sent by removing the information that can be regenerated by the receiver.  As an extreme example, we all know the shape of human hair so even if you can, can you send lower quality and reconstruct the hair in the receiver?  One example of this application can be seen in the RAISR demos by Google.


The same process could be also used to improve the readability or increase the detail level of objects that are too far or out of focus.

It is also possible to apply ML in the video codec implementations to optimize the processing required to encode the frames as you can see in this code included in the VP9 codebase.

Optimizing audio quality

We should be able to eliminate redundant audio data that can be regenerated in the receiver side in the same way we describe for video in the previous section.  In an extreme case we would only need to send the text and the accent / speed... of the speaker and the receiver should be able to reconstruct almost the same voice based on a previous learning process.

One of the problems with audio quality is the intelligibility in noisy environments.   ML algorithms can also help with this as shown by RRNoise project by Mozilla/Xiph by learning how to better differentiate and suppress noise vs voice.
Banner

Tuning transmission settings

The amount of parameters involved in a RTC session is really big, it goes from codecs with tons of settings, bitrates, packetization sizes, buffers, timeouts....  Deciding which ones to use at a given time is not trivial and can even require per user/network adjusts that change dynamically.  A ML based system could learn what is the best combination of those parameters for a specific user in some specific conditions.
In case of multiparty calls it is critical to include some algorithms to decide how the available bitrate is distributed for the different streams in the room.  For example is it better to send 2 videos at 50kbps or disable one and send the other one at 100kbps.  A ML algorithm able to make those decisions (bitrates, framerates, resolutions, codecs...) in real time based on the uplink/downlink characteristics of all the participants in the call, but also based on the type of conversation and on who is/are the active speaker/s could provide a much better quality of experience.

Resource allocation and planning

Most of the RTC infrastructures include the concept of Media Servers.  Those are the servers routing the audio and video packets between the different participants and are specially important in case of multiparty calls.

For these calls you can use ML algorithms to decide what is the best server to be used for a specific call based on the location of the server, the location of the participant/s and the status of the servers (basically the load and network status).

In the same way it can use for forecasting to predict load and make sure the amount of available resources is the optimum one.

Diagnostics and Monitoring

Most of the people who have been working building RTC platforms for a while has probably experienced how painful it is to debug issues.  Same way that ML is started to be used for medical diagnostics in e-health it can be also used to debug and find the root cause when there are problems.  For example based on quality metrics from different participants and the status of the servers can diagnose if it is a bug or a network issue, and in the second case which network was the responsible.

We can also use the classification algorithms available in ML to classify calls according to quality scores or other parameters to generate reports or monitoring purposes.

We can also use unsupervised learning to detect anomalies in the system automatically and trigger alerts.


As you can see there are many ways to use ML to improve the quality and/or reliability of our RTC platforms.  And these were only some examples and I'm sure you have many other ideas.


You can follow me in Twitter if you are interested in Real Time Communications.

Comments

  1. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.

    digital training in chennai

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. This post contains all about application of Artificial Intelligence In Speech Recognition. Thanks for the article.

    ReplyDelete
  4. I can't match inside your head however if you happen to noticed me, that is the place I would be because of the photons reflecting off of me and into and thru your eye and on to your retina therefore transformed to electrical impulses which transmit as electrical indicators into your mind which reconstructs identical again right into a digital actuality model of me that now can match inside your cranium.This is great blog. If you want to know more about this visit here Machine Learning Model.

    ReplyDelete
  5. Thanks For Sharing Excellent Blog. Machine Learning is steadily moving away from abstractions and engaging more in business problem solving with support from AI and Deep Learning. With Big Data making its way back to mainstream business activities, now smart (ML) algorithms can simply use massive loads of both static and dynamic data to continuously learn and improve for enhanced performance. Pridesys IT Ltd

    ReplyDelete
  6. Keep blogging.!!

    quite informative, thanks for sharing about new things here,

    - Learn Digital Academy

    ReplyDelete
  7. Hmm, it seems like your site ate my first comment (it was extremely long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog.
    nebosh igc course in chennai

    ReplyDelete
  8. This comment has been removed by the author.

    ReplyDelete
  9. Awesome post. Thanks for sharing this post with us.to form a pc checkers application that was one amongst the primary programs that would learn from its own mistakes and improve its performance over time.Machine learning course one step any - it changes its program's behavior supported what it learns.

    ReplyDelete
  10. thank you for giving such an great information.






    https://excelr.com.my/course/certification-on-industrial-revolution-4-0/

    ReplyDelete
  11. I’ve I’m a I’m planning to start my blog soon, but little lost on everything. Would you suggest starting with a free platform like Word Press or go for a paid option?
    nebosh course in chennai
    offshore safety course in chennai

    ReplyDelete
  12. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.
    nebosh course in chennai
    offshore safety course in chennai

    ReplyDelete
  13. your article on data science is very good keep it up thank you for sharing.
    Data Science coaching in Hyderabad

    ReplyDelete
  14. Great Job keep publishing such good articles, I would like to read more such articles going ahead.

    data science training in aurangabad
    data science course in aurangabad

    ReplyDelete
  15. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging…



    Dot Net Training in Chennai | Dot Net Training in anna nagar | Dot Net Training in omr | Dot Net Training in porur | Dot Net Training in tambaram | Dot Net Training in velachery


    ReplyDelete
  16. Glad to chat your blog,I seem to be forward to more reliable articles and i think we all wish to thank so many good articles,blog to share with us.
    learn data scientist course
    360digitmg best data science courses

    ReplyDelete
  17. This was not just great in fact this was really perfect your talent in writing was great.
    learn360digitmg data science training

    ReplyDelete
  18. This comment has been removed by the author.

    ReplyDelete
  19. it's really cool blog. Linking is very useful thing.you have really helped
    360digitmg ai online course

    ReplyDelete
  20. it's really cool blog. Linking is very useful thing.you have really helped
    360digitmg ai online course

    ReplyDelete
  21. Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.

    Data Science Online Training

    Data Science Classes Online

    Data Science Training Online

    Online Data Science Course

    Data Science Course Online

    ReplyDelete
  22. This comment has been removed by the author.

    ReplyDelete
  23. Industry reports are available through Federation of Indian Chambers of Commerce and Industry and Confederation of Indian Industry. artificial intelligence course in hyderabad

    ReplyDelete
  24. This comment has been removed by the author.

    ReplyDelete
  25. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.
    machine learning course training in indore

    ReplyDelete
  26. Hi it's really nice blog with new information,
    Thanks to share with us and keep more updates,

    https://www.porurtraining.in/sap-training-course-in-chennai
    https://www.porurtraining.in/microsoft-azure-training-in-porur-chennai
    https://www.porurtraining.in/cyber-security-training-in-porur-chennai
    https://www.porurtraining.in/ethical-hacking-training-in-porur-chennai

    ReplyDelete
  27. This comment has been removed by the author.

    ReplyDelete
  28. Great post! I am actually getting ready to across this information, It’s very helpful for this blog. Also great with all of the valuable information you have Keep up the good work you are doing well.
    CRS Info Solutions Salesforce training for beginners         

    ReplyDelete
  29. After completing all this tasks I got the skills to start looking to find a job as digital marketer! digital marketing training in hyderabad

    ReplyDelete
  30. Very good points you wrote here..Great stuff...I think you've made some truly interesting points.Keep up the good work.
    about us

    ReplyDelete
  31. Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative Rajasthan Budget Tours

    ReplyDelete
  32. Very nice blogs!!! i have to learning for lot of information for this sites…Sharing for wonderful information.Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing, data science online training

    ReplyDelete
  33. Primarily, data science refers to a field of study that uses the scientific approach to get an insight into the given data. The rapid growth in this field of science has resulted in the development of universities that have introduced different graduate programs related to data science. In this article, we are going to know more about both the fields. best course to learn artificial intelligence

    ReplyDelete
  34. Really impressed! Everything is very open and very clear clarification of issues. It contains truly facts. Your website is very valuable. Thanks for sharing.
    data science course in Hyderabad

    ReplyDelete
  35. A person trained in Data Science has multiple options to choose from. He/she can work in a number of fields like Data Analytics, Software programmer, Machine learning engineer etc. And these options are going to grow by multiple folds in the coming future. data science course syllabus

    ReplyDelete
  36. It will also check for any recently introduced new services or discounts that are finding enhanced interest among the customers. Salesforce training in Hyderabad

    ReplyDelete
  37. Thanks For Sharing Excellent Blog. Machine Learning is steadily moving away from abstractions and engaging more in business problem solving with support from AI and Deep Learning.Digital Marketing Training in Chennai

    Digital Marketing Training in Velachery

    Digital Marketing Training in Tambaram

    Digital Marketing Training in Porur

    Digital Marketing Training in Omr
    Digital Marketing Training in Annanagar

    ReplyDelete
  38. Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!

    data science course in India

    ReplyDelete
  39. I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site.
    data science training in Hyderabad

    ReplyDelete
  40. This comment has been removed by the author.

    ReplyDelete
  41. Wow! Such an amazing and helpful post this is. I really really love it. It's so good and so awesome. I am just amazed. I hope that you continue to do your work like this in the future also.
    Artificial Intelligence Course

    ReplyDelete
  42. I am really happy to say it’s an interesting post to read . I learn new information from your article , you are doing a great job . Keep it up

    Devops Training in Hyderabad

    Hadoop Training in Hyderabad

    Python Training in Hyderabad

    ReplyDelete
  43. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    top workday online training

    ReplyDelete
  44. Thanks for sharing
    https://nareshit.com/salesforce-online-training/

    ReplyDelete
  45. Tech giants like Google along with Instagram have also made use of Python and its popularity continues to rise. Discussed below are some of the advantages offered by Python: data science course in india

    ReplyDelete
  46. This comment has been removed by the author.

    ReplyDelete
  47. Thanks for writhing this Real Time Communications improving with Machine Learning article. It is really true that this digitalize system of machine learning may help to run fast and develop the world. To know more about this system you can visit http://thetechnews.com/2020/09/10/ai-machine-learning-deep-learning-explained-in-5-minutes/

    ReplyDelete
  48. Wow, wonderful content,
    https://www.techdatasolution.co.in/data-science-course-in-pune

    ReplyDelete

  49. Thanks for sharing your views about the concept which you know much better. Its easy to read and understand by the way you wrote the blog contents.
    machine learning course in chennai

    ReplyDelete
  50. We are well established IT and outsourcing firm working in the market since 2013. We are providing training to the people ,
    like- Web Design , Graphics Design , SEO, CPA Marketing & YouTube Marketing.Call us Now whatsapp: +(88) 01537587949
    Digital marketing training
    good post Mobile XPRESS
    Free bangla sex video:Delivery Companies in UK
    good post Mobile XPRESS

    ReplyDelete
  51. You have got done a good job along with your information that creates our work simple since you're giving such great data. Keep sharing this kind of information with us. web development service provider

    ReplyDelete
  52. I had a nice experience on your website, thank you for sharing your knowledge and experience with us,

    best regards

    youtube
    youtube

    ReplyDelete
  53. The TABE is for the most part not considered difficult, particularly on the off chance that a satisfactory sum of examining is included. What is the reason for the TABE test? The reason for the TABE is to decide what your abilities are in perusing, math, and English. Tabe applied math practice

    ReplyDelete
  54. https://designingcourses.in/
    Very Informative and useful... Keep it up the great work. I really appreciate your post.

    graphic designing courses in Bangalore
    web designing course in Bangalore
    UX Design course in Bangalore

    ReplyDelete
  55. Of late, online IT training business is catching up world wide and Hyderabad is no exception. There are some major players in Hyderabad who have started their online training facilities couple of years ago. Salesforce Training in Hyderabad

    ReplyDelete
  56. Thank you for a great and inspiring blog post. I just want you to know that I enjoyed reading your post. Duluth mn seo

    ReplyDelete
  57. I had the option to discover great data from your blog articles. best interiors

    ReplyDelete
  58. It's instructive and you are clearly entirely educated here. You have made me fully aware of differing sees on this point with fascinating and strong substance.
    data scientist training in hyderabad

    ReplyDelete
  59. provides the quality service of customized fiber connections in the case of large businesses and government entities. Black Parade Jacket

    ReplyDelete
  60. While THC stays in your urine for between 2 and 10 days, it can stay for up to three months, even if you only smoke occasionally. This is because the drug metabolites in your blood spread across your body and stay there – including your hair. Most drug detox programs focus on your digestive system to eliminate THC, alcohol, and other toxins before they get absorbed by fat and stay in your body. However, if you smoke pot up to a week before your detox, there’s a good chance that the metabolites have already gotten to your scalp and are going to give the game away when you get tested. Even if it’s been a while since you last used weed, the lab will almost definitely detect traces of it in your system. Ultra Clean is a hair cleaner that may help with drug detoxification, including cannabis sativa Visit: https://www.urineworld.com/

    ReplyDelete
  61. After reading your article I was struck by how knowledgeable you are about the topic at hand. You explained things in such a clear and concise way that it would be no surprise to see someone else feel the same way after reading your article.
    Artificial Intelligence Training in Hyderabad
    Artificial Intelligence Course in Hyderabad

    ReplyDelete
  62. I tracked down your blog while looking for the updates, I am glad to be here. Exceptionally helpful substance and furthermore effectively reasonable giving…

    Data Science Training in Hyderabad

    ReplyDelete
  63. If you are looking IVR Service provider, it is also important to choose the right IVR Service provider for long-term growth. IVR Guru has been the leading IVR Service provider all over India. IVR Guru has been providing quality IVR Services with the best pricing. Check out the prices or call us at +91-9015350505

    ReplyDelete
  64. Incredible blog here! It's mind boggling posting with the checked and genuinely accommodating data. little joe cartwright jacket

    ReplyDelete
  65. Really an awesome blog and very useful information for many people. Keep sharing more blogs again soon. Thank you.
    AI Patasala Data Science Training in Hyderabad

    ReplyDelete
  66. En iyi oyun indirme sitesi olantorrent oyun sitemize herkesi bekleriz.

    ReplyDelete
  67. It was not first article by this author as I always found him as a talented author. Spiderman Homecoming Peter Parker High School Jacket

    ReplyDelete
  68. İnsan böyle şeyler görünce mutlu oluyor

    ReplyDelete
  69. You should take help from professionals who have immense experience on Microsoft Business Central. They will help you with Solutions easily. Read: business central license types

    ReplyDelete
  70. web designing is one of the best and top developing IT training course at this point. Learning web designing will help you to understand the concepts of HTML, CSS and JAVASCRIPT. After understanding all the concepts a web developer can be able to create his own website from scratch.

    A2N Academy, one of the top-tier IT Skill Development Institutes in Bangalore, offers the best Web Designing course. Our skilled trainers recruited by the academy offer quality-based Web Designing Courses. Steadily, we guide our students to get placed in prominent web design companies through our online web designing course.

    With a determined goal to nurture students for their skills and placement, we focus on developing web designing skills in students through our web designing course with a learning-by-doing approach and creating responsive websites from scratch for all platforms.


    web designing course

    ReplyDelete
  71. IVR GURU Provides Virtual Phone Number, IVR Services, IVRS, Cloud Telephony, Voice Calls, Voice Survey Customer Lead Management, Toll Free Number, etc. For more info call us 9015350505

    ReplyDelete
  72. This comment has been removed by the author.

    ReplyDelete
  73. "If you are eager to enter the IT sector, you should try our career accelerated program to know more visit our website
    https://www.premiumlearnings.com/
    To see our tutorials and success stores visit our Youtube page
    https://www.youtube.com/c/PremiumLearningssystem "

    ReplyDelete
  74. Expressing thanks to you for partaking such immensities of information within few momentous sentences in this content. I am really looking forward to read some more motivating articles.
    Data Science training in Mumbai
    Data Science course in Mumbai
    SAP training in Mumbai

    ReplyDelete
  75. 360DigiTMG is the top-rated institute for Data Science in Hyderabad and it has been awarded as the best training institute by Brand Icon. Click the link below to know about fee details.
    data science course in hyderabad

    ReplyDelete
  76. Wonderful to you that you have shared this information with us. Read more info about. Star Trek Picard Field Jacket

    ReplyDelete
  77. In fact, Artificial Intelligence & Machine Learning (AI / ML) plays big role in all sectors.

    Regards,
    BroadMind - IELTS coaching centre in Madurai

    ReplyDelete
  78. Integrated Risk Management is a comprehensive approach that organizations use to manage risks in a unified and coordinated manner across various departments, functions, and levels of the organization. IRM aims to break down silos and bring together different aspects of risk management, including financial, operational, compliance, strategic, and cyber risks, into a cohesive strategy. The goal is to create a holistic view of risks and opportunities, enabling more effective decision-making and resource allocation.

    ReplyDelete
  79. Enterprise Risk Management Software enables organizations to identify and assess risks across all areas of operation, including strategic, operational, financial, and compliance-related risks. This comprehensive view helps ensure that all potential risks are considered.

    ReplyDelete

Post a Comment

Popular posts from this blog

Bandwidth Estimation in WebRTC (and the new Sender Side BWE)

Controlling bandwidth usage in WebRTC (and how googSuspendBelowMinBitrate works)