May 15, 2020
The Internet of Things is burgeoning — and so is the amount of camera-based connected devices that produce real-time media data and can potentially send it to a PC or smartphone.
But there’s a catch.
Most users do not need a standalone app to play audio or video clips: they have browsers for that. To integrate an IP camera with a browser, you could use Adobe Flash, which will be dropped by major browsers in late 2020. There are also HLS and MPEG Dash, which are not tailored for low-latency live streaming. Finally, there is WebRTC — a relatively new technology, which might not ensure maximum device coverage at the moment, but easily beats its counterparts in terms of performance.
This article will give a brief overview of WebRTC, enumerate its advantages, explain how to integrate the browser-first technology with connected devices, and identify its potential use cases in IoT.
Web Real-time Communication (WebRTC) is an open-source technology, which allows developers to integrate real-time video and audio communication capabilities into web and mobile applications, and, as of lately, connected devices.
Created by Global IP Solutions, the pioneer of videoconferencing and VoIP, WebRTC was acquired by Google in 2010. Simply put, WebRTC establishes peer-to-peer communication between web or mobile browsers without any additional plugins, gains access to a gadget’s camera and microphone, and is capable of streaming media files with only a half second lag. In fact, it is regarded as the primary real-time media file transfer solution.The technology’s obvious use cases thus include audio and videoconferencing applications, and team collaboration tools.
In order to begin a communication session between two users, browsers should locate each other and obtain permission to exchange media data in real time.
The challenge here is that the users’ laptops or smartphones are protected by network access translation (NAT) devices or firewalls. Unlike HTTPS websites, whose location is known to the entire Internet, laptops and smartphones do not have static web addresses.
To set up a phone or video call with a user outside a home network, WebRTC utilizes Session Traversal Utilities for NAT (STUN) and Traversal Using Relays around NAT (TURN) servers, along with signaling/communication protocols. The latter include:
Before WebRTC, real-time media communications relied on C/C++. As a result, the development of custom conferencing and collaboration tools meant longer project timeframes and higher costs. Although WebRTC still uses the technology stack under the hood, developers no longer have to dig into those layers, as it has a JavaScript API on top, which is enough for building applications that interact with browsers.
The Internet of Things concept revolves around real-time data exchange — and that’s what WebRTC is particularly good at.
For instance, the technology could be a welcome addition to an RFID-based retail store security system: besides the label tags and security antennae, the solution could incorporate an IP camera, which monitors store traffic and streams video data in real time to a web application running on a security manager’s computer. When an item that hasn't been paid for sets the detectors off, the manager can correlate the alert signal with camera footage, and identify the shoplifter.
Although you can compile the WebRTC native code package to create a peerconnection without an intermediary media server, WebRTC is a browser-first technology, meaning it is not directly supported by most IoT devices and embedded systems.
Further on the subject of smart cameras, you usually can't access the data produced by smart security systems outside a proprietary network, which prevents users from monitoring industrial facilities and connected homes remotely.
As a result, no business or consumer-facing IoT solution currently provides WebRTC out-of-the-box.
GStreamer, an open-source pipeline-based framework for creating multimedia streaming applications for desktop, connected devices, and servers, has added a native WebRTC API to its feature set. The implementation is flexible and much easier to work with than the official WebRTC Native APIs.
GStreamer is designed for impeccable performance. To communicate the concept to a non-technical audience, we can compare the framework to a plumbing system with water instead of media data and GStreamer pipelines serving as the actual pipes. These pipes are capable of changing the quality and quantity of the water on the way from the public water supply (device one) to a home plumbing system (device two).
Suppose the source device can read video files. We can build a pipe bend (GStreamer demuxer) to separate outgoing traffic into audio and video data streams. The data is decoded via h264 (video) and Opus (audio) down the pipeline and delivered to the target device — namely, its video and audio output components, or to the cloud, where it can be analyzed using Machine Learning algorithms.
In GStreamer, those functional pipe bends are called elements. They can be divided into source elements, which produce data, and sink elements, which accept it. The elements, in their turn, have pads — the interfaces with the outside world, which allow developers to connect elements based on their capabilities. Additionally, GStreamer features built-in synchronization mechanisms ensuring audio and video samples are played in the correct sequence and at the right time.
...and more. Below you will find a list of embedded software development projects which can potentially use GStreamer:
As more embedded software engineers discover GStreamer, we will soon see more WebRTC-based connected products, including industrial solutions for remote equipment maintenance, Smart Home products, telemedicine applications, intelligent motor cars collecting real-time telemetry data, and wearables.
What makes us think so?
Thanks to restful APIs, third-party developers can create WebRTC-based embedded communications with relative ease, and use sensor data produced by connected devices to trigger alert notifications, audio calls, and video sessions. WebRTC introduces a new security layer to IoT solutions, and can be used as a safe data transfer channel. And finally, there’s much more to the technology than audio and video streaming: a WebRTC-based Smart Home security system, for instance, can both produce a live video feed and tell which door is open using the data from a connected lock.
Dmitry Zhadinets is an embedded software development specialist at Softeq. With a PhD degree in Speech Analysis and Synthesis and over ten years of experience in embedded system development, Dmitry is involved in multiple enterprise-grade CCTV projects, and is particularly interested in media file transfer technologies. He has been in the IT industry since 2002.