HLS

A Raspberry Pi Streaming Camera using MPEG-DASH, HLS or RTSP

This is an improvement on my previous article, Raspberry Pi Hardware Accelerated RTSP Camera, now with the option of using more modern technology, MPEG-DASH and HLS!

First off, if you don’t care about the technicalities and just want a script to do everything for you, here you go! If you’re still interested in how it all works or want to tweak the settings, read on.

This article will walk you through how to either copy or convert video from your webcam or pi camera, set it up as a systemd service, and finally view it on a webpage or access it remotely.

MPEG-DASH vs HLS vs RSTP

So to clear this up first of all, these are “containers” that wrap around the actual video, which is a particular “codec” (such as h264). DASH and RTSP are fully codec agnostic, meaning they are capable of wrapping around any type of video codec. The big gotcha is what type of videos the viewer supports (and in RTSP’s case the middleman server as well.)

So if DASH and RTSP can handle everything, why even bother with HLS? Long story short, Apple, who developed HLS, is a bully, so they don’t support the open MPEG-DASH on their devices. Meaning if you are trying to share these video streams with the public or view on an Apple device, you will get the most compatibility with HLS.

So now the difference really comes down to how DASH/HLS are HTTP based protocols that can easily be supported in browser. This makes it super easy to set up an all-in-one device that can host it’s own webpage to view a video at.

Whereas RTSP requires additional software, such as VLC or a security system to view it. The real advantage with RTSP is the fact it really is nearly “real time” compared to DASH/HLS. Using my Raspberry Pis DASH/HLS seem to have a 10~20 second delay, compared to about 1 second for RTSP. As I already went over how to set that up, I won’t repeat it here and only go over DASH. But I personally use RSTP for my own home setup still.

Setting up required software

The two programs you will need are a file server (nginx, apache, python -m http.server, etc…) to host the DASH/HLS content and ffmpeg. And you don’t have to hand compile either!

Super short version:

sudo apt install nginx ffmpeg -y

To better understand why we need them and how to test them to make sure they are working properly, read on. Otherwise, skip ahead to “Gather Camera Details”.

File Server

Last time we use RTSP which required a special service of it’s own. Now we are using HLS and MPEG-DASH, which produce manifest and accompany stream files on the local system. For example, MPEG-DASH will create a manifest.mpd file that contains links to *.m4s files in the same directory which are the chunked up video files.

That means if we make those files accessible remotely, we can use standard HTTP to transport the video. Hence the need for a basic file server. I personally use nginx for the final setup, as it’s fast, easy to use, and has defaults we can use out of the box. So lets install it!

 sudo apt install nginx -y

Now all you need to do is open up a web browser on another computer on that network and connect to http://raspberrypi (If you changed hostname, or having trouble connecting, run hostname -I to see it’s IP address and use http://<ip_address> instead.) You should see a simple webpage that says “Welcome to nginx!”

FFmpeg

Since the last article came out, FFmpeg has finally started shipping with hardware acceleration built in! If you still want to compile in some custom libraries or try and optimize it for your needs, check out my Raspberry Pi FFmpeg compile guide. Otherwise, just download it from the distribution repositories.

sudo apt install ffmpeg -y

You can verify it’s part of the package by checking the encoders for h264_omx.

ffmpeg -hide_banner -encoders | grep omx

Should produce V….. h264_omx OpenMAX IL H.264 video encoder (codec h264) or similar. If for some reason it doesn’t have that or other libraries you are looking for, such as the popular fdk-aac, look into my article onto compiling FFmpeg yourself, or use the helper script with the option --compile-ffmpeg.

Gather camera details

If you have the helper script, simply run it with the option --camera-details and it will print out each device and their formats with their highest resolution for each.

python3 streaming_setup.py --camera-details
# /dev/video0: {'yuyv422': '1280x800', 'mjpeg': '1280x720'}

Under the hood, this is running the following command for every device found.

ffmpeg -hide_banner -f video4linux2 -list_formats all -i /dev/video0

To also see what frame rates are supported per resolution, you will have to run the v4l2-ctl command for that device.

v4l2-ctl -d /dev/video0 --list-formats-ext

Create the FFmpeg command

The FFmpeg command is particular about order when talking about input and output details. Ours will be broken down into the following blocks:

ffmpeg <incoming video details> -i <device> <conversion details> <output>

So let’s say you are using a raspberry pi camera and want to stream 1080p video without re-encoding it. We first have to tell FFmpeg about the camera details it will pull from.

Applying the Camera Details

In longhand, it would look like this:

-input_format h264 -f video4linux2 -video_size 1920x1080 -framerate 30 -i /dev/video0

Hopefully each of those parts are pretty self explanatory. We can also reduce -video_size, aka the incoming resolution to -s, and -framerate, aka the fps to -r.

Network Bandwidth Considerations

Internet streamers, beware you may not be able to upload directly from the camera’s full 1080p at 30fps. I did a quick test using vnstat over a wired connection with a Pi Zero, and found my 5MP OV5647 camera was using almost 20Mbit/s. Keep in mind the official Pi Camera with the sony sensor is 8MP so may be even higher than that.

The following tests were done at two minute averages while the stream was being watched. The averages were recorded, and generally the peaks were 2x the average.

ResolutionfpsAverage Mbit/s
1920×10803020.12
1920×1080155.12
1280×7206015.81
1280×720303.94
1280×720152.75
640×480905.66
640×480604.43
640×480300.93
640×480150.43
Using a Pi Zero with 5MP OV5647 camera over ethernet

Conversion options

Since this already is h264 we don’t need anything other than to say copy the incoming stream. So that option is -codec:v copy or shorthand -c:v copy. Which is saying set the codec of v for video tracks to copy aka don’t convert.

-c:v copy

If you instead had a webcam that only supported mjpeg input, or if you needed to add text overlay to the video, you would have to recompile FFmpeg. With the Raspberry Pi, you’ll want to use the built in hardware encoder, h264_omx. You would then also have to set the bitrate (-b:v) of the outgoing video. That is really camera / network dependent, but my rule of thumb is use video width x hight x 2. So 1920x1080x2 ==4,147,200, so I would set the bitrate to 4M (aka ~4000kb, or ~4000000 bytes).

-c:v h264_omx -b:v 4M

The Raspberry Pi OpenMAX (omx) hardware encoder has very limited options, and doesn’t support constant quality or rate factors like libx264 does. So the only way to adjust quality is with the bitrate. As for general quality, it sits between libx264s ultrafast and superfast presets, which is somewhat disappointing but not surprising for a real-time hardware encoder.

MPEG-DASH and HLS output

Personally I would never recommend HLS to a friend, as MPEG-DASH is all around a more open and powerful muxer. But I understand some legacy systems don’t have DASH support yet. Thankfully, FFmpeg’s dash module gives us HLS for free! (Note that some systems don’t even support that, and you may end up having to use only the hls muxer.)

DASH and HLS both crate playlist files locally, with chucked up video files beside them. This creates a few problems, first is the cleanup and management of those files. Thankfully, the DASH model has options to delete all those files on exit, as well as the ability to only keep so many video chunks on disk at a time.

The bigger problem is the constant writing to the disk. In this case an SD card that is a wear item and has higher error rates with the more writes it experiences. So to save the SD card, and ourselves future headaches, we are going to write these files to memory instead!

sudo mkdir -p /dev/shm/streaming/

Tada, we now have a folder in shared memory space we can use. The caveat is it will be removed after restarts, so we will have to make sure it’s recreated before or FFmpeg service is started. But lets not get ahead of ourselves. We just need to know the rest of our FFmpeg command.

-f dash -window_size 10 -remove_at_exit 1 -hls_playlist 1 /dev/shm/streaming/manifest.mpd

We are using just a few options of what FFmpeg’s DASH muxer can do if you do need further customization, but I doubt it for most cases.

I am setting the max number of video chuncks to be kept at 10 via -window_size and telling FFmpeg to delete them and the manifest file when it stops running with -remove_at_exit 1. Then we enable HLS with -hls_playlist 1 which creates a master.m3u8 file in the same directory as the manifest.mpd (Feel free to disable HLS if you don’t need it.)

Putting it all together

If you have that camera with native h264 encoding, like the Pi Camera, here is your copy and paste code!

# sudo mkdir -p /dev/shm/streaming/
sudo ffmpeg -input_format h264 -f video4linux2 -video_size 1920x1080 -framerate 30 -i /dev/video0 -c:v copy -f dash -window_size 10 -remove_at_exit 1 -hls_playlist 1 /dev/shm/streaming/manifest.mpd

You should soon start seeing messages about the manifest and chucks being updated and the current frame rate.

[dash @ 0x20bff00] Opening '/dev/shm/streaming/manifest.mpd.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/media_0.m3u8.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/chunk-stream0-00003.m4s.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/manifest.mpd.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/media_0.m3u8.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/chunk-stream0-00004.m4s.tmp' for writing
frame=  631 fps= 30 q=-1.0 size=N/A time=00:00:20.95 bitrate=N/A speed=1.01x

If you are showing errors like Operation not permitted or Cannot find a proper format please check your input formats and try lower resolutions. Sometimes cameras list their photo taking resolutions which are much higher than their streaming resolutions. If you are still receiving the errors even with the right codec selected, turn the Pi off, check the connections to the camera and turn it back on, as the camera can sometimes get in a bad state or have a loose wire.

To add audio or a text overlay like a timestamp please refer to those linked sections of my previous guide!

Setting up Remote Viewing

First we need to allow nginx to serve up that manifest file. By default nginx is serving up the /var/www/html directory. So it is easy enough to link our in memory folder as a sub folder there.

ln -s /dev/shm/streaming /var/www/html/streaming

Then we need to either have a way to view it via a webpage, or connect to it with a remote player such as VLC. If you have VLC or a viewer for DASH content, you can point it at http://raspberrypi/streaming/manifest.mpd and should start seeing the stream! (If you have a custom hostname or want to use IP, can use hostname -I command to use that in place of raspberrypi). To create a webpage to view the content, we will have to put it in a folder that won’t be deleted on reboot.

I personally chose /var/lib/streaming/index.html as I will also be putting a script in there that will help up set things up again each reboot. Make sure to create the directory first:

mkdir -p /var/lib/streaming

So open up your favorite text editor and copy the following html code into /var/lib/streaming/index.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Raspberry Pi Camera</title>
    <style>
        html body .page {
            height: 100%;
            width: 100%;
        }

        video {
            width: 800px;
        }

        .wrapper {
            width: 800px;
            margin: auto;
        }
    </style>
</head>
<body>
<div class="page">
    <div class="wrapper">
        <h1> Raspberry Pi Camera</h1>
        <video data-dashjs-player autoplay controls type="application/dash+xml"></video>
    </div>
</div>
<script src="https://cdn.dashjs.org/latest/dash.all.min.js"></script>
<script>
    var player = init();

    function init() {
        var video = document.querySelector("video");
        player = dashjs.MediaPlayer().create();
        player.initialize(video, "manifest.mpd", true);
        player.updateSettings({
            'streaming': {
                'lowLatencyEnabled': true,
                'liveDelay': 1,
                'liveCatchUpMinDrift': 0.05,
                'liveCatchUpPlaybackRate': 0.5
            }
        });
        return player;
    }
</script>
</body>
</html>

Now lets link it up to the nginx directory.

ln -s /var/lib/streaming/index.html /var/www/html/streaming/index.html

Now you should be able to view your streaming camera webpage at http://raspberrypi/streaming!

We are using the very expansive dash.js open source library that has a lot of customization options. We are using a few basic ones here to set it up to be better at live streaming, but please check out their project and see how best to tweak it for your needs.

Reboot Script

Now that we have a sexy webpage and working ffmpeg command, we need to save ’em and make sure they can survive reboots. Therefor, we need to create a script that will run on restart to recreate the folder in memory and copy the index file over. I put mine right beside the permanent index file, with /var/lib/streaming/setup_streaming.sh add the following text.

# /var/lib/streaming/setup_streaming.sh
mkdir -p /dev/shm/streaming
if [ ! -e /var/www/html/streaming ]; then
    ln -s  /dev/shm/streaming /var/www/html/streaming
fi 
if [ ! -e /var/www/html/streaming/index.html ]; then
    ln -s /var/lib/streaming/index.html /var/www/html/streaming/index.html
fi 

Don’t forget to make it executable.

chmod +x /var/lib/streaming/setup_streaming.sh

Now to run it on restart, we are going to add this script to /etc/rc.local. Open the /etc/rc.local file and add these lines before the exit 0 at the bottom:

# Streaming Shared Memory Setup
if [ -f /var/lib/streaming/setup_streaming.sh ]; then
    /bin/bash /var/lib/streaming/setup_streaming.sh|| true
fi

Notice we are being extra extra careful to not throw errors here, as at the top of the rc.local file it makes it clear that it should never exit without a clean exit code of 0.

Camera Streaming Service

After we have the location in memory setup, we can start the camera. I chose to do so via a systemd service, so it can restart on errors and easy to manage. In this example it’s called stream_camera but you can change the actual service file name to suit your fancy.

Add a new file at /etc/systemd/system/stream_camera.service.

# /etc/systemd/system/stream_camera.service
[Unit]
Description=Camera Streaming Service
After=network.target rc-local.service

[Service]
Restart=always
RestartSec=20s
ExecStart=ffmpeg -input_format h264 -f video4linux2 -video_size 1920x1080 -framerate 30 -i /dev/video0 -c:v copy -f dash -window_size 10 -remove_at_exit 1 -hls_playlist 1 /dev/shm/streaming/manifest.mpd

[Install]
WantedBy=multi-user.target

Notice we set it to be run after rc-local to make sure we are ready for ffmpeg to write to /dev/shm/streaming.

Personal Cloud Media Server – Encrypted, Streamable, Affordable…Possible?

Is it possible to build a personal media server that is hosted in the cloud, while making privacy, security, and accessibility paramount?I wanted to find out, and this post will dive into the options available to achieve such a possibly as well. (Spoiler: I did end up making my own software to do just this!)  

First, of course, is the why even try this when other options already exist? For example, Plex and Subsonic are some great options if you want to host a media server from your own home. The catch is then you have to have good upload speeds, storage space, an always running server or NAS, and concerned about how private your data really is. Because at the end of the day these are companies, not just software, and they are beholden to the requests of government agencies. They also have all user data in a single, potentially hackable, silo.

Cost Breakdown

So first, fast upload speeds. If you got it, you’re golden, but if your ISP doesn’t offer high enough speeds, you’re screwed. And even if you do have fast upload speeds you now need a server with hard drives that is always being fed electricity. Time to figure out what it costs to remove that need entirely.

On the flip side, a media server is pretty simple, logistics wise. You need a server to host the web page or API, and a storage provider. These could even be the same thing, however I have yet to find a price conscious option that includes both. Instead, for my personal needs, I priced out the difference between buying a 2-bay NAS (as it offers low electric and data redundancy) and using a local server, and constantly paying for an online one storage roughly 2TBs of data (BackBlaze for storage, DigitalOcean for webserver).

 Local Low  High     Cloud Low  High 
2-Bay NAS $150 $300   2TB
Storage 
$140 $600
2TB HDDs $70 $130   Web
server 
$25 $80
Electric $5 $30        
Internet  $0 $50        
             
Immediate
Cost
 $220  $430        
 Yearly Cost $5 $80     $165 $680

And the numbers speak for themselves. It is much, much cheaper,and more viable to buy a NAS and use the established software. The low end of the yearly cloud costs would match the cost of the high end NAS after only five years. Only an idiot with a paranoiac need for total control and security would even think about making their own software and paying so much to host it online.

I naturally started working on the architecture for how to build my cloud media player after the price analysis.

Design

Having content that is both easily streamable and encrypted is a doozy. It wouldn’t really have been possible for the his and hers at home a few years ago. But thanks to MPEG DASH and HLS, we now have video formats with those features built in!

HLS is far more common, but it is a proprietary format developed by Apple and doesn’t have nearly the same feature set as DASH. (Note: Apple should rename their company to Sour Apple, because they refuse to support the internationally standardized DASH format because they hate competition.) So for my own purposes, I chose MPEG DASH.

The real downsize to either of these formats though, is now you have to re-encode all your videos before uploading them, ugh. But it really can’t be helped, and then at least it standardizes your library. After figuring out a bit more of how DASH works, I created a super basic structure I wanted to follow:

The webserver needs to allow for finding and playing the encrypted movies. DASH supports multiple DRM methods, but the best option for a home user is ‘cleartext DRM’ aka a password. Well, you don’t want to store raw passwords in the database, so that means anything in there also has to be encrypted. Oh, and if you really want to storage provider to have no clue what’son there if they scan your stuff, that means subtitles and cover files need to be encrypted too. Oy.

But I really wanted to see if this was possible and learn this new tech, so I plowed on. I also was heavily working with JavaScript at work,so I wrote it to use Node instead of my beloved Python. Two weeks, forty dependencies and a job change later, I had ZABAVA!

Solution?

Zabava translates to “fun” or “entertainment” from Bosnian / Czech / Croatian(and maybe more?) and I thought it was a cool sounding word. Every time I say it aloud, I imitate Jim Carrey from Ace Ventura saying “shikaka”. (No idea if that’s correct, but no one’s stopped me yet.)

It has user authentication with JWT tokens. Thought, admittedly just single user right now with admin rights.

It of course supports editing video information and changing cover file.

And has a script that allows for automatic converting videos to DASH, adding them to the DB and uploading them to the storage provider. I only designed the backend for BackBlaze B2 currently, as that is what I use, but it has a fairly agnostic provider setup to allow easy creation of others.

Sexy right? I am quite proud I was able to get nearly everything to work as envisioned.

Of course, not all fairy tales have the ending we imagine. Some videos still don’t like being converted or played via DASH format and it takes forever and a day to convert and upload terabytes of media files. The code is also not up to my personal quality standard, as a lot was written while figuring out how the tech worked without consideration to overall architecture.

In the end, after uploading all my media and not using it for a few months, I stopped work on the project and have bought a home NAS anyways (as I needed some solution for tons other files as well.)

I may go back and refactor it some weekend I am in a crazy mood, but I don’t think it will fit my person standard for code quality. However, if you are interested in the code and working on it yourself, you can find it on my github

Summary

I learned a lot about building a streaming site and different security methodologies for it, so I even though I wouldn’t qualify the code as a success, it’s surely a personal win.

If I were to do this again I would of course do things a little differently:

  • Try using HLS instead of DASH for wider support
  • Write the backend in Python instead of Node
  • SQL instead of Mongo (in my defense I was using Node at the time)

So, lets go over the criteria again:

  • Encrypted – Yes!
    • Everything was secure
  • Streamable – Yes!
    • Some conversions might have issues until tweaked right, but majority of the content worked as expected.
  • Affordable – No
    • The NAS that I ended up buying cost more than streaming the media, but it was used for a lot more than just the few VHS backups I have. So not the cheapest option, but not a bank breaker.
    • Oh wait, factor in the time spent building the new app… yeah, it ain’t cheap.

As expected, you can’t have all the perks with no downsides, but if Security and Accessibility are your goals more than cost, something like this might interest you.