This post is hardly coding tangential, even if you can programmatically generate 3D prints, but I thought would be a fun thing to share at the end of the year. These prints were all printed on an Ender-3 Pro using PLA or PLA+.
Raspberry Pi Cases
I think this is something we can all appreciate, a well designed and cheap Raspberry Pi case. After printing about a half dozen different ones this year, my favorite is the Pi 4 Honeycomb (free).
I liked their color render so much, I threw some acrylic paint into the comb holes myself. The best part is how it doesn’t require any screws and still holds together snugly.
There is a similar one available for Raspberry Pi 3B in this pack (free).
Scooby-Doo
This mystery solving dog by Exclusive 3D Prints on Patreon (paid, not linking because of adult content), is my overall favorite print of the year.
Sitting at half a foot tall, he was a comfortable size to paint and display.
My Queen’s Crown
I actually bought the 3D printer for my wife, (don’t look at me like that, she uses it some too!), and she makes and prints her own 3D models. Her most impressive is the crown she designed (free) and then hand beaded.
Dice Tray Holder
3D Prints don’t have to just be about themselves, and I love creating stuff for a purpose, so I designed a inset the dice could set into that would fit into a larger handmade dice tray.
This was an iterative process, finally after three designs and five prints I achieved the one I wanted.
Miniatures
I am no artist, but I still find it fun to paint minis. I didn’t start until this year when we bought the 3D printer, yet already have over fifty minis. Most of the ones I printed are from mz4250 (paid via Patreon). There isn’t a recent group photo, but I do have a few of the ones we enjoyed painting the most.
And If you’re looking to paint your own 3D prints, just grab a can of cheap spray paint primer (white, gray or black all work fine), then a few primary acrylic colors and some appropriately sized brushes and get to it! You don’t need fancy mixing trays or washes or Citadel paints. Just whatever is cheapest at your local art store works fine. Then find some paper towels or a spare plate to mix on. And finally a red solo cup (or similar) to fill with water and wash off the brush between colors.
Failures
Sometimes the best attempts still end up crashing and burning. At least with 3D prints you can learn and laugh at them.
Wobbly Photo Turntable
I think my biggest failure was attempting to create a photo turner.
It actually fit together better than expected for a first attempt, holding a Raspberry Pi Zero and servo to drive the top. The top was free-floating, only sitting atop five ball bearings. Sadly the design itself left something to be desired. I ended up just grabbing a cheap turning during Black Friday, but maybe someday I’ll get back to perfecting this design.
I can at least share the silly code I used for this project to keep this article a little coding related:
import time
import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
GPIO.setup(14, GPIO.OUT) # Servo
GPIO.setup(17,GPIO.OUT) # Power Light
GPIO.setup(18, GPIO.IN, pull_up_down=GPIO.PUD_DOWN) # Power button
# Turn light off
GPIO.output(17, False)
# Init servo
pwm=GPIO.PWM(14, 50)
light_on = False
def press(channel):
global light_on
if light_on:
GPIO.output(17, False)
pwm.stop()
else:
GPIO.output(17, True)
pwm.start(0)
pwm.ChangeDutyCycle(10)
light_on = not light_on
GPIO.add_event_detect(18, GPIO.RISING, callback=press)
while True:
try:
time.sleep(0.01)
except KeyboardInterrupt:
GPIO.cleanup()
pwm.stop()
GPIO.output(17, False)
break
Default doesn’t mean Best
Plenty of times the models don’t make the bad print, just having the wrong settings can do the trick. In this case, a stackable Raspberry Pi print turned into more of a wrestling arena from stringification.
Just keep that in mind if you yourself are looking to grab a 3D printer. There will be a lot of bumps along the road, but given time and effort can really make some amazing things!
I wish you all the best and hope you and yours are safe and have a wonderful new year!
A lot of people have pass around a internet video of someone using a crazy high core count computer to display a video via Task Manager’s CPU usage. The one problem with it: it’s probably fake. Task Manager displays the last 60 seconds of activity, not a instant view like shown in the video. But what if we kept the usage the same for sixty seconds, could we at least make an image?
So the first problem, is how do we generate enough load for a CPU usage to noticeably increase? Luckily there is an old goto into the benchmark world that is really simple to implement. Square Roots. Throw a few million square roots at a CPU and watch it light up.
from itertools import count
import math
def run_benchmark():
# Warning, no way to stop other than Ctrl+C or killing the process
for num in count():
math.sqrt(num)
run_benchmark()
The real problem is making sure we have a way to stop it. The easiest way is with a timeout, but just the slight work of gathering system time may throw off how much work we want to do in the future. So lets only do that, say, every 100,000 operations.
from itertools import count
import math
import time
def run_benchmark(timeout=60):
start_time = time.perf_counter()
for num in count():
if num % 100_000 == 0.0:
if time.perf_counter() > start_time + timeout:
return
math.sqrt(num)
run_benchmark()
Excellent, now we can pump up a core to max workload. However, we currently have no control over which CPU it will run on! We have to set the “affinity” of the process. There is no easy way to do that with Python directly, and in this case we have to go straight to the Win32 API. In this case we will use the pywin32 library to access the win32process.
pip install pywin32
We will then use the SetProcessAffinityMask function to lock our program to a specific CPU, which will also require grabbing some details from GetCurrentProcess Win32 API function. One thing not really documented anywhere I could find, is the fact you set the CPU core affinity not by it’s actual number, but by the mask itself. Which we can create by taking 2 ** cpu_core.
from itertools import count
import math
import time
import win32process # pywin32
def run_benchmark(cpu_core=0, timeout=60):
start_time = time.perf_counter()
process_id = win32process.GetCurrentProcess()
win32process.SetProcessAffinityMask(process_id, 2 ** cpu_core)
for num in count():
if num % 100_000 == 0.0:
if time.perf_counter() > start_time + timeout:
return
math.sqrt(num)
run_benchmark()
So now every time we run the program, it should only max out the first core. If we want to do this across multiple cores, we will have to create multiple processes at the same time. My favorite way to do that is with multiprocessing maps. So instead of lighting up a single processor, let’s pump them all to the roof.
from itertools import count
import math
import time
from multiprocessing.pool import Pool
from multiprocessing import cpu_count
import win32process # pywin32
def run_benchmark(cpu_core=0, timeout=60):
start_time = time.perf_counter()
process_id = win32process.GetCurrentProcess()
win32process.SetProcessAffinityMask(process_id, 2 ** cpu_core)
for num in count():
if num % 100_000 == 0.0:
if time.perf_counter() > start_time + timeout:
return
math.sqrt(num)
if __name__ == '__main__':
arguments = [(core, 60) for core in range(cpu_count())]
# [(0, 60), (1, 60), (2, 60), (3, 60)]
# each tuple will be as arguments to run_benchmark as its arguments
with Pool(processes=cpu_count()) as p:
p.starmap(run_benchmark, arguments)
run_benchmark()
We have to throw the multiprocessing after the `if __name__ == ‘__main__’: block due to how CPython starts up on Windows. (It’s also just a good idea for any scripts.)
At this point you could also change up which cores it is running on to see how they correspond on the Task Manager. For example, you could change the range increments to only launch on each other core. range(0, cpu_count(), 2)
On my 8 core machine (so 16 logical cores) I can make a quick X shape by selecting certain cores
arguments = [(core, 100) for core in [0, 3, 5, 6, 9, 10, 12, 15]]
Now remember there are two types of CPU cores according to the operating system, physical and logical. Physical is the number of actual cores on the CPU, but if the CPU has SMT (Simultaneous multi-threading) it will double that. So that means every odd number core is actually a fake one (remember cores start at 0). Which means it has to use some of previous core’s resources. Hence why cores 2, 4, 8 and 14 are showing higher usage.
But what if we want even cooler graphics and don’t want to be limited to just 100% cpu usage? Well then we need to tell the computer to not work for very small amounts of time. Aka sleep. So lets try adding a sleep every 100K ops for oh say, 60 milliseconds.
def run_benchmark(cpu_core=0, timeout=60):
start_time = time.perf_counter()
process_id = win32process.GetCurrentProcess()
win32process.SetProcessAffinityMask(process_id, 2 ** cpu_core)
for num in count():
if num % 100_000 == 0.0:
if time.perf_counter() > start_time + timeout:
return
time.sleep(0.06)
math.sqrt(num)
This time I am also going to run it just on physical cores.
arguments = [(core, 100, 80) for core in range(0, cpu_count(), 2)]
How about that, now it’s using just about 50% usage on each core on my computer. If you’re trying this on your own, this is where the fun begins. I suggest trying out different time offsets to see if you can get a list of times for 10~90% usage. For example, mine is close to:
Then lets throw that into the run_benchmark function to be able to set a precise amount of usage per core.
def run_benchmark(cpu_core=0, timeout=60, usage=100):
start_time = time.perf_counter()
process_id = win32process.GetCurrentProcess()
win32process.SetProcessAffinityMask(process_id, 2 ** cpu_core)
for num in count():
if num % 100_000 == 0.0:
if time.perf_counter() > start_time + timeout:
return
if usage != 100:
time.sleep(usage_to_sleep_time[usage])
math.sqrt(num)
Then if you have enough cores, you can see them all in action at the same time.
arguments = [(core, 100, usage) for core, usage in enumerate(usage_to_sleep_time)]
(I was impatient and didn’t do this one while the computer was idle, hence the messy lower ones.)
From here I am sure some of you could go crazy making individual cores ramp up and done, and create something truly spectacular, but my whistle was wetted. It is totally possible to have control over exactly how much CPU core usage is being shown in Task Manager with Python.
Here is the full final code, hope you enjoyed!
from itertools import count
import math
import time
from multiprocessing.pool import Pool
from multiprocessing import cpu_count
import win32process # pywin32
usage_to_sleep_time = {
100: 0,
90: 0.014,
80: 0.02,
70: 0.03,
60: 0.04,
50: 0.06,
40: 0.08,
30: 0.1,
20: 0.2,
10: 0.5
}
def run_benchmark(cpu_core=0, timeout=60, usage=100):
start_time = time.perf_counter()
process_id = win32process.GetCurrentProcess()
win32process.SetProcessAffinityMask(process_id, 2 ** cpu_core)
for num in count():
if num % 100_000 == 0.0:
if time.perf_counter() > start_time + timeout:
return num
if usage != 100:
time.sleep(usage_to_sleep_time[usage])
math.sqrt(num)
if __name__ == '__main__':
arguments = [(core, 100, usage) for core, usage in enumerate(usage_to_sleep_time)]
with Pool(processes=len(arguments)) as p:
print(p.starmap(run_benchmark, arguments))
I have read several articles on the differences between what all these different types of High Dynamic Range (HDR) video formats are, but few address it from a technical standpoint.
So lets do a quick high level (not so technical) overview of each:
HDR: Larger color space for a bigger range of colors
HDR10: single sheet of color and light information to set upper and lower bounds
HDR10+: a list of sheets for each scene to change those bounds dynamically
Dolby Vision: a proprietary list of bigger sheets
HLG: Regular color space with magic sprinkles on top
Now into a bit more detail. Keep in mind I will be talking about consumer consumption of these concepts, not the prosumer or creator original formats.[1]
HDR
HDR is the starting point for all other types of color and light depth enhancements (expect HLG, discussed below). It uses a 10-bit or 12-bit pixel format and larger color space (bt.2020) instead of the traditional 8-bit (bt.709) to encompass a much larger array of color and light depth.
Images from Wikimedia Commons[2][3]
This new standard is not directly backwards compatible, so sometimes you may come across devices that view what should be a beautifully colorful scene, as painfully muted and off color. For example, here is a HDR video as decoded proper (left) and a direct conversion to 8-bit (right).
Stills taken from Dolby’s Glass Blowing Demo[4]
This is not much of an issue for movies, as HDR is only currently on 4K Blu Rays, so the players have to support BT.2020. However this is an issue for broadcasters who need to provide to older and newer hardware, that’s where HLG comes into play. But first, lets go deeper into “true” HDR options.
HDR10
HDR10 is built on top of HDR as a small set of information given to the video to tell the player the “Mastering Display Color Volume” for the entire video. The Mastering Display consists of static Maximum Frame Average Light Level (MaxFALL) and Maximum Content Light Level (MaxCLL). If you extract these metadata elements, stored in the video’s SEI messages, it will look something like this:
Encoders such as x265 (for HEVC / H.265) and rav1e (AV1) can support mastering display, and will take the arguments in a simplified format. For example they will take MaxFALL as “G(x,y)B(x,y)R(x,y)WP(x,y)L(max,min)”.
Expanding upon HDR10, HDR10+, also called HDR10 plus, supports that information for each frame, or scene by scene. Videos that have HDR10+ support will generally include HDR10 static metadata as well. HDR10+ is specifically for 10-bit video and up to 8K resolution.
The movie will then take a large array of information, for example here is a single frame’s extra metadata.
The HEVC encoder x265 is capable of taking such a metadata JSON file and applying HDR10+ to a video. Right now I know of no AV1 or H.266/VVC encoders that can do so.
Dolby Vision
Dolby’s take on dynamic HDR is their proprietary “Dolby Vision” that unlike all other options, requires a royalty cost. However, it does support 12-bit video for a larger range of color and brightness than HDR10+ is capable of supporting currently.
Right now most 4K Blu-rays and TVs support either Dolby Vision or HDR10+, however it is possible to support both in the same video.
Also unlike all previous options, it is not possible to re-encode a video with Dolby Vision without the original metadata file, as there is no current way to extract the proprietary information. (The x265 encoder does support encoding with Dolby Vision for content creators.)
HLG
Over in the ever dying world of direct-to-house TV broadcasting, they need a way to support both 30 year old TVs and receivers, as well as the brand new hotness of HDR. So they applied some black magic (math) and figured out how to send signals using both gamma and logarithmic curves, hence the name “Hybrid Log-Gamma.” Older SDR TVs read the gamma curve and produce a proper picture while receivers and TVs supporting HLG will show the HDR video. That way they don’t run into the issue shown in the glass blowing images above.
Learn More
Obviously this was a “getting your feet wet” post, and I wanted to include links to a lot of great resources to learn more on the various aspects of HDR and video formats.
[1] I will be referring to these formats as they are used in consumer releases, such as 4K Blu-Rays and online streaming services. As such, they use a more compressed video format “yuv42010le” and are HEVC encoded using standard RECs. Just be aware that these ideas may exist and be represented differently in creator and prosumer product that have full color ranges.
First off, if you don’t care about the technicalities and just want a script to do everything for you, here you go! If you’re still interested in how it all works or want to tweak the settings, read on.
This article will walk you through how to either copy or convert video from your webcam or pi camera, set it up as a systemd service, and finally view it on a webpage or access it remotely.
MPEG-DASH vs HLS vs RSTP
So to clear this up first of all, these are “containers” that wrap around the actual video, which is a particular “codec” (such as h264). DASH and RTSP are fully codec agnostic, meaning they are capable of wrapping around any type of video codec. The big gotcha is what type of videos the viewer supports (and in RTSP’s case the middleman server as well.)
So if DASH and RTSP can handle everything, why even bother with HLS? Long story short, Apple, who developed HLS, is a bully, so they don’t support the open MPEG-DASH on their devices. Meaning if you are trying to share these video streams with the public or view on an Apple device, you will get the most compatibility with HLS.
So now the difference really comes down to how DASH/HLS are HTTP based protocols that can easily be supported in browser. This makes it super easy to set up an all-in-one device that can host it’s own webpage to view a video at.
Whereas RTSP requires additional software, such as VLC or a security system to view it. The real advantage with RTSP is the fact it really is nearly “real time” compared to DASH/HLS. Using my Raspberry Pis DASH/HLS seem to have a 10~20 second delay, compared to about 1 second for RTSP. As I already went over how to set that up, I won’t repeat it here and only go over DASH. But I personally use RSTP for my own home setup still.
Setting up required software
The two programs you will need are a file server (nginx, apache, python -m http.server, etc…) to host the DASH/HLS content and ffmpeg. And you don’t have to hand compile either!
Super short version:
sudo apt install nginx ffmpeg -y
To better understand why we need them and how to test them to make sure they are working properly, read on. Otherwise, skip ahead to “Gather Camera Details”.
File Server
Last time we use RTSP which required a special service of it’s own. Now we are using HLS and MPEG-DASH, which produce manifest and accompany stream files on the local system. For example, MPEG-DASH will create a manifest.mpd file that contains links to *.m4s files in the same directory which are the chunked up video files.
That means if we make those files accessible remotely, we can use standard HTTP to transport the video. Hence the need for a basic file server. I personally use nginx for the final setup, as it’s fast, easy to use, and has defaults we can use out of the box. So lets install it!
sudo apt install nginx -y
Now all you need to do is open up a web browser on another computer on that network and connect to http://raspberrypi (If you changed hostname, or having trouble connecting, run hostname -I to see it’s IP address and use http://<ip_address> instead.) You should see a simple webpage that says “Welcome to nginx!”
FFmpeg
Since the last article came out, FFmpeg has finally started shipping with hardware acceleration built in! If you still want to compile in some custom libraries or try and optimize it for your needs, check out my Raspberry Pi FFmpeg compile guide. Otherwise, just download it from the distribution repositories.
sudo apt install ffmpeg -y
You can verify it’s part of the package by checking the encoders for h264_omx.
ffmpeg -hide_banner -encoders | grep omx
Should produce V….. h264_omx OpenMAX IL H.264 video encoder (codec h264) or similar. If for some reason it doesn’t have that or other libraries you are looking for, such as the popular fdk-aac, look into my article onto compiling FFmpeg yourself, or use the helper script with the option --compile-ffmpeg.
Gather camera details
If you have the helper script, simply run it with the option --camera-details and it will print out each device and their formats with their highest resolution for each.
Under the hood, this is running the following command for every device found.
ffmpeg -hide_banner -f video4linux2 -list_formats all -i /dev/video0
To also see what frame rates are supported per resolution, you will have to run the v4l2-ctl command for that device.
v4l2-ctl -d /dev/video0 --list-formats-ext
Create the FFmpeg command
The FFmpeg command is particular about order when talking about input and output details. Ours will be broken down into the following blocks:
ffmpeg <incoming video details> -i <device> <conversion details> <output>
So let’s say you are using a raspberry pi camera and want to stream 1080p video without re-encoding it. We first have to tell FFmpeg about the camera details it will pull from.
Hopefully each of those parts are pretty self explanatory. We can also reduce -video_size, aka the incoming resolution to -s, and -framerate, aka the fps to -r.
Network Bandwidth Considerations
Internet streamers, beware you may not be able to upload directly from the camera’s full 1080p at 30fps. I did a quick test using vnstat over a wired connection with a Pi Zero, and found my 5MP OV5647 camera was using almost 20Mbit/s. Keep in mind the official Pi Camera with the sony sensor is 8MP so may be even higher than that.
The following tests were done at two minute averages while the stream was being watched. The averages were recorded, and generally the peaks were 2x the average.
Resolution
fps
Average Mbit/s
1920×1080
30
20.12
1920×1080
15
5.12
1280×720
60
15.81
1280×720
30
3.94
1280×720
15
2.75
640×480
90
5.66
640×480
60
4.43
640×480
30
0.93
640×480
15
0.43
Using a Pi Zero with 5MP OV5647 camera over ethernet
Conversion options
Since this already is h264 we don’t need anything other than to say copy the incoming stream. So that option is -codec:v copy or shorthand -c:v copy. Which is saying set the codec of v for video tracks to copy aka don’t convert.
-c:v copy
If you instead had a webcam that only supported mjpeg input, or if you needed to add text overlay to the video, you would have to recompile FFmpeg. With the Raspberry Pi, you’ll want to use the built in hardware encoder, h264_omx. You would then also have to set the bitrate (-b:v) of the outgoing video. That is really camera / network dependent, but my rule of thumb is use video width x hight x 2. So 1920x1080x2 ==4,147,200, so I would set the bitrate to 4M (aka ~4000kb, or ~4000000 bytes).
-c:v h264_omx -b:v 4M
The Raspberry Pi OpenMAX (omx) hardware encoder has very limited options, and doesn’t support constant quality or rate factors like libx264 does. So the only way to adjust quality is with the bitrate. As for general quality, it sits between libx264s ultrafast and superfast presets, which is somewhat disappointing but not surprising for a real-time hardware encoder.
MPEG-DASH and HLS output
Personally I would never recommend HLS to a friend, as MPEG-DASH is all around a more open and powerful muxer. But I understand some legacy systems don’t have DASH support yet. Thankfully, FFmpeg’s dash module gives us HLS for free! (Note that some systems don’t even support that, and you may end up having to use only the hlsmuxer.)
DASH and HLS both crate playlist files locally, with chucked up video files beside them. This creates a few problems, first is the cleanup and management of those files. Thankfully, the DASH model has options to delete all those files on exit, as well as the ability to only keep so many video chunks on disk at a time.
The bigger problem is the constant writing to the disk. In this case an SD card that is a wear item and has higher error rates with the more writes it experiences. So to save the SD card, and ourselves future headaches, we are going to write these files to memory instead!
sudo mkdir -p /dev/shm/streaming/
Tada, we now have a folder in shared memory space we can use. The caveat is it will be removed after restarts, so we will have to make sure it’s recreated before or FFmpeg service is started. But lets not get ahead of ourselves. We just need to know the rest of our FFmpeg command.
We are using just a few options of what FFmpeg’s DASH muxer can do if you do need further customization, but I doubt it for most cases.
I am setting the max number of video chuncks to be kept at 10 via -window_size and telling FFmpeg to delete them and the manifest file when it stops running with -remove_at_exit 1. Then we enable HLS with -hls_playlist 1 which creates a master.m3u8 file in the same directory as the manifest.mpd (Feel free to disable HLS if you don’t need it.)
Putting it all together
If you have that camera with native h264 encoding, like the Pi Camera, here is your copy and paste code!
You should soon start seeing messages about the manifest and chucks being updated and the current frame rate.
[dash @ 0x20bff00] Opening '/dev/shm/streaming/manifest.mpd.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/media_0.m3u8.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/chunk-stream0-00003.m4s.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/manifest.mpd.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/media_0.m3u8.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/chunk-stream0-00004.m4s.tmp' for writing
frame= 631 fps= 30 q=-1.0 size=N/A time=00:00:20.95 bitrate=N/A speed=1.01x
If you are showing errors like Operation not permitted or Cannot find a proper format please check your input formats and try lower resolutions. Sometimes cameras list their photo taking resolutions which are much higher than their streaming resolutions. If you are still receiving the errors even with the right codec selected, turn the Pi off, check the connections to the camera and turn it back on, as the camera can sometimes get in a bad state or have a loose wire.
First we need to allow nginx to serve up that manifest file. By default nginx is serving up the /var/www/html directory. So it is easy enough to link our in memory folder as a sub folder there.
ln -s /dev/shm/streaming /var/www/html/streaming
Then we need to either have a way to view it via a webpage, or connect to it with a remote player such as VLC. If you have VLC or a viewer for DASH content, you can point it at http://raspberrypi/streaming/manifest.mpd and should start seeing the stream! (If you have a custom hostname or want to use IP, can use hostname -I command to use that in place of raspberrypi). To create a webpage to view the content, we will have to put it in a folder that won’t be deleted on reboot.
I personally chose /var/lib/streaming/index.html as I will also be putting a script in there that will help up set things up again each reboot. Make sure to create the directory first:
mkdir -p /var/lib/streaming
So open up your favorite text editor and copy the following html code into /var/lib/streaming/index.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Raspberry Pi Camera</title>
<style>
html body .page {
height: 100%;
width: 100%;
}
video {
width: 800px;
}
.wrapper {
width: 800px;
margin: auto;
}
</style>
</head>
<body>
<div class="page">
<div class="wrapper">
<h1> Raspberry Pi Camera</h1>
<video data-dashjs-player autoplay controls type="application/dash+xml"></video>
</div>
</div>
<script src="https://cdn.dashjs.org/latest/dash.all.min.js"></script>
<script>
var player = init();
function init() {
var video = document.querySelector("video");
player = dashjs.MediaPlayer().create();
player.initialize(video, "manifest.mpd", true);
player.updateSettings({
'streaming': {
'lowLatencyEnabled': true,
'liveDelay': 1,
'liveCatchUpMinDrift': 0.05,
'liveCatchUpPlaybackRate': 0.5
}
});
return player;
}
</script>
</body>
</html>
Now you should be able to view your streaming camera webpage at http://raspberrypi/streaming!
We are using the very expansive dash.js open source library that has a lot of customization options. We are using a few basic ones here to set it up to be better at live streaming, but please check out their project and see how best to tweak it for your needs.
Reboot Script
Now that we have a sexy webpage and working ffmpeg command, we need to save ’em and make sure they can survive reboots. Therefor, we need to create a script that will run on restart to recreate the folder in memory and copy the index file over. I put mine right beside the permanent index file, with /var/lib/streaming/setup_streaming.sh add the following text.
# /var/lib/streaming/setup_streaming.sh
mkdir -p /dev/shm/streaming
if [ ! -e /var/www/html/streaming ]; then
ln -s /dev/shm/streaming /var/www/html/streaming
fi
if [ ! -e /var/www/html/streaming/index.html ]; then
ln -s /var/lib/streaming/index.html /var/www/html/streaming/index.html
fi
Don’t forget to make it executable.
chmod +x /var/lib/streaming/setup_streaming.sh
Now to run it on restart, we are going to add this script to /etc/rc.local. Open the /etc/rc.local file and add these lines before the exit 0 at the bottom:
# Streaming Shared Memory Setup
if [ -f /var/lib/streaming/setup_streaming.sh ]; then
/bin/bash /var/lib/streaming/setup_streaming.sh|| true
fi
Notice we are being extra extra careful to not throw errors here, as at the top of the rc.local file it makes it clear that it should never exit without a clean exit code of 0.
Camera Streaming Service
After we have the location in memory setup, we can start the camera. I chose to do so via a systemd service, so it can restart on errors and easy to manage. In this example it’s called stream_camera but you can change the actual service file name to suit your fancy.
Add a new file at /etc/systemd/system/stream_camera.service.
I talked about this before with my encoding setting for handbrake post, but there is was a fundamental flaw using Handbrake for HDR 10-bit video….it only has had a 8-bit internal pipeline! It and most other GUIs don’t yet support dynamic metadata, such as HDR10+ or Dolby Vision though.
2021-02 Update: Handbrake’s latest code has HDR10 static metadata support.
Thankfully, you can avoid some hassle and save HDR10 or HDR10+ by using FastFlix instead, or directly use FFmpeg. (To learn about extracting and converting with HDR10+ or saving Dolby Vision by remuxing, skip ahead.) If you want to do the work yourself, here are the two most basic commands you need to save your juicy HDR10 data. This will use the Dolby Vision (profile 8.1) Glass Blowing demo.
Extract the Mastering Display metadata
First, we need to use FFprobe to extract the Mastering Display and Content Light Level metadata. We are going to tell it to only read the first frame’s metadata -read_intervals "%+#1" for the file GlassBlowingUHD.mp4
I chose to output it with json via the -print_format json option to make it more machine parsible, but you can omit that if you just want the text.
We are now going to take all that data, and break it down into groups of <color abbreviation>(<x>, <y>) while leaving off the right side of the in most cases*, so for example we combine red_x"35400/50000"and red_y"14600/50000" into R(35400,14600).
*If your data for colors is not divided by /50000 or luminescence not divided by 10000 and have been simplified, you will have to expand it back out to the full ratio. For example if yours lists 'red_x': '17/25', 'red_y': '8/25' you will have to divide 50000 by the current denominator (25) to get the ratio (2000) and multiply that by the numerator (17 and 8) to get the proper R(34000,16000).
This data, as well as the Content light level <max_content>,<max_average> of 0,0 will be fed into the encoder command options.
Convert the video
This command converts only the video, keeping the HDR10 intact. We will have to pass these arguments not to ffmpeg, but to the x265 encoder directly via the -x265-params option. (If you’re not familiar with FFmpeg, don’t fret. FastFlix, which I talk about later, will do the work for you!)
Let’s break down what we are throwing into the x265-params:
hdr-opt=1 we are telling it yes, we will be using HDR
repeat-headers=1we want these headers on every frame as required
colorprim, transfer and colormatrix the same as ffprobe listed
master-display this is where we add our color string from above
max-cll Content light level data, in our case 0,0
During a conversion like this, when a Dolby Vision layer exists, you will see a lot of messages like [hevc @ 000001f93ece2e00] Skipping NAL unit 62 because there is an entire layer that ffmpeg does not yet know how to decode.
For the quality of the conversion, I was setting it to -crf 20 with a -preset veryfast to convert it quickly without a lot of quality loss. I dig deeper into how FFmpeg handles crf vs preset with regards to quality below.
All sound and data and everything will be copied over thanks to the -map 0 option, that is a blanket statement of “copy everything from the first (0 start index) input stream”.
That is really all you need to know for the basics of how to encode your video and save the HDR10 data!
FFmpeg conversion settings
I covered this a bit before in the other post, but I wanted to go through the full gauntlet of presets and crfs one might feel inclined to use. I compared each encoding with the original using VMAF and SSIM calculations over a 11 second clip. Then, I created over a 100 conversions for this single chart, so it is a little cramped:
First takeaways are that there is no real difference between veryslow and slower, nor between veryfast and faster, as their lines are drawn on top of each other. The same is true for both VMAF and SSIM scores.
Second, no one in their right mind would ever keep a recording stored by using ultrafast. That is purely for real time streaming use.
Now for VMAF scores, 5~6 points away from source is visually distinguishable when watching. In other words it will have very noticeable artifacts. Personally I can tell on my screen with just a single digit difference, and some people are even more sensitive, so this is by no means an exact tell all. At minimum, lets zoom in a bit and get rid of anything that will produce video with very noticeable artifacts.
SSIM Graphs
From these chart, it seems clear that there is obviously no reason whatsoever to ever use anything other than slow. Which I personally do for anything I am encoding. However, slow lives up to its namesake.
Encoding Speed and Bitrate
I had to trim off veryslow and slower from the main chart to be able to even see the rest, and slow is still almost three times slower than medium. All the charts contain the same data, just with some of the longer running presets removed from each to better see details of the faster presets.
Please note, the first three crf datapoints are little dirty, as the system was in use for the first three tests. However, there is enough clean data to see how that compares down the line.
To see a clearer picture of how long it takes for each of the presets, I will exclude those first three times, and average the remaining data. The data is then compared against the medium (default) present and the original clip length of eleven seconds.
Preset
Time
vs “medium”
vs clip length(11s)
ultrafast
11.204
3.370x
0.982x
superfast
12.175
3.101x
0.903x
veryfast
19.139
1.973x
0.575x
faster
19.169
1.970x
0.574x
fast
22.792
1.657x
0.482x
medium
37.764
1.000x
0.291x
slow
97.755
0.386x
0.112x
slower
315.900
0.120x
0.035x
veryslow
574.580
0.066x
0.019x
What is a little scary here is that even with “ultrafast” preset we are not able to get realtime conversion, and these tests were run on a fairly high powered system wielding an i9-9900k! While it might be clear from the crf graph that slow is the clear winner, unless you have a beefy computer, it may be a non-option.
Use the slowest preset that you have patience for
FFmpeg encoding guide
Also unlike VBR encoding, the average bitrate and filesize using crf will wildly differ based upon different source material. This next chart is just showing off the basic curve effect you will see, however it cannot be compared to what you may expect to see with your file.
The two big jumps are between slow and medium as well as veryfast and superfast. That is interesting because while slow and medium are quite far apart on the VMAF comparison, veryfast and superfast are not. I expected a much larger dip from superfast to ultrafast but was wrong.
FastFlix, doing the heavy lifting for you!
I have written a GUI program, FastFlix, around FFmpeg and other tools to convert videos easily to HEVC, AV1 and other formats. While I won’t promise it will provide everything you are looking for, it will do the work for you of extracting the HDR10 details of a video and passing them into a FFmpeg command. FastFlix can even handle HDR10+ metadata! It also has a panel that shows you exactly the command(s) it is about to run, so you could copy it and modify it to your hearts content!
FastFlix 3.0
If you have any problems with it please help by raising an issue!
Extracting and encoding with HDR10+ metadata
First off, this is not for the faint of heart. Thankfully the newest FFmpeg builds for Windows now support HDR10+ metadata files by default, so this process has become a lot easier. Here is a quick overview how to do it, also a huge shutout to “Frank” in the comments below for this tool out to me!
You will have to download a copy of hdr10plus_parser from quietviod’s repo.
Check to make sure your video has HDR10+ information it can read.
Option 2: (original article) Using custom compiled x265
Now the painful part you’ll have to work on a bit yourself. To use the metadata file, you will need to custom compiled x265 with the cmake option “HDR10_PLUS” . Otherwise when you try to convert with it, you’ll see a message like “x265 [warning]: –dhdr10-info disabled. Enable HDR10_PLUS in cmake.” in the the output, but it will still encode, just without HDR10+ support.
Once that is compiled, you will have to use x265 as part of your conversion pipeline. Use x265 to convert your video with the HDR10+ metadata and other details you discovered earlier.
Then you will have to use that output.hevc file with ffmpeg to combine it with your audio conversions and subtitles and so on to repackage it.
Saving Dolby Vision
Thanks to Jesse Robinson in the comments, it seems it now may be possible to extract and convert a video with Dolby Vision without the original RPU file. Like the original HDR10+ section, you will need the x265 executable directly, as it does not support that option through FFmpeg at all.
Note I say “saving” and not “converting”, because unless you have the original RPU file for the DV to pass to x265, you’re out of luck as of now and cannot convert the video. The x265 encoder is able to take a RPU file and create a Dolby Vision ready movie.
Just like the manual HDR10+ process, first you have to use an extraction tool, in this case quietvoid’s dov_tool to extract the RPU.
There are a few extra caveats when converting a Dolby Vision file with x265. First you need to use the appropriate 10 or 12 bit version. Second, change pix_fmt, input-depth and output-depth as required. Finally you MUST provide a dobly-vision-profile level for it to read the RPU as well as enable VBV by providing vbv-bufsize and vbv-maxrate (read the x265 command line options to see what settings are best for you.)
Once the glass_dv.hevc file is created, you will need to use tsMuxerR ( this version is recommended by some, and the newest nightly is linked below) to put it back into a container as well as add the audio back.
Currently trying to add the glass_dv.hevc back into a container with FFmpeg or MP4Box will break the Dolby Vision. Hence the need for the special Muxer. This will result in video that has BL+RPU Dolby Vision.
It is possible to also only convert the audio and change around the streams with remuxers. For example tsMuxeR (nightly build, not default download) is popular to be able to take mkv files that most TVs won’t recognize HDR in, and remux them into into ts files so they do. If you also have TrueHD sound tracks, you may need to use eac3to first to break it into the TrueHD and AC3 Core tracks before muxing.
Easily Viewing HDR / Video information
Another helpful program to quickly view what type of HDR a video has is MediaInfo. For example here is the original Dolby Vision Glass Blowing video info (some trimmed):
Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L5.1@Main
HDR format : Dolby Vision, Version 1.0, dvhe.08.09, BL+RPU, HDR10 compatible / SMPTE ST 2086, HDR10 compatible
Codec ID : hev1
Color space : YUV
Chroma subsampling : 4:2:0 (Type 2)
Bit depth : 10 bits
Color range : Limited
Color primaries : BT.2020
Transfer characteristics : PQ
Matrix coefficients : BT.2020 non-constant
Mastering display color primaries : BT.2020
Mastering display luminance : min: 0.0050 cd/m2, max: 4000 cd/m2
Codec configuration box : hvcC+dvvC
And here it is after conversion:
Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L5.1@Main
HDR format : SMPTE ST 2086, HDR10 compatible
Codec ID : V_MPEGH/ISO/HEVC
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 10 bits
Color range : Limited
Color primaries : BT.2020
Transfer characteristics : PQ
Matrix coefficients : BT.2020 non-constant
Mastering display color primaries : BT.2020
Mastering display luminance : min: 0.0050 cd/m2, max: 4000 cd/m2
Notice we have lost the Dolby Vision, BL+RPU information but at least we retained the HDR10 data, which Handbrake can’t do!
That’s a wrap!
Hope you found this information useful, and please feel free to leave a comment for feedback, suggestions or questions!
