Code Calamity - Hodgepodges from a coder's mind

A Raspberry Pi Streaming Camera using MPEG-DASH, HLS or RTSP

Posted on August 10, 2020 by Chris Griffith Tutorial 0

This is an improvement on my previous article, Raspberry Pi Hardware Accelerated RTSP Camera, now with the option of using more modern technology, MPEG-DASH and HLS!

First off, if you don’t care about the technicalities and just want a script to do everything for you, here you go! If you’re still interested in how it all works or want to tweak the settings, read on.

This article will walk you through how to either copy or convert video from your webcam or pi camera, set it up as a systemd service, and finally view it on a webpage or access it remotely.

MPEG-DASH vs HLS vs RSTP

So to clear this up first of all, these are “containers” that wrap around the actual video, which is a particular “codec” (such as h264). DASH and RTSP are fully codec agnostic, meaning they are capable of wrapping around any type of video codec. The big gotcha is what type of videos the viewer supports (and in RTSP’s case the middleman server as well.)

So if DASH and RTSP can handle everything, why even bother with HLS? Long story short, Apple, who developed HLS, is a bully, so they don’t support the open MPEG-DASH on their devices. Meaning if you are trying to share these video streams with the public or view on an Apple device, you will get the most compatibility with HLS.

So now the difference really comes down to how DASH/HLS are HTTP based protocols that can easily be supported in browser. This makes it super easy to set up an all-in-one device that can host it’s own webpage to view a video at.

Whereas RTSP requires additional software, such as VLC or a security system to view it. The real advantage with RTSP is the fact it really is nearly “real time” compared to DASH/HLS. Using my Raspberry Pis DASH/HLS seem to have a 10~20 second delay, compared to about 1 second for RTSP. As I already went over how to set that up, I won’t repeat it here and only go over DASH. But I personally use RSTP for my own home setup still.

Setting up required software

The two programs you will need are a file server (nginx, apache, python -m http.server, etc…) to host the DASH/HLS content and ffmpeg. And you don’t have to hand compile either!

Super short version:

sudo apt install nginx ffmpeg -y

To better understand why we need them and how to test them to make sure they are working properly, read on. Otherwise, skip ahead to “Gather Camera Details”.

File Server

Last time we use RTSP which required a special service of it’s own. Now we are using HLS and MPEG-DASH, which produce manifest and accompany stream files on the local system. For example, MPEG-DASH will create a manifest.mpd file that contains links to *.m4s files in the same directory which are the chunked up video files.

That means if we make those files accessible remotely, we can use standard HTTP to transport the video. Hence the need for a basic file server. I personally use nginx for the final setup, as it’s fast, easy to use, and has defaults we can use out of the box. So lets install it!

 sudo apt install nginx -y

Now all you need to do is open up a web browser on another computer on that network and connect to http://raspberrypi (If you changed hostname, or having trouble connecting, run hostname -I to see it’s IP address and use http://<ip_address> instead.) You should see a simple webpage that says “Welcome to nginx!”

FFmpeg

Since the last article came out, FFmpeg has finally started shipping with hardware acceleration built in! If you still want to compile in some custom libraries or try and optimize it for your needs, check out my Raspberry Pi FFmpeg compile guide. Otherwise, just download it from the distribution repositories.

sudo apt install ffmpeg -y

You can verify it’s part of the package by checking the encoders for h264_omx.

ffmpeg -hide_banner -encoders | grep omx

Should produce V….. h264_omx OpenMAX IL H.264 video encoder (codec h264) or similar. If for some reason it doesn’t have that or other libraries you are looking for, such as the popular fdk-aac, look into my article onto compiling FFmpeg yourself, or use the helper script with the option --compile-ffmpeg.

Gather camera details

If you have the helper script, simply run it with the option --camera-details and it will print out each device and their formats with their highest resolution for each.

python3 streaming_setup.py --camera-details
# /dev/video0: {'yuyv422': '1280x800', 'mjpeg': '1280x720'}

Under the hood, this is running the following command for every device found.

ffmpeg -hide_banner -f video4linux2 -list_formats all -i /dev/video0

To also see what frame rates are supported per resolution, you will have to run the v4l2-ctl command for that device.

v4l2-ctl -d /dev/video0 --list-formats-ext

Create the FFmpeg command

The FFmpeg command is particular about order when talking about input and output details. Ours will be broken down into the following blocks:

ffmpeg <incoming video details> -i <device> <conversion details> <output>

So let’s say you are using a raspberry pi camera and want to stream 1080p video without re-encoding it. We first have to tell FFmpeg about the camera details it will pull from.

Applying the Camera Details

In longhand, it would look like this:

-input_format h264 -f video4linux2 -video_size 1920x1080 -framerate 30 -i /dev/video0

Hopefully each of those parts are pretty self explanatory. We can also reduce -video_size, aka the incoming resolution to -s, and -framerate, aka the fps to -r.

Network Bandwidth Considerations

Internet streamers, beware you may not be able to upload directly from the camera’s full 1080p at 30fps. I did a quick test using vnstat over a wired connection with a Pi Zero, and found my 5MP OV5647 camera was using almost 20Mbit/s. Keep in mind the official Pi Camera with the sony sensor is 8MP so may be even higher than that.

The following tests were done at two minute averages while the stream was being watched. The averages were recorded, and generally the peaks were 2x the average.

Resolution	fps	Average Mbit/s
1920×1080	30	20.12
1920×1080	15	5.12
1280×720	60	15.81
1280×720	30	3.94
1280×720	15	2.75
640×480	90	5.66
640×480	60	4.43
640×480	30	0.93
640×480	15	0.43

Using a Pi Zero with 5MP OV5647 camera over ethernet

Conversion options

Since this already is h264 we don’t need anything other than to say copy the incoming stream. So that option is -codec:v copy or shorthand -c:v copy. Which is saying set the codec of v for video tracks to copy aka don’t convert.

-c:v copy

If you instead had a webcam that only supported mjpeg input, or if you needed to add text overlay to the video, you would have to recompile FFmpeg. With the Raspberry Pi, you’ll want to use the built in hardware encoder, h264_omx. You would then also have to set the bitrate (-b:v) of the outgoing video. That is really camera / network dependent, but my rule of thumb is use video width x hight x 2. So 1920x1080x2 ==4,147,200, so I would set the bitrate to 4M (aka ~4000kb, or ~4000000 bytes).

-c:v h264_omx -b:v 4M

The Raspberry Pi OpenMAX (omx) hardware encoder has very limited options, and doesn’t support constant quality or rate factors like libx264 does. So the only way to adjust quality is with the bitrate. As for general quality, it sits between libx264s ultrafast and superfast presets, which is somewhat disappointing but not surprising for a real-time hardware encoder.

MPEG-DASH and HLS output

Personally I would never recommend HLS to a friend, as MPEG-DASH is all around a more open and powerful muxer. But I understand some legacy systems don’t have DASH support yet. Thankfully, FFmpeg’s dash module gives us HLS for free! (Note that some systems don’t even support that, and you may end up having to use only the hls muxer.)

DASH and HLS both crate playlist files locally, with chucked up video files beside them. This creates a few problems, first is the cleanup and management of those files. Thankfully, the DASH model has options to delete all those files on exit, as well as the ability to only keep so many video chunks on disk at a time.

The bigger problem is the constant writing to the disk. In this case an SD card that is a wear item and has higher error rates with the more writes it experiences. So to save the SD card, and ourselves future headaches, we are going to write these files to memory instead!

sudo mkdir -p /dev/shm/streaming/

Tada, we now have a folder in shared memory space we can use. The caveat is it will be removed after restarts, so we will have to make sure it’s recreated before or FFmpeg service is started. But lets not get ahead of ourselves. We just need to know the rest of our FFmpeg command.

-f dash -window_size 10 -remove_at_exit 1 -hls_playlist 1 /dev/shm/streaming/manifest.mpd

We are using just a few options of what FFmpeg’s DASH muxer can do if you do need further customization, but I doubt it for most cases.

I am setting the max number of video chuncks to be kept at 10 via -window_size and telling FFmpeg to delete them and the manifest file when it stops running with -remove_at_exit 1. Then we enable HLS with -hls_playlist 1 which creates a master.m3u8 file in the same directory as the manifest.mpd (Feel free to disable HLS if you don’t need it.)

Putting it all together

If you have that camera with native h264 encoding, like the Pi Camera, here is your copy and paste code!

# sudo mkdir -p /dev/shm/streaming/
sudo ffmpeg -input_format h264 -f video4linux2 -video_size 1920x1080 -framerate 30 -i /dev/video0 -c:v copy -f dash -window_size 10 -remove_at_exit 1 -hls_playlist 1 /dev/shm/streaming/manifest.mpd

You should soon start seeing messages about the manifest and chucks being updated and the current frame rate.

[dash @ 0x20bff00] Opening '/dev/shm/streaming/manifest.mpd.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/media_0.m3u8.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/chunk-stream0-00003.m4s.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/manifest.mpd.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/media_0.m3u8.tmp' for writing
[dash @ 0x20bff00] Opening '/dev/shm/streaming/chunk-stream0-00004.m4s.tmp' for writing
frame=  631 fps= 30 q=-1.0 size=N/A time=00:00:20.95 bitrate=N/A speed=1.01x

If you are showing errors like Operation not permitted or Cannot find a proper format please check your input formats and try lower resolutions. Sometimes cameras list their photo taking resolutions which are much higher than their streaming resolutions. If you are still receiving the errors even with the right codec selected, turn the Pi off, check the connections to the camera and turn it back on, as the camera can sometimes get in a bad state or have a loose wire.

To add audio or a text overlay like a timestamp please refer to those linked sections of my previous guide!

Setting up Remote Viewing

First we need to allow nginx to serve up that manifest file. By default nginx is serving up the /var/www/html directory. So it is easy enough to link our in memory folder as a sub folder there.

ln -s /dev/shm/streaming /var/www/html/streaming

Then we need to either have a way to view it via a webpage, or connect to it with a remote player such as VLC. If you have VLC or a viewer for DASH content, you can point it at http://raspberrypi/streaming/manifest.mpd and should start seeing the stream! (If you have a custom hostname or want to use IP, can use hostname -I command to use that in place of raspberrypi). To create a webpage to view the content, we will have to put it in a folder that won’t be deleted on reboot.

I personally chose /var/lib/streaming/index.html as I will also be putting a script in there that will help up set things up again each reboot. Make sure to create the directory first:

mkdir -p /var/lib/streaming

So open up your favorite text editor and copy the following html code into /var/lib/streaming/index.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Raspberry Pi Camera</title>
    <style>
        html body .page {
            height: 100%;
            width: 100%;
        }

        video {
            width: 800px;
        }

        .wrapper {
            width: 800px;
            margin: auto;
        }
    </style>
</head>
<body>
<div class="page">
    <div class="wrapper">
        <h1> Raspberry Pi Camera</h1>
        <video data-dashjs-player autoplay controls type="application/dash+xml"></video>
    </div>
</div>
<script src="https://cdn.dashjs.org/latest/dash.all.min.js"></script>
<script>
    var player = init();

    function init() {
        var video = document.querySelector("video");
        player = dashjs.MediaPlayer().create();
        player.initialize(video, "manifest.mpd", true);
        player.updateSettings({
            'streaming': {
                'lowLatencyEnabled': true,
                'liveDelay': 1,
                'liveCatchUpMinDrift': 0.05,
                'liveCatchUpPlaybackRate': 0.5
            }
        });
        return player;
    }
</script>
</body>
</html>

Now lets link it up to the nginx directory.

ln -s /var/lib/streaming/index.html /var/www/html/streaming/index.html

Now you should be able to view your streaming camera webpage at http://raspberrypi/streaming!

We are using the very expansive dash.js open source library that has a lot of customization options. We are using a few basic ones here to set it up to be better at live streaming, but please check out their project and see how best to tweak it for your needs.

Reboot Script

Now that we have a sexy webpage and working ffmpeg command, we need to save ’em and make sure they can survive reboots. Therefor, we need to create a script that will run on restart to recreate the folder in memory and copy the index file over. I put mine right beside the permanent index file, with /var/lib/streaming/setup_streaming.sh add the following text.

# /var/lib/streaming/setup_streaming.sh
mkdir -p /dev/shm/streaming
if [ ! -e /var/www/html/streaming ]; then
    ln -s  /dev/shm/streaming /var/www/html/streaming
fi 
if [ ! -e /var/www/html/streaming/index.html ]; then
    ln -s /var/lib/streaming/index.html /var/www/html/streaming/index.html
fi

Don’t forget to make it executable.

chmod +x /var/lib/streaming/setup_streaming.sh

Now to run it on restart, we are going to add this script to /etc/rc.local. Open the /etc/rc.local file and add these lines before the exit 0 at the bottom:

# Streaming Shared Memory Setup
if [ -f /var/lib/streaming/setup_streaming.sh ]; then
    /bin/bash /var/lib/streaming/setup_streaming.sh|| true
fi

Notice we are being extra extra careful to not throw errors here, as at the top of the rc.local file it makes it clear that it should never exit without a clean exit code of 0.

Camera Streaming Service

After we have the location in memory setup, we can start the camera. I chose to do so via a systemd service, so it can restart on errors and easy to manage. In this example it’s called stream_camera but you can change the actual service file name to suit your fancy.

Add a new file at /etc/systemd/system/stream_camera.service.

# /etc/systemd/system/stream_camera.service
[Unit]
Description=Camera Streaming Service
After=network.target rc-local.service

[Service]
Restart=always
RestartSec=20s
ExecStart=ffmpeg -input_format h264 -f video4linux2 -video_size 1920x1080 -framerate 30 -i /dev/video0 -c:v copy -f dash -window_size 10 -remove_at_exit 1 -hls_playlist 1 /dev/shm/streaming/manifest.mpd

[Install]
WantedBy=multi-user.target

Notice we set it to be run after rc-local to make sure we are ready for ffmpeg to write to /dev/shm/streaming.

Encoding UHD 4K HDR10 videos with FFmpeg

I talked about this before with my encoding setting for handbrake post, but there is a fundamental flaw using Handbrake for HDR 10-bit video….it only has a 8-bit internal pipeline! So while you still get a 10-bit x265 video, you are losing the HDR10 data.

Thankfully, you can avoid that and save the HDR by using FFmpeg instead. (To learn about saving Dolby Vision or HDR10+ by remuxing, skip ahead.) So cutting straight to the chase, here are the two most basic commands you need to save your juicy HDR10 data. This will use the Dolby Vision (profile 8.1) Glass Blowing demo.

Extract the Mastering Display metadata

First, we need to use FFprobe to extract the Mastering Display and Content Light Level metadata. We are going to tell it to only read the first frame’s metadata -read_intervals "%+#1" for the file GlassBlowingUHD.mp4

ffprobe -hide_banner -loglevel warning -select_streams v -print_format json -show_frames -read_intervals "%+#1" -show_entries "frame=color_space,color_primaries,color_transfer,side_data_list,pix_fmt" -i GlassBlowingUHD.mp4

A quick breakdown of what we are sending ffprobe:

-hide_banner -loglevel warning Don’t display what we don’t need
-select_streams v We only want the details for the video (v) stream
-print_format json Make it easier to parse
-read_intervals "%+#1" Only grab data from the first frame
-show_entries ... Pick only the relevant data we want
-i GlassBlowingUHD.mp4 input (-i) is our Dobly Vision demo file

That will output something like this:

{ "frames": [
        {
            "pix_fmt": "yuv420p10le",
            "color_space": "bt2020nc",
            "color_primaries": "bt2020",
            "color_transfer": "smpte2084",
            "side_data_list": [
                {
                    "side_data_type": "Mastering display metadata",
                    "red_x": "35400/50000",
                    "red_y": "14600/50000",
                    "green_x": "8500/50000",
                    "green_y": "39850/50000",
                    "blue_x": "6550/50000",
                    "blue_y": "2300/50000",
                    "white_point_x": "15635/50000",
                    "white_point_y": "16450/50000",
                    "min_luminance": "50/10000",
                    "max_luminance": "40000000/10000"
                },
                {
                    "side_data_type": "Content light level metadata",
                    "max_content": 0,
                    "max_average": 0
} ] } ] }

I chose to output it with json via the -print_format json option to make it more machine parsible, but you can omit that if you just want the text.

We are now going to take all that data, and break it down into groups of <color abbreviation>(<x>, <y>) while leaving off the right side of the \, so for example we combine red_x "35400/50000"and red_y "14600/50000" into R(35400, 14600).

G(8500,39850)B(6550,2300)R(35400, 14600)WP(15635,16450)L(50,40000000)

This data, as well as the Content light level (<max_content>,<max_average>) of (0,0) will be fed into the encoder command options.

Convert the video

This command converts only the video, keeping the HDR10 intact. We will have to pass these arguments not to ffmpeg, but to the x265 encoder directly via the -x265-params option. (If you’re not familiar with FFmpeg, don’t fret. FastFlix, which I talk about later, will do the work for you!)

ffmpeg  -i GlassBlowingUHD.mp4 -map 0 -c:v libx265 -x265-params hdr-opt=1:repeat-headers=1:colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc:master-display=G(8500,39850)B(6550,2300)R(35400,14600)WP(15635,16450)L(40000000,50):max-cll=(0,0) -crf 20 -preset veryfast GlassBlowingConverted.mkv

Let’s break down what we are throwing into the x265-params:

hdr-opt=1 we are telling it yes, we will be using HDR
repeat-headers=1we want these headers on every frame as required
colorprim, transfer and colormatrix the same as ffprobe listed
master-display this is where we add our color string from above
max-cll Content light level data, in our case (0,0)

During a conversion like this, when a Dolby Vision layer exists, you will see a lot of messages like [hevc @ 000001f93ece2e00] Skipping NAL unit 62 because there is an entire layer that ffmpeg does not yet know how to decode.

For the quality of the conversion, I was setting it to -crf 20 with a -preset veryfast to convert it quickly without a lot of quality loss. I dig deeper into how FFmpeg handles crf vs preset with regards to quality below.

All sound and data and everything will be copied over thanks to the -map 0 option, that is a blanket statement of “copy everything from the first (0 start index) input stream”.

That is really all you need to know for the basics of how to encode your video and save the HDR10 data!

FFmpeg conversion settings

I covered this a bit before in the other post, but I wanted to go through the full gauntlet of presets and crfs one might feel inclined to use. I compared each encoding with the original using VMAF and SSIM calculations over a 11 second clip. Then, I created over a 100 conversions for this single chart, so it is a little cramped:

First takeaways are that there is no real difference between veryslow and slower, nor between veryfast and faster, as their lines are drawn on top of each other. The same is true for both VMAF and SSIM scores.

Second, no one in their right mind would ever keep a recording stored by using ultrafast. That is purely for real time streaming use.

Now for VMAF scores, 5~6 points away from source is visually distinguishable when watching. In other words it will have very noticeable artifacts. Personally I can tell on my screen with just a single digit difference, and some people are even more sensitive, so this is by no means an exact tell all. At minimum, lets zoom in a bit and get rid of anything that will produce video with very noticeable artifacts.

From these chart, it seems clear that there is obviously no reason whatsoever to ever use anything other than slow. Which I personally do for anything I am encoding. However, slow lives up to its namesake.

Encoding Speed and Bitrate

I had to trim off veryslow and slower from the main chart to be able to even see the rest, and slow is still almost three times slower than medium. All the charts contain the same data, just with some of the longer running presets removed from each to better see details of the faster presets.

Please note, the first three crf datapoints are little dirty, as the system was in use for the first three tests. However, there is enough clean data to see how that compares down the line.

To see a clearer picture of how long it takes for each of the presets, I will exclude those first three times, and average the remaining data. The data is then compared against the medium (default) present and the original clip length of eleven seconds.

Preset	Time	vs “medium”	vs clip length (11s)
ultrafast	11.204	3.370x	0.982x
superfast	12.175	3.101x	0.903x
veryfast	19.139	1.973x	0.575x
faster	19.169	1.970x	0.574x
fast	22.792	1.657x	0.482x
medium	37.764	1.000x	0.291x
slow	97.755	0.386x	0.112x
slower	315.900	0.120x	0.035x
veryslow	574.580	0.066x	0.019x

What is a little scary here is that even with “ultrafast” preset we are not able to get realtime conversion, and these tests were run on a fairly high powered system wielding an i9-9900k! While it might be clear from the crf graph that slow is the clear winner, unless you have a beefy computer, it may be a non-option.

Use the slowest preset that you have patience for
FFmpeg encoding guide

Also unlike VBR encoding, the average bitrate and filesize using crf will wildly differ based upon different source material. This next chart is just showing off the basic curve effect you will see, however it cannot be compared to what you may expect to see with your file.

The two big jumps are between slow and medium as well as veryfast and superfast. That is interesting because while slow and medium are quite far apart on the VMAF comparison, veryfast and superfast are not. I expected a much larger dip from superfast to ultrafast but was wrong.

FastFlix, doing the heavy lifting for you!

I have written a GUI program, FastFlix, around FFmpeg and other tools to convert videos easily to HEVC, AV1 and other formats. While I won’t promise it will provide everything you are looking for, it will do the work for you of extracting the HDR details of a video and passing them into a FFmpeg command. It also has a panel that shows you exactly the command(s) it is about to run, so you could copy it and modify it to your hearts content!

Currently I only automatically produce windows builds, as it was originally designed for SVT-AV1 which also only had Windows builds, but it should work on other systems by cloning the git repo and running via python.

If you have any problems with it please help by raising an issue!

Saving Dolby Vision or HDR10+

Note I say “saving” and not “converting”, because unless you have the original RPU file for the DV to pass to x265, you’re out of luck as of now and cannot convert the video. (If you or anyone you know figured out a way to parse / use baked in RPU info, let me know!) However, the x265 encoder is able to take a RPU file and create a Dolby Vision ready movie, but we don’t have anything to extract that (yet).

Thankfully, it is possible to at least convert the audio and change around the streams with remuxers. For example tsMuxeR (nightly build, not default download) is popular to be able to take mkv files that most TVs won’t recognize HDR in, and remux them into into ts files so they do. If you also have TrueHD sound tracks, you may need to use eac3to first to break it into the TrueHD and AC3 Core tracks before muxing.

Easily Viewing HDR / Video information

Another helpful program to quickly view what type of HDR a video has is MediaInfo. For example here is the original Dolby Vision Glass Blowing video info (some trimmed):

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5.1@Main
HDR format                               : Dolby Vision, Version 1.0, dvhe.08.09, BL+RPU, HDR10 compatible / SMPTE ST 2086, HDR10 compatible
Codec ID                                 : hev1
Color space                              : YUV
Chroma subsampling                       : 4:2:0 (Type 2)
Bit depth                                : 10 bits
Color range                              : Limited
Color primaries                          : BT.2020
Transfer characteristics                 : PQ
Matrix coefficients                      : BT.2020 non-constant
Mastering display color primaries        : BT.2020
Mastering display luminance              : min: 0.0050 cd/m2, max: 4000 cd/m2
Codec configuration box                  : hvcC+dvvC

And here it is after conversion:

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5.1@Main
HDR format                               : SMPTE ST 2086, HDR10 compatible
Codec ID                                 : V_MPEGH/ISO/HEVC
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 10 bits
Color range                              : Limited
Color primaries                          : BT.2020
Transfer characteristics                 : PQ
Matrix coefficients                      : BT.2020 non-constant
Mastering display color primaries        : BT.2020
Mastering display luminance              : min: 0.0050 cd/m2, max: 4000 cd/m2

Notice we have lost the Dolby Vision, BL+RPU information but at least we retained the HDR10 data, which Handbrake can’t do!

That’s a wrap!

Hope you found this information useful, and please feel free to leave a comment for feedback, suggestions or questions!

Until next time, stay safe and love each other!

Let it Djan-go!

Posted on June 17, 2020 by Chris Griffith Coding Opinion 0

Do you know someone who hails from the strange land of Django development? Where there are code based settings files, is customary to use a ORM for all database calls, and classes reign supreme. It’s a dangerous place that sits perilously close to the Javalands. Most who originally traveled there were told there would be riches, but they lost themselves along the way and simply need our help to come home again.

Classless classes

The first bad habit that must be broken is the horrendous overuse of classes. While classes are considered the building blocks of most object oriented languages, Python has the power of first-class functions and a clean namespacing system which negates many the usual reasons for classes.

When using Django’s framework, where a lot of its powerful operations are performed by subclassing, classes actually makes sense. When creating a small class for a one use instance in a library or script it’s most likely simply over complicating and slowing down your code.

Let’s use an example I have seen recently, where someone wanted to run a command, but needed to know if a custom shell wrapper existed before running the command.

import subprocess
import os

class DoIt:

    def __init__(self, command):
        self.command = command
        self.shell = "/bin/secure_shell" if os.path.exists("/bin/secure_shell") else "/bin/bash"

    def execute(self):
        return subprocess.run([self.shell, "-c", self.command], stdout=subprocess.PIPE)

runner = DoIt('echo "order 66" ')
runner.execute()

Most of this class is setup and unnecessary. Especially if the state of the system won’t change while the script is running (i.e. if “/bin/secure_shell” is added or removed during execution). Let’s clean this up a little.

import subprocess
import os

shell = "/bin/secure_shell" if os.path.exists("/bin/secure_shell") else "/bin/bash"


def do_it(command):
    return subprocess.run([shell, "-c", command], stdout=subprocess.PIPE)

do_it('echo "order 66" ')

Shorter, cleaner, faster, easier to read. It doesn’t get much better than that. Now if the system state may change, you would need to throw the shell check inside the command itself, but is also possible.

To learn more view the great PyCon video entitled “stop writing classes.” The video goes over more details of why classes can stink up your code.

Help me ORM-Kenobi!

Now onto probably the hardest change that will have the most significant impact. If you or someone you know has ~~mesothelioma~~ learned how to interact databases by using an ORM (Django’s Models) and don’t know how to hand write SQL queries, it’s time to make an effort to learn.

An ORM is always going to be slower than its raw SQL equivalent, and in a many cases it will be much much slower. ORMs are brilliant, but still can’t always optimize queries. Even if they do, they then load the retrieved data into Python objects which takes longer and uses more memory. Their abstraction also comes at the price of not always being clear on what is actually happening under the hood.

I am not suggesting never using an ORM. They have a lot of benefits for simple datasets such as ease of development, cross database compatibility, custom field transforms and so on. However if you don’t know SQL to begin with, your ORM will most likely be extremely inefficient, under-powered and the SQL tables will be non-normalized.

For example, one of the worst sins in SQL is to use SELECT * in production. Which is the default behavior for ORM queries. In fact, the Django main query page doesn’t even go over how to limit columns returned!

Now, I am not going to tell when to use an ORM or not, or link to a dozen other articles that fight over this point further. All I want is for everyone that thinks they should use an ORM to at least sit down and learn a moderate amount of SQL before making that decision.

Text Config is King

When was the last time you had to convince an IT admin to update a source code file? Yeah, good luck with that.

By separate settings from code you gain a lot of flexibility and safety. It’s the difference between “Mr. CEO, you need the default from email changed? So for our Django docker deploy we’ll need to make the change in code, commit it to the feature branch, PR it to develop, pray we don’t have other changes and require a hotfix for this instead, then PR again for release to master, wait for the CI/CD to finish and then the auto-deployment should be good by end of the day.” vs. “One sec, let me update the config file and restart it. Okay, good to go.”

“Code based setting files are more powerful!” – Old Sith Proverb

That’s the path to the dark side. If you think you need programmatic help in a configuration file, your logic is in the wrong place. A config file shouldn’t validate itself, not have if statements, nor have to find the absolute path to the relative directory structured the user entered.

The best configuration handling I have seen are text files that are read, sanity checked, loaded into memory and have paths extrapolated as needed, run any necessary safety checks, then any additional logic run. This allows almost anyone to update a setting safely and quickly.

The scary Django way

Lets take a chunk from the “one django settings.py file to rule them all”

import os 
import json 

DEBUG = os.environ.get("APP_DEBUG") == "1"

ALLOWED_HOSTS = os.environ \
  .get("APP_ALLOWED_HOSTS", "localhost") \
  .split(",")

DEFAULT_DATABASES = {...}
DATABASES = (
  json.loads(os.environ["APP_DATABASES"])
  if "APP_DATABASES" in os.environ
  else DEFAULT_DATABASES
)

The basic design is to allow for overrides on runtime based on environment variables, which is very common and good practice. However, the way this example has been implemented honestly scares me. If the json loads fails, you are going to have an exception raised, in your freaking settings file. Which may even happen before logging is setup, project dependent.

Second, say you have “APP_ALLOWED_HOSTS” (or worse “APP_DATABASES”) set to an empty string accidentally. Then the “ALLOWED_HOSTS” would be set to [""] !

Finally spending all that work to grab environment variables makes it very unclear where to actually update the settings. For example, to update the ALLOWED_HOSTS you would have to change the default in the os.environ.get which wouldn’t make much sense to someone who didn’t know Python.

The safe way

Why not a much more easily readable file, in this case a YAML file paired with python-box (lots of other options as well, this was simply fast to write and easy to digest.)

# config.yaml
---
debug: false
allowed_hosts:
  - 'localhost'
databases:
  - 'sqlite3'

Wonderful, now we have a super easy to digest config file that would be easy to update. Let’s implement the parser for it.

import os
from box import Box

config = Box.from_yaml(filename="config.yaml")

config.debug |= os.getenv("DEBUG") == "1"

# Doing it this way will make sure it exists and is not empty
if os.getenv("ALLOWED_HOSTS"):
    # This is overwritten like in settings.py above instead of extended 
    config.allowed_hosts = os.getenv("ALLOWED_HOSTS").split(",")

if os.getenv("APP_DATABASES"):
    try:
        databases = Box.from_json(filename=os.environ["APP_DATABASES"])
    except Exception as err:
        # Can add proper logging or print message as well / instead of exception
        raise Exception(f"Could not load database set from {os.environ['APP_DATABASES']}: {err}")
    else:
        config.databases.extend(databases)

Tada! Now this is a lot easier to understand the logic that is happening, much easier to update the config file, as well as a crazy amount safer.

Now the only thing to argue about with your coworkers is what type of config file to use. cfg is old-school, json doesn’t support comments, yaml is slow, toml is fake ini, hocon is overcomplicated, and you should probably just stop coding if you’re even considering xml. But no matter which you go with, they are all better (except xml) than using a language based settings file.

Wrap Up

So you’ve heard me prattle on and on of the dangers Djangoers might bring with them to your next project, but also keep in mind the good things they will bring. Django helps people learn how to document well, keep a rigid project structure and has a very warm and supportive community that we should all strive to match.

Thanks for reading, and I hope you enjoyed and found this informative or at least amusing!

Passing the CKAD (Certified Kubernetes Administrator)

Posted on June 2, 2020 by Sam Griffith Certification DevOps 0

June 2^nd, 2020 written by Sam Griffith

Originally Published on alta3.com

I was the first person to become a Certified Kubernetes Application Developer at my company, so I naturally have received a lot of questions from my coworkers about it and how to pass it.

Of course, I told them you need to practice. And you need to have a fairly good idea about how different resources in Kubernetes work together. You also should repeat all of the hands-on exercises that I helped to create until you can do it all from muscle memory.

But trying to do all of the required tasks for the exam in two hours is hard. Especially if you have not been using Kubernetes for a long time. So, if your goal is to pass the CKAD exam, here is my suggestion:

Use Your Resources!!!

The CKAD exam is an open-book test. Although you are not allowed anything on or near your desk and no physical books are allowed, you are allowed one tab open to the test and one tab open to the Official Kubernetes Documentation page.

Using the documentation is a good way to look up anything that you have forgotten, like how to use a ConfigMap inside of a Pod, or how much to indent a field in your manifest.

But here is an even better plan:

Know what you are expected to know, and set your browser up for success with the Kubernetes Documentation.

So first question is:

What topics are you expected to know?

Thankfully, the Cloud Native Computing Foundation (CNCF) has put together an outline of “Knowledge, Skills, and Abilities that a Certified Kubernetes Application Developer (CKAD) can be expected to demonstrate.” If you want to see it on their GitHub page, take a look at this link:

CKAD Curriculum v1.18

The information provided in this outline is a good starting point. But it does not show you where to go to find this information.

Links to Use During the CKAD exam

Below are all of the categories and specific topics that you will be tested on during the CKAD exam, as well as links to the official documentation that you can use during the exam.

These are also excellent links to read through and practice with as you prepare for exam day.

13% – Core Concepts

You need to understand what the core components of a Kubernetes cluster are and how they interact with each other.

Understand Kubernetes API Primitives https://kubernetes.io/docs/concepts/overview/components/
Create and configure basic Pods https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/#pod-templates

10% – Multi-Container Pods

You need to be capable of altering a Pod manifest to include multiple containers, which then can interact with each other.

Understand Multi-Container Pod design patterns (e.g. ambassador, adapter, sidecar) https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns/

20% – Pod Design

You need to be able to comfortably use Labels and Selectors inside of your Pods in order to get Deployments, Jobs, and other resources to work with your Pods.

Understand Deployments and how to perform rolling updates https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#updating-a-deployment
Understand Deployments and how to perform rollbacks https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-back-a-deployment
Understand Jobs and CronJobs https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#running-an-example-job & https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cronjob
Understand how to use Labels, Selectors, and Annotations https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set & https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set

8% – State Persistence

You need to be capable of configuring a Pod to use a PersistentVolumeClaim to call dibs on some storage (a PersistentVolume).

Understand PersistentVolumeClaims for storage https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims

18% – Configuration

You need to be able to alter the basic Pod configuration to fit your cluster’s specifications for security and resource usage.

Understand ConfigMaps https://kubernetes.io/docs/concepts/configuration/configmap/#configmaps-and-pods
Understand SecurityContexts https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
Define an application’s resource requirements https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory
Create & Consume Secrets https://kubernetes.io/docs/concepts/configuration/secret/#creating-your-own-secrets
Understand ServiceAccounts https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server

18% – Observability

You need to be able to troubleshoot Pods and other resources using data that you are able to gather and inspect.

Understand LivenessProbes and ReadinessProbes https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-liveness-http-request & https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes
Understand container logging https://kubernetes.io/docs/concepts/cluster-administration/logging/#basic-logging-in-kubernetes
Understand how to monitor applications in Kubernetes https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/#resource-metrics-pipeline
Understand debugging in Kubernetes https://kubernetes.io/docs/tasks/debug-application-cluster/

13% – Services & Networking

You need to be able to use a NetworkPolicy to allow certain types of communication between Pods and other sources.

Understand Services https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service
Demonstrate basic understanding of NetworkPolicies https://kubernetes.io/docs/concepts/services-networking/network-policies/#networkpolicy-resource

My final tip: Bookmark these links!

Instead of searching through the Kubernetes documentation to try to find these topics, do yourself a favor: remove all of those old bookmarks you haven’t looked at in six months and prepare your browser by bookmarking every single one of the links above and any other Kubernetes Documentation that you find useful.

Step 1 Download Alta3’s CKAD BookmarksNote: This is really an HTML file. But browsers think they have to display all html files, so in order to download the file, it has to have a non .html file extension, hence the .download. If you would prefer to use your browser to view the original html, you can click here.
Step 2 Rename the fileRemove the .download file extension from the file name.
Step 3 Import ALL THE BOOKMARKS YOU NEED TO PASS THE CKAD

Once you do that, you are essentially walking in to an open-book test with your book in hand and all of the answers marked.

Good luck on your exam!

How NOT to call Robinhood’s secret API with Python

Posted on February 11, 2020 by Chris Griffith Coding Opinion 0

This article is meant to go over the basics of calling an API via Python, as well as a critique of a top google result I recently ran across. We will start by using Robinhood’s undocumented API, as I saw this article that had python code so bad I had to try and fix it.

Also, because I can get us each a free stock if you sign up with this Robinhood referral link (seems to be usually around a $10 stock, but they like to advertise you might get Facebook or Visa.)

So at the base of the bad example (non-working call), he was using curl calls like:

curl -v https://api.robinhood.com/quotes/XIV/ -H "Accept: application/json"

Which is a great use of curl. However, then the author simply decided to wrap it up in a subprocess call and manually parsed the output. He then states “You could also use PYCurl but I didn’t feel like learning it.” Well, thankfully there is no need to learn it.

How to interact with a REST API, the basic GET and POST

The most common text based web APIs are JSON REST-like nowadays. I add the “like” because there are usually cases a company does not fully stick to the standard, and to be honest is a good decision a lot of the time. Thankfully that doesn’t usually change the behavior of the most simple calls of GET and POST.

A properly implemented GET call on the server’s side won’t change anything on their server. So they are best to practice with. It is possible to do this all with pure python, however the requests library is so well known and has exactly what we want already, so we will stick to that library for the examples. To install it, simply pip install requests.

Example `GET` call

import requests

response = requests.get('https://api.robinhood.com')
# Check to make sure we got 'good' response, aka in HTTP code 2XX range
if response.ok: 
    # Display the result of the parsed json
    print(response.json())

This should simply print out {}.

Next I thought it would be a good idea to try and log in using that article’s example login. If you don’t have an account (not necessary), you can sign up here which uses a referral link, gives you and me both a free random stock.

Example (Non-working) `POST` call

import requests

response = requests.post('https://api.robinhood.com/api-token-auth/', 
                         json={'username': YOUR_USERNAME, 'password': YOUR_PASSWORD})
if response.ok:
    print(response.json())
else:
    print(response.status_code)
    print(response.text)

Now wait a minute, we are getting an error.

404
<h1>Not Found</h1><p>The requested resource was not found on this server.</p>

Oh gosh darn it. Not only did the author have bad python code, they have outdated examples! How dare they! (*quickly scans my most recent articles for anything obviously outdated*)

Now we get a quick lesson on the dangers of using undocumented APIs. They have absolutely no guarantees. Anything can change at anytime or even remove access to them. In this case, some people reversed engineered how to login to robinhood using a different endpoint, and have an unofficial API wrapped around it.

So for now, lets use Postman’s excellent echo API that will return what you send it. This time, lets put it in a more reusable function.

Example (good) `POST` call

import requests

api_root = 'https://postman-echo.com'

def post(path='/post', payload=None, timeout=10, **request_args):
    response = requests.post(f'{api_root}/{path}', 
                   json=payload, timeout=timeout, **request_args)
    if response.ok:
            return response.json()
    raise Exception(f'error while calling "{path}", returned status "{response.status_code}" with message: "{response.text}"')

print(post(payload={'username': 'chris'}))

The endpoint will return a JSON object, and requests will automatically translate that into a python dictionary when called with .json() on the response.

{'args': {}, 'data': {'username': 'chris'}, 'files': {}, 'form': {},  'json': {'username': 'chris'}, 'url': 'https://postman-echo.com/post'}

Lot of fluff, but it is nice that Postman’s API spells out exactly what that endpoint receives. For example, you can test sending form data by changing the post parameter call from json to data.

Now this is doing a better job of checking response and using much more pythonic methods. But let’s take it one step further and do better error catching. Because if this is something you are putting into a bigger program, you probably have custom exceptions and want to be able to catch them in a certain way.

This is going to be a little exception heavy, but is showcasing all the ways you might need to deal with certain errors. I would probably only have one wrapper around the call and the parsing JSON personally, but it is really program dependent.

import requests

api_root = 'https://postman-echo.com'

class MyError(Exception):
    """Exception for anything in my program"""

class MyConnectionIssue(MyError):
    """child exception for issues with connections"""


def post(path='/post', payload=None, timeout=10, **request_args):
    try:
        response = requests.post(f'{api_root}/{path}', json=payload, timeout=timeout, **request_args)
    except requests.Timeout:
        raise MyConnectionIssue(f'Timeout of {timeout} seconds reached while calling "{path}"') from None
    except requests.RequestException as err:
        raise MyConnectionIssue(f'Error while calling {path}: {err}')
    else:
        if response.ok:
            try:
                return response.json()
            except ValueError:
                raise MyError(f'Expected output from {path} was not in JSON format. Output: {response.text}')
        raise MyError(f'error while calling "{path}", returned status "{response.status_code}" with message: "{response.text}"')

print(post(payload={'username': 'chris'}))

If you’re a little rusty on exceptions, feel free to brush up on them here.

How NOT to parse JSON

In the terrible code below, we see the other article’s code manually parsing JSON output. Please for the love of all that is holy do not copy!

import subprocess

class data():
    def __init__(self, stock):
            parameter_list = ['open', 'high', 'low', 'volume', 'average_volume', 'last_trade_price', 'previous_close']

            # replacing CURL call
            out = '{"open": 54 "high": 55 "low": 52 "volume": 6000 "average_volume": 6000 "last_trade_price": 52 "previous_close": 54 "}'
            # back to copied code

            string = out  
            for p in parameter_list:  
                parameter = p  
                if parameter in string:  
                    x = 0  
                    output = ''  
                    iteration = 0  
                    for i in string:  
                        if i != parameter[x]:  
                            x = 0  
                        if i == parameter[x]:  
                            x = x+1  
                        if x == len(parameter):  
                            eowPosition = iteration  
                            break  
                        iteration = iteration + 1

                    target_position = eowPosition + 4
                    for i in string[target_position:]:  
                        if i == '"':  
                            break  
                        elif i == 'u':
                            output = 'NULL'  
                            break  
                        else:  
                            output = output+i

                    if output != 'NULL':  
                        output = float(output)  
                    if p == 'open':  
                        self.open = output  
                    if p == 'high':  
                        self.high = output  
                    if p == 'low':  
                        self.low = output  
                    if p == 'volume':  
                        self.volume = output  
                    if p == 'average_volume':  
                        self.average_volume = output  
                    if p == 'last_trade_price':  
                        self.current = output  
                    if p == 'previous_close':  
                        self.close = output  
XIV = data('XIV')

print('Current price: ', XIV.current)

So let’s do a critique starting from the top. class data():
First, classes should be CamelCase, second, this is a single function, shouldn’t even be a class. However does make a tiny bit of sense considering they are basically using it as a namespace, but there are better ways.

Next is them using subprocess to do a curl command instead of the built in urllib or requests. (Omitted so this code runs.)

Then onto string = out which is just…why? No reason to copy it to a new variable. Just name it what you want in the first place.

Finally the for loop that actually parses the JSON. Which is just painful to look at. If you ever need to parse JSON, without using requests method, just use the built-in json library!

import json

data = json.loads('{"open": 54}')
print(data["open"]) 
# 54

Bam, 42 lines into 3.

Finishing on a high note

Now for as much grief as I am giving this author, I also respect him a lot. He was able to accomplish what he wanted to achieve using Python, a language he had little experience with. To his credit he was clear up front that “I’m not an expert in stock trading nor coding so I can’t say the code is the cleanest”.

He also was willing to write up an article on how to do it for others, which most people don’t dream of doing, so in all earnestness, good job Spencer!

Last mention, free stock (legit actual money) from robinhood, just use this referral!

MPEG-DASH vs HLS vs RSTP

Setting up required software

File Server

FFmpeg

Gather camera details

Create the FFmpeg command

Applying the Camera Details

Network Bandwidth Considerations

Conversion options

MPEG-DASH and HLS output

Putting it all together

Setting up Remote Viewing

Reboot Script

Camera Streaming Service

Share this:

Extract the Mastering Display metadata

Convert the video

FFmpeg conversion settings

Encoding Speed and Bitrate

FastFlix, doing the heavy lifting for you!

Saving Dolby Vision or HDR10+

Easily Viewing HDR / Video information

That’s a wrap!

Share this:

Classless classes

Help me ORM-Kenobi!

Text Config is King

The scary Django way

The safe way

Wrap Up

Share this:

Use Your Resources!!!

Links to Use During the CKAD exam

13% – Core Concepts

10% – Multi-Container Pods

20% – Pod Design

8% – State Persistence

18% – Configuration

18% – Observability

13% – Services & Networking

My final tip: Bookmark these links!

Share this:

How to interact with a REST API, the basic GET and POST

Example GET call

Example (Non-working) POST call

Example (good) POST call

How NOT to parse JSON

Finishing on a high note

Share this:

Search

Recent Posts

Categories

Tags

Staying Legal

Example `GET` call

Example (Non-working) `POST` call

Example (good) `POST` call