Coding

Everything related to code, from programming to review and usage.

Raspberry Pi Hardware Accelerated RTSP Camera

Raspberry Pi’s are wonderful little computers, just sometimes they lack the umph to get stuff done. That may change with the new Raspberry Pi 4, but what to do with all those old ones? Or how about that pile of old webcams? Well this article will help turn all those into a full on security system. (Can also use a raspberry pi camera if you got one!)

Other posts I have read on this subject often only use motion to capture detection events locally. Sometimes they go a bit further and set the Raspberry PI to stream MJPEG as an IP camera. Or set up MotionEyeOS and make it into a singular video surveillance system.

With our IP camera, we are going to take it further and encode the video stream locally. Then we will send it over the network via rtsp. This will save huge amounts of bandwidth! It also does not require the client to re-encode the stream before saving, distributing the work. That way we can also hook it into a larger security suite without draining any of its resources, in this case I will use Blue Iris.

Now, the first thing I am going to do is discourage you. If you don’t already have a Pi and a webcam or pi camera for the cause, don’t run out to buy them just for this. It’s just not economical. A 1080p WiFi camera that has has ONVIF capabilities can be had for less than $50. So why do this at all? Well because A.) It’s all under your control and no worry about Chinaware, 2.) If you already got the equipment, it’s another free security eye, and 𓀀) It’s fun.

Standard Raspbian setup

Not going to go into too much detail here. If you haven’t already, download Raspbian and get it onto a SD Card. (I used raspbian buster for this tutorial) If you aren’t going to connect a display and keyboard to it, make sure to add an empty file named ssh on the root of the boot (SD Card) drive. That way you can just SSH to the raspberry pi via command line or PuTTY on Windows.

# Default settings
host: raspberrypi 
username: pi
password: raspberry

Remember to run sudo raspi-config, change your password and don’t forget to set up wifi, then reboot. Also, good idea to update the system before continuing.

sudo apt update --fix-missing
sudo apt upgrade -y 
sudo reboot 

Install node rtsp server

To start with, we need a place for ffmpeg to connect to for the rtsp connection. Most security systems expect to connect to a rstp server, instead of listening as a server themselves, so we need a middleman.

There are a lot of rstp server options out there, I wanted to go with a lightweight one we can just run on the pi itself that is easy to install and run easily. This is what I run at my own house, so don’t think I’m skimping out for this post 😉

First of, we need to install Node JS. The easiest way I have found is to use the pre-created scripts to add the proper package links to the apt system for us.

If you are on an arm6 based system, such as the pi zero, you should stick to Node 10, and is safe to do so for arm7 based systems as well, such as the pi 3, so that is the default version we are going to go with. However, arm7 also has Node 12 builds if you want a newer Node version. Find out your arm version with uname -a command and seeing if the string “arm6” or “arm7” appears.

Now, lets install Node JS and other needed libraries, such as git and coffeescript. If you want to view the script itself before running it, it is available to view here.

# Note, if you are using an arm7 system, you can use setup_12.x instead
# Run "uname -a" to find out if your system is arm6 or arm7 

curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -

sudo apt-get install -y nodejs git
sudo npm install -g coffeescript

Once that is complete, we want to download the node rtsp server code and install all it’s dependencies. Note, I am assuming you are doing this in the root of your home folder, which will later use as the base for the directory for the service.

cd ~ 
git clone https://github.com/iizukanao/node-rtsp-rtmp-server.git --depth 1
cd node-rtsp-rtmp-server
npm install -d

Now you should be good to go, you can test it out by running:

sudo coffee server.coffee

It takes about 30 seconds or more to start up, so give it minute before you will see any text. At this point you can also go into the server.coffee file and edit it to your hearts content, however I do keep it standard myself.

Create rtsp server service

You probably want this to always start this on boot, so lets add it as a systemd service. Copy and paste the following code into /etc/systemd/system/rtsp_server.service.

# /etc/systemd/system/rtsp_server.service

[Unit]
Description=rtsp_server
After=network.target rc-local.service

[Service]
Restart=always
WorkingDirectory=/home/pi/node-rtsp-rtmp-server
ExecStart=coffee server.coffee

[Install]
WantedBy=multi-user.target

Now we can start it up via the service, and enable it to start on boot.

sudo systemctl start rtsp_server
# Can make sure it works with sudo systemctl status rtsp_server
sudo systemctl enable rtsp_server

Compile FFMPEG with Hardware Acceleration

If you are just using the raspberry pi camera, or another one with h264 or h265 built in support, you can use the distribution version of ffmpeg instead.

This is going to take a while to make. I suggest reading a good blog post or watching some Red vs Blue while it builds. Again, this guide is just small modifications from another one. We are also adding libfreetype font package so we can add text (like a datetime) to the video stream, as well as the default libx264 so that we can use it with the Pi Camera if you have one.

sudo apt-get install libomxil-bellagio-dev libfreetype6-dev libmp3lame-dev \
 checkinstall libx264-dev fonts-freefont-ttf libasound2-dev -y
cd ~
git clone https://github.com/FFmpeg/FFmpeg.git --depth 1
cd FFmpeg
sudo ./configure --arch=armel --target-os=linux --enable-gpl --enable-omx \
 --enable-omx-rpi --enable-nonfree --enable-libfreetype --enable-libx264 \
 --enable-libmp3lame --enable-mmal --enable-indev=alsa --enable-outdev=alsa

# For old hardware / Pi zero remove the `-j4` 
sudo make -j4

When that is finally done, run the steps below that will install it. We take the additional precaution of turning it into a standard system package and hold it so we don’t overwrite our ffmpeg version.

sudo checkinstall --pkgname=ffmpeg -y 
sudo apt-mark hold ffmpeg 
echo "ffmpeg hold" | sudo dpkg --set-selections

Figure out your camera details

If you haven’t already, plug the webcam into the raspberry pi. Then we are going to use video4linux2 to discover what it’s capable of.

v4l2-ctl --list-devices

Mine lists my webcam and two paths it’s located at. Sometimes a camera will have multiple devices for different types of formats it supports, so it’s a good idea to check each one out.

Microsoft® LifeCam Cinema(TM): (usb-3f980000.usb-1.2):
        /dev/video0
        /dev/video1

Now we need to see what resolutions and FPS it can handle. Be warned MJPEG streams are much more taxing to encode them some of their counterparts. In this example we are going to specifically try to find YUYV 4:2:2 streams, as they are a lot easier to encode. (Unless you see h264, then use that!)

In my small testing group, MJPEG streams averaged only 70% of the FPS of the YUYV, while running the CPU up to 60%. Comparatively, YUYV encoding only took 20% of the CPU usage on average.

v4l2-ctl -d /dev/video0 --list-formats-ext

This pumps out a lot of info. Basically you want to find the subset under YUYV and figure out which resolution and fps you want. Here is an example of some of the ones my webcam supports.

ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'YUYV' (YUYV 4:2:2)
                Size: Discrete 640x480
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.050s (20.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.100s (10.000 fps)
                        Interval: Discrete 0.133s (7.500 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.100s (10.000 fps)
                        Interval: Discrete 0.133s (7.500 fps)
                Size: Discrete 960x544
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.100s (10.000 fps)
                        Interval: Discrete 0.133s (7.500 fps)

I am going to be using the max resolution of 1280×720 and the highest fps of 10. Now if it looks perfect as is, you can skip to the next section. Though if you need to tweak the brightness, contrast or other camera settings, read on.

Image tweaks

Let’s figure out what settings we can play with on the camera.

v4l2-ctl -d /dev/video0 --all
brightness (int)                : min=30 max=255 step=1 default=-8193 value=135
contrast (int)                  : min=0 max=10 step=1 default=57343 value=5
saturation (int)                : min=0 max=200 step=1 default=57343 value=100
power_line_frequency (menu)     : min=0 max=2 default=2 value=2
sharpness (int)                 : min=0 max=50 step=1 default=57343 value=27
backlight_compensation (int)    : min=0 max=10 step=1 default=57343 value=0
exposure_auto (menu)            : min=0 max=3 default=0 value=3
exposure_absolute (int)         : min=5 max=20000 step=1 default=156 value=156
pan_absolute (int)              : min=-201600 max=201600 step=3600 default=0 
tilt_absolute (int)             : min=-201600 max=201600 step=3600 default=0 
focus_absolute (int)            : min=0 max=40 step=1 default=57343 value=12
focus_auto (bool)               : default=0 value=0
zoom_absolute (int)             : min=0 max=10 step=1 default=57343 value=0

Plenty of options, excellent. Now, if you don’t have a method to look at the camera display just yet, come back to this part after you have the live stream going. You can change these settings while it is going thankfully.

v4l2-ctl -d /dev/video0 --set-ctrl <setting>=<value>

The main problems I had with my camera was that it was a little dark and liked to auto-focus every 5~10 seconds. So I added the following lines of code to my rc.local file, but there are various way to run commands on startup.

#  I added these lines right before the exit 0

# dirty hack to make sure v4l2 has time to initialize the cameras
sleep 10 

v4l2-ctl -d /dev/video0 --set-ctrl focus_auto=0
v4l2-ctl -d /dev/video0 --set-ctrl focus_absolute=12
v4l2-ctl -d /dev/video0 --set-ctrl brightness=135

Now onto the fun stuff!

Real Time Encoding

Now we are going to use hardware accelerated ffmpeg library h264_omx to encode the webcam stream. That is, unless you happen to already be using a camera that supports h264 already. Like the built-in raspberry pi camera. If you are lucky enough to have one, you can just copy the output directly to the rtsp stream.

# Only for cameras that support h264 natively!
ffmpeg -input_format h264 -f video4linux2 -video_size 1920x1080 -framerate 30 \
 -i /dev/video0 -c:v copy -an -f rtsp rtsp://localhost:80/live/stream

If at any point you receive the error ioctl(VIDIOC_STREAMON) failure : 1, Operation not permitted, go into raspi-config and up the video memory (memory split) to 256 and reboot.

In the code below, make sure to change the -s 1280x720 to your video resolution (can also use -video_size instead of -s) and both -r 10 occurrences to your frame rate (can also use -framerate).

ffmpeg -input_format yuyv422 -f video4linux2 -s 1280x720 -r 10 -i /dev/video0 \
 -c:v h264_omx -r 10 -b:v 2M -an -f rtsp rtsp://localhost:80/live/stream

So lets brake this down. The first part is telling ffmpeg what to expect from your device, -i /dev/video0. Which means all those arguments must go before the declaration of the device itself.

-input_format yuyv422 -f video4linux2 -s <your resolution> -r <your framerate>

We are making clear that we only want the yuyv format as it is best available for my two cameras, yours may be different. Then specifying what resolution and fps we want it at. Be warned, that if you set one of them wrong, it may seem like it works (still encodes) but will give an error message to look out for:

[video4linux2] The V4L2 driver changed the video from 1280x8000 to 1280x800
[video4linux2] The driver changed the time per frame from 1/30 to 1/10

The next section is our conversion parameters.

-c:v h264_omx -r <your framerate> -b:v 2M

Here -c:v h264_omx we are saying the video codex to use h264, with the special omx hardware encoder. We are then telling it what the frame rate will be out as well, -r 10, and specifying the quality with -b:v 2M (aka bitrate) which determines how much bandwidth will be used when transmitting the video. Play around with different settings like -b:v 500k to see where you want it to be at. You will need a higher bitrate for higher resolution and framerate, and a lot less for lower resolution.

After that, we are telling it to disable audio with -an for the moment. If you do want audio, there is an optional section below going over how to enable that.

-f rtsp rtsp://localhost:80/live/stream

Finally we are telling it where to send the video, and to send it in the expected rtsp format (rstp is the video wrapper format, the video itself is still mp4). Notice that with the rstp server we can have as many camera with their own sub url, so instead of live/stream at the end could be live/camera1 and live/camera2.

Adding audio

Optional, not included in my final service script

As most webcams have built-in microphones, it makes it easy to add it to our stream if you want. First we need to identify our audio device.

arecord -l

You should get a list of possible devices back, in this case only my webcam is showing up as expected. If you have more than one, make sure you check out the ffmpeg article on “surviving the reboot” so they don’t get randomly re-ordered.

**** List of CAPTURE Hardware Devices ****
card 1: CinemaTM [Microsoft® LifeCam Cinema(TM)], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0

Notice it says card 1 at the very being of the webcam, and specifically device 0, that is the ID we are going to use to reference it with ffmpeg. I’m going to show the full command first like before and break it down again.

ffmpeg -input_format yuyv422 -f video4linux2 -s 1280x720 -r 10 -i /dev/video0 \
 -f alsa -ac 1 -ar 44100 -i hw:1,0 -map 0:0 -map 1:0 -c:a aac -b:a 96k \
 -c:v h264_omx -r 10 -b:v 2M -f rtsp \ 
 -rtsp_transport tcp rtsp://127.0.0.1:80/live/webcam

So to start with, we are adding a new input of type ALSA ( Advanced Linux Sound Architecture) -f alsa -i hw:1,0. Because it’s a webcam, which generally only has a single channel of audio (aka mono), it needs -ac 1 passed to it as it by defaults tries to interpret it as stereo (-ac 2). It you get the error cannot set channel count to 1 (Invalid argument) that means it probably actually does have stereo, so you can remove it or set it to -ac 2.

Finally, I am setting a custom sampling rate of 44.1kHz, -ar 44100, the same used on CDs. All that giving us the new input of -f alsa -ac 1 -ar 44100 -i hw:1,0 .

Next we do a custom mapping to make sure our output streams are set up as we expect. Now ffmpeg is usually pretty good about doing this by default if we have a single input with video, and a single input with audio, this is really just to make sure that nobody out there has weird issues. -map 0:0 -map 1:0 is saying that we want the first track from the first source 0:0 and the first track from the second source 1:0.

Finally our encoding for the audio is set up with -c:a aac -b:a 96k which is saying to use the AAC audio type, with a bitrate of 96k. Now this could be a lot higher, as the theoretical bitrate of this source is now 352k (sample rate X bit depth X channels), but I can’t tell the difference past 96k with my mic is why I stuck with that.

One gotcha with sound, is that if the ffmpeg encoding can’t keep up with the source, aka the fps output isn’t the same as the input, the audio will probably skip weirdly, so you may need to step it down to a lower framerate or resolution if it can’t keep up.

Adding timestamp

Optional, but is included in my service script

This is optional, but I find it handy to directly add the current timestamp to the stream. I also like to have the timestamp in a box so I can always read it in case the background is close to the same color as the font. Here is what we are going to add into the middle of our ffmpeg command.

-vf "drawtext=fontfile= \
/usr/share/fonts/truetype/freefont/FreeSerif.ttf: \
text='%{localtime}':x=8:y=8:fontcolor=white: box=1: boxcolor=black"

It’s a lot text, but pretty self explanatory. We specify which font file to use drawtext=fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf, that the text will be the local time (make sure you have set your locale right!). Next we are going to start the box 8 pixels in and 8 pixels down from the top left corner. Then we set the font’s color, and that it will have a box around it with a different color.

ffmpeg -input_format yuyv422 -f video4linux2 -s 1280x720 -r 10 \
-i /dev/video0 -c:v h264_omx -r 10 -b:v 2M \
-vf "drawtext=fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf: \
text='%{localtime}':x=8:y=8:fontcolor=white: box=1: boxcolor=black" \
-an -f rtsp rtsp://localhost:80/live/stream

Making it systemd service ready

When running ffmpeg as a service, you probably don’t want to pollute the logs with standard output info. I also had a random issue with it trying to read info from stdin when a service, so I also added the -nostdin for my own sake. You can add these at the start of the command.

-nostdin -hide_banner -loglevel error

You can hide even more if you want to up it to -loglevel panic, but I personally want to see any errors that come up just in case.

So now our full command is pretty hefty.

ffmpeg -nostdin -hide_banner -loglevel error -input_format yuyv422 \
 -f video4linux2 -s 1280x720 -r 10 -i /dev/video0 -c:v h264_omx -r 10 -b:v 2M \
 -vf "drawtext=fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf \
 :text='%{localtime}:x=8:y=8:fontcolor=white: box=1: boxcolor=black" -an \
 -f rtsp rtsp://localhost:80/live/stream

Our new full command is a lot in one line, but it gets the job done!

Viewing the stream

When you have the stream running, you can pull up VLC or other network enabled media players and point to rtsp://raspberrypi:80/live/stream (if you changed your hostname, will have to do it based off ip).

When you have the command massaged exactly how you want it, we are going to create a systemd file with it, just like we did for the rstp server. In this case we will save it to the file /etc/systemd/system/encode_webcam.service and we will also add the argument -nostdin right after ffmpeg safety. sudo vi /etc/systemd/system/ encode_webcam .service

# /etc/systemd/system/encode_webcam.service

[Unit]
Description=encode_webcam
After=network.target rtsp_server.service rc-local.service

[Service]
Restart=always
User=pi
ExecStart=ffmpeg -nostdin -hide_banner -loglevel error -input_format yuyv422 \
 -f video4linux2 -s 1280x720 -r 10 -i /dev/video0 -c:v h264_omx -r 10 -b:v 2M \
 -vf "drawtext=fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf \
 :text='%{localtime}:x=8:y=8:fontcolor=white: box=1: boxcolor=black" -an \
 -f rtsp rtsp://localhost:80/live/stream


[Install]
WantedBy=multi-user.target

Now start it up, and enable it to run on boot.

sudo systemctl start encode_webcam
sudo systemctl enable encode_webcam

Connect it to your security center

I have looked at a few different security suits for my personal needs. They included iSpy (Windows) and ZoneMinder (Linux) but I finally decided upon the industry standard Blue Iris (Windows). I like it because of feature set: mobile app, motion detection, mobile alerts, NAS and Cloud storage, etc… Blue Iris also has a 15 day evaluation period to try before you buy. You don’t even need to register or provide credit info!

For our needs, the best part about Blue Iris is that it supports direct to disk recording. That way we don’t have to re-encode the stream! So lets get rolling, on the top left, select the first menu and hit “Add new camera”.

add new camera

It will then have a popup to name and configure the camera, here make sure to select the last option “Direct to disk recording”.

Next it will need the network info for the camera, put in the same info as you did for VLC. Blue Iris should auto parse it into the fields it wants, and hit OK.

Volia! Your raspberry pi is now added to your security suite!

Now you can have fun setting up recording schedules, motion detection recording, mobile alerts, and more!

On demand gaming server for pennies a month!

The only downside with most dedicated gaming servers is the cost. If you need a 24/7 server or someone to manage it for you, you can’t avoid it. However, if you are self sufficient for setting up the gaming software, and only play certain games intermittently, your costs can be significantly reduced.

Let’s use Factorio as an example. To pay for a dedicated server it will cost around $5~10 per month, used or not. Now, if you are like me and play with friends and family only a few nights a month, that’s an unnecessary expense. Sure you can self host, but then everyone is dependent on a single person to have it up and running. Instead, I switched to using a Digital Ocean Droplet, and have paid less than a quarter this past month for an on demand gaming server.

Even if we played every night after work and weekends, it would only be $1 or $2 a month. You may be thinking “oh, you just turn off the droplet when you’re not using it!”, but alas, it is not that simple. A turned off droplet or server is still holding onto resources, so it still incurs cost, which is standard across all the hosting giants I looked into.

The trick on how to save cash? When you’re not playing, turn the droplet off, snapshot it, then destroy the droplet. When you want to play again, restore the snapshot to a new droplet. That way you are only paying server costs while it is running, then paying the super cheap snapshot storage the rest of the time.

This is painful to do by hand, but really easy with a helper script. (If you want to take it further, you could make it a website with access controls so only who you want could control it whenever they wanted.)

Digital Ocean Gaming Service (DOGS)

The code repo is available on github and the instructions are boringly standard.

git clone https://github.com/cdgriffith/dogs.git
# Alternatively, just download and extract the zip file
# https://github.com/cdgriffith/dogs/archive/master.zip

# Create a venv if you are python savvy
pip install requirements.txt
cp config.yaml.example config.yaml
# Update the config file to match your digital ocean settings
python -m dogs

This script does have some prerequisites. You need to have a Digital Ocean account token, and will have to manually create and startup the server once before you can manage it via this script. (Pull Requests always considered if you want to ease that pain for others.)

Droplet setup

When you do create the droplet you want, make sure to pick the smallest specs you think you will need. You can always upgrade to larger disk size, but cannot go back down to smaller. Also during this process, record the hostname which we will use as our server name in the config file.

You will also need your SSH key id, which are the digits at the very end of the public key. For example if the key ends with rsa-key-20190721 the ssh id is 20190721. You can always find this info in the Accounts > Security section as well by selecting a key and hitting “Edit”.

Also if you hook a firewall up to the droplet you will also need that ID, which can you retrieve via the API.

DOGS Server Config

When you got all that info, add it to the config.yaml file.

token: <your 64 character hex string>
 servers:
   ubuntu-1804-factorio:
     region: nyc1
     size: s-1vcpu-2gb
     firewall_id: <unique hex string separated by dashes>
     snapshot_max: 2
     ssh_key: <8 digits from end of ssh public cert>

You can get a full list of regions and sizes from their API as well.

Again, to run you just need to be in the dogs directory and run python -m dogs.

Binary (EXE) files

If you want to package it into an easy to use exe (can also modify for mac or linux binaries), just use the included build scripts.

# Windows specific requirements, otherwise just install 'pyinstaller'
pip install -r requirements-build.txt
python dogs\build.py

And now you should have a super handy dogs.exe in the dist directory. Don’t forget to keep your config.yaml in the same directory as it!

Gaming Setup Scripts

I have a directory to put files for setting up and updating game servers. Right now I only have factorio, but feel free to add your own, a PR would be very appreciated! All you need is a way to near automatically create / install the service and a way to have it auto start (in my example using systemd for standard Ubuntu servers)

Check the scripts out on github.

Top 10ish Python standard library modules

When interviewing Python programming candidates, my wife always likes to ask the simple question, “can you name ten Python standard library modules?” This is harder than most think, as many people will completely blank out and others will be dead wrong. “Requests?” one poor soul answered. It’s a good interview question, as it gives insight onto what people are familiar with and may use regularly. So I sat down and though of of which ones I use and enjoy the most. So here are my top ten(ish) useful, favorite and unordered standard modules.

pathlib

Back in the dark days, you would have to store your path as a string, and call obscure functions under os.path to figure anything out about it. Pathlib removes the headache.

from pathlib import Path

my_path = Path('text_file.txt')
if not my_path.exists():
    my_path.write_text('File Content')
assert my_path.exists()
assert my_path.is_file()

Read more at the pathlib python docs.

tempfile

There are a boatload of uses for a temporary file or directory. Hence why it’s in the standard library. I find myself using them together, inside context managers more often than not.

from pathlib import Path
from tempfile import TemporaryDirectory, TemporaryFile


with TemporaryDirectory(prefix='Code_', suffix='_Calamity') as temp_dir:
    with TemporaryFile(dir=temp_dir) as temp_file:
        temp_file.write(b'Test')

        temp_file_path = Path(temp_file.name)
        assert temp_file_path.exists()

# Make sure file only exists within the context
assert not temp_file_path.exists()

I usually end up using this when a tool or library wants to work with a file rather than standard input, so short lived files in a context manager make life a lot easier. Tempfile python docs.

subprocess

Python is pretty amazing, but sometimes you do need to call other programs. Subprocess makes it easy to execute and interact with other executable across operating systems. Check out my other post on it!

from subprocess import run, PIPE
 
response = run("echo 'Join the Dark Side!'", shell=True, stdout=PIPE)

print(response.stdout.decode('utf-8'))
# 'Join the Dark Side!'

logging

This is probably the most useful built in library for debugging there is, and I see it either unused or misused more than anything else.

import logging
import sys
 
logger = logging.getLogger(__name__)
my_stream = logging.StreamHandler(stream=sys.stdout)
my_stream.setLevel(logging.DEBUG)
my_stream.setFormatter(
    logging.Formatter("%(asctime)s - %(name)-12s  "
                      "%(levelname)-8s %(message)s"))
logger.addHandler(my_stream)
logger.setLevel(logging.DEBUG)
 
logger.info("We the people")

If you haven’t already, go on your first date with python logging! It’s also possible to put all the configuration details into a separate ini or json file, learn more from the logging python docs.

threading and multithreading

Two very different things for widely different uses, but they have very similar interfaces and easy to talk about at the same time. Quick and dirty difference: Use threading for IO heavy tasks (writing to files, reading websites, etc) and multithreading for CPU heavy tasks.

from multiprocessing.pool import ThreadPool, Pool
 
def square_it(x):
    return x*x
 
# On Windows, make sure that multiprocessing doesn't start
# until after "if __name__ == '__main__'" 
 
# Pool and ThreadPool are interchangable in this example 
with Pool(processes=5) as pool:
   results = pool.map(square_it, [5, 4, 3, 2 ,1])
 
print(results) 
# [25, 16, 9, 4, 1]

I did a post on ThreadPools and Multithreading Pools, as I find them the easiest way to work with (multi)threading in Python.

os and sys

The Python world would not exist if we didn’t have all the power and functionality these built-ins bring. You really haven’t coded in Python if you haven’t used these yet, so I won’t even bother elaborating them here.

random and uuid

Maybe you’re making a game…

import random
random.choice(['Sneak Attack', 'High Kick', 'Low Kick'])

Or debugging a webserver…

from uuid import uuid4

# Bad example, but not writing out a whole webserver to prove a point
def get(*args):
request = uuid4()
logger.info(f'Request {request} called with args {args}')

Or turning your webserver into your own type of game…

if user_name == 'My Boss':
    time.sleep(random.randint(1, 5))

No matter which you are doing, it’s always handy to have randomly generated or unique numbers.

socket

The internet runs because of sockets. Modern technology exists because of sockets. Sockets are life, sockets are….annoying low level at times but its good to know the basics so you can appreciate everything written on top of them.

Thankfully the Python docs have good examples of them in use.

hashlib

Need to check a file’s integrity? Hashlib is there with md5 and sha hashes! I created a reusable function to easily reference when I need to do it.

Need to securly store people’s password hashes for a website? Hashlib now has scrypt support! Heck, here is my own function I always use to generate the scrypt hashes.

from collections import namedtuple
import hashlib
import os


Hashed = namedtuple('Hashed', ['hash', 'salt', 'n', 'r', 'p', 'key_length'])


def secure_hash(value: bytes, salt: bytes = None, key_length: int = 128, n: int = 2 ** 16, r: int = 8, p: int = 1):
    maxmem = n * r * 2 * key_length
    salt = salt or os.urandom(16)
    hashed = hashlib.scrypt(value, salt=salt, n=n, r=r, p=p, maxmem=maxmem, dklen=key_length)
    return Hashed(hash=hashed.hex(), salt=salt.hex(), n=n, r=r, p=p, key_length=key_length)

venv

You probably only think of it as a command when you run python -m venv python_virtual_env to create your environments, but it’s run that way because it’s a standard library. Every new project you start or Python program you install should be using this library, so it is used a lot!

Summary

There ya go, 10 or so can’t live without standard libraries! Isn’t it so nice that Python comes “batteries included”?

Introducing FastFlix – AV1 encoder GUI and more!

First, straight to the fun, download FastFlix here to try it out! (Windows only builds right now). FastFlix started out to be a small clip and GIF maker for myself, but quickly realized I could expand it for larger usage. And it just so happens that AV1 is an emerging codec that doesn’t have a lot of GUI options yet, it’s like it was meant to be!

The main GUI

Before going out and re-encoing everything with AV1, which will be the next standard codec as everyone is on board with it, there are a few catches, so make sure to read on

What makes FastFlix unique?

AV1 Support for multiple libraries

FastFlix is designed as a general command wrapper, so it can support multiple different programs. Right now it supports both the libaom-av1 and SVT-AV1 libraries.

Totally MIT open source code, reuse to your hearts content!

Unlike most converts that are limited to the GPL license thanks to the libx265/libx264 libraries and others, FastFlix has been designed in a way to keep it legal to steal use any of it’s core code in your own projects without forcing them to be open source.

Extensible

FastFlix was designed to have a plug in architecture. That way anyone can develop or use their own plugins on top of what is already available to bring additional functionally.

What’s the catch?

New program using an experimental codec

There will be bugs. Both on the GUI side and on the codec side. Report anything weird you see and we’ll try to figure out it’s a GUI problem or needs to be passed along to the codec team.

SVT-AV1 makes it difficult to convert videos

Right now SVT-AV1 requires the source input to be broken up to smaller chucks (if it is longer than the segment size) as raw YUV video, which can take up gigabytes of space. They are automatically cleaned up as it goes along, but it is still silly that SVT-AV1 cannot take a regular video file as input yet.

I’m just one guy, and this is a side project

I don’t make any money (nor take donations. $10K+ bribes, please email me 😉) and FastFlix is not something I will spend all my free time on. So I am always looking for help and feedback!

Wrapping up

Please give a github star if you like FastFlix and be sure to send your love to SVT-AV1 as well if you find their program useful!

Again, you can download FastFlix on the Github release page.

For those of you interested more in how FastFlix works or was created I hope to do a follow up post that goes into using PySide2 and the full workflow of using Appveyor to deliver releases.

Encoding settings for HDR 4K videos using 10-bit x265

There is currently a serious lack of data on compressing 4K HDR videos out there, so I took it upon myself to get learned in the ways of the x265 encoding world.

I have historically been using the older x264 mp4s for my videos, as it just works on everything. However most devices finally have some native h.265 decoding. (As a heads up h.265 is the specification, and x265 is encoder for it. I may mix it up myself in this article, don’t worry about the letter, just the numbers.)

Updated: 4/14/2019 – New Preset Setting (tl;dr: use slow)

What are the best settings for me to use when encoding x265 videos?

The honest to god true answer is “it depends”, however I find that answer unsuitable for my own needs. I want a setting that I can use on any incoming 4K HDR video I buy.

I mainly use Handbrake to encode my videos, so I went straight to their documentation. It states that for 4K videos with x265 they suggest a Constant Rate Factor (CRF) encoding in the range of 22-28 (the larger the number the lower the quality).

Through some experimentation I found that I personally never can really see a difference between anything lower than 22 using a Slow present. Therefore I played it safe, bump it down a notch and just encode all of my stuff with x265 10-bit at CRF of 20 on Slow preset. That way I know I should never be disappointed.

Then I recently read YouTubes suggest guidelines for bitrates. They claim that a 4K video coming into their site should optimally be 35~45Mbps when encoded with the older x264 codecs.

Now I know that x265 can be around 50% more efficient than x264, and that YouTube needs it higher quality coming in so when they re-compress it it will still look good. But when I looked at the videos I was enjoying just fine at CRF 22, they were mostly coming out with less than a 10Mbps bitrate. So I had to ask myself:

How much better is x265 than x264?

To find out I would need a lot of comparable data. I started with a 4K HDR example video. First thing I did was to chop out a minute segment and promptly remove the HDR. Thus comparing the two encoders via their default 8-bit compressors.

I found this code to convert the 10-bit “HDR” yuv420p10le colorspace down to the standard yuv420p 8-bit colorspace from the colourspace blog so props to them for having a handy guide just for this.

ffmpeg -y -ss 07:48 -t 60 -i my_movie.mkv-vf zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable:desat=0,zscale=t=bt709:m=bt709:r=tv,format=yuv420p -c:v libx265 -preset ultrafast -x265-params lossless=1 -an -sn -dn -reset_timestamps 1 movie_non_hdr.mkv

Average Overall SSIM

Then I ran multiple two pass ABR runs using ffmpeg for both x264 and x265 using the same target bitrate. Afterwards compared them to the original using the Structural Similarity Index (SSIM). Put simply, the closer the result is to 1 the better. It means there is less differences between the original and the compressed one

Generated via Python and matplotlib
(Click to view larger version)

The SSIM result is done frame by frame, so we have to average them all together to see which is best overall. On the section of video I chose, x264 needed considerably more bitrate to achieve the same score. The horizontal line shows this where x264 needs 14Mbps to match x265’s 9Mbps, a 5000kbps difference! If we wanted to go by YouTube’s recommendations for a video file that will be re-encoded again, you would only need a 25Mbps x265 file instead of a 35Mbps x264 video.

Sample commands I used to generate these files:

ffmpeg -i movie.mkv -c:v libx265 -b:v 500k -x265-params pass=1 -an -f mp4 NUL

ffmpeg -i movie.mkv -c:v libx265 -b:v 500k -x265-params pass=2 -an h265\movie_500.mp4

ffmpeg -i my_movie.mkv -i h265\movie_500.mp4 -lavfi  ssim=265_movie_500_ssim.log -f null -

Lowest 1% SSIM

However the averages don’t tell the whole story. Because if every frame was that good, we shouldn’t need more than 6Mbps x265 or 10Mbps x264 4K video. So lets take a step back and look at the lowest 1% of the frames.

Generated via Python and matplotlib
(Click to view larger version)

Here we can see x264 has a much harder time at lower bitrates. Also note that the highest marker on this chart is 0.98, compared the total average chart’s 0.995.

This information alone confirmed for me that I will only be using x265 or newer encodings (maybe AV1 in 2020) for storing videos going forward.

Download the SSIM data as CSV.

How does CRF compare to ABR?

I have always read to use Constant Rate Factor over Average BitRate for stored video files (and especially over Constant Quality). CRF is the best of both worlds. If you have an easily compressible video, it won’t bloat the encoded video to meet some arbitrary bitrate. And bitrate directly correlates to file size. It also won’t be constrained to that limit if the video requires a lot more information to capture the complex scene.

But that is all hypothetical. We have some hard date, lets use it. So remember, Handbrake recommends a range of 22-28 CRF, and I personally cannot see any visual loss at CRF 20. So where does that show up on our chart?

Generated via Python and matplotlib
(Click to view larger version)

Now this is an apples to oranges comparison. The CRF videos were done via Handbrake using x265 10-bit, whereas everything else was done via ffmpeg using x265 or x264 8-bit. Still, we get a good idea of where these show up. At both CRF 24 and CRF 22, even the lowest frames don’t dip below SSIM 0.95. I personally think the extra 2500kbps for the large jump in minimum quality from CRF 24 to CRF 22 is a must. To some, including myself, it could be worth the extra 4000kbps jump from CRF 22 to CRF 20.

So let’s get a little more apples to apples. In this test, I encoded all videos with ffmpeg using the default presents. I did three CRF videos first, at 22, 20, and 18, then using their resulting bitrates created three ABR videos.

Generated via Python and matplotlib
(Click to view larger version)

Their overall average SSIM scores were near as identical. However, CRF shows its true edge on the lowest 1%, easily beating out ABR at every turn.

To 10-bit or not to 10-bit?

Thankfully there is a simple answer. If you are encoding to x264 or x265, encode to 10-bit if your devices support it. Even if your source video doesn’t use the HDR color space, it compresses better.

There is only one time to not use it. When the device you are going to watch it on doesn’t support it.

Which preset should I use?

The normal wisdom is to use the the slowest you can stand for the encoding time. Slower = better video quality and compression. However, that does not mean smaller file size at the same CRF.

Even though others have tackled this issue, I wanted to use the same material I was already testing and make sure it held true with 4K HDR video.

Generated via Python and matplotlib
(Click to view larger version)

I used a three minute 4K HDR clip, using Handbrake to only modify which present was used. The results were surprising to me to be honest, I was expecting medium to have a better margin between fast and slow. But based on just the average, slow was the obvious choice, as even bumping up the CRF from 18 to 16 didn’t match the quality. Even thought the file size was much larger for the CRF 16 Medium encoding than it was than for the CRF 18 Slow! (We’ll get to that later.)

Okay, okay, lets back up a step and look at the bottom 1% again as well.

Generated via Python and matplotlib
(Click to view larger version)

Well well wishing well, that is even more definitive. The jump from medium to slow is very significant in multiple ways. Even though it does cost double the time of medium it really delivers in the quality department. Easily beating out the lowest 1% of even CRF 16 medium, two entire steps away.

Generated via Excel
(Click to view larger version)

The bitrates are as expected, the higher quality it gets the more bitrate it will need. What is interesting, is if we put CRF 16 - Medium encoding’s bitrate on this chart it would go shoot off the top at a staggering 15510kbps! Keep in mind that is while still being lesser quality than CRF 18 - Slow.

In this data set, slow is the clear winner in multiple ways. Which is very similar to other’s results as well, so I’m personally sticking too it. (And if I ran these tests first, I would have even used slow for all the other testing!)

Conclusion

If you want a single go to setting for encoding, based on my personal testing CRF 20 with Slow preset looks amazing (but may take too long if you are using older hardware).

Now, if I have a super computer and unlimited storage, I might lean towards CRF 18 or maybe even 16, but still wouldn’t feel the need to take it the whole way to CRF 14 and veryslow or anything crazy.

I hope you found this information as useful as I did, if you have any thoughts or feedback please let me know!