It’s finally happened, after years of complaining and told “it will never happen”, I got my darn switch statement! And honestly, calling it only a “switch” statement is really underselling it. It’s much more powerful than most people will ever use! You might have already heard about this, but I wanted to make sure it made it to at least the 3.10 beta phase, as the alphas are no guarantee you’ll actually see the new feature (still possible to remove before the RC phase, but less likely).
Python 3.10 will introduce “Structural Pattern Matching” as introduced in PEP622 which is a crazy advanced switch statement that can recognizing patterns. Instead of the keyword switch Python will introduce match instead (get ready to update your regex variable names!). Let’s do a little quick compare with it and JavaScript’s switch statement. First let’s start off with the Javascript example modified from w3schools.
Javascript’s Switch
switch (new Date().getDay()) {
case 5:
console.log("It's Friday!");
break;
case 0:
case 6:
console.log("Woo, Weekend!");
break;
default:
console.log("Work. Work. Work. Work.");
}
Python’s Structural Pattern Matching
We can accomplish the same with Python in less lines. Note that in Python the weekday starts at 0 for Monday.
from datetime import datetime
match datetime.today().weekday():
case 4:
print("It's Friday!")
case 5 | 6:
print("Woo, Weekend!")
case _:
print("Work. Work. Work. Work.")
Similarities and Differences
With Python’s new match style statements, the biggest change from tradition is no “drop through” cases. Aka you don’t have to manually specify break or anything to exit the matchcase for every instance. That does mean though that you can’t have multiple matches with the same result. Instead you can combine checks with | for “or” operations. Which I honestly think is a lot cleaner.
What I don’t like is there is no default keyword or similar, and you have to use the _ wildcard. The underscore is already overused in my opinion. However it does add a lot of power for more advanced cases. Let’s dive into some more examples.
Matching and Assigning
The very basic things to understand about the new Structural Pattern Matching is the flow (as seen in the example above) and the possibility of assignment. Because not only can you make sure things are equal to another literal, you can detect patterns (as the name suggests) and then work with the variables inside the pattern itself.
Pattern Recognition
Lets dip the toes in on this whole “pattern matching” thing, what does that mean?
Say you have a tuple with two objects, that could be in either order and you always want to print it right.
You could have a simple match case to straighten it out so it always prints “Purchase Number 574”. As the pattern matching support type checking.
match data_1:
case str(x), int(y):
print(x, y)
case int(x), str(y):
print(y, x)
Match on dict values
Just like with regular value comparisons, you can check dictionary values and still do or operations. In this case, either grade b or c will print out "Welcome to being average!"
match {'grade': 'B'}:
case {'grade': 'a' | 'A'}:
print("You're a star!")
case {'grade': 'b' | 'c' | 'B' | 'C'}:
print("Welcome to being average!")
Unpacking arbitrary data
You can use the standard * and ** unpackers to capture and unpack variable length lists and dicts.
my_dict = {'data': [{'action': 'move'}, {'action': 'stay'}],
'location': "square_10"}
match my_dict:
case {'data': [*options], **remainder }:
print(options)
print(remainder)
case _:
raise ValueError("Data not as expected")
That can really come in handy if dealing with variable length data.
Adding the “if”
You can also add in some inline checks.
cords = (30.25100365332043, -97.86221371827051, "Workplace")
cords_2 = (44.40575746530253, 8.947747627912994)
match cords:
case lat, lon, *_ if lat > 0:
print("Northern Hemisphere")
case lat, lon, *_ if lat < 0:
print("Sothern Hemisphere")
Don’t forget about “as”
Sometimes you might want one the many literals you are looking for to be usable in the case itself. Lets put a lot of our knowledge together and check out the added power of the as clause.
We are going to build a really simple command line game to look for random things.
import sys
import random
inventory = []
while True:
match input().lower().split():
# Remember `split` makes the input into a list.
# These first two cases are functionally the same.
case ["exit"]:
sys.exit(0)
case "stop", *_:
sys.exit(0)
case "travel", "up" | "down" | "left" | "right" as direction:
print(f"Going {direction}")
case "search", "area" | "backpack" as thing, _, *extra:
item = " ".join(extra)
if thing == "backpack":
if item in inventory:
print(f"Yes you have {item}")
else:
print(f"No you don't have {item}")
elif thing == "area":
if random.randint(1, 10) > 5:
inventory.append(item)
print(f"You found {item}!")
else:
print(f"No {item} here")
else:
print(f"Sorry, you cannot search {thing}")
case _:
print("sorry, didn't understand you! type 'exit' or 'stop' to quit")
Lets do a quick playthrough:
> search area for diamonds
You found diamonds!
> search backpack for diamonds
Yes you have diamonds
> take a short rest and try to eat the diamonds
sorry, didn't understand you! type 'exit' or 'stop' to quit
> Travel Up
Going up
> search area for candy
No candy here
> search backpack for candy
No you don't have candy
> stop
# Process finished with exit code 0
Notice our special case for searching around.
case "search", "area" | "backpack" as thing, _, *extra:
we are looking for the first keyword “Search”, then either “area” or “backpack” and saving which to the variable thing. The _ will ignore the next word as we expect them to type for or something useless there. Finally we grab everything else and treat it as a single item.
Classes and More
Using dataclasses with match is super easy.
from dataclasses import dataclass
@dataclass
class Car:
model: str
top_speed: int
car_1 = Car("Jaguar F-Type R", 186)
car_2 = Car("Tesla Model S", 163)
car_3 = Car("BMW M4", 155)
match car_1:
case Car(x, speed) if speed > 180:
print(f"{x} is a super fast car!")
case Car(x, speed) if speed > 160:
print(f"{x} is a pretty fast car")
case _:
print("Regular Car")
Using a regular class, you have to be a pit more explicit about the varaibles that are being used.
class Car:
def __init__(self, model, top_speed):
self.model = model
self.top_speed = top_speed
car_1 = Car("F-Type R", 186)
car_2 = Car("Model S", 163)
car_3 = Car("BMW M4", 155)
match car_1:
case Car(model=x, top_speed=speed) if speed > 180:
print(f"{x} is a super fast car!")
case Car(model=x, top_speed=speed) if speed > 160:
print(f"{x} is a pretty fast car")
case _:
print("Regular Car")
However, there is a way around that! You can add __match_args__ to the class itself to define which arguments you will want to use for the pattern recognition.
class Car:
__match_args__ = ["model", "top_speed"]
def __init__(self, model, top_speed):
self.model = model
self.top_speed = top_speed
car_1 = Car("F-Type R", 186)
car_2 = Car("Model S", 163)
car_3 = Car("BMW M4", 155)
match car_1:
case Car(x, speed) if speed > 180:
print(f"{x} is a super fast car!")
case Car(x, speed) if speed > 160:
print(f"{x} is a pretty fast car")
case _:
print("Regular Car")
Summary
Structural Pattern Matching is Python’s sexy new “switch” statement that will bring a lot of power, and removal of a lot of old ifelif blocks. The only real downside is that it’s brand new, and will take a few years for most people to be able to use it with their default interpreter.
I have previously covered how to upload large files with dropzone.js, but it didn’t allow for parallel chunk uploads. In this article we will go over that new addition, as well as several other improvements.
This is the final result for now, and can be customized if you so chose. Obviously I’m no graphic design, but I’d pick function over style anyway.
Design
Before we dive into code, lets think about the design. We will need to somehow handle multiple parts of a single file being uploaded at the same time in a random order. How do we keep track of that?
Thankfully, Dropzone will provide the server with a few different pieces of information which each chunk, they include:
dzuuid – unique ID per upload file
dzchunkindex – the chunk number of the current upload
dztotalfilesize – Total size of the upload
dzchunksize – Max size per chunk
dztotalchunkcount – The number of chunks in this file
dzchunkbyteoffset – The place in the file this chunk starts
In my mind there are two clear ways to approach the problem. First option is to create a sparse file of the full size to start with, using dztotalchunkcount and then with every incoming chunk, set the position of the file using dzchunkbyteoffset and write the data starting there.
The advantage of this method is that it only requires a single file on disk. The disadvantage is you have to worry about multiple threads accessing the same file at the same time.
The second choice is to write each chunk to a separate file, then when they are all uploaded concatenate them all to a single file and remove the individual chunks. The disadvantage are that that you require twice the space for a short time, and have to deal with cleanup of temporary files.
I personally preferred the second option, as it seemed a bit safer.
Upload Function
As a quick warning, I am now using Bottle instead of Flask for this upload, so a bit of the form syntax has changed since the last post.
from pathlib import Path
from threading import Lock
from collections import defaultdict
import shutil
import uuid
from bottle import route, run, request, error, response, HTTPError, static_file
from werkzeug.utils import secure_filename
lock = Lock()
chucks = defaultdict(list)
chunk_path = Path(__file__).parent / "chunks"
storage_path = Path(__file__).parent / "storage"
chunk_path.mkdir(exist_ok=True, parents=True)
storage_path.mkdir(exist_ok=True, parents=True)
@route("/upload", method="POST")
def upload():
file = request.files.get("file")
if not file:
raise HTTPError(status=400, body="No file provided")
dz_uuid = request.forms.get("dzuuid")
if not dz_uuid:
# Assume this file has not been chunked
with open(storage_path / f"{uuid.uuid4()}_{secure_filename(file.filename)}", "wb") as f:
file.save(f)
return "File Saved"
# Chunked download
try:
current_chunk = int(request.forms["dzchunkindex"])
total_chunks = int(request.forms["dztotalchunkcount"])
except KeyError as err:
raise HTTPError(status=400, body=f"Not all required fields supplied, missing {err}")
except ValueError:
raise HTTPError(status=400, body=f"Values provided were not in expected format")
# Create a new directory for this file in the chunks dir, using the UUID as the folder name
save_dir = chunk_path / dz_uuid
if not save_dir.exists():
save_dir.mkdir(exist_ok=True, parents=True)
# Save the individual chunk
with open(save_dir / str(request.forms["dzchunkindex"]), "wb") as f:
file.save(f)
# See if we have all the chunks downloaded
with lock:
chucks[dz_uuid].append(current_chunk)
completed = len(chucks[dz_uuid]) == total_chunks
# Concat all the files into the final file when all are downloaded
if completed:
with open(storage_path / f"{dz_uuid}_{secure_filename(file.filename)}", "wb") as f:
for file_number in range(total_chunks):
f.write((save_dir / str(file_number)).read_bytes())
print(f"{file.filename} has been uploaded")
shutil.rmtree(save_dir)
return "Chunk upload successful"
if __name__ == "__main__":
run(server="paste")
Hopefully the code is decently self documented. We do a few checks at the start as we pull in the required parameters. Then we prepare the directory for where the temporary chunks will be stored, and write the incoming chunk there. We gather information on all the chunks and when then have been completed in a global dictionary, and when they are all uploaded they are assembled into the final file.
File Downloading
Now that we can put files on the server, what about getting them back? I personally don’t want people to host random files on my server, but others may. To accomplish that, we shouldn’t just list all the files to everyone that visits the site, but only to whoever uploaded it. Thankfully we can just store the uuid in a cookie on the frontend, and then have a very basic download function.
@route("/download/<dz_uuid>")
def download(dz_uuid):
for file in storage_path.iterdir():
if file.is_file() and file.name.startswith(dz_uuid):
return static_file(file.name, root=file.parent.absolute(), download=True)
return HTTPError(status=404)
This does complicate our frontend a bit, as we want to save both UUID and filename as text fields in a cookie. There are a lot of great libraries out there to make life easier with JavaScript and cookies, but I wanted to keep it simple and pure JS other than Dropzone, making the code a bit more complicated than last time.
Dropzone frontend
Instead of being a standalone file, I have also put this directly into the python file to make using it as a f-string a lot easier, but makes it a little harder to read.
Notice we are using a slew of python variables that we are going to allow to be configurable upon launch.
Command line options
import argparse
...
allow_downloads = False
dropzone_cdn = "https://cdnjs.cloudflare.com/ajax/libs/dropzone"
dropzone_version = "5.7.6"
dropzone_timeout = "120000"
dropzone_max_file_size = "100000"
dropzone_chunk_size = "1000000"
dropzone_parallel_chunks = "true"
dropzone_force_chunking = "true"
...
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("-p", "--port", type=int, default=16273, required=False)
parser.add_argument("--host", type=str, default="0.0.0.0", required=False)
parser.add_argument("-s", "--storage", type=str, default=str(storage_path), required=False)
parser.add_argument("-c", "--chunks", type=str, default=str(chunk_path), required=False)
parser.add_argument(
"--max-size",
type=str,
default=dropzone_max_file_size,
help="Max file size (Mb)",
)
parser.add_argument(
"--timeout",
type=str,
default=dropzone_timeout,
help="Timeout (ms) for each chuck upload",
)
parser.add_argument("--chunk-size", type=str, default=dropzone_chunk_size, help="Chunk size (bytes)")
parser.add_argument("--disable-parallel-chunks", required=False, default=False, action="store_true")
parser.add_argument("--disable-force-chunking", required=False, default=False, action="store_true")
parser.add_argument("-a", "--allow-downloads", required=False, default=False, action="store_true")
parser.add_argument("--dz-cdn", type=str, default=None, required=False)
parser.add_argument("--dz-version", type=str, default=None, required=False)
return parser.parse_args()
if __name__ == "__main__":
args = parse_args()
storage_path = Path(args.storage)
chunk_path = Path(args.chunks)
dropzone_chunk_size = args.chunk_size
dropzone_timeout = args.timeout
dropzone_max_file_size = args.max_size
try:
if int(dropzone_timeout) < 1 or int(dropzone_chunk_size) < 1 or int(dropzone_max_file_size) < 1:
raise Exception("Invalid dropzone option, make sure max-size, timeout, and chunk-size are all positive")
except ValueError:
raise Exception("Invalid dropzone option, make sure max-size, timeout, and chunk-size are all integers")
if args.dz_cdn:
dropzone_cdn = args.dz_cdn
if args.dz_version:
dropzone_version = args.dz_version
if args.disable_parallel_chunks:
dropzone_parallel_chunks = "false"
if args.disable_force_chunking:
dropzone_force_chunking = "false"
if args.allow_downloads:
allow_downloads = True
if not storage_path.exists():
storage_path.mkdir(exist_ok=True)
if not chunk_path.exists():
chunk_path.mkdir(exist_ok=True)
print(f"""Timeout: {int(dropzone_timeout)
Chunk Size: {int(dropzone_chunk_size)
Max File Size: {int(dropzone_max_file_size)} Mb
Force Chunking: {dropzone_force_chunking}
Parallel Chunks: {dropzone_parallel_chunks}
Storage Path: {storage_path.absolute()}
Chunk Path: {chunk_path.absolute()}
""")
run(server="paste", port=args.port, host=args.host)
As this will become an executable, to be configurable we want to pass parameters upon launch.
Favicon
Now this is getting into the realm of silly. But to be an all in one script, we need to provide a binary file (the favicon) in the script itself. Thankfully ico files can be compressed rather easily, so we are going to compress it in the script itself, and decompress it when requested.
Here is the culmination of everything we talked about put into a script.
This may not always be the newest version, if you want to use it yourself please download the final executable or view github to see the latest code.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from pathlib import Path
from threading import Lock
from collections import defaultdict
import shutil
import argparse
import uuid
import zlib
from bottle import route, run, request, error, response, HTTPError, static_file
from werkzeug.utils import secure_filename
storage_path: Path = Path(__file__).parent / "storage"
chunk_path: Path = Path(__file__).parent / "chunk"
allow_downloads = False
dropzone_cdn = "https://cdnjs.cloudflare.com/ajax/libs/dropzone"
dropzone_version = "5.7.6"
dropzone_timeout = "120000"
dropzone_max_file_size = "100000"
dropzone_chunk_size = "1000000"
dropzone_parallel_chunks = "true"
dropzone_force_chunking = "true"
lock = Lock()
chucks = defaultdict(list)
@error(500)
def handle_500(error_message):
response.status = 500
response.body = f"Error: {error_message}"
return response
@route("/")
def index():
index_file = Path(__file__) / "index.html"
if index_file.exists():
return index_file.read_text()
return f"""
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8">
<link rel="stylesheet" href="{dropzone_cdn.rstrip('/')}/{dropzone_version}/min/dropzone.min.css"/>
<link rel="stylesheet" href="{dropzone_cdn.rstrip('/')}/{dropzone_version}/min/basic.min.css"/>
<script type="application/javascript"
src="{dropzone_cdn.rstrip('/')}/{dropzone_version}/min/dropzone.min.js">
</script>
<title>pyfiledrop</title>
</head>
<body>
<div id="content" style="width: 800px; margin: 0 auto;">
<h2>Upload new files</h2>
<form method="POST" action='/upload' class="dropzone dz-clickable" id="dropper" enctype="multipart/form-data">
</form>
<h2>
Uploaded
<input type="button" value="Clear" onclick="clearCookies()" />
</h2>
<div id="uploaded">
</div>
<script type="application/javascript">
function clearCookies() {{
document.cookie = "files=; Max-Age=0";
document.getElementById("uploaded").innerHTML = "";
}}
function getFilesFromCookie() {{
try {{ return document.cookie.split("=", 2)[1].split("||");}} catch (error) {{ return []; }}
}}
function saveCookie(new_file) {{
let all_files = getFilesFromCookie();
all_files.push(new_file);
document.cookie = `files=${{all_files.join("||")}}`;
}}
function generateLink(combo){{
const uuid = combo.split('|^^|')[0];
const name = combo.split('|^^|')[1];
if ({'true' if allow_downloads else 'false'}) {{
return `<a href="/download/${{uuid}}" download="${{name}}">${{name}}</a>`;
}}
return name;
}}
function init() {{
Dropzone.options.dropper = {{
paramName: 'file',
chunking: true,
forceChunking: {dropzone_force_chunking},
url: '/upload',
retryChunks: true,
parallelChunkUploads: {dropzone_parallel_chunks},
timeout: {dropzone_timeout},
maxFilesize: {dropzone_max_file_size},
chunkSize: {dropzone_chunk_size},
init: function () {{
this.on("complete", function (file) {{
let combo = `${{file.upload.uuid}}|^^|${{file.upload.filename}}`;
saveCookie(combo);
document.getElementById("uploaded").innerHTML += generateLink(combo) + "<br />";
}});
}}
}}
if (typeof document.cookie !== 'undefined' ) {{
let content = "";
getFilesFromCookie().forEach(function (combo) {{
content += generateLink(combo) + "<br />";
}});
document.getElementById("uploaded").innerHTML = content;
}}
}}
init();
</script>
</div>
</body>
</html>
"""
@route("/favicon.ico")
def favicon():
return zlib.decompress(
b"x\x9c\xedVYN\xc40\x0c5J%[\xe2\xa3|q\x06\x8e1G\xe1(=ZoV"
b"\xb2\xa7\x89\x97R\x8d\x84\x04\xe4\xa5\xcb(\xc9\xb3\x1do"
b"\x1d\x80\x17?\x1e\x0f\xf0O\x82\xcfw\x00\x7f\xc1\x87\xbf"
b"\xfd\x14l\x90\xe6#\xde@\xc1\x966n[z\x85\x11\xa6\xfcc"
b"\xdfw?s\xc4\x0b\x8e#\xbd\xc2\x08S\xe1111\xf1k\xb1NL"
b"\xfcU<\x99\xe4T\xf8\xf43|\xaa\x18\xf8\xc3\xbaHFw\xaaj\x94"
b"\xf4c[F\xc6\xee\xbb\xc2\xc0\x17\xf6\xf4\x12\x160\xf9"
b"\xa3\xfeQB5\xab@\xf4\x1f\xa55r\xf9\xa4KGG\xee\x16\xdd\xff"
b"\x8e\x9d\x8by\xc4\xe4\x17\tU\xbdDg\xf1\xeb\xf0Zh\x8e"
b"\xd3s\x9c\xab\xc3P\n<e\xcb$\x05 b\xd8\x84Q1\x8a\xd6Kt\xe6"
b"\x85(\x13\xe5\xf3]j\xcf\x06\x88\xe6K\x02\x84\x18\x90"
b"\xc5\xa7Kz\xd4\x11\xeeEZK\x012\xe9\xab\xa5\xbf\xb3@i\x00"
b"\xce\xe47\x0b\xb4\xfe\xb1d\xffk\xebh\xd3\xa3\xfd\xa4:`5J"
b"\xa3\xf1\xf5\xf4\xcf\x02tz\x8c_\xd2\xa1\xee\xe1\xad"
b"\xaa\xb7n-\xe5\xafoSQ\x14'\x01\xb7\x9b<\x15~\x0e\xf4b"
b"\x8a\x90k\x8c\xdaO\xfb\x18<H\x9d\xdfj\xab\xd0\xb43\xe1"
b'\xe3nt\x16\xdf\r\xe6\xa1d\xad\xd0\xc9z\x03"\xc7c\x94v'
b"\xb6I\xe1\x8f\xf5,\xaa2\x93}\x90\xe0\x94\x1d\xd2\xfcY~f"
b"\xab\r\xc1\xc8\xc4\xe4\x1f\xed\x03\x1e`\xd6\x02\xda\xc7k"
b"\x16\x1a\xf4\xcb2Q\x05\xa0\xe6\xb4\x1e\xa4\x84\xc6"
b"\xcc..`8'\x9a\xc9-\n\xa8\x05]?\xa3\xdfn\x11-\xcc\x0b"
b"\xb4\x7f67:\x0c\xcf\xd5\xbb\xfd\x89\x9ebG\xf8:\x8bG"
b"\xc0\xfb\x9dm\xe2\xdf\x80g\xea\xc4\xc45\xbe\x00\x03\xe9\xd6\xbb"
)
@route("/upload", method="POST")
def upload():
file = request.files.get("file")
if not file:
raise HTTPError(status=400, body="No file provided")
dz_uuid = request.forms.get("dzuuid")
if not dz_uuid:
# Assume this file has not been chunked
with open(storage_path / f"{uuid.uuid4()}_{secure_filename(file.filename)}", "wb") as f:
file.save(f)
return "File Saved"
# Chunked download
try:
current_chunk = int(request.forms["dzchunkindex"])
total_chunks = int(request.forms["dztotalchunkcount"])
except KeyError as err:
raise HTTPError(status=400, body=f"Not all required fields supplied, missing {err}")
except ValueError:
raise HTTPError(status=400, body=f"Values provided were not in expected format")
save_dir = chunk_path / dz_uuid
if not save_dir.exists():
save_dir.mkdir(exist_ok=True, parents=True)
# Save the individual chunk
with open(save_dir / str(request.forms["dzchunkindex"]), "wb") as f:
file.save(f)
# See if we have all the chunks downloaded
with lock:
chucks[dz_uuid].append(current_chunk)
completed = len(chucks[dz_uuid]) == total_chunks
# Concat all the files into the final file when all are downloaded
if completed:
with open(storage_path / f"{dz_uuid}_{secure_filename(file.filename)}", "wb") as f:
for file_number in range(total_chunks):
f.write((save_dir / str(file_number)).read_bytes())
print(f"{file.filename} has been uploaded")
shutil.rmtree(save_dir)
return "Chunk upload successful"
@route("/download/<dz_uuid>")
def download(dz_uuid):
if not allow_downloads:
raise HTTPError(status=403)
for file in storage_path.iterdir():
if file.is_file() and file.name.startswith(dz_uuid):
return static_file(file.name, root=file.parent.absolute(), download=True)
return HTTPError(status=404)
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("-p", "--port", type=int, default=16273, required=False)
parser.add_argument("--host", type=str, default="0.0.0.0", required=False)
parser.add_argument("-s", "--storage", type=str, default=str(storage_path), required=False)
parser.add_argument("-c", "--chunks", type=str, default=str(chunk_path), required=False)
parser.add_argument(
"--max-size",
type=str,
default=dropzone_max_file_size,
help="Max file size (Mb)",
)
parser.add_argument(
"--timeout",
type=str,
default=dropzone_timeout,
help="Timeout (ms) for each chuck upload",
)
parser.add_argument("--chunk-size", type=str, default=dropzone_chunk_size, help="Chunk size (bytes)")
parser.add_argument("--disable-parallel-chunks", required=False, default=False, action="store_true")
parser.add_argument("--disable-force-chunking", required=False, default=False, action="store_true")
parser.add_argument("-a", "--allow-downloads", required=False, default=False, action="store_true")
parser.add_argument("--dz-cdn", type=str, default=None, required=False)
parser.add_argument("--dz-version", type=str, default=None, required=False)
return parser.parse_args()
if __name__ == "__main__":
args = parse_args()
storage_path = Path(args.storage)
chunk_path = Path(args.chunks)
dropzone_chunk_size = args.chunk_size
dropzone_timeout = args.timeout
dropzone_max_file_size = args.max_size
try:
if int(dropzone_timeout) < 1 or int(dropzone_chunk_size) < 1 or int(dropzone_max_file_size) < 1:
raise Exception("Invalid dropzone option, make sure max-size, timeout, and chunk-size are all positive")
except ValueError:
raise Exception("Invalid dropzone option, make sure max-size, timeout, and chunk-size are all integers")
if args.dz_cdn:
dropzone_cdn = args.dz_cdn
if args.dz_version:
dropzone_version = args.dz_version
if args.disable_parallel_chunks:
dropzone_parallel_chunks = "false"
if args.disable_force_chunking:
dropzone_force_chunking = "false"
if args.allow_downloads:
allow_downloads = True
if not storage_path.exists():
storage_path.mkdir(exist_ok=True)
if not chunk_path.exists():
chunk_path.mkdir(exist_ok=True)
print(
f"""Timeout: {int(dropzone_timeout)
Chunk Size: {int(dropzone_chunk_size)
Max File Size: {int(dropzone_max_file_size)} Mb
Force Chunking: {dropzone_force_chunking}
Parallel Chunks: {dropzone_parallel_chunks}
Storage Path: {storage_path.absolute()}
Chunk Path: {chunk_path.absolute()}
"""
)
run(server="paste", port=args.port, host=args.host)
Make it yours, and give back if you can!
What will you add to this script? Set a max time for how long you can see the uploaded files? A way to ensure the file exists on the server before trying to download it? Checksum comparison to avoid using space for duplicate files?
However you make it better, please consider to add a pull request for your features so anyone can benefit from it!
Do you know someone who hails from the strange land of Django development? Where there are code based settings files, is customary to use a ORM for all database calls, and classes reign supreme. It’s a dangerous place that sits perilously close to the Javalands. Most who originally traveled there were told there would be riches, but they lost themselves along the way and simply need our help to come home again.
Classless classes
The first bad habit that must be broken is the horrendous overuse of classes. While classes are considered the building blocks of most object oriented languages, Python has the power of first-class functions and a clean namespacing system which negates many the usual reasons for classes.
When using Django’s framework, where a lot of its powerful operations are performed by subclassing, classes actually makes sense. When creating a small class for a one use instance in a library or script it’s most likely simply over complicating and slowing down your code.
Let’s use an example I have seen recently, where someone wanted to run a command, but needed to know if a custom shell wrapper existed before running the command.
Most of this class is setup and unnecessary. Especially if the state of the system won’t change while the script is running (i.e. if “/bin/secure_shell” is added or removed during execution). Let’s clean this up a little.
Shorter, cleaner, faster, easier to read. It doesn’t get much better than that. Now if the system state may change, you would need to throw the shell check inside the command itself, but is also possible.
To learn more view the great PyCon video entitled “stop writing classes.” The video goes over more details of why classes can stink up your code.
Help me ORM-Kenobi!
Now onto probably the hardest change that will have the most significant impact. If you or someone you know has mesothelioma learned how to interact databases by using an ORM (Django’s Models) and don’t know how to hand write SQL queries, it’s time to make an effort to learn.
An ORM is always going to be slower than its raw SQL equivalent, and in a many cases it will be much much slower. ORMs are brilliant, but still can’t always optimize queries. Even if they do, they then load the retrieved data into Python objects which takes longer and uses more memory. Their abstraction also comes at the price of not always being clear on what is actually happening under the hood.
I am not suggesting never using an ORM. They have a lot of benefits for simple datasets such as ease of development, cross database compatibility, custom field transforms and so on. However if you don’t know SQL to begin with, your ORM will most likely be extremely inefficient, under-powered and the SQL tables will be non-normalized.
For example, one of the worst sins in SQL is to use SELECT * in production. Which is the default behavior for ORM queries. In fact, the Django main query page doesn’t even go over how to limit columns returned!
Now, I am not going to tell when to use an ORM or not, or link to a dozen other articles that fight over this point further. All I want is for everyone that thinks they should use an ORM to at least sit down and learn a moderate amount of SQL before making that decision.
Text Config is King
When was the last time you had to convince an IT admin to update a source code file? Yeah, good luck with that.
By separate settings from code you gain a lot of flexibility and safety. It’s the difference between “Mr. CEO, you need the default from email changed? So for our Django docker deploy we’ll need to make the change in code, commit it to the feature branch, PR it to develop, pray we don’t have other changes and require a hotfix for this instead, then PR again for release to master, wait for the CI/CD to finish and then the auto-deployment should be good by end of the day.” vs. “One sec, let me update the config file and restart it. Okay, good to go.”
“Code based setting files are more powerful!” – Old Sith Proverb
That’s the path to the dark side. If you think you need programmatic help in a configuration file, your logic is in the wrong place. A config file shouldn’t validate itself, not have if statements, nor have to find the absolute path to the relative directory structured the user entered.
The best configuration handling I have seen are text files that are read, sanity checked, loaded into memory and have paths extrapolated as needed, run any necessary safety checks, then any additional logic run. This allows almost anyone to update a setting safely and quickly.
The basic design is to allow for overrides on runtime based on environment variables, which is very common and good practice. However, the way this example has been implemented honestly scares me. If the json loads fails, you are going to have an exception raised, in your freaking settings file. Which may even happen before logging is setup, project dependent.
Second, say you have “APP_ALLOWED_HOSTS” (or worse “APP_DATABASES”) set to an empty string accidentally. Then the “ALLOWED_HOSTS” would be set to [""] !
Finally spending all that work to grab environment variables makes it very unclear where to actually update the settings. For example, to update the ALLOWED_HOSTS you would have to change the default in the os.environ.get which wouldn’t make much sense to someone who didn’t know Python.
The safe way
Why not a much more easily readable file, in this case a YAML file paired with python-box (lots of other options as well, this was simply fast to write and easy to digest.)
Wonderful, now we have a super easy to digest config file that would be easy to update. Let’s implement the parser for it.
import os
from box import Box
config = Box.from_yaml(filename="config.yaml")
config.debug |= os.getenv("DEBUG") == "1"
# Doing it this way will make sure it exists and is not empty
if os.getenv("ALLOWED_HOSTS"):
# This is overwritten like in settings.py above instead of extended
config.allowed_hosts = os.getenv("ALLOWED_HOSTS").split(",")
if os.getenv("APP_DATABASES"):
try:
databases = Box.from_json(filename=os.environ["APP_DATABASES"])
except Exception as err:
# Can add proper logging or print message as well / instead of exception
raise Exception(f"Could not load database set from {os.environ['APP_DATABASES']}: {err}")
else:
config.databases.extend(databases)
Tada! Now this is a lot easier to understand the logic that is happening, much easier to update the config file, as well as a crazy amount safer.
Now the only thing to argue about with your coworkers is what type of config file to use. cfg is old-school, json doesn’t support comments, yaml is slow, toml is fake ini, hocon is overcomplicated, and you should probably just stop coding if you’re even considering xml. But no matter which you go with, they are all better (except xml) than using a language based settings file.
Wrap Up
So you’ve heard me prattle on and on of the dangers Djangoers might bring with them to your next project, but also keep in mind the good things they will bring. Django helps people learn how to document well, keep a rigid project structure and has a very warm and supportive community that we should all strive to match.
Thanks for reading, and I hope you enjoyed and found this informative or at least amusing!
I’m an artist and a python developer, two things that rarely occupy the same worlds, let alone the same sentence. However, I have recently found a way to combine these two passions: Panoramas.
My current smartphone takes excellent pictures. It does a great job at figuring out colors, lighting, and focus, even in low lighting. As an artist, this is important to me because I often use my phone to snap quick pictures of a scene as a reference to take back to my studio. It’s a huge improvement in the technology I had in my hands even five years ago. There is one thing about my old phone that I miss though – its ‘panorama’ photo mode, but not because it was better.
I miss how amazingly awful it used to be, and more importantly, the freedom to make awful pictures it allowed. I’d point the lens out the window of the car as it sped along (as a passenger of course) to make jagged and confusing images of tiny bits of the landscape that the phone struggled to hodgepodge together. I’d tilt and move the phone in random directions to make weird swirls of the horizon. Even when being used ‘as directed’, it would usually struggle with focus and lighting coming up with spontaneously and wonderfully terrible photos with abstract light glare or menacing dark patches. It’s hard to explain, but sometimes as an artist, a terrible photo can be just as inspiring as those picture perfect reference pics I take with me back to the studio.
My current phone is too smart for that though, and it snatches away any joy of bad photography by making consistently beautiful and seamless panoramas. Not only that, but it accomplishes this mostly by yelling at you (“You’re going too fast!”) or by using angry arrows to make sure you can only move the phone in one direction, and then abruptly ending the photograph when you don’t cooperate. So, I did what anyone does when they get nostalgic for awful photography – I made a python script to make my own terrible panoramas.
My plan was simple. First, I would shoot short videos where my phone wouldn’t yell at me for moving, tilting, and spinning the image as much as I wanted. Next, use Python to convert each frame of the video clip to an image, crop the image into a tiny sliver out of the center of the image and then glue them all together. The results are imperfect. And gloriously so.
Side note: Although I used my smartphone to shoot some video, this script could be applied to any video. Think of the wild panoramas you could create from some Russian dash cam footage, or a GoPro strapped to a fish, or a tiny clip from the Lord of the Rings. However, this script works best on videos that are less than 10 seconds long or else it produces mile long panoramic images. Currently, I don’t bother limiting the image size at all, but theoretically I could by using one out of every five frames for instance, or by cutting down the image slice size based on video length.
The Python Image Library is the only other requirement, installed with pip.
pip3 install pillow
The script works by pulling all videos out of a source directory based on file suffix and creating a panorama for each. This could easily be modified to convert just one video at a time by removing the loops and passing the path to the desired video directly to ffmpeg.
directory = Path('my\\videos\\dir')
vids = []
for vid in directory.iterdir():
if vid.suffix.lower() in ('.mp4', '.mkv'):
vids.append(vid)
Every frame pulled out by ffmpeg is stored in a file. I delete the directory and recreate it before ffmpeg runs to delete the old frames from the last run.
for vid in vids:
shutil.rmtree("pics", ignore_errors=True)
os.makedirs("pics", exist_ok=True)
print(f'Creating panoramic {vid.stem}')
result = run(
f'ffmpeg -i {vid.absolute()} '
f'-y pics\\thumb%04d.jpg -hide_banner',
shell=True, stderr=PIPE)
result.check_returncode()
print(result.stderr.decode('utf-8'))
After it finishes pulling out all the frames, I start the panorama by creating an empty image. I need to have the dimensions of the finished image to create it. To get the final width, I multiply the number of frames ffmpeg pulled out by the width of my image slice (40 pixels). For the height, I open up one of the frames and use it size as a reference. I also use the sample image’s dimensions to figure out the center of the image for cropping everything down later.
Then, I loop through all the frame images in reverse order (because … long story short, it usually looks better that way) and then work on slicing each image down to 40 pixels wide to glue into the panorama.
sample = Image.open("pics/{}".format(os.listdir("pics")[0]))
width, height = sample.size
center = width / 2
panoramic = Image.new('RGB', (len(os.listdir("pics")*40), height))
# This offset is so PIL knows where to start adding
# each image slice to the panorama
x_offset = 0
for i in reversed(os.listdir("pics")):
img = Image.open("pics/{}".format(i))
area = (center - 20, 0, center + 20, height)
cropped_img = img.crop(area)
panoramic.paste(cropped_img, (x_offset, 0))
x_offset += 40
panoramic.save(f'{vid.stem}.jpeg')
panoramic.close()
The Painting
So far, I am quite happy with the results of this adorable little script. It has definitely given me the creative inspiration I was missing. In the past two weeks, I have done three series of paintings based on panoramas I have created using it, with plans for many more. Here’s an example of how I used it to create some artwork!
I took this video:
turned it into this panorama:
played with some paint and smudged around some charcoal and pastels:
and came up with this:
Final Thoughts
It feels really amazing to apply Python to unusual problems, even if that challenge is finding a unique way of creating original art. Plus, if the inspiration ever dries up, I have some ideas for making this script even more fun:
grab each slice from a random spot rather than dead center of the image for something much more jumbled and abstract
options to not use every frame for longer videos
PIL ‘effects’, like a black and white mode, over saturation, or extra blurry images
an ‘up and down’ mode for tall panoramas
I hope you enjoyed! Feel free to check out my website or my instagram for more artwork if you are interested.
The trek from Python 2 to Python 3 has been drawn-out, arduous and fraught with perils. How close are our dear Knights developers to all reaching the long sought glory of Python 3?
Quest for the Python 3 – Artwork by Clara Griffith (Link may contain NSFW art)
PIP Downloads
Let’s first jump into what is being used the most currently. This data examines fifteen different libraries downloaded via PIP for a particular Python version. We are only including 2.7 and 3.4+, the Python Versions that are currently supported.
The libraries analyzed are ones that have over 10K stars on github and have been downloaded via PIP. The contenders are: celery, django, flask, ipython, keras, mitmproxy, numpy, pandas, python-box, requests, scrapy, selenium, tensorflow, and tornado. (To be fair, numpy and python-box didn’t have 10K stars, but I used them in the script to make these graphics, so gave them some spotlight too.)
As of January 2019, Python 3 downloads are eclipsing Python 2 by over 20% with Python 3.6 bringing over 39% of it, almost directly matching Python 2.7’s total.
That is good, but not great news. Thankfully Python 2 won’t just stop working at the end of this year, but those are rookie Python 3 numbers, we got to pump them up!
Of course, we have to remember this is a small subset of all downloads. Subsequently, pip downloads themselves don’t tell the whole tale, but this does give us an idea of how things of are going.
This is accomplished by using the PyPI BigQuery data and some SQL (adapted from Artem Golubin’s post about this from last year), then throwing it into matplotlib.
SELECT
SUBSTR(details.python, 0, 3) as python_version,
COUNT(*) as download_count
FROM
TABLE_DATE_RANGE(
[the-psf:pypi.downloads],
DATE_ADD(CURRENT_TIMESTAMP(), -30, "DAY"),
CURRENT_TIMESTAMP()
)
WHERE
details.installer.name='pip' and
file.project = 'requests' -- change project name here
GROUP BY
python_version
ORDER BY
download_count DESC
LIMIT 100
Library Brawl: Who’s the Python 3 champs?
In this head to head, we are going to compare two similar libraries, and see how they are doing on the switch to Python 3.
Web Frameworks
The first two up are very popular web frameworks to develop in, Flask and Django.
It’s a dead heat! Both libraries are doing well at attracting developers with a fresh mindset.
Machine Learning
The most popular github package by far was tensorflow with over a hundred thousand stars. Here it’s paired against it’s younger brother keras, which actually depends on it (or other AI tools) to operate.
Machine learning needs to teach it’s developers how to update! It’s a sad day for AI.
Hacker vs Web Scraper
Okay, not really directly comparable tools with a man-in-the-middle proxy and a web scraper, but it’s still an interesting match up.
With this duo I was surprised they didn’t have a higher correlation. I was honestly expecting the mitm tool to have less Python 3 love, as a lot of “hacker” tools depend on the broken way Python 2 handles strings vs unicode, thus are hard to update.
Good job hackers, always keep your tool belt fresh! Scrapers….scrape it together.
Data Science
The last head to head is for the data scientists out there, and you got science in your name and numbers in your veins, you should be at the bleeding edge of tech!
Ouch, yinz need to get with the times.
Python Version Developers Use More Often
This is some hard to gather data as an individual, so I’m going to have to cheat and just base this information off JetBrain’s yearly state of the ecosystem reports from 2017 and 2018.
In 2017, 53% of devs reported using Python 3 as their main language, which went up 22% in 2018 to 75%. Based on those two points of data, we can come to a crystal clear, no doubt conclusion to how many developers will be using Python 3 as their main language in 2019.
That’s right, based on the past two year trend, 97% of developers should be using Python 3 in 2019.
Okay, well, maybe not. But I personal expect that number to be over 90% by the time Python 2 is EOL, which is excellent news.
Operating System Default Language
OSes have a fun time of being in the cross hairs of everyone from desktop to server users, trying to figure out the right combo of what’s best for their users and for their own technology stack going forward. Every major Linux distribution agrees Python 3 is the way to the future and they will need to change over. The hard part is deciding when it will impact the users least and best for their own release cycle. This has caused lots of headaches over the years. So where do we stand now?
OS
Python Version
Windows 10
None
OSX 10.8
2.7
Debian 9
2.7
RedHat 8*
3.6
Fedora 29
3.7
Ubuntu 19.04*
3.7
(* denotes upcoming releases this year)
Windows has the easy stance of just saying “do it yourself” and Mac is, as usual, not bothering to innovate and just hum along until it breaks. Thankfully most Linux distros, which power the internet, are either already updated or updating this year. I haven’t seen for sure that Debian 10 will be released with Python 3 or that it w ill be out before year’s end, but I would be surprised if either were not true. Then there’s Arch linux. Arch has had Python 3 as the standard for almost as long as it existed, good boy!
Are we ready?
In all honesty, we are. We are far more prepared for this than the financial sector was ready for Y2K, and we all survived that. Moreover, there are always going to be code bases that can’t update to the latest version easily, but that’s true across the entire software development world. That and the fact the Python Software Foundation has given an extended eleven years which has allowed for even the slowest of companies to have ample time to migrate to Python 3.
