Chris Griffith

Python Box 3 – The reddit release

Box has surpassed my expectations in popularity, even appearing in the Python Weekly newsletter and on Python Bytes podcast. I am very grateful for everyone that uses it and hope that you find it to be another great tool in your Python tool-belt! Also a huge shot out to everyone that has provided feedback or code contributes!

Box logoI first posted about Box about a month ago and linked to it on /r/python, which had great feedback and critique. With such interest, I have been working diligently the past few weeks to try and make it better, and I feel it confident it isn’t just better, it’s substantially improved.

Box 3 brings two huge changes to the table. First is that can take arguments to allow it to work in new and different ways, such as becoming a default_box or a  frozen_box. Second, Box does not longer deals with converting all sub dicts into Boxes upon creation, rather upon retrieval. These two things means that Box is now more powerful, customizable and faster than ever before.

New Features

 Conversion Box

This turns all those pesky keys that normally could not be accessed as an attribute into a safe string. automatically ‘fixes’ keys that can’t become attributes into valid lookups.

my_box = Box({"Send him away!": True})

my_box["Send him away!"] == my_box.Send_him_away

To be clear, this NEVER modifies the actual dictionary key, only allows attribute lookup to search for the converted name. This is the only feature that is enabled by default. Also keep it mind it does not modify case or attempt to find ascii equivalent for extended charset letters, it simply removes anything outside the allowed range.

You can also set those values through the attribute. The danger with that is if two improper strings would transform into the same attribute, such as a~b and a!b would then both equal the attribute a_b, but the Box would only modify or retrieve the first one it happened to find.

my_box.Send_him_away = False

my_box.items()
# dict_items([('Send him away!', False)])

Default Box

If you don’t like getting errors on failed lookups, or bothering to create all those in-between boxes, this is for you!

default_box = Box(default_box=True)

default_box.Not_There 
# <Box: {}>

default_box.a.b.c.d.c.e.f.g.h = "x"
# <Box: {'a': {'b': {'c': {'d': {'c': {'e': {'f': {'g': {'h': 'x'}}}}}}}}}>

You can specify what you want the default return value to be with default_box_attr which by default is a Box class. If something callable is specified, it will return it as an instantiated version, otherwise it will return the specified result as is.

bx3 = Box(default_box=True, default_box_attr=3)

bx3.hello
# 3

However it now only acts like a standard default dict, and you do lose the magical recursion creation.

bx3.hello.there
# AttributeError: 'int' object has no attribute 'there'

Frozen Box

Brrrrr, you feel that? It’s a box that’s too cool to modify. Handy if you want to ensure the incoming data is not mutilated during it’s usage. An immutable box cannot have data added or delete,  like a tuple it can be hashed if all content is immutable, but if it has mutable content can still modify those objects and is unhashable.

frigid_box = Box({"Kung Fu Panda": "Pure Awesome"}, frozen_box=True)

frigid_box.Kung_Fu_Panda = "Overrated kids movie"
# Traceback (most recent call last):
# ...
# box.BoxError: Box is frozen

hash(frigid_box)
# -156221188

Camel Killer Box

DontYouHateCamelCaseKeys? Well I do too, kill them with fire camel_killer_box=True.

snake_box = Box({"BadKey": "silly value"}, camel_killer_box=True) 
assert snake_box.bad_key == "silly_value"

This transformation, including lower casing, is applied to everything. Which means the above example for just conversion box would also be lower case.

snake_box["Send him away!"] == snake_box.send_him_away

Box It Up

Force the conversion of all sub objects on creation, just like old times. This is useful if you expect to be referencing every value, and every sub-box value and want to front load the time at creation instead of during operations. Also a necessity if you plan to keep the reference or re-use the original dictionary that was boxed.

boxed = Box({"a": {"b": [{"c": "d"}]}}, box_it_up=True)

 

Dangerous Waters

As stated above, sub box objects are now not recreated until they are retrieved or modified. Which means if you have references to the original dict and modify it, you may inadvertently be updating the same objects that are in the box.

a = {"a": {"b": {"c": {}}}}
a_box = Box(a)
a_box 
# <Box: {'a': {'b': {'c': {}}}}>

a["a"]["b"]["d"] = "2"

a_box
# <Box: {'a': {'b': {'c': {}, 'd': '2'}}}>

So if you plan to keep the original dict around, make sure to box_it_up or do a deepcopy first.

safe_box = Box(a, box_it_up=True)
a["a"]["b"]["d"] = "2"

safe_box
# <Box: {'a': {'b': {'c': {}}}}>

Obviously not an issue if you are creating it from scratch, or with a new and unreferenced dict, just something to be aware of.

Speed and Memory Usage

Something a lot of people were worried about was speed and memory footprint compared to alternatives and the built-in dict. With the previous version of Box, I was more of the mindset that it lived to be a scripter, sysadmin or developer aid in writing in the code, and less performant worried. Simply put, it was foolish to think that way. If I really want this to be the go to dot notation library, it needs to be as fast as, if not faster, than it’s peers while still providing more functionality.

Because Box is the only dot notation library that I know of that works by converting upon retrieval, it is hard to do a true 1 to 1 test verse the others. To the best of my abilities, I have attempted to capture the angles that show where the performance differences are most noticeable.

This test is using a modified version of HearthStone card data, available on github, which is 6.57MBs and has 2224 keys, with sub dictionaries and data. To gather metrics, I am using reusables.time_it and memory_profiler.profile and the file can be run yourself. Box is being compared against the two most applicable competitors I know of, addict and DotMap (as Bunch and EasyDict do not account for recursion or have other common features), as well as the built in dict type. All charts except the memory usage are in seconds.

Lower is always better, these are speed and memory footprint charts

Creation and Conversion

So first off, how long does it take to convert a JSON file to the object type, and back again.

Figure 1: Convert JSON file to object and back to dict

 

The load is the first thing done, between then and the transformation back into a dictionary, all first level items are retrieved and 1000 new elements were added to the object. Thanks to the new style of only transforming sub items upon retrieval, it greatly speeds up Box, making very comparative to the built in dict type, as seen in figure 1. 

Load code comparison:

# addict
with open("hearthstone_cards.json", encoding="utf-8") as f:
    ad = Dict(json.load(f))

# DotMap
with open("hearthstone_cards.json", encoding="utf-8") as f:
    dm = DotMap(json.load(f))

# Box
bx = Box.from_json(filename="hearthstone_cards.json", encoding="utf-8")

# dict
with open("hearthstone_cards.json", encoding="utf-8") as f:
    dt = json.load(f)

Value Lookup

The load speed up does come at the price of having a heavier first initial lookup call for each value. The following graph, figure 2, shows a retrieval of all 2224 first level values from the object.

Figure 2: Lookup of 2224 first level values

 

As expected, Box took far and away the most time, but only on the initial lookup, and we are talking about a near order of magnitude less than the creation times for the other libraries.

The lookup is done the same with every object type.

for k in obj:
    obj[k]

Inserting items

Inserting a 1000 new entities into the objects deals with very small amounts of time all around.

Figure 3: Inserting a 1000 new dict objects that will be accessible as those native object types

 

Insert looks the most different from the code, as the different objects behave differently on insertion. Box will insert it as a plain dict, and not convert it until retrieval. Addict will not automatically make new items into addict.Dicts, causing the need to manually. DotMap behaves as a default dict out of the box, so we can simply access the attribute we want to create directly off the bat.

# addict 
for i in range(1000):
    ad["new {}".format(i)] = Dict(a=i)

# DotMap
for i in range(1000):
    dm["new {}".format(i)]['a'] = i

# Box
for i in range(1000):
    bx["new {}".format(i)] = {'a': i}

# dict
for i in range(1000):
    dt["new {}".format(i)] = {'a': i}

Total Times

To summarize the times, dict is obviously the fastest, but Box is in a comfortable second fastest.

Figure 4: The total amount of seconds for all above scenarios

 

Memory Footprint

What I was most surprised with was memory usage. I have not had a lot of experience measuring this, so I leave room open for something obvious I am missing, but on first look, it seems that Box uses less memory than even the built in dict. 

Figure 5: Total memory usage in MBs

 

"""
        Line #    Mem usage    Increment   Line Contents
        ================================================
        85     28.9 MiB      0.0 MiB   @profile()
        86                             def memory_test():
        87     41.9 MiB     13.0 MiB       ad = load_addict()
        88     57.1 MiB     15.2 MiB       dm = load_dotmap()
        89     64.5 MiB      7.4 MiB       bx = load_box()
        90     74.6 MiB     10.1 MiB       dt = load_dict()
        91                             
        92     74.6 MiB      0.0 MiB       lookup(ad)
        93     74.6 MiB      0.0 MiB       lookup(ad)
        94                             
        95     74.6 MiB      0.0 MiB       lookup(dm)
        96     74.6 MiB      0.0 MiB       lookup(dm)
        97                             
        98     75.6 MiB      1.0 MiB       lookup(bx)
        99     75.6 MiB      0.0 MiB       lookup(bx)
       100                             
       101     75.6 MiB      0.0 MiB       lookup(dt)
       102     75.6 MiB      0.0 MiB       lookup(dt)
       103                             
       104     76.0 MiB      0.3 MiB       addict_insert(ad)
       105     76.6 MiB      0.6 MiB       dotmap_insert(dm)
       106     76.8 MiB      0.2 MiB       box_insert(bx)
       107     76.9 MiB      0.2 MiB       dict_insert(dt)
"""

 

Questions I can answer

How does it work like a traditional dict and still take keyword arguments?

You can create a Box object just like you would create a dict as normal, just those new arguments will not be part of dict object (they are stored in a hidden ._box_config attribute). They were also made rather unique, such as ‘conversion_box’ and ‘frozen_box’ so that it would be very rare for others to be using those kwargs.

Is it safe to wait to convert dictionaries until retrieval?

As long as you aren’t storing malformed objects that subclassed dict, yes.

Python Decorators

Often times in python there comes a need to have multiple functions act in a similar manner. It could be anything from making sure that similar functions output the right type of result, they all log when they are called or all report exceptions in the same way.

decorators An easy and repeatable way to accomplish this is . They look like:

@decorator
def my_function(print_string="Hello World"):
    print(print_string)

my_function()
# Hello World

Decorators are simply wrapping a function with another function. In other words, it is exactly the same as doing:

def my_function(print_string="Hello World"):
    print(print_string)

decorator(my_function)("My message")
# My message

So what does a decorator look like?

Decorator Template

from functools import wraps

def decorator(func):
    """This is an example decorators"""
    @wraps(func)
    def container(*args, **kwargs):
        # perform actions before running contained function
        output = func(*args, **kwargs)
        # actions to run after contained function
        return output
    return container

Running the code above does nothing extra currently. It is showing how a decorator runs another function within itself.

@decorator
def my_function():
    print "Hello World"

my_function()
# "Hello World"

Line by line breakdown

def decorator(func):

The name of the decorator function, which takes the wrapped function as its single argument.

@wraps(func)

Wraps is a built-in method that replaces the decorators name and docstring with that of the wrapped function, the section after the line by line breakdown explains why this is necessary.

def container(*args, **kwargs):

This inner function collects the parameters and keyword parameters that are going to be passed to the original function. This allows the decorator access to incoming arguments to verify or modify before the function is ever run.

output = func(*args, **kwargs)

This runs the function with the original arguments and captures the output. It is also possible to return a completely different result. It is more common to check or modify the output, like if you wanted to make sure everything returned was an integer.

return output

Don’t forget to actually return the original function’s output (or a custom one). Otherwise the function will simply return None.

return container

The container function is the actual function being called,  hence why *args, **kwargs are passed to it. It is necessary to return it from the outside decorator so it can be called.

The importance of Wraps

We need to incorporate wraps so that the function name and docstring appear to be from the wrapped function, and not those of the wrapper itself.

@decorator
def my_func():
    """Example function"""
    return "Hello"

@decorator_no_wrap
def second_func():
    """Some awesome docstring"""
    return "World"

help(my_func)
# Help on function my_func:

# my_func()
#    Example function

help(second_func)
# Help on function container:

# container(*args, **kwargs)

It is possible, though more work to accomplish the same thing yourself.

def decorator(func):
    """This is an example decorators"""
    def container(*args, **kwargs):
        return func(*args, **kwargs)
    container.__name__ = func.__name__
    container.__doc__ = func.__doc__
    return container

Useful Example

Now lets turn it into something useful. In this example we will make sure that the function returns the expected type of result. Otherwise it will raise an exception for us so there are not hidden complications down the road.

from functools import wraps

def isint(func):
    """ 
    This decorator will make sure the resulting value 
    is a integer or else it will raise an exception.
    """
    @wraps(func)
    def container(*args, **kwargs):
        output = func(*args, **kwargs)
        if not isinstance(output, int):
            raise TypeError("function did not return integer")
        return output
    return container

@isint
def add(num1, num2):
    """Add two numbers together and return the results"""
    return num1 + num2

print(add(1, 2))
# 3

print(add("this", "that"))
# Type Error: function did not return integer

Regular decorators are already called on execution, so you do not need to add ()s after their name, such as @isint. However, if the decorator accepts arguments, aka a meta-decorator, it will require () even if nothing additional is passed to it.

Meta-decorators

Passing arguments to a decorator turns it into a Meta-decorator. To pass these arguments in, it requires either another function wrapped around the decorator or turn the entire thing into a class.

from functools import wraps

def istype(instance_type=int):
    def decorator(func):
        """ 
        This decorator will make sure the resulting value is the 
        type specified or else it will raise an exception. 
        """
        @wraps(func)
        def container(*args, **kwargs):
            output = func(*args, **kwargs)
            if not isinstance(output, instance_type):
                raise TypeError("function did not return proper type")
            return output
        return container
    return decorator

@istype()
def add(num1, num2):
    """Add two numbers together and return the results"""
    return num1 + num2

@istype(str)
def reverse(forward_string):
    """Reverse and return incoming string"""
    return forward_string[::-1]


print(add(1, 2))
# 3

print(reverse("Hello"))
# "olleH"

print(add("this", "that"))
# Type Error: function did not return proper type

Remember running a decorator is equivalent to:

decorator(my_function)("My message")

Running a meta-decorator adds an additional layer.

reversed_string = istype(str)(reverse)("Reverse Me")

Hence why @decorator doesn’t require to be called when put above a function, but @istype() does.

You can also create this meta-decorator as a class instead of another function.

class IsType: # Instead of 'def istype'

    def __init__(self, inc_type=int):
        self.inc_type = inc_type

    def __call__(self, func): # Replaces 'def decorator(func)'
        @wraps(func)
        def container(*args, **kwargs):
            output = func(*args, **kwargs)
            if not isinstance(output, self.inc_type):
                raise TypeError("function did not return proper type")
            return output
        return container

In functionality they are the same, but be aware they are technically different types. This will really only impact code inspectors and  those trying to manually navigate code, so it is not a huge issue, but is something to be aware of.

type(IsType)
<class 'type'>

type(istype)
<class 'function'>

 Things to keep in mind

  1. Order of operation does matter. If you have multiple decorators around a single function keep in mind they are run from top to bottom
  2. Once a function is wrapped, it cannot be run without the wrapper.
  3.  Meta-decorators require the additional parentheses, regular decorators do not.

Introducing Box – Python dictionaries with recursive dot notation access

Box logo

Everyone loves Python’s dictionaries; they’re fast, easy to create and quite handy for a range of reasons. However, there are times that ["typing"]["out"]["all"]["those"] extra quotes and  brackets seems excessive. Wouldn’t it be nicer to access.them.like.class.methods?

Say hello to box.

Box logo

from box import Box

movie_data = {
  "movies": {
    "Spaceballs": {
      "imdb_stars": 7.1,
      "rating": "PG",
      "length": 96,
      "Director": "Mel Brooks",
      "Stars": [{"name": "Mel Brooks", "imdb": "nm0000316", "role": "President Skroob"},
                {"name": "John Candy","imdb": "nm0001006", "role": "Barf"},
                {"name": "Rick Moranis", "imdb": "nm0001548", "role": "Dark Helmet"}
      ]
    },
    "Robin Hood: Men in Tights": {
      "imdb_stars": 6.7,
      "rating": "PG-13",
      "length": 104,
      "Director": "Mel Brooks",
      "Stars": [
                {"name": "Cary Elwes", "imdb": "nm0000144", "role": "Robin Hood"},
                {"name": "Richard Lewis", "imdb": "nm0507659", "role": "Prince John"},
                {"name": "Roger Rees", "imdb": "nm0715953", "role": "Sheriff of Rottingham"},
                {"name": "Amy Yasbeck", "imdb": "nm0001865", "role": "Marian"}
      ]
    }
  }
}


my_box = Box(movie_data)

my_box.movies.Spaceballs.rating
'PG'

my_box.movies.Spaceballs.Stars[0].name
'Mel Brooks'

my_box.movies.Spaceballs.Stars[0]
# <Box: {'name': 'Mel Brooks', 'imdb': 'nm0000316', 'role': 'President Skroob'}>

Box is a creation I made over three years ago, originally in the reusables code base named Namespace, inspired by JavaScript Object access methods.

Install is super simple:

pip install python-box

Or just grab the file box.py directly from the github project.

Every Box is usable as a drop in replacement to dictionaries in 99%* of cases. And every time you add a dictionary or list to a Box object, they become Box (subclass of dict) or BoxList (subclass of list) objects as well.

type(my_box)
# box.Box
assert isinstance(my_box, dict)

type(my_box.movies.Spaceballs.Stars)
# box.BoxList
assert isinstance(my_box.movies.Spaceballs.Stars, list)

my_box.movies.Spaceballs.Stars[0].additional_info = {'Birth name': 'Melvin Kaminsky', 'Birthday': "05/28/1926"}

my_box.movies.Spaceballs.Stars[0].additional_info
# <Box: {'Birth name': 'Melvin Kaminsky', 'Birthday': '05/28/1926'}>

At any level you can change a Box object back into a standard dictionary.

my_box.movies.Spaceballs.to_dict()

{'Director': 'Mel Brooks',
 'Stars': [
  {'additional_info': {'Birth name': 'Melvin Kaminsky', 'Birthday': '05/28/1926'},
   'imdb': 'nm0000316',
   'name': 'Mel Brooks',
   'role': 'President Skroob'},
  {'imdb': 'nm0001006', 'name': 'John Candy', 'role': 'Barf'},
  {'imdb': 'nm0001548', 'name': 'Rick Moranis', 'role': 'Dark Helmet'},
  {'imdb': 'nm0000597', 'name': 'Bill Pullman', 'role': 'Lone Starr'}],
 'imdb_stars': 7.1,
 'length': 96,
 'rating': 'PG'}

You can also run to_list() on lists in the Box to return them to a standard list, with all inner Box and BoxList objects transformed back to normal.

Box also has built in functions for dealing with json and yaml**.

my_box.movies.Spaceballs.to_json()

# {
#    "imdb_stars": 7.1,
#    "rating": "PG",
#    "length": 96,
#    "Director": "Mel Brooks",
#    "Stars": [
# ...


my_box.movies.Spaceballs.to_yaml()

# Director: Mel Brooks
# imdb_stars: 7.1
# length: 96
# rating: PG
# Stars:
# - imdb: nm0000316
#   name: Mel Brooks
#   role: President Skroob
# ...


Calling a Box object will return it’s keys. It’s also possible to access the attributes the standard dictionary method, which is required for keys that are numeric or have spaces.

my_box.movies()
# ('Spaceballs', 'Robin Hood: Men in Tights')

my_box.movies['Robin Hood: Men in Tights']
# <Box: {'imdb_stars': 6.7, 'rating': 'PG-13', 'length': 104, ...

Unlike addict it does not act as a default dictionary, so you will get built-in errors if you try to access something that isn’t there.

my_box.tv_shows

# Traceback (most recent call last):
# ...
# AttributeError: tv_shows

Another power previously mentioned is that you can add dictionaries into lists and they will automatically be converted into Box objects.

my_box.movies.Spaceballs.Stars.append(
    {"name": "Bill Pullman", "imdb": "nm0000597", "role": "Lone Starr"})

my_box.moves.Spaceballs.Stars[-1].name
'Bill Pullman'

It also protects itself from having its functions overwritten accidentally.

my_box.to_dict = '3'
# AttributeError: Key name 'to_dict' is protected

Box is also a substitute for the Namespace used by argparse, making it super easy to convert incoming arguments to a dict if wanted. This allows incoming arguments to be easily passed to function arguments.

import argparse
from box import Box

parser = argparse.ArgumentParser()
parser.add_argument('floats', metavar='N', type=float, nargs='+')
parser.add_argument("-v", "--verbosity", action="count", default=0)

args = parser.parse_args(['1', '2', '3', '-vv'], namespace=Box())

print(args.to_dict())
{'floats': [1.0, 2.0, 3.0], 'verbosity': 2}


def example_func(floats, verbosity):
    print(verbosity)

example_func(**args)
2

If you have any questions, suggestions or feedback, please open a github issue and let me know!

Hope you enjoy!

Caveats

*  Based off nothing but pure guess and personal experience. Only time drop in replacement doesn’t work is when converting or dumping. So make sure do use  first for those cases.  

** If you don’t have PyYAML installed, the to_yaml function will not be available.

Run, Subprocess, Run!

Python is awesome, and can pretty much do everything you ever wanted, but on rare occasion, you may want to call an external program. The original way to do this with Python was to use os.system. 

import os

return_code = os.system("echo 'May the force be with you'")

The message “May the force be with you” would be printed to the terminal via stdout, and the return code variable would be 0 as it did not error. Great for running a program, not so great if you need to capture its output.

So the Secret Order of the Pythonic Brotherhood* meet, performed the required rituals to appease our Benevolent Dictator for Life**, and brought fourth subprocess.                                                                                                         * not real  ** real 

Subprocess is a module dedicated to running other processes. You’ve probably already have used or encountered it in it’s many forms. subprocess.call , subprocess.check_callsubprocess.check_output or even the direct call to the process constructor subprocess.Popen.

These made life a lot easier, as you could now have easy interaction with the process pipes, running it as a direct call or opening a new shell first, and more. The downside was the inconvenience of having to switch between them depending on needs, as they all had different outputs considering how you interacted with them.

That all changed in Python 3.5, with the introduction of subprocess.run (for older versions check out reusables.run). It is simply the only subprocess command you should ever need! Let’s look at a quick example.

import subprocess

response = subprocess.run("echo 'Join the Dark Side!' 1>&2", 
                          stderr=subprocess.PIPE)

# CompletedProcess(args="echo 'Join the Dark Side!' 1>&2",
#                  returncode=0, 
#                  stderr=b"'Join the Dark Side!' \r\n")

Now check that response out. It’s an organized class, that stores what args you sent to start the subprocess, the returncode as well as stdout and/or stderr if there was a pipe specified for them. (If something was sent to stdout or stderr and there wasn’t a pipe specified, it would send it to the current terminal.)

As the return value is a class, you can access any of those attributes as normal.

print(response.stderr)
# Join the Dark Side!
print(response.returncode)
# 0
response.check_returncode() # Would return None in this case

It also includes a check_returncode function that will raise subprocess.CalledProcessError if the return code is not 0. 

Basically, you should use subprocess.run and never look back.  It’s only real limitation is that it is equivalent to using Popen(...).communicate() , which means you cannot provide multiple inputs, wait for certain output, or behave interactively in any manner.

There are plenty of additional capabilities that are good to know, this article will cover:

  1. Timeouts
  2. Shell
  3. Passing arguments as string or list
  4. Pipes and Buffers
  5. Input
  6. Working Directory
  7. Environment Variables

Timeout

In Python 2 it’s a real pain to have a timeout for a subprocess. You could potentially do a poll for a max amount of time before calling it quits. But if you had input it was much harder. On Linux you could use signals, but Windows required either a forever running background thread or run in a separate process.

Thankfully in the modern world we can simply specify one to the run command.

subprocess.run("ping 127.0.0.1", shell=True, timeout=1)

You’ll see some ping responses being printed to the terminal (as we didn’t send it to a pipe) then in a second (literally) see a traceback.

subprocess.TimeoutExpired: Command 'ping 127.0.0.1' timed out after 1 seconds

No crazy multiprocessing or signaling needed, only need to pass a keyword argument.

Shell

I see shell being overused and misunderstood a lot, so I want to define it’s behavior very clearly here. When shell=False  is set, there is no system shell started up., so the first argument must be a path to an executable file or else it will fail.

Setting shell=True will first spin up a system dependent shell process (commonly \bin\sh on Linux or cmd.exe on Windows) and run the command within it. With a shell you can use environment variables, shell built-in commands and have glob “*” expansion.

Also keep in mind a lot of programs are actual files on Linux, whereas they are shell built-ins on Windows. That’s why “echo” with shell=False will work on Linux but will break on Windows:

subprocess.run(["echo", "hi"])

# Linux: CompletedProcess(args=['echo', 'hi'], returncode=0)
# Windows: FileNotFoundError: 
#          [WinError 2] The system cannot find the file specified

“So, just always use shell?” Nope, it’s actually better to avoid it whenever possible. It’s costly, aka slower, to spin up a new shell, and it’s susceptible to shell injection vulnerabilities.

If you are going to be calling an executable file, it’s best to always keep shell=False unless you need one of the shell’s features.

Arguments as string or list

There seems to be very odd behavior with the first argument being passed to subprocess functions, including .run that changes from a list to a string if you use shell=True . In general if shell=False  (the default behavior) pass in a list of arguments. If shell=True, then pass in a string.

subprocess.run(['echo', 'howdy'])             # List when shell=False

subprocess.run('echo "howdy"', shell=True)    # String when shell=True

However it’s important to know why you should do that. It’s because of because of how Python has to create the new processes and send them information, which differs across operating systems.

On Windows you can get away with murder pass either a list or string for either case.  Because when creating a new process, Python has to interpret the list of arguments into a string anyways when shell=False; Otherwise, when shell=True, the string is sent directly to the shell as-is.

On Linux, the same scenario happens when shell=True. The string will be passed directly to the newly spawned shell as is, so it can expand globs and environment variables.  However, if a list is passed, it is sent as positional arguments to the shell. So if you have:

subprocess.run(['echo', 'howdy'], shell=True)

It is not sending “howdy” as an argument to echo , but rather to /bin/sh.

/bin/sh -c "echo" "howdy"

Which will result in confusing behavior of nothing being returned to stdout and no error.

And going the other direction can be a pain on Linux. When shell=False and a string is provided, the entire thing is treated as the path to the program. Which is helpful if you want to run something without passing any arguments, but can be confusing at first when it  returns a FileNotFoundError .

subprocess.run('echo "howdy"')

# FileNotFoundError: [Errno 2] No such file or directory: 'echo "howdy"'

So to be safe, simply remember:

subprocess.run(['echo', 'howdy'])             # List when shell=False

subprocess.run('echo "howdy"', shell=True)    # String when shell=True

You can also “cheat” by always building a string, then use shlex.split on it if you don’t need to use a shell.

import shlex

args = shlex.split("conquer --who 'mine enemy'" 
                   "--when 'Sometime in the next, eh, \"6\" minutes'")

print(args)
# ['conquer ', '--who', 'mine enemy', 
#  '--when', 'Sometime in the next, eh, "6" minutes']


subprocess.run(args)

(Note that shlex.split should also be sent posix=False when being used on Windows)

Stream, Pipes and Buffers

Pipes and buffers are both sections of memory used for the storage and exchange of data between processes. Pipes are designed to transfer and hold the data, while buffers act as temporary vessels to transfer data to files.

Setting stdout or stderr streams to subprocess.PIPE will save any output from the program to memory, and then stored in the CompletedProcess class under the corresponding attribute name.  If you do not set them, they will write to the corresponding default file descriptors, which are same as sys.stdout (aka file descriptor 1) and so on. So if you redirect sys.stdout the subprocess stdout will also be redirected there.

Another common use case is to send their output to a file, i.e:

subprocess.run('sh info_gathering.sh', 
               stdout=open('comp_info.txt', 'w'), 
               shell=True, 
               encoding='utf-8', 
               bufsize=4096)

That way the output is stored into a file, using a buffer to temporarily hold information in memory until a large enough section is worth writing to the file.

If encoding is specified (Python 3.6+), the incoming bytes will be decoded and the buffer will be treated as text, aka “text mode”. This can also happen if either errors or universal_newlines keyword arguments are specified.

There are multiple different ways to use buffering:

bufsizeshorthand description
0unbufferedData will be directly written to file
1line bufferedText mode only, will write out buffer on `\n`
-1system defaultLine buffered if text mode, otherwise will generally be 4096 or 8192
>= 1sized bufferWrite out when (approx) that amount of bytes are in the buffer

Input

With run there is a simple keyword argument of input  the same as Popen().communicate(input) . It is a one time dump to standard input of the new process, which can read any of it at it’s choosing. However it is not possible to wait for certain output for event before sending input, that is more suited to pexpect or similar.

Check

This allows run to be a drop in replacement of check_call. It makes sure the status return code is 0, or will raise subprocess.CalledProcessError.

s.run('exit 1', check=True, shell=True)
# subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1.
s.check_call('exit 1', shell=True)
# subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1.

Working directory

To start the process from a certain directory, pass in the argument cwd with the directory you want to start at.

s.run('dir', shell=True, cwd="C:\\")
# Volume in drive C has no label.
# Volume Serial Number is 58F1-C44C
# Directory of C:\

Environment Variables

To send environment variables to the program, it needs to be run in a shell.

s.run('echo "hey sport, here is a %TEST_VAR%"', 
      shell=True, 
      env={'TEST_VAR': 'fun toy'})

"hey sport, here is a fun toy"
# CompletedProcess(args='echo "hey sport, here is a %TEST_VAR%"',
#                  returncode=0)

It is not uncommon to want to pass the current environment variables as well as your own.

s.run('echo "hey sport, here is a %TEST_VAR%. Being run on %OS%"', 
      shell=True, 
      env=dict(os.environ, TEST_VAR='fun toy'))

"hey sport, here is a fun toy. Being run on Windows_NT"
# CompletedProcess(args='echo "hey sport, here is a %TEST_VAR%. Being run on %OS%"',
#                  returncode=0)

 

 

Reusables – Part 1: Overview and File Management

Reusables 0.8 has just been released, and it’s about time I give it a proper introduction.

I started this project three years ago, with a simple goal of keeping code that I inevitably end up reusing grouped into a single library. It’s for the stuff that’s too small to do well as it’s own library, but common enough it’s handy to reuse rather than rewrite each time.

It is designed to make the developer’s life easier in a number of ways. First, it requires no external modules, it’s possible to supplement some functionality with the modules specified in the requreiments.txt file, but are only required for specific use cases; for example: rarfile is only used to extract, you guessed it, rar files.

Second, everything is tested on both Python 2.6+ and Python 3.3+, also tested on pypy. It is cross platform compatible Windows/Linux, unless a specific function or class specifies otherwise.

Third, everything is documented via docstrings, so they are available at readthedocs, or through the built-in help() command in python.

Lastly, all functions and classes are all available at the root level (except CLI helpers), and can be broadly categorized as follows:

  • File Management
    • Functions that deal with file system operations.
  • Logging
    • Functions to help setup and modify logging capabilities.
  • Multiprocessing
    • Fast and dynamic multiprocessing or threading tools.
  • Web
    • Things related to dealing with networking, urls, downloading, serving, etc.
  • Wrappers
    • Function wrappers.
  • Namespace
    • Custom class to expand the usability of python dictionaries as objects.
  • DateTime
    • Custom datetime class primarily for easier formatting.
  • Browser Cookie Management
    • Find, extract or modify cookies of Firefox and Chrome on a system.
  • Command Line Helpers
    • Bash analogues to help system admins perform faster operations from inside an interactive python shell.

In this overview, we will cover:

  1. Installation
  2. Getting Started
  3. File, Folder and String Management
    1. Find Files Fast
    2. Archives (Extraction and Compression)
    3. Run Command
    4. File Hashing
    5. Finding Duplicate Files
    6. Safe File and Folder Names
    7. Touch (ing a file)
    8. Simple JSON and CSV
    9. Cut (ing a string into equal lengths)
    10. Config to dictionary

Installation

Very straightforward install, just do a simple pip or easy_install from PyPI.

pip install reusables

OR

easy_install reusables

If you need to install it on an offline computer, grab the appropriate Python 2.x or 3.x wheel from PyPI, and just pip install it directly.

There are no additional modules required for install, so if either of those don’t work, please open an issue at github.

Getting Started

import reusables 

reusables.add_stream_handler('reusables', level=10)

The logger’s name is ‘reusables’, and by default does not have any handlers associated with it. For these examples we will have logging on debug level, if you aren’t familiar with logging, please read my post about logging.

File, Folder and String Management

Everything here deals with managing something on the disk, or strings that relate to files. From checking for safe filenames to saving data files.

I’m going to start the show off with my most reused function, that is also one of the most versatile and powerful, find_files. It is basically an advanced implementation of os.walk.

Find Files Fast

reusables.find_files_list("F:\\Pictures",
                              ext=reusables.exts.pictures, 
                              name="sam", depth=3)

# ['F:\\Pictures\\Family\\SAM.JPG', 
# 'F:\\Pictures\\Family\\Family pictures - assorted\\Sam in 2009.jpg']

With a single line, we are able to search a directory for files by a case insensitive name, a list (or single string) of extensions and even specify a depth.  It’s also really fast, taking under five seconds to search through 70,000 files and 30,000 folders, taking just half a second longer than using the windows built in equivalent dir /s *sam* | findstr /i "\.jpg \.png \.jpeg \.gif \.bmp \.tif \.tiff \.ico \.mng \.tga \.xcf \.svg".

If you don’t need it as a list, use the generator itself.

for pic in reusables.find_files("F:\\Pictures", name="*chris *.jpg"):
    print(pic)

# F:\Pictures\Family\Family pictures - assorted\Chris 1st grade.jpg
# F:\Pictures\Family\Family pictures - assorted\Chris 6th grade.jpg
# F:\Pictures\Family\Family pictures - assorted\Chris at 3.jpg

That’s right, it also supports glob wildcards. It even supports using the external module scandir for older versions of Python that don’t have it nativity (only if enable_scandir=True is specified of course, its one of those supplemental modules). Check out the full documentation and more examples at readthedocs.

Archives

Dealing with the idiosyncrasies between the compression libraries provided by Python can be a real pain. I set out to make a super simple and straight forward way to archive and extract folders.

reusables.archive(['reusables',    # Folder with files 
                   'tests',        # Folder with subfolders
                   'AUTHORS.rst'], # Standalone file
                   name="my_archive.bz2")

# 'C:\Users\Me\Reusables\my_archive.bz2'

It will compress everything, store it, and keep folder structure in the archives.

To extract files, it is very similar behavior. Given a ‘wallpapers.zip’ file like this:

It is trivial to extract it to a location without having to specify it’s archive type.

reusables.extract("wallpapers.zip",
                  path="C:\\Users\\Me\\Desktop\\New Folder 6\\")
# ... DEBUG File wallpapers.zip detected as a zip file
# ... DEBUG Extracting files to C:\Users\Me\Desktop\New Folder 6\
# 'C:\\Users\\Me\\Desktop\\New Folder 6'

We can see that it extracted everything and again kept it’s folder structure.

The only support difference between the two is that you can extract rar files if you have installed rarfile and dependencies (and specified enable_rar=True), but cannot archive them due to licensing.

Run Command

Ok, so it many not always deal with the file system, but it’s better here than anywhere else. As you may or may not know, in Python 3.5 they released the excellent subprocess.run which is a convenient wrapper around Popen that returns a clean CompletedProcess class instance. reusables.run is designed to be a version agnostic clone, and will even directly run subprocess.run on Python 3.5 and higher.

reusables.run("cat setup.cfg", shell=True)

# CompletedProcess(args='cat setup.cfg', returncode=0, 
#                 stdout=b'[metadata]\ndescription-file = README.rst')

It does have a few subtle differences that I want to highlight:

  • By default, sets stdout and stderr to subprocess.PIPE, that way the result is always is in the returned CompletedProcess instance.
  • Has an additional copy_local_env argument, which will copy your current shell environment to the subprocess if True.
  • Timeout is accepted, buy will raise a NotImplimentedError if set on Python 2.x.
  • It doesn’t take positional Popen arguments, only keyword args (2.6 limitation).
  • It returns the same output as Popen, so on Python 2.x stdout and stderr are strings, and on 3.x they are bytes.

Here you can see an example of copy_local_env  in action running on Python 2.6.

import os

os.environ['MYVAR'] = 'Butterfly'

reusables.run("echo $MYVAR", copy_local_env=True, shell=True)

# CompletedProcess(args='echo $MYVAR', returncode=0, 
#                 stdout='Butterfly\n')

File Hashing

Python already has nice hashing capabilities through hashlib, but it’s a pain to rewrite the custom code for being able to handle large files without a large memory impact.  Consisting of opening a file and iterating over it in chunks and updating the hash. Instead, here is a convenient function.

reusables.file_hash("reusables\\reusables.py", hash_type="sha")

# '50c5425f9780d5adb60a137528b916011ed09b06'

By default it returns an md5 hash, but can be set to anything available on that system, and returns it in the hexdigest format, if the kwargs hex_digest is set to false, it will be returned as bytes.

reusables.file_hash("reusables\\reusables.py", hex_digest=False)

# b'4\xe6\x03zPs\xf5\xe9\x8dX\x9c/=/<\x94'

Starting with python 2.7.9, you can quickly view the available hashes directly from hashlib via hashlib.algorithms_available.

# CPython 3.6
import hashlib

print(f"{hashlib.algorithms_available}")
# {'sha3_256', 'MD4', 'sha512', 'sha3_512', 'DSA-SHA', 'md4', ...

reusables.file_hash("wallpapers.zip", "sha3_256")

# 'b7c357d582f8932977d785a24f728b267cef1de87537076aadac5049f4e4fa70'

Duplicate Files

You know you’ve seen this picture  before, you shouldn’t have to safe it again, where did that sucker go? Wonder no more, find it!

list(reusables.dup_finder("F:\\Pictures\\20131005_212718.jpg", 
                          directory="F:\\Pictures"))

# ['F:\\Pictures\\20131005_212718.jpg',
#  'F:\\Pictures\\Me\\20131005_212718.jpg',
#  'F:\\Pictures\\Personal Favorite\\20131005_212718.jpg']

dup_finder is a generator that will search for a given file at a directory, and all sub-directories. This is a very fast function, as it does a three step escalation to detect duplicates, if a step does not match, it will not continue with the other checks, they are verified in this order:

  1. File size
  2. First twenty bytes
  3. Full SHA256 compare

That is excellent for finding a single file, but how about all duplicates in a directory? The traditional option is to create a dictionary of hashes of all the files to compares against. It works, but is slow. Reusables has directory_duplicates function, which first does a file size comparison first, and only moves onto hash comparisons if the size matches.

reusables.directory_duplicates(".")

# [['.\\.git\\refs\\heads\\master', '.\\.git\\refs\\tags\\0.5.2'], 
#  ['.\\test\\empty', '.\\test\\fake_dir']]

It returns a list of lists, each internal list is a group of matching files.  (To be clear “empty” and “fake_dir” are both empty files used for testing.)

Just how much faster is it this way? Here’s a benchmark on my system of searching through over sixty-six thousand (66,000)  files in thirty thousand (30,000) directories.

The comparison code (the Reusables duplicate finder is refereed to as ‘size map’)

import reusables

@reusables.time_it(message="hash map took {seconds:.2f} seconds")
def hash_map(directory):
    hashes = {}
    for file in reusables.find_files(directory):
        file_hash = reusables.file_hash(file)
        hashes.setdefault(file_hash, []).append(file)

    return [v for v in hashes.values() if len(v) > 1]


@reusables.time_it(message="size map took {seconds:.2f} seconds")
def size_map(directory):
    return reusables.directory_duplicates(directory)


if __name__ == '__main__':
    directory = "F:\\Pictures"

    size_map_run = size_map(directory)
    print(f"size map returned {len(size_map_run)} duplicates")

    hash_map_run = hash_map(directory)
    print(f"hash map returned {len(hash_map_run)} duplicates")

The speed up of checking size first in our scenario is significant, over 16 times faster.

size map took 40.23 seconds
size map returned 3511 duplicates

hash map took 642.68 seconds
hash map returned 3511 duplicates

It jumps from under a minute for using reusables.directory_duplicates to over ten minutes when using a traditional hash map. This is the fastest pure Python method I have found, if you find faster, let me know!

Safe File Names

There are plenty of instances that you want to save a meaningful filename supplied by a user, say for a file transfer program or web upload service, but what if they are trying to crash your system?

Reusables has three functions to help you out.

  • check_filename: returns true if safe to use, else false
  • safe_filename: returns a pruned filename
  • safe_path: returns a safe path

These are designed not off of all legally allowed characters per system, but a restricted set of letters, numbers, spaces, hyphens, underscores and periods.

reusables.check_filename("safeFile?.text")
# False

reusables.safe_filename("safeFile?.txt")
# 'safeFile_.txt'

reusables.safe_path("C:\\test'\\%my_file%\\;'1 OR 1\\filename.txt")
# 'C:\\test_\\_my_file_\\__1 OR 1\\filename.txt'

Touch

Designed to be same as Linux touch command. It will create the file if it does not exist, and updates the access and modified times to now.

time.time()
# 1484450442.2250443

reusables.touch("new_file")

os.path.getmtime("new_file")
# 1484450443.804158

Simple JSON and CSV save and restore

These are already super simple to implement in pure python with the standard library, and are just here for convince of not having to remember conventions.

List of lists to CSV file and back

my_list = [["Name", "Location"],
           ["Chris", "South Pole"],
           ["Harry", "Depth of Winter"],
           ["Bob", "Skull"]]

reusables.list_to_csv(my_list, "example.csv")

# example.csv
#
# "Name","Location"
# "Chris","South Pole"
# "Harry","Depth of Winter"
# "Bob","Skull"


reusables.csv_to_list("example.csv")

# [['Name', 'Location'], ['Chris', 'South Pole'], ['Harry', 'Depth of Winter'], ['Bob', 'Skull']]

Save JSON with default indent of 4

my_dict = {"key_1": "val_1",
           "key_for_dict": {"sub_dict_key": 8}}

reusables.save_json(my_dict,"example.json")

# example.json
# 
# {
#     "key_1": "val_1",
#     "key_for_dict": {
#         "sub_dict_key": 8
#     }
# }

reusables.load_json("example.json")

# {'key_1': 'val_1', 'key_for_dict': {'sub_dict_key': 8}}

Cut a string into equal lengths

Ok, I admit, this one has absolutely nothing to do with the file system, but it’s just to handy to not mention right now (and doesn’t really fit anywhere else). One of the features I was most surprised to not be included in the standard library was to a have a function that could cut strings into even sections.

I haven’t seen any PEPs about it either way, but I wouldn’t be surprised if one of the reasons is ‘why do to with leftover characters?’. Instead of forcing you to stick with one, Reusables has four different ways it can behave for your requirement.

By default, it will simply cut everything into even segments, and not worry if the last one has matching length.

reusables.cut("abcdefghi")
# ['ab', 'cd', 'ef', 'gh', 'i']

The other options are to remove it entirely, combine it into the previous grouping (still uneven but now last item is longer than rest instead of shorter) or raise an IndexError exception.

reusables.cut("abcdefghi", 2, "remove")
# ['ab', 'cd', 'ef', 'gh']

reusables.cut("abcdefghi", 2, "combine")
# ['ab', 'cd', 'ef', 'ghi']

reusables.cut("abcdefghi", 2, "error")
# Traceback (most recent call last):
#     ...
# IndexError: String of length 9 not divisible by 2 to splice

Config to Dictionary

Everybody and their co-worker has written a ‘better’ config file handler of some sort, this isn’t trying to add to that pile, I swear. This is simply a very quick converter using the built in parser directly to dictionary format, or to a python object  I call a Namespace (more on that in future post.)

Just to make clear, this only reads configs, not writes any changes. So given an example config.ini file:

[General]
example=A regular string

[Section 2]
my_bool=yes
anint=234
exampleList=234,123,234,543
floatly=4.4

It reads it as is into a dictionary. Notice there is no automatic parsing or anything fancy going on at all.

reusables.config_dict("config.ini")
# {'General': {'example': 'A regular string'},
#  'Section 2': {'anint': '234',
#                'examplelist': '234,123,234,543',
#                'floatly': '4.4',
#                'my_bool': 'yes'}}

You can also take it into a ConfigNamespace.

config = reusables.config_namespace("config.ini")
# <ConfigNamespace: {'General': {'example': 'A regular string'}, 'Section 2': ...

Namespaces are special dictionaries that allow for dot notation, similar to Bunch but recursively convert dictionaries into Namespaces.

config.General
# <ConfigNamespace: {'example': 'A regular string'}>

ConfigNamespace has handy built-in type specific retrieval.  Notice that dot notation will not work if item have spaces in them, but the regular dictionary key notation works as well.

config['Section 2'].bool("my_bool")
# True

config['Section 2'].bool("bool_that_doesn't_exist", default=False)
# False
# If no default specified, will raise AttributeError

config['Section 2'].float('floatly')
# 4.4

It supports booleans, floats, ints, and unlike the default config parser, lists. Which even accepts a modifier function.

config['Section 2'].list('examplelist', mod=int)
# [234, 123, 234, 543]

Finale

That’s all for this first overview,. hope you found something useful and will make your life easier!

Related links: