Python Decorators

Often times in python there comes a need to have multiple functions act in a similar manner. It could be anything from making sure that similar functions output the right type of result, they all log when they are called or all report exceptions in the same way.

An easy and repeatable way to accomplish this is decorators. They look like:

@decorator
def my_function(print_string="Hello World"):
    print(print_string)

my_function()
# Hello World

what that is actually doing is passing the lower function into another function called decorator. In other words it is exactly the same as doing:

def my_function(print_string="Hello World"):
    print(print_string)

decorator(my_function)("My message")
# My message

This post will go over the basics of how to create them, if you want to understand better how and why they work as they do, check out Guide to: Learning Python Decorators by Matt Harrison.

So what does a decorator look like?

Decorator Template

from functools import wraps

def decorator(func):
    """This is an example decorators"""
    @wraps(func)
    def container(*args, **kwargs):
        # perform actions before running contained function
        output = func(*args, **kwargs)
        # actions to run after contained function
        return output
    return container

Running the code in this sate would look exactly the same as if there were no decorator.

@decorator
def my_function():
    print "Hello World"

my_function()
# "Hello World"

line by line breakdown

def decorator(func):

The name of the decorator function, which takes the wrapped function as its single argument.

@wraps(func)

Wraps is a built-in method that replaces the decorators name and docstring with that of the wrapped function, the section after the line by line breakdown explains why this is necessary.

def container(*args, **kwargs):

This inner function collects the parameters and keyword parameters that are going to be passed to the original function. This allows the decorator access to incoming arguments to verify or modify before the function is ever run.

output = func(*args, **kwargs)

this runs the function with the original arguments and captures the output. This output could then be checked or changed before being returned, or even return an alternative output entirely.

return output

Don’t forget to actually return the original function’s output (or a custom one) or else the function will simply return None.

return container

The container function is the actual function being called,  hence why *args, **kwargs are passed to it. It is necessary to return it from the outside decorator so it can be called.

The importance of Wraps

We need to incorporate wraps so that the function name and docstring appear to be from the wrapped function, and not those of the wrapper itself.

@decorator
def my_func():
    """Example function"""
    return "Hello"

@decorator_no_wrap
def second_func():
    """Some awesome docstring"""
    return "World"

help(my_func)
# Help on function my_func:

# my_func()
#    Example function

help(second_func)
# Help on function container:

# container(*args, **kwargs)

It is possible, though more work to accomplish the same thing yourself.

def decorator(func):
    """This is an example decorators"""
    def container(*args, **kwargs):
        return func(*args, **kwargs)
    container.__name__ = func.__name__
    container.__doc__ = func.__doc__
    return container

Useful Example

Now lets turn it into something useful. In this example we will make sure that the function returns the expected type of result, or else it will raise an exception for us so there are not hidden complications down the road.

from functools import wraps

def isint(func):
         """ 
         This decorator will make sure the resulting value 
         is a integer or else it will raise an exception.
         """
    @wraps(func)
    def container(*args, **kwargs):
        output = func(*args, **kwargs)
        if not isinstance(output, int):
            raise TypeError("function did not return integer")
        return output
    return container

@isint
def add(num1, num2):
    """Add two numbers together and return the results"""
    return num1 + num2

print add(1, 2) 
# 3

print add("this", "that")
# Type Error: function did not return integer

Regular decorators are already called on execution, so you do not need to add ()s after their name, such as @isint. However, if the decorator accepts arguments, aka a meta-decorator, it will require () even if nothing additional is passed to it.

Meta-decorators

Meta-decorators can be created with either yet another function around the decorator, or built as a class.

from functools import wraps

def istype(instance_type=int):
    def decorator(func):
        """ 
        This decorator will make sure the resulting value is the 
        type specified or else it will raise an exception. 
        """
        @wraps(func)
        def container(*args, **kwargs):
            output = func(*args, **kwargs)
            if not isinstance(output, instance_type):
                raise TypeError("function did not return proper type")
            return output
        return container
    return decorator

@istype()
def add(num1, num2):
    """Add two numbers together and return the results"""
    return num1 + num2

@istype(str)
def reverse(forward_string):
    """Reverse and return incoming string"""
    return forward_string[::-1]


print add(1, 2) 
# 3

print reverse("Hello") 
# "olleH"

print add("this", "that")
# Type Error: function did not return proper type

Remember running a decorator is equivalent to:

decorator(my_function)("My message")

Running a meta-decorator adds an additional layer.

reversed_string = istype(str)(reverse)("Reverse Me")

Hence why @decorator doesn’t require to be called when put above a function, but @istype() does.

You can also create this meta-decorator as a class instead of another function.

class IsType: # Instead of 'def istype'

    def __init__(self, inc_type=int):
        self.inc_type = inc_type

    def __call__(self, func): # Replaces 'def decorator(func)'
        @wraps(func)
        def container(*args, **kwargs):
            output = func(*args, **kwargs)
            if not isinstance(output, self.inc_type):
                raise TypeError("function did not return proper type")
            return output
        return container

In functionality they are the same, but be aware they are technically different types. This will really only impact code inspectors and  those trying to manually navigate code, so it is not a huge issue, but is something to be aware of.

type(IsType)
<class 'type'>

type(istype)
<class 'function'>

 Things to keep in mind

  1. Order of operation does matter. If you have multiple decorators around a single function keep in mind they are run from top to bottom
  2. Once a function is wrapped, it cannot be run without the wrapper.
  3.  Meta-decorators require the additional parentheses, regular decorators do not.

This content is an updated version of what was original posted on my old personal site Cyber Curios, which has been shut down.

Creating a successful project – Part 3: Development Tools/Equipment

Every single year that I’ve been doing this, I hear about the next “totally awesome” way to write code.  And more often than not, the new thing is certainly very shiny.

When it comes to projects, with the exception of coding standards (which will be part 4 of this series) I am not a fan of telling developers how to write code.  If you’ve got someone who likes to write code using Notepad on a Microsoft Windows machine, more power to them.  Oh, you like coding in SublimeText3 on Mac – go for it.

If you work on one of my projects there are only a few rules I have about how you write your code:

  1. It must maintain the agreed-upon standard (such as PEP8)
  2. Your code – under penalty of my ire – must work on the designated system.  If it WFM, “Works for Me” then you must get it working on the chosen system. (More on this topic in the test and build posts) And trust me, there’s plenty of people out there – including other contributors to this site – who would shudder to think of my ire directed singly upon them.
  3. Use whatever the agreed upon (preferably Git) source code control system.
  4. Use whatever build system is in play.  Usually, this is done via a Jenkins server, but I’m not picky.  I want consistency, and I want to make sure that the output of the project is reliable.  More on build systems in the CI/CD section.

Notice something odd in there: nowhere did I say you had to use this particular editor or debugger.  I honestly couldn’t care less if you like to write your code using Comic Sans or SourceCodePro.  I really don’t care if you like to code using EMACS or Sublime.  The tools one uses to write code should be selected through a similar vetting process to purchasing a good chef’s knife: use what you feel most comfortable using.

But, in the interest of showing what a rather more seasoned coder uses, here’s my setup:

Keyboard – Microsoft Natural Ergonomic Keyboard – I spend 8-16 hours a day on a keyboard, so I want my keyboard to be comfortable and able to handle heavy use.  The good thing (besides that this is a great keyboard) they’re nice and cheap.  So when one dies, I just buy another.

Mouse – ROCCAT Kone Pure Color – This is just a really great mouse.

Editor- Vim or, as of recent Neovim – I’ve used Vi/Vim for decades so I’m a bit of an old hat at using them.

Operating System – Debian Linux – When you want the best and you don’t want extra crap getting in your way; accept only the best.

I use that same setup at work as well as home.  I am not endorsed by any of the product manufacturers; I just know what works for me.  If I find a keyboard in the same form-factor as the one I’m using with Cherry MX Browns, I’ll buy two of them in a heartbeat.

I have also made use of PyCharm and Atom.  Both of which I still use with Vim Keybindings.

 

Creating a successful project – Part 2: Project Organization

Ok, so you’ve familiarized yourself with the rules.  Good.  This isn’t Fight Club, but they are important rules. You want to be a professional in all the projects on which you want to participate.  If properly managed, your GitHub account should be a functional part of your ever-evolving resume.

Code organization:

So, code itself really isn’t hard to organize.  But you’d be surprised how many people just don’t get it.  Since I usually develop in python, here is my standard project directory structure:

./new_project.
 ├── CONTRIBUTING.md
 ├── doc
 ├── LICENSE.md
 ├── README.md
 ├── scripts
 ├── setup.py
 ├── src
 └── tests

If you code in a different language which has different structural requirements, make changes.  This is how I do it.

First, the markdown files:

I’m not going to worry over the preference for ReStructuredText, Markdown, LaTeX, or basic text for these files.  Feel free to do your own research and come to your own conclusions.

CONTRIBUTING.md

This document is optional for simple projects.  It is possible to have this as part of the README.  In this document, you should spell out how people can contribute to your project.  Should they fork and submit pull requests?  How should they report issues and/or request new features?  These are important questions.  Not just for potential compatriots, but for you.

LICENSE.md

This should have the text of the license under which you are placing the project.  You can cut and paste this from any number of reputable licensing sources.  I recommend reading about the different licenses here. I will also be going over the primary players in a future post.

README.md

I feel that the project README is non-optional.  This single document should have key information about the project.  It should (ideally) address the following:

  1. What is the purpose of this project?  Think of it as the two-to-three sentence description of the project.  What problem were you trying to solve?
  2. Where to find more documentation.
  3. Installation instructions
  4. Simple configuration settings – if appropriate.
  5. Possibly simple use-cases and examples.

Optional Docs:

DONATIONS.md/SPONSOR.md – Where/how to send any monetary donations.  A good example of what might be found in a donations/sponsorship doc can be found here.

THANKS.md – If you want to thank anyone for help or encouragement, this would be alright to add.  Completely optional.

Now the directories:

doc

This is where your project documentation should reside.  If you do this correctly, you can easily get them posted to readthedocs.org.

scripts

This is optional for projects which have no need for command-line scripts.  But, if your project has a command-line tool associated with it, you should really consider having them in this directory.

src

This is where your code should reside.

tests

This is where the tests for your code should be found.  Having tests is a big indicator of the health and maturity of the project.  The tests here should show that you take the writing and maintaining of the code very seriously.

Optional Directories:

examples – Some people like to house some usage examples here.  If appropriate, feel free.

 

 

 

Introducing Box – Python dictionaries with recursive dot notation access

Box logo

Everyone loves Python’s dictionaries; they’re fast, easy to create and quite handy for a range of reasons. However, there are times that ["typing"]["out"]["all"]["those"] extra quotes and  brackets seems excessive. Wouldn’t it be nicer to access.them.like.class.methods?

Say hello to box.

Box logo

from box import Box

movie_data = {
  "movies": {
    "Spaceballs": {
      "imdb_stars": 7.1,
      "rating": "PG",
      "length": 96,
      "Director": "Mel Brooks",
      "Stars": [{"name": "Mel Brooks", "imdb": "nm0000316", "role": "President Skroob"},
                {"name": "John Candy","imdb": "nm0001006", "role": "Barf"},
                {"name": "Rick Moranis", "imdb": "nm0001548", "role": "Dark Helmet"}
      ]
    },
    "Robin Hood: Men in Tights": {
      "imdb_stars": 6.7,
      "rating": "PG-13",
      "length": 104,
      "Director": "Mel Brooks",
      "Stars": [
                {"name": "Cary Elwes", "imdb": "nm0000144", "role": "Robin Hood"},
                {"name": "Richard Lewis", "imdb": "nm0507659", "role": "Prince John"},
                {"name": "Roger Rees", "imdb": "nm0715953", "role": "Sheriff of Rottingham"},
                {"name": "Amy Yasbeck", "imdb": "nm0001865", "role": "Marian"}
      ]
    }
  }
}


my_box = Box(movie_data)

my_box.movies.Spaceballs.rating
'PG'

my_box.movies.Spaceballs.Stars[0].name
'Mel Brooks'

my_box.movies.Spaceballs.Stars[0]
# <Box: {'name': 'Mel Brooks', 'imdb': 'nm0000316', 'role': 'President Skroob'}>

Box is a creation I made over three years ago, originally in the reusables code base named Namespace, inspired by JavaScript Object access methods.

Install is super simple:

pip install python-box

Or just grab the file box.py directly from the github project.

Every Box is usable as a drop in replacement to dictionaries in 99%* of cases. And every time you add a dictionary or list to a Box object, they become Box (subclass of dict) or BoxList (subclass of list) objects as well.

type(my_box)
# box.Box
assert isinstance(my_box, dict)

type(my_box.movies.Spaceballs.Stars)
# box.BoxList
assert isinstance(my_box.movies.Spaceballs.Stars, list)

my_box.movies.Spaceballs.Stars[0].additional_info = {'Birth name': 'Melvin Kaminsky', 'Birthday': "05/28/1926"}

my_box.movies.Spaceballs.Stars[0].additional_info
# <Box: {'Birth name': 'Melvin Kaminsky', 'Birthday': '05/28/1926'}>

At any level you can change a Box object back into a standard dictionary.

my_box.movies.Spaceballs.to_dict()

{'Director': 'Mel Brooks',
 'Stars': [
  {'additional_info': {'Birth name': 'Melvin Kaminsky', 'Birthday': '05/28/1926'},
   'imdb': 'nm0000316',
   'name': 'Mel Brooks',
   'role': 'President Skroob'},
  {'imdb': 'nm0001006', 'name': 'John Candy', 'role': 'Barf'},
  {'imdb': 'nm0001548', 'name': 'Rick Moranis', 'role': 'Dark Helmet'},
  {'imdb': 'nm0000597', 'name': 'Bill Pullman', 'role': 'Lone Starr'}],
 'imdb_stars': 7.1,
 'length': 96,
 'rating': 'PG'}

You can also run to_list() on lists in the Box to return them to a standard list, with all inner Box and BoxList objects transformed back to normal.

Box also has built in functions for dealing with json and yaml**.

my_box.movies.Spaceballs.to_json()

# {
#    "imdb_stars": 7.1,
#    "rating": "PG",
#    "length": 96,
#    "Director": "Mel Brooks",
#    "Stars": [
# ...


my_box.movies.Spaceballs.to_yaml()

# Director: Mel Brooks
# imdb_stars: 7.1
# length: 96
# rating: PG
# Stars:
# - imdb: nm0000316
#   name: Mel Brooks
#   role: President Skroob
# ...


Calling a Box object will return it’s keys. It’s also possible to access the attributes the standard dictionary method, which is required for keys that are numeric or have spaces.

my_box.movies()
# ('Spaceballs', 'Robin Hood: Men in Tights')

my_box.movies['Robin Hood: Men in Tights']
# <Box: {'imdb_stars': 6.7, 'rating': 'PG-13', 'length': 104, ...

Unlike addict it does not act as a default dictionary, so you will get built-in errors if you try to access something that isn’t there.

my_box.tv_shows

# Traceback (most recent call last):
# ...
# AttributeError: tv_shows

Another power previously mentioned is that you can add dictionaries into lists and they will automatically be converted into Box objects.

my_box.movies.Spaceballs.Stars.append(
    {"name": "Bill Pullman", "imdb": "nm0000597", "role": "Lone Starr"})

my_box.moves.Spaceballs.Stars[-1].name
'Bill Pullman'

It also protects itself from having its functions overwritten accidentally.

my_box.to_dict = '3'
# AttributeError: Key name 'to_dict' is protected

Box is also a substitute for the Namespace used by argparse, making it super easy to convert incoming arguments to a dict if wanted. This allows incoming arguments to be easily passed to function arguments.

import argparse
from box import Box

parser = argparse.ArgumentParser()
parser.add_argument('floats', metavar='N', type=float, nargs='+')
parser.add_argument("-v", "--verbosity", action="count", default=0)

args = parser.parse_args(['1', '2', '3', '-vv'], namespace=Box())

print(args.to_dict())
{'floats': [1.0, 2.0, 3.0], 'verbosity': 2}


def example_func(floats, verbosity):
    print(verbosity)

example_func(**args)
2

If you have any questions, suggestions or feedback, please open a github issue and let me know!

Hope you enjoy!

Caveats

*  Based off nothing but pure guess and personal experience. Only time drop in replacement doesn’t work is when converting or dumping. So make sure do use  first for those cases.  

** If you don’t have PyYAML installed, the to_yaml function will not be available.

Creating a successful project – Part 1: The Rules

So, you want to create a project.  That’s great!  Welcome to the party.  I’d like to pass along some general knowledge about how best to approach creating/running a project.  Also, I’m not going to say that I know everything there is to know about running a successful project.  I have run a few projects in my time, some successful, some less so;  I also have some hard-fought lessons learned that I am going to share.  Perhaps you’ll use some of this knowledge to improve your projects.

One side note on what I am teaching/advocating here:  I am not telling you some secret to making your project the next docker or requests.  I’m showing you best practices for success.  These ideas and concepts are good for any size or purpose of a software project.  They have been successfully used on single-person projects all the way up to projects with 70+ contributors.  Good ideas and buy-in from your team are critical to success, but they are never going to be enough to replace plain old elbow grease to write the code, docs, and tests for a project.  That stuff takes discipline and time.

General Project Structure Rules

  1. If you want you (and your project) to be taken seriously?  Be organized.
  2. Tests are non-optional.
  3. Documentation is also non-optional.
  4. Documentation should read like it was written by someone with knowledge who cares about informing a newcomer to the project.
  5. You should understand contributor roles if any.
  6. You MUST understand the licenses that are in regular use and decide which license to use.  This is more about long-term CYA than anything else.

Well, there’s the first entry in this series.  I hope people find it interesting.