Uncategorized

Is it easier to ask for forgiveness in Python?

Yes, we are looking for graphic artists

Yes, we are looking for graphic artists

Is Python more suited for EAFP1 or LBYL2  coding styles? It has been debated across message boards, email chains, stackoverflow, twitter and workplaces. It’s not as heated as some other battles; like how spaces are better than tabs, or that nano is indisputably the best terminal editor 3; but people seem to have their own preference on the subject. What many will gloss over is the fact that Python as a language doesn’t have a preference. Rather, different design patterns lend themselves better to one or the other.

I disagree with the position that EAFP is better than LBYL, or “generally recommended” by Python – Guido van Rossum 4

If you haven’t already, I suggest taking a look at Brett Cannon’s blog post about how EAFP is a valid method with Python that programmers from other languages may not be familiar with. The benefits are EAFP are simple: Explicit code, fail fast, succeed faster, and DRY (don’t repeat yourself). In general I find myself using it more than LBYL, but that doesn’t mean it’s always the right way.

Use Case Dependent

First, a little code comparison to see the differences between them. Lets try to work with a list of list of strings. Our goal is that if the inner list is at least three items long, copy the third item into another list.

messages = [["hi there", "how are you?", "I'm doing fine"]]
out = []
 
# LBYL
if messages and messages[0] and len(messages[0]) >= 3:
    out.append(messages[0][2])

# EAFP
try:
    out.append(messages[0][2])
except IndexError as err:
    print(err)  # In the real world, this should be a logging `.exception()`

Both pieces of code are short and easy to follow what they are accomplishing. However, when there are at least three items in the inner list, LBYL will take almost twice the amount of time! When there aren’t any errors happening, the checks are just additional weight slowing the process down.

But if we prune the messages down to [["hi there", "how are you?"]]   then EAFP will be much slower, because it will always try a bad lookup, fail and then have to catch that failure. Whereas LBYL will only perform the check, see it can’t do it, and move on. Check out my speed comparison gist yourself.

Some people might think putting both the append and lookup in the same try block is wrong. However, it is both faster; as we only have to do the lookup once; and the proper way; as we are only catching the possible IndexError and not anything that would arise from the append.

The real takeaway from this is to know which side of the coin you area dealing with beforehand. Is your incoming data almost always going to be correct? Then I’d lean towards EAFP, whereas if it’s a crap shoot, LBYL may make more sense.

Sometimes one is always faster

There are some instances when one is almost always faster than the other (though speed isn’t everything). For example, file operations are faster with EAFP because the less  IO wait, the better.

Lets try making a directory when we are not sure if it was created yet or not, while pretending it’s still ye olde times and os.makedirs(..., exist_ok=True) doesn’t exist yet.

import os 

# LBYL 
if not os.path.exists("folder"):
    os.mkdir("folder")

# EAFP 
try:
    os.mkdir("folder")
except FileExistsError:
    pass

In this case, it’s always preferential to use EAFP, as it’s faster (even when executed on a m.2 SSD) , there are no side effects, and the error is highly specific so there is no need for special handling.

Be careful though, many times when dealing with files you don’t want to leave something in a bad state, in those cases, you would use LBYL.

What bad things can happen when you don’t ask permission?

Side effects

In many cases, if something fails to execute, the state of the program or associated files might have changed. If it’s not easy to revert to a known good state in the exception clause, EAFP should be avoided.

# DO NOT DO

with open("file.txt", "w") as f:
    try: 
        f.write("Hi there!\n") 
        f.write(message[3])   
    except IndexError: 
        pass  # Don't do, need to be able to get file back to original state

Catching too much

If you are wrapping something with except Exception or worse, the dreaded blank except:, then you shouldn’t be using EAFP (if you’re using black except: still, you need to read up more on that and stop it!)

# DO NOT DO

try:
    do_something()
except:  # If others see this you will be reported to the Python Secret Police
    pass 

Also, sometimes errors seem specific, like an OSError, but could raise multiple different child errors that must be parsed through.

# DO 
import errno

try:
    os.scandir("secret_files")
except FileNotFoundError: # A child of OSError
    # could be put as 'elif err.errno == errno.ENOENT' below
    log.error("Yo dawg, that directory doesn't exist, "
              "we'll try the backup folder")
except OSError as err:
    if err.errno in (errno.EACCES, errno.EPERM):
        log.error("We don't have permission to access that capt'n!"
                  "We'll try the backup folder")
    else:
        log.exception(f"Unexpected OSError: {err}") 
        raise err # If you don't expect an error, don't keep going. 
                  # This isn't Javascript

When to use what?

EAFP (Easier to Ask for Forgiveness than Permission)

  • IO operations (Hard drive and Networking)
  • Actions that will almost always be successful
  • Database operations (when dealing with transactions and can rollback)
  • Fast prototyping in a throw away environment

LBYL (Look Before You Leap):

  • Irrevocable actions, or anything that may have a side effect
  • Operation that may fail more times than succeed
  • When an exception that needs special attention could be easily caught beforehand

Sometime you might even find yourself using both for complex or mission critical applications. Like programming the space shuttle, or getting your code above that 500 lines the teacher wants.

 

Python Box 3 – The reddit release

Box has surpassed my expectations in popularity, even appearing in the Python Weekly newsletter and on Python Bytes podcast. I am very grateful for everyone that uses it and hope that you find it to be another great tool in your Python tool-belt! Also a huge shot out to everyone that has provided feedback or code contributes!

Box logoI first posted about Box about a month ago and linked to it on /r/python, which had great feedback and critique. With such interest, I have been working diligently the past few weeks to try and make it better, and I feel it confident it isn’t just better, it’s substantially improved.

Box 3 brings two huge changes to the table. First is that can take arguments to allow it to work in new and different ways, such as becoming a default_box or a  frozen_box. Second, Box does not longer deals with converting all sub dicts into Boxes upon creation, rather upon retrieval. These two things means that Box is now more powerful, customizable and faster than ever before.

New Features

 Conversion Box

This turns all those pesky keys that normally could not be accessed as an attribute into a safe string. automatically ‘fixes’ keys that can’t become attributes into valid lookups.

my_box = Box({"Send him away!": True})

my_box["Send him away!"] == my_box.Send_him_away

To be clear, this NEVER modifies the actual dictionary key, only allows attribute lookup to search for the converted name. This is the only feature that is enabled by default. Also keep it mind it does not modify case or attempt to find ascii equivalent for extended charset letters, it simply removes anything outside the allowed range.

You can also set those values through the attribute. The danger with that is if two improper strings would transform into the same attribute, such as a~b and a!b would then both equal the attribute a_b, but the Box would only modify or retrieve the first one it happened to find.

my_box.Send_him_away = False

my_box.items()
# dict_items([('Send him away!', False)])

Default Box

If you don’t like getting errors on failed lookups, or bothering to create all those in-between boxes, this is for you!

default_box = Box(default_box=True)

default_box.Not_There 
# <Box: {}>

default_box.a.b.c.d.c.e.f.g.h = "x"
# <Box: {'a': {'b': {'c': {'d': {'c': {'e': {'f': {'g': {'h': 'x'}}}}}}}}}>

You can specify what you want the default return value to be with default_box_attr which by default is a Box class. If something callable is specified, it will return it as an instantiated version, otherwise it will return the specified result as is.

bx3 = Box(default_box=True, default_box_attr=3)

bx3.hello
# 3

However it now only acts like a standard default dict, and you do lose the magical recursion creation.

bx3.hello.there
# AttributeError: 'int' object has no attribute 'there'

Frozen Box

Brrrrr, you feel that? It’s a box that’s too cool to modify. Handy if you want to ensure the incoming data is not mutilated during it’s usage. An immutable box cannot have data added or delete,  like a tuple it can be hashed if all content is immutable, but if it has mutable content can still modify those objects and is unhashable.

frigid_box = Box({"Kung Fu Panda": "Pure Awesome"}, frozen_box=True)

frigid_box.Kung_Fu_Panda = "Overrated kids movie"
# Traceback (most recent call last):
# ...
# box.BoxError: Box is frozen

hash(frigid_box)
# -156221188

Camel Killer Box

DontYouHateCamelCaseKeys? Well I do too, kill them with fire camel_killer_box=True.

snake_box = Box({"BadKey": "silly value"}, camel_killer_box=True) 
assert snake_box.bad_key == "silly_value"

This transformation, including lower casing, is applied to everything. Which means the above example for just conversion box would also be lower case.

snake_box["Send him away!"] == snake_box.send_him_away

Box It Up

Force the conversion of all sub objects on creation, just like old times. This is useful if you expect to be referencing every value, and every sub-box value and want to front load the time at creation instead of during operations. Also a necessity if you plan to keep the reference or re-use the original dictionary that was boxed.

boxed = Box({"a": {"b": [{"c": "d"}]}}, box_it_up=True)

 

Dangerous Waters

As stated above, sub box objects are now not recreated until they are retrieved or modified. Which means if you have references to the original dict and modify it, you may inadvertently be updating the same objects that are in the box.

a = {"a": {"b": {"c": {}}}}
a_box = Box(a)
a_box 
# <Box: {'a': {'b': {'c': {}}}}>

a["a"]["b"]["d"] = "2"

a_box
# <Box: {'a': {'b': {'c': {}, 'd': '2'}}}>

So if you plan to keep the original dict around, make sure to box_it_up or do a deepcopy first.

safe_box = Box(a, box_it_up=True)
a["a"]["b"]["d"] = "2"

safe_box
# <Box: {'a': {'b': {'c': {}}}}>

Obviously not an issue if you are creating it from scratch, or with a new and unreferenced dict, just something to be aware of.

Speed and Memory Usage

Something a lot of people were worried about was speed and memory footprint compared to alternatives and the built-in dict. With the previous version of Box, I was more of the mindset that it lived to be a scripter, sysadmin or developer aid in writing in the code, and less performant worried. Simply put, it was foolish to think that way. If I really want this to be the go to dot notation library, it needs to be as fast as, if not faster, than it’s peers while still providing more functionality.

Because Box is the only dot notation library that I know of that works by converting upon retrieval, it is hard to do a true 1 to 1 test verse the others. To the best of my abilities, I have attempted to capture the angles that show where the performance differences are most noticeable.

This test is using a modified version of HearthStone card data, available on github, which is 6.57MBs and has 2224 keys, with sub dictionaries and data. To gather metrics, I am using reusables.time_it and memory_profiler.profile and the file can be run yourself. Box is being compared against the two most applicable competitors I know of, addict and DotMap (as Bunch and EasyDict do not account for recursion or have other common features), as well as the built in dict type. All charts except the memory usage are in seconds.

Lower is always better, these are speed and memory footprint charts

Creation and Conversion

So first off, how long does it take to convert a JSON file to the object type, and back again.

Figure 1: Convert JSON file to object and back to dict

 

The load is the first thing done, between then and the transformation back into a dictionary, all first level items are retrieved and 1000 new elements were added to the object. Thanks to the new style of only transforming sub items upon retrieval, it greatly speeds up Box, making very comparative to the built in dict type, as seen in figure 1. 

Load code comparison:

# addict
with open("hearthstone_cards.json", encoding="utf-8") as f:
    ad = Dict(json.load(f))

# DotMap
with open("hearthstone_cards.json", encoding="utf-8") as f:
    dm = DotMap(json.load(f))

# Box
bx = Box.from_json(filename="hearthstone_cards.json", encoding="utf-8")

# dict
with open("hearthstone_cards.json", encoding="utf-8") as f:
    dt = json.load(f)

Value Lookup

The load speed up does come at the price of having a heavier first initial lookup call for each value. The following graph, figure 2, shows a retrieval of all 2224 first level values from the object.

Figure 2: Lookup of 2224 first level values

 

As expected, Box took far and away the most time, but only on the initial lookup, and we are talking about a near order of magnitude less than the creation times for the other libraries.

The lookup is done the same with every object type.

for k in obj:
    obj[k]

Inserting items

Inserting a 1000 new entities into the objects deals with very small amounts of time all around.

Figure 3: Inserting a 1000 new dict objects that will be accessible as those native object types

 

Insert looks the most different from the code, as the different objects behave differently on insertion. Box will insert it as a plain dict, and not convert it until retrieval. Addict will not automatically make new items into addict.Dicts, causing the need to manually. DotMap behaves as a default dict out of the box, so we can simply access the attribute we want to create directly off the bat.

# addict 
for i in range(1000):
    ad["new {}".format(i)] = Dict(a=i)

# DotMap
for i in range(1000):
    dm["new {}".format(i)]['a'] = i

# Box
for i in range(1000):
    bx["new {}".format(i)] = {'a': i}

# dict
for i in range(1000):
    dt["new {}".format(i)] = {'a': i}

Total Times

To summarize the times, dict is obviously the fastest, but Box is in a comfortable second fastest.

Figure 4: The total amount of seconds for all above scenarios

 

Memory Footprint

What I was most surprised with was memory usage. I have not had a lot of experience measuring this, so I leave room open for something obvious I am missing, but on first look, it seems that Box uses less memory than even the built in dict. 

Figure 5: Total memory usage in MBs

 

"""
        Line #    Mem usage    Increment   Line Contents
        ================================================
        85     28.9 MiB      0.0 MiB   @profile()
        86                             def memory_test():
        87     41.9 MiB     13.0 MiB       ad = load_addict()
        88     57.1 MiB     15.2 MiB       dm = load_dotmap()
        89     64.5 MiB      7.4 MiB       bx = load_box()
        90     74.6 MiB     10.1 MiB       dt = load_dict()
        91                             
        92     74.6 MiB      0.0 MiB       lookup(ad)
        93     74.6 MiB      0.0 MiB       lookup(ad)
        94                             
        95     74.6 MiB      0.0 MiB       lookup(dm)
        96     74.6 MiB      0.0 MiB       lookup(dm)
        97                             
        98     75.6 MiB      1.0 MiB       lookup(bx)
        99     75.6 MiB      0.0 MiB       lookup(bx)
       100                             
       101     75.6 MiB      0.0 MiB       lookup(dt)
       102     75.6 MiB      0.0 MiB       lookup(dt)
       103                             
       104     76.0 MiB      0.3 MiB       addict_insert(ad)
       105     76.6 MiB      0.6 MiB       dotmap_insert(dm)
       106     76.8 MiB      0.2 MiB       box_insert(bx)
       107     76.9 MiB      0.2 MiB       dict_insert(dt)
"""

 

Questions I can answer

How does it work like a traditional dict and still take keyword arguments?

You can create a Box object just like you would create a dict as normal, just those new arguments will not be part of dict object (they are stored in a hidden ._box_config attribute). They were also made rather unique, such as ‘conversion_box’ and ‘frozen_box’ so that it would be very rare for others to be using those kwargs.

Is it safe to wait to convert dictionaries until retrieval?

As long as you aren’t storing malformed objects that subclassed dict, yes.

Creating a successful project – Part 3: Development Tools/Equipment

Every single year that I’ve been doing this, I hear about the next “totally awesome” way to write code.  And more often than not, the new thing is certainly very shiny.

When it comes to projects, with the exception of coding standards (which will be part 4 of this series) I am not a fan of telling developers how to write code.  If you’ve got someone who likes to write code using Notepad on a Microsoft Windows machine, more power to them.  Oh, you like coding in SublimeText3 on Mac – go for it.

If you work on one of my projects there are only a few rules I have about how you write your code:

  1. It must maintain the agreed-upon standard (such as PEP8)
  2. Your code – under penalty of my ire – must work on the designated system.  If it WFM, “Works for Me” then you must get it working on the chosen system. (More on this topic in the test and build posts) And trust me, there’s plenty of people out there – including other contributors to this site – who would shudder to think of my ire directed singly upon them.
  3. Use whatever the agreed upon (preferably Git) source code control system.
  4. Use whatever build system is in play.  Usually, this is done via a Jenkins server, but I’m not picky.  I want consistency, and I want to make sure that the output of the project is reliable.  More on build systems in the CI/CD section.

Notice something odd in there: nowhere did I say you had to use this particular editor or debugger.  I honestly couldn’t care less if you like to write your code using Comic Sans or SourceCodePro.  I really don’t care if you like to code using EMACS or Sublime.  The tools one uses to write code should be selected through a similar vetting process to purchasing a good chef’s knife: use what you feel most comfortable using.

But, in the interest of showing what a rather more seasoned coder uses, here’s my setup:

Keyboard – Microsoft Natural Ergonomic Keyboard – I spend 8-16 hours a day on a keyboard, so I want my keyboard to be comfortable and able to handle heavy use.  The good thing (besides that this is a great keyboard) they’re nice and cheap.  So when one dies, I just buy another.

Mouse – ROCCAT Kone Pure Color – This is just a really great mouse.

Editor- Vim or, as of recent Neovim – I’ve used Vi/Vim for decades so I’m a bit of an old hat at using them.

Operating System – Debian Linux – When you want the best and you don’t want extra crap getting in your way; accept only the best.

I use that same setup at work as well as home.  I am not endorsed by any of the product manufacturers; I just know what works for me.  If I find a keyboard in the same form-factor as the one I’m using with Cherry MX Browns, I’ll buy two of them in a heartbeat.

I have also made use of PyCharm and Atom.  Both of which I still use with Vim Keybindings.

 

Creating a successful project – Part 2: Project Organization

Ok, so you’ve familiarized yourself with the rules.  Good.  This isn’t Fight Club, but they are important rules. You want to be a professional in all the projects on which you want to participate.  If properly managed, your GitHub account should be a functional part of your ever-evolving resume.

Code organization:

So, code itself really isn’t hard to organize.  But you’d be surprised how many people just don’t get it.  Since I usually develop in python, here is my standard project directory structure:

./new_project.
 ├── CONTRIBUTING.md
 ├── doc
 ├── LICENSE.md
 ├── README.md
 ├── scripts
 ├── setup.py
 ├── src
 └── tests

If you code in a different language which has different structural requirements, make changes.  This is how I do it.

First, the markdown files:

I’m not going to worry over the preference for ReStructuredText, Markdown, LaTeX, or basic text for these files.  Feel free to do your own research and come to your own conclusions.

CONTRIBUTING.md

This document is optional for simple projects.  It is possible to have this as part of the README.  In this document, you should spell out how people can contribute to your project.  Should they fork and submit pull requests?  How should they report issues and/or request new features?  These are important questions.  Not just for potential compatriots, but for you.

LICENSE.md

This should have the text of the license under which you are placing the project.  You can cut and paste this from any number of reputable licensing sources.  I recommend reading about the different licenses here. I will also be going over the primary players in a future post.

README.md

I feel that the project README is non-optional.  This single document should have key information about the project.  It should (ideally) address the following:

  1. What is the purpose of this project?  Think of it as the two-to-three sentence description of the project.  What problem were you trying to solve?
  2. Where to find more documentation.
  3. Installation instructions
  4. Simple configuration settings – if appropriate.
  5. Possibly simple use-cases and examples.

Optional Docs:

DONATIONS.md/SPONSOR.md – Where/how to send any monetary donations.  A good example of what might be found in a donations/sponsorship doc can be found here.

THANKS.md – If you want to thank anyone for help or encouragement, this would be alright to add.  Completely optional.

Now the directories:

doc

This is where your project documentation should reside.  If you do this correctly, you can easily get them posted to readthedocs.org.

scripts

This is optional for projects which have no need for command-line scripts.  But, if your project has a command-line tool associated with it, you should really consider having them in this directory.

src

This is where your code should reside.

tests

This is where the tests for your code should be found.  Having tests is a big indicator of the health and maturity of the project.  The tests here should show that you take the writing and maintaining of the code very seriously.

Optional Directories:

examples – Some people like to house some usage examples here.  If appropriate, feel free.