In the latest Python Weekly issue 607, there was a well written article by James Turk called You Don’t Need __all__ (waybackmachine backup link) which seemed to miss an important point I wanted to discuss. With no way to comment on his blog, I decided might as well turn it into a learning experience here!
Where We Agree
James makes a lot of good points throughout his article and explains how imports work well, so please do give it a quick read through! We are step in step with the mindset that you probably shouldn’t be using import * anyways. It’s messy and unclear.
However everyone writing Python code isn’t in a perfect dev environment. A lot are people just trying to run stuff quickly in notebooks to produce a report, new to Python, or frankly just like the convenience of using import *.
Where You Need __all__
Something not covered in the article is what __all__ is really handy for. It’s too limit the imports from a module. Including other imports.
Let’s say you have this messy.py code.
from json import loads
import shutil
import pathlib
def my_reverse_function(my_var: str) -> str:
return my_var[::-1]
If you use from messy import * you aren’t only going to get my_reverse_function you will also get all the other imports from that file, including loads, shutil and pathlib.
>>> from messy import * >>> print(globals().keys()) dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', 'loads', 'shutil', 'pathlib', 'my_reverse_function'])
This is instantly solved by adding __all__
from json import loads
import shutil
import pathlib
__all__ = ["my_reverse_function"]
def my_reverse_function(my_var: str) -> str:
return my_var[::-1]
Notice that loads and the other imports are no longer imported.
>>> from messy import * >>> print(globals().keys()) dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', 'my_reverse_function'])
Name Collisions
Say you have a simple file with a custom loads function, I’ll call it my_stuff.py
import os
def loads(item, value):
"""Load item into environment variables"""
os.environ[item] = value
So you import it and run it and all is good.
>>> from my_stuff import *
>>> loads("MY_VAR", "WORLD")
Then later in your notebook you want that handy reverser function, so you import everything from messy.py. Then you try running loads again and suddenly it broke!
>>> from messy import *
>>> loads("MY_VAR", "WORLD")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: loads() takes 1 positional argument but 2 were given
Suddenly it’s using the loads from json imported from messy.py because there was no __all__.
Real World Example
So if you are writing a library that other people are going to use, say like python-box that other people use and some will surely misuse it in some way, and don’t want them to hurt themselves or mess up their globals, it’s good to add the __all__ .
Look at all these imports needed in that library.
import copy
import re
import warnings
from keyword import iskeyword
from os import PathLike
from typing import Any, Dict, Generator, List, Optional, Tuple, Type, Union
from inspect import signature
try:
from typing import Callable, Iterable, Mapping
except ImportError:
from collections.abc import Callable, Iterable, Mapping
try:
from IPython import get_ipython
except ImportError:
ipython = False
else:
ipython = True if get_ipython() else False
import box
from box.converters import (
BOX_PARAMETERS,
_from_json,
_from_msgpack,
_from_toml,
_from_yaml,
_to_json,
_to_msgpack,
_to_toml,
_to_yaml,
msgpack_available,
toml_read_library,
toml_write_library,
yaml_available,
)
from box.exceptions import BoxError, BoxKeyError, BoxTypeError, BoxValueError, BoxWarning
__all__ = ["Box"]
But because it uses the __all__ it give the user a very neat import if they call this directly.
>>> from box.box import * >>> print(globals().keys()) dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', 'Box'])
If It didn’t have that, it would look like:
>>> from box.box import * >>> print(globals().keys()) dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', 'copy', 're', 'warnings', 'iskeyword', 'PathLike', 'Any', 'Dict', 'Generator', 'List', 'Optional', 'Tuple', 'Type', 'Union', 'signature', 'Callable', 'Iterable', 'Mapping', 'box', 'BOX_PARAMETERS', 'msgpack_available', 'toml_read_library', 'toml_write_library', 'yaml_available', 'BoxError', 'BoxKeyError', 'BoxTypeError', 'BoxValueError', 'BoxWarning', 'NO_DEFAULT', 'NO_NAMESPACE', 'Box'])
Defining imports in __init__.py
Again, James does also touch on a great design principal that you should just declare the exports explicitly in your __init__.py file. Notice I am doing from box.box import * ? You shouldn’t be doing that anyways, rather from box import *. Because python-box is already using the technique he described, by just defining the imports they want exposed in the root __init__.py
from box.box import Box from box.box_list import BoxList from box.config_box import ConfigBox from box.exceptions import BoxError, BoxKeyError from box.from_file import box_from_file, box_from_string from box.shorthand_box import SBox, DDBox
So if you did from box import * you would only get those specific items, even if they didn’t have the __all__ in the box.py file.
>>> from box import * >>> print(globals().keys()) dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', 'Box', 'BoxList', 'ConfigBox', 'BoxError', 'BoxKeyError', 'box_from_file', 'SBox', 'DDBox'])
Why does it matter?
Explicit is better than implicit.PEP 20
Because if you have code used by other people, this could save them someday without them ever realizing it. Avoiding a hidden name collision, or prevent using wrong function that’s named the same elsewhere. Just think about it, I would bet you’ve seen a lot of functions just named loads, dumps, run or similar.
Yes, it is better to avoid import *, and yes it’s better to have a well structured project, but you can’t stop other’s from using Python as they see fit. And import * is fast and valid. You shouldn’t ignore convention because it’s not convenient to your way of coding if your library is used by others.
