Opinion

A subject that is in debated more than standardized

How NOT to call Robinhood’s secret API with Python

This article is meant to go over the basics of calling an API via Python, as well as a critique of a top google result I recently ran across. We will start by using Robinhood’s undocumented API, as I saw this article that had python code so bad I had to try and fix it.

Also, because I can get us each a free stock if you sign up with this Robinhood referral link (seems to be usually around a $10 stock, but they like to advertise you might get Facebook or Visa.)

Murder – Artwork by Clara Griffith

So at the base of the bad example (non-working call), he was using curl calls like:

curl -v https://api.robinhood.com/quotes/XIV/ -H "Accept: application/json"

Which is a great use of curl. However, then the author simply decided to wrap it up in a subprocess call and manually parsed the output. He then states “You could also use PYCurl but I didn’t feel like learning it.” Well, thankfully there is no need to learn it.

How to interact with a REST API, the basic GET and POST

The most common text based web APIs are JSON REST-like nowadays. I add the “like” because there are usually cases a company does not fully stick to the standard, and to be honest is a good decision a lot of the time. Thankfully that doesn’t usually change the behavior of the most simple calls of GET and POST.

A properly implemented GET call on the server’s side won’t change anything on their server. So they are best to practice with. It is possible to do this all with pure python, however the requests library is so well known and has exactly what we want already, so we will stick to that library for the examples. To install it, simply pip install requests.

Example GET call

import requests

response = requests.get('https://api.robinhood.com')
# Check to make sure we got 'good' response, aka in HTTP code 2XX range
if response.ok: 
    # Display the result of the parsed json
    print(response.json())

This should simply print out {}.

Next I thought it would be a good idea to try and log in using that article’s example login. If you don’t have an account (not necessary), you can sign up here which uses a referral link, gives you and me both a free random stock.

Example (Non-working) POST call

import requests

response = requests.post('https://api.robinhood.com/api-token-auth/', 
                         json={'username': YOUR_USERNAME, 'password': YOUR_PASSWORD})
if response.ok:
    print(response.json())
else:
    print(response.status_code)
    print(response.text)

Now wait a minute, we are getting an error.

404
<h1>Not Found</h1><p>The requested resource was not found on this server.</p>

Oh gosh darn it. Not only did the author have bad python code, they have outdated examples! How dare they! (*quickly scans my most recent articles for anything obviously outdated*)

Now we get a quick lesson on the dangers of using undocumented APIs. They have absolutely no guarantees. Anything can change at anytime or even remove access to them. In this case, some people reversed engineered how to login to robinhood using a different endpoint, and have an unofficial API wrapped around it.

So for now, lets use Postman’s excellent echo API that will return what you send it. This time, lets put it in a more reusable function.

Example (good) POST call

import requests

api_root = 'https://postman-echo.com'

def post(path='/post', payload=None, timeout=10, **request_args):
    response = requests.post(f'{api_root}/{path}', 
                   json=payload, timeout=timeout, **request_args)
    if response.ok:
            return response.json()
    raise Exception(f'error while calling "{path}", returned status "{response.status_code}" with message: "{response.text}"')

print(post(payload={'username': 'chris'}))

The endpoint will return a JSON object, and requests will automatically translate that into a python dictionary when called with .json() on the response.

{'args': {}, 'data': {'username': 'chris'}, 'files': {}, 'form': {},  'json': {'username': 'chris'}, 'url': 'https://postman-echo.com/post'}

Lot of fluff, but it is nice that Postman’s API spells out exactly what that endpoint receives. For example, you can test sending form data by changing the post parameter call from json to data.

Now this is doing a better job of checking response and using much more pythonic methods. But let’s take it one step further and do better error catching. Because if this is something you are putting into a bigger program, you probably have custom exceptions and want to be able to catch them in a certain way.

This is going to be a little exception heavy, but is showcasing all the ways you might need to deal with certain errors. I would probably only have one wrapper around the call and the parsing JSON personally, but it is really program dependent.

import requests

api_root = 'https://postman-echo.com'

class MyError(Exception):
    """Exception for anything in my program"""

class MyConnectionIssue(MyError):
    """child exception for issues with connections"""


def post(path='/post', payload=None, timeout=10, **request_args):
    try:
        response = requests.post(f'{api_root}/{path}', json=payload, timeout=timeout, **request_args)
    except requests.Timeout:
        raise MyConnectionIssue(f'Timeout of {timeout} seconds reached while calling "{path}"') from None
    except requests.RequestException as err:
        raise MyConnectionIssue(f'Error while calling {path}: {err}')
    else:
        if response.ok:
            try:
                return response.json()
            except ValueError:
                raise MyError(f'Expected output from {path} was not in JSON format. Output: {response.text}')
        raise MyError(f'error while calling "{path}", returned status "{response.status_code}" with message: "{response.text}"')

print(post(payload={'username': 'chris'}))

If you’re a little rusty on exceptions, feel free to brush up on them here.

How NOT to parse JSON

In the terrible code below, we see the other article’s code manually parsing JSON output. Please for the love of all that is holy do not copy!

import subprocess

class data():
    def __init__(self, stock):
            parameter_list = ['open', 'high', 'low', 'volume', 'average_volume', 'last_trade_price', 'previous_close']

            # replacing CURL call
            out = '{"open": 54 "high": 55 "low": 52 "volume": 6000 "average_volume": 6000 "last_trade_price": 52 "previous_close": 54 "}'
            # back to copied code

            string = out  
            for p in parameter_list:  
                parameter = p  
                if parameter in string:  
                    x = 0  
                    output = ''  
                    iteration = 0  
                    for i in string:  
                        if i != parameter[x]:  
                            x = 0  
                        if i == parameter[x]:  
                            x = x+1  
                        if x == len(parameter):  
                            eowPosition = iteration  
                            break  
                        iteration = iteration + 1

                    target_position = eowPosition + 4
                    for i in string[target_position:]:  
                        if i == '"':  
                            break  
                        elif i == 'u':
                            output = 'NULL'  
                            break  
                        else:  
                            output = output+i

                    if output != 'NULL':  
                        output = float(output)  
                    if p == 'open':  
                        self.open = output  
                    if p == 'high':  
                        self.high = output  
                    if p == 'low':  
                        self.low = output  
                    if p == 'volume':  
                        self.volume = output  
                    if p == 'average_volume':  
                        self.average_volume = output  
                    if p == 'last_trade_price':  
                        self.current = output  
                    if p == 'previous_close':  
                        self.close = output  
XIV = data('XIV')

print('Current price: ', XIV.current)

So let’s do a critique starting from the top. class data():
First, classes should be CamelCase, second, this is a single function, shouldn’t even be a class. However does make a tiny bit of sense considering they are basically using it as a namespace, but there are better ways.

Next is them using subprocess to do a curl command instead of the built in urllib or requests. (Omitted so this code runs.)

Then onto string = out which is just…why? No reason to copy it to a new variable. Just name it what you want in the first place.

Finally the for loop that actually parses the JSON. Which is just painful to look at. If you ever need to parse JSON, without using requests method, just use the built-in json library!

import json

data = json.loads('{"open": 54}')
print(data["open"]) 
# 54

Bam, 42 lines into 3.

Finishing on a high note

Now for as much grief as I am giving this author, I also respect him a lot. He was able to accomplish what he wanted to achieve using Python, a language he had little experience with. To his credit he was clear up front that “I’m not an expert in stock trading nor coding so I can’t say the code is the cleanest”.

He also was willing to write up an article on how to do it for others, which most people don’t dream of doing, so in all earnestness, good job Spencer!

Last mention, free stock (legit actual money) from robinhood, just use this referral!

Top 10ish Python standard library modules

When interviewing Python programming candidates, my wife always likes to ask the simple question, “can you name ten Python standard library modules?” This is harder than most think, as many people will completely blank out and others will be dead wrong. “Requests?” one poor soul answered. It’s a good interview question, as it gives insight onto what people are familiar with and may use regularly. So I sat down and though of of which ones I use and enjoy the most. So here are my top ten(ish) useful, favorite and unordered standard modules.

pathlib

Back in the dark days, you would have to store your path as a string, and call obscure functions under os.path to figure anything out about it. Pathlib removes the headache.

from pathlib import Path

my_path = Path('text_file.txt')
if not my_path.exists():
    my_path.write_text('File Content')
assert my_path.exists()
assert my_path.is_file()

Read more at the pathlib python docs.

tempfile

There are a boatload of uses for a temporary file or directory. Hence why it’s in the standard library. I find myself using them together, inside context managers more often than not.

from pathlib import Path
from tempfile import TemporaryDirectory, TemporaryFile


with TemporaryDirectory(prefix='Code_', suffix='_Calamity') as temp_dir:
    with TemporaryFile(dir=temp_dir) as temp_file:
        temp_file.write(b'Test')

        temp_file_path = Path(temp_file.name)
        assert temp_file_path.exists()

# Make sure file only exists within the context
assert not temp_file_path.exists()

I usually end up using this when a tool or library wants to work with a file rather than standard input, so short lived files in a context manager make life a lot easier. Tempfile python docs.

subprocess

Python is pretty amazing, but sometimes you do need to call other programs. Subprocess makes it easy to execute and interact with other executable across operating systems. Check out my other post on it!

from subprocess import run, PIPE
 
response = run("echo 'Join the Dark Side!'", shell=True, stdout=PIPE)

print(response.stdout.decode('utf-8'))
# 'Join the Dark Side!'

logging

This is probably the most useful built in library for debugging there is, and I see it either unused or misused more than anything else.

import logging
import sys
 
logger = logging.getLogger(__name__)
my_stream = logging.StreamHandler(stream=sys.stdout)
my_stream.setLevel(logging.DEBUG)
my_stream.setFormatter(
    logging.Formatter("%(asctime)s - %(name)-12s  "
                      "%(levelname)-8s %(message)s"))
logger.addHandler(my_stream)
logger.setLevel(logging.DEBUG)
 
logger.info("We the people")

If you haven’t already, go on your first date with python logging! It’s also possible to put all the configuration details into a separate ini or json file, learn more from the logging python docs.

threading and multithreading

Two very different things for widely different uses, but they have very similar interfaces and easy to talk about at the same time. Quick and dirty difference: Use threading for IO heavy tasks (writing to files, reading websites, etc) and multithreading for CPU heavy tasks.

from multiprocessing.pool import ThreadPool, Pool
 
def square_it(x):
    return x*x
 
# On Windows, make sure that multiprocessing doesn't start
# until after "if __name__ == '__main__'" 
 
# Pool and ThreadPool are interchangable in this example 
with Pool(processes=5) as pool:
   results = pool.map(square_it, [5, 4, 3, 2 ,1])
 
print(results) 
# [25, 16, 9, 4, 1]

I did a post on ThreadPools and Multithreading Pools, as I find them the easiest way to work with (multi)threading in Python.

os and sys

The Python world would not exist if we didn’t have all the power and functionality these built-ins bring. You really haven’t coded in Python if you haven’t used these yet, so I won’t even bother elaborating them here.

random and uuid

Maybe you’re making a game…

import random
random.choice(['Sneak Attack', 'High Kick', 'Low Kick'])

Or debugging a webserver…

from uuid import uuid4

# Bad example, but not writing out a whole webserver to prove a point
def get(*args):
request = uuid4()
logger.info(f'Request {request} called with args {args}')

Or turning your webserver into your own type of game…

if user_name == 'My Boss':
    time.sleep(random.randint(1, 5))

No matter which you are doing, it’s always handy to have randomly generated or unique numbers.

socket

The internet runs because of sockets. Modern technology exists because of sockets. Sockets are life, sockets are….annoying low level at times but its good to know the basics so you can appreciate everything written on top of them.

Thankfully the Python docs have good examples of them in use.

hashlib

Need to check a file’s integrity? Hashlib is there with md5 and sha hashes! I created a reusable function to easily reference when I need to do it.

Need to securly store people’s password hashes for a website? Hashlib now has scrypt support! Heck, here is my own function I always use to generate the scrypt hashes.

from collections import namedtuple
import hashlib
import os


Hashed = namedtuple('Hashed', ['hash', 'salt', 'n', 'r', 'p', 'key_length'])


def secure_hash(value: bytes, salt: bytes = None, key_length: int = 128, n: int = 2 ** 16, r: int = 8, p: int = 1):
    maxmem = n * r * 2 * key_length
    salt = salt or os.urandom(16)
    hashed = hashlib.scrypt(value, salt=salt, n=n, r=r, p=p, maxmem=maxmem, dklen=key_length)
    return Hashed(hash=hashed.hex(), salt=salt.hex(), n=n, r=r, p=p, key_length=key_length)

venv

You probably only think of it as a command when you run python -m venv python_virtual_env to create your environments, but it’s run that way because it’s a standard library. Every new project you start or Python program you install should be using this library, so it is used a lot!

Summary

There ya go, 10 or so can’t live without standard libraries! Isn’t it so nice that Python comes “batteries included”?

Is the world ready for Python 3?

The trek from Python 2 to Python 3 has been drawn-out, arduous and fraught with perils. How close are our dear Knights developers to all reaching the long sought glory of Python 3?

Quest for the Python 3 – Artwork by Clara Griffith (Link may contain NSFW art)

PIP Downloads

Let’s first jump into what is being used the most currently. This data examines fifteen different libraries downloaded via PIP for a particular Python version. We are only including 2.7 and 3.4+, the Python Versions that are currently supported.

The libraries analyzed are ones that have over 10K stars on github and have been downloaded via PIP. The contenders are: celery, django, flask, ipython, keras, mitmproxy, numpy, pandas, python-box, requests, scrapy, selenium, tensorflow, and tornado. (To be fair, numpy and python-box didn’t have 10K stars, but I used them in the script to make these graphics, so gave them some spotlight too.)

As of January 2019, Python 3 downloads are eclipsing Python 2 by over 20% with Python 3.6 bringing over 39% of it, almost directly matching Python 2.7’s total.

That is good, but not great news. Thankfully Python 2 won’t just stop working at the end of this year, but those are rookie Python 3 numbers, we got to pump them up!

Of course, we have to remember this is a small subset of all downloads. Subsequently, pip downloads themselves don’t tell the whole tale, but this does give us an idea of how things of are going.

This is accomplished by using the PyPI BigQuery data and some SQL (adapted from Artem Golubin’s post about this from last year), then throwing it into matplotlib.

SELECT
  SUBSTR(details.python, 0, 3) as python_version,
  COUNT(*) as download_count
FROM
  TABLE_DATE_RANGE(
    [the-psf:pypi.downloads],
    DATE_ADD(CURRENT_TIMESTAMP(), -30, "DAY"),
    CURRENT_TIMESTAMP()
  )
WHERE
 details.installer.name='pip' and
 file.project = 'requests' -- change project name here
GROUP BY
  python_version
ORDER BY
  download_count DESC
LIMIT 100

Library Brawl: Who’s the Python 3 champs?

In this head to head, we are going to compare two similar libraries, and see how they are doing on the switch to Python 3.

Web Frameworks

The first two up are very popular web frameworks to develop in, Flask and Django.

It’s a dead heat! Both libraries are doing well at attracting developers with a fresh mindset.

Machine Learning

The most popular github package by far was tensorflow with over a hundred thousand stars. Here it’s paired against it’s younger brother keras, which actually depends on it (or other AI tools) to operate.

Machine learning needs to teach it’s developers how to update! It’s a sad day for AI.

Hacker vs Web Scraper

Okay, not really directly comparable tools with a man-in-the-middle proxy and a web scraper, but it’s still an interesting match up.

With this duo I was surprised they didn’t have a higher correlation. I was honestly expecting the mitm tool to have less Python 3 love, as a lot of “hacker” tools depend on the broken way Python 2 handles strings vs unicode, thus are hard to update.

Good job hackers, always keep your tool belt fresh! Scrapers….scrape it together.

Data Science

The last head to head is for the data scientists out there, and you got science in your name and numbers in your veins, you should be at the bleeding edge of tech!

Ouch, yinz need to get with the times.

Python Version Developers Use More Often

This is some hard to gather data as an individual, so I’m going to have to cheat and just base this information off JetBrain’s yearly state of the ecosystem reports from 2017 and 2018.

In 2017, 53% of devs reported using Python 3 as their main language, which went up 22% in 2018 to 75%. Based on those two points of data, we can come to a crystal clear, no doubt conclusion to how many developers will be using Python 3 as their main language in 2019.

That’s right, based on the past two year trend, 97% of developers should be using Python 3 in 2019.

Okay, well, maybe not. But I personal expect that number to be over 90% by the time Python 2 is EOL, which is excellent news.

Operating System Default Language

OSes have a fun time of being in the cross hairs of everyone from desktop to server users, trying to figure out the right combo of what’s best for their users and for their own technology stack going forward. Every major Linux distribution agrees Python 3 is the way to the future and they will need to change over. The hard part is deciding when it will impact the users least and best for their own release cycle. This has caused lots of headaches over the years. So where do we stand now?

OSPython Version
Windows 10None
OSX 10.82.7
Debian 92.7
RedHat 8*3.6
Fedora 293.7
Ubuntu 19.04*3.7

(* denotes upcoming releases this year)

Windows has the easy stance of just saying “do it yourself” and Mac is, as usual, not bothering to innovate and just hum along until it breaks. Thankfully most Linux distros, which power the internet, are either already updated or updating this year. I haven’t seen for sure that Debian 10 will be released with Python 3 or that it w ill be out before year’s end, but I would be surprised if either were not true. Then there’s Arch linux. Arch has had Python 3 as the standard for almost as long as it existed, good boy!

Are we ready?

In all honesty, we are. We are far more prepared for this than the financial sector was ready for Y2K, and we all survived that. Moreover, there are always going to be code bases that can’t update to the latest version easily, but that’s true across the entire software development world. That and the fact the Python Software Foundation has given an extended eleven years which has allowed for even the slowest of companies to have ample time to migrate to Python 3.

Python 3 everywhere? Bring it on!


Stop using plus signs to concatenate strings!

In Python, using plus signs to concatenate strings together is one of the first things you learn, i.e. print("hello" + "world!"), and it should be one of the first things you stop using. Using plus signs to “add” strings together is inherently more error prone, messier and unprofessional. Instead you should be using .format() or f-strings.

Hunter – Artwork by Clara Griffith

Before diving into what’s really wrong with + plus sign concatenation, we are going to take a quick step back and look at the possible different ways to merge strings together in Python, so we can get a better understanding of where to use what.

Concatenating strings

When to useWhen to avoid
+NeverAlways
%Legacy code, logging modulePython 3+
formatEverywhere
f-stringPython 3.6+When you need to escape characters inside the {}s
joinOn an iterable (list, tuple, etc) of strings

Here is a quick demo of each of those methods in action using the same tuple of strings. For an already existing iterate of strings, join makes the most sense if you want them to have the same character(s) between all of them. However, in most other cases join won’t be applicable so we are going to ignore it for the rest of this post.

variables = ("these", "are", "strings")

print(" ".join(variables))
print("%s %s %s" % variables)
print("{} {} {}".format(*variables))
print(f"{variables[0]} {variables[1]} {variables[2]}")
print(variables[0] + " " + variables[1] + " " + variables[2])

# They all print "these are strings"

In many cases you will have other words or strings not in the same structure you will be concatenating together, so even though something like f-strings here looks more cumbersome than the others, it wins out in simplicity in other scenarios. I honestly use f-strings more than anything else, but .format does have advantages we will look at later. Anyways, back to why using plus signs with strings is bad.

Errors lurking in the shadows

Consider the following code, which has four different perfectly working examples of string concatenation.

wait_time = "0.1"
time_amount = "seconds"

print("We are going to wait {} {}".format(wait_time, time_amount))

print(f"We are going to wait {wait_time} {time_amount}")

print("We are going to wait %s %s" % (wait_time, time_amount))

print("We are going to wait " + wait_time + " " + time_amount)

# We are going to wait 0.1 seconds
# We are going to wait 0.1 seconds
# We are going to wait 0.1 seconds
# We are going to wait 0.1 seconds

Everything works as expected, but wait, if we are going to put a time.sleep in there, it takes the wait time as a float. Let’s update that and add the sleep.

Concatenation TypeErrors

import time

wait_time = 0.1 # Changed from string to float
time_amount = "seconds"

print("We are going to wait {} {}".format(wait_time, time_amount))

print(f"We are going to wait {wait_time} {time_amount}")

print("We are going to wait %s %s" % (wait_time, time_amount))

print("We are going to wait " + wait_time + " " + time_amount)

time.sleep(wait_time)

print("All done!")


# We are going to wait 0.1 seconds
# We are going to wait 0.1 seconds
# We are going to wait 0.1 seconds
# Traceback (most recent call last):
#    print("We are going to wait " + wait_time + " " + time_amount)
# TypeError: can only concatenate str (not "float") to str

That’s right, the only method of string concatenation to break our code was using + plus signs. Now here it was very obvious it was going to happen. But what about going back to your code a few weeks or months later? Or even worse, if you are using someone else’s code as a library and they do this. It can become quite an avoidable headache.

Formatting issues

Another common issue that you will run into frequently using plus signs is unclear formatting. It’s very easy to forget to add white space around variables when you aren’t using a single string with replace characters like every other method. What can look very similar will yield two different results:

print(f"{wait_time} {time_amount}")
print(wait_time + time_amount)

# 0.1 seconds
# 0.1seconds

Did you even notice we had that issue in the very first paragraph’s code? print("hello" + "world!")

Messy

This is the most subjective of my reasons to avoid it, but I personally think it becomes very unreadable compared to any other methods, as shown with the following example.

mixed_type_vars = {
    "a": "My",
    "b": 2056,
    "c": "bodyguards",
    "d": {"have": "feelings"}
}


def plus_string(variables):
    return variables["a"] + " " + str(variables["b"]) + \
           " " + variables["c"] + " " + str(variables["d"])


def format_string(variables):
    return "{a} {b} {c} {d}".format(**variables)


def percent_string(variables):
    return "%s %d %s %s" % (variables["a"], variables["b"], 
                            variables["c"], variables["d"])

print(plus_string(mixed_type_vars))
print(format_string(mixed_type_vars))
print(percent_string(mixed_type_vars))

String format is very powerful because it is a function, and can take positional or keyword args and replace them as such in the string. In the example above .format(**variables) is equivalent to

.format(a="My", b=2056, c="bodyguards", d={"have": "feelings"})

That way in the string you can reference them by their keywords (in this case single characters a through d).

"Thing string is {opinion} formatted".format(opinion="very nicely")

Which means with format you have a lot of options to make the string a lot more readable, or you can reuse positional or named variables easily.

print("{0} is not {1} but it is {0} just like "
      "{fruit} is not a {vegetable} but is a {fruit}"
      "".format(1, 2, fruit="apple", vegetable="potato"))

Slower string conversion

Using the functions from the Messy section we can see that it is also slower when concatenation a mix of types.

import timeit
plus = timeit.timeit('plus_string(mixed_type_vars)',
                     number=1000000,
                     setup='from __main__ import mixed_type_vars, plus_string')

form = timeit.timeit('format_string(mixed_type_vars)',
                     number=1000000,
                     setup='from __main__ import mixed_type_vars, format_string')

percent = timeit.timeit('percent_string(mixed_type_vars)',
                     number=1000000,
                     setup='from __main__ import mixed_type_vars, percent_string')

print("Concatenating a mix of types into a string one million times:")
print(f"{plus:.04f} seconds - plus signs")
print(f"{form:.04f} seconds - string format")
print(f"{percent:.04f} seconds - percent signs")

# Concatenating a mix of types into a string one million times:
# 1.9958 seconds - plus signs
# 1.3123 seconds - string format
# 1.0439 seconds - percent signs

On my machine, percent signs were slightly faster than string format, but both smoked using plus signs and explicit conversion.

Unprofessional

This isn’t only something to call out teammates on during code review, but can even negatively impact you if you’re applying for Python jobs. Using “+” everywhere for strings is a red flag that you are still a novice. I don’t know anyone personally that has been turned away because of something so trivial, but it does show that you unfamiliar with Python’s awesome feature rich strings and haven’t had a lot of experience in group coding.

If you ever saw Batman or James Bond coding in Python, they wouldn’t be using +s in their string concatenation, and nor should you!

Summary

"If" + "πŸ‘" + "you" + "πŸ‘" +"use" + "πŸ‘" + "plus signs" + "πŸ‘" + "to" + 
"πŸ‘" + "concatenate"  + "πŸ‘" + "your"  + "πŸ‘" + "strings"  + "πŸ‘" + "you" 
 + "πŸ‘" + "are"  + "πŸ‘" + "more"  + "πŸ‘" + "annoying"  + "πŸ‘" + "than"  + 
"πŸ‘" + "this"  + "πŸ‘" + "meme!"

Exploit the work of others for profit! (Vega 64 Edition)

As I sit here, anxious for the new AMD Vega 64 to be released, I decide to keep myself busy writing some Python code….designed to text me as soon as a new “rx vega 64” search term showed up on Amazon (I have the patience of a child on Christmas Eve, so sue me.)

When writing code, I try to be asΒ lazy efficient as possible. That means I look for other’s to do the hard part for me. Other people might phrase it more kindly like “don’t reinvent the wheel,” but let’s be real, you are receiving benefit for no cost. So next time a project saves your bacon, consider sending a little cash or cryptocoin to the dev(s). Or throw your hat into the open source community and provide dev work yourself, it’s a great way to learn a lot more about the coding community and gain a lot of experience along the way while still giving back.

Back to the Vega 64 stock tracking tool. It would totally be possible to do that all with the Python internals; using urllibΒ and reΒ to download and find stuff on the page, then using emailΒ to send a message to my phone’s SMS; but that would take forever, and is honestly stupid to do. There are much better tools for that at this point, like requestsΒ and BeautifulSoupΒ , then using some gmail or other common email provider library.

But as Amazon is a rather big website with an API available, there are Python libraries for that API. There are also different ways to easily send a text message to yourself via online services, like twilo. In the end, I created the script using Python 3.6 on Windows (should be cross-platform compatible), and the libraries I used for this were:

python-amazon-simple-product-api
twilio
reusables
python-box

If you are interested in using this as well (comes as-is, no promises it works or won’t bite you) before using the script you will need to get AWS access keysΒ and sign up for twilio,Β then fill in the appropriate variables at the top of the script.

from datetime import datetime
from time import sleep

from amazon.api import AmazonAPI
from twilio.rest import Client
from box import Box, BoxList
from reusables import setup_logger

amazon_access_key = "XXXXXXXXXXXXXXXXXXXX"
amazon_secret = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
amazon_associate_tag = "codecalamity-20" 

twilio_from = "+"  # Twilio phone number
twilio_to = "+"  # Phone number to text
twilio_key = "ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
twilio_secret = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

search_product_keywords = "rx vega 64"
search_product_name_includes = "vega"
search_product_brand = "amd"
search_index = 'Electronics'
search_region = 'US'

log = setup_logger("vega")


def update(search, saved_data):
    new_prods = BoxList()
    for x in (product for product in search if
              search_product_name_includes in product.title.lower()
              and (search_product_brand in product.brand.lower() or
                   search_product_brand in product.manufacturer.lower())):
        if x.title not in saved_data:
            data = Box({"price": float(x.price_and_currency[0]),
                        "url": x.detail_page_url,
                        "updated": datetime.now().isoformat()})
            log.info(f"New item: {x.title} - {data}")
            new_prods.append((x.title, data))
            saved_data[x.title] = data

    saved_data.to_json(filename="products.json")
    return new_prods


def format_message(new_prods):
    product_list = [f"{prod[0]}-{prod[1].price}-{prod[1].url}"
                    for prod in new_prods]
    return f"New Prods: {', '.join(product_list)}"


def send_message(client, message):
    log.info(f"About to send message: '{message}'")
    client.messages.create(to=twilio_to,
                           from_=twilio_from,
                           body=message)


def main():
    amazon = AmazonAPI(amazon_access_key, amazon_secret, amazon_associate_tag)
    # Only search the first two pages to not spam the server
    products = amazon.search_n(2, Keywords=search_product_keywords,
                               SearchIndex=search_index, region=search_region)
    twilio_client = Client(twilio_key, twilio_secret)

    try:
        prods = Box.from_json(filename="products.json")
    except FileNotFoundError:
        prods = Box()

    while True:
        new_prods = update(products, prods)
        if new_prods:
            message = format_message(new_prods)
            send_message(twilio_client, message)
        sleep(60)


if __name__ == '__main__':
    main()

So now that possibly huge pure python standard library multifaceted application has turned into fifty lines of code (not counting imports / globals) designed to do nothing but feed my anxiety as efficiently as possible. Luckily it’s self testing to make sure it works, as it will find the result for the Vega Frontier Edition first, so if you chose to use it, make sure you get that text.

2017-08-07 22:09:23,938 - vega             INFO      About to send message: 
'New Prods: Radeon Vega Frontier Edition Liquid Retail-1579.48-
https://www.amazon.com/Radeon-Vega-Frontier-Liquid-Retail/dp/B072XLR2K7?SubscriptionId=AKIAIF3WXFESZ53UZKDQ&tag=codecalamity-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=B072XLR2K7'

Be warned that this may spam your phone a lot when the product does drop, and that the code doesn’t have a lot of safety checks as-is so may fail and stop reporting. It also doesn’t check if it is available to buy yet, just that the product page for the Vega 64 exists.

If you don’t want to have your own script constantly running, try out sites like nowinstock.net, as they have great options to text or email you when products are available.