Coding

Replacing NZXT’s CAM software on Windows for Kraken

NZXT Kraken coolers are awesome for CPUs or GPUs. Their CAM software on the other hand is slow, bloated and possibly stealing your data. Thankfully, there are open source alternatives available.

The option that I will walk you through using is a command line tool that doesn’t need to be constantly running in the background, and doesn’t require any internet connection to work.

APRIL 2019 UPDATE: liquidctl now has the best support all around and automated windows builds! Use this and forget about everything below!

UPDATE: I have created a standalone executable for Windows for those that do not want to bother with the steps below. It is available on github. (For x61 and other second gen users, you will still need to update the driver as part of step five below.)

There are a few different libraries to control third generation coolers ( Kraken x62, x72, x52 and x42 ). The only one that supports fan control for second generation ( Kraken x61, x41 and x31 ) as well as third on Windows that I have found is liquidctl (as of 2/6/2019 still in an experimental dev branch, can check on the issue directly for updates ).

Please note: none of this is my own software, and this is only a guide based on my own experience. This is fully “as-is”, no warranty or guarantee it won’t harm your hardware or other software.

I am going to get a little more detailed for these steps than usual, as I want to make this easy for anyone’s skill level. For power users, here are the abbreviated steps:

  1. Download and install Python 3.7+ x86 version
  2. Create Python virtual env
    • run command python -m venv venv
    • Activate it venv/Scripts/activate.bat
  3. Downloaded libusb
    • extracted the files from MS32/dll into the venv/Scripts folder
    • You may need an extractor program like 7zip
  4. Downloaded liquidctl
    • Make sure you are in the activated venv
    • pip install liquidctl
  5. Kraken x61 / gen two users only – Downloaded zadig and install libusbK driver for the Kraken device
    • WARNING – CAM will no longer be able to use this device unless you uninstall this new driver
    • Select Options > List all Devices
    • Find the device “690LC” in the dropdown list, should have USB ID of 2433 B200
  6. Run liquidctl to change your fan speed!
    • liquidctl set fan speed 60

As of time of writing their software does not by default include the code for changing the logo color for the second generation Krakens (x61, etc..), but I’ll show you how to add it easily. Checkout the liquidctl repo, that now has great support for older devices, as well as some EVGA and Corsair support!

Download Python

For those of you who do not already have it, we will need to make sure there is a version of Python 3 on your system. Go to the official download page from the Python Software Foundation at https://www.python.org/downloads/windows/. Click on the top link for “Latest Python 3 Release” (version may be newer than shown below.)

Scroll down to the bottom of the next page, and select the “Windows x86 Executable Installer”. Yes, install the x86 version, aka 32-bit , even if you have 64-bit Windows. If you do happen to download and install the x86_64 version instead, you just will have to use a different libusb dll (thankfully included in same download bundle).

After the exe is downloaded, double click on it, and lets make a few changes during the installation to make your life easier. First make sure to add Python to PATH. This will make it possible to run python from the command line without specifying the full path to it’s executable. Then click on “Customize installation”.

Next page we don’t need to touch anything. If you are only going to use python for only this, you can remove Documentation and Python test suite and it will still work fine and reduce bloat.

Final page we just want to make sure it’s installed the program files directory instead of the obscure user app dir that it uses by default. Then hit install.

You should now be able to open python easily in the command prompt. You can open it by hitting the Windows key on your keyboard (or manually opening start menu), typing “Command” and then click on “Command Prompt”.

C:\Users\me>python --version
Python 3.7.2

Create a virtual environment

By creating a virtual environment you isolate the new libraries you are going to install from the system version of Python, eliminating possible future headaches if you need to install conflicting packages.

Thankfully it is super simple to do, open a command prompt, navigate to the directory you want a new folder with the virtual environment in, and type:

python -m venv venv

This is simply saying “Run python, using module venv -m venv to create a virtual env at the venv directory. Then you want to “activate” it so that your command prompt will instead use that new isolated version of Python and Python tools (like pip).

venv\Scripts\activate.bat

This will cause your command prompt to give a nice little notification that you are now using that venv:

C:\Users\me> venv\Scripts\activate.bat

(venv) C:\Users\me>

Remember the path you installed the virtual env too, we will use it later!

Install libusb DLLs

Just because life has to be a little extra difficult, libusb only pushes their releases in 7z and tar.bz2 archives. Which means, if you don’t already have a tool that can open these type of files, head over to https://www.7-zip.org/download.html and download either the exe or msi for your version of Windows (if you are unsure, just get the 32-bit exe one).

Installing it should be fine with the defaults, feel free to change as you want. When done, go over and download libusb. The developers of
liquidctl suggest you use 1.0.21 at time of writing so we are linking directly to that version https://github.com/libusb/libusb/releases/tag/v1.0.21.

After download, extract those files with 7zip.

Then you will want to go into the new directory, and find the proper DLLs for your installed version of Python. If you followed this guide so far, that will be the MS32 folder.

You are going to copy those into your virtual environment directory Scripts directory, as that is by default added to the PATH when you activate the virtual env. So if I was in the C:\Users\me directory when I ran python -m venv venv I would copy those three files into C:\Users\me\venv\Scripts.

They look a little out of place, but meh, it works.

Change the Kraken drivers (x61, x41, x31 Only)

Sadly (or not so sadly) we can’t use the default second gen Kraken drivers installed by CAM. Which also mean that when we switch these, CAM will no longer be able to interface with the Kraken unless this new driver is removed via the Driver Manager.

ONLY FOR KRAKEN x61, x41 and x31! Skip ahead to the next section for anything else.

Go to https://zadig.akeo.ie/ and download their awesome driver switcher tool.

Run it as an administrator, then go to Options and select “List all Devices”.

Chose 690LC from the drop down list. It’s USB ID should be 2433 B200. Select either the libusb-win32 or libusbK driver from the right hand tiny select buttons. (I had trouble the first time timing out on the libusbdrivers so I switched to libusbK, your mileage may vary.) Then hit ‘Replace Driver’.

This will probably take a while. Hopefully it pops up a success window in a few seconds or minutes. If not, I suggest waiting a full ten minutes, then switching to the other driver type (aka select libusbK if you first tried libusb-win32) and trying again before running to their github for help.

Install liquidctl

We’re almost there!

Switch back to the command prompt and make sure you are in the active virtual env. If you don’t see (venv) at the start of the line, go back and take a look at the “Create a virtual environment” section.

If you are using this for a newer third generation device, like the Kraken x62, you can use the regular module.

pip install liquidctl

However, if you do need second generation support for the Kraken x61, x41 or x31, you will need to grab the experimental branch. Please be aware that this may be at an unstable state at time of your download. You can follow current issues directly in the pull request.

pip install git+https://github.com/jonasmalacofilho/liquidctl.git@add-second-gen-krakens

This will install the command in the active virtual environment as liquidctl! Keep in mind, you will need to activate this virtual environment to use this command.

Using liquidctl

You can find the full list of what you can do via --help or from their github. But to start you want to list out devices detected on the system, initialize them, and view their status.

(venv) C:\>liquidctl list
Device 0, Asetek 690LC (NZXT, EVGA or other) (experimental)

(venv) C:\>liquidctl initialize
report: failed to (pre) open (control: open), ignoring

(venv) C:\>liquidctl status
Device 0, Asetek 690LC (NZXT, EVGA or other) (experimental)
report: failed to (pre) open (control: open), ignoring
Liquid temperature          31.2  °C
Fan speed                   1320  rpm
Pump speed                  3600  rpm
Firmware version         0.7.0.0

For the Kraken x61 and gen-two devices, the only thing you can do* is set the fan speed. (* As of time of writing, could have come a long way by whenever ‘now’ is. Also, you can used my “hack” below to change the logo color.)

(venv) C:\> liquidctl set fan speed 100

If you are using third gen devices, you can have fun with pump speeds or colors as well!

liquidctl set ring color fading 350017 ff2608
liquidctl set logo color spectrum-wave --speed slowest
liquidctl set pump speed 90

Multiple Devices

If you happen to have multiple supported devices on your system, you will have to select which one you want to use. First figure out which device is numbered as which.

(venv) C:\> liquidctl list
Device 0, Asetek 690LC (NZXT, EVGA or other) (experimental)
Device 1, NZXT Kraken X (X42, X52, X62 or X72

Then just append --device <num> to any follow up commands.

liquidctl --device 1 set ring color spectrum-wave
liquidctl --device 0 set fan speed 100

You can see more detailed options including color modes for Kraken devices here.

Getting color working for the Kraken x61

If you don’t code and are never going to use this virtual environment for anything other than this tool, don’t fret. It’s a copy paste operation! (For others that don’t want to modify their site-packages file and know what that means, you can do a local develop mode install via setup.py and play to your heart’s content.)

Okay, so all we need to do is simply add a new section of code to one of the files. You’ll have to navigate a few layers deep into the virtual environment you created: venv\Lib\site-packages\liquidctl\driver there is a file called asetek.py that you need to open. If you installed the full Python suite, you should be able to easily right click and edit in IDLE.

Scroll to the bottom of the file, and paste the following block of code:

    def set_color(self, channel, mode, colors, speed):
        """Set the color of the logo."""
        modes = ('fixed', 'alternating', 'blinking', 'off')
        speeds = {
            'fastest': 1,
            'faster': 2,
            'normal': 3,
            'slower': 4,
            'slowest': 5
        }
        try:
            speed = int(speed)
        except ValueError:
            if speed not in speeds:
                LOGGER.warning('Speed must be a value between 1 and 255, setting to 1')
                speed = 1
            else:
                speed = speeds[speed]
        else:
            if speed < 1 or speed > 255:
                speed = 1
                LOGGER.warning('Speed must be a value between 1 and 255, setting to 1')

        if channel != 'logo':
            LOGGER.warning('Only "logo" channel supported for this device, falling back to that')

        if mode not in modes:
            raise NotImplementedError('Modes available are: {}'.format(",".join(modes)))

        if mode == 'off':
            color1, color2 = [0x00, 0x00, 0x00], [0x00, 0x00, 0x00]
        else:
            color1, *color2 = colors
            if len(color2) > 1:
                LOGGER.warning('Only maximum of 2 colors supported, ignoring further colors')
            color2 = color2[0] if color2 else [0x00, 0x00, 0x00]

        self._begin_transaction()
        data = ([0x10] +
                color1 +
                color2 +
                [0xff, 0x00, 0x00, 0x37,
                 speed,
                 speed,
                 0x00 if mode == 'off' else 0x01,
                 0x01 if mode == 'alternating' else 0x00,
                 0x01 if mode == 'blinking' else 0x00,
                 0x01, 0x00, 0x01])
        data.to_yaml(filename=self.data_file)
        self._write(data)
        self._end_transaction_and_read()

Because Python is spacing specific, make sure the def set_color lines up with the def _write above it. It should look something like this (code may change vs what is in the image):

Then, make sure to save the file, by either hitting Ctrl+s or go to File > Save. The options for colors are quite limited for second gen devices versus third gen, but at least you can use them!

ModeColors to supply
off0
fixed1
alternating2
blinking1
liquidctl --device 0 set logo color blinking ffffff --speed 2

Now you can issue a command to set the colors like normal. Provide --speed to set number of seconds between alternating colors or blinking.

If you have any issues with the colors command, please let me know. If you have issues with the program itself, please open an issue directly on liquidctl github page.

Windows Auto-start

Alright, I’ve played around and have everything set perfectly. But wait, I rebooted and it’s all gone! NOOOO!

Thankfully, it’s rather simple to have your configuration auto load on startup. We simply have to put the commands into a script, and put it into the windows auto start directory.

Open your run menu by hitting the Windows Key plus r at the same time (or typing in run to windows search). When the little box pops up, type in shell:startup and hit OK. It will take you to a deep dark folder like C:\Users\me\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup

Open your favorite text editor (or if you know what you’re doing, just right click in the folder, create a new text file, then rename it with a .bat extension.) Keep that window open, so we can copy it’s path.

Now to create the small script. This script’s first command is to go into the venv\Scripts directory, so make sure to change to the full path of where that is in your version. Then, simply modify the liquidctl commands to the ones you want to issue.

cd C:\YOUR_OWN_FULL_PATH_HERE_MAKE_SURE_TO_CHANGE\venv\Scripts
call activate.bat

liquidctl --device 0 set logo color fixed ffffff
liquidctl --device 0 set fan speed 30 25  40 50  50 100

liquidctl --device 1 set logo color fixed ffffff
liquidctl --device 1 set ring color fixed ffffff
liquidctl --device 1 set pump speed 30 50  40 75  50 100
liquidctl --device 1 set fan speed 30 25  50 50  60 75  70 100

Now when you go to save this new file, copy that long path from the open shell:startup directory into the save bar so it is saved into that directory. Then save the file as a name you will remember, like kraken_control.bat just make sure it ends in .bat. In notepad, you may have to change the dropdown from Save as type to All files.

You should now see that file in the startup directory. Go ahead and double click on it to make sure it works. Easiest way to test is first manually change your Kraken colors to something different and make sure running the script changes them back.

If it doesn’t work, most likely a command is miss typed. Copy line for line into a cmd prompt and make sure it works for you there. (Or just run the .bat file from an open cmd prompt if you know how so you don’t have to copy-paste.)

That’s all for this tutorial. I hope you found it useful and stress free! If there is something that needs improved please let me know in the comments below.

Personal Cloud Media Server – Encrypted, Streamable, Affordable…Possible?

Is it possible to build a personal media server that is hosted in the cloud, while making privacy, security, and accessibility paramount?I wanted to find out, and this post will dive into the options available to achieve such a possibly as well. (Spoiler: I did end up making my own software to do just this!)  

First, of course, is the why even try this when other options already exist? For example, Plex and Subsonic are some great options if you want to host a media server from your own home. The catch is then you have to have good upload speeds, storage space, an always running server or NAS, and concerned about how private your data really is. Because at the end of the day these are companies, not just software, and they are beholden to the requests of government agencies. They also have user all data in a single, potentially hackable, silo.

Cost Breakdown

So first, fast upload speeds. If you got it, you’re golden, but if your ISP doesn’t offer high enough speeds, you’re screwed. And even if you do have fast upload speeds you now need a server with hard drives that is always being fed electricity. Time to figure out what it costs to remove that need entirely.

On the flip side, a media server is pretty simple, logistics wise. You need a server to host the web page or API, and a storage provider. These could even be the same thing, however I have yet to find a price conscious option that includes both. Instead, for my personal needs, I priced out the difference between buying a 2-bay NAS (as it offers low electric and data redundancy) and using a local server, and constantly paying for an online one storage roughly 2TBs of data (BackBlaze for storage, DigitalOcean for webserver).

 Local Low  High     Cloud Low  High 
2-Bay NAS $150 $300   2TB
Storage 
$140 $600
2TB HDDs $70 $130   Web
server 
$25 $80
Electric $5 $30        
Internet  $0 $50        
             
Immediate
Cost
 $220  $430        
 Yearly Cost $5 $80     $165 $680

And the numbers speak for themselves. It is much, much cheaper,and more viable to buy a NAS and use the established software. The low end of the yearly cloud costs would match the cost of the high end NAS after only five years. Only an idiot with a paranoiac need for total control and security would even think about making their own software and paying so much to host it online.

I naturally started working on the architecture for how to build my cloud media player after the price analysis.

Design

Having content that is both easily streamable and encrypted is a doozy. It wouldn’t really have been possibly for the his and hers at home a few years ago.But thanks to MPEG DASH and HLS, we now have video formats with those features built in!

HLS is far more common, but it is a proprietary format developed by Apple and doesn’t have nearly the same feature set as DASH. (Note: Apple should rename their company to Sour Apple, because they refuse to support the internationally standardized DASH format because they hate competition.) So for my own purposes, I chose MPEG DASH.

The real downsize to either of these formats though, is now you have to re-encode all your videos before uploading them, ugh. But it really can’t be helped, and then at least it standardizes your library. After figuring out a bit more of how DASH works, I created a super basic structure I wanted to follow:

The webserver needs to allow for finding and playing the encrypted movies. DASH supports multiple DRM methods, but the best option for a home user is ‘cleartext DRM’ aka a password. Well, you don’t want to store raw passwords in the database, so that means anything in there also has to be encrypted. Oh, and if you really want to storage provider to have no clue what’son there if they scan your stuff, that means subtitles and cover files need to be encrypted too. Oy.

But I really wanted to see if this was possible and learn this new tech, so I plowed on. I also was heavily working with JavaScript at work,so I wrote it to use Node instead of my beloved Python. Two weeks, forty dependencies and a job change later, I had ZABAVA!

Solution?

Zabava translates to “fun” or “entertainment” from Bosnian / Czech / Croatian(and maybe more?) and I thought it was a cool sounding word. Every time I say it aloud, I imitate Jim Carrey from Ace Ventura saying “shikaka”. (No idea if that’s correct, but no one’s stopped me yet.)

It has user authentication with JWT tokens. Thought, admittedly just single user right now with admin rights.

It of course supports editing video information and changing cover file.

And has a script that allows for automatic converting videos to DASH, adding them to the DB and uploading them to the storage provider. I only designed the backend for BackBlaze B2 currently, as that is what I use, but it has a fairly agnostic provider setup to allow easy creation of others.

Sexy right? I am quite proud I was able to get nearly everything to work as envisioned.

Of course, not all fairy tales have the ending we imagine. Some videos still don’t like being converted or played via DASH format and it takes forever and a day to convert and upload terabytes of media files. The code is also not up to my personal quality standard, as a lot was written while figuring out how the tech worked without consideration to overall architecture.

In the end, after uploading all my media and not using it for a few months, I stopped work on the project and have bought a home NAS anyways (as I needed some solution for tons other files as well.)

I may go back and refactor it some weekend I am in a crazy mood, but I don’t think it will fit my person standard for code quality. However, if you are interested in the code and working on it yourself, you can find it on my github

Summary

I learned a lot about building a streaming site and different security methodologies for it, so I even though I wouldn’t qualify the code as a success, it’s surely a personal win.

If I were to do this again I would of course do things a little differently:

  • Try using HLS instead of DASH for wider support
  • Write the backend in Python instead of Node
  • SQL instead of Mongo (in my defense I was using Node at the time)

So, lets go over the criteria again:

  • Encrypted – Yes!
    • Everything was secure
  • Streamable – Yes!
    • Some conversions might have issues until tweaked right, but majority of the content worked as expected.
  • Affordable – No
    • The NAS that I ended up buying cost more than streaming the media, but it was used for a lot more than just the few VHS backups I have. So not the cheapest option, but not a bank breaker.
    • Oh wait, factor in the time spent building the new app… yeah, it ain’t cheap.

As expected, you can’t have all the perks with no downsides, but if Security and Accessibility are your goals more than cost, something like this might interest you.

Is the world ready for Python 3?

The trek from Python 2 to Python 3 has been drawn-out, arduous and fraught with perils. How close are our dear Knights developers to all reaching the long sought glory of Python 3?

Quest for the Python 3 – Artwork by Clara Griffith (Link may contain NSFW art)

PIP Downloads

Let’s first jump into what is being used the most currently. This data examines fifteen different libraries downloaded via PIP for a particular Python version. We are only including 2.7 and 3.4+, the Python Versions that are currently supported.

The libraries analyzed are ones that have over 10K stars on github and have been downloaded via PIP. The contenders are: celery, django, flask, ipython, keras, mitmproxy, numpy, pandas, python-box, requests, scrapy, selenium, tensorflow, and tornado. (To be fair, numpy and python-box didn’t have 10K stars, but I used them in the script to make these graphics, so gave them some spotlight too.)

As of January 2019, Python 3 downloads are eclipsing Python 2 by over 20% with Python 3.6 bringing over 39% of it, almost directly matching Python 2.7’s total.

That is good, but not great news. Thankfully Python 2 won’t just stop working at the end of this year, but those are rookie Python 3 numbers, we got to pump them up!

Of course, we have to remember this is a small subset of all downloads. Subsequently, pip downloads themselves don’t tell the whole tale, but this does give us an idea of how things of are going.

This is accomplished by using the PyPI BigQuery data and some SQL (adapted from Artem Golubin’s post about this from last year), then throwing it into matplotlib.

SELECT
  SUBSTR(details.python, 0, 3) as python_version,
  COUNT(*) as download_count
FROM
  TABLE_DATE_RANGE(
    [the-psf:pypi.downloads],
    DATE_ADD(CURRENT_TIMESTAMP(), -30, "DAY"),
    CURRENT_TIMESTAMP()
  )
WHERE
 details.installer.name='pip' and
 file.project = 'requests' -- change project name here
GROUP BY
  python_version
ORDER BY
  download_count DESC
LIMIT 100

Library Brawl: Who’s the Python 3 champs?

In this head to head, we are going to compare two similar libraries, and see how they are doing on the switch to Python 3.

Web Frameworks

The first two up are very popular web frameworks to develop in, Flask and Django.

It’s a dead heat! Both libraries are doing well at attracting developers with a fresh mindset.

Machine Learning

The most popular github package by far was tensorflow with over a hundred thousand stars. Here it’s paired against it’s younger brother keras, which actually depends on it (or other AI tools) to operate.

Machine learning needs to teach it’s developers how to update! It’s a sad day for AI.

Hacker vs Web Scraper

Okay, not really directly comparable tools with a man-in-the-middle proxy and a web scraper, but it’s still an interesting match up.

With this duo I was surprised they didn’t have a higher correlation. I was honestly expecting the mitm tool to have less Python 3 love, as a lot of “hacker” tools depend on the broken way Python 2 handles strings vs unicode, thus are hard to update.

Good job hackers, always keep your tool belt fresh! Scrapers….scrape it together.

Data Science

The last head to head is for the data scientists out there, and you got science in your name and numbers in your veins, you should be at the bleeding edge of tech!

Ouch, yinz need to get with the times.

Python Version Developers Use More Often

This is some hard to gather data as an individual, so I’m going to have to cheat and just base this information off JetBrain’s yearly state of the ecosystem reports from 2017 and 2018.

In 2017, 53% of devs reported using Python 3 as their main language, which went up 22% in 2018 to 75%. Based on those two points of data, we can come to a crystal clear, no doubt conclusion to how many developers will be using Python 3 as their main language in 2019.

That’s right, based on the past two year trend, 97% of developers should be using Python 3 in 2019.

Okay, well, maybe not. But I personal expect that number to be over 90% by the time Python 2 is EOL, which is excellent news.

Operating System Default Language

OSes have a fun time of being in the cross hairs of everyone from desktop to server users, trying to figure out the right combo of what’s best for their users and for their own technology stack going forward. Every major Linux distribution agrees Python 3 is the way to the future and they will need to change over. The hard part is deciding when it will impact the users least and best for their own release cycle. This has caused lots of headaches over the years. So where do we stand now?

OSPython Version
Windows 10None
OSX 10.82.7
Debian 92.7
RedHat 8*3.6
Fedora 293.7
Ubuntu 19.04*3.7

(* denotes upcoming releases this year)

Windows has the easy stance of just saying “do it yourself” and Mac is, as usual, not bothering to innovate and just hum along until it breaks. Thankfully most Linux distros, which power the internet, are either already updated or updating this year. I haven’t seen for sure that Debian 10 will be released with Python 3 or that it w ill be out before year’s end, but I would be surprised if either were not true. Then there’s Arch linux. Arch has had Python 3 as the standard for almost as long as it existed, good boy!

Are we ready?

In all honesty, we are. We are far more prepared for this than the financial sector was ready for Y2K, and we all survived that. Moreover, there are always going to be code bases that can’t update to the latest version easily, but that’s true across the entire software development world. That and the fact the Python Software Foundation has given an extended eleven years which has allowed for even the slowest of companies to have ample time to migrate to Python 3.

Python 3 everywhere? Bring it on!


Stop using plus signs to concatenate strings!

In Python, using plus signs to concatenate strings together is one of the first things you learn, i.e. print("hello" + "world!"), and it should be one of the first things you stop using. Using plus signs to “add” strings together is inherently more error prone, messier and unprofessional. Instead you should be using .format() or f-strings.

Hunter – Artwork by Clara Griffith

Before diving into what’s really wrong with + plus sign concatenation, we are going to take a quick step back and look at the possible different ways to merge strings together in Python, so we can get a better understanding of where to use what.

Concatenating strings

When to useWhen to avoid
+NeverAlways
%Legacy code, logging modulePython 3+
formatEverywhere
f-stringPython 3.6+When you need to escape characters inside the {}s
joinOn an iterable (list, tuple, etc) of strings

Here is a quick demo of each of those methods in action using the same tuple of strings. For an already existing iterate of strings, join makes the most sense if you want them to have the same character(s) between all of them. However, in most other cases join won’t be applicable so we are going to ignore it for the rest of this post.

variables = ("these", "are", "strings")

print(" ".join(variables))
print("%s %s %s" % variables)
print("{} {} {}".format(*variables))
print(f"{variables[0]} {variables[1]} {variables[2]}")
print(variables[0] + " " + variables[1] + " " + variables[2])

# They all print "these are strings"

In many cases you will have other words or strings not in the same structure you will be concatenating together, so even though something like f-strings here looks more cumbersome than the others, it wins out in simplicity in other scenarios. I honestly use f-strings more than anything else, but .format does have advantages we will look at later. Anyways, back to why using plus signs with strings is bad.

Errors lurking in the shadows

Consider the following code, which has four different perfectly working examples of string concatenation.

wait_time = "0.1"
time_amount = "seconds"

print("We are going to wait {} {}".format(wait_time, time_amount))

print(f"We are going to wait {wait_time} {time_amount}")

print("We are going to wait %s %s" % (wait_time, time_amount))

print("We are going to wait " + wait_time + " " + time_amount)

# We are going to wait 0.1 seconds
# We are going to wait 0.1 seconds
# We are going to wait 0.1 seconds
# We are going to wait 0.1 seconds

Everything works as expected, but wait, if we are going to put a time.sleep in there, it takes the wait time as a float. Let’s update that and add the sleep.

Concatenation TypeErrors

import time

wait_time = 0.1 # Changed from string to float
time_amount = "seconds"

print("We are going to wait {} {}".format(wait_time, time_amount))

print(f"We are going to wait {wait_time} {time_amount}")

print("We are going to wait %s %s" % (wait_time, time_amount))

print("We are going to wait " + wait_time + " " + time_amount)

time.sleep(wait_time)

print("All done!")


# We are going to wait 0.1 seconds
# We are going to wait 0.1 seconds
# We are going to wait 0.1 seconds
# Traceback (most recent call last):
#    print("We are going to wait " + wait_time + " " + time_amount)
# TypeError: can only concatenate str (not "float") to str

That’s right, the only method of string concatenation to break our code was using + plus signs. Now here it was very obvious it was going to happen. But what about going back to your code a few weeks or months later? Or even worse, if you are using someone else’s code as a library and they do this. It can become quite an avoidable headache.

Formatting issues

Another common issue that you will run into frequently using plus signs is unclear formatting. It’s very easy to forget to add white space around variables when you aren’t using a single string with replace characters like every other method. What can look very similar will yield two different results:

print(f"{wait_time} {time_amount}")
print(wait_time + time_amount)

# 0.1 seconds
# 0.1seconds

Did you even notice we had that issue in the very first paragraph’s code? print("hello" + "world!")

Messy

This is the most subjective of my reasons to avoid it, but I personally think it becomes very unreadable compared to any other methods, as shown with the following example.

mixed_type_vars = {
    "a": "My",
    "b": 2056,
    "c": "bodyguards",
    "d": {"have": "feelings"}
}


def plus_string(variables):
    return variables["a"] + " " + str(variables["b"]) + \
           " " + variables["c"] + " " + str(variables["d"])


def format_string(variables):
    return "{a} {b} {c} {d}".format(**variables)


def percent_string(variables):
    return "%s %d %s %s" % (variables["a"], variables["b"], 
                            variables["c"], variables["d"])

print(plus_string(mixed_type_vars))
print(format_string(mixed_type_vars))
print(percent_string(mixed_type_vars))

String format is very powerful because it is a function, and can take positional or keyword args and replace them as such in the string. In the example above .format(**variables) is equivalent to

.format(a="My", b=2056, c="bodyguards", d={"have": "feelings"})

That way in the string you can reference them by their keywords (in this case single characters a through d).

"Thing string is {opinion} formatted".format(opinion="very nicely")

Which means with format you have a lot of options to make the string a lot more readable, or you can reuse positional or named variables easily.

print("{0} is not {1} but it is {0} just like "
      "{fruit} is not a {vegetable} but is a {fruit}"
      "".format(1, 2, fruit="apple", vegetable="potato"))

Slower string conversion

Using the functions from the Messy section we can see that it is also slower when concatenation a mix of types.

import timeit
plus = timeit.timeit('plus_string(mixed_type_vars)',
                     number=1000000,
                     setup='from __main__ import mixed_type_vars, plus_string')

form = timeit.timeit('format_string(mixed_type_vars)',
                     number=1000000,
                     setup='from __main__ import mixed_type_vars, format_string')

percent = timeit.timeit('percent_string(mixed_type_vars)',
                     number=1000000,
                     setup='from __main__ import mixed_type_vars, percent_string')

print("Concatenating a mix of types into a string one million times:")
print(f"{plus:.04f} seconds - plus signs")
print(f"{form:.04f} seconds - string format")
print(f"{percent:.04f} seconds - percent signs")

# Concatenating a mix of types into a string one million times:
# 1.9958 seconds - plus signs
# 1.3123 seconds - string format
# 1.0439 seconds - percent signs

On my machine, percent signs were slightly faster than string format, but both smoked using plus signs and explicit conversion.

Unprofessional

This isn’t only something to call out teammates on during code review, but can even negatively impact you if you’re applying for Python jobs. Using “+” everywhere for strings is a red flag that you are still a novice. I don’t know anyone personally that has been turned away because of something so trivial, but it does show that you unfamiliar with Python’s awesome feature rich strings and haven’t had a lot of experience in group coding.

If you ever saw Batman or James Bond coding in Python, they wouldn’t be using +s in their string concatenation, and nor should you!

Summary

"If" + "👏" + "you" + "👏" +"use" + "👏" + "plus signs" + "👏" + "to" + 
"👏" + "concatenate"  + "👏" + "your"  + "👏" + "strings"  + "👏" + "you" 
 + "👏" + "are"  + "👏" + "more"  + "👏" + "annoying"  + "👏" + "than"  + 
"👏" + "this"  + "👏" + "meme!"

Truffle: going from ganache to testnet (ropsten)

Truffle is an amazing suite of tools created by Consensys to develop smart contracts for the Ethereum blockchain network. However, it can be a bit jarring to make the leap from local development to the real test network, ropsten.

Required Setup

For this walk through, I have installed:

I will be using the default example truffle project, MetaCoin, that you can walk through how to unbox here or follow along using your own project.

First things first, if you do NOT have a package.json file yet, make sure to run npm init. This will turn the directory into a node package that we can easily manage with the npm package manager. So we don’t have to download dependices into the global package scope.

Now we can download all the things we are going to need:

npm install bip39 dotenv --save
  • bip39 – used to generate wallet mnemonic
  • dotenv – simple way to read environment variable files

We got everything development wise we need now.

Storing Secrets outside the code

We will have to create a private key or mnemonic, and that means we need somewhere relatively secure to store it. For testnet stuff, this can be as simple as making sure it’s not being put into version control alongside the code. To that end, we are going to use Environment Variables, and will to store them in a file called .env (that’s it, just an extension basically. Make sure to add it to your .gitignore if you’re using git). To learn more, check out the github page for dotenv. But for our purposes, all you need to know is that this file will have a format of:

ENV_VARIABLE_NAME=someting
ANOTHER_ENV=something else

Accessing testnet

The easiest way to reach out to testnet is by using a provider. I personally like using infura.io (free, just requires registration).  After you register and have your API key emailed to you, make sure you select the URL for the test network and add to the .env file using a variable named ROPSTEN_URL.

ROPSTEN_URL=https://ropsten.infura.io/<your-api-key>

It’s also possible to use your own geth node set to testnet, but that is not required.

Next we are going to create our own wallet, if you already have one set up, like with MetaMask, you can skip this next part.

Creating your testnet wallet

So now you have an place to put your secrets, lets create some. This is where bip39 comes in, it will create random mnemonics which can be used as the basis for private key of a wallet. It will be a series of 12 random words.

We could put this generation in a file, but it’s easy enough to just do straight from the command line:

node -e "console.log(require('bip39').generateMnemonic())"

This will output 12 words, DO NOT SHARE THESE ANYWHERE. The ones I am using below are example ones, and also shout NOT be used. Put them in .env file as the variable MNEMONIC. So now your .env file should now contain:

MNEMONIC=candy maple cake sugar pudding cream honey rich smooth crumble sweet treat
ROPSTEN_URL=https://ropsten.infura.io/<your-api-key>

We have our seed, so it’s time to hook it into our code. In your truffle.js or truffle-config.js file, you will need to now import the environment variables and a wallet provider at the top of the file.

require('dotenv').config()
const HDWalletProvider = require('truffle-hdwallet-provider')

After that is added, we will move down to the the exports section, we are going to add a new network, named ropsten. Then are going to use the HDWalletProvider and supply it with the mnemonic and Ifura url provided via environment variables.

module.exports = {
  networks: {
    ropsten: {
      provider: () => new HDWalletProvider(
        process.env.MNEMONIC,
        process.env.ROPSTEN_URL),
      network_id: 3
    },
  },
}

Test and make sure everything’s working by opening a truffle console, specifying our new network.

truffle console --network ropsten

We can then get our public account address via the console.

truffle(ropsten)> web3.eth.getAccounts((err, accounts) => console.log(accounts))
[ '0x627306090abab3a6e1400e9345bc60c78a8bef57' ]

If you are seeing this same wallet address, you did it wrong. Go back and make your own mnemonic, don’t copy the candy one from above.

Funding the wallet

In your development environment, the wallet already has ETH in it to pay for gas and deploying the contract. On the mainnet, you will have to buy some real ETH. On testnet, you can get some for free by using a Faucet, such as https://faucet.ropsten.be/ or if you’re using MetaMask just use https://faucet.metamask.io/.

Make sure to use the address you gathered from the console for the faucet,  and soon you should have test funds to play around with and actually deploy your contract.

Deploying the Contract

Now where the rubber meets the road, getting your contract out into the real (test) world.

truffle deploy --network ropsten

If everything is successful, you’ll get messages like these:

Using network 'ropsten'.

Running migration: 1_initial_migration.js
  Deploying Migrations...
  ... 0xefe70115c578c92bfa97154f70f9c3fbaa2b8400b1da1ee7cdxxxxxxxxxxxxxx
  Migrations: 0x6eeedefb64bd6ee6618ac54623xxxxxxxxxxxxxx
Saving successful migration to network...
  ... 0xd4294e35c166e2dca771ba9bf5eb3801bc1793e30db6a53d4dxxxxxxxxxxxxxx
Saving artifacts...
Running migration: 2_deploy_contracts.js
  Deploying Capture...
  ... 0x446d5e92d6976bb05c85bb95b243d6f7405af6bb12b3b6fe08xxxxxxxxxxxxxx
  Capture: 0x1d2f60c6ef979ca86f53af1942xxxxxxxxxxxxxx
Saving successful migration to network...
  ... 0x0b6f918ccc8e3b82cdf43038a2c32fe1fef66d0fa9aeb2260bxxxxxxxxxxxxxx
Saving artifacts...

Tada! You now have your custom contracts deployed to testnet!

Or, you got an out of gas error, as it is not uncommon to have to adjust the gas price to get it onto the network, as truffle does not automatically figure that out for you. A follow up post will show how to calculate and adjust gas price as needed.