Software complexity and Python problems

ElectrumSV is written in Python, but recently using Python has become a more error prone and problematic experience.

I’m taking a break from testing the upcoming ElectrumSV 1.3.7 release, so will try and see if I crank out my thoughts on this topic. This article also serves as a place for me to summarise these problems for my own later reference. Maybe other Python developers will benefit from them as well.

As a project grows in complexity it becomes harder and harder to know the effect of a change. Python is of course an open source project that has come to be used by a lot of people and businesses, and has accrued a lot of useful functionality. Many of the valuable tools and projects, and even development of the Python distribution itself, is very likely subsidised through the personal efforts of the people doing the work.

And this subsidisation isn’t limited to Python. It happens with Linux distributions, and their packages. It happens with Go, Rust, Ruby and almost every other language or project that has allows community involvement. Many of the people who contribute their time very likely use what they create personally, even if they don’t use them professionally.

Recent Python problems

For the last couple of years, we haven’t had that many problems that can be directly attributed to the Python project, but over the last couple of months this has changed.

One of our Linux users, who have to do a lot more work than Windows or MacOS users to get ElectrumSV to work, pointed out that he couldn’t even create a wallet. Looking into this it turns out that the recently release of Python 3.8.6 introduced a regression where format strings no longer made use of the integer values but rather the descriptive name for an “IntFlag” enumeration value.

A Python f-string using an IntFlag member:

f"You should see a number here: {Constants.SomeValue}"

If Constants.SomeValue has a value of 5, then you should see:

"You should see a number here: 5"

But instead the Linux user was seeing the side effect of it working like this:

"You should see a number here: Constants.SomeValue"

Now where we mostly use this functionality is SQL statements, so what was happening was that this was making the database operations to create a new wallet invalid and the process would error.

This isn’t just a problem in 3.8.6, it’s also a problem in every other Python release that has been made including Python 3.9. So when you find something like this, now you have to take on the cost of reporting the bug. You can’t just go to the bug reporting site and say “I found a bug where this thing is kind of happening”, you have to get your ducks in a row. You need to test where it happens, what it is limited to, whether it’s already been reported and fixed and so on.

There’s no satisfaction to reporting it, as it won’t be fixed in any timely manner and you need to deal with users having broken Python version for years to come. And the developers are probably sitting at home with their families reacting to the report in idle moments, not necessarily being paid to professionally handle it.

Anyway, so I report the problem and they’re very nice about my rushed report and offer a workaround. That’s as much as you can hope for. The project benefits and I pay back a little of the benefit that I gain from the huge collection of collaborative work that is the Python language and ecosystem.

I develop on Windows, so when I went to open a console window, I used the Developer Command Prompt for Visual Studio which is an enhanced version set up as an environment for easier development.

As an ElectrumSV developer I need to use both the 32-bit and 64-bit versions of the Python language. Python has a packaging system, and in theory it detects that you are using the Python interpreter for a given architecture. The platform is in this case Windows, and the architecture is either of 32-bit or 64-bit. Python on Windows has a very useful command py which allows easy starting of Python of different versions and architectures.

py -3.7-32 py -3.7-64 py -3.8-32 py -3.8-64 etc..

Unfortunately, the packaging system under some cases ignores the architecture of the interpreter and pays attention to some environmental aspect of the Developer Command Prompt for Visual Studio. This results in corrupt package installation along the lines of some 32-bit packages try and import 64-bit dependencies, and some 64-bit packages try and import 32-bit packages. This errored.

So now I no longer open the Developer Command for Visual Studio unless I explicitly need it for a non-Python reason. This in theory should ensure that I do not get the corruption between one architecture trying to use things intended for another architecture.

I uninstalled Python and deleted the installation locations from disk. Then I reinstalled both 32-bit and 64-bit Pythons, and installed ElectrumSV’s dependencies. What I found out very quickly was that the problems were exactly the same. There was still cross-architectural contamination.

I had to uninstall a dependency once to uninstall some upper layer version, then uninstall it again to uninstall the underlying problematic older version.

At this point, I have given up on using both architectures and just focus on 32-bit as that is what our builds have to target as the lowest common denominator.

The packaging system for Python is called “pip”. You execute a rather standard installation command, which has much in common with similar packaging systems for Linux, and other programming languages.

pip install -r contrib\deterministic-requirements\requirements.txt

In our case the above command installs pinned versions for a range of different dependencies at once. However, it doesn’t always work sometimes it seems to succeed and shows weird errors in installing things related to those dependencies. So you run it again, and it seems to install the rest. So now you don’t know whether it broke and partially installed things.

One of the tasks on my plate is to come up with a new build system for Windows, and the first step is to package a standalone Python environment with ElectrumSV included, that runs as an ElectrumSV application. The standard way this is done is to either embed or extend Python. In our case we can embed if necessary.

It turns out that there is an official Python embedded build released along with all the other builds they do, that you can use as a base to build your application over.

The embedded builds for Python 3.9.

Things were going swimmingly until it turns out that there’s something different about the embedded builds that prevents the packaging system working reliably against it. The packaging system maintainers won’t fix this, which kind of undermines the value of the official embeddable releases, but makes sense given that they have to limit the scope of what they support to be able to do the more common cases better.

This problem however isn’t like the other problems, in that it’s documented and official, if not likely obvious until you’ve encountered it.

Summing up

Development relies on the work of others I don’t pay and whose efforts I use for free. These are the recent problems ElectrumSV has encountered using Python. Using other programming languages hasn’t exactly been a bed of roses either.

ElectrumSV developer