Software complexity and Python problems

ElectrumSV is written in Python, but recently using Python has become a more error prone and problematic experience.

I’m taking a break from testing the upcoming ElectrumSV 1.3.7 release, so will try and see if I crank out my thoughts on this topic. This article also serves as a place for me to summarise these problems for my own later reference. Maybe other Python developers will benefit from them as well.

As a project grows in complexity it becomes harder and harder to know the effect of a change. Python is of course an open source project that has come to be used by a lot of people and businesses, and has accrued a lot of useful functionality. Many of the valuable tools and projects, and even development of the Python distribution itself, is very likely subsidised through the personal efforts of the people doing the work.

And this subsidisation isn’t limited to Python. It happens with Linux distributions, and their packages. It happens with Go, Rust, Ruby and almost every other language or project that has allows community involvement. Many of the people who contribute their time very likely use what they create personally, even if they don’t use them professionally.

Recent Python problems

A regression in string formatting

A Python f-string using an IntFlag member:

f"You should see a number here: {Constants.SomeValue}"

If Constants.SomeValue has a value of 5, then you should see:

"You should see a number here: 5"

But instead the Linux user was seeing the side effect of it working like this:

"You should see a number here: Constants.SomeValue"

Now where we mostly use this functionality is SQL statements, so what was happening was that this was making the database operations to create a new wallet invalid and the process would error.

This isn’t just a problem in 3.8.6, it’s also a problem in every other Python release that has been made including Python 3.9. So when you find something like this, now you have to take on the cost of reporting the bug. You can’t just go to the bug reporting site and say “I found a bug where this thing is kind of happening”, you have to get your ducks in a row. You need to test where it happens, what it is limited to, whether it’s already been reported and fixed and so on.

There’s no satisfaction to reporting it, as it won’t be fixed in any timely manner and you need to deal with users having broken Python version for years to come. And the developers are probably sitting at home with their families reacting to the report in idle moments, not necessarily being paid to professionally handle it.

Anyway, so I report the problem and they’re very nice about my rushed report and offer a workaround. That’s as much as you can hope for. The project benefits and I pay back a little of the benefit that I gain from the huge collection of collaborative work that is the Python language and ecosystem.

Packaging platform detection problems

As an ElectrumSV developer I need to use both the 32-bit and 64-bit versions of the Python language. Python has a packaging system, and in theory it detects that you are using the Python interpreter for a given architecture. The platform is in this case Windows, and the architecture is either of 32-bit or 64-bit. Python on Windows has a very useful command py which allows easy starting of Python of different versions and architectures.

py -3.7-32 py -3.7-64 py -3.8-32 py -3.8-64 etc..

Unfortunately, the packaging system under some cases ignores the architecture of the interpreter and pays attention to some environmental aspect of the Developer Command Prompt for Visual Studio. This results in corrupt package installation along the lines of some 32-bit packages try and import 64-bit dependencies, and some 64-bit packages try and import 32-bit packages. This errored.

Dual architecture installation problems

I uninstalled Python and deleted the installation locations from disk. Then I reinstalled both 32-bit and 64-bit Pythons, and installed ElectrumSV’s dependencies. What I found out very quickly was that the problems were exactly the same. There was still cross-architectural contamination.

I had to uninstall a dependency once to uninstall some upper layer version, then uninstall it again to uninstall the underlying problematic older version.

At this point, I have given up on using both architectures and just focus on 32-bit as that is what our builds have to target as the lowest common denominator.

Random pip errors

pip install -r contrib\deterministic-requirements\requirements.txt

In our case the above command installs pinned versions for a range of different dependencies at once. However, it doesn’t always work sometimes it seems to succeed and shows weird errors in installing things related to those dependencies. So you run it again, and it seems to install the rest. So now you don’t know whether it broke and partially installed things.

Embedded Python release problems

It turns out that there is an official Python embedded build released along with all the other builds they do, that you can use as a base to build your application over.

The embedded builds for Python 3.9.

Things were going swimmingly until it turns out that there’s something different about the embedded builds that prevents the packaging system working reliably against it. The packaging system maintainers won’t fix this, which kind of undermines the value of the official embeddable releases, but makes sense given that they have to limit the scope of what they support to be able to do the more common cases better.

This problem however isn’t like the other problems, in that it’s documented and official, if not likely obvious until you’ve encountered it.

Summing up

ElectrumSV developer