ElectrumSV progress report

For the most part there have been no critical problems with ElectrumSV that require releases. So we have been working on our next big technical upgrade. This document is intended to describe what we have been doing for that upgrade so far, and what else has to be done.

This is a technically oriented update.

Goal

The focus of this effort is to clean up the wallet code and prepare for a P2P future, removing a lot of the legacy approaches we are stuck with.

Progress so far

We’ve made a lot of great progress getting rid of a lot of technical debt and getting the ground work in place for a lot of the remaining changes.

When we were looking at moving to a database what seems like a long time ago now, we considered encrypting the contents of the database. This turned out to make things a lot more complicated, so we gave up on it. But this factored into the database storage design and resulted in wallet code being much harder to write than it had to be. We also came up with custom ways to do things, that can actually be done much simpler using standard Python mechanics like “thread pool executors”.

The first part of our current refactoring effort was tearing out a lot of this complexity and making things a lot simpler.

The SQLite writer thread

The database solution we use is SQLite, which is wonderful. However for various reasons you kind of need to do database writes on a designated thread, and not on the thread your logic happens to be running on.

For this reason we came up with a callback mechanism that would block in the logic thread until the writer thread completed it’s work.

Our quickly made concurrency tool for database writes.

Python actually has a mechanism for doing this, called executors where work can be delegated to a thread pool. In our case, this is to the designated writer thread. With a minimal rewrite we could switch to using this, simplify our code and make use of the standard concept of Python futures for synchronisation.

The new “proper” concurrency tool for database writes.

Database caching

We had a whole lot of complexity in our database code that cached state in memory, and this was a layer of abstraction getting in the way of making better use of the database. This was the primary legacy of our initial foray into encrypted wallet information in the database, and in our development branch it has now been torn out and replaced with direct database access.

Instead of writing lots of Python logic to ensure consistent data in your wallet, we can just rely on database constraints doing what they should. And we can offload data-based operations onto SQL queries. It is a lot easier to rewrite a query than many layers of Python abstraction. Here’s a database function that calculates the wallet balance.

Previously we supported Python 3.7 or 3.8, but the current release of Python is 3.9.1. Additionally, we have had people complain about Python 3.9 working and they are always Linux users, which is normally where users have some very old version of Python that their distribution provides. And beyond that, the only Python release that supported the latest version of SQLite which had a feature we wanted (probably the new UPDATE FROM support) was 3.9.1. Also it is a waste of our available development time supporting older Python versions for the sake of supporting them.

Windows users get their Python version bundled with their executable, so never think about it. And the same applies for MacOS users.

The original Electrum code we inherited, which Electron Cash still uses, reflects the blockchain. If the blockchain changes, the wallet changes the transactions in the wallet to match. This can be observed by finding a server that accepts higher fee transactions, and broadcasting a transaction. Then switching to a server that does not, and observing the transaction you broadcast disappearing.

The multi-signature refactoring

In early 2020, we spent a lot of time making the formal move to switching from JSON-based storage to database storage. This was part of the move to upgrade to a more future proof model of data storage, and enabled us to provide bare multi-signature as a replacement for P2SH before the Genesis upgrade. But to do it in a way where it was future proof, and not just hacked onto what we had. However, because of the amount of work required we did not spend much time on replacing the Electrum model. Instead we kind of recreated it in a modified form just to limit the scope of the work we were doing to primarily the move to the database and solid multi-signature support.

Peer to peer refactoring

As part of this second refactoring effort, we have now switched away from monitoring the blockchain and observing gap limits as the default way the wallet works. There is no actual peer to peer yet, but the core of the code is ready for when there is.

A user no longer copies an address and gives it to someone, instead they must formally reserve it in the wallet. Then the wallet knows to monitor it and will register it’s payment, and then stop monitoring it for new payments. This will for one thing put an end to spam payments, as we will no longer be monitoring to find them. Similarly, as monitoring the blockchain is how we identify if a transaction is mined, we will still be doing so for that purpose but we will only be observing it for the transactions we are interested in.

This provides the base for our remaining P2P oriented refactoring.

Project X

Nothing mentioned here yet.

Upcoming tasks

If you use ElectrumSV and you want to make sure that what we are planning still suits your needs, you might want to read the following and think it through.

One of the core goals for ElectrumSV is to make sure that people who have coins they have not accessed in a long time, can still access them. This might also be latecomers who have BTC coins, but want to access the original BSV coins they forked from. For this reason it is essential we still support seed based wallet recovery, as an option of last resort.

The vaguest attempt at an example UI.

But we will only be offering blockchain scanning in two cases:

  1. When a user creates an account using existing addresses, keys or seed words, they will be presented a user interface that will do an initial scan. It will only scan while it is open, and will show some form of progress.
  2. Any user who wants to rescan their wallet will be able to manually open the scanner and do so.

It will be a formal process the user will manage, and will not happen automatically.

Users still rely on watch only wallets to observe the movement of the coins. A user will be able to designate a wallet with no spending power to more openly monitor the use of it’s keys and coins than we allow for other types of wallet. It will only work for on-chain payments and will not be guaranteed to work at all moving on towards P2P. For other types of wallet we will not support this at all. This will be the old gap limit-based approach emulated.

Work has already started on MAPI support as an alternate method of transaction broadcast. However, as ElectrumSV does not have a hosted service, we cannot receive merkle proof or double spend notifications. So the work to be done here will be to merge it with the existing ElectrumX-based broadcast and merkle proof support, in a way that is future proof for when we do have hosted services offering SPV channel support among other things.

A random screenshot from the MAPI web site.

The reorg support needs to be rewritten to work with the new subscription model. This is not that complex and is basically “delete proofs above fork height and reacquire”.

Final thoughts

There are a number of tasks we have added to this work related to:

  • Code quality.
  • Modifying ElectrumX to make it have a longer life until the APIs we need.
  • Writing lots of tests to ensure things work the way they should.
  • Polishing the new Windows build, and starting signing our builds.

The end however is in sight. The hardest part is done, and we just need to get the rest done.

We have sprint boards and backlogs on Azure Devops, however they do not allow public access to more than just the work items. You can however do construct queries like this one, if you want to see how we are going.

The sprint board Microsoft will only show ElectrumSV developers and not you.

ElectrumSV developer