You may have noticed we’re not releasing many new features and new releases for ElectrumSV (ESV). There’s a good reason for this, and this post is intended to both explain why, and to start laying out where we currently see ourselves heading with the ElectrumSV wallet.
Our current focus
We’re based on the open source Electrum wallet (BTC) created by Thomas Voegtlin, which provided us with both a reasonably solid base (given use on a toy blockchain like Bitcoin Core or Bitcoin Cash ABC) of common wallet functionality, and a reasonably solid base of blockchain interaction (again given the same constraints) functionality.
The problem is already stated, both Bitcoin Core and Bitcoin Cash ABC are toy blockchains. People kind of send money back and forwards, and they watch their money, and maybe they use a hardware wallet, but the limits imposed by toy blockchains meant that all the Electrum (BTC)/Electron Cash (BCHABC) had to do was kind of keep looking like they worked for those same purposes.
Electrum (BTC) already had aspirations to serving as a wallet management service for developers, but it never really saw or supported more than token use as far as I could tell. Now that we have real on-chain products being developed in the Bitcoin SV ecosystem, I’ve started building several use cases pumping transactions through ElectrumSV (ESV), and what it confirms is something my fellow developer Neil (AKA Kyuupichan) and I already knew.
What we knew was that while our Electrum project heritage may have provided us with a popular and useful wallet, it requires a lot of the foundations to be cleaned up to support it becoming more than a simple wallet for toy blockchains that believe they are not for use, but rather for store of value.
So this is what we have been focused on for the last couple of months. It’s slow progress but it’s happening.
What is in ElectrumSV’s future?
There are many things we want to achieve with ElectrumSV (ESV).
Separating the user interface from the wallet
At this time the UI code is built directly on top of the wallet. This makes it take more work to do ESV development, and also means that if you want to directly manage a wallet with a UI you have to be able to display the user-interface on the machine the wallet is running on.
If the UI code interacted with the wallet code through a layer of abstraction, then we would have a range of other possibilities, among them being:
- We could write an Electron-based UI which because it also embeds the Chromium browser perhaps even provides a good base for sandboxed browsing of websites from the blockchain.
- We could allow the existing Qt5-based Python UI to attach to any running ElectrumSV instance you can access. This might mean you can run a headless ElectrumSV wallet server on your web host, and tunnel an SSH connection to that host, then access and manage that wallet from your home or work desktop.
There’s a lot of overlap between these possibilities, and even other indirect goals, as well. The web-page UI very likely could be doable in conjunction with the cross-platform Electron-based application, sharing a code-base. Perhaps it will even streamline development of future mobile versions and extensions of ElectrumSV, given a wider range of suitable frameworks and tooling.
Acting as a wallet server for applications
Someone who just uses ESV for a desktop wallet might wonder whether focusing on automated wallet use distracts us from working on things that improve their experience, and the answer is a definitive “no, it doesn’t.”
This uses exactly the same wallet functionality a desktop user uses, it just uses it more extensively, and much more heavily. Already this has highlighted shortcomings in ESV, and areas of work that need to be focused on and provided real world test cases that stress the code in ways that UI-based testing never can.
Bitcoin Core’s Electrum offers one limited application using it as a wallet server. This is it’s merchant server, intended to help users pay payment requests. The user gets a QR code for payment which in turn takes them to a custom page on the merchant server, and then they can use that to see the status of their payment. In reality, it doesn’t do much, and doesn’t require much of Electrum.
For the merchant server, the viewer of the web page actually connects using web sockets to the instance of Electrum running on the web server. This provides the visual update allowing the web page to reflect the viewer’s payment status.
Allowing a merchant’s customers to connect to the instances of ESV running on their web server, is not how I think this sort of thing should be done. So what are the alternatives?
Electrum actually provides an API for usage, and it’s done in a general fashion. Commands are defined, like “sign_transaction”, “broadcast” and “getaddressbalance”, and they are exposed on several interfaces. Interfaces like the command-line, the in-wallet Python console and also the JSON-RPC interface for the built-in JSON-RPC API that runs in the background.
At this time, I believe the path forward to be refactoring this API to provide a solid set of fundamental operations that allow an application access and control over a wallet. The merchant server would connect to this and both direct it to perform wallet operations, and in turn receive events back. One of these would be changes in the status of customer’s payments. Then it is the responsibility of the web site to update the page for the customer.
This actually brings us back to the earlier task of also separating the UI from the wallet server, as this same interface and control/access abstraction could and likely should be used by the UI to connect to the wallet server.
So let’s move on to what it has become obvious we need to do, based on experiments using ESV as a wallet server from an automated application:
More scalable wallet storage
Imagine you’re storing files on the blockchain, and uploading them through your ESV wallet. ESV keeps all the transactions you create or receive. And it keeps them all in one ever increasing in size compressed and encrypted JSON file. Upload a 2 MB photo? It’s probably compressed already, that’s a 2 MB larger wallet to be loaded and saved. Upload a 256 MB video? It’s probably compressed already, that’s a 256 MB larger wallet to be loaded and saved.
The work to move the wallet content out of the JSON file into an encrypted database has already started, and mostly working. As this will change the wallet format for users we need to proceed carefully, and it’s best we drag in as much of what needs to change about the existing wallet storage to reduce the number of migrations. Because we encrypt the data we can’t try and find stuff in real-time, there’s going to be a need for some indexing to be implemented as well.
It’ll also likely make a lot of sense to prune transaction data from the storage and fetch it again as needed, some design will be needed to flesh this out.
More intelligent transaction management
ESV knows about one thing, transactions you have received from the blockchain (those “in the mempool” or mined). This might be transactions you have just made and sent to the network via one of the electrumx servers you are connected to, or it might be transactions that are payments others have made to you.
This is unfortunately something that currently only works for a toy blockchain. Consider the case that you send a second transaction before you receive the first back. ESV has no way of knowing what inputs it used for the first transaction and it is almost guaranteed that the second transaction will naively pick the same inputs therefore creating a double-spend!
What if you created a transaction and gave it to someone else to broadcast, and you’re still waiting for them to do so? Same problem! If you spend inputs in the meantime you could turn that into a double-spend unwittingly.
Design is needed to flesh out the range of different types of managed transactions and how this affects other resources like used inputs.
It’s interesting to note that Electrum has actually implemented unsent transactions, and does lock inputs for those. ESV however never received those changes, and in any case given how much we’ve diverged from them code-wise, we’re going to both implementing it differently and more extensively than Electrum.
More intelligent address management
Addresses are currently generated for the wallet using standard approaches, defined in the specifications BIP32 and BIP44. This is how ESV can import a private key from any of the other wallets in the eco-system and find the payments made to and from that wallet.
Given that there are receiving addresses and change addresses, and a separate linear path through which each type is created, a wallet examines usage down that path until it no longer finds usage. Then it uses addresses past that point as fresh addresses for future use.
At this time ESV subscribes to events relating to every single one of these used and soon to be used addresses, with the electrumx servers it is connected to. As the number of addresses increases this increases the load on the electrumx servers, and as a result slows the service those servers provide this user.
If some functionality allocates a lot of addresses to a transaction, perhaps creating thousands of UTXOs so that they are available for use as needed so that lots of OP_RETURN operations can be made as needed, and that transaction ends up being rejected or discarded what happens to these addresses that were all pre-allocated but never used? We need to have active address management in much the same way that active transaction management ensures we do not double-spend.
So what do we do here? We need a design for this, right? In an ideal world we could assume no address reuse and only subscribe and pay attention to some of the later addresses. Maybe the user could say, hey check this address for me, or go over my past addresses at least this often and I’ll pay for the extra quality of service from an electrumx service, or even use my own electrumx server to do it all. Anyway, this needs to be addressed.
Providing a better user experience
The original Electrum like many open source projects is incrementally developed. It depends on the kindness of strangers who have an interest in something working differently, and them convincing the maintainers that it should be added. Then something resembling that makes it in. The wallet becomes very flexible, but there’s not necessarily an ability, or spent energy making it a better user experience or one that’s intuitive. Or that reduces any confusion from what is already a complicated tool to use, making use of a complicated technology.
So we need to rethink what the interface allows, and how it can be used.
A better Windows executable
Electrum also provides us with the build system that creates the Windows executables. The problem is that it uses pyinstaller, which is also notorious for being used by malware. This means as we make releases users who use virus checkers tend to report that the virus checkers always flag our releases. This has actually gotten worse, in that Windows Defender has started flagging our latest releases.
As long as we use pyinstaller, this is going to continue. We’ll make a release. Then a week or so later, Windows Defender will start flagging and quarantining them. Then I’ll have to upload the “this is a false positive” binaries to Microsoft and within a day they will sort it out.
So what are the alternatives?
Well, there are no other similar packaging tools for Python that are cross-platform to the same level. I’ve looked and gone over the complaints and problems with the alternatives. We’re lucky to have pyinstaller to use in the first place.
But what we could do is write a simple C or C++ executable (here’s some prior thoughts on the subject) that embeds the Python interpreter, and loads a signed bootstrap package of Python code. This would then look to the blockchain for a signed update to download and run, or just run the last update it has. This might be enough. It might even be that if the core application can be built in Visual Studio it could probably even take care of doing builds for the Windows Store for us.
Is it possible that this would solve the problems for us completely? Even if unstable BCHABC zealots were to report us as malware for the executables downloaded from our web site, the users could always opt to use the store to get ESV.
A new user-friendly wallet wizard
When you start ESV sometimes it will show a wallet selection/creation wizard, and sometimes it will launch straight into a wallet. What is actually going on is that it tries to open the last wallet you used. When it succeeds and the wallet has no password, it can open that wallet immediately. When the wallet has a password, it will be ready for you to enter it so that it can then be opened. But you get this same bland “software installer” user interface regardless.
We can do a lot better than this, and plan to. In fact I’ve already started development of a replacement. But that bland looking wallet wizard does a lot of things, and we want to handle the same range of functionality, but do it in a way that less technical users can feel more comfortable in. And we also want to do a lot more automatically, and have the users doing a lot less.
- You know your seed words you are importing are from Centbee? Just click the Centbee option and paste it.
- You have some seed words and you don’t remember where they originated, are too lazy to say, or even just want to watch ESV try and work it out? Just paste them in the “find my wallet” option and ESV should search out all the different possibilities for usage, and then report back on the results. It should take Electrum seed words, BIP 39 seed words, xprivs and even xpubs. Just paste them all in and let it work it out and offer you options.
Here’s a desktop recording of me stepping through it, after I last worked on it, courtesy of the new Windows 10 application recording functionality.
More flexible wallet history access
While Electrum has moved on from it’s old transaction history view, we’re still there using it. This displays all your transactions. And this as the number of transactions increases, make changing the contents of the history view quite slow.
When I worked on Electron Cash, I spent quite a bit of time with a large wallet trying to optimise this and the conclusion I came to was that the problem was that Qt could not handle the given number of entries. It was faster to rebuild the list each time it changed, than to try and selectively add or remove entries.
The very near future of this user interface is some form of paging. This is a necessity with the move away from storing the wallet data in the one large loaded JSON file, to the encrypting of individual chunks in a database that I mentioned earlier.
While the use of database makes wallet startup faster, the opaqueness of the data in the database means that it needs to be indexed to display in some intelligent and interactive manner.
This can be looked at as not only an extension of the database task, but a requirement for it to be usable. But it doesn’t just stop at replacing the existing complete list, with a paged list. Why don’t we make it searchable if we have indexing anyway? Filterable based on dates, amounts and descriptions? Why not make it a real view and not a tab, and allow it to “pop out” and let the user have multiple different views open at the same time? What if we let the user define tags and tag payments? Filter a view then select specific rows, and group tag them? What if you exported your transactions from some other service like an exchange and it tagged based on presence in imported data? Which of these are worth doing?
The possibilities for an improved history just require time and effort, and the database work to be completed first of course.
We see a lot of people uploading files to the blockchain, primarily using unwriter’s protocols. But our requirements go beyond that. Once we have the database storage and other scalability fixes in place, perhaps even including transaction pruning to prevent the wallet from tracking everything which gets uploaded, we have a lot of potential for what we can do here.
When I first started supporting attaching files to transactions, it was intended to be an early stage, which we’d iterate on. However it quickly became clear that as I posted more and more files to the blockchain, the wallet bloated and became slower and slower. So in order to dissuade people from using ESV for file upload and bloating their wallets they used for other purposes, I stopped development on the in-wallet functionality. It does not support the unwriter style small uploads using the B protocol, and it does not support the Bcat protocol for splitting larger files. It is just a naive tool for attaching binary pushdata to a transaction. You could attach the right stuff to make it do both, but it’s going to be a clunky experience.
In order to push the wallet as a service feature forward and experiment with APIs for wallet usage, I added something akin to ESV-based applications. This allows you to bootstrap a wallet server with a custom JSON-RPC API, and to expose the functionality you need. This was ideally to support the new merchant web site I was going to write, but due to priorities that was side-lined. Instead to push improvement of the wallet fundamentals, I instead started work on a command-line based file upload application.
Beyond stressing the wallet fundamentals, it also serves as a prototype of how we might integrate file uploading, resuming of incomplete uploads and tracking of existing uploads and so on, in the future. The UI is also going to have to be designed to take into account a lot of interesting things, and allow the user to manage and track them in a scalable way.
Encrypted messaging / A notification center
You’re connected to the blockchain. Someone else is connected to the blockchain. You both have public keys through which you can contact each other (actually it’s more complicated than public keys, but it comes down to that eventually). So why can’t you send them a message and have the wallet synchronise it, and display it?
How would we integrate these into the UI? This and other things kind of push us in the direction of a notification center, kind of like Windows 10 has now and likely other operating systems also have. Or for that matter, even Slack shows indications of messages and allows streamlined access to them.
The indicator might be coloured to represent the urgency of the notifications and when clicks might pop up some kind of message area. We’d probably even want some kind of message history so that users could go back and find one they dismissed or look for something they half remember. And then there’s the ability to deal with messages that require action, or to filter out notifications of specific types in order to reduce noise and distractions.
This needs a design and to be integrated with the wallet, and may turn out to be quite well integrated with the full extent of available functionality.
Extended wallet restoration
When you restore a wallet from seed words, or private key, or whatever, the blockchain allows deterministic (reproducible) restoration of the payments made and received in that wallet according to the BIP32 and BIP44 specifications.
Something you may not have noticed is that when you restore a wallet where you’ve edited payment descriptions in ESV in the past, it also restores the descriptions. This is done through a service that the Bitcoin Core Electrum wallet provides, we and other Electrum-based wallets can use it also because in theory there’s no way for the service to discern what crypto-currency the encrypted descriptions belong to.
But this shouldn’t be done using some third party service which we hope is provided on an ongoing basis. What if the data is corrupted? What if they work out how to differentiate currencies and (rightfully) make a decision they just want to offer it for Bitcoin Core?
It should be persisted to the blockchain and restored deterministically, like the payments. And similarly, so should metadata about files, messages and really anything the wallet needs to track. Have you ever used a SPV wallet based on bitcoinj? What they used to do (and probably mostly still do) to restore a wallet, was process the whole blockchain. I think they would optimise it by storing the creation date with the private key, and this would mean they would only have to start processing from the block around that creation time, as no payments would predate the wallet.
This gets more complicated, and I’ll go into it further later below.
Our current multi-signature signing is quite arcane, and for automated involvement of participants also relies on a third party server providing communication between participating wallets, the “cosigner pool”.
This is another form of inter-wallet communication that will be a subset of the encrypted messaging, rather than using that server. And it needs the UI completely redone exploring the variety of different ways multi-signature transactions can be signed by the different parties.
Multi-signature signing is also going to be changing. P2SH (pay to script hash) transactions are being sunsetted, and will at some point no longer be allowed on Bitcoin SV. As multi-signatures are done using P2SH, we are going to be migrating to a new multi-signature regime likely using threshold signatures.
A mobile wallet
ESV cannot just be just a desktop wallet. When we’ve gotten things in shape we need to look at mobile versions. But until then we do not have the attention or time to support multiple platforms.
It might be that we support the full range of functionality on mobile wallets, or we treat the ESV wallet as the master wallet and delegate sub-keys and/or information for payments and other more advanced uses.
It might even be that a mobile ESV app can act as a hardware wallet replacement, doing external signing of some or all of the payments ESV can make. Could this be done by split keys?
A new contact / identity system
I doubt anyone uses the existing ESV contact system. It’s pretty much the ability to name an address. But that’s only scalable to the naive toy blockchain approach where everyone uses the same address over and over.
There’s another more extensive identity system also there (it’s since been removed in the latest code in our source control repository), called OpenAlias. OpenAlias was a DNS-based system that also mapped to addresses, where you’d enter “email@example.com” and it’d go away and ask that website for the address for the given user. Or something like that, I don’t even think anyone use it in Electrum, let alone ESV.
We see a future where you don’t reuse addresses, and there’s a new more flexible identity system in place. This is pending the release of specifications from the wallet workshop. This is also the reason we haven’t integrated HandCash handles, because the understanding is that all wallets will use the same basic identity system creating one glorious BSV future.
Supporting other protocols
We need our users to be able to manage all their assets in our wallet, so they don’t have to use ESV for what it provides and another wallet for what it doesn’t.
Payment requests as primarily represented by BIP70 were originally implemented using complicated tools like protocol buffers and signed with DNS certificates. BIP270 is JSON-based and will also be extended in interesting ways.
We already support BIP270, but it is not usable until a new signing mechanism is implemented. That’s going to be based on Bitcoin keys, not DNS certificates. And it might very well be the way most payments are done on Bitcoin SV. Someone hands you a payment request, you hand them back a payment with the merkle proofs for all your UTXOs and maybe also any required headers.
This will ideally be integrated with previously mentioned things like connecting the decoupled UI to do remote management of a headless ElectrumSV installation over an SSH tunnel, or via a web-based UI, and so more.
We need to write the replacement example merchant server for this, and if I end up doing it, it’ll likely be done in C# based on ASP.NET core. Python web development is a little too complicated for me.
Ideally users should be able to manage their Tokenized tokens in our wallet. This is going to require design on how to best fit these sorts of things into the wallet, and a lot of work to make it happen.
Becoming more than a stream of payments
We’ve talked about encrypted messaging, we’ve talked about file management, we’ve talked about a lot of things and where we see ESV going is becoming more than a stream of payments.
Think about your wallet as defined by your seed words as no longer mapping to payments, but more securing your data. You should be able to add folders under which you categorise different aspects of your life. Perhaps each of these is a indirectly derived or even completely independent private key.
In your employment folder you might have messages you have sent, and been sent, relating to your job. Perhaps there are multiple sub-folders including archived content from previous jobs. For the current jobs you may have been given keys, whether generated just for you by your employer, or even split keys with metadata about how they are used and how they have been used. Perhaps each job has a sub-folder tracking all known encrypted on-chain documents you have access to.
In your personal folder, more of the same. Folders tracking keys of your own, keys you share with others in some way, and also the metadata relating to them. Maybe some of them are access keys you’ve assigned to your mobile ESV wallet, or even your HandCash or Centbee wallet should there be a standard which they support, and they turn the mobile device through the wallet into a tool which can provide on the go access. Who knows!
We don’t know how this is going to go. But we do feel strongly that ESV is going to evolve into a desktop tool with a lot of different uses beyond payments, this also leads to a range of related topics which need to be considered. It can’t be described clearly because we won’t know exactly the best way to do it, until we get to the point we can start exploring the possibilities.
It’s not enough to store restoration data on chain, over and above BIP32/44 it needs to be done in a way where other wallets can use it to restore the ESV state. This requires new specifications.
It may not even matter if other wallets implement just the BIP32/44, those whole keys can be implemented as a sub-wallet of the ESV wallet. Want to make a payment? Choose a sub-wallet to use, as you would choose a bank account.
Maybe there is even some overlap between how we persist the in-use wallet to the encrypted database we’re going to be using soon, and how the wallet should be persisted on chain. If this can be reconciled, then we can write blobs of restoration state to the blockchain as we write them to the in-use wallet storage. The only difference might be that we strip the transaction data from the version written to the blockchain, and keep a lot more indexing data in the live wallet that any restoring wallet can regenerate.
The wallet would in theory be managing a large part of the user’s information and finances. Ideally it should keep track of fiat prices at the time of transactions and allow the user to keep tax records, even cross-reference expenses from the employment folders where applicable. As the wallet gets more comprehensive, it will need to work to keep things manageable for the user.
We’re really excited about where ElectrumSV is headed. There’s a lot of work involved, but we’re starting to make some serious headway on the behind the scenes bits. The wallet workshop was an invaluable process, with useful solutions we can’t wait to use when they are finalised. There’s a huge amount of potential in what we can do with the wallet as we get it in shape.
We’re also really excited about where Bitcoin SV is headed and the opportunities it provides both for developers and users. If you’re interested in helping out with developing the world’s best open source desktop wallet that is currently written in Python using PyQt5, let us know! It’d be a great way to get some blockchain related experience, which might just be a way to get a foothold in the industry. It’s a good place to target as the future is not only blockchain, but it’s the only real Bitcoin.