16th October 2021
We are seeing increasingly larger blocks on Bitcoin SV. Users still need to use the legacy technologies we inherited from Bitcoin Core, and without them they will be unable to restore their funds or hear about incoming transactions. The current way this works is by the kindness of strangers, who run public free versions of the ElectrumX indexer for anyone to use. ElectrumX is going to stop working likely within a year. This document is about replacing what ElectrumX allows ElectrumSV to do so it can continue working into the future.
A large part of the future communication and transaction exchange will happen by SPV. This will use Paymail, Peer channels, MAPI and similar technologies. This document is not about that approach, it is the other part you hear less about.
ElectrumSV is also planning on trying to make wallet backups as painless as possible. This document is not about that, although it allows techniques to help with it and even do worst case scenario recovery.
The ElectrumX indexer
ElectrumX was designed for small blocks of maybe around 4 megabytes, not terabyte blocks. At some point it will be unable to process blocks because they are too large, or it may not even be able to keep up and will fall behind more and more rendering it useless. This indexer cannot technically cope with the larger blocks. And beyond that as block sizes increase, so do the storage costs for those who run servers.
Until recently, up to four different wallet applications have relied on these indexers. Some have already indicated they are building their own infrastructure to replace it in a way that will work with larger blocks, perhaps others are oblivious or they have not revealed their plans.
Remember that the indexer software was written by a community member Neil Booth using their professional knowledge, and made available at no cost to the kind people who run the publicly accessible indexers. These kind people make it available to users of not only ElectrumSV, but a small number of other wallets and applications, also at no cost to those users. No-one has the time or interest in writing and giving away new free indexer software that works with terabyte blocks — for good reason. It is expected all replacement services will be commercially operated in order to pay for the used resources.
The primary goal of the ElectrumSV project is to allow users to restore their older wallets and locate both their unspent coins, and their transaction history. Most users of the restoration functionality will be providing seed words, and restoring their wallet using those. There will however be users who have anything from private keys, master public keys through to addresses that wish to locate transactions.
When the ElectrumX servers become useless for indexing the blockchain, either because they break or because they fall behind to the point they no longer update, wallet restoration will become fixed at a certain height if the server operators even keep them running. At best restoration will no longer work past a certain height and at worst it won’t work at all.
Why transaction history?
It is a lot easier to just restore the UTXOs that a wallet contains, but ElectrumSV only intends supporting full restoration with history. Our highest priority is to ensure that users can obtain a fully restored wallet, and supporting less useful subsets incurs additional work and support requirements. It makes a lot of sense for us to just stick to full restoration with history.
Users will have a strong interest in proving that the coins they have, they obtained legitimately. This means that the ability to correlate receipt of initial coins from a third party or an exchange, and the payment made for those coins, will be invaluable. The unspent coins they still have in their wallet, may be many transactions distant from the originally received funds. This is why the restoration of full wallet history is important.
There are two different classes of users who will be affected by the legacy seed-based restoration ending. There are the users that have yet to bring their existing Bitcoin Cash or Bitcoin Core coins to Bitcoin SV. And then there are the users that were told that seed-based restoration would always work, who have obtained or spent Bitcoin SV and who within reason we should help access their coins.
It seems like it comes down to partial indexing to allow immediate restoration and on-demand scanning to allow delayed restoration.
A capped height indexer will have the data necessary to find any transaction that was broadcast up to and including that capped height. Any of a user’s transactions that are before or at this height, and associated with their account, should be able to be immediately located and obtained — or restored.
The simplest way to do this would be to cap the height to the point at which Bitcoin Cash forked away from Bitcoin SV. This would be good enough for the users of Bitcoin Cash or Bitcoin Core, who have not brought their coins to Bitcoin SV. However, a lot of the users who have brought their coins to Bitcoin SV still expect that wallet restoration, especially using seed words, will find their Bitcoin SV coin usage. We need to provide them with alternatives, and there will come a date when we release a new version of ElectrumSV and declare a block height beyond which attempts at wallet restoration should be expected to be incomplete.
This API is already based on push data hashes, where the user submits all the push data hashes they want to know the restoration data for, and it streams back matches to the user. This API will likely be very cheap to use. If price did vary for it’s usage, the later larger blocks before the cap would likely be more expensive than the earlier smaller blocks.
- It is not intended that this restoration index be a complete pushdata index. It would instead only index receipt and spending of standard type scripts. P2PK, P2PKH, P2MS and P2SH pre-Genesis. Unlike tip scanning which is mentioned later, this would also handle the 65 byte pushdata used for uncompressed public keys.
It is expensive to keep an index of the full blockchain, and we do not expect any service to do so. If it turns out that some service does, they can expose it to services and ElectrumSV as immediate restoration. What is more likely however, is that if a user needs to know if something exists in the blockchain above the capped restoration height, they will pay for a full blockchain scan. This might be very expensive depending on the distance from the tip, and it might be deferred until multiple users have submitted requests to be completed at the same time. How long a user has to wait might depend on how much they are willing to pay to have their scan performed.
It is intended that this API will be based on user-provided cuckoo filters, where the user also provides a peer channel for results to be sent. It is similar to the tip scanning API which is described below for transaction state.
Cuckoo filters reduce the number of checks to see if the user’s pushdata hashes are present in the blockchain, and the amount of space required for the user to provide that pushdata hash data to the scanning service. Cuckoo filters also give privacy. Passing in a list of pushdata hashes reveals everything the user is looking for. A cuckoo filter just picks out possible items of interest when they are encountered.
It is inevitable that payments on Bitcoin SV will primarily move to Paymail and other similar solutions. However, Paymail is not yet ready to replace legacy payments and beyond that support for legacy payments is still required for a number of reasons. It is important to recognise that we are dealing with transactions that will not always be for the primary purpose of payment, so speaking in terms of payments alone instead of transactions is not correct.
If there is no full index of the blockchain due to the size of the blocks, which the ElectrumX indexers are currently able to provide, then how can ElectrumSV find out whether there are any new incoming transactions related to a user’s accounts? Or whether a recipient has broadcast transactions they were given?
Not all transactions a wallet needs to know about are direct immediate payments between two parties and it is unclear what the range of situations are in which a wallet needs to know whether one of their transactions have been broadcast in. This will not just be a legacy requirement.
It is envisioned the cheapest way to keep track of transactions that appear in the blockchain and are related to your accounts is to have registered filters with a tip scanning service. This would then post these to a peer channel you have created for the purpose. When your wallet is next loaded, it can effectively synchronise your account state (which is the same process as a restoration) by reading from the peer channel.
It is intended that this API will be based on user-provided cuckoo filters, where the user also provides a peer channel for results to be sent. Their filter along with the filters of other users will all be run over each incoming block. As it is expected that focusing most of the filtering on the same latest block will reduce costs, this API should be priced reasonably.
- The user provides parameters with their filter, which might alter the default types of pushdata that their filter is checked against. Filters by default might only process 20 bytes (public key hash), 33 bytes (compressed public key with uncompressed being excluded intentionally) and 32 byte pushdata (SHA 256 hash) but a user could opt to pay more and process other sizes. They might just process 32 byte pushdata by default, and none of the other sizes.
It might make sense to index popular protocols. This might be the Tokenized and other token protocols. It might be the DID decentralised identity protocol. It could be anything where there is demand. These might be indexed for immediate access, like how it works for immediate restoration. These might be kept as separate indexes each accessed by those who need it. For protocols that require proof of current state because of the “back to genesis” problem, the availability of these indexers might be invaluable.
These are more of a longer term option, rather than being in the initial service offering, and require some design as to how they should work and be accessed.
The above approaches should cover both existing and what we perceive at this time to be the future needs of ElectrumSV. An index of earlier blocks will allow users to restore full account history for existing transactions up to a fixed height. Beyond that the user will be required to pay for a cuckoo filter to observe the blockchain and select transactions to be posted to their peer channel. If they do not install a cuckoo filter, then they will have to pay to scan older blocks.
An Open API
What seems like a long time ago, the ElectrumSV developers Roger Taylor and AustEcon published a request for indexing services for us to use. We came up with an example API that the services might use, which we thought we needed. It wasn’t quite like what we describe above, but the reason we published it was because ElectrumSV is an open source project and it benefits from the community providing services it can use.
Unfortunately, none of the services or developers who were planning on developing an indexer managed to do so. There were other applications and developers who were interested in using one, should anyone develop one. I expect they have also come up with their own solutions. Instead we had to invest effort outside of our work on ElectrumSV on a custom indexer that suits our needs. This indexer currently has a working restoration API and the systems are being refined, and the tip and archive scanners are still coming.
Anyone could run an ElectrumX server. They could run it privately for their own usage, or publicly for others to use. Moving beyond ElectrumX, it is intended that anyone who comes up with an indexer that provides APIs that are compatible with ours, should be usable in ElectrumSV. ElectrumSV should be considered completely independent from our services.
If you are working on an indexer or a scanner and wish to work with us on a common API and be considered as an option for ElectrumSV users, then please get in touch.
Payment for usage
It is envisioned that all of these APIs will be monetised. It will become very expensive to run an indexer, scanner and retain an index of various aspects of the blockchain. General API usage might be done through payment channels, however if a user is restoring their existing coins from seed words, it might even be that they pay through Paypal or through some non-Bitcoin method.
You can discuss this using the following forums:
We do not discuss on Twitter. It is a painful way to communicate. Most articles like this are advanced published in the #electrumsv channel on the Metanet.ICU slack.