Extending the ElectrumSV SDK
--
This article considers some additions to the ElectrumSV SDK, including an SPV channel server, a blockchain scanner and also frozen blockchains.
These changes are necessary for ElectrumSV, but they also round out the functionality available in the SDK for any developers who use it. They are not the only changes coming for the SDK.. but we will cover those at some later point in time.
What is the SDK?
The ElectrumSV SDK is not an SDK for developing ElectrumSV, it is an SDK for developing Bitcoin SV applications locally without having to use the mainnet, testnet or scaling testnet blockchains. Each of those blockchains has a use, but the power as a developer is in running your own local version of Bitcoin SV you can reset at will and mine blocks on whenever you want is unbeatable.
All the supported components are prebuilt for Windows, MacOS and Linux. Start a node, start a MAPI server and start ElectrumSV (if you want). Then you have a more complete development environment that you control locally, and can even use when your computer is offline.
FYI 99.9% of the programming work on the SDK was done by AustEcon [SV].
Existing extensions
One part of the value in the ElectrumSV SDK is that it can provide the ability to run all the public services that a developer will use when their application goes live, in addition to a node.
The currently supported services are:
- A MAPI server.
- An instance of the WhatsOnChain blockchain indexer.
- The ElectrumX blockchain indexer.
- The ElectrumSV wallet.
MAPI server
The future model of transaction broadcasting is that you post your transactions to a MAPI service, and it has the ability to notify you when your transactions are mined and provide you with a merkle proof. It also has the ability to notify you if your coins are double spent. This should replace broadcasting a transaction yourself to the blockchain through the use of a node you have access to.
WhatsOnChain blockchain explorer
If you are going to run a local blockchain, it is very useful to have a blockchain explorer you can run against it. Thankfully the WhatsOnChain team open sourced their explorer, and it is possible to run this in the SDK.
ElectrumX indexing server (will be removed)
The main reason for the ElectrumX indexer was to provide Electrum wallets a way to do seed-based restoration, and to detect incoming payments, both of which use the same approach. It also sits in front of a node, and provides wallets with a way of broadcasting transactions before MAPI came about.
However, other applications and wallets than ElectrumSV have found the APIs it provides useful, and have made use of the data these servers provide. In most cases, SDK users will not need to run this unless they are also intending to run ElectrumSV.
These are currently run on mainnet (the real Bitcoin SV blockchain) by people who make them available for free use, but as the size of blocks increase they will start disappearing. This software was written for 1 MB blocks, not 2 GB blocks, and they will stop working in the next 6–12 months.
ElectrumSV
It is not necessary to run ElectrumSV if you use this SDK, but it is useful to have a wallet with a GUI you can run with the SDK.
New extensions
Again, one part of the value in the ElectrumSV SDK is that it can provide the ability to run all the public services that a developer will use when their application goes live, in addition to a node. Here we are going to consider the next pieces to add that will allow application developers to move towards the SPV model and more.
SPV channel server
The availability of this service is an essential building block for any Bitcoin applications to develop on. It provides a way for services or the applications users use to connect to each other. A user’s application or service can negotiate with another user’s application or service on their behalf, and the user merely interacts with a user friendly interface.
The reference implementation of the SPV channel server is currently difficult to include in the SDK. It requires a Postgres database server to be running, which in turn requires us to bundle a version of Postgres in the SDK or abandon the goal of being as close to zero configuration for users as possible. We may include it or we may create a simple, minimal and disposable server for use in the short term, and integrate that.
This does not matter, the key takeaway is that all applications need is a “post to channel” URL, in much the same way as a MAPI service makes use of. The larger API that ElectrumSV can automate the usage of to manage channels on behalf of the user, can if we need be ad-hoc for now with minimised technical debt. There is no finalised API at this time to adopt, so any applications that need to manage channels and channel messages are in the same boat as ElectrumSV for now.
If we do go the ad-hoc approach it is expected that we will replace this component with something standardised as an option becomes available, although we may have to modify that to fit our zero configuration requirements.
Blockchain indexer/scanner
It has already been mentioned that ElectrumX is going to die, and we cannot rely on free public blockchain indexers. This is not intended to be a replacement indexer, there are however reasons we are adding it. The indexer we add will be very basic and slow and will not be usable outside of the local testing/development blockchain (also known as “regtest” blockchain) you are running in the SDK.
Wallet restoration
Before Bitcoin SV removed all the limitations that the Bitcoin Core developers put in place, there was an idea sold to users about seed-based restoration. It was claimed that with just your seed words (also known as “mnemonic” and “seed phrase”) you can recover your entire wallet. However, this only worked because very few people can use Bitcoin Core. The block size is small, and the fees are high. This means that for the oddly hobbled Bitcoin Core, the blockchain is never going to be used much, it is never going to grow very large, and so the free indexers don’t get worked very hard.
Now on Bitcoin SV we are seeing gigabyte blocks and the free indexers are going to die. However, ElectrumSV is funded by the Bitcoin Association and the single most important goal the project has is that users who have Bitcoin SV can access their coins. Some of these people will be late arriving Bitcoin Core users, some Bitcoin Cash users, and some even Bitcoin SV users who had believed the fantasy seed-based restoration would always work. But if the indexers are failing as the blockchain size grows and this is how the restoration worked, how do we do this?
Only restore before a given blockchain height
The only way is to declare a height at which restoration stops being supported. This should give every one of those users who relies on it the guarantee that they can find both their transaction history and their unspent coins. It is expected that there will be paid services where the user will have to pay a nominal fee to recover that history and the unspent coins.
Recovering transaction history in addition to unspent coins
If you have the keys to coins, it does not prove your ownership of the coins. You need to be able to show where you obtained the coins. This is why you should never never use a blockchain mixer, all it does is mix your clean coins with someone else’s tainted coins and lose you that proof of ownership. By recovering a transaction history, a user will be able to map their unspent coins back to where they purchased them from an exchange, business or other user and prove provenance.
Restoration after the capped height
There are going to be users who are not paying attention and need to recover their coins after the capped height. They will either have to find some business that runs a full indexer or engage a blockchain scanner that will comb through the blocks above the capped height looking for their history.
Indexing support
This component will index “standard” script (P2SH, P2PKH, P2MS) push data in scripts and collect enough information for wallet restoration. It will not have to worry about blockchain size, and if the local blockchain is overloaded by the SDK user, it will just slowly work it’s way through the blocks. There will be an optional capped height at which it stops working, to emulate the restoration limits that will exist in similar mainnet services.
Scanning support
This component will take cuckoo filters and apply them to some subset of the push data in new blocks, any matches will be posted to an SPV channel. No records will be maintained. There may also be a older block scanning mode, which paid services will charge more for than the new block scanning mode.
APIs
There will be a push data-based API for access to restoration data. There will be a cuckoo filter-based API for notifications about new block inclusions or scanning older blocks. We will define our own API for these things. The general shape of the APIs should be pretty intuitive, but there may be nuances that we need to decide on. These APIs will likely change, including a monetisation or payment channel protocol.
Frozen blockchains
If you want to test against existing blockchain data, you can create all the transactions you need in the different blocks you need them in, spending coins in the ways you want. Another way is to do that, then save all the blocks to disk. Then the next time you reset your local blockchain, you can just replay all the saved blocks and get that transaction history back. You will already have seed words that restore (before any capped height), coins that you have spent using different transaction types and so on ready to go.
It should be very possible to write an SDK command that saves and restores frozen blockchain state. This could if the user wanted, pretty much just bring up an ElectrumSV wallet at that base recorded blockchain state ready to work from (maybe in CI, maybe some local automation, maybe manual testing). It doesn’t have to be ElectrumSV, it could be their own local custom components that only they run against the SDK.
There’s a lot of potential nuances to what we could do with this down the track, but that should all be deferred. Things like stepping through the blocks with manual user prompting for each one, or fixed delays before each. This might be useful for observing the effect of reorgs for example. Who knows, the main benefit of this is restoring the SDK state to a fresh known transaction history.
Final thoughts
The SDK needs to support the future functionality we will use on Bitcoin SV so that ElectrumSV (or any application developer) can develop for it. If it does not do this, it becomes less and less useful instead of staying the extremely valuable developer productivity tool that it currently is.
Currently the SDK is installed using a local version of Python, but we will be moving away from using this mechanism and towards a complete download which a user can run in place. This will make it more accessible to developers than it currently is. There are also a range of other improvements in the works for the SDK, including a single-click to start desktop GUI that we have already mentioned before. This is currently AustEcon [SV]’s current focus and work is proceeding well. Maybe even a name change so it is not tied to ElectrumSV.
You can comment on this article on Medium, but the best place to discuss the SDK with us, is in our Github discussions forum.
