This document aims to record the plan for moving the focus of ElectrumSV away from monitoring the blockchain to a P2P focused approach. Due to the complicated nature of this, and the fact that it cannot be done wholesale, it is even more necessary to ensure this design document records both the reasoning and the nuances of the planned approach taken for later reference.
It is a copy of an original document shared on Google Drive, and is not guaranteed to be updated to match the original.
The way SPV was interpreted for the original Electrum wallet was that a user would broadcast a transaction to the blockchain, get notified it was present in the nodes, get notified it was mined in a block and then fetch a merkle proof and verify it against the given block. ElectrumSV still has to do this to interoperate with incoming payments still made in this way, but it will stop doing it in all other cases where possible.
This document should outline the complete model that ElectrumSV will use going forward. Monitoring the blockchain is still the only API we have to find out about the appearance of transactions paying a user on the blockchain, and whether a transaction has been mined. But we will only be monitoring it for specific reasons, rather than in all cases.
Script hash events
The API ElectrumSV has at its disposal is the ability to register script hashes with the ElectrumX server it is connected to, it then gets the current state and after that receives any notifications relating to those script hashes being used. It gets told about what transactions use those script hashes, and what height in the blockchain they are. This indicates their presence in the mempool, a block or their movement from the mempool into a block as they get mined.
Detecting new blockchain payments
Examining the script hash history obtained from the server, ElectrumSV can reconcile which transactions it already has and which it does not. Those that it does not have, are new payments and it can then obtain and import them.
Merkle proofs and transaction verification
A transaction in the script hash history that has a height indicates it has been mined, and is not just in a mempool. ElectrumSV can then request the merkle proof and verify it against the block header.
Dispensing payment destinations
A user can give out payment destinations from their account and be sure that if a payment is made to it their wallet will pick it up from the blockchain. They might do this by going to the Receiving tab and copying the destination (most often in the form of an address) shown there. They might go to the keys tab and copy a range of destinations available there. Or they might go to their account menu, and generate any number on request via the menu option provided for that purpose.
The receive tab features the ability to save an expected payment with optional description, amount or expiry date. These are internally known as payment requests. This is mostly for the user’s records, although we have hooked it up to the processing of incoming payments and expiration, either possibly marking an entry as paid automatically. But for the most part, it is mostly useless and I expect it is also mostly unused.
Scanning the blockchain
All accounts with deterministic key sequences have a gap limit for both change and receiving payment destinations. This is a maintained space of unused keys at the end of the current sequence of enumerated payment destinations. As all payment destinations are monitored on the blockchain, transactions are discovered, imported and keys become used. In turn the gap of unused addresses shifts further out in the sequence of enumerated payment destinations — with more being enumerated/created as the need arises.
With the gap address based approach, there is no need for the user to manually scan the blockchain for usage of their keys and the payment destinations that feature them. It happens naturally and the account is bound to the central mechanic of monitoring the blockchain.
P2P focused approach
ElectrumSV still needs to use an indexer for a P2P focused approach, and the only available indexer is ElectrumX which indexes script hashes. Every key that is active has a number of different script types that it can be used in, and each of these produces an output script which is also known as a payment destination.
Addresses and blockchain payments
What about addresses? There were only ever two types of addresses, P2PKH (historically for single recipient payments) and P2SH (historically for multi-signature payments). The classic P2SH output script is now rejected by nodes, so that leaves P2PKH. But there are many more types of output script than those two, including P2PK and bare multi-signature. In fact, there is no reason why your application or wallet cannot dynamically construct custom output scripts on a case by case basis.
ElectrumSV still allows a user to copy and paste an address, to give to a second party, for them to pay by the blockchain. We will still continue to allow that and support it, even if we discourage it and sideline it in favour of putting P2P front and center. But remember, only P2PKH and P2SH were addresses, and other forms of output script have no way of being used by wallets and applications for payment. For this reason, we came up with BIP27⁶¹ which allows the encoding of payment destinations as text. But it is not supported by anything else, and will likely only find use between ElectrumSV wallets until there are hosting servers available to provide something better.
So who might copy and paste a payment destination (whether it is a P2PKH address or BIP276 encoded output script) to give to another person? It can only be used for blockchain-based payment, after all.
- Receiving payments from exchanges and other businesses rather than via other secure P2P mechanisms which do not exist yet.
- All ElectrumSV users until hosting services are available to provide either Paymail hosting, or further evolved replacements for it.
Detecting new blockchain payments
As part of normal wallet management, the only time we will ever pay attention to unknown transaction ids in the script hash history is if the script is a payment destination the user has created a payment request for in the Receiving tab, and that payment request is still unpaid. As transactions are processed and the payment criteria are met, namely that any designated amount is paid or that any designated expiry date has passed, any remaining transactions are ignored.
We still leave some room for users who want to openly monitor key usage through several avenues.
- A payment request can be created without an amount. All discovered transactions paying to the given payment destination will be processed regardless of how many.
- A payment request can be created without an expiration date. The payment destination will be monitored until the specified amount is paid, or indefinitely if there is no specified amount.
- The user can go to the Keys tab and set a key as “user active”. This will process all discovered transactions making payments to payment destinations that use the given key.
Our initial implementation will only monitor the account’s designated script type for a given active key, but when we get time we will allow the user to opt to monitor any of the possible script types. If we ever get pushdata indexers available to use, we will match on the public key, public key hash or other data that the output script is built around and have more options available to us.
Merkle proofs and transaction verification
For every transaction that we detect, obtain and import into the wallet, it is known that it is related to at least one of the accounts in the wallet. The point of SPV is that the user obtains the merkle proof of any transaction that they plan to spend coins from, so that they can provide those transactions and their merkle proofs with any child transaction that spends them. The recipient can then have some proof given they can verify the spent coins against the blockchain headers, that the spent coins are legitimate.
The only avenue we currently have available for discovering if a transaction has been mined, is the script hash history. The appearance of a transaction there with a height above zero, indicates that the transaction is in the block with that height. So we must continue to subscribe to the script hash for the given script types using that key, in those transactions where we want these notifications. This can be a fairly heavyweight way to receive these notifications if keys are reused, and for heavily reused keys the servers have refused to serve the histories in the past. For those users, the only recourse is to run their own indexer. But key reuse is something we support as best we can for legacy reasons, but do not consider it our responsibility how fast or reliable the experience is for those who choose to do it.
MAPI, SPV channels and callbacks
Merchant API 1.2.⁰² provides the ability to both submit a transaction to be mined, and to get callbacks either when it is double spent or mined. However these callbacks are through SPV channels and at this time, ElectrumSV does not have businesses that are providing SPV channel hosting for it’s users. Even the older form of direct URL-based callbacks requires a hosting service acting on the user’s behalf. The only way to obtain a merkle proof is through this callback. While there is also a polling mechanism to see the status of a transaction, polling is not desirable and there is no path to acquiring the merkle proof. And beyond that, polling only indicates if that transaction is mined or double spent, and that does not cover all cases.
ElectrumX has and still provides an API that allows getting the merkle proof³ for arbitrary transactions, which is another reason why ElectrumSV has to continue using it for now. However, it would be better if it also provided an API to register for a notification when a transaction is mined, in order to allow ElectrumSV to use the script hash history notifications solely for learning about new transactions of interest.
Tracking relevant state
It is not enough to say, “Here is my known transaction hash, is it in a block?” We must also know if it has been double spent. We must also know if it has been malleated, in which case the hash will no longer be the same. The simplest initial approach would have been to use an API that wraps the getrawtransaction RPC method⁴ of a node, but this will only tell you whether it is in the mempool or contains an unspent output (which is lined up to be deprecated) unless your node is unpruned.
Until there are better APIs provided by services we need to use script hash history notifications to learn both about new transactions, when a transaction is mined and even both when a user forces a key active. This then means that we need to work out the algorithm to use to ensure we do register for events when needed, and do not unregister prematurely.
When we want to register for script hash notifications for a given key:
- When there is a payment request that uses that key that is still unpaid.
- When the user has forcibly set that key as active.
- When we have a transaction that uses that key and want to know when it is mined, and the merkle proof is available. This may be just after we receive the transaction, after we learn of it on the blockchain or just after a reorg that affects the given transaction.
Separation of concerns
Attempting to treat monitored key usage and monitored transaction state as a combined system over complicates things. By separating both systems, and abstracting the subscription mechanism so that both can interact with it independently, we can construct a generic mechanism that allows all the systems to be simpler and more manageable.
The subscription system can manage multiple concurrent subscriptions for different systems for the same resource, and dispatch notifications to any who have expressed interest. It can unsubscribe when the last interested client system has removed its expression of interest. And subscribe when the first expression is received.
Satisfying transaction state needs
If we have the ideal API, it will allow us to get notifications about whether outpoints have been spent. ElectrumX does not provide this functionality, and will not, although interestingly Bitcoin Core’s fork has a pending pull request⁵ for it. If we were to subscribe to notifications for every outpoint that is in transactions we are interested in, whether in off-chain transactions or otherwise, it is possible we could get the full picture. We should be able to identify double spends if outpoints are spent in conflicting transactions, we should be able to identify malleation if outpoints are all spent in a matching transaction with a different hash, we should be able to identify valid spends by all being spent in the same transaction with the expected hash.
Returning to the API ElectrumSV has available to it, script hash notifications, we have two options.
- We could monitor the script hash of the first output of our transaction, and this would detect expected spending, malleation but not necessarily double spending.
- In order to monitor a script hash to know whether our transaction’s spent outputs are spent in the mempool or mined, we would need to fetch all the parent transactions and identify the script hashes of the outputs that are spent from them in our transaction.
Observing the first output
This is the simplest approach to detect if a transaction has been mined or malleated. In nearly all types of SIGHASH the outputs in the transaction are signed into place. The exception to this is SIGHASH_NONE⁶ which we can ignore for now, by the time we add support for it it is possible that there will be APIs more suited to our needs.
This approach cannot detect double spending, but that is not that important anyway. The custody and context of the original transaction can serve as proof that the payer defrauded the wallet owner. Later APIs can also address this.
Obtaining the parent transactions
At first glance, this approach sounds cumbersome. But our goal is to move to a P2P-based model that is based on SPV. This should ensure we have all the parent transactions and merkle proofs of the outpoints that are not our own for the transactions we receive. In the case where we cannot obtain parent transactions, we can of course fall back on observing the first output of our transaction.
An exception to this would be detecting incoming transactions via the blockchain, and the script hash history provides block height for all key usage in all related transactions which negates the need for all parent transactions historically. It is the yet to be mined transactions that we would need to obtain them for. There is however, no guarantee that the spent outpoints come from recent transactions so it is possible that services may have pruned them. As mentioned above, while the getrawtransaction node API method currently allows access to transactions with spent outpoints, this is marked as deprecated — something that should be a large red flag to anyone hoping to use the blockchain UTXO as storage.
It is necessary to do a reality check at this point, there is no real SPV happening. There are no bundled parent transactions and merkle proofs (perhaps together referred to as SPV proofs) with the transactions ElectrumSV receives. The only P2P-esque way to exchange transactions is via the Paymail P2P endpoint, and it does not provide SPV proofs. This will be touched on in a related “Transaction exchange” design document.
The conclusion of this thought experiment is that while we may not need the parent transactions because we should have them, we do not have them because SPV is something the ecosystem is still working towards. It might be that the API to monitor spent outputs arrives before the infrastructure for SPV proofs to accompany transactions. Even in the short term, if we were to fetch all parent transactions from our unpruned ElectrumX nodes we might have to decide whether to bother depending on how many spent outpoints there are in a given transaction. It also adds an unknown amount of bandwidth requirements for an online wallet, as there is no way to determine the size of a transaction without first fetching it.
- The BIP276 standard for encoding payment destinations in a form users can copy and paste.
- The Merchant API v1.2.0 specification.
- The ElectrumX get_merkle RPC method.
- The Bitcoin SV node getrawtransaction RPC method.
- The Bitcoin Core ElectrumX spent output notification pending pull request.
- The Bitcoin SV wiki SIGHASH page reference for SIGHASH_NONE.
This document was modified with malleation taken into account and not naively looking for transaction state by whatever hash we may have for it, based on advice from Kyuupichan developer of ElectrumX.