Refactoring project #2
This document is intended to define the necessary work required to do our next refactoring pass, where we address problems with our wallet state management to help it better suit our needs into the future.
The previous refactoring project was to move to database storage of wallets and also to prepare for different forms of multi-signature transactions other than P2SH. It was not intended to fix all the things that need to be fixed in ElectrumSV. This means that in order to limit the scope it did not do all the desired work to decouple from the older model of the blockchain seen through a local node as the definitive source of wallet state.
This is the scope of this project, completely decoupling from the flawed network broadcast model of reconciling wallet state. It needs to move away from that fully, to a model that facilitates a P2P first approach when reconciling that state.
As usual, I will add a disclaimer. This is part of an ElectrumSV design document that I am sharing in order to try and solicit feedback and engage discussion. I am not a Bitcoin expert, nor do I play one in a movie.
And only download ElectrumSV from electrumsv.io.
Exploring transaction exchange
Let’s start with how ElectrumSV obtains transactions, or should obtain transactions.
Limited blockchain usage detection
With a model where there are paths to directly exchange transactions, whether through Paymail or P2P between the sender and recipient, the wallet may still have some situations where they want to monitor for usage on the blockchain. It should not be registering to be notified about usage of any key that it may have used in the past or could use in the future, but rather for exceptional cases when necessary. This might be automated by the wallet, or on the specific direction of the user.
In the short term this should be able to emulate automatic blockchain usage detection, so that it can be prepared and used in the short term, and allow easy phasing out of the full model of automatic blockchain usage detection.
Emulation of automatic blockchain detection
This is necessary in the short term where many transactions are still delivered over the blockchain, and received through the use of indexers that monitor the blockchain. As P2P forms of transaction delivery become more common, it will be possible to disable this.
Explicit key reuse
We are slowly building out the infrastructure that the Bitcoin SV society can build on, and at this time are still transitioning from the legacy model. As more services and protocols are developed that allow direct exchange of transactions, there should be fewer and fewer reasons a user needs to force reuse of a given key. An exception to this might be users who have historical addresses they have given out publicly and no easy path to migrating users away from these.
Watch only wallets
The whole model of wallet state being based on monitoring the blockchain for key usage, enables watch only wallets to be created. As derivation of keys moves to using methods that are not in linear sequences, like an Xpub provides, the ability to create a watch only wallet disappears. It is likely that the cases where this provides a benefit will reduce over time.
Automatic blockchain usage detection
This is the legacy model where it was believed that the blockchain was the method through which a user receives and sends transactions related to their wallet. It is documented here to give a more complete picture, and to help flesh out the full picture for the move away from this model.
Definitive source of state
This is where a wallet matches all the possible keys that belong to it against usage of those keys on the blockchain, and populates the wallet with the transactions that use those keys. This provides them with a picture of both the previously spent and available to be spent coins in the wallet.
In fact an odd artifact of this is observed on Bitcoin Cash where different ElectrumX indexing servers each have their own blockchain node, and each may also have differing settings depending on the domain knowledge or intention of the server operator. This means that a low fee transaction may be signed and broadcast to a server that accepts it, then the wallet software switches to another server with a higher fee setting. The transactions then disappear from the wallet. The only way to get them back is to switch back to the original server. Whether this is desirable, or even a friendly experience for users is questionable.
Adherence to the longest chain
ElectrumSV connects to up to 10 servers and designates the one with the longest chain as it’s main server. The main server is the one that it uses as the definitive source of state, and it employs the historically incorrect form of SPV, where servers provide merkle proofs along with transactions, to link them to the block they were mined in.
Each of the connected servers provides blockchain headers to the wallet software, as the nodes behind them have new blocks propagated to them. The wallet software calculates the work for each blockchain that it is made aware of through the servers, and as it identifies one that is longer than the one it currently follows it designates that server with the new longest chain as the new main server and reconciles it’s wallet state against it.
Watch only wallets
With blockchain observation employed to detect wallet usage, it works for both the holder of the master private key and holders of the master public key alone. This means that observation only, or “watch only” wallets, can accurately track all related wallet usage. And it allows a user to create a wallet that can detect unexpected usage of their coins without the risk of it being able to be compromised.
The heart of this model is that when the wallet software connects to or switches to a main server, it reconciles its state against what that server claims is the current state. In some cases this will be adoption of the longer chain used by that server, extending on from the chain it already uses. In other cases this will result in addition or removal of transactions depending on the mempool contents of the different node behind the server. And in other cases this will mean the state will be reverted and replaced to some degree to adhere to a fork and reorganisation.
Reorganisation now becomes an artifact of the larger naïve model of the server being the definitive source of state.
Direct transaction delivery
In an ideal future world one user’s wallet software should be able to connect to another user’s wallet software and to do so in a private and secure manner. It is envisioned that the IPv6 should allow this, but unfortunately IPv6 is not reliably available in any sense where it can be adopted at this current time. In this current day we are forced to deal with the inability to reliably connect using existing technologies, due to routers, firewalls and other things.
ElectrumSV depends on the common acceptance of new protocols and solutions that allow this to work, and like every other developer in society already has an overwhelming workload of things that can be done now. This means that direct transaction delivery is currently deferred. It should be a variation of other forms of P2P delivery in any case.
Paymail transaction delivery
The current Paymail specification was a starting point and is based on trust in the server hosting an identity. The hosting server provides payment destinations for software to use when constructing transactions directed at a hosted identity, and an endpoint for delivery of these transactions to that hosted identity. This is not P2P transaction delivery, but both payment and delivery through the designated trusted service.
All the mainstream wallets that support Paymail are hosted wallets that implicitly require their users also have their payments routed through their hosting services.
- Money Button.
- Simply Cash.
This is for a very good reason, but ElectrumSV is the only real non-custodial wallet and any user should be able to opt to use the Paymail hosting service of their choice. There is no designated single hosting service run by ElectrumSV as a business, and while the wallet may direct users as to how to choose trustworthy hosted services, it will not require usage of it’s own service.
Custodial transaction delivery
Use of the above mentioned mainstream wallets requires use of a designated hosted service, again for good reason. Delivery of a transaction to a user of one of those wallets, is inherently it’s delivery to their wallet software’s designated hosted service. This can be considered P2P delivery of a transaction in a custodial sense.
P2P transaction delivery
As the sole wallet without a designated hosting service ElectrumSV is the only wallet that will allow its users to choose to use arbitrary hosting services for any of the user’s identities. It will connect to the designated services for any wallet identities to receive incoming transactions for those identities. And connect to the hosting service for an identity that is to be paid, to deliver outgoing transactions for those identities.
Based on the listed methods of transaction exchange, the requirements for refactoring are as follows:
- Tracking wallet state should be designed around the longer term approach of P2P transaction exchange.
- Detection of transaction usage by observing the blockchain should be fully decoupled as the definitive source of wallet state.
- While it is still necessary to track wallet state by observing the blockchain it still has to be reconciled and factored into the wallet state.
Exploring state storage
Given that the wallet becomes aware of a transaction through some known method, once it has the transaction it needs to store it in a way that suits the possible ranges of use that ElectrumSV will make of it.
Blockchain as the source of truth
With a P2P first design and an expectation that ElectrumSV is the sole agent in control of the private key, the user’s actions should be definitive whether transactions they sign are retained privately or broadcast to the blockchain. It should not be necessary to reconcile any difference in the blockchain.
There are however some situations we should handle.
One of the recent observations of wallet corruption we had was related to a user having two wallet softwares that used the same keys. They created a local transaction with ElectrumSV and did not broadcast it, and then created and broadcast a transaction in the other wallet software that used some but not all of the same keys. The fault here lies in the hands of the user, who should accept all responsibility for loading the same seed words into different wallet software.
ElectrumSV will never support this other wallet software using the keys that it is in control of. If the user is doing that, then the problems they will create and encounter go beyond the above observation. However it is a requirement that we support overlapping transactions for other reasons. It should be possible to have multiple transactions that use the same keys recorded and tracked in the wallet, one rendered invalid by later key usage and one rendered the sole valid one due to it having been mined. Only the valid one should be associated with the state (and therefore balance) of the account it’s keys are owned by.
There are other situations where the user might find their wallet unexpectedly receives an overlapping transaction, including where their keys have been stolen and their coins spent from under them.
This is a larger subject that covers construction of a transaction the wallet user is signing for their own use, to participation and persistence in the signing of a transaction involving external devices or parties.
A stable ID
A complete transaction does not change and has a stable identifier in the form of it’s SHA256 hash. Each alteration to an incomplete transaction changes its hash. Rather than having separate states for incomplete transactions and complete transactions and trying to reconcile the two, or moving the state as the hash of the incomplete transaction changes, it makes a lot more sense to just allocate a random hash sized value and use that as a stable ID for an incomplete transaction. When the transaction is finalised or completed, it’s state can be moved under its final hash.
The wallet should never be in a situation where it allocates available coins for multiple purposes at the same time. If a coin is available, and it is selected to fulfil the requirements of an incomplete transaction, it should become unavailable regardless of how long it takes for that transaction to be completed. The work required becomes storing the spending of a coin by an incomplete transaction in a way that works with how it is done for complete transactions.
Additionally the wallet should never consider the change from incomplete transactions as available coins that can be spent.
These can be considered a variation on incomplete transactions where the transaction is complete but not necessarily in a final form. It can be placed in the non-final mempool of miners to commit the latest updated version of the transaction for mining. Coins spent in the transaction cannot be double spent, and after a period of time without further update the transaction will become finalised through it’s lock time and get mined.
A core requirement of non-final transactions is that they have the same set of inputs in the same order, so the implication is that coins have to be committed to it for the duration of its lifetime as until it becomes final. Presumably this constraint is part of the negotiation between payers and payees, where both parties have an interest in finalised transactions being settled at a minimal value and a new non-final transaction being created for subsequent payments. This is of course perhaps most obviously useful for payment channels.
Much like incomplete transactions these do not have a stable hash before finalisation, but rather than using a random Id it should be possible to use a preimage hash that has a guarantee of uniqueness based on the unique set of inputs and the requirement that at least one input in an updated transaction must feature an nSequence value that increases.
Representing a full model of an incomplete multi-signature transaction in the database is outside the scope of this refactoring work. That is a larger problem, which requires further analysis.
ElectrumSV currently only supports a specific kind of multi-signature, where the cosigners have a mutual arrangement to manage a pool of funds. Within the scope of refactoring, having a stable ID and coin allocation against it should be sufficient.
These are the finalised items, which spend existing coins and produce additional unspent coins. Any unspent coins produced by valid transactions are available for spending, however as in the overlapping transaction case not all complete transactions are necessarily valid.
What makes an invalid transaction?
The following list is an incomplete range of possibilities:
- A signed but not shared transaction that was deleted, but retained by the wallet and not deleted. It might be that this is what happens to all removed transactions, and they might not be deleted until the wallet designates it safe, or the user runs some sort of manual database compaction process.
- An existing signed transaction that conflicted with a transaction sourced from the blockchain, perhaps because the private keys were stolen and the coins were spent out from under the user.
Monitoring for irregularities
Just because the wallet has shared state externally, does not mean that what is expected to happen to that state inevitably happens. The wallet has an interest in monitoring whatever has to be monitored and following up as necessary.
Merchant API usage
When a wallet receives a transaction from another party, it might be that the Merchant API service of a miner was selected for broadcasting based on fee estimates. The wallet then waits for either merkle proof notifications or double spend notifications. Given that the wallet has obtained a fee quote and the fee quote has not expired, there is an expectation that the transaction will be mined in a reasonable time period long before it expires from the mempool. However it is in the best interest of the wallet to keep track of submitted transactions and be aware of any potential problems.
If Merchant API gives a wallet a way to have assurance their transaction will be mined in an upcoming block by a given miner, then it is worth identifying how a wallet knows their transaction has been relayed into the non-final mempool. This should in theory be exactly the same as submitting a final transaction, but perhaps the difference would be that the wallet has no guarantee of knowing if, or perhaps any way of ascertaining, how long the transaction stays in the non-final mempool. This however is getting lost in the weeds, it is a detail that becomes more important when ElectrumSV is making use of non-final transactions.
For now it is enough to know how non-final transactions will factor into the larger model of transaction state management. And that we can do custom monitoring and management of the payment process and reconciling the implications of their presence in the non-final mempool when it matters more.
At this point we should have some picture in mind of how transactions are obtained, and how we expect to make use of them, intentionally constrained by the chosen scope of this refactoring effort. The next step is to define how the refactoring work will be done.
This copy of the design document intentionally excludes implementation related details, in order to reduce the length.
Feel free to comment about this document on PowPing: