Unspent output notifications
Work is progressing moving ElectrumSV from using the existing ElectrumX servers, to using our new reference server. A lot of the functionality in the reference server is complete, but one piece that isn’t is unspent output notifications.
The old model
How ElectrumX allows a wallet to find existing transactions and hear about new transactions in both the mempool and in blocks, is by monitoring script hashes. The wallet has to know how coins were spent to find them. And the server has to index all relevant transactions in the blockchain, so it can provide any data the user is asking for.
This older model provided an awkward way to do a lot of the things, but it also covered some but not all of what unspent output notifications could also be used for, given it was a brute force approach.
This older model is what we have to move away from, because it is not a viable one as blocks get bigger and bigger. We could see some killer app arrive at any time to take us past the point where it is no longer feasible for our servers to continue to run, and ElectrumSV could stop working.
The new model
We will no have longer a full index of the blockchain. It will be an expensive action to ask “look back through the blockchain and find me this”. The user needs to know what has already happened, and keep that data, and provision services to monitor ongoing usage to add to their existing data and react to.
This is less relevant for being able to tell what outputs are or are not spent, and to hear about when they get spent, as most future-looking indexers will have enough information to know about the UTXO set. But having that UTXO set and being able to build services on it, fill outs a range of the functionalities that full blockchain index provided and provides more that it could not.
This is not intended to be a definitive list of the applications that could be made of APIs that provide unspent output functionality, but just the ones we know we are more than likely going to make use of.
Detecting transaction spends
If you want to know if a transaction has been spent, the simple approach seems to be to go away and ask a service “have you seen the transaction with this hash”. The service will either say yes, in which case you will know it has been broadcast or mined. Or the service will say no, in which case you do not really know if it has been broadcast or mined.
Malleation is where a fully signed transaction is altered in some way that does not make it invalid. The hash of the transaction changes with the alteration and this means the transaction can be broadcast but under that different hash. There are some nuances to this like most of the current approaches in which interesting (and not bad!) applications of malleation are currently disabled, relating to things named like clean stack and push data only script validation constraints. And generally, people won’t malleate. But why choose a solution that is not future proof? Better to use one which is, and monitor the spent outputs in your transactions to detect if the transaction is spent. We need to build for the future, not for the present.
One case where you want to know if a transaction is broadcast, is where you give a transaction to an external party, but they might not broadcast it right away or even in the near future. An example of this is where the payee in a payment channel has a refund transaction created in the process of opening that channel, and the payer may want to know if the payee broadcasts it.
Detecting alternative spends
If you do have a local transaction that was partially or wholly funded by another party, where a local transaction is one that has not been broadcast or mined, you may want to detect if they spend the coins they committed to you. This does not have to be malicious either, although it could be. I am sure there are lots of cases where the payee will be forgiving if there is a non-critical payment and payer reallocates the funds they need elsewhere. We do not live in robot world, people will be dealing with people.
We can formalise a process for this if necessary where the payer’s wallet notifies the payee as part of the process of reallocating the funds.
Detecting these is likely to not even be necessary to do explicitly, and can be done as part of just observing the results of “detecting transaction spends”.
Detecting coin spends
One thing that a full blockchain index gave us, was the ability to know for sure if a wallet sent or received funds. But this was always a benefit of the broken Bitcoin blockchain crippled with the false limitations imposed on it. It is not a given that two independent wallets can monitor the same keys and know how they are used, if those false limitations are removed.
This was relevant in implementing watch-only wallets where the user could create a secondary read-only wallet elsewhere and have it pick up received and spend funds. It was also used in multi-signature wallets to detect whether there were any changes, but this was a rough approximation of a much better user experience that is not based on this approach. The whole concept of monitoring key usage for detecting spends and receipts is inherently flawed and regressive, and serves to hold any application doing it back. It is possible the only future for watch-only wallets is monitoring mutually known unspent coins, to catch if they move. And for this, the unspent outputs notifications are sufficient.
Another naive usage that users made, simply because it worked and because those false limitations made it possible, was that they could operate multiple synchronised copies of a wallet in different locations. This only worked because of those limitations, and in the Bitcoin SV future of wholesale enterprise and everyday integration into life, it either holds a wallet or application back or does not work.
Synchronised wallets can be done properly, although whether we can afford the attention and manpower to support it is another matter. We would do this through adding the proper backup to ElectrumSV where a user stores their full wallet data via a backup API and all the high level metadata that defines unpredictable spends, receipts and anything else can be synchronised.
This also highlights a need for the implementation to serve as a way to find out how a UTXO is spent.
The highest priority is to get this working in our sample indexer, it does not have to be long term implementation. As long as it serves our needs, we can integrate it and hook it up. If at a later time it is apparent there is a better way, we’ll just implement it and change the reference server API.
The simplest approach is to just have the wallet register for notifications for any outpoint it is interested in, and if they are spent it immediately gets the status back, and if they are not the registrations are recorded and observed. This is what I will most likely implement, given it is an unexplored space of potential applications there is no known path forward for this. It is as effective to just pick one and see how it pans out and pivot if necessary.
In the longer term, arbitrary queries for what spent what are also worth supporting.