Random development notes #1

My notes aren’t as comprehensive as I’d like. I can tell this, because when I went to write up this article, I knew I didn’t know some things I did know when I did the main work on researching these topics. The following topics are either thoughts I’ve had, or the result of discussions with fellow ElectrumSV developers. You’re welcome to use the Slack links at the bottom of this article to contact us and discuss this with us.

Databases, PostgreSQL and SQLite

The beta version 1.3.0 of ElectrumSV currently depends on SQLite as the database provider used for our wallet storage, but we should also be able to allow users to select PostgreSQL as an alternative. Anyone who uses ElectrumSV as a UI-based desktop wallet would get SQLite chosen as the default, but more advanced users like those that use ESV as a wallet server via it’s REST API might get PostgreSQL chosen otherwise.

One problem with this is that our SQL is SQLite-specific, and to some degree the code is too. In theory we could switch to using an ORM like peewee, which abstracts table creation and migration, and it limits dependencies which is very important for auditing updates. The main downside is that the cost arises in that any query that isn’t simplistic becomes much harder to write. This pushes developers to write the SQL directly, which then implies that it has to be written and tested for every provider.

It might be that it is best to do all the database abstractions ourselves, and test all our database code against all the supported providers. It might be that it is best to choose peewee, and have it do the database abstractions outside of select queries, which we might test explicitly.

This isn’t a priority yet. It requires a lot of work, and a lot of testing will be necessary to verify it works correctly.

Document generation

With the addition of some preliminary in-wallet help in both the wallet wizard (used for creating new wallets) and the account wizard (used for adding new accounts to the current wallet), it seemed like a good time to investigate solutions.

The goal was to have text documents, and to have some form of pipeline output them as HTML. This would be used for generating the web site, formal documentation and in-wallet help documents. And both the text documents and the output created from them should be checked into the main ElectrumSV github repository, so that they could be updated and present for any given related code commit in that same repository.

I evaluated some of the options for this, and came to the conclusion that the best solution was Pelican. The requirements for suitability were:

  1. Based on Python 3.
  2. Regularly and recently updated.
  3. Able to take both reST and HTML formatted text as input and output styled HTML using Jinja templates.

I had a notion that the in-wallet help would be turned into HTML, which would be pulled into the Pelican process as inputs. Perhaps the in-wallet help would pass through Jinja2 directly, and the web site and documentation would then pass through Pelican.

The web site should include ElectrumSV-related articles and announcements that would be generated using this method, rather than me posting them in my Medium account. After all, Medium is supposed to be unprofitable and have an uncertain future. Also, it’s best to have people come to the ElectrumSV web site for ElectrumSV news and information.

This is on the back-burner and when a break is needed from other more important tasks, it can be picked up and some more work done on it.

Wallet encryption and privacy

When the entire contents of a wallet are in one large JSON document, which gets read in when a wallet is opened and written out when the wallet is closed, it’s easy to just encrypt the whole thing. But when you store the wallet in a database, you lose all the advantages of using a database if you encrypt everything for privacy.

ElectrumSV 1.2.5 (the current official release) uses the large JSON document. Electrum 1.3.0 (the current in-development beta release) uses the database. This means that anyone moving to 1.3.0 will lose the privacy that they had in 1.2.5. Keys, seeds and other important information are still encrypted in the database and can be viewed using the ‘View Secured Data’ options and providing the wallet password for decryption.

If someone has access to your computer, they can do much worse than read your balances, public keys and even addresses should they open and execute queries against your wallet database. They can install a key logger or something, and steal your bank password and so on. They can get your wallet password and bypass even the full privacy of your fully encrypted and “private” 1.2.5 wallet.

This is not a priority at all. It’s not even on the radar.

A dumber blockchain indexer

ElectrumX on the STN is regularly falling behind due to the large blocks that occur there. It also indexes by script hash, which means that if you want to look for key usage, you need to form all the possible scripts that could have been seen using that key and register their hashes. ElectrumX will send you a list of all their usages, both the spending and receiving transactions, and as long as you maintain the connection it will send any new usages that happen from then on. It also tracks UTXOs which ElectrumSV does not care about.

The vague plan is to write a new blockchain indexer that indexes pushdata. But we only care about pushdata that is public key sized or public key hash sized. This should allow us to find:

  • Pay to public key (can match on public key).
  • Pay to public key hash (can match on public key hash).
  • Bare multi-signature (can match on public key).
  • Accumulator multi-signature (can match on public key hash).
  • Pay to script hash (can match on the script hash).

We will put all this in a database, and link the matches to both spending and receiving transactions. Searching should allow paging, and we should also support the block header notifications and other functionality that ElectrumSV expects from it’s ElectrumX server.

One confounder is that indexable protocols like Tokenized may not identify their protocols with keys or key hashes, and we’d have to add custom special case indexers for those that cannot be matched using the key/key hash indexing. In Tokenized’s case we’d need to match on a string (e.g. “tokenized”) at some pushdata offset. I think unless we’re going to index full pushdata, or all smaller pushdata within some reasonable criteria, this is just something we’ll have to deal with.

No work has started on this, and it’s mostly just being mulled over at this time.

Links

Web site and downloads

Official web site: https://electrumsv.io
Official web site downloads: https://electrumsv.io/download.html

Support

If you need some assistance with something, please submit an issue at the following link. But please, fill out a template otherwise we will not have the information we need and you will have to wait longer for assistance.

Report an issue: github.com/electrumsv/electrumsv/issues

We do not provide support over Twitter, and will request you submit an issue. Support over twitter is much more difficult, and we prefer to avoid it completely.

Discussion

You can reach us for discussion in one of two possible locations.

ElectrumSV developer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store