Migrate legacy wallets to descriptor wallets (wallet, rpc/rest/zmq)


Host: achow101  -  PR author: achow101

The PR branch HEAD was df1dca43c277704d3959253037cba651ba2c0b46 at the time of this review club meeting.


  • PR #17261 introduced the ScriptPubKeyMan interface for classes for managing keys and scripts. All of the key and script management code for the Legacy wallet was moved into a LegacyScriptPubKeyMan.

  • PR #16528 introduced Descriptor wallets, an entirely new type of wallet which uses output script descriptors to manage keys and scripts in a wallet. This was implemented by introducing a DescriptorScriptPubKeyMan.

  • The old type of wallets are referred to as Legacy wallets and they are slowly being deprecated and removed. However there needs to be a way to turn a Legacy wallet into a Descriptor wallet so that users do not find themselves unable to use their wallet.

  • In addition to changing how keys and scripts are tracked, Descriptor wallets also redefine the concept (and implementation) of IsMine. Instead of allowing both private keys and watch-only addresses in a single wallet, Descriptor wallets can either always have private keys, or never have private keys.

  • PR #19602 implements an RPC which migrates a Legacy wallet into a descriptor wallet by computing descriptors for everything the legacy wallet would have considered to belong to it. These descriptors are subsequently added to a newly created descriptor wallet with the same name as the original Legacy wallet.

  • Legacy wallets determine whether a script belongs to it by checking to see if it would be able to sign for the script, or the script is explicitly being watched. Descriptor wallets determine this by comparing the script to the set of scripts computed from the stored descriptors.

  • This PR is a step in the roadmap for the eventual removal of the Legacy wallet and the Berkeley DB dependency.


  1. Did you review the PR? Concept ACK, approach ACK, tested ACK, or NACK?

  2. Try creating a Legacy wallet and then migrating it using migratewallet. How can you tell if the migration was successful?

    • A Legacy wallet can be created with bitcoin-cli -named createwallet wallet_name=<name> descriptors=false
  3. In your own words, briefly describe the migration process.

  4. What kind of scripts will a Legacy wallet consider to belong to it for a single key (Hint: Look at LegacyScriptPubKeyMan::IsMine)?

  5. Under what circumstances will a Legacy wallet watch for a multisig?

  6. Why is LegacyScriptPubKeyMan::GetAllScriptPubKeys() needed (Hint: Consider how DescriptorScriptPubKeyMan::IsMine works))?

  7. Why isn’t the HD seed ignored when looking at all of the keys in a Legacy wallet?

  8. What happens to watch-only addresses, scripts, and pubkeys that were in the Legacy wallet?

  9. What happens if the migration fails for some reason? How does the user recover their wallet?

Meeting Log

  117:00 <achow101> #startmeeting
  217:00 <svav> Hi
  317:01 <achow101> Welcome to PR review club. Today we're looking at #19602 wallet: Migrate legacy wallets to descriptor wallets
  417:01 <bitplebpaul> hi
  517:01 <B_1o1> hi, all
  617:01 <effexzi> Hey every1
  717:02 <achow101> did everyone get a chance to review the PR and the notes? y/n
  817:02 <svav> Read the notes
  917:02 <michaelfolkson> hi
 1017:03 <larryruane> hi
 1117:03 <hernanmarino> hi
 1217:03 <glozow> hi
 1317:03 <B_1o1> y/y, basic test of new wallet migration
 1417:03 <Kaizen_Kintsugi_> hi
 1517:03 <Kaizen_Kintsugi_> n, just gonna lurk
 1617:03 <michaelfolkson> y
 1717:03 <glozow> read notes, still reviewing
 1817:04 <hernanmarino> read notes only
 1917:04 <michaelfolkson> B_1o1: You generated a legacy-bdb wallet, sent funds to it and then used the migratewallet RPC?
 2017:05 <svav> Can someone describe the nature of the Berkeley DB dependency?
 2117:05 <svav> *briefly* XD
 2217:05 <B_1o1> michaelfolkson: just generated the legacy and migrated, didn't sent funds
 2317:06 <glozow> it's an evil, evil dependency that we must dispel
 2417:06 <achow101> svav: It is used as the database system for the legacy wallet.
 2517:06 <achow101> it's also unmaintained
 2617:06 <brunoerg> did I miss it?
 2717:06 <svav> So what would it be replaced by?
 2817:06 <achow101> sqlite
 2917:06 <svav> ok thx
 3017:07 <achow101> did anyone try to migrate a legacy wallet?
 3117:07 <glozow> i found this video to be a nice intro to descriptor wallets https://www.youtube.com/watch?v=xC25NzIjzog
 3217:08 <michaelfolkson> B_1o1: Cool. I thought to test the migration you'd need to have some funds (testnet/signet maybe) at an address. Otherwise you are migrating an empty wallet to a different empty wallet
 3317:08 <B_1o1> yeah I migh
 3417:08 <B_1o1> might try later
 3517:09 <glozow> or use regtest and generate to an address owned by the wallet
 3617:09 <michaelfolkson> Is that right achow101? Is there value in testing migration without any funds in that wallet?
 3717:10 <achow101> michaelfolkson: certainly. the migration deals with keys and scripts, not really transactions. Even without funds you can check that the keys and scripts were migrated correctly
 3817:11 <michaelfolkson> But keys with no value attributed to them? They were just previously generated but no Bitcoin sent to them?
 3917:11 <achow101> we must still watch for funds sent to them all the same
 4017:11 <hernanmarino> you might receive funds in the future, i guess
 4117:11 <michaelfolkson> Ok cool
 4217:12 <michaelfolkson> B_1o1: You should definitely comment on the PR with what you tested then :)
 4317:12 <B_1o1> michaelfolkson: +1
 4417:13 <achow101> could someone briefly describe the migration process?
 4517:13 <glozow> Basically pack everything into a `MigrationData` struct - keys, keyids, hd seeds, key origins, scripts, etc
 4617:14 <svav>  descriptors are computed for everything the legacy wallet would have considered to belong to it
 4717:14 <svav> These descriptors are subsequently added to a newly created descriptor wallet with the same name as the original Legacy wallet
 4817:15 <glozow> and then create a descriptor wallet, generate descriptors using all this stuff
 4917:16 <glozow> also make a sqlite db and write everything
 5017:16 <achow101> svav: indeed!
 5117:16 <achow101> how do we compute everything the legacy wallet would have considered to belong to it?
 5217:17 <Kaizen_Kintsugi_> GetLegacyScriptPubKeyMan?
 5317:18 <achow101> Kaizen_Kintsugi_: not quite. That just retrieves the ScriptPubKeyMan object for all the legacy wallet logic
 5417:18 <B_1o1> if it's able to sign for the script or if the Script is being watched?
 5517:19 <Kaizen_Kintsugi_> Hmmm m_database->makeBatch? this seems to scoop up all the records in the database
 5617:19 <larryruane> glozow: "and then write everything" ... If the node crashes at a random time during the migration, are we guaranteed to be okay? Start up the node again and it finishes the migration?
 5717:19 <glozow> is it everything that isn't ISMINE_NO?
 5817:19 <achow101> B_1o1: that's part of it
 5917:20 <achow101> glozow: and how do we determine that?
 6017:21 <Kaizen_Kintsugi_> ISMINE_No determines if it is a watchonly?
 6117:22 <glozow> should i be looking at `IsMineInner()` ?
 6217:22 <achow101> yes
 6317:23 <achow101> Kaizen_Kintsugi_: ISMINE_NO means that the script does not belong to the wallet. The trick is finding the finite set of scripts that are not ISMINE_NO, as there is an inifinite number of scripts that are.
 6417:24 <glozow> aha. we look at the spk, and depending on the type, see if we have the key/script for it in our `keystore`
 6517:24 <Kaizen_Kintsugi_> good to know
 6617:24 <achow101> glozow: right! and what about multisigs?
 6717:24 <glozow> we need to own all of the keys involved
 6817:25 <glozow> https://github.com/bitcoin/bitcoin/blob/47bbd3ff4f5e0c04da4731c5d26d23d97cfa0bf1/src/wallet/scriptpubkeyman.cpp#L187-L192
 6917:26 <achow101> indeed. so how do we go from IsMineInner to a set of scriptPubKeys?
 7017:26 <achow101> or rather a set of descriptors
 7117:28 <Kaizen_Kintsugi_> Is it in APplyMigrationData?
 7217:28 <achow101> no
 7317:28 <Kaizen_Kintsugi_> MigrateToDescriptor?
 7417:29 <achow101> it's in GetAllScriptPubKeys
 7517:29 <achow101> sorry, GetScriptPubKeys
 7617:30 <michaelfolkson> Ha was gonna say, that isn't in scriptpubkeyman.cpp
 7717:32 <svav> Get the DescriptScriptPubKeyMans that have the same scriptPubKeys as this LegacyScriptPubKeyMan ??
 7817:33 <glozow> ok so here we're going through `mapKeys`, `mapCryptedKeys`, and `mapScripts`, and adding them to spks if `IsMine()`
 7917:34 <glozow> er - we only call `IsMine()` for the scripts
 8017:35 <glozow> and then in `MigrateToDescriptor()`, after we gather this set of spks, we create `desc_spkms` etc
 8117:36 <achow101> so a related question is why we have to do this at all? what is the difference between LegacyScriptPubKeyMan::IsMine and DescriptorScriptPubKeyMan::IsMine?
 8217:37 <Kaizen_Kintsugi_> DescriptorScripts seem like they are only spendable?
 8317:37 <glozow> `DescriptorScriptPubKeyMan` doesn't include watchonly 🤔
 8417:38 <Kaizen_Kintsugi_> +1 glozow
 8517:39 <glozow> so the watchonly stuff is put in a different `CWallet`?
 8617:39 <ls55> I think `DescriptorScriptPubKeyMan` also doesn't include `FillableSigningProvider::mapKeys`.
 8717:39 <glozow> std::shared_ptr<CWallet> watchonly_wallet{nullptr};
 8817:39 <glozow> std::shared_ptr<CWallet> solvable_wallet{nullptr};
 8917:39 <achow101> Kaizen_Kintsugi_: kind of, it depends on the wallet type, but Descriptor wallets get rid of the concept of mixed watchonly and spendable. Of course we have to deal with this somehow during migration
 9017:40 <svav> Descriptor wallets can either have private keys or be watch only, not both ??
 9117:40 <achow101> glozow: right
 9217:40 <achow101> svav: yes
 9317:40 <michaelfolkson> Why (get rid of this concept)?
 9417:40 <Kaizen_Kintsugi_> single responsibility pricipal?
 9517:40 <michaelfolkson> Just cleaner? It isn't descriptor related...
 9617:41 <glozow> definitely think this is simpler. why were watchonly and owned mixed together before?
 9717:41 <achow101> michaelfolkson: it really simplifies IsMine. Also separation of funds/duties/etc.
 9817:41 <Kaizen_Kintsugi_> watchonly wallets wouldn't need to store scripts?
 9917:41 <achow101> glozow: because it used to not be possible to have different wallet files for different purposes
10017:42 <achow101> multiwallet is relatively recent
10117:43 <Kaizen_Kintsugi_> This is to help organizational purposes I assume?
10217:44 <achow101> yes
10317:44 <glozow> are there other multiple wallets to separate beyond watchonly and owned?
10417:45 <ls55> External Signer ?
10517:45 <achow101> glozow: yes. you pointed out watchonly_wallet and solvable_wallet, but solvable_wallet is not the same as the spendable wallet. it's watchonly, but also slightly different
10617:46 <Kaizen_Kintsugi_> o rly? I thought solvable and spendable were sonoymous
10717:46 <Kaizen_Kintsugi_> I guess that has to do with multisig? where I would have 1 of N keys? I can partially solve but can't spend?
10817:47 <michaelfolkson> Kaizen_Kintsugi_: https://bitcoin.stackexchange.com/questions/63198/why-outputs-spendable-and-solvable-are-false
10917:47 <achow101> Kaizen_Kintsugi_: solvable means that we would be able to sign if we knew the keys
11017:47 <glozow> is spendable a subset of solvable?
11117:47 <achow101> but the split between watchonly and solvable actually goes back to LegacyScriptPubKeyMan::IsMine
11217:47 <achow101> there's actually a solvable, but not watchonly set of scripts
11317:47 <svav> Some background info on Descriptors https://github.com/bitcoin/bitcoin/blob/master/doc/descriptors.md
11417:48 <achow101> could anyone describe how that might happen?
11517:48 <Kaizen_Kintsugi_> missing script?
11617:49 <achow101> Kaizen_Kintsugi_: missing the script would mean it is not solvable
11717:50 <achow101> (this requires looking at SignStep in src/script/sign.cpp)
11817:50 <michaelfolkson> glozow: That's my understanding. You can't spend if not solvable. But it can be solvable without you being able to spend
11917:51 <ls55> `if scriptPubKey could not be completely satisfied.` ?
12017:51 <glozow> multisig but don't know all pubkeys?
12117:51 <glozow> wait is that a thing
12217:52 <vnprc> does it have to do with standard script types?
12317:52 <ls55> `
12417:52 <ls55> case TxoutType::NONSTANDARD:
12517:52 <ls55> case TxoutType::NULL_DATA:
12617:52 <ls55> case TxoutType::WITNESS_UNKNOWN:
12717:52 <ls55> `
12817:52 <Murch> Yeah, achow101: is there a way for an output to be spendable but not solvable?
12917:52 <glozow> no
13017:52 <achow101> It is not possible for a script to be spendable but not solvable
13117:53 <achow101> glozow: I think that might be true
13217:53 <michaelfolkson> glozow: As a general concept you do need to know the pubkeys of the multisig to spend from it
13317:54 <achow101> we could have a multisig script, but not all the keys for it, and also not necessarily be watching the spk
13417:54 <michaelfolkson> At least with CHECKMULTISIG, with CHECKSIGADD....same deal?!
13517:54 <Kaizen_Kintsugi_> makes sense
13617:55 <achow101> michaelfolkson: no, that's taproot so descriptors only
13717:55 <michaelfolkson> Indeed, thanks
13817:55 <glozow> okay yeah so if it was a 1 of 2 multisig script, you have a key and it's solvable, but you're not watching the script
13917:55 <ls55> Isn't legacy wallet compatible with Taproot ? why ?
14017:56 <michaelfolkson> Taproot is only descriptors (as Andrew said). Just design choice
14117:56 <achow101> ls55: the legacy wallet does not support taproot. we decided not to implement taproot for it because it will be going away soon(tm)
14217:56 <ls55> Ok. Thanks.
14317:57 <achow101> related to weird IsMine things, can anyone say what we do with the key for the HD seed?
14417:57 <ls55> Will `LegacyScriptPubKeyMan` be removed from codebase ?
14517:57 <achow101> ls55: that's the plan
14617:57 <svav> What is an HD seed?
14717:57 <ls55> Very good. It simplifies the code.
14817:58 <hernanmarino> achow101: it gets migrated ?
14917:58 <Kaizen_Kintsugi_> seed of an HD wallet
15017:58 <ls55> I think not
15117:58 <hernanmarino> that was an affirmation not a question .
15217:58 <ls55> `Note that we do not ignore the seeds themselves because they are considered IsMine!`
15317:58 <Murch> svav: The main secret of a wallet using hierarchical deterministic derivation
15417:58 <Murch> See BIP32
15517:59 <achow101> ls55: right! what's the implication of that?
15617:59 <svav> Thanks Murch:
15717:59 <hernanmarino> it is a valid key
15817:59 <ls55> A new seed is created in the new descriptor wallet ?
15918:00 <achow101> hernanmarino: yes, and as a valid key, you could receive Bitcoin to it even though it's corresponding addresses would never be given out
16018:00 <hernanmarino> exactly :)
16118:01 <Murch> So, we generate one wallet from the seed and another from the descriptor that replaces the derived addresses of the seed?
16218:01 <achow101> Murch: the same wallet with multiple descriptors
16318:02 <achow101> we're at the top of the hour, so thank you all for coming and for reviewing the pr.
16418:02 <achow101> #endmeeting