#15931 Remove GetDepthInMainChain dependency on locked chain interface (wallet)


Host: jnewbery


  • This PR is the latest in a sequence of PRs to clean up the node-wallet interface.
  • We previously reviewed PR 15713 in Bitcoin Core review club. See the notes and logs for that meeting for more information about the interface and the recent work to tidy it up.
  • One of the main goals of that work is to remove the wallet’s ability to lock cs_main. PR 16426 is a proof-of-concept PR which does that.
  • This PR is a big step towards removing the wallet’s ability (and requirement) to lock cs_main. It removes the locked_chain dependency from the CWalletTx::GetDepthInMainChain() function.
  • For a given wallet transaction, GetDepthInMainChain() returns how many confirmations that transaction has in the block chain.
  • When a wallet transaction is included in a block, the block’s hash is stored in the CWalletTx object (see hashBlock and SetMerkleBranch().
  • GetDepthInMainChain() previously worked by taking that hashBlock and checking its depth in the block chain. That requires locking cs_main since block chain state is being accessed.
  • After this PR, each wallet transaction stores the height of the block that it was confirmed in, and the wallet stores the height of the best block in the block chain. By storing these values internally, the wallet no longer needs to query the block chain state to calculate the transaction’s number of confirmation.
  • Part of this PR has been split off into a separate PR, wallet: encapsulate transactions state to make review easier. Reviewers should leave comments on that PR before reviewing this PR.


  • An early version of this PR added an m_block_height field the wallet transaction serialization (comment). Why wouldn’t this work?
  • The PR author offers two ways for the wallet to populate the wallet transactions’ heights (save the transaction height to disk or calculate the height for each transactions at wallet load time). What are the trade-offs? Which approach do you prefer?
  • How does the wallet learn about new transactions in the mempool or included in blocks?
  • What are the wallet’s expectations about block notifications? Is it a problem if the wallet is informed of the same block more than once? If blocks arrive in the wrong order? If a block is skipped? If a block is re-orged out of the main chain?

Meeting Log

13:00 < jnewbery> hi!
13:00 < digi_james> Hello!
13:00 < kanzure> hi
13:00 < emilengler> hi
13:00 < dergigi> hi
13:00 < ariard> hi
13:00 < lightlike> hello
13:00 < peevsie> hi
13:00 < jonatack> hi
13:01 < michaelfolkson> Hey
13:01 < jnewbery> jkcqyq dergigg: thanks for leaving review comments on the PR!
13:01 < jnewbery> sorry, jkczyz
13:02 < jnewbery> what did people think of the PR this week?
13:02 < jkczyz> hi
13:02 < jnewbery> perhaps jkczyz or dergigi (or anyone else) could give some initial thoughts about the PR
13:02 < nehan> hi
13:04 < jnewbery> I like this PR. It's a good next step to decoupling the wallet from the node
13:04 < dergigi> I don't have any specific questions unfortunately - just ran into the compilation error (it's the first PR I'm looking at, still trying to wrap my head around the code)
13:04 < jkczyz> Initial thoughts were having separate members for hash and height could be prone to bugs, but noticed there was a suggestion to combine
13:04 < jnewbery> dergigi: that's ok. Thanks for trying to test and leaving a comment - that's useful in itself
13:04 < jkczyz> did not have a chance to look at the PR that was made for that thought
13:04 < jkczyz> s/thought/though
13:05 < jnewbery> jkczyz: you mean #16624?
13:05 < jkczyz> yes
13:05 < jnewbery> https://github.com/bitcoin/bitcoin/pull/16624 wallet : encapsulate transactions state
13:05 < dergigi> I like the concept - seems like a good idea to store the tx block height internally so we don't have to look into the chain for every tx
13:06 < jnewbery> right - the PR author pulled out a small part of this PR and made it into its own PR. I'd already chosen 15931 for today's discussion and didn't want to change it in case you'd all started reviewing it
13:06 < michaelfolkson> It is certainly one that is interlocking with a bunch of other PRs and requires some organizational roadmap so they are merged in the right order. Seems like a high quality PR by itself.
13:07 < jnewbery> dergigi: yeah, it's good to reduce the wallet's reliance on locking the node's chain state
13:07 < nehan> i found it pretty difficult to chase the chain of PRs and understand everything together
13:07 < lightlike> I found this PR not so easy to review, but obiviously the concept of not having to query the block height makes a lot of sense.
13:07 < jnewbery> nehan: do you think you figured it out in the end? Any questions?
13:07 < jnewbery> lightlike: what made it difficult? Was the PR just too large?
13:08 < jnewbery> I think https://github.com/bitcoin/bitcoin/pull/16426 gives a good summary of what the end goal is
13:08 < nehan> jnewbery: no, but looking forward to discussing. the goal is great, but given i'm pretty unfamiliar with the wallet code i can't convince myself that these PRs don't change behavior (or that the behavior they change is 'safe')
13:08 < lightlike> jnewbery: It was large, but also some of the elements (the callback mechanism) were not so trivial to understand for me.
13:09 < jnewbery> "I can't convince myself that these PRs don't change behavior (or that the behavior they change is 'safe')" - I agree this is very difficult!
13:10 < jnewbery> What would you want to do to convince yourself of that?
13:12 < jnewbery> anyone?
13:12 < ariard> yes sorry for the big PR and splitting it between 15931 and 16624, it happened that wallet state transitions weren't clear for anyone
13:12 < jkczyz> Have test coverage for the behaviors
13:12 < jnewbery> jkczyz: yes please!
13:12 < lightlike> familiarize myself more with the wallet in general
13:13 < jnewbery> One thing that we're really missing is testing of old wallet.dat files
13:13 < jnewbery> There's an issue here: https://github.com/bitcoin/bitcoin/issues/14536
13:13 < jnewbery> I think it'd be really useful to have wallet files produced by lots of old versions of bitcoin core and tests for them
13:14 < ariard> jkczyz: have a look on 16624 listed issues in opening message, I think some behaviors aren't tested
13:14 < jnewbery> for PRs like 15931 and 16624, where there are changes in serialization (or at least in the way we deserialize and hold data at runtime), being able to do regression testing on old wallet files would be really useful
13:15 < jnewbery> lightlike: when you talk about 'callback mechanism', is that things like the BlockConnected/BlockDisconnected/TransactionAddedToMempool stuff?
13:16 < jnewbery> first question I had was: An early version of this PR added an m_block_height field the wallet transaction serialization (comment). Why wouldn’t this work?
13:16 < jnewbery> any thoughts about that?
13:16 < nehan> breaks compatibility with old wallet.dat files?
13:17 < jnewbery> nehan: yes exactly
13:18 < provoostenator> I have a PR that tests both forward and backward compatibility of wallets, though not ancient: https://github.com/bitcoin/bitcoin/pull/12134
13:18 < jnewbery> serialization is a bit tricky with the wallet. We store wallet objects in bdb, which is a key value store.
13:18 < lightlike> jnewbery: yes, that's what I meant. Not the changes done, in this PR, which are clear, but how the whole mechanism really works level.
13:18 < lightlike> *at a low level
13:18 < jnewbery> The serialization for each individual object is defined in the object declaration in the header file
13:19 < jnewbery> eg https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/wallet.h#L492
13:19 < jonatack> provoostenator: nice!
13:19 < jnewbery> but there's also a bunch of code in walletdb.cpp that fiddles around with that serialization
13:20 < jnewbery> here: https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/walletdb.cpp#L200
13:21 < jnewbery> serialization just serializes various data elements from the wallet in a specified order: https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/wallet.h#L505
13:21 < jnewbery> which makes it very difficult to version the serialization code
13:22 < jnewbery> ok, next question: The PR author offers two ways for the wallet to populate the wallet transactions’ heights (save the transaction height to disk or calculate the height for each transactions at wallet load time). What are the trade-offs? Which approach do you prefer?
13:22 < bcribles> you didn't miss any discussion while disconnected
13:23 < michaelfolkson> What version does it test back to <provoostenator>
13:23 < michaelfolkson> ?
13:23 < jnewbery> thanks bcribles!
13:24 < jonatack> jnewbery, ariard: how hard do you think it would be to clean up that serialization?
13:24 < jnewbery> jonatack: what do you mean by clean up?
13:25 < jonatack> untangle it to be easier to version
13:26 < dergigi> side note: i really like the tx status (suggested by ryanofsky) in 16624: unconfirmed/confirmed/conflicted/abandoned - the comments on that are very clear, good job on that ariard
13:26 < jnewbery> jonatack: quite difficult. We absolutely need to remain compatible with old wallet.dat files, and it's also important that new wallet.dat files are compatible with old versions, so we couldn't just change it completely
13:26 < lightlike> jnewbery: the serialization link https://github.com/bitcoin/bitcoin/pull/15931#discussion_r295434058 you gave pointed to CMerkleTx. Isn't that only for old wallet files, so why try to add block height there? Or did the change to CWalletTx happen only recently?
13:27 < ariard> could we version wallet.dat files in a backward compatible way ?
13:27 < jnewbery> if we were designing it from scratch, we might use a type-length-value scheme so wallet software could just ignore fields that it doesn't recognize, but we're constrained by all the existing wallet.dat files already in existence
13:28 < jnewbery> lightlike: the change to CWalletTx happened recently, resulting from a suggestion in this PR
13:28 < jnewbery> https://github.com/bitcoin/bitcoin/pull/16451
13:28 < lightlike> oh ok, then it makes sense.
13:29 < jnewbery> ariard: I'm not sure. Does the serialization for the wallet include the software version that wrote it?
13:29 < jnewbery> In any case, we'd need to leave the code for deserializing 'legacy' wallet.dat files in place for a long time
13:30 < fjahr> fg
13:31 < ariard> yes version with birthday would be cool, and pumping a message to user with which core version to use
13:31 < hugohn> brainstorming: perhaps one way to get around the lack of versioning in older Core versions is to write new stuff to a new .dat file? and start versioning in the new .dat file?
13:31 < hugohn> not sure if that's feasible
13:33 < ariard> let's say a new frontend serializer you may try to read first byte of files, if there is a magic number showing versioning support you use new serializer if not the old one
13:33 < jnewbery> lightlike: going back to your previous point: all of the wallet callbacks come from CValidationInterface functions. I think there are only 5 of those methods that are overridden by the wallet:  TransactionAddedToMempool, TransactionRemovedFromMempool, BlockConnected, BlockDisconnected, UpdatedBlockTip, ChainStateFlushed
13:34 < jnewbery> it's worth looking at the NotificationsHandlerImpl in src/interfaces/chain and understanding how those interface methods are called
13:34 < ariard> jnewbery: it's a bit more complicated given we use Chain::Notifications as an adaptor between CValidationInterface and wallets
13:35 < jnewbery> ariard: that's right. It's a bit of indirection, but they're direct function calls from Chain::Notifications
13:35 < jnewbery> I haven't seen any answers to: The PR author offers two ways for the wallet to populate the wallet transactions’ heights (save the transaction height to disk or calculate the height for each transactions at wallet load time). What are the trade-offs? Which approach do you prefer?
13:35 < fjahr> sorry, wrong window :/
13:36 < jnewbery> One potential disadvantage to calculating height at wallet load time is that it might impact performance for a large wallet
13:37 < jnewbery> ryanofsky thought it wouldn't be too much of a problem: https://github.com/bitcoin/bitcoin/pull/15931#issuecomment-519099417
13:38 < ariard> I think it's a performance hit at load time but a win at run time, you don't have to query chain anymore
13:38 < hugohn> saving tx height to disk seems to make more sense in the long-term, doesn't make sense to recompute the height each time the process starts, which doesn't change.
13:38 < jnewbery> ariard: I'm comparing doing the calculation at start up, or serializing it to disk. Both are a win at run time
13:39 < nehan> what is the likelihood that the heights have changed since the process was restarted?
13:39 < jkczyz> saving to disk would require a serialization change, right?
13:39 < jnewbery> jkczyz: exactly correct
13:39 < jnewbery> which we should be cautious about doing without thorough testing
13:40 < jnewbery> nehan: very slim, but possible. There would have to be a re-org over that block while the wallet was offline
13:41 < nehan> jnewbery: and if that happened, would there be a process to update the txn heights if they were loaded from disk?
13:41 < provoostenator> @michaelfolkson at the moment it tests back to 0.17.1, but more could be added. The main drawback is having too many switch statements in Python test framework to deal with ancient RPC stuff
13:42 < nehan> it seems to me it's not safe to rely on the serialized txn heights unless you're sure that on restart you'll get the appropriate notifications to update them (which you might. i'm not sure)
13:42 < michaelfolkson>  <provoostenator>: So how far do you think it could be reasonably extended back to?
13:42 < bcribles> possibly because I haven't read enough of the PR but the malleability of using height caused when a reorg happens (like nehan was saying) is something I need to work through
13:42 < ariard> nehan: yes you need others change before, where the wallet send a block locator of its current highest tip
13:42 < jnewbery> nehan: yes, the wallet knows its own best block hash and tries to find the point that it diverges from the main chain, then rescans from there
13:42 < jnewbery> https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/wallet.cpp#L4504
13:42 < hugohn> is it possible to save the tx height to a different field in wallet.dat?
13:42 < ariard> and if this one is on a forked branch, a reorg should happen by replaying old blocks
13:42 < ariard> and connecting new ones
13:42 < hugohn> without affecting existing CWalletTx serialization
13:43 < jnewbery> ariard: that's correct
13:44 < nehan> ariard: jnewbery: thanks
13:44 < jnewbery> to add some more detail: the wallet has a 'locator', which is a sparse list of blockhashes in its view of the block chain. It uses that to try to find a fork point with the nodes view of the block chain at startup
13:45 < provoostenator> michaelfolkson: in theory back to when regtest was added, in practice you'll just have to try one by one
13:45 < jnewbery> https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/primitives/block.h#L122
13:45 < bcribles> jnewbery e.g. a wallet would be able to reason about if the block a txn was confirmed in has experienced a reorg?
13:46 < jnewbery> Locators are also used in GETBLOCKS and GETHEADERS messages on the P2P network: https://btcinformation.org/en/developer-reference#getblocks
13:46 < hugohn> to me the decision to persist to disk or not probably shouldn't impact how the wallet reacts to reorgs - one is a memory management issue, one is a consensus issue. reorgs could happen _after_ we have recomputed the height at startup anyway, so the issues should be completely orthogonal. but I could be wrong.
13:47 < jnewbery> bcribles: not directly, but it would know that there had been a re-org since it went offline, and therefore rescan the block chain from a height where it knows it shares history with the node
13:48 < jnewbery> if a re-org happened when the wallet is online, we'd expect the node to inform the wallet of the blocks being rewound with BlockDisconnected calls, followed by BlockConnected calls for the new chain
13:48 < nehan> hugohn: agreed! i just wanted to understand how invariants are maintained, and expectations on how a newly loaded wallet will be updated
13:48 < jnewbery> next question: How does the wallet learn about new transactions in the mempool or included in blocks?
13:49 < jnewbery> (should be easy, since we've already talked about it)
13:49 < hugohn> nehan: yeah, great question!
13:49 < hugohn> CValidationInterface?
13:49 < digi_james> ValidationInterface Callbacks
13:49 < jkczyz> It registers for notifications from the chain and implementing interfaces::Chain::Notifications
13:50 < jnewbery> good, you've all been paying attention :)
13:50 < jnewbery> ok, final question: What are the wallet’s expectations about block notifications? Is it a problem if the wallet is informed of the same block more than once? If blocks arrive in the wrong order? If a block is skipped? If a block is re-orged out of the main chain?
13:52 < jnewbery> ok, it's a difficult question to answer without reading a lot of code!
13:52 < jkczyz> It would seem that AddToWalletIfInvolvingMe handles many of these cases
13:53 < jnewbery> your entry points are those ValidationInterface methods
13:53 < jnewbery> jkczyz: exactly. AddToWalletIfInvolvingMe() is key
13:53 < jnewbery> https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/wallet.cpp#L1204
13:54 < jnewbery> how does a wallet know if a transaction is interesting to it?
13:54 < hugohn> by calling IsMine()
13:54 < hugohn> or IsFromMe()
13:55 < jnewbery> hugohn: yes, IsMine() will determine if that transaction sends money to the wallet
13:55 < jnewbery> hugohn: and yes, IsFromMe() determines if the transaction sends money from the wallet
13:56 < jnewbery> important to note: IsFromMe() can only work if the transaction input is spending from a transaction that the wallet already knows about
13:56 < hugohn> the legacy IsMine() logic is quite involving, it does pattern-matching and recursive lookups to determine if something belongs to the wallet
13:56 < jnewbery> say tx A sends money to a wallet, and tx B spends an output from tx A
13:57 < jkczyz> It seems odd that BlockDisconnected uses this with a null block hash / position. At very least it's hard to follow what the code is doing in this case given the reuse for adding blocks
13:57 < jnewbery> if the wallet doesn't know about tx A, then it has no way of knowing that tx B is a spend from the wallet
13:57 < nehan> jnewbery: it doesn't look like AddToWalletIfInvolvingMe() handles blocks in the wrong order
13:57 < jnewbery> nehan: right, the wallet's expectation is that blocks are served sequentially
13:58 < nehan> jnewbery: good that aligns with my understanding of CValidationInterface
13:58 < lightlike> A comment in ValidationInterface says that it is guaranteed that calls arrive in the same order as events are generated in validation - in that case, how is it possible that a block is skipped?
13:58 < jnewbery> jkczyz: that's needed so the wallet can change confirmed transactions to unconfirmed in the case of a reord
13:58 < jnewbery> lightlike: it shouldn't be possible
13:59 < hugohn> could the arrive-in-order assumption still hold true once we have node/wallet process separation? and block notifications served over IPC?
13:59 < jnewbery> however, I believe the wallet should be fine if it receives a block more than once
13:59 < nehan> jnewbery: re "What are the wallet’s expectations about block notifications?..." I think the wallet does not expect or handle any of that
13:59 < digi_james> hugohn: I had the same question
13:59 < nehan> jnewbery: oh
13:59 < jnewbery> hugohn: blocks would still need to arrive in the correct order
14:00 < lightlike> isn't the arrive-in-order assumptions broken during IBD?
14:00 < ariard> lightlike: do you mean receiving block C before A and B ?
14:00 < jnewbery> because there's not really the concept of a 'from address' in a transaction, the wallet needs to know about all prior transactions to know if a new tx is spending from it
14:01 < lightlike> ariard: yes.
14:01 < jnewbery> I have to run now. Sorry we've run out of time.
14:01 < jkczyz> jnewbery: ah, I think the fUpdate flag threw me off
14:01 < jnewbery> Feel free to continue chatting in here.
14:02 < jnewbery> Thanks for everyone's input. Great discussion this week!
14:02 < lightlike> thanks!
14:02 < ariard> lightlike: it shouldn't if ActivateBestChain is right
14:02 < lightlike> ariard: you are right, I guess blocks can be downloaded out of order, but obviously not connected out of order
14:02 < nehan> jkczyz: what did you mean when you said that AddToWalletIfInvolving me handleds many of these cases? What cases does it handle?
14:03 < ariard> lightlike: exactly
14:05 < jkczyz> nehan: Possibly informed of the same block more than once (BlockConnected) and re-org (vis BlockDisconnected), though I'm not sure if the former happens in practice
14:06 < jonatack> ugh, lost connection for last 15
14:06 < hugohn> `bool fExisted = mapWallet.count(tx.GetHash()) != 0`
14:06 < hugohn> `if (fExisted && !fUpdate) return false;`
14:06 < hugohn> this logic^ should take care of redundant block?
14:07 < hugohn> AddToWalletIfInvolving bails if it's already seen the txn
14:09 < hugohn> unless it's a forced update (fUpdate = true when rescanning)
14:12 < hugohn> not sure of behavior when blocks are skipped