Remove GetDepthInMainChain dependency on locked chain interface (wallet)

https://github.com/bitcoin/bitcoin/pull/15931

Host: jnewbery  -  PR author: ariard

Notes

  • This PR is the latest in a sequence of PRs to clean up the node-wallet interface.
  • We previously reviewed PR 15713 in Bitcoin Core review club. See the notes and logs for that meeting for more information about the interface and the recent work to tidy it up.
  • One of the main goals of that work is to remove the wallet’s ability to lock cs_main. PR 16426 is a proof-of-concept PR which does that.
  • This PR is a big step towards removing the wallet’s ability (and requirement) to lock cs_main. It removes the locked_chain dependency from the CWalletTx::GetDepthInMainChain() function.
  • For a given wallet transaction, GetDepthInMainChain() returns how many confirmations that transaction has in the block chain.
  • When a wallet transaction is included in a block, the block’s hash is stored in the CWalletTx object (see hashBlock and SetMerkleBranch().
  • GetDepthInMainChain() previously worked by taking that hashBlock and checking its depth in the block chain. That requires locking cs_main since block chain state is being accessed.
  • After this PR, each wallet transaction stores the height of the block that it was confirmed in, and the wallet stores the height of the best block in the block chain. By storing these values internally, the wallet no longer needs to query the block chain state to calculate the transaction’s number of confirmation.
  • Part of this PR has been split off into a separate PR, wallet: encapsulate transactions state to make review easier. Reviewers should leave comments on that PR before reviewing this PR.

Questions

  • An early version of this PR added an m_block_height field the wallet transaction serialization (comment). Why wouldn’t this work?
  • The PR author offers two ways for the wallet to populate the wallet transactions’ heights (save the transaction height to disk or calculate the height for each transactions at wallet load time). What are the trade-offs? Which approach do you prefer?
  • How does the wallet learn about new transactions in the mempool or included in blocks?
  • What are the wallet’s expectations about block notifications? Is it a problem if the wallet is informed of the same block more than once? If blocks arrive in the wrong order? If a block is skipped? If a block is re-orged out of the main chain?

Meeting Log

  113:00 <jnewbery> hi!
  213:00 <digi_james> Hello!
  313:00 <kanzure> hi
  413:00 <emilengler> hi
  513:00 <dergigi> hi
  613:00 <ariard> hi
  713:00 <lightlike> hello
  813:00 <peevsie> hi
  913:00 <jonatack> hi
 1013:01 <michaelfolkson> Hey
 1113:01 <jnewbery> jkcqyq dergigg: thanks for leaving review comments on the PR!
 1213:01 <jnewbery> sorry, jkczyz
 1313:02 <jnewbery> what did people think of the PR this week?
 1413:02 <jkczyz> hi
 1513:02 <jnewbery> perhaps jkczyz or dergigi (or anyone else) could give some initial thoughts about the PR
 1613:02 <nehan> hi
 1713:04 <jnewbery> I like this PR. It's a good next step to decoupling the wallet from the node
 1813:04 <dergigi> I don't have any specific questions unfortunately - just ran into the compilation error (it's the first PR I'm looking at, still trying to wrap my head around the code)
 1913:04 <jkczyz> Initial thoughts were having separate members for hash and height could be prone to bugs, but noticed there was a suggestion to combine
 2013:04 <jnewbery> dergigi: that's ok. Thanks for trying to test and leaving a comment - that's useful in itself
 2113:04 <jkczyz> did not have a chance to look at the PR that was made for that thought
 2213:04 <jkczyz> s/thought/though
 2313:05 <jnewbery> jkczyz: you mean #16624?
 2413:05 <jkczyz> yes
 2513:05 <jnewbery> https://github.com/bitcoin/bitcoin/pull/16624 wallet : encapsulate transactions state
 2613:05 <dergigi> I like the concept - seems like a good idea to store the tx block height internally so we don't have to look into the chain for every tx
 2713:06 <jnewbery> right - the PR author pulled out a small part of this PR and made it into its own PR. I'd already chosen 15931 for today's discussion and didn't want to change it in case you'd all started reviewing it
 2813:06 <michaelfolkson> It is certainly one that is interlocking with a bunch of other PRs and requires some organizational roadmap so they are merged in the right order. Seems like a high quality PR by itself.
 2913:07 <jnewbery> dergigi: yeah, it's good to reduce the wallet's reliance on locking the node's chain state
 3013:07 <nehan> i found it pretty difficult to chase the chain of PRs and understand everything together
 3113:07 <lightlike> I found this PR not so easy to review, but obiviously the concept of not having to query the block height makes a lot of sense.
 3213:07 <jnewbery> nehan: do you think you figured it out in the end? Any questions?
 3313:07 <jnewbery> lightlike: what made it difficult? Was the PR just too large?
 3413:08 <jnewbery> I think https://github.com/bitcoin/bitcoin/pull/16426 gives a good summary of what the end goal is
 3513:08 <nehan> jnewbery: no, but looking forward to discussing. the goal is great, but given i'm pretty unfamiliar with the wallet code i can't convince myself that these PRs don't change behavior (or that the behavior they change is 'safe')
 3613:08 <lightlike> jnewbery: It was large, but also some of the elements (the callback mechanism) were not so trivial to understand for me.
 3713:09 <jnewbery> "I can't convince myself that these PRs don't change behavior (or that the behavior they change is 'safe')" - I agree this is very difficult!
 3813:10 <jnewbery> What would you want to do to convince yourself of that?
 3913:12 <jnewbery> anyone?
 4013:12 <ariard> yes sorry for the big PR and splitting it between 15931 and 16624, it happened that wallet state transitions weren't clear for anyone
 4113:12 <jkczyz> Have test coverage for the behaviors
 4213:12 <jnewbery> jkczyz: yes please!
 4313:12 <lightlike> familiarize myself more with the wallet in general
 4413:13 <jnewbery> One thing that we're really missing is testing of old wallet.dat files
 4513:13 <jnewbery> There's an issue here: https://github.com/bitcoin/bitcoin/issues/14536
 4613:13 <jnewbery> I think it'd be really useful to have wallet files produced by lots of old versions of bitcoin core and tests for them
 4713:14 <ariard> jkczyz: have a look on 16624 listed issues in opening message, I think some behaviors aren't tested
 4813:14 <jnewbery> for PRs like 15931 and 16624, where there are changes in serialization (or at least in the way we deserialize and hold data at runtime), being able to do regression testing on old wallet files would be really useful
 4913:15 <jnewbery> lightlike: when you talk about 'callback mechanism', is that things like the BlockConnected/BlockDisconnected/TransactionAddedToMempool stuff?
 5013:16 <jnewbery> first question I had was: An early version of this PR added an m_block_height field the wallet transaction serialization (comment). Why wouldn’t this work?
 5113:16 <jnewbery> any thoughts about that?
 5213:16 <nehan> breaks compatibility with old wallet.dat files?
 5313:17 <jnewbery> nehan: yes exactly
 5413:18 <provoostenator> I have a PR that tests both forward and backward compatibility of wallets, though not ancient: https://github.com/bitcoin/bitcoin/pull/12134
 5513:18 <jnewbery> serialization is a bit tricky with the wallet. We store wallet objects in bdb, which is a key value store.
 5613:18 <lightlike> jnewbery: yes, that's what I meant. Not the changes done, in this PR, which are clear, but how the whole mechanism really works level.
 5713:18 <lightlike> *at a low level
 5813:18 <jnewbery> The serialization for each individual object is defined in the object declaration in the header file
 5913:19 <jnewbery> eg https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/wallet.h#L492
 6013:19 <jonatack> provoostenator: nice!
 6113:19 <jnewbery> but there's also a bunch of code in walletdb.cpp that fiddles around with that serialization
 6213:20 <jnewbery> here: https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/walletdb.cpp#L200
 6313:21 <jnewbery> serialization just serializes various data elements from the wallet in a specified order: https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/wallet.h#L505
 6413:21 <jnewbery> which makes it very difficult to version the serialization code
 6513:22 <jnewbery> ok, next question: The PR author offers two ways for the wallet to populate the wallet transactions’ heights (save the transaction height to disk or calculate the height for each transactions at wallet load time). What are the trade-offs? Which approach do you prefer?
 6613:22 <bcribles> you didn't miss any discussion while disconnected
 6713:23 <michaelfolkson> What version does it test back to <provoostenator>
 6813:23 <michaelfolkson> ?
 6913:23 <jnewbery> thanks bcribles!
 7013:24 <jonatack> jnewbery, ariard: how hard do you think it would be to clean up that serialization?
 7113:24 <jnewbery> jonatack: what do you mean by clean up?
 7213:25 <jonatack> untangle it to be easier to version
 7313:26 <dergigi> side note: i really like the tx status (suggested by ryanofsky) in 16624: unconfirmed/confirmed/conflicted/abandoned - the comments on that are very clear, good job on that ariard
 7413:26 <jnewbery> jonatack: quite difficult. We absolutely need to remain compatible with old wallet.dat files, and it's also important that new wallet.dat files are compatible with old versions, so we couldn't just change it completely
 7513:26 <lightlike> jnewbery: the serialization link https://github.com/bitcoin/bitcoin/pull/15931#discussion_r295434058 you gave pointed to CMerkleTx. Isn't that only for old wallet files, so why try to add block height there? Or did the change to CWalletTx happen only recently?
 7613:27 <ariard> could we version wallet.dat files in a backward compatible way ?
 7713:27 <jnewbery> if we were designing it from scratch, we might use a type-length-value scheme so wallet software could just ignore fields that it doesn't recognize, but we're constrained by all the existing wallet.dat files already in existence
 7813:28 <jnewbery> lightlike: the change to CWalletTx happened recently, resulting from a suggestion in this PR
 7913:28 <jnewbery> https://github.com/bitcoin/bitcoin/pull/16451
 8013:28 <lightlike> oh ok, then it makes sense.
 8113:29 <jnewbery> ariard: I'm not sure. Does the serialization for the wallet include the software version that wrote it?
 8213:29 <jnewbery> In any case, we'd need to leave the code for deserializing 'legacy' wallet.dat files in place for a long time
 8313:30 <fjahr> fg
 8413:31 <ariard> yes version with birthday would be cool, and pumping a message to user with which core version to use
 8513:31 <hugohn> brainstorming: perhaps one way to get around the lack of versioning in older Core versions is to write new stuff to a new .dat file? and start versioning in the new .dat file?
 8613:31 <hugohn> not sure if that's feasible
 8713:33 <ariard> let's say a new frontend serializer you may try to read first byte of files, if there is a magic number showing versioning support you use new serializer if not the old one
 8813:33 <jnewbery> lightlike: going back to your previous point: all of the wallet callbacks come from CValidationInterface functions. I think there are only 5 of those methods that are overridden by the wallet: TransactionAddedToMempool, TransactionRemovedFromMempool, BlockConnected, BlockDisconnected, UpdatedBlockTip, ChainStateFlushed
 8913:34 <jnewbery> it's worth looking at the NotificationsHandlerImpl in src/interfaces/chain and understanding how those interface methods are called
 9013:34 <ariard> jnewbery: it's a bit more complicated given we use Chain::Notifications as an adaptor between CValidationInterface and wallets
 9113:35 <jnewbery> ariard: that's right. It's a bit of indirection, but they're direct function calls from Chain::Notifications
 9213:35 <jnewbery> I haven't seen any answers to: The PR author offers two ways for the wallet to populate the wallet transactions’ heights (save the transaction height to disk or calculate the height for each transactions at wallet load time). What are the trade-offs? Which approach do you prefer?
 9313:35 <fjahr> sorry, wrong window :/
 9413:36 <jnewbery> One potential disadvantage to calculating height at wallet load time is that it might impact performance for a large wallet
 9513:37 <jnewbery> ryanofsky thought it wouldn't be too much of a problem: https://github.com/bitcoin/bitcoin/pull/15931#issuecomment-519099417
 9613:38 <ariard> I think it's a performance hit at load time but a win at run time, you don't have to query chain anymore
 9713:38 <hugohn> saving tx height to disk seems to make more sense in the long-term, doesn't make sense to recompute the height each time the process starts, which doesn't change.
 9813:38 <jnewbery> ariard: I'm comparing doing the calculation at start up, or serializing it to disk. Both are a win at run time
 9913:39 <nehan> what is the likelihood that the heights have changed since the process was restarted?
10013:39 <jkczyz> saving to disk would require a serialization change, right?
10113:39 <jnewbery> jkczyz: exactly correct
10213:39 <jnewbery> which we should be cautious about doing without thorough testing
10313:40 <jnewbery> nehan: very slim, but possible. There would have to be a re-org over that block while the wallet was offline
10413:41 <nehan> jnewbery: and if that happened, would there be a process to update the txn heights if they were loaded from disk?
10513:41 <provoostenator> @michaelfolkson at the moment it tests back to 0.17.1, but more could be added. The main drawback is having too many switch statements in Python test framework to deal with ancient RPC stuff
10613:42 <nehan> it seems to me it's not safe to rely on the serialized txn heights unless you're sure that on restart you'll get the appropriate notifications to update them (which you might. i'm not sure)
10713:42 <michaelfolkson> <provoostenator>: So how far do you think it could be reasonably extended back to?
10813:42 <bcribles> possibly because I haven't read enough of the PR but the malleability of using height caused when a reorg happens (like nehan was saying) is something I need to work through
10913:42 <ariard> nehan: yes you need others change before, where the wallet send a block locator of its current highest tip
11013:42 <jnewbery> nehan: yes, the wallet knows its own best block hash and tries to find the point that it diverges from the main chain, then rescans from there
11113:42 <jnewbery> https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/wallet.cpp#L4504
11213:42 <hugohn> is it possible to save the tx height to a different field in wallet.dat?
11313:42 <ariard> and if this one is on a forked branch, a reorg should happen by replaying old blocks
11413:42 <ariard> and connecting new ones
11513:42 <hugohn> without affecting existing CWalletTx serialization
11613:43 <jnewbery> ariard: that's correct
11713:44 <nehan> ariard: jnewbery: thanks
11813:44 <jnewbery> to add some more detail: the wallet has a 'locator', which is a sparse list of blockhashes in its view of the block chain. It uses that to try to find a fork point with the nodes view of the block chain at startup
11913:45 <provoostenator> michaelfolkson: in theory back to when regtest was added, in practice you'll just have to try one by one
12013:45 <jnewbery> https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/primitives/block.h#L122
12113:45 <bcribles> jnewbery e.g. a wallet would be able to reason about if the block a txn was confirmed in has experienced a reorg?
12213:46 <jnewbery> Locators are also used in GETBLOCKS and GETHEADERS messages on the P2P network: https://btcinformation.org/en/developer-reference#getblocks
12313:46 <hugohn> to me the decision to persist to disk or not probably shouldn't impact how the wallet reacts to reorgs - one is a memory management issue, one is a consensus issue. reorgs could happen _after_ we have recomputed the height at startup anyway, so the issues should be completely orthogonal. but I could be wrong.
12413:47 <jnewbery> bcribles: not directly, but it would know that there had been a re-org since it went offline, and therefore rescan the block chain from a height where it knows it shares history with the node
12513:48 <jnewbery> if a re-org happened when the wallet is online, we'd expect the node to inform the wallet of the blocks being rewound with BlockDisconnected calls, followed by BlockConnected calls for the new chain
12613:48 <nehan> hugohn: agreed! i just wanted to understand how invariants are maintained, and expectations on how a newly loaded wallet will be updated
12713:48 <jnewbery> next question: How does the wallet learn about new transactions in the mempool or included in blocks?
12813:49 <jnewbery> (should be easy, since we've already talked about it)
12913:49 <hugohn> nehan: yeah, great question!
13013:49 <hugohn> CValidationInterface?
13113:49 <digi_james> ValidationInterface Callbacks
13213:49 <jkczyz> It registers for notifications from the chain and implementing interfaces::Chain::Notifications
13313:50 <jnewbery> good, you've all been paying attention :)
13413:50 <jnewbery> ok, final question: What are the wallet’s expectations about block notifications? Is it a problem if the wallet is informed of the same block more than once? If blocks arrive in the wrong order? If a block is skipped? If a block is re-orged out of the main chain?
13513:52 <jnewbery> ok, it's a difficult question to answer without reading a lot of code!
13613:52 <jkczyz> It would seem that AddToWalletIfInvolvingMe handles many of these cases
13713:53 <jnewbery> your entry points are those ValidationInterface methods
13813:53 <jnewbery> jkczyz: exactly. AddToWalletIfInvolvingMe() is key
13913:53 <jnewbery> https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/wallet.cpp#L1204
14013:54 <jnewbery> how does a wallet know if a transaction is interesting to it?
14113:54 <hugohn> by calling IsMine()
14213:54 <hugohn> or IsFromMe()
14313:55 <jnewbery> hugohn: yes, IsMine() will determine if that transaction sends money to the wallet
14413:55 <jnewbery> hugohn: and yes, IsFromMe() determines if the transaction sends money from the wallet
14513:56 <jnewbery> important to note: IsFromMe() can only work if the transaction input is spending from a transaction that the wallet already knows about
14613:56 <hugohn> the legacy IsMine() logic is quite involving, it does pattern-matching and recursive lookups to determine if something belongs to the wallet
14713:56 <jnewbery> say tx A sends money to a wallet, and tx B spends an output from tx A
14813:57 <jkczyz> It seems odd that BlockDisconnected uses this with a null block hash / position. At very least it's hard to follow what the code is doing in this case given the reuse for adding blocks
14913:57 <jnewbery> if the wallet doesn't know about tx A, then it has no way of knowing that tx B is a spend from the wallet
15013:57 <nehan> jnewbery: it doesn't look like AddToWalletIfInvolvingMe() handles blocks in the wrong order
15113:57 <jnewbery> nehan: right, the wallet's expectation is that blocks are served sequentially
15213:58 <nehan> jnewbery: good that aligns with my understanding of CValidationInterface
15313:58 <lightlike> A comment in ValidationInterface says that it is guaranteed that calls arrive in the same order as events are generated in validation - in that case, how is it possible that a block is skipped?
15413:58 <jnewbery> jkczyz: that's needed so the wallet can change confirmed transactions to unconfirmed in the case of a reord
15513:58 <jnewbery> lightlike: it shouldn't be possible
15613:59 <hugohn> could the arrive-in-order assumption still hold true once we have node/wallet process separation? and block notifications served over IPC?
15713:59 <jnewbery> however, I believe the wallet should be fine if it receives a block more than once
15813:59 <nehan> jnewbery: re "What are the wallet’s expectations about block notifications?..." I think the wallet does not expect or handle any of that
15913:59 <digi_james> hugohn: I had the same question
16013:59 <nehan> jnewbery: oh
16113:59 <jnewbery> hugohn: blocks would still need to arrive in the correct order
16214:00 <lightlike> isn't the arrive-in-order assumptions broken during IBD?
16314:00 <ariard> lightlike: do you mean receiving block C before A and B ?
16414:00 <jnewbery> because there's not really the concept of a 'from address' in a transaction, the wallet needs to know about all prior transactions to know if a new tx is spending from it
16514:01 <lightlike> ariard: yes.
16614:01 <jnewbery> I have to run now. Sorry we've run out of time.
16714:01 <jkczyz> jnewbery: ah, I think the fUpdate flag threw me off
16814:01 <jnewbery> Feel free to continue chatting in here.
16914:02 <jnewbery> Thanks for everyone's input. Great discussion this week!
17014:02 <lightlike> thanks!
17114:02 <ariard> lightlike: it shouldn't if ActivateBestChain is right
17214:02 <lightlike> ariard: you are right, I guess blocks can be downloaded out of order, but obviously not connected out of order
17314:02 <nehan> jkczyz: what did you mean when you said that AddToWalletIfInvolving me handleds many of these cases? What cases does it handle?
17414:03 <ariard> lightlike: exactly
17514:05 <jkczyz> nehan: Possibly informed of the same block more than once (BlockConnected) and re-org (vis BlockDisconnected), though I'm not sure if the former happens in practice
17614:06 <jonatack> ugh, lost connection for last 15
17714:06 <hugohn> `bool fExisted = mapWallet.count(tx.GetHash()) != 0`
17814:06 <hugohn> `if (fExisted && !fUpdate) return false;`
17914:06 <hugohn> this logic^ should take care of redundant block?
18014:07 <hugohn> AddToWalletIfInvolving bails if it's already seen the txn
18114:09 <hugohn> unless it's a forced update (fUpdate = true when rescanning)
18214:12 <hugohn> not sure of behavior when blocks are skipped