#15931 Remove GetDepthInMainChain dependency on locked chain interface (wallet)
- This PR is the latest in a sequence of PRs to clean up the node-wallet interface.
- We previously reviewed PR 15713 in Bitcoin Core review club. See the notes and logs for that meeting for more information about the interface and the recent work to tidy it up.
- One of the main goals of that work is to remove the wallet’s ability to lock
cs_main. PR 16426 is a proof-of-concept PR which does that.
- This PR is a big step towards removing the wallet’s ability (and requirement)
cs_main. It removes the
locked_chaindependency from the
- For a given wallet transaction,
GetDepthInMainChain()returns how many confirmations that transaction has in the block chain.
- When a wallet transaction is included in a block, the block’s hash is stored
GetDepthInMainChain()previously worked by taking that
hashBlockand checking its depth in the block chain. That requires locking
cs_mainsince block chain state is being accessed.
- After this PR, each wallet transaction stores the height of the block that it was confirmed in, and the wallet stores the height of the best block in the block chain. By storing these values internally, the wallet no longer needs to query the block chain state to calculate the transaction’s number of confirmation.
- Part of this PR has been split off into a separate PR, wallet: encapsulate transactions state to make review easier. Reviewers should leave comments on that PR before reviewing this PR.
- An early version of this PR added an
m_block_heightfield the wallet transaction serialization (comment). Why wouldn’t this work?
- The PR author offers two ways for the wallet to populate the wallet transactions’ heights (save the transaction height to disk or calculate the height for each transactions at wallet load time). What are the trade-offs? Which approach do you prefer?
- How does the wallet learn about new transactions in the mempool or included in blocks?
- What are the wallet’s expectations about block notifications? Is it a problem if the wallet is informed of the same block more than once? If blocks arrive in the wrong order? If a block is skipped? If a block is re-orged out of the main chain?
13:00 < jnewbery> hi! 13:00 < digi_james> Hello! 13:00 < kanzure> hi 13:00 < emilengler> hi 13:00 < dergigi> hi 13:00 < ariard> hi 13:00 < lightlike> hello 13:00 < peevsie> hi 13:00 < jonatack> hi 13:01 < michaelfolkson> Hey 13:01 < jnewbery> jkcqyq dergigg: thanks for leaving review comments on the PR! 13:01 < jnewbery> sorry, jkczyz 13:02 < jnewbery> what did people think of the PR this week? 13:02 < jkczyz> hi 13:02 < jnewbery> perhaps jkczyz or dergigi (or anyone else) could give some initial thoughts about the PR 13:02 < nehan> hi 13:04 < jnewbery> I like this PR. It's a good next step to decoupling the wallet from the node 13:04 < dergigi> I don't have any specific questions unfortunately - just ran into the compilation error (it's the first PR I'm looking at, still trying to wrap my head around the code) 13:04 < jkczyz> Initial thoughts were having separate members for hash and height could be prone to bugs, but noticed there was a suggestion to combine 13:04 < jnewbery> dergigi: that's ok. Thanks for trying to test and leaving a comment - that's useful in itself 13:04 < jkczyz> did not have a chance to look at the PR that was made for that thought 13:04 < jkczyz> s/thought/though 13:05 < jnewbery> jkczyz: you mean #16624? 13:05 < jkczyz> yes 13:05 < jnewbery> https://github.com/bitcoin/bitcoin/pull/16624 wallet : encapsulate transactions state 13:05 < dergigi> I like the concept - seems like a good idea to store the tx block height internally so we don't have to look into the chain for every tx 13:06 < jnewbery> right - the PR author pulled out a small part of this PR and made it into its own PR. I'd already chosen 15931 for today's discussion and didn't want to change it in case you'd all started reviewing it 13:06 < michaelfolkson> It is certainly one that is interlocking with a bunch of other PRs and requires some organizational roadmap so they are merged in the right order. Seems like a high quality PR by itself. 13:07 < jnewbery> dergigi: yeah, it's good to reduce the wallet's reliance on locking the node's chain state 13:07 < nehan> i found it pretty difficult to chase the chain of PRs and understand everything together 13:07 < lightlike> I found this PR not so easy to review, but obiviously the concept of not having to query the block height makes a lot of sense. 13:07 < jnewbery> nehan: do you think you figured it out in the end? Any questions? 13:07 < jnewbery> lightlike: what made it difficult? Was the PR just too large? 13:08 < jnewbery> I think https://github.com/bitcoin/bitcoin/pull/16426 gives a good summary of what the end goal is 13:08 < nehan> jnewbery: no, but looking forward to discussing. the goal is great, but given i'm pretty unfamiliar with the wallet code i can't convince myself that these PRs don't change behavior (or that the behavior they change is 'safe') 13:08 < lightlike> jnewbery: It was large, but also some of the elements (the callback mechanism) were not so trivial to understand for me. 13:09 < jnewbery> "I can't convince myself that these PRs don't change behavior (or that the behavior they change is 'safe')" - I agree this is very difficult! 13:10 < jnewbery> What would you want to do to convince yourself of that? 13:12 < jnewbery> anyone? 13:12 < ariard> yes sorry for the big PR and splitting it between 15931 and 16624, it happened that wallet state transitions weren't clear for anyone 13:12 < jkczyz> Have test coverage for the behaviors 13:12 < jnewbery> jkczyz: yes please! 13:12 < lightlike> familiarize myself more with the wallet in general 13:13 < jnewbery> One thing that we're really missing is testing of old wallet.dat files 13:13 < jnewbery> There's an issue here: https://github.com/bitcoin/bitcoin/issues/14536 13:13 < jnewbery> I think it'd be really useful to have wallet files produced by lots of old versions of bitcoin core and tests for them 13:14 < ariard> jkczyz: have a look on 16624 listed issues in opening message, I think some behaviors aren't tested 13:14 < jnewbery> for PRs like 15931 and 16624, where there are changes in serialization (or at least in the way we deserialize and hold data at runtime), being able to do regression testing on old wallet files would be really useful 13:15 < jnewbery> lightlike: when you talk about 'callback mechanism', is that things like the BlockConnected/BlockDisconnected/TransactionAddedToMempool stuff? 13:16 < jnewbery> first question I had was: An early version of this PR added an m_block_height field the wallet transaction serialization (comment). Why wouldn’t this work? 13:16 < jnewbery> any thoughts about that? 13:16 < nehan> breaks compatibility with old wallet.dat files? 13:17 < jnewbery> nehan: yes exactly 13:18 < provoostenator> I have a PR that tests both forward and backward compatibility of wallets, though not ancient: https://github.com/bitcoin/bitcoin/pull/12134 13:18 < jnewbery> serialization is a bit tricky with the wallet. We store wallet objects in bdb, which is a key value store. 13:18 < lightlike> jnewbery: yes, that's what I meant. Not the changes done, in this PR, which are clear, but how the whole mechanism really works level. 13:18 < lightlike> *at a low level 13:18 < jnewbery> The serialization for each individual object is defined in the object declaration in the header file 13:19 < jnewbery> eg https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/wallet.h#L492 13:19 < jonatack> provoostenator: nice! 13:19 < jnewbery> but there's also a bunch of code in walletdb.cpp that fiddles around with that serialization 13:20 < jnewbery> here: https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/walletdb.cpp#L200 13:21 < jnewbery> serialization just serializes various data elements from the wallet in a specified order: https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/wallet.h#L505 13:21 < jnewbery> which makes it very difficult to version the serialization code 13:22 < jnewbery> ok, next question: The PR author offers two ways for the wallet to populate the wallet transactions’ heights (save the transaction height to disk or calculate the height for each transactions at wallet load time). What are the trade-offs? Which approach do you prefer? 13:22 < bcribles> you didn't miss any discussion while disconnected 13:23 < michaelfolkson> What version does it test back to <provoostenator> 13:23 < michaelfolkson> ? 13:23 < jnewbery> thanks bcribles! 13:24 < jonatack> jnewbery, ariard: how hard do you think it would be to clean up that serialization? 13:24 < jnewbery> jonatack: what do you mean by clean up? 13:25 < jonatack> untangle it to be easier to version 13:26 < dergigi> side note: i really like the tx status (suggested by ryanofsky) in 16624: unconfirmed/confirmed/conflicted/abandoned - the comments on that are very clear, good job on that ariard 13:26 < jnewbery> jonatack: quite difficult. We absolutely need to remain compatible with old wallet.dat files, and it's also important that new wallet.dat files are compatible with old versions, so we couldn't just change it completely 13:26 < lightlike> jnewbery: the serialization link https://github.com/bitcoin/bitcoin/pull/15931#discussion_r295434058 you gave pointed to CMerkleTx. Isn't that only for old wallet files, so why try to add block height there? Or did the change to CWalletTx happen only recently? 13:27 < ariard> could we version wallet.dat files in a backward compatible way ? 13:27 < jnewbery> if we were designing it from scratch, we might use a type-length-value scheme so wallet software could just ignore fields that it doesn't recognize, but we're constrained by all the existing wallet.dat files already in existence 13:28 < jnewbery> lightlike: the change to CWalletTx happened recently, resulting from a suggestion in this PR 13:28 < jnewbery> https://github.com/bitcoin/bitcoin/pull/16451 13:28 < lightlike> oh ok, then it makes sense. 13:29 < jnewbery> ariard: I'm not sure. Does the serialization for the wallet include the software version that wrote it? 13:29 < jnewbery> In any case, we'd need to leave the code for deserializing 'legacy' wallet.dat files in place for a long time 13:30 < fjahr> fg 13:31 < ariard> yes version with birthday would be cool, and pumping a message to user with which core version to use 13:31 < hugohn> brainstorming: perhaps one way to get around the lack of versioning in older Core versions is to write new stuff to a new .dat file? and start versioning in the new .dat file? 13:31 < hugohn> not sure if that's feasible 13:33 < ariard> let's say a new frontend serializer you may try to read first byte of files, if there is a magic number showing versioning support you use new serializer if not the old one 13:33 < jnewbery> lightlike: going back to your previous point: all of the wallet callbacks come from CValidationInterface functions. I think there are only 5 of those methods that are overridden by the wallet: TransactionAddedToMempool, TransactionRemovedFromMempool, BlockConnected, BlockDisconnected, UpdatedBlockTip, ChainStateFlushed 13:34 < jnewbery> it's worth looking at the NotificationsHandlerImpl in src/interfaces/chain and understanding how those interface methods are called 13:34 < ariard> jnewbery: it's a bit more complicated given we use Chain::Notifications as an adaptor between CValidationInterface and wallets 13:35 < jnewbery> ariard: that's right. It's a bit of indirection, but they're direct function calls from Chain::Notifications 13:35 < jnewbery> I haven't seen any answers to: The PR author offers two ways for the wallet to populate the wallet transactions’ heights (save the transaction height to disk or calculate the height for each transactions at wallet load time). What are the trade-offs? Which approach do you prefer? 13:35 < fjahr> sorry, wrong window :/ 13:36 < jnewbery> One potential disadvantage to calculating height at wallet load time is that it might impact performance for a large wallet 13:37 < jnewbery> ryanofsky thought it wouldn't be too much of a problem: https://github.com/bitcoin/bitcoin/pull/15931#issuecomment-519099417 13:38 < ariard> I think it's a performance hit at load time but a win at run time, you don't have to query chain anymore 13:38 < hugohn> saving tx height to disk seems to make more sense in the long-term, doesn't make sense to recompute the height each time the process starts, which doesn't change. 13:38 < jnewbery> ariard: I'm comparing doing the calculation at start up, or serializing it to disk. Both are a win at run time 13:39 < nehan> what is the likelihood that the heights have changed since the process was restarted? 13:39 < jkczyz> saving to disk would require a serialization change, right? 13:39 < jnewbery> jkczyz: exactly correct 13:39 < jnewbery> which we should be cautious about doing without thorough testing 13:40 < jnewbery> nehan: very slim, but possible. There would have to be a re-org over that block while the wallet was offline 13:41 < nehan> jnewbery: and if that happened, would there be a process to update the txn heights if they were loaded from disk? 13:41 < provoostenator> @michaelfolkson at the moment it tests back to 0.17.1, but more could be added. The main drawback is having too many switch statements in Python test framework to deal with ancient RPC stuff 13:42 < nehan> it seems to me it's not safe to rely on the serialized txn heights unless you're sure that on restart you'll get the appropriate notifications to update them (which you might. i'm not sure) 13:42 < michaelfolkson> <provoostenator>: So how far do you think it could be reasonably extended back to? 13:42 < bcribles> possibly because I haven't read enough of the PR but the malleability of using height caused when a reorg happens (like nehan was saying) is something I need to work through 13:42 < ariard> nehan: yes you need others change before, where the wallet send a block locator of its current highest tip 13:42 < jnewbery> nehan: yes, the wallet knows its own best block hash and tries to find the point that it diverges from the main chain, then rescans from there 13:42 < jnewbery> https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/wallet.cpp#L4504 13:42 < hugohn> is it possible to save the tx height to a different field in wallet.dat? 13:42 < ariard> and if this one is on a forked branch, a reorg should happen by replaying old blocks 13:42 < ariard> and connecting new ones 13:42 < hugohn> without affecting existing CWalletTx serialization 13:43 < jnewbery> ariard: that's correct 13:44 < nehan> ariard: jnewbery: thanks 13:44 < jnewbery> to add some more detail: the wallet has a 'locator', which is a sparse list of blockhashes in its view of the block chain. It uses that to try to find a fork point with the nodes view of the block chain at startup 13:45 < provoostenator> michaelfolkson: in theory back to when regtest was added, in practice you'll just have to try one by one 13:45 < jnewbery> https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/primitives/block.h#L122 13:45 < bcribles> jnewbery e.g. a wallet would be able to reason about if the block a txn was confirmed in has experienced a reorg? 13:46 < jnewbery> Locators are also used in GETBLOCKS and GETHEADERS messages on the P2P network: https://btcinformation.org/en/developer-reference#getblocks 13:46 < hugohn> to me the decision to persist to disk or not probably shouldn't impact how the wallet reacts to reorgs - one is a memory management issue, one is a consensus issue. reorgs could happen _after_ we have recomputed the height at startup anyway, so the issues should be completely orthogonal. but I could be wrong. 13:47 < jnewbery> bcribles: not directly, but it would know that there had been a re-org since it went offline, and therefore rescan the block chain from a height where it knows it shares history with the node 13:48 < jnewbery> if a re-org happened when the wallet is online, we'd expect the node to inform the wallet of the blocks being rewound with BlockDisconnected calls, followed by BlockConnected calls for the new chain 13:48 < nehan> hugohn: agreed! i just wanted to understand how invariants are maintained, and expectations on how a newly loaded wallet will be updated 13:48 < jnewbery> next question: How does the wallet learn about new transactions in the mempool or included in blocks? 13:49 < jnewbery> (should be easy, since we've already talked about it) 13:49 < hugohn> nehan: yeah, great question! 13:49 < hugohn> CValidationInterface? 13:49 < digi_james> ValidationInterface Callbacks 13:49 < jkczyz> It registers for notifications from the chain and implementing interfaces::Chain::Notifications 13:50 < jnewbery> good, you've all been paying attention :) 13:50 < jnewbery> ok, final question: What are the wallet’s expectations about block notifications? Is it a problem if the wallet is informed of the same block more than once? If blocks arrive in the wrong order? If a block is skipped? If a block is re-orged out of the main chain? 13:52 < jnewbery> ok, it's a difficult question to answer without reading a lot of code! 13:52 < jkczyz> It would seem that AddToWalletIfInvolvingMe handles many of these cases 13:53 < jnewbery> your entry points are those ValidationInterface methods 13:53 < jnewbery> jkczyz: exactly. AddToWalletIfInvolvingMe() is key 13:53 < jnewbery> https://github.com/bitcoin/bitcoin/blob/6dfa9efa3f558deaca0f01f673c79cce2b92a2b3/src/wallet/wallet.cpp#L1204 13:54 < jnewbery> how does a wallet know if a transaction is interesting to it? 13:54 < hugohn> by calling IsMine() 13:54 < hugohn> or IsFromMe() 13:55 < jnewbery> hugohn: yes, IsMine() will determine if that transaction sends money to the wallet 13:55 < jnewbery> hugohn: and yes, IsFromMe() determines if the transaction sends money from the wallet 13:56 < jnewbery> important to note: IsFromMe() can only work if the transaction input is spending from a transaction that the wallet already knows about 13:56 < hugohn> the legacy IsMine() logic is quite involving, it does pattern-matching and recursive lookups to determine if something belongs to the wallet 13:56 < jnewbery> say tx A sends money to a wallet, and tx B spends an output from tx A 13:57 < jkczyz> It seems odd that BlockDisconnected uses this with a null block hash / position. At very least it's hard to follow what the code is doing in this case given the reuse for adding blocks 13:57 < jnewbery> if the wallet doesn't know about tx A, then it has no way of knowing that tx B is a spend from the wallet 13:57 < nehan> jnewbery: it doesn't look like AddToWalletIfInvolvingMe() handles blocks in the wrong order 13:57 < jnewbery> nehan: right, the wallet's expectation is that blocks are served sequentially 13:58 < nehan> jnewbery: good that aligns with my understanding of CValidationInterface 13:58 < lightlike> A comment in ValidationInterface says that it is guaranteed that calls arrive in the same order as events are generated in validation - in that case, how is it possible that a block is skipped? 13:58 < jnewbery> jkczyz: that's needed so the wallet can change confirmed transactions to unconfirmed in the case of a reord 13:58 < jnewbery> lightlike: it shouldn't be possible 13:59 < hugohn> could the arrive-in-order assumption still hold true once we have node/wallet process separation? and block notifications served over IPC? 13:59 < jnewbery> however, I believe the wallet should be fine if it receives a block more than once 13:59 < nehan> jnewbery: re "What are the wallet’s expectations about block notifications?..." I think the wallet does not expect or handle any of that 13:59 < digi_james> hugohn: I had the same question 13:59 < nehan> jnewbery: oh 13:59 < jnewbery> hugohn: blocks would still need to arrive in the correct order 14:00 < lightlike> isn't the arrive-in-order assumptions broken during IBD? 14:00 < ariard> lightlike: do you mean receiving block C before A and B ? 14:00 < jnewbery> because there's not really the concept of a 'from address' in a transaction, the wallet needs to know about all prior transactions to know if a new tx is spending from it 14:01 < lightlike> ariard: yes. 14:01 < jnewbery> I have to run now. Sorry we've run out of time. 14:01 < jkczyz> jnewbery: ah, I think the fUpdate flag threw me off 14:01 < jnewbery> Feel free to continue chatting in here. 14:02 < jnewbery> Thanks for everyone's input. Great discussion this week! 14:02 < lightlike> thanks! 14:02 < ariard> lightlike: it shouldn't if ActivateBestChain is right 14:02 < lightlike> ariard: you are right, I guess blocks can be downloaded out of order, but obviously not connected out of order 14:02 < nehan> jkczyz: what did you mean when you said that AddToWalletIfInvolving me handleds many of these cases? What cases does it handle? 14:03 < ariard> lightlike: exactly 14:05 < jkczyz> nehan: Possibly informed of the same block more than once (BlockConnected) and re-org (vis BlockDisconnected), though I'm not sure if the former happens in practice 14:06 < jonatack> ugh, lost connection for last 15 14:06 < hugohn> `bool fExisted = mapWallet.count(tx.GetHash()) != 0` 14:06 < hugohn> `if (fExisted && !fUpdate) return false;` 14:06 < hugohn> this logic^ should take care of redundant block? 14:07 < hugohn> AddToWalletIfInvolving bails if it's already seen the txn 14:09 < hugohn> unless it's a forced update (fUpdate = true when rescanning) 14:12 < hugohn> not sure of behavior when blocks are skipped