When a block is disconnected, update transactions that are no longer conflicted (wallet)

https://github.com/bitcoin/bitcoin/pull/27145

Host: josibake  -  PR author: ishaanam

The PR branch HEAD was 89df798 at the time of this review club meeting.

Notes

  • A transaction is considered conflicted when one or more of its inputs has been spent by another confirmed transaction. A conflicted transaction is marked with negative depth equal to the number of confirmations on the conflicting transaction.

  • It’s possible for a transaction to have previously been in a block that used to be part of the most-work chain but has since been reorged out.

  • The wallet keeps track of relevant transactions and their confirmation status. This information is used to calculate the wallet’s balance(s). For example:

    • If a transaction that is 100 blocks deep in the most-work chain, the wallet can reasonably include its UTXOs in the balance displayed to the user.

    • If a transaction conflicts with another transaction 100 blocks deep in the most-work chain, the wallet can be equally sure that, even though the transaction may have a valid signature, its UTXOs do not count towards the user’s balance.

    • If a transaction is unconfirmed and in the node’s mempool, the wallet should account for its UTXOs, but not consider them as safe as confirmed ones.

  • The author has provided more notes on transaction states and their effects on balance calculation here.

  • Wallet Transaction Conflict Tracking across chainstate and mempool events is tricky. As described in Issue #7315, when a block is disconnected, the wallet should be marking conflicted transactions as inactive, but isn’t currently doing so. PR #27145 updates the behavior to mark transactions that are no longer conflicting after a reorg as inactive.

Questions

  1. Did you review the PR? Concept ACK, approach ACK, tested ACK, or NACK? What was your review approach?

  2. What is the issue this PR addresses? Can you reproduce the problem described? (Hint: try running wallet_conflicts.py on master. What error do you get?)

  3. What are the different states a CWalletTx can be in?

  4. Which, if any, of the TxStates are “final,” i.e. once a transaction reaches this state, it will never change again?

  5. Where in net_processing.cpp is CWallet::blockDisconnected() triggered to be executed (Hint: it is not directly called. See CValidationInterface)? Which thread executes this wallet function?

  6. What does the wallet do with TxUpdate::NOTIFY_CHANGED? (Who is notifying whom, of what?)

  7. What does RecursiveUpdateTxState() do and why is it “recursive”? What are its callsites? (Before you grep, where do you think this function should be called?)

  8. What is tested in wallet_conflicts.py? Can you think of any other cases that should be tested?

Meeting Log

  117:00 <josie> #startmeeting
  217:00 <josie> hi!
  317:00 <abubakarsadiq> Hi
  417:00 <kevkevin> hi
  517:00 <glozow> hi
  617:00 <LarryRuane> hi
  717:00 <Pins> hi
  817:00 <ranemirus> hi
  917:00 <pablomartin> hello
 1017:00 <hernanmarino> Hi
 1117:00 <effexzi> Hi every1
 1217:01 <SebastianvStaa> hi
 1317:01 <josie> welcome to this week's PR review club. any first timers here?
 1417:01 <Pins> Me
 1517:01 <josie> Pins: welcome!
 1617:01 <Pins> Thanks!
 1717:01 <glozow> Pins: welcome!
 1817:02 <josie> just a general reminder: don't hesitate to ask a question or speak up (no need to ask if its okay to ask)
 1917:02 <josie> first question: did you get a chance to review the PR?
 2017:03 <josie> (can respond with a y/n)
 2117:03 <SebastianvStaa> y
 2217:03 <kevkevin> n :(
 2317:03 <abubakarsadiq> y
 2417:03 <hernanmarino> n, just lurking today
 2517:03 <pablomartin> same
 2617:03 <Pins> y
 2717:03 <josie> if you did review it, what was your approach? and what's your conclusion? concept/approach ack, or nack?
 2817:03 <josie> lurking is always fine :)
 2917:04 <hernanmarino> From a light reading , approach ACK .
 3017:05 <abubakarsadiq> Tested Ack I, run the test on the PR it passed, and also run the functional test on master to ensure the test fail.
 3117:05 <SebastianvStaa> same here
 3217:05 <ishaana> hi
 3317:05 <LarryRuane> abubakarsadiq: same
 3417:05 <josie> abubakarsadiq: nice! I was curious if anyone got a chance to run the test on master before the PR got merged
 3517:06 <stickies-v> (and if you didnt before it got merged, you still can, of course: https://github.com/bitcoin-core-review-club/website/pull/685#discussion_r1211727602)
 3617:07 <josie> stickies-v: thanks for the link!
 3717:07 <abubakarsadiq> Yeah, thats what I did, thanks for the link and help stickies-v
 3817:07 <josie> so question 2: what issue does this PR address? can you reproduce the problem?
 3917:08 <josie> (for those who ran the test on master pre-merge, this should be an easy question :D)
 4017:09 <abubakarsadiq> This PR address issue whereby if a block is disconnected, the state of all the transaction in the block that our node/wallet know will change to inactive and have 0 confirmations.
 4117:09 <josie> (also, forgot to post this at the beginning! we are discussing: https://bitcoincore.reviews/27145)
 4217:10 <SebastianvStaa> On master, formerly conflicted txns are not set from conflicted to inactive after reorg
 4317:10 <SebastianvStaa> (reorginging out of the conflicted txn)
 4417:13 <josie> SebastianvStaa: yep! before this PR, txs marked as conflicted were not being updated to the correct state of inactive when relevant
 4517:15 <josie> abubakarsadiq: I didn't actually verify what the behavior was before this PR, so you maybe correct. I'd say the main issue this PR attempts to address is *not* marking txs as inactive on blockDisconnect, when they should be marked inactive
 4617:15 <josie> unrelated question: did anyone get a chance to read the wiki or the gist linked in the notes? if not, I'd recommend it! I learned a lot about re-orgs and transaction states by reading them
 4717:16 <glozow> indeed! https://github.com/bitcoin-core/bitcoin-devwiki/wiki/Wallet-Transaction-Conflict-Tracking
 4817:16 <glozow> https://gist.github.com/ishaanam/846adf3b453c3a85fe6e15c882c57ae0#locations-where-tx-states-are-updated
 4917:17 <josie> okay, moving on to question 3: what are the different states a CWalletTx (a transaction known by the wallet) can be in?
 5017:17 <josie> glozow: thanks!
 5117:17 <abubakarsadiq> thanks josie
 5217:17 <SebastianvStaa> TxStateConfirmed: Contains corresponding block information
 5317:17 <SebastianvStaa> TxStateInMempool
 5417:17 <SebastianvStaa> TxStateConflicted: Contains corresponding conflicting block information
 5517:17 <SebastianvStaa> TxStateInactive: Can be abandoned or not
 5617:17 <SebastianvStaa> TxStateUnrecognized: Treated as inactive
 5717:18 <abubakarsadiq> Inactive, Confirmed, Conflicted, InMempool, and Unrecognized
 5817:19 <josie> SebastianvStaa, abubakarsadiq: yep! curious what you think about Unrecognized? I was trying to think of an example of an Unrecognized state and couldn't come up with one
 5917:20 <SebastianvStaa> josie don't know yet. Just looked up the state definitons in the code
 6017:20 <abubakarsadiq> Josie, Are they transaction whose inputs are known to the node?
 6117:21 <josie> yeah, same. Unrecognized stuck out to me tho, as it seems like this is an area we should be able to account for everything
 6217:21 <glozow> what does a tx state start as if you created but couldnt broadcast it (yet)?
 6317:22 <glozow> or would that never go to mapwallet?
 6417:22 <josie> abubakarsadiq: it's certainly possible to have external inputs in a tx, but I don't think this would apply here as the inputs would either be in a confirmed or unconfirmed state. if its confirmed the node definitely knows about it since it appears in a block
 6517:22 <ishaana> glozow: TxStateInactive
 6617:22 <glozow> ishaana: ah thanks
 6717:22 <ishaana> and I think it would be added to mapWallet
 6817:23 <josie> glozow, ishaana: interesting! my gut reaction was to say it wouldn't be added to mapWallet yet
 6917:24 <josie> seems like Unrecognized might be a better state than inactive for a tx yet to be broadcasted? anyways, don't want to go on too far a tangent
 7017:25 <josie> question 4: what transaction states are considered "final" ?
 7117:26 <josie> by final, we mean once a tx reaches this state it will never (or it is extremely unlikely) change states again
 7217:26 <SebastianvStaa> josie since there is only statistical finality in Bitcoin, probably no txns state is final forever
 7317:26 <abubakarsadiq> considering reorg happens i dont think there is
 7417:26 <SebastianvStaa> abubakarsadiq +1
 7517:27 <Pins> +1
 7617:27 <josie> SebastianvStaa, abubakarsadiq: reading the question from the notes, I agree :) which is why I added the "extremely unlikely" qualifier
 7717:27 <ishaana> glozow, josie: see https://github.com/bitcoin/bitcoin/blob/f08bde7f715cf84ef050c3f6902bc75fb90cedb3/src/wallet/wallet.cpp#L2314
 7817:29 <glozow> thanks for the link!
 7917:29 <ishaana> I don't think that an "abandoned" transaction can change tx state, but technically that would just be a TxStateInactive transaction
 8017:29 <josie> but I think we have to assume some notion of probabilistic finality, otherwise something like showing a wallet balance would be impossible
 8117:30 <josie> so given that we accept some notion of probabilistic finality, what states would you consider "final"/
 8217:30 <josie> ?*
 8317:30 <abubakarsadiq> I also noticed while running the test on master, that even though the conflicting transaction state does not change to inactive 0 confirmations it was accepted to the mempool and mined in the next block after rebroadcast. https://github.com/bitcoin/bitcoin/blob/f08bde7f715cf84ef050c3f6902bc75fb90cedb3/test/functional/wallet_conflicts.py#L119 this line passes when running test/functional/wallet_conflicts.py
 8417:30 <abubakarsadiq> on master before this pr was merged, just asking conceptually the real problem that will require the state to change to inactive
 8517:32 <SebastianvStaa> josie TxStateConfirmed could be considered final (unless reorg)
 8617:33 <josie> SebastianvStaa: correct! TxStateConfirmed is considered (increasingly) final the more confirmations it has. The PR specifically mentions 100 blocks as a number where a TxState is definitely considered final
 8717:34 <Pins> +1 (considering it is extremely unlikely the reorg)
 8817:34 <josie> any other states we might consider "final" (especially if we assume > 100 confirmations)
 8917:35 <wim96> spend?
 9017:35 <Pins> TxStateConflicted
 9117:35 <SebastianvStaa> yes, that is also the time span after which the coinbase transaction becomes spendable
 9217:35 <abubakarsadiq> +1 Josie
 9317:36 <josie> abubakarsadiq: I'm not sure I follow your question? The conflicted transaction shouldn't be mine-able as it would be spending inputs that are already confirmed spent in the longest chain
 9417:36 <josie> wim96: I would consider "spent" as TxConfirmed
 9517:37 <SebastianvStaa> josie maybe TxStateConflicted could be considered final after conflicting txn is mined 100 blocks deep
 9617:38 <josie> Pins, SebastianvStaa: ah! so it seems we agree that TxConflicted is not really a "final" state. based on this PR, what state does a TxConflicted tx get updated to? (e.g on blockDisconnect)
 9717:39 <abubakarsadiq> Josie the conflicting transaction's block was disconnected.
 9817:39 <Pins> TxStateInactive
 9917:40 <abubakarsadiq> TxStateInactive
10017:40 <SebastianvStaa> josie TxStateInactive
10117:40 <josie> Pins: yep! from my understanding of the PR, I'd say TxStateInactive is a "final" state for a tx in our wallet
10217:41 <Pins> Agreed
10317:41 <josie> so we have TxConfirmed (the tx in a block that is part of the heaviest chain), and TxInactive (txs that were at one point in a conflicting block, but that block is no longer part of the longest chain)
10417:42 <josie> okay, question 5: where in `net_processing.cpp` is `CWallet::blockDisconnected()` triggered to be executed?
10517:42 <josie> which thread executes this wallet function?
10617:43 <SebastianvStaa> josie: didn'T we state earlier that TxStateInactive is also the inital state for a txn after creation?
10717:43 <SebastianvStaa> [7:21:45 PM] <glozow> what does a tx state start as if you created but couldnt broadcast it (yet)?
10817:43 <SebastianvStaa> [7:22:13 PM] <glozow> or would that never go to mapwallet?
10917:43 <SebastianvStaa> [7:22:32 PM] <ishaana> glozow: TxStateInactive
11017:43 <SebastianvStaa> [7:22:55 PM] <glozow> ishaana: ah thanks
11117:44 <SebastianvStaa> so I don't see why this state could be considered 'final'
11217:45 <josie> SebastianvStaa: yep, which is a bit surprising to me, tbh. I suppose "final" almost always is accompanied with some notion of confirmations
11317:46 <SebastianvStaa> josie ok makes sense. the state by itself is not final, the txn need more properties for that
11417:47 <josie> yep! that's how I think about it. An Inactive tx which spends inputs in the UTXO set wouldn't be considered final
11517:47 <josie> but an Inactive tx which spends inputs that were spent many blocks back by a different transaction I would definitely consider to be in a final state
11617:50 <josie> feel free to throw out any ideas regarding question 5. I'll admit, I don't think I know the answer to this one
11717:50 <SebastianvStaa> I'm curious about question 5 as well, as I don't understand thread concurrency in Bitcoin Core (yet)
11817:50 <josie> (altho, I did learn that subscribers to the ValidationInterface can assume that events happen in a sequential order, which is kinda cool)
11917:53 <josie> ishaana: curious if you have any insights for question 5?
12017:54 <josie> SebastianvStaa: threading in bitcoin core is quite the beast! I'm still stumbling my way through understanding it
12117:54 <SebastianvStaa> this link on the topic seems quite cool. Currently reading it: https://diyhpl.us/wiki/transcripts/scalingbitcoin/tokyo-2018/edgedevplusplus/overview-bitcoin-core-architecture/
12217:54 <SebastianvStaa> josie +1
12317:55 <josie> let's move on to question 6: what does the wallet do with TxUpdate::NOTIFY_CHANGED?
12417:55 <yashraj> nice
12517:55 <josie> (who is notifying whom, of what?)
12617:56 <josie> SebastianvStaa: great link! thanks for sharing. https://obc.256k1.dev/ is also a great architecture overview, which might be slightly more up to date
12717:57 <SebastianvStaa> very nice resource! Thanks
12817:57 <lightlike> as for q5: looks like ActivateBestChain() is called in various places in net_processing, which can lead to DisconnectTip() in validation being called, which then creates the BlockDisconnected() signal which is picked up by the wallet later.
12917:58 <abubakarsadiq> Josie: After running the test on master before this pr was merged the error was
13017:58 <abubakarsadiq> `AssertionError: not(-15 == 0)` for both https://github.com/bitcoin/bitcoin/blob/f08bde7f715cf84ef050c3f6902bc75fb90cedb3/test/functional/wallet_conflicts.py#LL115C5-L115C5 and https://github.com/bitcoin/bitcoin/blob/f08bde7f715cf84ef050c3f6902bc75fb90cedb3/test/functional/wallet_conflicts.py#LL117C3-L117C3
13117:58 <abubakarsadiq> when I commented the two lines the test passes this>> https://github.com/bitcoin/bitcoin/blob/f08bde7f715cf84ef050c3f6902bc75fb90cedb3/test/functional/wallet_conflicts.py#LL119C1-L119C1, My question is that If the state of the conflicting transaction did not change before the PR from TxConflicted to TxInactive why does it this pass on master
13217:59 <josie> lightlike: thanks! I got as far as DisconnectTip() -> BlockDisconnected() but didn't realize it was originating with ActivateBestChain()
13318:00 <ishaana> josie: yeah it looks like it is called during ActivateBestChain()
13418:00 <josie> that's time, so we'll stop here, but I'd encourage everyone to work through the remaining questions!
13518:00 <josie> #endmeeting