Only support compact blocks with witnesses (p2p)

https://github.com/bitcoin/bitcoin/pulls/20799

Host: jnewbery  -  PR author: jnewbery

The PR branch HEAD was f8ff628 at the time of this review club meeting.

Notes

  • Compact blocks are a way to relay blocks across the Bitcoin P2P network with reduced bandwidth usage. They can also reduce block propagation latency when used in BIP152 high-bandwidth mode. BIP152 is the specification for compact blocks.

  • BIP152 was originally developed and deployed before segwit was activated; Bitcoin Core’s implementation was merged in PR 8068 and included in the v0.13.0 release. The specification was made extensible so that compact blocks could be used for non-witness serialized blocks (version 1) and witness serialized blocks (version 2).

  • Segwit (BIP141) was activated in August 2017. Every version of Bitcoin Core since v0.13.1 supports segwit and will fully validate blocks according to the BIP141 consensus rules. To fully validate a block with segwit transactions, it must be serialized with witnesses.

  • Version 1 (non-witness) compact blocks are therefore no longer useful to peers on the network. This PR removes support for serving version 1 compact blocks.

Questions

  1. How does using compact blocks save bandwidth? What data is not downloaded when relaying blocks using compact blocks?

  2. What is BIP152 high-bandwidth mode? How many of our peers can we choose to be high-bandwidth peers?

  3. How do we choose which peers should be high-bandwidth peers? In which function does that logic exist?

  4. If a peer chooses us to be its high-bandwidth peer, how does it signal that to us?

  5. BIP152 states: “high-bandwidth mode permits relaying of CMPCTBLOCK messages prior to full validation (requiring only that the block header is valid before relay).” In which PeerManager function do we relay compact blocks to peers that have chosen us to be a high-bandwidth peer?

  6. How is that PeerManager function invoked? In which thread is it called?

Meeting Log

  118:00 <jnewbery> #startmeeting
  218:00 <emzy> hi
  318:00 <jnewbery> Hi folks. Welcome to the first Bitcoin Core PR Review Club of 2021 🎉
  418:00 <jarolrod> hi
  518:00 <schmidty> Hi!
  618:00 <troygior1hev> hi
  718:00 <ccdle12> hi
  818:00 <Legousk> Hello!
  918:00 <lightlike> hi
 1018:00 <theStack> hi
 1118:00 <andozw> hiiii
 1218:00 <elle> hi!
 1318:00 <joelklabo> good morning 👋
 1418:00 <anir> hey everyone!
 1518:00 <AnthonyRonning> Hi all!
 1618:00 <amiti> hello!
 1718:00 <felixweis> hi!
 1818:00 <larryruane_> hi!
 1918:00 <jnewbery> I hope you all had a good break, and are feeling ready to learn even more about Bitcoin Core this year :)
 2018:00 <jnewbery> feel free to say hi to let everyone know you're here
 2118:00 <sunon> Hey!! Elle yay!
 2218:01 <norisg> Hi !
 2318:01 <fodediop> hi
 2418:01 <willcl_ark> hi
 2518:01 <jnewbery> Notes and questions are in the normal place: https://bitcoincore.reviews/20799
 2618:01 <elle> hey sunon! :)
 2718:01 <glozow> hi!
 2818:01 <michaelfolkson> hi
 2918:01 <jnewbery> Is anyone here for the first time?
 3018:01 <olympics> hi
 3118:01 <AnthonyRonning> first time here
 3218:01 <pinheadmz> hi
 3318:01 <jnewbery> Welcome AnthonyRonning! Thanks for joining us
 3418:01 <keyboardwarrior> hi :)
 3518:01 <sunon> Trying via a matrix.org proxy for the first time. Let’s see how this goes
 3618:01 <fodediop> Welcome Anthony! 🎉
 3718:01 <olympics> welcome AnthonyRonning
 3818:02 <AnthonyRonning> 🙌
 3918:02 <Caralie> hi :) Welcome Anthony!
 4018:02 <thomasb06> hi
 4118:02 <jnewbery> A couple of reminders on the format: i have some prepared questions to guide the conversation, but feel free to jump in at any time. Don't worry if you have a question about something that we covered earlier or whatever.
 4218:03 <jnewbery> also, don't ask to ask. Just jump right in.
 4318:03 <jnewbery> we're all here to learn and all questions are welcome
 4418:03 <jnewbery> ok, onto the PR. Who had a chance to review it? (y/n)
 4518:03 <sunon> n
 4618:03 <joelklabo> y
 4718:03 <Legousk> n
 4818:03 <AnthonyRonning> y
 4918:03 <ccdle12> y
 5018:03 <troygior1hev> n
 5118:03 <fodediop> y
 5218:03 <norisg> n
 5318:03 <felixweis> n
 5418:03 <thomasb06> y
 5518:03 <willcl_ark> n
 5618:03 <pinheadmz> 1/2
 5718:03 <theStack> 0.5 y (did just skim over the commits w/o detailled code review)
 5818:04 <emzy> y/n
 5918:04 <Legousk> Where can I see beforehand which one will be reviewed?
 6018:04 <jnewbery> nice. No problem if you didn't have time this week
 6118:04 <elle> 0.5 y (spent most time on context)
 6218:04 <glozow> y-ish
 6318:04 <jarolrod> .5y
 6418:04 <lightlike> 1/2
 6518:04 <jnewbery> Legousk: they're announced on the website, usually the Friday before.
 6618:04 <olympics> Legousk https://bitcoincore.reviews/
 6718:04 <jnewbery> any initial thoughts about the PR? Concept ACK/NACKs?
 6818:04 <anir> 1.2
 6918:04 <anir> 1/2*
 7018:05 <AnthonyRonning> concept ack, love removing code
 7118:05 <nehan> hi
 7218:05 <sunon> Concept ACK
 7318:05 <jnewbery> YES! Removing code is the best!
 7418:05 <emzy> concept ack, not needed anymore.
 7518:05 <pinheadmz> ack yes
 7618:05 <jarolrod> concept ack makes sense
 7718:05 <thomasb06> not acquainted with the code enough
 7818:05 <Legousk> Thanks
 7918:05 <joelklabo> ack 👍
 8018:05 <theStack> concept ack!
 8118:05 <michaelfolkson> Concept ACK but remind me how many old versions it is Core policy to support?
 8218:05 <anir> Concept ack, streamlines the codebase
 8318:06 <jonatack> hi
 8418:06 <fodediop> Concept but still trying to digest the codebase
 8518:06 <fodediop> Ack
 8618:06 <michaelfolkson> 0.13 is pretty old
 8718:06 <jnewbery> michaelfolkson: that's a very broad question. There are lots of different versionings (eg p2p protocol, wallet, file format, ...)
 8818:06 <jnewbery> First question. How does using compact blocks save bandwidth? What data is not downloaded when relaying blocks using compact blocks?
 8918:07 <ccdle12> the transactions aren't downloaded? it
 9018:07 <jarolrod> Compact Blocks only includes a short transaction id for each transaction contained within it. The node on the receiving end of a compact block must match a transaction id with a transaction in its mempool and reconstruct the full block. Since we are not sending all of the transaction data, we can reduce block relay bandwidth around ~90%
 9118:07 <sunon> Most of the actual transactions
 9218:07 <pinheadmz> full transaction data is not downloaded, just short IDs
 9318:07 <jnewbery> The BIP is here: https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki
 9418:07 <pinheadmz> in hopes the recipient has the tx data already
 9518:07 <joelklabo> transactions are assumed to be in the mempool already
 9618:07 <jonatack> Legousk: if you're on twitter, you can get updates by following https://twitter.com/BitcoinCorePRs
 9718:07 <michaelfolkson> jnewbery: For this type of p2p change how many old versions should be supported?
 9818:07 <thomasb06> only the transactions missing are transmitted
 9918:07 <thomasb06> the block isn't
10018:07 <emzy> We only send the block header and short IDs of the transactions. If the peer has all the transactions in the mempool it can reconstruct the block without additional data.
10118:07 ℹ sdaftuar_ is now known as sdaftuar
10218:08 <sunon> Some transaction predicted to not be known to receiver are transmitted too
10318:08 <glozow> PrefilledTransactions for txns we expect the peer to be missing, e.g. the coinbase tx
10418:08 <pinheadmz> michaelfolkson i think an important point is that no one will ever need version 1 compact blocks because the last non witness block was so long ago, and compacts only work for recent blocks
10518:08 <Legousk> jonatack: I'm not on twitter, but will follow it tho
10618:08 <emzy> pinheadmz: exactly.
10718:08 <pinheadmz> even non upgraded nodes (non segwit) it wont affect them, they arent getting v1 compacts any more anyway
10818:08 <jnewbery> Lots of great answers there. Yes, when the node sends a CMPCTBLOCK it'll include shortids for the transactions, and some prefilled transactions, including the coinbase
10918:08 <michaelfolkson> pinheadmz: If you are running 0.13 and want to use compact blocks?
11018:08 <pinheadmz> i still wont send them to you
11118:08 <pinheadmz> the last non witness block was SO LONG ago, you definitely dont have those txs in your mempool :-)
11218:09 <pinheadmz> and having tx in the mempool is required for compact blocks to be useful
11318:09 <jnewbery> michaelfolkson: if you're running 0.13.0, then you're not enforcing all consensus rules
11418:09 <sdaftuar> pinheadmz: that's not exactly right
11518:09 <sipa> michaelfolkson: if you're running 0.13.0, yes; not 0.12.x (because it has no compact blocks), and not 0.13.1+ (because it has segwit activated)
11618:10 <sdaftuar> pinheadmz: currently an 0.13.0 node could attempt to use compact block relay, and it would get short-ids based on txid rather than wtxid
11718:10 <sunon> So it’s really only very specifically 0.13.0 that’s the weird case?
11818:10 <olympics> so it only affects node IFF they run 0.13.0
11918:10 <olympics> ?
12018:10 <sipa> olympics: i believe so
12118:10 <sdaftuar> pinheadmz: but you are right that reconstruction would always require a roundtrip because those witness transactions would be nonstadnard to an 0.13.0 node
12218:10 <sdaftuar> however, the protocol would work
12318:10 <sipa> (or any other codebase that behaves similarly)
12418:10 <pinheadmz> sdaftuar i see ok thanks
12518:10 <jnewbery> or theoretically some other implementation that implemented BIP 152 but not segwit
12618:10 <jnewbery> maybe some old version of an alternative implementation
12718:11 <sunon> Oh yes of course
12818:11 <norisg> did the block structure also change with segwit, you are talking about "non witness blocks", non witness blocks have no segwit transactions in it?
12918:11 <pinheadmz> sdaftuar non standard TXs require a roundtrip? even if the tx is in the mempool (by txid, not wtxid) ?
13018:11 <sdaftuar> michaelfolkson: importantly, this doesn't fundamentally break blockrelay to 0.13.0 nodes -- just won't use compact blocks for it
13118:11 <sdaftuar> so to your question about "supporting" older nodes, this change wouldn't cause 0.13.0 nodes to fall out of consensus
13218:11 <jnewbery> norisg: yes, the serialization of blocks changed with segwit. See BIP 144 for details
13318:12 <larryruane_> so a segwit block could happen to have no segwit transactions?
13418:12 <olympics> sdaftuar the worst effect would be, marginally longer txn propagation to nodes on 0.13.0?
13518:12 <sdaftuar> pinheadmz: well non-standard transactions are (by definition) not in the mempool -- so reconstruction would fail
13618:12 <sdaftuar> olympics: yep
13718:12 <olympics> ty
13818:12 <sdaftuar> block propagation
13918:12 <sipa> larryruane_: then by definition it is not a segwit block :)
14018:12 <pinheadmz> sdaftuar ah yes yes ty
14118:13 <sdaftuar> pinheadmz: (well we also have the extra pool, too -- but that is relatively small)
14218:13 <olympics> sdaftuar should I have said block propagation not txn propagation ?
14318:13 <sdaftuar> olympics: yes
14418:14 <olympics> bc the shortID expedites relaying the txns in the block rather than shipping out txns the peer may already have
14518:14 <jnewbery> Next question: What is BIP152 high-bandwidth mode? How many of our peers can we choose to be high-bandwidth peers?
14618:14 <ccdle12> 3 peers as high bandwith, high-bandwith means the nodes send compacted blocks by default
14718:15 <emzy> We send the compact block without the peer asking for it, also before we validated it ourself. Max is 3 peers.
14818:15 <thomasb06> 2 max
14918:15 <anir> peers send new block announcements with the short transaction IDs already via a cmpctblock message, and it’s enabled by setting the first boolean to 1 in a sendcmpct message
15018:15 <AnthonyRonning> High bandwidth mode is available so that peers can receive blocks automatically and asap (`1.5 * RTT`), even before full block validation takes place.
15118:15 <sunon> High bandwidth mode allows peers to send compact blocks without sending inv msg / asking permission
15218:15 <willcl_ark> look like 3 peers
15318:15 <jnewbery> olympics: tx relay (for unconfirmed transactions) is unaffected by this. compact blocks makes block propogation faster based on the observation that the receiver probably has most of the transactions in their mempool already
15418:15 <pinheadmz> high bandwidth sends unannounced compcat block
15518:15 <willcl_ark> from `MaybeSetPeerAsAnnouncingHeaderAndIDs`
15618:15 <jarolrod> high bandwidth mode is when you send cmpct blocks without asking for permission, can lead to higher bandwidth because you can receive the same block more than once, but reduced latency because potentially less round trips
15718:15 <norisg> we are removing code, but no functionality ?
15818:16 <pinheadmz> i was confused by this at first bc if oyu look at bip152 graphic "low bandwiwdth" appears to have more messages
15918:16 <glozow> I got 3 from this line in MaybeSetPeerAsAnnouncingHeaderAndIDs: https://github.com/bitcoin/bitcoin/blob/417f95fa453d97087a33d4176523ab278bef21a1/src/net_processing.cpp#L553
16018:16 <thomasb06> high bandwidth mode means the nodes will only get the missing transactions
16118:16 <lightlike> when it was introduced, why wasn't it decided to schedule compact blocks after segwit? Seems like a lot of effort for just one version 0.13.0
16218:16 <sipa> norisg: wouldn't that be great? no, we are removing functionality: dropping support for the non-segwit version of compact blocks
16318:16 <sunon> Also yeah only header needs to be validated before they’re sent in high bandwidth mode right?
16418:16 <AnthonyRonning> one thing I was unsure of, bip152 says "Nodes MUST NOT send such sendcmpct messages to more than three peers" but sendcmpct can be for low bandwitdth too, so is the three limit total or high bandwidth?
16518:16 <sipa> lightlike: things get merged when they're ready, and compact blocks did not require a softfork or anything
16618:16 <sdaftuar> lightlike: segwit was also a big change, and it was unclear when it would be finally ready
16718:16 <anir> upto 3 peers recommended I believe
16818:17 <norisg> sipa because the probability of the block without segwit tx is aganist 0?
16918:17 <theStack> it should not only affect 0.13.0 but also other higher versions that don't have activated segwit yet, right? (afair segwit came with v0.17, so there must be some versions v0.14.x and v0.15.x etc. with non-latest minor number that are affected)
17018:17 <lightlike> ah ok, thanks!
17118:17 <sipa> lightlike: so at the time it was included, it worked immediately on the network, and would keep doing so regardless of whether segwit activated or not (which turned out to be... uncertain for a while)
17218:17 <sunon> I also believe I saw 3 peers max
17318:17 <thomasb06> it's >= indeed
17418:17 <sipa> norisg: no no
17518:17 <sipa> norisg: the new protocol works even with non-segwit blocks
17618:17 <jnewbery> ... or indeed if it would ever be activated. Segwit and compact blocks are indepedently good things, so it doesn't make sense to gate one on the other
17718:17 <michaelfolkson> You just need one SegWit transaction for it to be a SegWit block
17818:18 <sdaftuar> theStack: no, everything from 0.13.1 on is segwit aware/activated
17918:18 <sipa> norisg: it should be less ambigious and say "segwit-supporting compact block version" and "non-segwit-supporting compact block version" perhaps
18018:18 <theStack> sdaftuar: how so? in 0.14.0 there was no segwit
18118:18 <sipa> theStack: there absolutely was
18218:18 <sdaftuar> theStack: that is not true!
18318:19 <theStack> ah, right
18418:19 <sipa> theStack: wallet support for segwit came in 0.16 iirc, but that's unrelated
18518:19 <ccdle12> i was curious what was the specific reason for a max of 3 high bandwith nodes?
18618:19 <michaelfolkson> As pinheadmz said it must have been a long time since a mined block didn't have a single SegWit transaction
18718:19 <norisg> sipa ok you you are saying that the "segwit-supporting compact block version" can also propagate non-segwit blocks?
18818:19 <sunon> Is it not for DoS mitigation?
18918:19 <jnewbery> theStack: the code for segwit was merged in bitcoin 0.13.1. Activation was later, but any node on 0.13.1 or higher can understand/validate/relay segwit blocks/txs
19018:20 <theStack> for some reason i thought it was with 0.17, sorry :x
19118:20 <sipa> norisg: correct
19218:20 <jonatack> was afk, catching up, we can select up to 3, not 2, peers to be bip152 high-bandwidth
19318:20 <norisg> and this pull request we are talking about implements "segwit-supporting compact block version"
19418:20 <willcl_ark> ccdle12: as per BIP152: "Nodes MUST NOT send such sendcmpct messages to more than three peers, as it encourages wasting outbound bandwidth across the network."
19518:21 <jnewbery> norisg: no. It removes support for non-segwit-supporting compact block version
19618:21 <ccdle12> willcl_ark: thanks
19718:21 <sipa> norisg: no; the current codebase supports both; what we're discussing is removing support for the "non-segwit-supporting compact block version"
19818:21 <nR2Cncdm> Hello, my first time. Just lurking today. Hope to make an impact before I die!
19918:21 <AnthonyRonning> that's what I'm confused about willcl_ark, doesn't low bandwidth send sendcmpct too?
20018:21 <glozow> we can't tell if a node has more than 3, can we?
20118:21 <nR2Cncdm> (i'm not expecting death btw)
20218:21 <sipa> AnthonyRonning: yes, but with an extra roundtrip
20318:21 <thomasb06> jonatack: yep, misread the code
20418:21 <sipa> AnthonyRonning: the extra roundtrip means that the receiver only asks for the actual compact block once
20518:22 <jnewbery> to everyone who answered "3 peers". You're right! Bonus point to Gloria for pointing out the line of code: https://github.com/bitcoin/bitcoin/blob/417f95fa453d97087a33d4176523ab278bef21a1/src/net_processing.cpp#L553
20618:22 <AnthonyRonning> sipa: so the three limit applies to low too? not just high?
20718:22 <jonatack> if you'd like to see your bip152 high-bandwidth peers in real time, they can be visualized with https://github.com/bitcoin/bitcoin/pull/20764
20818:22 <emzy> nR2Cncdm: good to hear. You will make an impect with the right mindset.
20918:22 <sipa> AnthonyRonning: all the ones that are not hb are lb
21018:22 <michaelfolkson> The message comparison is here AnthonyRonning: https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#intended-protocol-flow
21118:22 <jnewbery> The BIP RECOMMENDS that a node chooses up to 3 peers to request HB mode from: https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#implementation-notes
21218:23 <jnewbery> and that's exactly what Bitcoin Core implements
21318:23 <troygior1hev> jonatack: nice!
21418:23 <sipa> AnthonyRonning: the hb peers will just send the compact block without being asked for it; the lb peers will announce "hey i have a compact block for you, want it?" - and then the receiver gets to pick which of the lb peers to ask it from
21518:23 <jonatack> or via getpeerinfo in master now that theStack's PR exposing these two fields was merged
21618:23 <jnewbery> Next question (should be easy given the link above): How do we choose which peers should be high-bandwidth peers? In which function does that logic exist?
21718:23 <AnthonyRonning> sipa: thanks!
21818:23 <jonatack> troygior1hev: ty :)
21918:23 <AnthonyRonning> High bandwidth mode is for peers that have a past performance in providing blocks quickly. I believe it exists in `MaybeSetPeerAsAnnouncingHeaderAndIDs`
22018:23 <norisg> sipa: what was the reason why we had to implement it in the first place, but did't remove it immediatly when introducing "segwit-supporting compact block version"
22118:24 <pinheadmz> jnewbery in the updated code, if i send you SENDCMPCT with version 1, the function just returns. So you will not send me compact blocks - is my node "ok with this"? Or do we disconnect from peers that dont send compact blocks after we ask them to ?
22218:24 <sipa> norisg: because it was necessary at the time; segwit wasn't active on the network
22318:24 <norisg> thx sipa
22418:24 <jnewbery> norisg: because there's no rush, and potentially there could have been peers on 0.13.0 who still wanted to use compact blocks
22518:25 <michaelfolkson> norisg: Why remove functionality for 0.13.0 nodes if Core current version is 0.14?
22618:25 <jnewbery> norisg: You may want to look at the motivation in the PR again. It answers a lot of your questions: https://github.com/bitcoin/bitcoin/pull/20799
22718:25 <thomasb06> the function is "connman.ForNode(.,.)"
22818:25 <anir> I think peers be selected based on their past performance in providing blocks quickly
22918:26 <willcl_ark> jnewbery: this is in src/net_processing.cpp#MaybeSetPeerAsAnnouncingHeaderAndIDs called by PeerManager::BlockChecked
23018:26 <pinheadmz> MaybeSetPeerAsAnnouncingHeaderAndIDs sets `pfrom->m_bip152_highbandwidth_to = true;`
23118:26 <jonatack> anir: right
23218:27 <jonatack> for the peers our node chooses
23318:27 <pinheadmz> high bandwidth peers are chosen if they are the first peer to inform us of a new block
23418:27 <jnewbery> pinheadmz: Very good question. I think Bitcoin Core would be "ok with this", but it's maybe worth a quick check in the Bitcoin Core 0.13.0 code. If it's not ok with it, then maybe a polite disconnect would be better.
23518:28 <AnthonyRonning> if I forked core to connect to all my peers with high bandwith mode, is that allowed? can the network see I have connected to more than 3?
23618:29 <michaelfolkson> AnthonyRonning: No feel free!
23718:29 <jnewbery> To those that answered BlockChecked -> MaybeSetPeerAsAnnouncingHeaderAndIDs : you're right!
23818:29 <sdaftuar> AnthonyRonning: no one is in charge...
23918:29 <willcl_ark> AnthonyRonning: I don't think anyone except you would know, it's just bad practice
24018:29 <sipa> AnthonyRonning: yes, you'll primarily be wasting your own bandwidth
24118:29 <sunon> I don’t think that would be any issue
24218:29 <thomasb06> arg...
24318:29 <emzy> AnthonyRonning: I think there is no way. But what will you gain. Execpt of more bandwith incomming?
24418:29 <jnewbery> AnthonyRonning: no-one can stop you, but the BIP says you shouldn't: Nodes MUST NOT send such sendcmpct messages to more than three peers, as it encourages wasting outbound bandwidth across the network.
24518:30 <AnthonyRonning> 👍
24618:30 <glozow> AnthonyRonning: If your question is whether nodes detect/punish this behavior I don't think they do
24718:30 ⚡ emzy has unlimited bandwith on his node. Be my guest :)
24818:30 <glozow> how would they 🤔
24918:30 <AnthonyRonning> glozow: yeah!
25018:30 <norisg> does the network equally distribute the high bath width connection, lets say we have 5 nodes on the network and each one of them is connected to the same other 3 in high band with, how do we get the other one in
25118:30 <jnewbery> Any questions about BlockChecked() or MaybeSetPeerAsAnnouncingHeaderAndIDs()?
25218:30 <sunon> I mean it’s ok to play a little haha
25318:31 <jnewbery> How about this question: "Where does BlockChecked() get called from? On which thread?"
25418:31 <ccdle12> ThreadMessageHandler
25518:32 <ccdle12> ProcessNewBlock I think
25618:33 <jnewbery> ccdle12: yes, it can be called from ProcessNewBlock
25718:33 <jnewbery> it can also be called from ConnectTip, if we tried to connect the block, but it failed at that point (e.g. if it spent coins that didn't exist)
25818:34 <larryruane_> thread b-loadblk? (validation)
25918:34 <jnewbery> And yes, those functions will almost always be called on ThreadMessageHandler
26018:34 <jnewbery> larryruane_: I think you might be right that it could also be on that thread
26118:35 <norisg> ccdle12 which codefile and line ?
26218:35 <jnewbery> the interesting thing to note here is that CheckBlock is one of the two validation interface callbacks that are called synchronously. Does anyone know the other one?
26318:35 <glozow> where can I find all the threads? grep for std::thread ?
26418:35 <ccdle12> norisg: net.cpp::CConnman::ThreadMessageHandler() and net_processing.cpp:ProcessMessage()
26518:35 <norisg> thx ccdle12
26618:36 <troygior1hev> This is useful for anyone wanting to explore the different threads:
26718:36 <ccdle12> norisg: ProcessMessage() is hugh fyi lol
26818:36 <troygior1hev> https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#threads
26918:36 <glozow> <troygior1hev> big thank 🙏
27018:36 <larryruane_> glozow I think a good way to do it actually is to run the debugger, set a bp, then see which thread you're on (info threads helps)
27118:37 <jnewbery> troygior1hev: good tip!
27218:37 <glozow> larryruane_ oo nice!
27318:37 <jnewbery> Also, try running with -logthreadnames and look at your debug log
27418:37 <jonatack> doc/developer-notes.md has a section on the threads
27518:38 <pinheadmz> jnewbery tracing through net_processing I think the requesting node doesnt care if it gets a block instead of a compact block. Even if the inv has type MSG_CMPCT_BLOCK, the remote node just tests for IsGenBlkMsg which includes all block tpe messages, and then decides later to send compact or not
27618:38 <jnewbery> especially with the validation category enabled, because then you can see the validation interface callbacks being set and fired
27718:38 <jonatack> troygior1hev: what you wrote ^
27818:38 <jnewbery> pinheadmz: thanks! Quick work
27918:39 <jnewbery> ok, we can come back to the validation interface in a bit. Let's move on to the next question. If a peer chooses us to be its high-bandwidth peer, how does it signal that to us?
28018:39 <emzy> Sending sendcmpct with first byte set to1 (true).
28118:39 <theStack> by sending the SENDCMPCT message with its payload byte set to 1
28218:39 <AnthonyRonning> sending a 1 in the `sendcmpct` command
28318:39 <norisg> how do we test this compact block feature, there should already be test files I guess
28418:40 <jnewbery> yes. exactly right. The first byte of sendcmpct is for high bandwidth mode.
28518:40 <ccdle12> norisg: test/functional/p2p_compactblocks.py
28618:40 <michaelfolkson> norisg: There is already a functional test which is changed in the PR
28718:40 <jnewbery> norisg: good question. Take a look at the commits in the PR and see which functional test is being updated
28818:40 <theStack> norisg: see test/functional/p2p_compatblocks.py
28918:40 <theStack> *p2p_compactblocks.py
29018:41 <jnewbery> Next question. BIP152 states: “high-bandwidth mode permits relaying of CMPCTBLOCK messages prior to full validation (requiring only that the block header is valid before relay).” In which PeerManager function do we relay compact blocks to peers that have chosen us to be a high-bandwidth peer?
29118:41 <AnthonyRonning> I think `NewPoWValidBlock` , determined by the `m_sendcmpct_hb` flag
29218:41 <jarolrod> PeerManager::NewPOWValidBlock line 1250 of src/net_processing.cpp
29318:42 <jnewbery> AnthonyRonning jarolrod: exactly right. It's in NewPOWValidBlock(). Can anyone tell me more about NewPOWValidBlock? Where is it called from?
29418:43 <ccdle12> AcceptBlock()?
29518:43 <AnthonyRonning> yeah that was my guess as well https://github.com/bitcoin/bitcoin/blob/c37600777eea9fc248ad0cffcc1a90df9af2410c/src/validation.cpp#L3741
29618:44 <AnthonyRonning> not sure the thread info though, haven't reviewed the threads doc yet
29718:44 <jnewbery> yeah, AcceptBlock() has this call: `GetMainSignals().NewPoWValidBlock(pindex, pblock)`. What does that mean?
29818:44 <norisg> why is propagation more important than verification
29918:45 <larryruane_> Is it because that function is going to run in a different thread?
30018:45 <ccdle12> actually that's something I was studying earlier, GetMainSignals() gets called in init.cpp, so is it like a main thread in a way?
30118:45 <jnewbery> larryruane_: good guess. You're almost there
30218:45 <larryruane_> (I meant the GetMainSignals)
30318:45 <jarolrod> norisg: block propagation is important to prevent network splits
30418:46 <sipa> norisg: it's not; but verification takes time, and latency on the network is important, so we want blocks to be able to propagate without suffering a delay from validation at every hop
30518:46 <sipa> ccdle12: validation.cpp and net_processing.cpp together used to be main.cpp; i think that's where the name comes from
30618:47 <ccdle12> sipa: aahh interesting, thanks
30718:47 <larryruane_> also we don't want to give anyone an incentive to run a modified client
30818:47 <jnewbery> ok, GetMainSignals is the way that validation fires a validation interface notification
30918:47 <glozow> at this point though we've already verified the PoW, which would be very hard to fake, so it's not as risky right?
31018:48 <larryruane_> john could you elaborate a bit?
31118:48 <emzy> I think a valid Block header is enough to be relayed. It is enough work to come up with one.
31218:48 <norisg> sipa: how long do we allow peers to send us false blocks until we close the connection
31318:48 <jnewbery> other components subscribe to those notifications. Look for any class that inherits the CValidationInterface interface to see examples
31418:48 <norisg> in high band with mode
31518:48 <sdaftuar> glozow: yep the PoW prevents DoS. however it's worth looking at what happens if a block is invalid
31618:48 <sipa> norisg: it is very expensive for them to do so, due to PoW
31718:49 <sipa> (even unvalidated blocks must pass PoW)
31818:49 <norisg> sipa perfect thanks
31918:49 <jnewbery> many of those notifications are asynchronous (i.e. validation sends them, and then the subscibers get notified by a background thread), but there are a couple that are synchronous (i.e. the subscribers are called directly by the thread in validation)
32018:49 <jnewbery> BlockChecked was one synchronous callback. What's the other?
32118:50 <jnewbery> They're defined in https://github.com/bitcoin/bitcoin/blob/68196a891056f98c1df0debacf09fb2ea4682a43/src/validationinterface.h
32218:51 <jonatack> jnewbery: i'd gladly review a commit adding that info in a GetMainSignals() doxygen doc in validationinterface.h
32318:51 <pinheadmz> sdaftuar glozow related: i dont see in net_processing how we deal with unsolicited BLOCK messages? is that not a DoS risk?
32418:52 <jnewbery> look for "called on a background thread"
32518:52 <AnthonyRonning> ah so NewPoWValidBlock is sync too then?
32618:52 <jnewbery> YES!
32718:53 <jnewbery> AnthonyRonning: correct
32818:53 <jnewbery> why do you think that is?
32918:53 <larryruane_> ChainStateFlushed() also
33018:53 <glozow> pinheadmz: I suppose we check blocks in an order that prioritizes failing early
33118:53 <jnewbery> ChainStateFlushed is asynchronous
33218:54 <glozow> i.e. first non-contextual and look at PoW before doing any expensive verification
33318:54 <AnthonyRonning> does BlockChecked not call NewPoWValidBlock? So they'd both be sync?
33418:54 <pinheadmz> glozow ah yeah i thikn i see in CChainState::AcceptBlock() if we didnt request the block we check chainwork in the header before anything
33518:54 <jnewbery> AnthonyRonning: No. NewPOWValidBlock is called by AcceptBlock()
33618:55 <glozow> i think we do that for blocks we did request too, right?
33718:55 <jnewbery> The reason it's important is that we want to send out the high-bandwidth compact blocks as quickly as possible
33818:55 ℹ sanketcell_ is now known as sanketcell
33918:55 <larryruane_> Why is there such a thing as synchronous notifications? How is that different from just a normal function call?
34018:55 <pinheadmz> glozow sorry we just check against nMinimumChainWork
34118:55 <jnewbery> so we immediately (synchronously) call the NewPoWValidBlock() callback in net_processing, which sends the cmpctblock message out immediately
34218:56 <jnewbery> larryruane_: validation doesn't hold a reference/pointer to peermanager. Everything that peermanager gets from validation is through the validation interface
34318:56 <AnthonyRonning> interesting, so do async notifications go through a queue of some sort? And calling it syncronously skips that?
34418:57 <jnewbery> AnthonyRonning: basically yes, async notifications are all serviced on the background scheduler thread
34518:57 <nehan> jnewbery: so the documentation that NewPoWValidBlock is synchronous vs asynchronous is the fact that it does *not* have "Called in a background thread" in its defn comments in validationinterface.h?
34618:57 <nehan> how else might one confirm that?
34718:57 <jnewbery> whereas synchronous notifications are just regular function calls by the thread that's doing the validation
34818:58 <AnthonyRonning> gotcha, very cool, thanks
34918:58 <nehan> just that its function doesn't call ENQUEUE_AND_LOG_EVENT?
35018:58 <jnewbery> nehan: that's the documentation, yes! You could look in validationinterface.cpp to see how those notifications are invoked, or you could run a node with thread debug logs enabled and validation category debug logs enabled and see which threads are servicing the validation interface callbacks
35118:59 <jnewbery> We've covered the last question already (How is that PeerManager function invoked? In which thread is it called?)
35218:59 <jnewbery> And that's about time.
35319:00 <jnewbery> Next week, dhruv is hosting on PR 19825. We'll have notes up by Friday.
35419:00 <jnewbery> Thanks everyone!
35519:00 <jnewbery> #endmeeting
35619:00 <pinheadmz> thanks jnewbery !
35719:00 <theStack> thanks for hosting jnewbery
35819:00 <AnthonyRonning> thank you!
35919:00 <troygior1hev> thanks jnewbery!
36019:00 <olympics> thanks jnewbery
36119:00 <norisg> thank you !
36219:00 <ccdle12> thanks jnewbery
36319:00 <thomasb06> thanks jnewbery
36419:00 <emzy> Thank you jnewbery and all others. Happy, healthy and productive 2021 to you all.
36519:00 <lightlike> thanks jnewbery
36619:00 <glozow> thanks jnewbery!
36719:00 <fodediop> Thanks jnewbery!
36819:00 <larryruane_> thank you all, this was great!
36919:00 <anir> Thanks jnewbery and everyone else!
37019:01 <jarolrod> thanks