Use legacy relaying to download blocks in blocks-only mode (
p2p) Aug 4, 2021
The PR branch HEAD was 0c2f9346024cfca29d5f0880cb4471961ffd112b at the time of this review club meeting.
After a block is mined it is broadcast to the p2p network where it will eventually
be relayed to all nodes on the network. There are two methods available
for relaying blocks: legacy relay and compact block relay.
Legacy Relay: A node participating in legacy relaying will always send or
request entire blocks. For nodes that maintain a mempool this is quite
bandwidth inefficient, since they probably already have most of the
transactions from a new block in their mempool.
Compact Block Relay: Compact block relay is specified in BIP 152.
The goal is to address the bandwidth inefficiencies of legacy relaying by
only relaying the transactions of a new block that the requesting peer has not
yet seen. Check out this Compact Blocks FAQ
for bechmarks and more info.
Bitcoin Core 0.12 introduced a
-blocksonly setting that can reduce a node’s
bandwidth usage by 88%. The reduction is achieved by not participating in
transaction relay. For more info check out
on blocksonly mode by Gregory Maxwell. Blocksonly nodes currently use
compact block relaying to download blocks even though they don’t maintain
a full mempool.
This PR makes blocksonly nodes use legacy relaying to download new blocks.
Did you review the PR?
Concept ACK, approach ACK, tested ACK, or NACK?
What is the sequence of messages used in legacy and compact block relaying?
Why does compact block relay waste bandwidth for blocksonly nodes during
block download? How much bandwidth is waisted?
Can and should a blocksonly node still serve compact blocks?
Where and how does this PR use it to achieve its goals?
What do you think of
on the usage of
sendrawtransaction? Can you think of other exceptions in
which a blocksonly node would still want to download compact blocks?
1 10:00 <dergoegge> #startmeeting
3 10:00 <dergoegge> Hi everyone! Welcome to this week's PR Review Club!
7 10:00 <BlockHead> Whats up
9 10:00 <dergoegge> Feel to say hi to let people know you are here (lurkers are also welcome)
13 10:00 <dergoegge> Anyone here for the first time?
14 10:00 <merkle_noob[m]> Hi everyone...
18 10:00 <dergoegge> today we are looking at #22340 - Use legacy relaying to download blocks in blocks-only mode
20 10:01 <JanB> firstimer , will be watching only for this time
23 10:01 <jnewbery> JanB: welcome :)
24 10:01 <glozow> welcome JanB :D
25 10:01 <BlockHead> also my 1st time
26 10:01 <dergoegge> JanB: welcome!
27 10:01 <dergoegge> we have a couple of prepared questions but feel free to jump in at any point if you have any questions or points you want to add
28 10:01 <jnewbery> BlockHead: Welcome!
29 10:01 <Azorcode> Hello Everyone
31 10:02 <dergoegge> Ok lets get started, Who had a chance to review the PR? Concept ACK, approach ACK, tested ACK, or NACK?
34 10:02 <amiti> started reviewing, concept ACK!
35 10:02 <dergoegge> BlockHead: Welcome!
36 10:02 <raj> initial pass. Concept ACK..
37 10:02 <theStack> y, concept ACK
38 10:02 <larryruane> review y, tested n, ACK
40 10:02 <glozow> concept ACK, needs a functional test in p2p_compactblocks_hb.py imo
41 10:02 <murchandamus> Sounds like a concept ack though
42 10:02 <lightlike> concept ACK
43 10:02 <amiti> yeah agree
44 10:02 <jnewbery> reviewed && ACKed
45 10:02 <raj> glozow, +1..
46 10:02 <willcl_ark> ACK from me
47 10:03 <amiti> (agree with glozow)
48 10:03 <dergoegge> glozow: good point, i agree
49 10:03 <dergoegge> First question: What is the sequence of messages used in legacy and compact block relaying?
50 10:03 <willcl_ark> Yeah I also thought a test would be nice, to avoid a future change inadvertantly reverting this nice win :)
51 10:04 <amiti> I started tinkering with the functional test, but I'm getting surprising results. I thought one difference would be a blocksonly node sending sendcmpct with 0 vs 1, but that's not what I'm getting in the tests 🤔
52 10:04 <amiti> but maybe we can get to that later :)
53 10:05 <dergoegge> amiti: interesting, we can get to that later
54 10:05 <larryruane> for the sequence, there's a really nice diagram in the PR description
55 10:06 <BlockHead> larryruane yeah i think that sequence diagram is taken from BIP 152 doc
56 10:06 <glozow> both would do headers first,
57 10:06 <glozow> low-bandwidth compact block would request GETDATA(CMPCT_BLOCK) and get a compact block and then do GETBLOCKTXN?
58 10:06 <glozow> legacy would do a GETDATA(BLOCK) and then a BLOCK
59 10:06 <glozow> high-bandwdith compact block sends a compact block straight away, with coinbase prefilled?
60 10:07 <dergoegge> glozow: yes, i think the coinbase is prefilled
62 10:08 <glozow> headers first for low bandwidth compact block and for legacy*?
63 10:08 <dergoegge> glozow: i think both inv and headers is possible but unsure
64 10:08 <larryruane> i did have a question about case C, low bandwidth relaying ... since node B is sending a getdata(CMPCT) message, why is it necessary for that node to announce ahead of time that it wants to use this mode (the sendcmpct(0) message?
65 10:09 <glozow> larryruane: i think you'd need a version of bitcoin client that understands compact block p2p messages
66 10:09 <larryruane> so for example, let's say node B doesn't send the sendcmpct(0) message ... then later sends a getdata(CMPCT)? is that just a protocol error?
67 10:10 <dergoegge> larryruane: yes the peer would not respond with a cmpctblock message
68 10:10 <lightlike> GETBLOCKTXN is not always sent - if we have all the txes in our mempool, would it be skipped?
69 10:10 <murchandamus> IIRC, only really old nodes would announce blocks with INV messages, any newer nodes would always announce it with the header
70 10:10 <larryruane> glozow: okay, but if it's sending a getdata(CMPCT), doesn't that indicate that it understands compact block p2p messages?
71 10:10 <BlockHead> glozow it looks like headers/inv are sent via legacy and low band. and not highband.
72 10:11 <dergoegge> lightlike: i think it is sent and it requests all the txs that are missing
75 10:12 <glozow> BlockHead: ye!
76 10:12 <lightlike> dergoegge: yes, but what if there are no txs missing (because we have them all in our mempool)?
77 10:12 <glozow> lightlike: then we don't send any GETBLOCKTXN
78 10:12 <jnewbery> murchandamus: we'll announce blocks using INV to any peers that don't send us a `sendheaders` during version handshake
79 10:12 <murchandamus> jnewbery: Okay, thanks
80 10:13 <dergoegge> lightlike: oh right, i am not sure in that case but i guess it could be omitted
81 10:13 <glozow> lightlike: also, could get them from `vExtraTxnForCompact`
82 10:13 <dergoegge> Ok next question: Why does compact block relay waste bandwidth for blocksonly nodes during block download? How much bandwidth is wasted?
83 10:14 <lightlike> we can also revert to inv-mode - at least, this seems to happen sometimes in the functional tests intermittently, when the CI has strange delays for some reason
84 10:14 <glozow> larryruane: I think we usually disconnect a peer that sends a message we don't expect? not always tho, would need to look at what BIP152 says
86 10:15 <larryruane> dergoegge: is it because the node will need to request a bunch of tx that we would have gotten with the full block?
87 10:15 <BlockHead> dergoegge block only nodes don't have a mempool, so it always needs all the detail about new blocks
88 10:16 <dergoegge> larryuane: yes the requesting of the txs adds a bit of overhead but there is also another message that causes some waste
90 10:18 <dergoegge> BlockHead: yes blocksonly nodes don't have a mempool
91 10:18 <lightlike> i think most of the overhead is them sending us all the short-ids for the txes, when we'd request all of them anyway. that is unnecessary
92 10:18 <dergoegge> lightlike: exactly
93 10:19 <dergoegge> the unnecessary shortids are in the cmpctblock and getblocktxn message
94 10:19 <jnewbery> larryruane: I think you can think of a sendcmpct(0, version) to mean "I can provide compact blocks" and a sendcmpct(1, version) to mean "I want you to use HB compact blocks to announce new blocks"
95 10:20 <dergoegge> one thing to note here is that bandwidth is only wasted when downloading blocks
96 10:20 <dergoegge> which ties into the next question: Can and should a blocksonly node still serve compact blocks to their peers?
97 10:20 <lightlike> dergoegge: Are you sure? I thought they do have a mempool as well, it is just not updated via p2p? for example, if I restart a normal node with a full mempool into blocks-only mode, the mempool will still be full?!
98 10:21 <larryruane> jnewbery: +1 thanks
99 10:21 <jnewbery> lightlike: yes, you're right. There is still a mempool, but the node won't request transactions from peers when it receives an inv
100 10:21 <amiti> lightlike: that's true, but rare right? you'd have a mempool initially, but then after some time it'd likely clear out?
101 10:22 <lightlike> amiti: yes, I agree. with time, it will clear out.
102 10:22 <amiti> dergoegge: yes, I think blocksonly nodes serving compact blocks makes sense
103 10:22 <jnewbery> I don't think there's currently a config option to switch the mempool off entirely, although it shouldn't be too much work to implement that now that so much work has been done to separate out the components
104 10:23 <theStack> i'd also say serving compact blocks from blocksonly nodes makes sense
105 10:23 <larryruane> but even in blocks-only, don't i need a mempool for transactions that i'm initiating?
106 10:23 <glozow> larryruane: yes, you do. which is why you have one i think
107 10:24 <amiti> larryruane: you can initiate transactions in blocksonly node but its not recommended. its a dead give away that they are your own =P
108 10:24 <dergoegge> amiti theStack: yes so blocksonly nodes can and should still serve compact blocks since they really only the entire block to be able to do that
109 10:24 <larryruane> amiti: glozow: gotcha thanks makes sense
111 10:26 <dergoegge> What is PeerManagerImpl::m_ignore_incoming_txs? Where and how does this PR use it to achieve its goals?
112 10:27 <larryruane> hey in case this may be useful to anyone else, to measure the bandwidth of your local bitcoind, run `sudo iftop -f 'port 8333' in another window (on linux, you may have to apt install it)
113 10:27 <theStack> m_ignore_incoming_txs it is set to true if the node was started with -blocksonly
114 10:27 <larryruane> sorry that didn't format right `sudo iftop -f 'port 8333'`
115 10:28 <dergoegge> larryruane: to collect the data for this PR i used -debug=all and then later parsed the logs
117 10:28 <dergoegge> theStack: correct!
118 10:30 <lightlike> I think the naming of the variable m_ignore_incoming_txs is a bit misleading. we don't just ignore them, we'll disconnect the peer if they send us txes.
119 10:31 <jnewbery> lightlike: good point. Maybe m_ignore_tx_announcements would have been more appropriate.
121 10:33 <amiti> what about m_please_don't_send_me_txns ?? :)
122 10:33 <dergoegge> So one thing we do in this PR is not send sendcmpct(1) if m_ignore_incoming_txs=true
123 10:33 <lightlike> jnewbery: but we seem to disconnect even for sending INVs. maybe just m_blocksonly_mode ?
124 10:33 <raj> amiti, I like it.. :D
125 10:33 <dergoegge> what does that achieve?
126 10:34 <jnewbery> lightlike: oh yes, you're right that we'll disconnect if they send an INV
127 10:35 <larryruane> just for us newbies, INV used to be used to announce both blocks and transactions, but now is used only for transactions?
128 10:36 <jnewbery> (the reason we disconnect is that it's a violation of the protocol. We've sent them fRelay=false in our version message to them, requesting that they don't relay txs to us)
129 10:36 <willcl_ark> Agree the name could be changed, but I quite like the behaviour, it's better for low-bandwidth nodes to disconnect if people start sending you unsolicited messages, wasting your bandwidth
130 10:36 <lightlike> larryruane: you are right, i meant tx INVs, there are two types of INVs, and Block INVs are ok in blocksonly mode
131 10:38 <merkle_noob[m]> Please, I have a question: What is the difference between sending a headers announcement and an INV announcement?
132 10:38 <sipa> in an INV announcement, you send just the block hash
133 10:38 <sipa> in a headers announcement, the block header (80 bytes) is sent instead
134 10:39 <larryruane> notice too that at least by default, we set up a block-relay-only connection to some of our peers, even without the local `-blocksonly` flag ... does that cause `ignores_incoming_txs` to be set to true?
135 10:39 <merkle_noob[m]> sipa: Thanks...
136 10:39 <JanB> dergoegge by not sending sendcmpct(1) we default to Legacy relaying (?)
137 10:39 <amiti> larryruane: no, -blocksonly is a mode & sets ignore_incomping_txs
138 10:39 <amiti> but block-relay-only is an attribute of a connection
139 10:40 <larryruane> but should the changes this PR is making apply to the latter case too?
141 10:41 <dergoegge> JanB: almost correct, there is a second step to defaulting to legacy relay. by not sending sendcmpct(1) we don't request high bandwidth mode, so our peer won't send us cmcptblock messages to announce blocks.
142 10:42 <JanB> dergoegge: ah yes, i c
143 10:43 <dergoegge> So now that high bandwidth mode is covered what do we need to do for low bandwidth mode?
144 10:43 <amiti> larryruane: I don't think so. I could be running a node with a full mempool and have a block-relay-only connection to you. Just because you don't send me txns doesn't mean I necessarily don't already have them to reconstruct the block from short ids.
145 10:43 <amiti> with -blocksonly mode, although its possible I have mempool txns based on edge cases, most likely I don't have mempool txns, so will need to get all the transactions
146 10:44 <raj> dergoegge, we should also ensure not to send sendcmpct(0)?
147 10:44 <theStack> dergoegge: if we receive an announcement via headers or inv, request the full block rather than compact blocks
148 10:44 <lightlike> a weird thing is that I think even in blocksonly mode, one has two block-relay-only connections ;)
149 10:44 <amiti> lightlike: hahhaha, yeah that's true =P
150 10:45 <larryruane> amiti: yes, you're exactly right, i see now, +1
151 10:45 <dergoegge> raj: we actually need to still send sendcmpct(0) otherwise a peer won't request compact blocks and a blocksonly node still wants to serve those
152 10:45 <dergoegge> theStack: correct!
153 10:46 <jnewbery> lightlike: amiti: (I know you know this alredy, but for everyone else) block-relay-only connections also don't gossip addrs, so even in blocksonly mode their behaviour is slightly different from the other connections
154 10:46 <merkle_noob[m]> jnewbery: Thanks for the link...
155 10:46 <raj> dergoegge, oh right.. thanks..
157 10:47 <dergoegge> amiti: since we are talking about sendcmpct what did you observe in your tests?
158 10:48 <JanB> dergoegge: that's done by the extra check added on line 2125 ? (to request the full block)
159 10:48 <theStack> so if i see that correctly, at the point of the diff where the second change occurs, the GETDATA message is already prepared to request the full block, and we add an additional condition (!m_ignore_incoming_txs) to when to modify the message to request compact blocks
160 10:50 <jnewbery> theStack: right - there's no point in requesting a compact block if we don't have transactions in our mempool
161 10:50 <amiti> dergoegge: even with the guard to MaybeSetPeersAsAnnouncing... commented out, the sendcmpct announce bool was set to False for a blocksonly node. I thought this is what that clause would change (so, true on master, false on PR). but I might be missing something in my test setup or my understanding =P
163 10:50 <dergoegge> JanB: yes so if m_ignore_incoming_txs = true then we don't send getdata(CMPCT) and instead send a normal getdata(BLOCK)
164 10:50 <larryruane> theStack: looks like to me too, and also I found it interesting (not changed by this PR) that only the first entry in the CInv vector needs to say MSG_CMPCT_BLOCK ... all the others can still say MSG_BLOCK
165 10:51 <lightlike> larryruane: One more thought about your earlier question: I think the main reason for having block-relay-only connections is to not be subjected to privacy issues with tx and addr relay. By allowing compactblock downloading, you do reveal something about your mempool to them which you otherwise wouldn't (which txes of the block you still need), but I think it's not really possibly to abuse this, because creating valid blocks we'd
166 10:51 <lightlike> download from you requires POW.
167 10:52 <dergoegge> amiti: sendcmpct is send multiple times and the first time it is send the announce flag defaults to false
169 10:52 <dergoegge> maybe that is what you are seeing?
170 10:52 <larryruane> lightlike: +1
171 10:53 <theStack> jnewbery: yes that makes sense
172 10:53 <amiti> I see two sendcmpct messages being sent & both have announce = false, whether or not the m_ignore_incoming_txs guard is set
173 10:53 <theStack> larryruane: hm is it even possible to mix different types within an INV? (i have to make myself more familiar with the protocol messages...)
174 10:53 <jnewbery> amiti: we'll only send a sendcmpct(hb=true) to a peer if that peer has sent us a valid block at the tip
176 10:54 <jnewbery> during version handshake we'll send two sendcmpct(hb=false) to the peer to let them know we support version 1 and version 2 compact blocks
177 10:54 <dergoegge> jnewbery: oh yes that is also something that is saw in my manual testing
179 10:55 <amiti> ok so in order to see the change in behavior, I should have the p2pconn send the blocksonly node a valid block at the tip in the test setup?
180 10:55 <dergoegge> in the interest of time lets move on: What do you think of Gleb’s comments on the usage of sendrawtransaction? Can you think of other exceptions in which a blocksonly node would still want to download compact blocks?
182 10:55 <jnewbery> but we only send sendcmpct(hb=true) if the peer is the first to provide a valid block at the tip. It's in the BlockChecked callback in the validation interface
183 10:55 <jnewbery> amiti: yes, that should do it as long as the node is out of IBD
184 10:56 <amiti> jnewbery: ok! that helps a lot, thanks :)
185 10:58 <amiti> dergoegge: I think its possible but highly unlikely
186 10:58 <lightlike> dergoegge: if a node in blocksonly mode somehow receives up-to-date txes from somewhere else. I don't really think that the scenario by gleb sounds very probable.
187 10:58 <jnewbery> dergoegge: I think it's unlikely that anyone is using the node in this way
188 10:59 <dergoegge> lightlike: yeah i can't think of another way for a node to receive all txs besides sendrawtransaction
189 10:59 <dergoegge> i also think gleb's scenario is pretty unlikely
190 10:59 <dergoegge> if you find other exceptions please comment on the PR :)
191 11:00 <dergoegge> #endmeeting