Reduce bandwidth during initial headers sync when a block is found (p2p)

https://github.com/bitcoin/bitcoin/pull/25720

Host: dergoegge  -  PR author: sdaftuar

The PR branch HEAD was 17f2822c at the time of this review club meeting.

Notes

  • On startup, if a node’s headers chain is more than a day behind the current time, it will pick one peer to sync headers with until its best headers chain is caught up. (See here) If a node’s headers chain is not caught up within 15 min + ~num headers missing * 1ms (either because the chosen peer was not able to or because they purposefully stalled the download), it will disconnect the chosen peer for stalling and pick a new peer to sync headers with. Blocks are only downloaded and validated after the node’s headers chain has sufficient work (nMinimumChainWork) to protect against DoS attacks.

  • When a node receives a block announcement via an inv during headers sync, it will add all announcing peers as additional headers sync peers. PR #25720 changes this logic such that only one of the announcing peers is added for headers sync.

  • BIP 130 (implemented in PR #7129) introduced the sendheaders message, which indicates that a node prefers receiving block announcements via a headers message rather than an inv.

Questions

  1. Did you review the PR? Concept ACK, approach ACK, tested ACK, or NACK?

  2. Why do nodes (mostly) receive inv block announcements while they are doing initial headers sync, even though they indicated preference for headers announcements (BIP 130)?

  3. Why is bandwidth wasted (during initial headers sync) by adding all peers that announce a block to us via an inv as headers sync peers?

  4. What would be your estimate (lower/upper bound) of how much bandwidth is wasted? (There is no one true answer here, the estimate depends on a couple of variables)

  5. What’s the purpose of CNodeState’s members fSyncStarted and m_headers_sync_timeout, and PeerManagerImpl::nSyncStarted? If we start syncing headers with peers that announce a block to us via an inv, why do we not increase nSyncStarted and set fSyncStarted = true and update m_headers_sync_timeout?

  6. An alternative to the approach taken in the PR would be to add an additional headers sync peer after a timeout (fixed or random). What is the benefit of the approach taken in the PR over this alternative?

  7. Can you think of any other alternatives?

Meeting Log

  117:00 <dergoegge> #startmeeting
  217:00 <dergoegge> Hi everyone, welcome to this week's PR review club!
  317:00 <effexzi> Hi every1
  417:00 <Lov3r_Of_Bitcoin> hello
  517:00 <vnprc> hi
  617:00 <dergoegge> Feel free to say hi to let people know you are here
  717:00 <svav> Hi
  817:00 <dergoegge> Anyone here for the first time?
  917:00 <BlueMoon> Hello!!
 1017:01 <lightlike> Hi
 1117:01 <juancama> Hi everyone
 1217:01 <dergoegge> This week we are looking at #25720 “Reduce bandwidth during initial headers sync when a block is found”
 1317:01 <larryruane_> hi
 1417:01 <dergoegge> Notes are in the usual place: https://bitcoincore.reviews/25720
 1517:01 <hernanmarino> Hi
 1617:01 <pablomartin> hello
 1717:02 <dergoegge> OK let's get started with the usual first question: Did you review the PR? Concept ACK, approach ACK, tested ACK, or NACK?
 1817:02 <juancama> yes
 1917:03 <hernanmarino> yes, tested ACK
 2017:04 <pablomartin> tested ACK
 2117:04 <adam2k> approach ACK
 2217:04 <dergoegge> hernanmarino: that's cool, i am curious how you tested it?
 2317:05 <dergoegge> did you run the functional tests or did you do anything more elaborate?
 2417:05 <hernanmarino> comilead and ran the tests. Also reviewed the code , I have some doubts but for later :)
 2517:06 <hernanmarino> compiled*
 2617:06 <pablomartin> same
 2717:06 <glozow> hi
 2817:06 <hernanmarino> I am actually thinking of adding a test later to answer my own doubt
 2917:06 <dergoegge> Ok lets get to the doubts later then :)
 3017:07 <dergoegge> Next question: Why do nodes (mostly) receive inv block announcements while they are doing initial headers sync, even though they indicated preference for headers announcements (BIP 130)?
 3117:07 <Amirreza> Hi
 3217:08 <hernanmarino> I'm not sure but ... Is it a fallback for some errors conditions or large reorgs or your peer not having the headers you requested ?
 3317:08 <svav> What does inv stand for?
 3417:08 <Amirreza> @dergoegge, I think it is because inv can be received without request. So some node may notify me that new block is received.
 3517:08 <juancama> inventory
 3617:09 <juancama> It’s important to blocks-first nodes that the blocks be requested and sent in order because each block header references the header hash of the preceding block. That means the IBD node can’t fully validate a block until its parent block has been received.
 3717:09 <dergoegge> svav: inv stand for inventory, it is usually used as a short announcement (like a tx id or block hash) for a larger message
 3817:10 <larryruane_> and `inv` can contain many tx or block hashes, right? (not just 1)
 3917:10 <glozow> up to 50000, of multiple types
 4017:11 <larryruane_> https://en.bitcoin.it/wiki/Protocol_documentation#inv
 4117:11 <dergoegge> amirreza: headers can also be received without request (see the description for BIP 130 in the notes)
 4217:11 <svav> OK thanks all
 4317:12 <Amirreza> @dergoegge, I mean during I'm syncing my headers, a new block can be mined, and why not other nodes notify me to get that newly mined block?
 4417:13 <larryruane_> an `inv` message can be a mixture of blocks (actually that i have knowledge of different formats of blocks) and transactions https://en.bitcoin.it/wiki/Protocol_documentation#Inventory_Vectors
 4517:14 <dergoegge> hernanmarino: you are close
 4617:14 <dergoegge> the reason is that a node will not announce a new block to a peer using a headers message if the peer has not previously sent a header to which the new header connects to
 4717:14 <dergoegge> See: https://github.com/bitcoin/bitcoin/blob/a6fc293c0a1f27ba1e573bfa16fd76d5f58988b2/src/net_processing.cpp#L4975-L4978
 4817:14 <hernanmarino> ohh, okey
 4917:15 <juancama> nodes (mostly) receive inv block announcements while they are doing initial headers sync bc block inventories show up in the inv message in the same order they appear in the chain, first inv message contains inventories for blocks 1 - 501. IBD node uses the received inventories to request 128 blocks from sync node in the “getdata” msg.
 5017:15 <dergoegge> the node only announces with a header if it is sure that the peer has already seen the parent block
 5117:16 <hernanmarino> ok, but this is surely the case on reorgs that our node is no t aware of, right ?
 5217:16 <hernanmarino> And also can happen without reorgs
 5317:16 <glozow> well this node is in IBD so it doesn't know the parent block of the new block
 5417:17 <vnprc> in this context what does "know" mean?
 5517:17 <ibaddesmukh> dergoegge if node doesn't find the header, it reverts to inventory?
 5617:17 <dergoegge> hernanmarino: yes, this can also happen for reorgs not just during initial sync
 5717:19 <dergoegge> ibaddesmukh: if a node doesn't have the header it self it won't announce it in any way
 5817:19 <ibaddesmukh> dergoegge thank you, got it!
 5917:19 <glozow> doesn't know = has not downloaded
 6017:21 <dergoegge> i mostly asked this question to show that nodes doing initial headers sync will almost always receive block announcements through invs
 6117:21 <dergoegge> so only adding the logic for additional headers sync peers there makes sense
 6217:23 <dergoegge> Next question: Why is bandwidth wasted (during initial headers sync) by adding all peers that announce a block to us via an inv as headers sync peers?
 6317:23 <Amirreza> I have difficulty understanding the answer of this question, does it need to know the bip-130?
 6417:24 <larryruane_> dergoegge: because you end up receiving duplicate headers (across multiple peers)
 6517:24 <Amirreza> @dergoegge, well not sure, but it may cause downloading same block headers from many peers.
 6617:24 <adam2k> In that situation are we getting repeated headers from some of the various message types?
 6717:24 <glozow> you'll probably get a lot of announcements -> a lot of headers syncing -> download the same headers repeatedly
 6817:25 <vnprc> I think you need some process to identify potential new headers sync peers in case your current one needs to be disconnected.
 6917:25 <pablomartin> yeah, dupes
 7017:25 <dergoegge> You are all correct! You only need to download the headers once
 7117:26 <glozow> we don't attempt to limit the number of peers we're requesting 1 header from, yes? unlike blocks
 7217:26 <larryruane_> This duplicate headers during IBD was attempted to be fixed back in 2016, but the fix had to be reverted: https://github.com/bitcoin/bitcoin/pull/8306
 7317:26 <dergoegge> larryruane_: oh interesting, didn't know about that
 7417:27 <larryruane_> I'll mention it in a comment on the current PR 25720, doesn't seem to be mentioned there yet
 7517:27 <dergoegge> glozow: yeah once our header chain is close to today we request headers from all peers
 7617:29 <lightlike> larryruane: I think that problem (multiple getheaders with one peer) was already fixed by #25454
 7717:30 <glozow> talking about https://github.com/bitcoin/bitcoin/issues/6755, yeah?
 7817:31 <larryruane_> lightlike: thanks, I didn't know about that, but the attempted fix that had to be reverted in 2016 involved not just one peer but across multiple peers (like the current PR)
 7917:32 <dergoegge> next question: What would be your estimate (lower/upper bound) of how much bandwidth is wasted? (There is no one true answer here, the estimate depends on a couple of variables)
 8017:33 <juancama> The amount of bandwidth wasted depends partially on if an inv for a block is received before headers chain is caught up, no?
 8117:34 <dergoegge> Probably best if the answer is a formula to estimate the waste
 8217:34 <larryruane_> wouldn't the upper bound of waste be something like 80 bytes times number of blocks (700k) times (number of peers - 1)?
 8317:34 <dergoegge> juancama: yes thats correct, you will download less duplicate data if a block is announced later in the initial sync
 8417:35 <hernanmarino> I don't have a number but my guess is (amount_of_peers - 1)* size_of_headers * remaining_headers
 8517:35 <dergoegge> larryruane, hernanmarino: good answers
 8617:36 <dergoegge> larryruane_: fun fact headers are actually 81 bytes on the wire not 80 :D
 8717:36 <larryruane_> oh cool, why is that (if not too much of a diversion)?
 8817:36 <hernanmarino> part of the protocol ?
 8917:36 <dergoegge> because they are serialized as empty CBlock's, so you have one extra byte for the empty transaction vector
 9017:37 <larryruane_> OH got it, thanks, makes sense
 9117:37 <sipa> The notion of a block header as a protocol concept didn't exist originally.
 9217:37 <sipa> headers were just blocks without transactions
 9317:37 <hernanmarino> great
 9417:37 <larryruane_> (tx vector length is zero, varint)
 9517:37 <dergoegge> sipa: ah ok i was wondering why it was implemented like that!
 9617:39 <dergoegge> the estimate could be improved by accounting for the `getheaders` messages but they are probably small in comparison
 9717:39 <lightlike> I think it's weird that we tie the act of adding more headers-sync peers to a random event (when a block is found). Sometimes it's happening multiple times within 10 minutes, sometimes it doesn't happen for an hour or so. Waiting for a fixed period (e.g. 10 minutes) before adding another peers would seem more natural to me.
 9817:40 <dergoegge> lightlike: we'll get there, question 6 will be about the approach :)
 9917:40 <larryruane_> lightlike: I think it's always been more of an accidental kind of thing... I traced through this code once, and when we get a new block from a peer, that naturally begins a new "thread" of getheaders and headers reply sequence to and from that peer
10017:41 <larryruane_> (not thread in the linux sense, but in the p2p sense!)
10117:41 <lightlike> oh ok, sorry, i forgot to read the notes...
10217:42 <dergoegge> question 5: What’s the purpose of CNodeState’s members fSyncStarted and m_headers_sync_timeout, and PeerManagerImpl::nSyncStarted? If we start syncing headers with peers that announce a block to us via an inv, why do we not increase nSyncStarted and set fSyncStarted = true and update m_headers_sync_timeout?
10317:42 <glozow> wait dergoegge, when you said "yeah once our header chain is close to today we request headers from all peers" were you saying that when we're not close to today we try fewer peers?
10417:44 <dergoegge> glozow: when we are not close to today, we choose one peer to sync the headers from (and add additional peers if an inv announcement is received)
10517:44 <dergoegge> if that chosen peer doesn't deliver within 20 minutes we will disconnect it and try a different one
10617:44 <hernanmarino> dergoegge : the purpose is to evict unresponsive nodes, but i don know the answer to the seconda part of the question
10717:45 <glozow> oh i got confused thinking that was about the additional inving peers, nvm
10817:45 <larryruane_> if we're in that mode, not close to today, would it be good to round-robin among the peers that we request headers from? (instead of just one peer)?
10917:45 <larryruane_> (that way we're not trusting a single peer as much)
11017:45 <glozow> larryruane_ is round-robin not 1 peer at a time?
11117:46 <larryruane_> yes, but i mean send getheaders to peer 1, get reply, then getheaders to peer 2, ... etc (still single-threaded)
11217:46 <juancama> fsyncstarted tells us whether we've started headers synchronization with this peer, m_headers_sync_timeout tells us when to potentially disconnect peer for stalling headers download
11317:46 <dergoegge> hernanmarino: yes, although m_headers_sync_timeout is only used for our initially chosen headers sync peer
11417:47 <lightlike> dergoegge: Do I understand it correctly that it doesn't really matter if the our peer sends us nothing at all for 20 minutes, or 99% of the headers within that time - they will get disconnected after 20 minutes for stalling either way.
11517:47 <adam2k> For question 5: is this to prevent sending headers when another request is in-flight?
11617:48 <dergoegge> juancama: yes! but m_headers_sync_timeout is only used on one peer and if nSyncStarted == 1
11717:49 <hernanmarino> might it be the case that we only don trust our randomly chosen node, beacuse it will eventually be replaced if unresponsive, and we do not want to do that with the other peers we are adding with this new logic ?
11817:49 <dergoegge> nSyncStarted corresponds to the number of peers with fSyncStarted = true
11917:49 <dergoegge> lightlike: yes that is my understanding as well, if they don't catch us up within ~20minutes we disconnect
12017:50 <glozow> `m_headers_sync_timeout` also depends on how much time we have between tip and today, right? https://github.com/bitcoin/bitcoin/blob/92f6461cfd39fff2fc885dd623fa47e7d8d53827/src/net_processing.cpp#L4904-L4910
12117:51 <lightlike> dergoegge: I think it could make sense to also require some progress for them, and disconnecting them much earlier than 20min if they are just completely unresponsive.
12217:51 <dergoegge> hernanmarino: I am actually not quite sure, i don't think anything would break if we did set those variables there
12317:52 <dergoegge> its a review question that i was asking myself and i don't have a definitive answer yet :D
12417:52 <dergoegge> lightlike: sounds reasonable
12517:52 <lightlike> glozow: yes I think so! It's 15 minutes + X, with X being close to 5 minutes currently if they sync from genesis.
12617:53 <sipa> Historically, I think the case is just that it was never intended that we'd be syncing headers from non-nSyncStarted peers.
12717:53 <hernanmarino> ohh, okey. Actually this question is related to my doubt I mentioned earlier so, I'm not sure
12817:53 <juancama> For the second part of question 5, do we not increase nSyncStarted and set fSyncStarted = true and update m_headers_sync_timeout if we start syncing headers with peers that announce a block to us via an inv because it would lead to even more wasted bandwidth?
12917:53 <sipa> But it turned outs that (a) we actually do and (b) this is actually somewhat advantageous because it prevents a situation that the singular chosen nSyncStarted peer is malicious/broken/infinitely slow, and stalls your headers sync forever.
13017:54 <sipa> (I can say this as the person who first introduced the headers syncing and nSyncStarted concept, until not too recently I wasn't actually aware that we'd start fetching headers from other peers as well if they'd announce a new block to us)
13117:55 <dergoegge> sipa: yea i think this happens somewhat by accident, there is a comment about reorgs but not about starting initial sync
13217:55 <larryruane_> Just for my historical understanding, it used to be that header and block download would proceed simultaneously (with headers obviously being ahead of blocks in height) ... then some years ago it was changed to download only headers until we're close to today, then start downloading blocks (this makes a lot more sense)
13317:55 <dergoegge> sipa: thanks for the explainer, that serves as an answer to me!
13417:56 <larryruane_> (i say makes more sense because the old way, we could end up downloading a bunch of blocks that turn out not to be part of the best chain)
13517:56 <dergoegge> lets use the last 5 minutes for approach discussion: An alternative to the approach taken in the PR would be to add an additional headers sync peer after a timeout (fixed or random). What is the benefit of the approach taken in the PR over this alternative?
13617:57 <larryruane_> less duplicate (wasted) bandwidth?
13717:57 <dergoegge> i think a fixed timer for adding new peers would have roughly the same bandwidth usage
13817:58 <dergoegge> suhas argues on the PR that peers that announce an inv to us have a higher probability of being responsive
13917:58 <lightlike> I think one benefit is that the peer that manages to send us the block inv first is often also a very fast peer. So we'd not pick another slow peer if for some reason our initial peer is slow.
14017:59 <dergoegge> lightlike: +1
14117:59 <glozow> after this timeout, would we accept any number of peers or just 1 peer?
14217:59 <adam2k> I think the benefit over the alternative suggestion is that we would reduce bandwidth usage over adding additional header sync peer?
14318:00 <lightlike> (but that could also be achieved in other ways, e.g. picking a fast peer after 10 minutes)
14418:00 <dergoegge> glozow: i meant just one peer
14518:01 <dergoegge> oh that's time! feel free to stick around and discuss some more
14618:01 <dergoegge> #endmeeting