Periodically make block-relay connections and sync headers (p2p)

https://github.com/bitcoin/bitcoin/pulls/19858

Host: jnewbery  -  PR author: sdaftuar

The PR branch HEAD was 93a0ec1a at the time of this review club meeting.

Notes

  • An eclipse attack is when an adversary is able to isolate a victim from the Bitcoin network by taking over all of its connections. We covered eclipse attacks in the review club meetings for PR 16702 and PR 17428.

  • Block-relay-only connections are a type of connection where nodes do not participate in transaction or address relay and only relay blocks. An effective way for a spy node to infer the network topology is to observe the timing and details of transaction and address relay, so these block-relay-only connections obfuscate network topology, helping to mitigate eclipse attacks.

  • Block-relay-only connections were introduced in #15759. After these changes, nodes by default open two outbound block-relay-only connections on startup. We discussed this PR in a previous review club. Refer to the notes and log from that meeting if you’re interested in learning more.

  • This week’s PR, #19858, proposes a more advanced use of block-relay-only connections to further mitigate eclipse attacks. After this PR, the node will periodically initiate an additional block-relay-only connection, and then sync headers to try to learn about new blocks.

  • After the node opens an additional block-relay-only connection to a peer, it begins to sync headers. If this reveals new blocks, the eviction logic will rotate out an existing block-relay-only connection. If no new blocks are discovered, the connection is closed. It’s important for this eviction logic to be carefully reviewed.

Questions

  1. Did you review the PRs? Concept ACK, approach ACK, ACK, or NACK?

  2. Describe a scenario in which periodic block-relay-only connections could help prevent an attack?

  3. What are potential downsides to the concept? Are there any ways it could reduce security guarantees?

  4. What happens if we learn about new blocks when opening one of these new block-relay-only connections?

  5. How does the existing stale tip detection work? What are the differences compared to this mechanism?

  6. How do we maintain the count for block-relay-only connections? What changes in this PR?

  7. At a high level, how does outbound eviction work? What changes in this PR?

Meeting Log

  117:01 <jnewbery> #startmeeting
  217:01 <emzy> hi
  317:01 <felixweis> hi all!
  417:01 <glozow> hi
  517:01 <dhruvm> hi
  617:01 <willcl_ark> hi
  717:01 <jesseposner> hi
  817:01 <blanching> hi
  917:01 <elle> hi
 1017:01 <pinheadmz> ¡hi!
 1117:01 <joelklabo> Hi!
 1217:01 <lightlike> hi!
 1317:01 <sipa> hi
 1417:01 <michaelfolkson> hi
 1517:01 <amiti> hi
 1617:01 <jnewbery> welcome all!
 1717:01 <dappdever> hi
 1817:01 <jnewbery> great turnout today. Must be a popular PR!
 1917:01 <jnewbery> Notes and questions are here: https://bitcoincore.reviews/19858
 2017:02 <jnewbery> anyone here for the first time?
 2117:02 <joelklabo> Me 👋
 2217:02 <glozow> it's because jnewbery is a popular person
 2317:02 <jnewbery> Hi joelklabo. Thanks for joining us!
 2417:03 <jnewbery> Normal reminders: feel free to ask any questions at any time. We're all here to learn and all are welcome
 2517:03 <jnewbery> don't ask to ask, just ask
 2617:03 <ajonas> Hi
 2717:03 <blanching> welcome joelklabo
 2817:03 <jnewbery> amiti prepared some notes and questions for this meeting, which we'll use to guide the conversation, but feel free to jump in at any time
 2917:03 <jnewbery> who had a chance to review the PR (y/n)?
 3017:03 <jesseposner> y
 3117:04 <joelklabo> y
 3217:04 <pinheadmz> y just on github
 3317:04 <willcl_ark> a brief skim over the top...
 3417:04 <lightlike> y
 3517:04 <sipa> y
 3617:04 <dappdever> y
 3717:04 <felixweis> n
 3817:04 <jonatack> hi
 3917:04 <michaelfolkson> y
 4017:05 <emzy> n
 4117:05 <amiti> 0.5 y
 4217:05 <jnewbery> ok, any first thoughts about the PR? concept ACK/NACK? What's it trying to achieve?
 4317:05 <pinheadmz> more eclipse attack mitigation, more condfidence that the node is on the mnost work chain by explosring the p2p netowrk
 4417:06 <jonatack> y. i've been collecting data from a node running the branch with custom logging. need to analyze it further, but connectivity does seem improved.
 4517:06 <dappdever> Are block-relay-only connections less observable than address and transaction relays?
 4617:06 <jnewbery> pinheadmz: exactly right!
 4717:06 <pinheadmz> seems like overkill to me but i dunno if its a nack
 4817:06 <joelklabo> periodically checking for new block-relay-only peers to make sure your not only connected to dishonest peers
 4917:06 <blanching> pinheadmz: what's "explosring"?
 5017:06 <emzy> concept ACK. Sounds good to me.
 5117:06 <murch> hi
 5217:07 <pinheadmz> blanching connecting to more nodes and checking for more-work blocks
 5317:07 <michaelfolkson> Overkill as in we don't need to worry about eclipse attacks pinheadmz?
 5417:07 <jnewbery> *exploring
 5517:07 <dhruvm> (1) Mitigation against eclipse attacks by periodically creating new relay-only connections to look for higher PoW elsewhere and (2) Honest peers helping bridge other trapped honest nodes.
 5617:07 <pinheadmz> michaelfolkson overkill in that we already have block-only connections and 8 peers
 5717:07 <glozow> it could be better for eclipse attacks, provided you have a healthy/diverse addrman
 5817:07 <michaelfolkson> It is incremental, sure. But incremental better than nothing
 5917:07 <jnewbery> dappdever: yes, relaying transactions and addrs reveals information about the network topology, which could be used by a spy to map your connections
 6017:08 <jnewbery> There have been a couple of other PR review clubs about the concepts: https://bitcoincore.reviews/15759 https://bitcoincore.reviews/17428
 6117:08 <michaelfolkson> Can we just double check I have terminology right? Stale tip is when your tip has a new block on top of the previous tip right?
 6217:08 <michaelfolkson> *I
 6317:09 <michaelfolkson> Nothing to do with orphan blocks or anything like that
 6417:09 <pinheadmz> stale tip means no new blocks in a while
 6517:09 <pinheadmz> orphan blocks like the name suggests menas a block with no parent
 6617:09 <elle> even if an attacker can map your connections, how can they actually perform the eclipse attack? how can the attacker get you to disconnect you other peers and connect to them instead?
 6717:09 <michaelfolkson> Just missing the latest block?
 6817:09 <pinheadmz> doesnt happen anymore since headers first
 6917:09 <lightlike> I'm still a a bit sceptical: if your addrman has been corrupted, the additional periodic block-only connections don't seem to help much (because you likely won't connect to a good peer). But if your addrman is healthy, how did you get eclipsed in the first place?
 7017:09 <joelklabo> I was curious why block-only peers reveal less about network topology but I'll check if that was covered in the previous reviews
 7117:09 <jnewbery> michaelfolkson: that's a really great question. Terminology is important here, and people often get it wrong or get confused
 7217:09 <glozow> lightlike: that's my bit of skepticism as well
 7317:10 <jnewbery> This is a good summary of what is meant by stale and orphan blocks: https://bitcoin.stackexchange.com/a/5869/26940
 7417:10 <michaelfolkson> Great, thanks
 7517:10 <pinheadmz> elle i think a real eclipse involves filling up a peers addrman with "poinson" entries and just waiting and waiting for those to get rotated in, or possibly joinging with a DoS attack to force the node to restart ?
 7617:10 <pinheadmz> *poison
 7717:11 <jnewbery> I'd say that 'stale' doesn't mean the block is old. It means that we know about a competing chain with more work
 7817:11 <elle> oh! ok cool. thanks pinheadmz :)
 7917:11 <michaelfolkson> lightlike, glozow: I think you are thinking in boolean rather than on a continuum. Your addrman isn't either perfectly pure or perfectly poisoined. It could be partially poisoned or under attack
 8017:12 <michaelfolkson> Any bolstered defence is better than nothing
 8117:12 <dhruvm> Since certain nodes are probably larger targets for eclipse attacks, small incremental benefits for them should have significant network benefits?
 8217:13 <jnewbery> Right, defense in depth. But I think a good question is whether the potential improved protection is worth the additional complexity
 8317:14 <amiti> joelklabo: I gave a talk about eclipse attacks & it covered some of the differences between relaying txns / blocks / addrs. if you're interested: https://www.youtube.com/watch?v=H-wH6mY9pZo&feature=youtu.be&t=252
 8417:14 <jnewbery> We've already started touching on the next question: Describe a scenario in which periodic block-relay-only connections could help prevent an attack?
 8517:14 <joelklabo> thanks amiti, I'll check that out
 8617:14 <sipa> one argument may be the symmetry: we create extra normal connections that may get promoted to normal ones... it feels natural to do the same with blockonly connections
 8717:15 <jnewbery> Thanks amiti! That was a great talk.
 8817:15 <jesseposner> This paper has a detailed description of an eclipse attack: https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-heilman.pdf
 8917:15 <ajonas> Would a case be when the node has been eclipsed but their addrman still has honest addresses in it?
 9017:16 <dappdever> A node could receive block-relay-only blocks that reveal the legit chain that was attempting to be obfuscated by an attacker
 9117:16 <amiti> building on the conversation so far- I guess if an adversary has been trying to poison your addrman, has taken over some of it eg. 60%. you could happen to have all your outbound connections to them, but then rotating through some of the others connects you to the honest network?
 9217:16 <jnewbery> ajonas: yes, I think that's the scenario where this would help.
 9317:16 <joelklabo> addrman, that's address manager I'm guessing?
 9417:17 <glozow> yup
 9517:17 <dhruvm> amiti: is the opposite true as well? if you are eclipsed, this makes it more likely that an honest node reaches out?
 9617:17 <jnewbery> joelklabo: right. It's where we store addresses of potential peers.
 9717:17 <willcl_ark> If we are worried about having a poisoned addrman, and then with a feeler we detect that we're on a stale tip, would we not want to promote that new ("honest") peer to a NODE_NETWORK peer, rather than blocksonly?
 9817:17 <sipa> ip addresses, to be clear; not bitcoin addresses
 9917:17 <pinheadmz> so this PR introduces kind of a slow churn through the addrman
10017:17 <jnewbery> sipa: thanks for the clarification!
10117:17 <willcl_ark> unless blocksonly also relays addrs?
10217:18 <amiti> dhruvm: ah fair, but I think it would be harder to be eclipsed if you're also enabling inbounds? although maybe not that much harder if you were just spinning up 🤔
10317:18 <pinheadmz> if theres at least one honest peer in there well eventually connect to it and learn the truth about reality
10417:18 <jnewbery> willcl_ark: no. Just blocks. We want block-relay-only peers to be as undetectable as possible, so they don't take part in address relay
10517:18 <jnewbery> (I assume you mean block-relay-only peers, not -blocksonly?)
10617:18 <dappdever> Where do the block-relay-only inbound connections come from? How do they discover your address initially?
10717:19 <willcl_ark> jnewbery: hmmm. So we would be OK with a "bad" Addrman as long as we keep receiving good blocks?
10817:19 <willcl_ark> (and yes that's what I mean!0
10917:19 <pinheadmz> dappdever all inbound connections just come from your public IP being gossiped by peers.
11017:19 <sipa> dappdever: they're just normal nodes, and their IP address gets rumoured
11117:19 <pinheadmz> from your nodes perspective, the block only peers that matter are always outbound conenctions
11217:19 <dhruvm> amiti: That means this PR helps even when your adddrman is 100% poisoned?
11317:19 <dappdever> got it, thanks
11417:20 <sipa> dappdever: it's not that these are secret nodes; just rhe existance of the connection that is less observable
11517:20 <joelklabo> sipa, what does rumoured mean?
11617:20 <jnewbery> Just so everyone is clear, -blocksonly mode and block-relay-only peers are two different concepts. Again, I think it's important to be precise with terminology because they mean different things and it's easy to get confused between them otherwise.
11717:20 <jnewbery> The thread from here lists some differences: https://github.com/bitcoin/bitcoin/pull/20217#issuecomment-727876700
11817:20 <sipa> joelklabo: nodes sharing data with each other... "hey i know a cool IP address of a bitcoin node"
11917:20 <sipa> joelklabo: address relay
12017:21 <joelklabo> I see, thanks
12117:21 <michaelfolkson> How long does it take for a newly mined block to propagate across the network? You would want your block-relay connection to have the chance to give you the latest block rather than being cut off instantly?
12217:21 <sipa> jnewbery: ah yes, thanks - i may be getting the name wrong sometimes
12317:21 <jnewbery> rumour/gossip/relay are mostly interchangeable
12417:22 <jnewbery> propogation/dissemination/promulgation. What a rich language we have.
12517:22 <dappdever> are the rumours just a different message type? Or does it happen on a different protocol?
12617:22 <dappdever> or different socket?
12717:22 <sipa> dappdever: all the same P2P protocol, same connections, same sockets
12817:23 <jnewbery> dappdever: addresses are 'rumoured' or 'gossipped' or 'relayed' using the `addr` message in the P2P protocol
12917:23 <dhruvm> dappdever: https://developer.bitcoin.org/reference/p2p_networking.html#addr
13017:23 <joelklabo> so the nodes you are connecting to are all the same you just ask those two for block only
13117:23 <dhruvm> dappdever: also https://developer.bitcoin.org/reference/p2p_networking.html#getaddr
13217:23 <jonatack> dappdever: you might find helpful the enum class ConnectionType documentation in src/net.h
13317:23 <jnewbery> joelklabo: yes. In fact, you're connecting to them and saying "no transactions please". They may still gossip addresses to you but you'd ignore them
13417:24 <willcl_ark> so what are the assumptions here: we are currently eclipsed, but we have _some_ honest addresses in our addrman. This PR will eventually rotate through enough block-relay-only peers that we will discover a more-work chain, then replace one of our current block-relay-only peers with the new peer?
13517:24 <michaelfolkson> And all connections are taken from the addrman right? If your addrman is 100 percent poisoned and all connections are controlled by adversary you're done unless you restart
13617:24 <jnewbery> willcl_ark: exactly right
13717:24 <jnewbery> michaelfolkson: you're done even if you restart
13817:24 <willcl_ark> jnewbery: ok thanks
13917:25 <jnewbery> because when you restart you'll connect to peers from your addrman
14017:25 <michaelfolkson> jnewbery: Restart with DNS seeds
14117:25 <sipa> willcl_ark: another way to put it is that with more connections, even just occasionally, the threshold of how much of addrman needs to be under attacker control increases
14217:25 <jonatack> michaelfolkson: or addnode an honest peer
14317:25 <joelklabo> is the reason we ask for blocks only to save bandwidth?
14417:25 <jnewbery> I'd need to look at it again, but I don't think we use DNS seeds if our addrman is already full
14517:25 <sipa> joelklabo: less bandwidth, and less leaking of network topology
14617:26 <sipa> joelklabo: from such a block-relay connection, the peer has a harder time inferring who you're connected to
14717:26 <dhruvm> michaelfolkson: If you are 100% poisoned, does this PR increase the chance that an honest peer reaches out if you have inbounds enabled?
14817:26 <jnewbery> joelklabo: I'd say mostly less leaking of network topology. The fact that they use less bandwidth/memory/CPU is a nice property that means we can potentially have more of them
14917:26 <pinheadmz> jnewbery actually that makes me think - what if this PR actually queried the DNS seeds? they are centralized endpoints but are trusted
15017:27 <sipa> pinheadmz: dns seeds are a last resort effort to find peers
15117:27 <pinheadmz> like, use a completely different protocol besides bitcoin p2p to get new peers
15217:27 <willcl_ark> sipa: thats an interesting way to look at it. To be honest, I am kind of suprised nodes don't handshake and request tip hash (or a getheaders) from more nodes before this PR
15317:27 <pinheadmz> sipa yes but they exist outside the p2p network and cant be eclipesed
15417:27 <pinheadmz> unless your DNS resolver is also under attack
15517:27 <joelklabo> harder to infer network topology because just less traffic between the nodes? Sorry if I'm repeating questions
15617:27 <pinheadmz> or a seed maintainer is under attack
15717:27 <michaelfolkson> dhruvm: This PR has no effect on that. But yeah an inbound request could save you generally (I think...)
15817:27 <jesseposner> There are many flavors of eclipse attacks, and I believe that the block-relay-only connections are trying to mitigate against attacks that target the connected peers directly. It doesn't protect against a 100% poisoned addrman state. It simply makes it more difficult to discover all the connected peers.
15917:27 <jnewbery> pinheadmz: all of this stuff is easy if you just trust a centralized authority!
16017:28 <jnewbery> you could just get your blocks directly from sipa
16117:28 <willcl_ark> :D
16217:29 <ajonas> joelklabo - transactions can be fingerprinted to see who is connected to who. It's easier to infer topology through tx relay.
16317:29 <jnewbery> sorry, that was a bit flippant. But using DNS seeds does have a downside. One is that you're potentially telling people that you're running a bitcoin node that you wouldn't otherwise be doing
16417:29 <emzy> I only trust blocks I mined myself ;)
16517:29 <dhruvm> michaelfolkson: ah, ok. i am trying to make sense of https://github.com/bitcoin/bitcoin/pull/19858/files#r483713328
16617:29 <lightlike> I'd hope that it would be much easier to corrupt the DNS nodes (that are on central places with actual, known operators) than our addrman.
16717:29 <pinheadmz> jnewbery point taken! see Vitaliks recent post about "weak subjectivity" i.e. check a block exploerer now and then 😬
16817:30 <jnewbery> emzy: that's an impressive level of self-sovereignty
16917:30 <jnewbery> ok, next question: What are potential downsides to the concept? Are there any ways it could reduce security guarantees?
17017:30 <sipa> after altcoins, egocoins; everyone has their own
17117:30 <ajonas> joelklabo see https://arxiv.org/abs/1812.00942
17217:31 <joelklabo> ajonas, thanks
17317:31 <michaelfolkson> dhruvm: "the behavior introduced here is beneficial to not just the node doing it, but to the whole network, as a node already connected to the honest network that is periodically connecting to new peers to sync tips with others is helping bridge the entire network"
17417:31 <michaelfolkson> dhruvm: I think what Suhas means there is that the fewer nodes that are owned/eclipsed the better for the network
17517:32 <sipa> joelklabo: tx relay and addr relay are relatively easy to use for identifying nodes and connectioms; there are various known techniques, and it's not so surprising... just construct a random tx or addr message, send it somewhere, and see where it appears
17617:32 <sipa> joelklabo: this is not true for blocks, as they're very infrequent by design, and very expensive to construct
17717:32 <sipa> joelklabo: look up txprobe
17817:32 <joelklabo> I see, thanks sipa
17917:33 <ajonas> michaelfolkson dhruvm - well there is also a subtle upgrade to find new block-only peers that are "useful" as in they deliver blocks that our current peers aren't delivering
18017:33 <sipa> jnewbery: reducing open listening sockets on the network seems the most obvious downside
18117:33 <jnewbery> We do make use of techniques to try to mask tx origination (e.g. randomized timer, batching invs), but I think those are really adding statistical noise. With a large enough sample set, I think it would still be possible to map the network
18217:34 <sipa> jnewbery: exactly
18317:34 <jonatack> joelklabo: if you run a node, you can observe the network interactions using -netinfo to help get an intuition for these things
18417:34 <jonatack> watch ./src/bitcoin-cli -netinfo 4
18517:34 <joelklabo> thanks, I'll check that out
18617:34 <pinheadmz> jnewbery i dont see how this could affect security but yeah perhaps privacy
18717:35 <lightlike> one downside would be that an attacker that we connect to and that can send us a new block would replace a potentially honest blocks-only connection - although this would be really costly for the attacker, because they would basically need to be a miner withholding a new block from the network just to eclipse us further.
18817:35 <jnewbery> sipa: yes, that does seem to be the main downside. Also the cost/complexity/maintenance that's associated with any change
18917:35 <michaelfolkson> ajonas dhruvm: So the more an individual node can find useful peers the easier it makes it for everybody else?
19017:35 <pinheadmz> and honestly, if bandwidth were free computaitonally and financially - wouldnt there be a benefit to connecting to all 9,000 nodes?
19117:35 <michaelfolkson> Not for privacy pinheadmz
19217:35 <dhruvm> michaelfolkson: ajonas: How is a node already connected to an honest network, helping bridge the entire network by making new "feeler" block-relay-only connections?
19317:35 <jnewbery> Next question. What happens if we learn about new blocks when opening one of these new block-relay-only connections?
19417:36 <ajonas> If we learn a new block, we rotate out an existing next-youngest block-relay peer in favor of the new peer.
19517:36 <blanching> batching invs only hides data flows / timing analysis which in some domains can obfuscate some topology but weakly, probabilistically (high resolution: eve knows x was sent before y and can conclude A)
19617:36 <amiti> dhruvm, michaelfolkson: block-relay-only connections also help with grander partition attacks than just eclipse attacks (eg. a malicious party trying to create a network split). which is more a network level concern (and everybody loses if that happens)
19717:36 <blanching> not sufficient to obfuscate topology well
19817:37 <jnewbery> blanching: indeed. It can make mapping the topology more costly/difficult but can't prevent it entirely
19917:37 <michaelfolkson> Good point amiti
20017:37 <blanching> +1
20117:37 <dhruvm> michaelfolkson: ajonas: amiti: So that comment is more about block propogation times rather than the honesty of the peer?
20217:38 <ajonas> dhruvm: that would make sense to me
20317:38 <jesseposner> ajonas: +1
20417:38 <michaelfolkson> How long does it take for a newly mined block to propagate across the network? You would want your block-relay connection to have the chance to give you the latest block rather than being cut off instantly?
20517:38 <murch> jnewbery: but without auth, how would you know they're from sipa?
20617:39 <blanching> michaelfolkson IIRC something like 6-18 seconds is considered healthy propagation time
20717:39 <jonatack> murch: i know when i get a block from sipa
20817:39 <sipa> for non-miners the propagation time really doesn't matter
20917:39 <michaelfolkson> So should you wait 18 seconds before kicking out a block-relay-only connection blanching?
21017:39 <sipa> as long as it's not close to interblock time, of course, as you want to see confirmations
21117:39 <blanching> 6-18 seconds. e.g. 1-3% of the average block creation window
21217:40 <jnewbery> ajonas: if we receive a block from the additional block-relay-only peer while we're connected, will we _always_ disconnect the second youngest?
21317:40 <sipa> these improvememts are primarily about making sure you get blocks at all
21417:40 <sipa> not necessarily fast
21517:40 <blanching> michaelfolkson I'd defer to someone else to answer that
21617:40 <willcl_ark> jnewbery: there is a comparison to see who gave us a block most recently too
21717:41 <blanching> sipa: agreed. it's about preventing peer ossification which would decrease attack costs
21817:41 <michaelfolkson> You don't want to kick out your block-relay-only connection for no reason
21917:41 <glozow> could someone elaborate on open listening sockets? how limited is the resource and what's the cost of taking up more?
22017:41 <pinheadmz> glozow there is a maxInbound setting, right?
22117:41 <blanching> for clarity: peer ossifying = decreases attack costs
22217:42 <sipa> glozow: it hasn't been a problem for years, but early in bitcoin's history there was a time when inbound sockets were hard to find
22317:42 <jnewbery> willcl_ark: right, we see which of our two youngest block-relay-only peers (which will be one of the existing block-relay-only peers and the 'additional' block-relay-only peer) gave us the block least recently and consider it for disconnection
22417:42 <ajonas> jnewbery: I guess not if we're under the respective conn limits?
22517:42 <sipa> this was when light clients with permanent p2p connections were far more popular than today
22617:43 <jnewbery> so we're really getting into the meat of the new functionality here. https://github.com/bitcoin/bitcoin/pull/19858/files#diff-6875de769e90cec84d2e8a9c1b962cdbcda44d870d42e4215827e599e11e90e3R3962-R3999 if you want to follow along
22717:43 <sipa> glozow: our prioritization logic for incoming connections makes it also harder to attack, unlesd you have lots of source IP networks
22817:43 <glozow> oh, you mean like, there are fewer total available inbounds in the network? so it'd be harder for a new node to get a good set of outbound connections?
22917:43 <sipa> glozow: yes
23017:43 <willcl_ark> ajonas: I think only if we currently have a feeler active
23117:43 <emzy> But we will free a conection if we find a better block-relay-only peer. So nothing changes.
23217:44 <dhruvm> blanching: are anchors evidence that some ossification is useful?
23317:44 <jnewbery> This function EvictExtraOutboundPeers() is called on a timer every 45 seconds
23417:44 <lightlike> jonatack: In your test on mainnet, how often per hour would you actually succesfully connect to a new outbound peer and compare headers? In a short run on testnet yesterday with a fresh node (no peers.dat), that was rather rare, most of the time I'd get no response and the 5min poisson timer would start again.
23517:44 <willcl_ark> as GetExtraBlockRelayCount will return > 0
23617:44 <sipa> glozow: 128 incoming slots per publicly reachable peer, by default... it's a finite resource that can be used up
23717:45 <sipa> years ago we'd just stop accepting incoming connections when the limit was reached... which made it kind of trivial to attack
23817:45 <jnewbery> willcl_ark: feelers aren't relevant here. GetExtraBlockRelayCount() doesn't count feelers.
23917:45 <emzy> btw. all my nodes have more then 128 connections incoming.
24017:45 <amiti> for the questions around how long it takes blocks to propagate, this is a cool resource:
24117:45 <amiti> https://dsn.tm.kit.edu/bitcoin/#propagation
24217:45 <amiti> https://dsn.tm.kit.edu/bitcoin/videos.html#blocks
24317:46 <willcl_ark> jnewbery: damn my terminology! I meant an extra block-relay-only peer over the limit (of two?)
24417:46 <pinheadmz> emzy would you be willing to share how you host those/ what the cost is?
24517:46 <glozow> sipa: ah okay i see, are inbounds less scarce now? does anyone ever have a problem getting 10 outbounds?
24617:46 <blanching> amiti
24717:46 <jonatack> lightlike: i've been busy with things for the 0.21 release and meaning to get back to the data, but IIRC, using this PR i did see greater peer diversity (e.g. higher peer id numbers) than usual. don't depend on that info though; i need to refresh and look at the data
24817:46 <blanching> amiti i didnt realize they were still updating that thanks
24917:46 <jnewbery> willcl_ark: ah! Terminology is important!!!
25017:46 <sipa> that's no longer the case, but still... if there are more than 128*public_nodes/10 nodes making connections, not everyone would get them
25117:46 <jnewbery> otherwise we're talking about different things and we don't realise it :)
25217:46 <sipa> glozow: yeah, i'm talkimg about 2012 or so
25317:47 <michaelfolkson> Thanks amiti
25417:47 <jonatack> lightlike: i don't remember more but will contact you, if you don't mind, when i look at it
25517:47 <sipa> glozow: i haven't seen evidence of scarcity since
25617:47 <emzy> pinheadmz: all only HDD. Costs are about $10 to $50.
25717:47 <sipa> jnewbery: misunderstandings are the basis of the best inventions
25817:47 <jnewbery> ok, so if we call EvictExtraOutboundPeers(), and we currently have an additional block-relay-only peer (i.e. 3 in total), and the youngest gave us a block more recently than the second youngest, what do we do?
25917:47 <amiti> sipa: 128 incoming? I thought 125 total - 8 (outbound) - 2 (block-relay-only)
26017:47 <pinheadmz> emzy ahhhh HDD, must be slow to sync - but these are VPS? like amazon, digital ocean, etc?
26117:48 <amiti> (searching code now)
26217:48 <sipa> amiti: you're right, i was simplifying
26317:48 <sipa> and/or misremembering
26417:48 <emzy> pinheadmz: some are VPN others are dedicated servers. IBD is like one day.
26517:48 <glozow> sipa: makes sense. i always thought the inbound : outbound ratio seemed really high, and people explained because not everyone enables inbounds, so the people who do have to make up for it.
26617:48 <jesseposner> jnewbery: disconnect the second youngest
26717:48 <lightlike> jonatack: if you had net logging enabled, you could also grep for the new log entries (e.g. "keeping block-relay-only peer=X chosen for eviction")
26817:48 <michaelfolkson> sipa glozow: Something to monitor if Neutrino becomes widely used
26917:48 <jnewbery> sipa: apparently so. The half-dose/full-dose oxford vaccine regime is more effective than the full-dose/full-dose regime and they only found out because they'd accidentally given the half dose to a bunch of people
27017:49 <jesseposner> jnewbery: unless the youngest hasn't given us a block, in which case disconnect the youngest
27117:49 <sipa> jnewbery: erlay was invented by gleb who misunderstood gmaxwell's idea
27217:50 <jonatack> lightlike: yes, i did have it enabled and was grepping for "enabling\|block-relay-only" and -i "enabling\|block-relay-only\|extra"
27317:50 <jonatack> lightlike: (based also on my added logging)
27417:50 <jnewbery> jesseposner: are there any conditions where we wouldn't disconnect either?
27517:50 <lightlike> jonatack: cool, I'd be interested in the statistics once you find the time for that!
27617:51 <jonatack> lightlike: happy to, i'm sure you'll have good ideas for interpreting it
27717:52 <willcl_ark> looks like we disconnect youngest_peer.first as default
27817:52 <willcl_ark> ah, if we are getting a block right now then we might not disconnect either
27917:53 <jnewbery> willcl_ark: youngest_peer.first refers to the peer id. (youngest_peer is a std::pair of peer id and the time of the latest block from that peer)
28017:54 <jnewbery> willcl_ark: yes, exactly. If we're currently downloading a block from that peer, then we won't disconnect it
28117:54 <michaelfolkson> sipa jnewbery: Misunderstandings are great for inventing things. Not so great for analyzing a robustness of a system and avoiding bugs ;)
28217:54 <jnewbery> which makes sense. We shouldn't disconnect a peer if it looks like it's about to give us a block
28317:54 <ajonas> Am I reading this right that we need to have been connected to those peers for more than 30 seconds?
28417:55 <willcl_ark> ajonas: looks like that to me: more than 30 seconds connection time and not currently receiveing a block
28517:55 <jnewbery> ajonas: exactly right. If the timer for EvictExtraOutboundPeers() pops a few seconds after connecting to the additional block-relay-only peer, then we haven't given it enough time to send us a block, so we don't disconnect
28617:56 <jnewbery> Ok, time for one last question. How do we maintain the count for block-relay-only connections? What changes in this PR?
28717:57 <jonatack> lightlike: looking at it right now, a few (~4 to 10) times an hour
28817:57 <jnewbery> hint: we're looking in ThreadOpenConnections()
28917:58 <jnewbery> here: https://github.com/bitcoin/bitcoin/blob/1e9e4b68f3cbd1af3de5806aeff952e9bf0ab8fc/src/net.cpp#L1843
29017:59 <jnewbery> ThreadOpenConnections() is running a loop to open new connections to peers. Whenever it runs through that loop it'll count the number of block-relay-only connections that we have here: https://github.com/bitcoin/bitcoin/blob/1e9e4b68f3cbd1af3de5806aeff952e9bf0ab8fc/src/net.cpp#L1904-L1929
29117:59 <jnewbery> That function is really long and really confusing, but it's essential to understanding how we manage our outbound connections, so definitly worth digging into if this stuff interests you
29218:00 <michaelfolkson> Cool
29318:00 <jnewbery> That's time everyone. Thanks for coming and chatting about p2p! And thanks again to amiti for preparing the notes and questions.
29418:00 <lightlike> jonatack: thanks, that sounds quite good to me, considering that at a 100% success rate we'd have 12 hits per hour on average.
29518:00 <blanching> thanks amiti jnewbery
29618:00 <lightlike> thanks!
29718:00 <jesseposner> Thanks!
29818:00 <ajonas> thanks jnewbery and amiti for notes
29918:00 <jnewbery> next week, glozow is hosting. Don't miss it!
30018:00 <jnewbery> #endmeeting
30118:01 <glozow> thanks jnewbery!!!
30218:01 <willcl_ark> thanks jnewbery!
30318:01 <jonatack> thanks amiti and john!
30418:01 <elle> thanks everyone!
30518:01 <dhruvm> Thank you, jnewbery, amiti and suhas!
30618:01 <jnewbery> thank you glozow!!!!!
30718:01 <amiti> thanks jnewbery, thanks all :)
30818:01 <emzy> tnx amiti and jnewbery
30918:01 <glozow> and thank you amiti!!!!!!!!!!!!
31018:01 <jnewbery> ok time out
31118:01 <jnewbery> enough !s
31218:01 <michaelfolkson> Cut the conversation log