Reduce addr blackholes (p2p)

https://github.com/bitcoin/bitcoin/pull/21528

Host: amitiuttarwar  -  PR author: amitiuttarwar

The PR branch HEAD was 0c194ff at the time of this review club meeting.

Notes

  • The Bitcoin network uses addr messages to communicate network addresses (the locations of nodes). See the Bitcoin wiki p2p documentation for more details.

  • Good address propagation improves network connectivity and increases the difficulty of executing an eclipse attack.

  • Bitcoin Core nodes will periodically self-announce (also known as self-advertise) their own network address to peers.

  • When a Bitcoin Core node receives an addr message that contains 10 addresses or fewer, it forwards them to 1 or 2 peers, selected at random. If we assume all nodes do this, then self-announcements should reach a large portion of the nodes on the network.

  • However, we know that some nodes on the network do not relay addr messages that they receive. Two known cases are block-relay-only connections from Bitcoin Core nodes, and connections from certain light clients. We refer to these connections as addr black holes. addr messages go in, but they never escape!

  • If a large portion of the connections on the network are addr black holes, then addr propagation is impacted. Self-announcements won’t reach a majority of nodes on the network. It’d be better if we could somehow avoid picking black holes as the 1 or 2 peers that we select for relaying addr messages to.

  • This PR defers initialization of m_addr_known of inbound peers until the peer sends an address related message (addr, addrv2, getaddr or sendaddrv2). The node uses the presence of m_addr_known to decide whether the peer is a candidate for relaying addr messages received from the network.

Questions

  1. Did you review the PR? Concept ACK, approach ACK, tested ACK, or NACK? What was your review approach?

  2. What is an addr black hole? Why is this a concern for addr propagation?

  3. How does Bitcoin Core implement self-announcements? How would you expect a single advertisement to propagate throughout the network?

  4. How does this PR propose to improve addr black holes? What are possible issues an approach like this could have? What does this approach not address?

  5. What are the addr related p2p messages in the bitcoin protocol? What do each of them mean? How does the logic to send and receive each message change with this PR?

  6. Could this change potentially be disruptive to other software on the network? How so?

  7. What are alternative approaches for achieving the intended goal?

Meeting Log

  119:00 <amiti> #startmeeting
  219:00 <jnewbery> hi!
  319:00 <amiti> hi!
  419:00 <glozow> hi
  519:00 <petroleum> hi
  619:00 <ccdle12> hi
  719:00 <genef> hi
  819:00 <svav> hi
  919:00 <tkc_> hi
 1019:00 <amiti> welcome everyone!
 1119:00 <b10c> hi
 1219:00 <lightlike> hi
 1319:00 <amiti> anyone here for the first time?
 1419:01 <ivanacostarubio> hi
 1519:01 <emzy> hi
 1619:01 <amiti> some reminders - all questions are welcome, we're here to learn :)
 1719:01 <amiti> and feel free to ask them whenever, no need to ask to ask
 1819:02 <amiti> who got a chance to review the PR this week? (y / n)
 1919:02 <genef> y
 2019:02 <b10c> y
 2119:02 <svav> y
 2219:02 <ccdle12> y
 2319:02 <jnewbery> (notes and questions here: https://bitcoincore.reviews/21528)
 2419:02 <jnewbery> y
 2519:02 <lightlike> y
 2619:02 <amiti> wow lots of review!
 2719:02 <ivanacostarubio> y
 2819:02 <sishir> y
 2919:03 <glozow> y
 3019:03 <amiti> first question: Did you review the PR? Concept ACK, approach ACK, tested ACK, or NACK? What was your review approach?
 3119:03 <emzy> n
 3219:04 <amiti> a good time for general thoughts about the PR :)
 3319:04 <ccdle12> concept ACK
 3419:04 <genef> Concept ACK, light code review
 3519:04 <b10c> concept ack
 3619:04 <ivanacostarubio> concept ACK.
 3719:04 <schmidty> y
 3819:05 <jnewbery> Concept ACK, approach ACK
 3919:05 <amiti> would anyone like to summarize the goal of the PR?
 4019:05 <amiti> related to question 2: What is an addr black hole? Why is this a concern for addr propagation?
 4119:05 <svav> Prevent eclipse attacks
 4219:05 <sishir> in some nodes set of addresses will go in but do not escape, they do not propagate in time
 4319:05 <ccdle12> reliable propagation of addr announcements
 4419:06 <amiti> svav: indeed, the goal of good addr relay is to be robust to eclipse attacks, can you describe more about what part specifically this PR contributes to?
 4519:06 <amiti> sishir, ccdle12: yes!
 4619:06 <petroleum> Does this PR *reduce the occurrence* of addr blackholes or is it more /reducing addr data propagation overhead/?
 4719:06 <glozow> i thought of black hole just in relation to our node (they wont forward addrs we send them) and not necessarily that they dont participate in addr relay at all
 4819:06 <petroleum> seems more the later, to me
 4919:07 <amiti> glozow: good clarification! yes they could be either
 5019:07 <amiti> does anyone want to answer petroleum's question ?
 5119:07 <ivanacostarubio> Make sure we won't send addr messages to peers that won't relay those messages
 5219:07 <amiti> ivanacostarubio: yup
 5319:08 <sishir> I thought this feature is not relaying addr at all to block-relay-only nodes and light clients so reduce occurrence?
 5419:08 <amiti> sishir: exactly
 5519:08 <amiti> petroleum: does that make sense?
 5619:08 <petroleum> amiti so the later part of my sentence?
 5719:08 <glozow> neither
 5819:08 <petroleum> e.g. communication overhead reduction
 5919:08 <petroleum> oh
 6019:09 <amiti> oh I see what you mean
 6119:09 <petroleum> I don't see how it's exclusively the later and none of the former (in my original question)
 6219:09 <amiti> I guess latter, but I'd phrase it differently
 6319:09 <petroleum> block-relay-only will always be an addr black hole
 6419:09 <petroleum> got it
 6519:09 <lightlike> I'd say that it does reduce the occurrence of black holes, because we relay a given ADDR to a limited number of nodes, and those messages that aren't sent to black holes will be sent to other nodes instead.
 6619:10 <amiti> lightlike: great description. thanks!
 6719:10 <petroleum> lightlike yeah good perspective
 6819:10 <amiti> ok, so lets dig in to how / why
 6919:10 <amiti> How does Bitcoin Core implement self-announcements? How would you expect a single advertisement to propagate throughout the network?
 7019:11 <sishir> Q. Does advertising mean that node is putting itself out to get connected to?
 7119:11 <svav> Periodic self announcement by nodes
 7219:12 <ccdle12> in SendMessages, our node will check if the peer can relay addrs, we are not in IBD and if the `m_next_local_addr` is expired
 7319:12 <amiti> sishir: yes
 7419:13 <genef> peer self-announces, addr gets fanned out by those nodes, then those nodes send off again to another send of nodes, repeat
 7519:13 <genef> set* of nodes
 7619:13 <amiti> ccdle12: yup! I believe you're referring to the periodic self announcements we initiate in SendMessages
 7719:13 <svav> Internode communication is dependent on Bitcoin protocol version
 7819:13 <amiti> that can be found here: https://github.com/bitcoin/bitcoin/blob/b14462083f82aeaa9a376978f210db5538db296f/src/net_processing.cpp#L4198-L4214
 7919:14 <svav> What bit of code determines node self announcement? How often does self announcement happen?
 8019:14 <amiti> genef: that relay pattern is true, but do you know what the "fan" factor is?
 8119:15 <amiti> svav: see the code I just linked :)
 8219:15 <glozow> not sure if someome already mentioned, we do one after connecting (outbound) to a node and receiving `VERSION` from them: https://github.com/bitcoin/bitcoin/blob/267b60f800cb298d6700ae54fdace595c0e80313/src/net_processing.cpp#L2435-L2458
 8319:15 <amiti> glozow: yes!
 8419:16 <amiti> so, there are two code paths that initiate self announcements, which glozow & I have just linked
 8519:16 <amiti> 1. when we receive a `VERSION` message from a outbound not-block-relay-only peer, we will announce our address
 8619:16 <lightlike> svav: once a day on average, AVG_LOCAL_ADDRESS_BROADCAST_INTERVAL = 24h in net_processing.cpp
 8719:16 <jnewbery> gleb had a PR to consolidate those two self-announcement mechanisms a bit: https://github.com/bitcoin/bitcoin/pull/19843 . The PR needs a bit of love and rebase now.
 8819:17 <amiti> 2. what lightlike just said, once a day on average per peer
 8919:17 <amiti> so, lets understand these two a bit better- why would we announce our own address after we've connected to a peer?
 9019:17 <glozow> is addr self-announcement on a poisson timer for privacy or for fanciness or?
 9119:18 <b10c> it could be that we just joined the network and nobody knows us yet?
 9219:18 <sishir> to ensure that newly connected node becomes well known and better connected
 9319:19 <amiti> glozow: good question. let me ask you one in return- is privacy a concern when announcing your address?
 9419:19 <amiti> b10c, sishir: right, the behavior is asymmetric. if node A connects to node B, node B might not actually know node A's address
 9519:20 <amiti> so node A announces
 9619:20 <sishir> Does newly connected node sends addr & getaddr at the same time?
 9719:20 <amiti> after that announcement, what would we expect the propagation pattern to look like on the network ?
 9819:20 <amiti> (that announcement, or any announcement really)
 9919:21 <b10c> amiti: oh right, because with TCP you connect FROM a different port that you are listening on (and B doesn't know if you are listening at all).
10019:21 <glozow> amiti: i can't really think of what you'd want to hide when self-announcing
10119:22 <amiti> sishir: relevant code here https://github.com/bitcoin/bitcoin/blob/b14462083f82aeaa9a376978f210db5538db296f/src/net_processing.cpp#L2435-L2464, the logic paths are fired at the same time, but we send the getaddr right away, and queue up the addr for a bit later (when we process in SendMessages)
10219:22 <sugarjig> Wouldn't node B only forward the addr message to 1 or 2 other nodes?
10319:23 <b10c> amiti: B sends addr-of-A to other nodes
10419:23 <amiti> b10c: exactly
10519:23 <amiti> glozow: me neither!
10619:23 <amiti> sugarjig: yup
10719:24 <b10c> but if all B's would send it to their peers we'd flood, right? that's why we only send to 1 or 2 peers
10819:24 <amiti> b10c: I don't understand this statement
10919:25 <amiti> so, a node initiates a self announcement, and then when a peer receives the announcement it forwards the address to 1-2 peers
11019:25 <amiti> see https://github.com/bitcoin/bitcoin/blob/b14462083f82aeaa9a376978f210db5538db296f/src/net_processing.cpp#L2680
11119:25 <amiti> and "addr black holes" are when it doesn't actually forward it
11219:26 <amiti> and this was already mentioned, some reasons for that could be: its a block-relay-only connection or its a light client
11319:26 <genef> why don't block-only relays participate in addr forwarding?
11419:26 <amiti> genef: good question, anyone know the answer?
11519:27 <svav> When you create a node, how does it determine the first address of another node to connect to?
11619:27 <genef> svav: DNS seeds
11719:27 <amiti> so, the reason black holes could be an issue is if the addresses are not really getting propagated well around the network
11819:27 <b10c> amiti: to rephrase RE propagation pattern: A self-announces to B, B only relays it to 1 or 2 peers as otherwise (compared to relay to e.g. all 50 peers) we'd flood the network with addr's, right?
11919:27 <emzy> amiti: to prevent mapping the network.
12019:28 <amiti> b10c: yup. correct
12119:28 <ccdle12> couldn't an attacker determine a block-relay-only node by seeing that they don't forward addrs?
12219:28 <sishir> genef why don't block-only relays participate in addr forwarding? +1
12319:29 <glozow> because block-relay-only is block-relay-only
12419:29 <genef> glozow: guess that makes sense, thought it just referred to they don't relay tx. really does mean "block"-only, lol
12519:30 <lightlike> ccdle12: block-relay-only is not a node property, it's a connection property, currently a node has only 2 outgoing block-relay only connections. you could think of it as an extra stealthy network within the network.
12619:30 <sishir> I though addr was just used for propagation? What do they do with the addr then
12719:30 <emzy> Via addr forwardings you can figure out which node is connected to which. Blocks-only prevents to find out about all connections of a node.
12819:31 <amiti> some good questions and answers here :)
12919:31 <sishir> lightlike âś…
13019:31 <amiti> addr forwarding leaks some info about node topology, so block-relay-only connections wanted to avoid that entirely
13119:31 <glozow> note difference between `-blocksonly` mode and a block-relay-only connection
13219:32 <amiti> ok, so hopefully this is making sense so far
13319:32 <amiti> lets move on to the next question:
13419:32 <amiti> How does this PR propose to improve addr black holes? What are possible issues an approach like this could have? What does this approach not address?
13519:32 <genef> maybe unrelated to this pr: could addr be forwarded using a Dandelion++-like protocol to obscure the origin-dest?
13619:32 <emzy> glozow: good point. I was talking about block-relay-onl
13719:32 <genef> ^background q
13819:32 <glozow> genef: why do we want to obscure origin of addr?
13919:33 <genef> this PR doesn't send to nodes that don't participate in addr propagation
14019:33 <amiti> so let's clarify this: when relaying addrs, what relevant information is private vs public?
14119:33 <genef> glozow: for the same reason block-only don't propagate, prevent addr mapping
14219:33 <sishir> IP address
14319:34 <amiti> so the contents of the addr message is my ip address / location
14419:35 <amiti> I want this to be public information
14519:35 <amiti> that's why I'm sending it out
14619:35 <amiti> but, the pattern of how it gets sent out might reveal node topology, aka which peers I am connected to
14719:35 <amiti> and that is something I want to keep private
14819:36 <amiti> because if an attacker knows the network topology, it could make an attack like causing a partition tangibly easier
14919:36 <amiti> does this make sense to people?
15019:37 <ivanacostarubio> https://developer.bitcoin.org/reference/p2p_networking.html#addr
15119:37 <genef> yes
15219:37 <sugarjig> Yes!
15319:37 <sishir> YES!
15419:37 <ivanacostarubio> Yes. It makes sense
15519:37 <amiti> awesome!
15619:37 <amiti> cool, so lets go back to question 4:
15719:37 <amiti> How does this PR propose to improve addr black holes? What are possible issues an approach like this could have? What does this approach not address?
15819:37 <sishir> Postpones initialize of m_addr_known until peers sends an address related message
15919:39 <amiti> sishir: yes! and then what do we do with that information? (of whether m_addr_known is initialized)
16019:39 <lightlike> for outbound connections, nothing changes
16119:39 <amiti> lightlike: good observation :)
16219:40 <sishir> for inbound initialize filter when we get addr messages
16319:40 <amiti> yup
16419:40 <b10c> does not change: one (or multiple) mallilicous peers could still be a addr-blackholes
16519:41 <amiti> b10c: correct
16619:41 <glozow> i also like that the PR makes nodes stop sending `SENDADDRV2` to block-relay-only peers
16719:41 <amiti> so, we defer initializing `m_addr_known` for inbound peers until they send us a message that has to do with addrs
16819:42 <amiti> how does this link back to not sending to black holes ?
16919:42 <sugarjig> If we've never gotten an addr-related message from a peer, there's a good chance they could be a black hole
17019:42 <genef> consider an inbound peer a black hole until they send addr info?
17119:42 <amiti> yup, exactly!
17219:43 <oldgoat5> "How does this PR propose to improve addr black holes?" - this pr appears to add a SetupAddressRelay flag, which can be set to true for full nodes, and false for light clients.  Currently some nodes are not likely to forward addresses (light clients), thus some announcements will be lost.  This pr wants nodes to declare whether they will forward
17319:43 <oldgoat5> a relay or not, so the source nodes can skip light clients.
17419:43 <oldgoat5> is this correct^?
17519:43 <amiti> oldgoat5: mostly, but some clarifications: 1. there are other types of connections that won't forward addresses, eg. block-relay-only conn or potentially other software on the network
17619:44 <amiti> 2. we use a heuristic to set the flag, and the heuristic is whether the conn is outbound, or inbound & send addr- related message
17719:45 <lightlike> where did a typical light client that doesn't participate in addr relay but somehow managed to connect to us get our IP? From the DNS seeds?
17819:45 <amiti> lightlike: great question, I don't know. does anybody else know ?
17919:47 <ccdle12> maybe from websites like bitnodes? an edge case but they must have a db of node ips
18019:47 <amiti> ok, we can keep this as an open question and keep moving :)
18119:47 <amiti> lets discuss this part of the question: What are possible issues an approach like this could have?
18219:48 <amiti> lightlike: you already brought up a potential problem at the bitcoin-dev meeting last week :)
18319:48 <genef> could prematurely exlude nodes from addr relay
18419:48 <b10c> I think BTCPayServer's NXExplorer does connect via P2P to your node and doesn't have anywhere to forward addr's
18519:48 <jnewbery> lightlike: good question! Maybe it connected to some hard-coded peers, sent a getaddr to get a diversity of peers and then connected to some of them (?)
18619:49 <amiti> genef: yes exactly
18719:50 <svav> How many other nodes is a given node typically connected to?
18819:50 <sugarjig> A node may not have any inbound peers that have announced an addr message, so could itself be a black hole
18919:50 <amiti> and that would suck, because this is the main technique for ongoing addr relay, so if a node doesn't hear about addrs, it would be more vulnerable to being eclipsed
19019:50 <amiti> sugarjig: not quite, what about outbounds?
19119:51 <genef> could it attempt to reseed from DNS peers again?
19219:51 <amiti> svav: default values in bitcoin core are 8 outbound full relay, 2 block relay only peers, 125 total
19319:51 <amiti> but there is also other software on the netwrok
19419:51 <sugarjig> amiti: yep, if it has outbounds, then we're good
19519:52 <amiti> sugarjig: oh interesting. I don't think bitcoin core can run on an inbound-only method. I'd have to check that you can't do something weird with startup flags though
19619:53 <amiti> genef: yes, but unlikely. we also have other methods for getting addresses such as sending GETADDR when we connect to outbound peers
19719:53 <genef> +1
19819:53 <amiti> so, I think the biggest concern with this proposal is that we don't accidentally exclude nodes from addr relay that are depending on it
19919:54 <amiti> one piece of feedback that has been given to me is that I should communicate about this on the bitcoin-dev mailing list & research the expectations of other software on the network
20019:55 <amiti> we can reasonably build confidence around behavior of bitcoin core nodes, but thats not sufficient
20119:55 <amiti> so this kinda covers question #6 around how we could disrupt other software
20219:55 <ccdle12> curious, what are the other pieces of software on the network that needs to be taken into account?
20319:56 <amiti> we have 5 minutes left, which I don't think is enough to dig into question #5 or 7, so I'll open the floor to any remaining questions about addr relay
20419:56 <glozow> i have a clarification question, if you're about to relay addrs and you have 0 candidates (e.g. no m_addr_known with any of your peers) for any of your non-source peers, do you also just not relay it?
20519:56 <amiti> ccdle12: anything else running bitcoin protocol. there are some open source ones and there would be private ones too.
20619:57 <svav> Why do you want to cut-off sending to blackhole addresses completely? Is that necessary?
20719:57 <genef> do you think a DHT or using dandelion++ could help with obfuscating network topology?
20819:57 <amiti> glozow: yes I believe so
20919:57 <amiti> glozow: we could edit the tests to see, but pretty sure the `sort_func` in `RelayAddress` would just come up empty
21019:57 <sishir> I gotta head out but thank you amiti! Learned a lot today
21119:57 <svav> Isn't it only a problem if a node sends messages out to ALL blackholes? So, can you just write the code to prevent this?
21219:57 <amiti> genef: what is DHT?
21319:57 <glozow> distributed hash table
21419:58 <genef> distributed hash table ^
21519:58 <amiti> genef: dandelion could definitely help obfuscate network topo through tx relay
21619:58 <lightlike> this touches #7, but I'm really interested in it: have you considered the alternative approach treating addr messages to potential black holes as additional messages (but not stopping them) - e.g. relaying to one more peer if we suspect our peer is a black hole?
21719:58 <amiti> oh, um, I guess depends on how it was used??
21819:58 <amiti> lightlike: great question! and yes, I'm trying to think that through right now
21919:59 <amiti> on one hand, that change can be considered "safer" because the observable addr propagation on the network shouldn't change
22019:59 <svav> Yes, it's not a problem if you send a msg to a blackhold, but it is if you are sending to only blackholes, right?
22120:00 <amiti> on the other hand, that behavior might be exploitable because you're relying on the peer to indicate to you whether or not they are a black hole, and maybe that could lead to them receiving more addrs than otherwise?
22220:00 <amiti> ok! that's time! thanks for playing everyone :)
22320:00 <amiti> #endmeeting