An autonomous system
(AS) is a
collection of connected Internet Protocol (IP) routing prefixes under the
control of one or more network operators (typically ISPs or large companies) on
behalf of a single administrative entity or domain that presents a common,
clearly defined routing policy to the internet.
An autonomous system number (ASN) is a globally unique identifier allocated
to each AS for use in BGP routing. The ASN uniquely identifies each network on
the Internet.
Border Gateway Protocol (BGP) addresses
the routing of packets among different autonomous systems to connect them. BGP
uses ASNs to uniquely identify each AS.
BGP hijacking (sometimes
referred to as “prefix hijacking”, “route hijacking” or “IP hijacking”) is the
illegitimate takeover of groups of IP addresses by the act of corrupting
Internet routing tables maintained using the Border Gateway Protocol (BGP).
Due to the non-uniformity of IP distribution among ASNs, bucketing 8 outbound
connections by /16 prefix network groups may result in connecting to 8 peers
from just 2 large ASNs.
The idea is that allowing nodes to connect to each globally unique ASN only
once should increase the security of the Bitcoin network by diversifying
connections. With this PR, instead of connecting to possibly as few as 2 ASNs,
nodes would connect to 8 different ASNs.
Diversifying network connections is motivated by the Erebus
Attack. The word Erebus is ancient Greek for
“shadow” or “darkness” and underscores the attack’s stealthy nature.
Resources for related network attacks (Eclipse
attack, BGP
hijacking) are included below.
Instead of relying on the /16 IP prefix to diversify the connections every
node creates, this PR proposes to rely instead on IP-to-ASN mapping, if the
mapping is provided.
The asmap is the IP-to-ASN mapping in the form of a .map file named
“ip_asn.map” by default. It would be created by every user independently based
on a router dump or provided along with the Bitcoin release. This PR currently
generates an asmap under 2MB in size with this Python
script by
Pieter Wuille.
Users would be able to toggle asmap use by passing the -asmap= argument when
launching bitcoind or by setting it in the bitcoin.conf configuration file.
“TL;DR. We present the Erebus attack, which allows large malicious Internet
Service Providers (ISPs) to isolate any targeted public Bitcoin nodes from the
Bitcoin peer-to-peer network. The Erebus attack does not require routing
manipulation (e.g., BGP hijacks) and hence it is virtually undetectable to any
control-plane and even typical data-plane detectors.”
What steps did you take to review this PR? Did you try the asmap effect
analysis
script,
custom prints/logging, or modifying the code/tests?
Did you read the Erebus website and paper? Did you study the Eclipse attack
and the BGP hijacking attack?
Briefly describe:
the Erebus attack: can it be detected; is it scaleable against multiple
Bitcoin nodes?
the BGP hijacking attack (Apostolaski et al.)
the Eclipse attack (Heilman et al.)
major differences between the three
What are potential countermeasures to defend against the Erebus attack and
their current implementation status?
This PR claims to increase Bitcoin’s network security by using ASN
information for addrman bucketing and for diversifying peer connections. Do
you agree? Do you see any tradeoffs?
What do you think of the implementation? How should the .map file be
distributed – in the binary or outside it? Should the data be hosted in the
Bitcoin Core repository?
Any thoughts on the test coverage? Do you see anything that is not tested or
could also be tested?
<jonatack> launch bitcoind with : bitcoind -asmap=<path-to>/asmap/demo.map and the log should output "2019-12-11T15:01:16Z Opened asmap file (932999 bytes) from disk."
<jonatack> We usually start Bitcoin Core IRC meetings with a 'hi' so it's clear who's at keyboard. Feel free to say hi, even if you arrive in the middle of the meeting!
<gleb> Pieter also promised to document asmap creation, which would make it more accessible, because right now it's a little bit of a black box, unless you spend a lot of time digging into it.
<gleb> jonatack: No, I think it's pretty short and everything should be understandable. Maybe not very optimized, so it might take some time given 60,000 nodes parameters and 100 experiments
<ethzero> It makes reasoning about addrmans behavior dependent on the data of asmap. For example lets say a person wants to know why their node is only making 6 outgoing connections. In the current world they could just reason about this from the IP addresses they are connected to. Now they have to read the asmap file and understand how the rules are followed.
<gleb> Analysis script is explained in the issue related to the pr. Basically I wanted to show that we're not making a network graph less "random". This might be important for several reasons.
<ethzero> Additionally since this file will be updated, now debugging issues requires know which version they had and if they used a custom asmap file.
<lightlike> I didn't really understand the internal binary asmap representation. aside from documenting the creation script (which afaik won't make it into the bitcoin repo) some documentation in asmap.h/cpp could be helpful.
<ethzero> @jnewberyI think logging would definitely help. One could imagine a bug in this system that requires reading the asmap to understand the behavior is not correct. Right now I could look at addrman log file and identify an issue without any additional context.
<gleb> ethzero: I see your point. It would be awesome if you provide a bit more extensive feedback in the PR body about that. Like, what kind of logs you would want to see? Something like "Okay, we have 1,000 addrs in database, but managed to connect to only 6 due to this and that"
<jonatack> _andrewtoth_: I think minimum once per release depending on how the it is distributed, but users should be able to generate their own asmap and use it.
<_andrewtoth_> and going further, could the asmap that shipped with bitcoin (if that happens) become out of date enough that an attacker could abuse knowledge of what most bitcoin nodes use
<gleb> Talked to matt a bit. I think the understanding is that asmap will be pretty stable, with exceptions minor enough so we won't be exposed to any big threats.
<gleb> _andrewtoth_: Yeah, so it's an open question, but our current intuition is *no*. Worst case, we get bad actors a little bit more connectivity than average.
<jonatack> i found suhas' comment on that interesting: "dishonest peers can gain a connectivity advantage by locating themselves in small AS groups, that seems potentially problematic"
<ethzero> Getting an ASN requires filling out a request form. If you just spam them it will not get approved. I have no done it myself but my impression is that it would be cheaper for an attacker to just rent IPv4s in a bunch of different ASes.
<gleb> nehan: It's probably a good idea to have a script of comparing 2 asmaps (or dumps used to create them) and detect things like this. Like, if there is 10% growth in ASNs over 6 months, it's probably suspicious.
<BlueMatt> its not perfect, but its better than today - /X is a completely bogus metric, asns are better. asns are sybilable (you really just need to create a corporate entity to get one), but we can also filter a bit more
<lightlike> if the bitcoin devs would create one with each release, the creation script would probably have to become more sophisticated to detect tampering like artificial creation of many new ASNs
<jonatack> practicalswift also proposed applying a whitelisting approach: allow only AS number ranges that have been been allocated to the five Regional Internet Registries (AFRINIC, APNIC, ARIN, LACNIC and RIPE) by IANA
<jonatack> Question: It's opt'in for now, but should all 8 outbound connections use the asmap? lukejr suggested using it for only 4 of the 8, for instance.
<BlueMatt> and its not like aws is famous for good latency performance (in fact, usually the opposite, as hosting companies go, aws is one of the worst for latency)
<survey> BlueMatt said another way: it's not believed that latency will materially affect block relay by more than current targets .5%? .005*10*60 = 3 seconds
<BlueMatt> survey: I mean I havent done any kind of formal analysis, but this doesnt change the fact that we dont do any kind of optimizing to connect to nearby/local peers
<survey> BlueMatt true there isn't code to promot local peering but is it not the case that due to previous prefix filtering network topology could produce topology that achieve network performance based on local peers and now due to ASN filtering nodes need to connect to peers in which higher latency will occur? or is your point that you think the marginal change is negligibile? I see the change having sticky dynamics
<survey> my point being: making ASN filtering must be factored into peering for P2P robustness but, let's consider how it'll affect network topology and mitigate as much as possible
<BlueMatt> survey: right, if I had to bet, Id think this biasing against the mega-providers means network-wide you *will* see higher latency, but in fact this is the same effect we're targeting (so we cant avoid it). we want more diverse peers, which means we want more latency for mega-provider nodes.
<BlueMatt> well-laid-out networks re *always* going to win, by a lot, and the p2p network shouldnt be trying to compete, it should be trying to be robust.
<jonatack> What I found interesting in prepping to review this PR is the amount of domain learning to do. I tried to convey some of that in the prep notes for this session.
<pinheadmz> re: testing - is there really any way to test "big picture" of the PR locally? I see the unit tests in C, but I guess we couldn't really add pyhton e2e tests since all peers would have the same IP :-) 127.0.0.1 ...
<gleb> Addrman handling Tor is not very healthy right now — Tor takes 16 buckets, meaning that there is a chance connecting only to Tor. That's pretty bad, because Tor is very sybillable. I hope to get to that somewhere later in Winter..
<survey> has anyone done any research on ASN deduping? e.g. single entities having several ASN registered to them (bit beyond my knowledge) E.g. before a merger two telecomms used two ASNs but now they merged and though they have seemingly distinct ASNs they're beholden to a single corporate entity
<pinheadmz> jonatack: I meant the tor relay nodes whose IPs are known in order to route messages - if I create a circuit I want to ensure that all 3 hops are different ASNs
<jonatack> nehan: your question probably excludes it, but in this PR, IP-to-ASN bucketing is off by default unless the user specifies it and provides a valid map file
<gleb> They also might do 0/10 in 10 years both, if everything is *completely* useless w.r.t diversity. But it seems unlikely to me that asmap can be worse than /16.
<gleb> The implementation of asmap integration is actually not that difficult — just require a bit of learning how AddrMan serialization currently works.
<jonatack> gleb: i find this to be a really interesting PR. were you considering it before the EREBUS paper was published, or was that paper the impetus to do it?
<survey> are there any design decisions that'll be made that are specific to bitcoind that wouldn't be applicable to other projects trying to use this effort?
<gleb> Thank you, it's been a bit like a sprint. Hope there was something useful. More people knowing these things can already contribute to p2p improvements we're trying to push forward.
<jonatack> everyone: next week we will tentatively be looking at a PR to improve valgrind integration into Bitcoin Core, to help detect issues like the one that recently caused the 0.19.0.1 patch release.