The PR branch HEAD was 3684fb7ae at the time of this review club meeting.
Notes
The REST interface
is a lightweight interface that serves public data over HTTP on the same port
as the JSON-RPC interface. It can be enabled using the -rest config option.
Endpoint examples:
Query transactions by their ID: /rest/tx/<TX-HASH>.<bin|hex|json>
What are blockfilters and what are they used for (hint: see BIP157 and BIP158)
Can you explain what REST is?
What are the main differences between the JSON-RPC and REST interface?
The JSON-RPC interface is already capable of serving blockfilters, why do we
want this ability for the REST interface?
There is a NACK
(#23259) on the PR
suggesting that the REST interface should be removed entirely in favour
of external proxy servers. Do you agree? Why/why not?
<jnewbery> dergoegge: ah. If you open a PR with just one commit, then github will use the commit log as the PR description by default. I guess that's where it came from.
<jnewbery> I think it's a shame that the github review process de-emphasizes commit logs. It'd be nice to be able to comment on them inline just like for the code. They're important!
<stickies-v> Blockfilters are a replacement to bloom filters that allow light nodes to significantly reduce bandwidth, storage and verification without sacrificing privacy like bloom filters did. A block filter is a compressed list of prevouts and UTXOs in a block
<sipa> svav: i wouldn't call it "representation"; they don't encode the full block - more like a fancy checksum, which allows you to quickly check whether the block may be interesting to you
<larryruane> "..may contain transactions that are interesting .." Also an important goal is to not leak information to the server (bitcoind node) about WHAT we (light client) are interested in, right?
<jnewbery> they're similar to BIP37 bloom filters in as much as they're a probabilistic filter of set inclusion. However, they're different in that everyone uses the same filter for each block, so they don't need to be recalculated for every client
<sipa> with BIP37, the filtering was done on the server side (client gives filter of what they're interested in to server); with BIP157, it's done on the client side (the server gives filter about the block's contents to the client)
<sipa> there is also a technical difference, in that BIP37 uses a bloom filter, while BIP157 uses a golomb-coded filter; the difference between those is mostly an implementation detail (bloom is bigger but faster to update)
<jnewbery> For anyone interested in learning about the Bitcoin Core implementation of block filters, we did a whole series of review clubs on them. Just look in https://bitcoincore.reviews/meetings/ for anything with "BIP 157" in the title
<stickies-v> it's also great for scaling, because now each full node only has to calculate one filter per block, whereas previously it would have to spend additional resources for each bloom filter (which was unique per light client)
<larryruane> malicious servers can't make things appear to be present in the block that aren't (since client should always download block and check), but server could *withhold* items (from the filter), I think?
<stickies-v> larryruane yes, but that's where block filter headers come into play. You should connect to multiple nodes, check if they all provide the same block filter headers (they commit to block filters), and if you see different filters amongst nodes investigate. You then also verify that the filter header matches the filter. So quite easy to catch attackers as long as you're onnected to one hoenst node
<stickies-v> A stateless representation of resources, typically (but I think not always?) communicated over HTTP with HTTP status codes. The stateless part basically means that the webserver has no state, i.e. each request is fully self contained and does not depend on e.g. previous requests. This makes for much more easily scalable services, since it doesn't matter which of many servers handles your request - there is noo state
<stickies-v> The REST api only offers a subset of functionality of the full fledged JSON-RPC. It is unauthenticated (but still weirdly somewhat trusted, so don't go ahead and open it up to the internet) and meant to expose public data in an easy and fast way.
<larryruane> One big difference probably is that REST (the way bitcoind uses it) is read-only (query various stuff), while RPC can change things (like submit transactions)
<glozow> I have a dumb question, I've never built a web app. I thought REST was just a concept, not itself a communication protocol or tool. if someone says "REST API" is that shorthand for "our web API which is RESTful" or?
<dergoegge> larryruane: exposing the content of your mempool is v bad for privacy, an attacker could figure out which transactions are broadcast by you
<jnewbery> I also think that it's just generally not designed to be exposed on a public network. If you wanted to serve REST clients on a public network, you should put the bitcoind server behind a proxy.
<Kaizen_K_> So why are there these two interfaces, that seem like they shouldn't even be exposed publicly, are available to use instead of just one? Is it a performance issue?
<sipa> michaelfolkson: obviously; you can't compare REST and JSON-RPC as concepts on their own, they're not comparable (one is a principle for client/server communication, the other is a specific protocol)
<sipa> < larryruane> I haven't checked but I assume the REST interface can't be used to extract secret material (such as keys)? <-- absolutely; only public data is available through REST
<dergoegge> just in short my understing of the two interfaces: The JSON-RPC interface is used to control your node through the “bitcoin-cli” binary while the REST interface is there to serve public data (blocks, txs, etc) to a trusted caller.
<jnewbery> sipa: right. I retract my statement about bitcoin-cli being the most common client. It's only the most commonly used client amongst Bitcoin Core developers.
<stickies-v> dergoegge REST apis are super easy to consume, especially with all the tooling built around it (e.g. OpenAPI, although I've not seen an OpenAPI spec for bitcoin yet). I think developer ease of use is the main reason?
<stickies-v> does anyone know if there has been any discussion around making an OpenAPI spec for the REST API? If not, I could look into contributing that
<jnewbery> The REST interface should also be more performant (at least in theory), since it can return binary data. The json-rpc interface always serializes its returned data into json text. If your application is going to immediately deserialize that, then it's unnecessary overhead.
<Kaizen_K_> larryruane: thats what I'm wondering too, I wonder if this is a scalability issue, RPC creates more network traffic through authentication? Rest maybe reduces that?
<Kaizen_K_> TIL: ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It's fast enough to be the
<Kaizen_K_> fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems.
<larryruane> Kaizen_K_: "are super informative" Yes, and be aware that you can go back and read all the old ones! (which I haven't done enough myself!)
<dergoegge> ok last question: There is a NACK (#23259) on the PR suggesting that the REST interface should be removed entirely in favour of external proxy servers. Do you agree? Why/why not?
<stickies-v> dergoegge I mostly agree. If we're trying to keep core reviewable and maintainable, I see this as an easy to carve out some code - even though it is tiny. Like Jeremy proposes, I think it would make sense to have a separate project to run a full fledged REST server (ideally with optional auth to access the full feature set that JSON RPC represents). However, I don't think the REST functionality is a particular
<michaelfolkson> Other than additional maintenance why not just add an external proxy server as an additional option rather than as a replacement for REST?
<stickies-v> stickies-v no I don't think it should stop this PR, this is a useful addition. I think they're separate. This PR is about getting the functionality up to date, Jeremy's PR is about carving out that functionality into a different project I think?
<jnewbery> the responsibility of the Bitcoin Core project ends at the interface. How those interfaces are consumed/used is the responsibility of the client user/application
<stickies-v> I'm not sure if anyone has any thoughts on this, but my Approach NACK is because I don't think <COUNT> should be a path parameter in `GET /rest/blockfilterheaders/<FILTERTYPE>/<COUNT>/<BLOCK-HASH>.<bin|hex|json>` - it's not restful.
<stickies-v> Instead, I think this should be a query parameter, for example `GET /rest/blockfilterheaders/<FILTERTYPE>/<BLOCK-HASH>.<bin|hex|json>?count=<COUNT>`. Thoughts?
<larryruane> BTW in case anyone hasn't tried fetching a block filter yet, I ran `bitcoin-cli getblockfilter 00000000000000000006f9a460e2f86f4262d8970902f7f38b0f86bf08bfc898` and got the error `Index is not enabled for filtertype basic`
<michaelfolkson> stickies-v: I think suggest it as a change and then if the author doesn't want to make the change it is up to you whether you would rather the PR wasn't merged because of it (whether it is worthy of an Approach NACK)
<larryruane> so i added `blockfilterindex=1` to my config file, restarted, and now it's building these filters, starting from block 0, in the background (still adding new blocks)
<jnewbery> stickies-v: you should definitely raise specific objections in review. I find that the word NACK tends to antagonize people, so I use it sparingly and only when I think the PR is harmful to the project.
<stickies-v> michaelfolkson sipa jnewbery got it, thanks for the advice on how to approach this- that makes sense! Will start with the suggestion first.