fetch multiple headers in getblockheader() (rpc)

https://github.com/bitcoin/bitcoin/pull/25261

Host: larryruane  -  PR author: natanleung

The PR branch HEAD was 053ccf0468e477283e80f78cc095ffb83bff9b95 at the time of this review club meeting.

Notes

  • The getblockheader RPC returns a block header, given its hash. The header data is returned as a JSON object (verbose=true, default) or in raw hex form (verbose=false).

  • The REST interface provides another way to query a bitcoind node. It’s not enabled by default; specify the bitcoind -rest command-line option or rest=1 in the config file to enable this service.

  • The REST interface also provides a blockheader endpoint to fetch block headers; on mainnet, try:
    curl -s localhost:8332/rest/headers/00000000000000000006c042058f7ff60003ae9a96ca2ac3065d91221b00f547.json
    

    This returns five block headers beginning with the specified block hash. You can specify the number of headers by appending ?count=nnn to the URL. The maximum number of results is 2000.

  • This PR proposes to allow the getblockheader RPC to return more than one header by adding an optional count argument, bringing it in line with the functionality offered in the REST interface.

Questions

  1. Did you review the PR? Concept ACK, approach ACK, tested ACK, or NACK?

  2. The new count argument is placed after the verbose argument. Why?

  3. Suppose you do not want to specify the verbose argument (that is, you prefer the default), but you do want to specify count. Is there a way to do that?

  4. Why is the type of the count argument an RPCArg::Type::AMOUNT rather than a RPCArg::Type::NUM as would seem more natural?

  5. The default number of headers to return is 1, yet there is a difference between specifying count=1 and not specifying a count. What is this difference? Why do these behave differently, and should they?

  6. Why is the count limited to 2000? Do you agree with this limit? What are the tradeoffs?

  7. What does this call to EnsureAnyChainman do? Why are the Ensure* family of functions often used in RPC handlers?

  8. Why does the PR modify client.cpp? What are the meanings of the values in the vRPCConvertParams table?

  9. Does the getblockheader RPC work on a pruned node? Why or why not? How does this compare with the getblock RPC?

  10. Why is getblockheader’s block specification argument a hash instead of a height? Related: How does the RPC determine the next header in the series (if more than a single header is being returned)? (Hint: how is this loop advanced?) What happens if you specify a block hash that isn’t part of the active (“best”) chain?

  11. Bonus question: The PR calls CChain::Next() without cs_main being held. Is this safe?

Meeting Log

  117:00 <pablomartin> hello!
  217:00 <stickies-v> hi everyone
  317:00 <LarryRuane> hi all, welcome to PR Review Club. Feel free to say hi to let people know you're here
  417:01 <brunoerg> hi!
  517:01 <Lov3r_Of_Bitcoin> hello
  617:01 <b_101> hi everyone!
  717:01 <svav> Hi all
  817:02 <LarryRuane> I'm having intermittant internet connection problems, so if there's a delay, that's probably why!
  917:02 <LarryRuane> Is anyone here for the first time?
 1017:02 <BlueMoon> Hello!!
 1117:02 <willcl_ark> Hi!
 1217:02 <LarryRuane> perhaps @stickies-v can take over if I disappear?
 1317:02 <LarryRuane> Feel free to ask questions at any point if anything is unclear. We're all here to learn together!
 1417:02 <LarryRuane> #startmeeting
 1517:03 <LarryRuane> https://www.irccloud.com/pastebin/p1n6FYfc/
 1617:03 <stickies-v> sure, i'll step in if we don't hear from you for too long
 1717:03 <BlueMoon> Thanks!!
 1817:03 <LarryRuane> oops if that formatted strangely, sorry about that
 1917:04 <LarryRuane> By the way, let me know if you're interested in volunteering to host a review club meeting, let me know!
 2017:04 <LarryRuane> Notes and questions are in the usual place: https://bitcoincore.reviews/25261
 2117:04 <LarryRuane> Sorry for the delay in getting the notes and questions posted
 2217:05 <LarryRuane> We’ll use those questions to guide the conversation, but feel free to jump in at any point if you have questions or comments
 2317:05 <LarryRuane> Also if we've moved on to question N, it's fine to continue discussing questions less than N
 2417:05 <LarryRuane> ok,
 2517:05 <LarryRuane> Who had a chance to review the PR and notes/questions this week? (y/n)
 2617:06 <stickies-v> y
 2717:06 <enel> Hi! This is my PR. Thanks for taking it on.
 2817:06 <Lov3r_Of_Bitcoin> Yes Concept Ack
 2917:06 <LarryRuane> enel: welcome! very glad you're here! Thank you!
 3017:06 <svav> I read the notes
 3117:06 <b_101> y/y concept Ack
 3217:07 <LarryRuane> great! Before we get into the questions, any comments or questions about the Notes themselves?
 3317:07 <willcl_ark> Yeah seems like a nice PR to help align the RPC and REST interfaces
 3417:09 <brunoerg> Concept ACK
 3517:09 <LarryRuane> Did anyone have a chance to run the REST interface, in particular the `headers` request?
 3617:09 <brunoerg> y
 3717:09 <stickies-v> yup!
 3817:10 <LarryRuane> I found that it helps to pipe the REST output into `jq` by the way, great tool in case anyone here isn't aware of it
 3917:10 <LarryRuane> (otherwise it can be difficult to read, since it's all on one long line)
 4017:11 <LarryRuane> Okay, let's get into the questions. Did you review the PR? Concept ACK, approach ACK, tested ACK, or NACK?
 4117:11 <LarryRuane> willcl_ark: +1
 4217:11 <stickies-v> Concept ACK
 4317:11 <brunoerg> Concept ACK
 4417:12 <b_101> Concept ACK
 4517:12 <Lov3r_Of_Bitcoin> Concept Ack
 4617:12 <willcl_ark> Concept ACK
 4717:13 <LarryRuane> cool, me too, I think this one should be pretty uncontroversial
 4817:13 <LarryRuane> question 2 - The new count argument is placed after the verbose argument. Why?
 4917:14 <Lov3r_Of_Bitcoin> So as to not interfere with previous implementations (?)
 5017:14 <stickies-v> backwards compatibility! people not aware of the update would be passing what they think is their verbose argument into the count parameter, which could fail silently or explicitly
 5117:15 <LarryRuane> Can you elaborate on "interfere"?
 5217:15 <brunoerg> stickies-v: +1
 5317:15 <LarryRuane> stickies-v: yes! -- @Lov3r_Of_Bitcoin I think that's what you were getting at too
 5417:16 <Lov3r_Of_Bitcoin> LarryRuane so old implementation can still be called
 5517:16 <Lov3r_Of_Bitcoin> what stickies-v said :)
 5617:16 <LarryRuane> Any time we add an argument to an RPC, it should always come at the end, and it should have a default -- yes exactly, so that if there are scripts or whatever, they continue to work (assuming the default is the old behavior)
 5717:17 <LarryRuane> Q3 - Suppose you do not want to specify the verbose argument (that is, you prefer the default), but you do want to specify count. Is there a way to do that?
 5817:17 <stickies-v> I don't agree that new parameters need to have a default - it can be preferable to not have a default and make the RPC call fail explicitly, e.g. to avoid unsafe behaviour
 5917:18 <stickies-v> it depends on the parameter/change, really. but yes generally if it is safe we should strive to not have new parameters break existing infrastructure too much
 6017:18 <LarryRuane> stickies-v: ok I hadn't thought of that, good point
 6117:19 <stickies-v> (and we always have the -deprecatedrpc` startup options to temporarily revert new behaviour)
 6217:20 <LarryRuane> Oh I'm not familiar with -deprecaterpc, can you elaborate?
 6317:21 <LarryRuane> that's a config (`bitcoind` command-line or config file) option?
 6417:21 <b_101> :stickies-v +1
 6517:23 <stickies-v> if we introduce a non-backwards compatible rpc change (e.g. deprecating a method, or some parameter(s)), users can temporarily revert to the old behaviour by starting bitcoind with `-deprecatedrpc=<method_name>` to use the previous version of that rpc, until it is usually completely removed in the next major release
 6617:23 <LarryRuane> stickies-v: +1 thanks!! TIL
 6717:24 <LarryRuane> so back to Q3 - Suppose you do not want to specify the verbose argument (that is, you prefer the default), but you do want to specify count. Is there a way to do that?
 6817:24 <brunoerg> `-depracatedrpc` can be specified multiple times, right?
 6917:24 <stickies-v> brunoerg: yes
 7017:24 <glozow> hi
 7117:25 <LarryRuane> I would guess some users may specify that and then forget about it ... until OOPS, now it's gone!
 7217:25 <willcl_ark> you can use `bitcoin-cli -named arg1=x arg2=y`
 7317:25 <LarryRuane> glozow: hi, thanks for dropping in!
 7417:25 <LarryRuane> willcl_ark: +1 yes ... I found some discussion of this here: https://github.com/BlockchainCommons/Learning-Bitcoin-from-the-Command-Line/blob/master/04_3_Creating_a_Raw_Transaction_with_Named_Arguments.md#43-creating-a-raw-transaction-with-named-arguments
 7517:26 <stickies-v> willcl_ark +1. and hopefully soon, you'll also be able to combine named and positional arguments for more ease-of-use: https://github.com/bitcoin/bitcoin/pull/19762
 7617:26 <LarryRuane> although it's referring to creating a raw transaction, `-named` works with any RPCs
 7717:27 <LarryRuane> stickies-v: +1 thanks, i'll add it to my review list!
 7817:27 <LarryRuane> note that `bitcoin-cli` arguments that start with a dash are for `bitcoin-cli` itself ... without the dash, they're just passed on (with any remaining arguments) to `bitcoind`
 7917:27 <willcl_ark> stickies-v: oh that's a cool PR!
 8017:29 <LarryRuane> ok let's go to the next question (but again, feel free to keep discussing previous things): 4 - Why is the type of the count argument an RPCArg::Type::AMOUNT rather than a RPCArg::Type::NUM as would seem more natural?
 8117:29 <LarryRuane> this link may help https://github.com/bitcoin-core-review-club/bitcoin/commit/053ccf0468e477283e80f78cc095ffb83bff9b95#diff-decae4be02fb8a47ab4557fe74a9cb853bdfa3ec0fa1b515c0a1e5de91f4ad0bR506
 8217:29 <b_101> stickies-v: yes that PR is great!, but I noticed it has been there for long time
 8317:30 <brunoerg> AMOUNT we can pass a float number instead of only int one?
 8417:31 <willcl_ark> 2 years are rookie numbers in Bitcoin Core (sometimes!) :P
 8517:31 <b_101> willcl_ark: lol
 8617:31 <brunoerg> I confess `RPCArg::Default{"null"}` is new for me, haven't seen it before for AMOUNT..
 8717:31 <LarryRuane> brunoerg: yes, allows a float, but since it's a count there's no need for that
 8817:32 <enel> This was a while ago, but I think from the PR discussion you can see the reviews called for "null". And I used AMOUNT to take in both "null" string and int values. This may not be the correct RPCArg::Type.
 8917:32 <b_101> LarryRuane: +1 , ciuld't find why use AMOUNT
 9017:32 <willcl_ark> I'm curious about the answer to this one... I couldn't immediately see why
 9117:33 <brunoerg> LarryRuane: Yes, for this logic I couldn't understand as well
 9217:33 <brunoerg> I thought it could be related to this "`null` by default", but not sure if it makes sense
 9317:33 <stickies-v> "-named` works with any RPCs" that's mostly but not entirely true, there are a few exceptions to this, e.g. "bitcoin-cli -netinfo -named level=1" does not work, you need to use positional "cli -netinfo"
 9417:34 <stickies-v> "-named" btw is purely handled by the cli tool, and is actually unrelated to the RPC server
 9517:34 <Lov3r_Of_Bitcoin> enel  is the use of AMOUNT allowing to pass the string “maximun: 2000”
 9617:34 <Lov3r_Of_Bitcoin> ?
 9717:34 <LarryRuane> stickies-v: great, thanks!
 9817:35 <willcl_ark> stickies-v: until the PR you linked...
 9917:35 <LarryRuane> I think the reason it's an AMOUNT is that an AMOUNT can be a string or a number ... and the reason we want it to be able to be a string is ... hint see the JSON spec https://www.json.org/json-en.html
10017:36 <stickies-v> willcl_ark: no I don't believe #19762 fixes that, as it doesn't affect `NetinfoRequestHandler`
10117:36 <LarryRuane> it is confusing though, maybe there should be a STRORNUM type or something, to use in this case instead of AMOUNT
10217:37 <brunoerg> seems not intuitive for me having an `AMOUNT` that could be a string
10317:37 <stickies-v> LarryRuane: so you're saying we use AMOUNT because we want to be able to pass both 2 and "2" (with and without quotes)?
10417:37 <LarryRuane> I think we try to make RPC arguments compatible with JSON (?)
10517:37 <willcl_ark> but numbers are compatible with JSON?
10617:37 <LarryRuane> stickies-v: yes ... and why is that?
10717:38 <b_101> brunoerg: +1
10817:38 <willcl_ark> So because we want legacy behaviour to use `null` rather than `0` (internally, but unlikely to come from the user) then we need to use an AMOUNT?
10917:38 <stickies-v> I'm not sure about that. When using the CLI, we would already automatically convert 2 into a NUM type. When using the RPC programatically, I think there's no reason to not specify the field as an integer?
11017:39 <LarryRuane> willcl_ark: yes exactly!
11117:39 <brunoerg> now I am understanding the `RPCArg::Default{"null"}` haha
11217:39 <LarryRuane> If you read the discussion in the PR, initially the default was going to be zero, but that would be strange because it would return one entry
11317:40 <LarryRuane> someone suggested making the default "null" -- which is a JSON-recognized token by the way
11417:40 <stickies-v> but you don't need to be able to pass a "null" string, you can just pass an actual JSON null object?
11517:40 <theStack> stickies-v: -netinfo is not a real RPC though, it's afair a pure bitcoin-cli helper that calls RPCs internally (like getnetworkinfo) and displays it in a user-friendly way
11617:40 <theStack> (hi and sorry for being late btw... still didn't get the time change :X)
11717:41 <LarryRuane> stickies-v: i'm pretty sure you do have to pass "null" but it's the default, so most people won't need to ever do that
11817:41 <stickies-v> well yes you need to pass null but it shouldn't be a string, it should just be an actual null object?
11917:42 <LarryRuane> how do you specify a null object?
12017:42 <LarryRuane> (on the command line)
12117:43 <LarryRuane> maybe "{}" .. that's an empty object, but this is a value, not an object
12217:43 <stickies-v> `bitcoin-cli method_name` null - if method_name isn't expecting a string then I'm pretty sure null would get parsed into an actual null object and passed to the RPC like that?
12317:43 <stickies-v> `bitcoin-cli method_name null` (sorry bad quoting)
12417:44 <stickies-v> anyway sorry I don't want to derail too much, I'll just try it out with NUM instead of AMOUNT and see where I'm wrong
12517:44 <LarryRuane> I don't think that generates a null object ...it's just the JSON token "null" (which is listed in the JSON spec) ... well this is getting pretty detailed, perhaps we can continue
12617:46 <LarryRuane> stickies-v: no that'sokay! If you can figure that out, I'd love to know, that would be better
12717:47 <LarryRuane> Q5 - The default number of headers to return is 1, yet there is a difference between specifying count=1 and not specifying a count. What is this difference? Why do these behave differently, and should they?
12817:48 <willcl_ark> specifying a count gets you an array response, whereas without a count gets you the legacy "flat" response
12917:48 <willcl_ark> in the case of `count=1` this means an array with one object, vs a flat response with 1 object if the parameter is omitted
13017:49 <LarryRuane> willcl_ark: yes, exactly, if the RPC always returned an array, even if it contained one entry, that wouldn't be backward-compatible
13117:50 <LarryRuane> so `count=null` (where null is a string) will return a non-array (single entry, as before), and this is also the default
13217:50 <enel> LarryRuane: yes, the intent was to emulate the REST behaviour
13317:51 <LarryRuane> Doesn't REST return 5 entries by default? (I thought that's kind of strange)
13417:52 <b_101> LarryRuane: I understood the same
13517:52 <stickies-v> LarryRuane: it seems to work fine with NUM. Code change here (https://github.com/bitcoin/bitcoin/pull/25261/files#r1024323103) and then cli works fine like this: `bitcoin-cli -signet getblockheader 00000086d6b2636cb2a392d45edc4ec544a10024d30141c9adf4bfd9de533b53 true null`
13617:52 <LarryRuane> stickies-v: oh that's cool! so that type can be changed (in the PR) to NUM?
13717:52 <stickies-v> yes, I think so
13817:53 <LarryRuane> Great, see, review club rocks! I just tried specifying a count of 1 to REST and it returns an array (with one entry) ... zero isn't allowed
13917:54 <brunoerg> maybe using NUM the default value could be -1? we can use as a "null"?
14017:54 <LarryRuane> stickies-v: That makes sense, actually, since JSON allows the use of null for any value type
14117:55 <stickies-v> LarryRuane: "Doesn't REST return 5 entries by default?" historically speaking, a big raison-d'etre for the REST interface was to make it faster to do a specific set of large requests, by e.g. reducing the amount of serialization needed, and by batching things.
14217:56 <stickies-v> brunoerg: but UniValues are already nullable, so wouldn't it be more explicit to just use null? not a fan of magic values when we can avoid them, personally
14317:56 <LarryRuane> great comments, okay almost out of time, Q6 - Why is the count limited to 2000? Do you agree with this limit? What are the tradeoffs?
14417:56 <LarryRuane> stickies-v: +1
14517:56 <brunoerg> stickies-v: So, we can set null in RPCArg::Default for num?
14617:57 <LarryRuane> null is somewhat like `std::optional` :)
14717:57 <stickies-v> brunoerg: see https://github.com/bitcoin/bitcoin/pull/25261/files#r1024323103 - can use `RPCArg::Optional::OMITTED`
14817:58 <stickies-v> that's equivalent to passing a null UniValue (just figured this out now so may be missing some nuance here)
14917:58 <willcl_ark> I guess the limit will naturally help minimise DoS risks for the server in being asked to return huge amounts of data, but shouldn't be a concern really on the CLI, more for the REST
15017:58 <willcl_ark> (but as we are matching REST here, makes sense)
15117:58 <brunoerg> stickies-v: yea, sorry! i didn't remember OMITTED when thinking about it
15217:59 <b_101> LarryRuane: can we get quickly into: What does this call to EnsureAnyChainman do?
15317:59 <LarryRuane> willcl_ark: yes, I agree with you.. there are really no DoS concerns with your RPC client, it's trusted in many ways anyway
15418:00 <stickies-v> re Q6: you generally also don't want to have huge network responses. clogs up resources
15518:00 <stickies-v> better to send multiple smaller requests
15618:00 <LarryRuane> I'm not sure, does REST allow arbitrary clients? Or is it limited (like RPC)?
15718:00 <LarryRuane> b_101: yes, that's a good question,
15818:01 <willcl_ark> REST is the "safe" interface that can be web-exposed
15918:01 <willcl_ark> AFAIK
16018:01 <adam2k> 👋 hello all
16118:01 <stickies-v> REST is unauthenticated so anyone on the network can query it. RPC requires authentication
16218:01 <LarryRuane> Can anyone explain the `Ensure` functions?
16318:01 <LarryRuane> stickies-v: willcl_ark: got it, thanks
16418:02 <LarryRuane> I think we better officially end here, but feel free to continue discussions!
16518:02 <LarryRuane> #endmeeting
16618:02 <stickies-v> willcl_ark: hmmm, I think safe is a misnomer. People can still use it to DDOS your node etc. It's safe in that it doesn't touch any of your wallet stuff etc, but it's very much not designed to be a public webservice
16718:02 <willcl_ark> hello adam2k !
16818:02 <LarryRuane> hi adam2k!
16918:02 <b_101> For what I understood digging into the code has to do to make sure we are using the most work chain, am I right?
17018:02 <adam2k> oh shoot...am I an hour late because of the time change 😰
17118:03 <willcl_ark> I meant what you said stickies 😋
17218:03 <LarryRuane> adam2k: Yes I think you're not the only one!
17318:03 <adam2k> LarryRuane ah ha!  I'll update my calendar entry for this.
17418:03 <stickies-v> thanks for hosting us today LarryRuane ! and thank you enel for the PR 👍
17518:04 <willcl_ark> I think the Ensures ensure you always get something back, be that the object you wanted, or a JSON error?
17618:04 <stickies-v> adam2k: not sure which calendar you're using, but some (e.g. google calendar) actually support the UTC timezone so you don't need to change it on every DST change
17718:04 <LarryRuane> As I understand it, the `Ensure*` functions ensure that the needed object is instantiated, and if not, throws an exception ... you don't want an RPC client to crash the node, so we don't want it to be an assert
17818:05 <brunoerg> b_101: I think `EnsureAnyChainman` allows to get a NodeContext to be used in EnsureAnyChainman to get a ChainstateManager
17918:05 <LarryRuane> stickies-v: +1 that's what I did on my google calendar, made the timezone UTC (which doesn't shift)
18018:05 <brunoerg> it's gonna return an error if it wasn't able to find it
18118:05 <glozow> thank you LarryRuane! was lurking heh
18218:06 <LarryRuane> brunoerg: actually it will throw an exception -- so node doesn't crash
18318:06 <adam2k> stickies-v Thanks!  I'll update it.
18418:06 <b_101> brunoerg: LarryRuane: thx!
18518:06 <LarryRuane> you're all very welcome, thanks for being here!!
18618:06 <enel> thanks everyone for the input
18718:07 <brunoerg> LarryRuane: yea! `throw JSONRPCError(RPC_INTERNAL_ERROR, "Node chainman not found")`
18818:07 <b_101> enel: thanks for the PR!
18918:07 <willcl_ark> Thanks all!
19018:07 <brunoerg> thanks LarryRuane for hosting it!
19118:09 <LarryRuane> sure, if anyone's still interesting, I like Q9 - Does the getblockheader RPC work on a pruned node? Why or why not? How does this compare with the getblock RPC?
19218:10 <pablomartin> thanks Larry!
19318:13 <brunoerg> LarryRuane: It works! pruned nodes download and store all headers
19418:15 <LarryRuane> brunoerg: yes exactly! `getblock` requires the full blocks, so will it work at all on pruned nodes?
19518:17 <b_101> LarryRuane: +1
19618:23 <LarryRuane> Q11 - Bonus question: The PR calls CChain::Next() without cs_main being held. Is this safe?