Testnet4 including PoW difficulty adjustment fix (tests, consensus)

https://github.com/bitcoin/bitcoin/pull/29775

Host: fjahr  -  PR author: fjahr

Notes

  • Recommended reading:
  • Bitcoin core supports three test networks out of the box: Regtest, Testnet, and Signet. There also exist some custom Signet variants like Mutinynet. At this point, the current Testnet has been running for 12 years. However, the current Testnet is actually Testnet 3. It was introduced in PR #1392. Documentation on how exactly Testnet 1 and 2 broke is not available but it appears that they fell victim to high fluctuation in mining power. Remember that around this time the first ASIC miners entered the market while Testnet was probably still mostly mined by CPUs and maybe the occasional GPU.
  • Testnet 3 features a Proof of Work exception rule, known as the 20-min exception. This rule was designed to prevent the chain from getting stuck again due to hash power fluctuation. However, a bug in this exception leads to so-called block storms, large numbers of blocks being mined in quick succession. This is the main reason Testnet 3 is so far ahead of mainnet even though it started much later. The bug was recently exploited on purpose for an extended period of time to highlight the issue.
  • Testnet 4 still includes the 20-min exception but adds a mitigation for the block storm issue.
  • The pull request also includes a fix for the timewarp attack, an attack that is still possible on mainnet today. A fix for this was proposed as part of the Great Consensus Cleanup but failed to get the necessary support as a softfork so far.

Questions

  1. Did you review the PR? Concept ACK, approach ACK, tested ACK, or NACK? What was your review approach?
  2. Why reset Testnet in the first place? Were there any arguments against the reset?
  3. What is the message in the Genesis block in Testnet 3 and why (reference the code)?
  4. Aside from the consensus changes, what differences do you see between Testnet 4 and Testnet 3, particularly the chain params?
  5. Pick a single chain param that you don’t know/remember the meaning of. Look up what it does and explain it in one sentence.
  6. How does the 20-min exception rule work in Testnet 3? How does this lead to the block storm bug? Please try to reference the code.
  7. How is the block storm bug fixed in the PR? What other fixes were discussed in the PR?
  8. Why was the time warp fix included in the PR? Hint: This came up in the PR discussion.
  9. How does the time warp fix work? Where does the fix originate from? Can you think of any other ways to fix it?
  10. How do you start your node with Testnet 4? What happens when you start it just with -testnet=1 after Testnet 4 is included?
  11. The PR and ML discussions included many further concerns and ideas that were not addressed in the code of the PR. Pick the one you found most interesting and give a short summary. Do you think this is still a concern and should be addressed?
  12. Do you have ideas for additional test cases? What makes Testnet 4 features tricky to test?
  13. Why is it interesting to embed special scripts into the chain as test cases? What makes this useful beyond bitcoin core?
  14. What expectations do you have for such a change before you would include it in a release? For example, would you reset the genesis block one more time?

Meeting Log

  117:00 <fjahr> #startmeeting
  217:00 <stickies-v> hi
  317:00 <pablomartin> hello
  417:00 <GregTonoski> #29520 add -limitdummyscriptdatasize option - I'm suggesting discussion about that PR in the next Bitcoin Core review monthly meeting, stickies-v and glozow. I'm contacting you in order to host the meeting (per instruction at https://bitcoincore.reviews.
  517:00 <fjahr> Hi everyone, welcome to the PR Review Club on the Testnet 4 PR #29775! I hope you enjoyed reviewing it as much as I enjoy working on it.
  617:00 <glozow> hi
  717:01 <fjahr> I came up with a lot of questions so we will jump right into it :)
  817:01 <lightlike> hi
  917:01 <fjahr> Did you review the PR? [Concept ACK, approach ACK, tested ACK, or NACK](https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#peer-review)? What was your review approach?
 1017:02 <emzy> tested ACK
 1117:02 <fjahr> emzy: whoop :D
 1217:03 <pablomartin> concept ack, light code review... read all relevant readings from the notes (i still need to re-read some)
 1317:03 <stickies-v> concept ACK on resetting testnet, the current testnet issues seem prohibitive for it to be a useful dev tool
 1417:04 <fjahr> pablomartin, stickies-v: cool!
 1517:04 <fjahr> I will get started with the rest of the questions because I think there are some interesting learnings even if you haven't reviewed everything.
 1617:04 <pablomartin> +1 stickies-v
 1717:05 <fjahr> Q: What is the message in the Genesis block in Testnet 3 and why (reference the code)?
 1817:06 <fjahr> by "code" I just mean the two functions involved
 1917:07 <emzy> https://github.com/bitcoin/bitcoin/blob/3714692644f45808a6480525abc36870aeee1de4/src/kernel/chainparams.cpp#L240
 2017:07 <glozow> Isn't it also the chancellor on the brink message/
 2117:07 <emzy> I think it is this.
 2217:08 <emzy> glozow: looks like it.
 2317:08 <abubakarsadiq> hi
 2417:08 <glozow> https://github.com/bitcoin/bitcoin/blob/3714692644f45808a6480525abc36870aeee1de4/src/kernel/chainparams.cpp#L238
 2517:08 <fjahr> Right, it’s the same as on mainnet, this actually surprised me which is why I added the question. Definitely good trivia knowledge ;)
 2617:09 <fjahr> can you read me? I just got a weird error message...
 2717:09 <glozow> fjahr: i can see you
 2817:10 <fjahr> ok, cool, so there are two versions of CreateGenesisBlock, one of which defaults to the chancellor message. I found that interesting :)
 2917:10 <fjahr> Q: Aside from the consensus changes, what differences do you see between Testnet 4 and Testnet 3, particularly the chain params?
 3017:11 <stickies-v> it's got a different genesis message (`testnet4_genesis_msg`)
 3117:12 <fjahr> stickies-v: yepp!
 3217:12 <lightlike> all softforks are already active at height 1.
 3317:13 <fjahr> lightlike: Right! The deployment heights of the past softforks are all set to 1, i.e. they are active from the beginning. While this might seem kind of trivial, these could have also been set to some later value allowing for some potential testing of deployment mechanisms, but there wasn’t that much appetite in that from what I remember.
 3417:14 <glozow> New default port 48333
 3517:14 <stickies-v> different messagestart too, so serialized data can be distinguished between testnets
 3617:15 <fjahr> Yepp, both right, and there are also some differences in the seed nodes and no checkpoints and assumeutxo data (yet).
 3717:15 <fjahr> Next Q: Pick a single chain param that you don’t know/remember the meaning of. Look up what it does and explain it in one sentence.
 3817:15 <lightlike> unrelated question: looking at the existing testnet4 chain, according to mempool.space, blocks 10000 and 20000 were mined just 5 hours apart. Was someone just pointing a ridiculous amount hash power at testnet, or was there still some funny stuff going on?
 3917:17 <emzy> IIRC there was a big reorg. Seems to be related to that.
 4017:17 <fjahr> I don't know about funny stuff, the difficulty has to ramp up initially and if someone pointed an ASIC at the chain that doesn't seem ridiculous. But still interesting to check.
 4117:18 <stickies-v> I had to look up `fPowNoRetargeting` during review, forgot about regtest not doing difficulty adjustments (luckily). So that's what it does: when `true`, don't adjust the required pow difficulty
 4217:18 <glozow> the coinbases do seem to suggest they have a common miner, from a quick glance
 4317:19 <fjahr> I kind of failed at this Q and looked at something that's only in Testnet 3 but I found it interesting: The BIP16 exception (script_flag_exceptions). BIP16 standardized P2SH transactions and defined 3 rules that transactions can not violate. The blockhash in the exception is block 394 in Testnet. I didn’t have time to check which transaction exactly violates which the rules though.
 4417:20 <abubakarsadiq> fjahr: you mean al derived class of `CChainParams` ?
 4517:20 <abubakarsadiq> "a"
 4617:21 <fjahr> Well, yeah, I was thinking of the Testnet 4 chainparams but the learning is more broadly because it may apply to other chains too.
 4717:22 <fjahr> Ok, so let's get to the meat: How does the 20-min exception rule work in Testnet 3? How does this lead to the block storm bug? Please try to reference the code.
 4817:23 <fjahr> (no offence to vegetarians 😉)
 4917:25 <stickies-v> the exception for testnet3 is made in `GetNextWorkRequired` by looking at the timestamp in a block's header: https://github.com/bitcoin/bitcoin/blob/3714692644f45808a6480525abc36870aeee1de4/src/pow.cpp#L26
 5017:27 <fjahr> right, and what is the effect?
 5117:28 <lightlike> if there is no block for 20 minutes, difficulty goes to 1 for the next block. For the ones after the next block, it goes back to whatever the difficulty was before the 20 minutes had passed.
 5217:28 <fjahr> hint: the bip has more information on this, which I added based on feedback after coming up with the questions :)
 5317:29 <fjahr> lightlike: right!
 5417:30 <fjahr> And how does this lead to the blockstorms? Which leads right into the next question: How is that fixed?
 5517:31 <stickies-v> if the last block in a difficulty period is min-difficulty, then the next block (i.e. the first of the next epoch) won't have any "lookback window" to find the true difficulty, so it'll just take the previous difficulty, which is min-difficulty
 5617:31 <stickies-v> https://github.com/bitcoin/bitcoin/blob/3714692644f45808a6480525abc36870aeee1de4/src/pow.cpp#L32
 5717:33 <fjahr> stickies-v: correct, and you already mentioned the lookback window which is coming into play again with the fix currently implemented.
 5817:34 <fjahr> Can someone name the alternative fix that was also contemplated? It can pretty much be derived from stickies answer.
 5917:34 <stickies-v> this can be exploited ad infinitum, right? so i guess the only reason block storms stop is because eventually attacker/trolls just decide to do so?
 6017:35 <fjahr> Yeah, like jameson lopp did recently on Testnet 3. I don't know how long it was, 2-3 weeks maybe? But it only got back to normal because he stopped.
 6117:35 <lightlike> don't you have to wait for 20 minutes regularly to avoid the difficulty from going back up (and in these 20 minutes, someone else could mine a block)?
 6217:36 <fjahr> In testnet 4 you do, in Testnet 3 you don't
 6317:37 <fjahr> (when you are within a blockstorm in testnet 3)
 6417:38 <fjahr> you can reorg someone who is a "fair" miner as well
 6517:38 <lightlike> ah, i see.
 6617:39 <stickies-v> oooh
 6717:39 <fjahr> Alright, the alternative fix I wanted to mention is just disallowing the last block in the difficulty to be min-difficulty. I think almost everyone was kind of indifferent between this and the look-back solution.
 6817:40 <fjahr> Let's talk about the time warp: Why was the time warp fix included in the PR? Hint: This came up in the PR discussion.
 6917:41 <stickies-v> since we're trying to fix it with the great consensus cleanup, test-running it in testnet seems sensible?
 7017:42 <lightlike> because that's an unrelated way to manipulate the block production rate, so why not fix both ways?
 7117:42 <stickies-v> and also it helps prevent block storms so
 7217:43 <fjahr> Right, the 20 min exception exploits in combination with this still are pretty annoying and test running it was the second entry in the pro column :)
 7317:44 <fjahr> And how does the time warp fix work?
 7417:47 <stickies-v> we check that the first block of a new difficulty adjustment period is not earlier than 2h before the previous block
 7517:47 <lightlike> is there any best practice on how to make exceptions for testnet? It used to be a flag (fPowAllowMinDifficultyBlocks), in the PR the genesis block is compared, elsewhere we use "chainparams.GetChainType() != ChainType::REGTEST" - are some ways better than others?
 7617:48 <fjahr> stickies-v: right!
 7717:50 <fjahr> lightlike: sjors gave the feedback that we should introduce a new helper method, I will do that probably when I retouch or as a follow-up. Using the hash was just an easy first step when I opened the PR.
 7817:50 <stickies-v> fjahr: i wonder why we don't check that with `if (nHeight % consensusParams.DifficultyAdjustmentInterval() == 0)` instead using the `pindexPrev->nHeight`? is it because of the genesis block handling?
 7917:52 <fjahr> stickies-v: Hm, I haven't thought about it to be honest. I used the code from Bluematt without unnecessary changes because it had many eyes on it already and I didn't think about this in particular
 8017:52 <stickies-v> okay, i'll leave a comment on the PR then, thanks
 8117:53 <fjahr> great!
 8217:53 <stickies-v> this code is never reached for genesis blocks (`assert(pindexPrev != nullptr);`) so I think it should be fine
 8317:53 <fjahr> Q: How do you start your node with Testnet 4? What happens when you start it just with -testnet=1 after Testnet 4 is included? Do you think that choice is sensible?
 8417:55 <emzy> It's not chaged it still runs in testnet3 mode.
 8517:55 <stickies-v> there's a new startup option `-testnet4`
 8617:55 <lightlike> by the way, using -testnet=4 also seems to result in testnet3 :)
 8717:56 <stickies-v> what about -testnet4=3? :-D
 8817:56 <emzy> :o)
 8917:56 <fjahr> lightlike: Interesting observation :D
 9017:56 <GregTonoski> Do you (achow101 and glozow) plan to reopen and unlock the "https://github.com/bitcoin/bitcoin/pull/28408&quot;, perhaps?
 9117:57 <fjahr> Right, it is planned that the default switches with the following release, does that make sense to everyone?
 9217:57 <fjahr> GregTonoski: I am not sure who you are asking but I am sure this meeting isn't the right place for discussing this.
 9317:58 <stickies-v> yeah, i think that's fair. defaulting users onto a net that's not widely used anymore doesn't seem helpful
 9417:58 <fjahr> Alright, I would like to skip 11 and 12 because they are pretty bike-sheddy and we can end roughtly on time still
 9517:59 <fjahr> But this is a good one I think: Why is it interesting to embed special scripts into the chain as test cases? What makes this useful beyond bitcoin core?
 9618:00 <stickies-v> I don't really understand the second part of the question, but having a single place to go to battle test your software for all kinds of weird cases is pretty helpful for devs
 9718:01 <fjahr> Right, maybe I didn't formulate it well: I think what makes this particularly interesting is that we force other implementations and tools to parse these transactions and scripts if they want to validate the chain. That means we are getting some tests for the whole ecosystem, not just bitcoin core.
 9818:02 <fjahr> Alright, I think the last one is also pretty bike-sheddy so I would say we finish up unless anyone has a comment on the last question or the ones we skipped :)
 9918:02 <lightlike> did that embedding happen in the existing testnet4 chain? seems like a bit of work to come up with all kind of special scripts that might be interesting and create txns for them.
10018:03 <fjahr> lightlike: Yeah, that isn't done and it's a project that is on my list but where I would also be interested to collaborate with someone else. Volunteers welcome :) There are a lot of ideas for sources, like the Taproot functional test, the fuzzing body, existing scripts on Testnet 3 etc.
10118:04 <fjahr> So we don't need to be creative but getting the existing scripts in there is already valuable
10218:04 <fjahr> Great! Thanks everyone for participating!
10318:04 <fjahr> #endmeeting