The PR branch HEAD was bcb0cac at the time of this review club meeting.
Notes
During Initial block download (IBD),
a new node automatically initializes and populates its data directory by fetching blocks from peers,
validating them, and storing them in the blocks directory within the data directory (default
$HOME/.bitcoin).
The blocks are stored in files named blknnnnn.dat (for example, blk01234.dat).
These files are limited to 128 MiB, so each can hold about 60 blocks or
more depending on their sizes.
The blknnnnn.dat files are in a custom format: a sequence of blocks, each preceded by a 4-byte
“marker” or “magic number” and a 4-byte integer indicating the block’s size or length in bytes.
The blocks need not be in height-order, either within a block file, or across block files. For
example, block 2000 could be stored in blk00010.dat while block 1500 could be stored in
blk00011.dat.
In order to save disk space, the node operator can enable
pruning
using -prune, so the node retains only the most recent few hundred blocks on disk, although IBD
still downloads and verifies all blocks since the beginning.
During IBD, besides storing the raw blocks, several kinds of state are derived from
the blocks and also stored the data directory as entries in LevelDB. The two most prominent are
the block index and the chainstate (UTXO set). The
block index,
contains an entry for every block (including pruned ones) and indexes the
locations (file and offset within the file) of unpruned raw blocks.
If corruption is suspected in the derived indices (the block index or
chainstate), the user has the option of starting over with an empty data directory
and performing IBD again, but that’s slow and uses a lot of network bandwidth.
If the node isn’t pruned, an alternative is to start the node with the -reindex option.
This will use the existing blocks files to rebuild all the derived state.
This is simliar to IBD but obtains blocks from local files instead of network peers.
PR #24858 fixes a long-standing bug that can
cause a form of mild corruption in the way blocks are stored within the blocks files following a
reindex.
What is reindexing, and how does it differ from IBD (initial block download)?
How does pruning interact with reindexing?
What is the format of the blknnnnn.dat files?
Are these files portable across CPU architectures (big-endian, little-endian)?
What happens if a blocks file becomes corrupted?
Partial hint: See
this call
to FindByte
Each network type (mainnet, testnet, signet, regtest) has its own “magic” bytes, for example,
mainnet).
Where else are these bytes used? Why do they differ across network types?
A review comment
suggested a slightly different way to fix the bug. Explain the
alternate approach. How does it compare?
(Bonus question) The definition of BLOCK_SERIALIZATION_HEADER_SIZE
must be the same across platforms (so that the blocks files are portable).
Why is it okay to assume an int is 4 bytes? Couldn’t it be different on some platforms?
<josie[m]> this PR is fixes a bug where extra bytes are added to a serialized during reindexing. the extra bytes cause an error to be printed to the log when it is later de-serialized
<adam2k> There is a deserialization error in the log file that appears when doing a reindexing of the blknnnn.dat files. Â It's not fatal, but it's confusing and possibly alarming for people that see the issue in their logs.
<michaelfolkson> Maybe this should be asked later but you can explain "A reindex, if it occurs, always happens immediately after the node starts up, before any blocks have been added from peers."?
<josie[m]> one thing that was a little unclear to me: the first few questions in the notes didn't seem directly related to the PR (tho helpful background knowledge to have)
<larryruane> michaelfolkson: Reindex is requested by the user (node operator) as a configuration option (command line or in the config file, tho you probably would never put it in the file, or else it would reindex on every startup!),
<larryruane> and if specified (`-reindex` or `-reindex=1`), it will happen when the node first starts up ... after that process completes (which takes hours usually), then the node syncs with its peers, and you'll add more blocks as usual
<larryruane> josie[m]: yes, that's my fault, I thought it would be helpful to make sure people had the background needed before jumping into this particular PR
<Amirreza> A question. Notes say that blk???.dat files are 128 MiG but store 60 or more blocks. Why 60? I imagined the number is much higher while block size are at most 1 MiG.
<larryruane> Thanks, BlueMoon: and michaelfolkson: - very useful links! Okay, here's question 2 (but feel free to bring up what we've already covered): Which parts of the bitcoind data directory are not derived from other parts of the data directory? What are some examples of parts that are?
<larryruane> michaelfolkson: that's really good, I actually hadn't thought of P2P! I was thinking more that the `blocks` directory is not derived from other stuff in the datadir, but like ... yes, as hernanmarino_: said, exactly
<larryruane> Things in the data directory that are derived are, in a way, to make performance reasonable ... if the node is looking for information about a block (and has its hash), it would be impractical to linearly search the blocks files (blknnnn.dat)!
<BlueMoon> It's not necessary from what I've read, and each of these block files has a marker indicating the size of the block file, I imagine they are accessed via an index.
<larryruane> adam2k: yes exactly, a long time ago, the blocks were ordered, but initial block download (IBD) got a major performance improvement by a feature called headers-first download
<larryruane> So the node first downloads only headers (which are only 80 bytes each), figures out the best chain (assuming the blocks turn out to be valid), knows order of the blocks, so can request many blocks simultaneously from different peers ... and their reply times are kind of random .. so the blocks end up out of order
<larryruane> nassersaazi: Yes the headers come in ordered by height, and there are many in a single message (getheaders P2P message) .. so the node basically says "I know about block hash X, give me up to N headers that build on block X"
<adam2k> it looks like that class is used in several places, like within the blockstorage.cpp there are methods like LoadBlockIndexDB and WriteBlockIndexDB.
<larryruane> josie[m]: I don't think it's json .. but yes, anytime you see those serialization methods, you know this data structure is getting saved to (and read from) disk or sent over (and received from) the network
<adam2k> According to the help cmd "Rebuild chain state and block index from the blk*.dat files on disk. This will also rebuild active optional indexes."
<josie[m]> +1 to other answers, but in how it's different from IBD, reindexing does not verify the blocks, right? its assumed the blk files on disk have already been verified?
<adam2k> I was looking at this earlier. Â I have a pruned client and it appears that I don't have blknnnn.dat files that go beyond the memory limit that I set.
<larryruane> michaelfolkson: I think reindexing does as much as it can (using the blk files), and then the node goes into its usual "sync with my peers" .. which may amount to IBD
<josie[m]> adam2k: i think that's correct, but im guessing you wouldnt store blk files on disk in the case of a pruned node, you would just use the blocks to build the index?
<michaelfolkson> larryruane: I'd guess the optimal behavior would be to after multiple corruptions or whatever just drop all blocks and start afresh without input from user
<adam2k> hmm...ok, maybe I don't have something configured correctly because I can see the blknnnn.dat files on disc and it seems like I can reindex without any failures.
<larryruane> michaelfolkson: Yes it does do that, or very close.. like say reindex is able to process the first 100,000 blocks and then ran into some kind of corruption ... it will automatically IBD starting at 100,001
<michaelfolkson> larryruane: So it detects when the corruption happened and drops blocks from the corruption onwards and requests them from peers? That's neat if so
<josie[m]> larryruane, adam2k: im pretty sure you can run reindex on a pruned node. i just tried by adding prune=1000 in bitcoin.conf and then restarted bitcoind with -reindex
<josie[m]> i think it behaves the same as IBD (again, TIL), but it deletes older blocks as normal once you hit the pruned size. but yeah, i also need to play with this a bit more