How to manually verify blk*.dat and rev*.dat files?

Josue

New member
I apologise if this had been asked and answered before, but I failed to find that on this forum.

Every time I found a new VPS provider offering cheaper and bigger resources, I move my full node to the new VPS. I make sure the integrity of blk*.dat and rev*.dat files by comparing their md5sum on the source and target VPS.

I recently decided not to renew the contract of one of my VPS'. Instead of letting it waiting for its contract expiry date and doing nothing, I configured another full node with pruning as it has only about 100 GB storage space.

When I compared the pruning full node with the main one, the most recent blk*.dat and rev*.dat files have different md5sum as below.
 

Ashton

Member
Your files are not corrupted.
a) BitcoinCore writes blocks into blk*.dat files not in their order.
b) BitcoinCore keeps orphan blocks in blk*.dat files
This causes differences.

The databases are equal - but the files can be different
 

Kaiden

Member
a) BitcoinCore writes blocks into blk*.dat files not in their order.
b) BitcoinCore keeps orphan blocks in blk*.dat files
Thanks a lot. That explains why my pruning full node and my main full node have different md5sum from blk01513.dat and rev01513.dat onward, as starting from those files they are running independently.

How about my other question? I am sorry for a basic question as I didn't realise that until now.

What will happen if my main full node (which has all blocks from blk00000.dat) fails to send valid data to its peers due to corrupted (for whatever reasons) blk*.dat and/or rev*.dat files? What kind of mechanism applies in bitcoind?

I am running bitcoin core 0.17.1 by the way.

And does it make sense to run for instance "bitcoin-cli verifychain 4 563927" once in a while to make sure the integrity of my full node?

On my VPS that might take about 7 hours as it took about 4 seconds to verify 100 blocks.

Edited:
Sorry... wrong calculation on "4 seconds to verify 100 blocks". It took about 1 minutes 14 seconds to verify 100 blocks. So to verify 563927 blocks will take about 5 days on my VPS
 

Giovanni

Member
What will happen if my main full node (which has all blocks from blk00000.dat) fails to send valid data to its peers due to corrupted (for whatever reasons) blk*.dat and/or rev*.dat files?
Nothing will happen.
Your peers will disconnect/ban your misbehaviour node and will try to get valid data from another sources.
(Nodes do not send rev*.dat information to each other. These files are local database files for
updating the current utxo set)
 

Jesus

Member
Your files are not corrupted.
a) BitcoinCore writes blocks into blk*.dat files not in their order.
b) BitcoinCore keeps orphan blocks in blk*.dat files
This causes differences.

The databases are equal - but the files can be different
Why the old orphans are not removed? Is it a flaw in architecture?
 

Calvin

Member
Your peers will disconnect/ban your misbehaviour node and will try to get valid data from another sources.
This is exactly what I want to avoid, hence my questions.

If something went wrong outside my control causing the corruption on blk*dat and/or rev*.dat files, e.g. power outage or glitch, my full node should not be categorised as "misbehave node". There must be better mechanism to avoid such full nodes from being banned. In my case, I want to be able to make sure the integrity of the blockchain files on my full node
 

Stone

Member
Out of curiosity, I just executed
Code:
[email protected]:~$ bitcoin-cli verifychain 1 10000
true
[email protected]:~$

And the result
Code:
2019-02-20T22:24:01Z Verifying last 10000 blocks at level 1
2019-02-20T22:24:01Z [0%]...[10%]...[20%]...[30%]...[40%]...[50%]...[60%]...[70%]...[80%]...[90%]...[DONE].
2019-02-20T22:32:38Z No coin database inconsistencies in last 10000 blocks (0 transactions)

It took 8 minutes and 37 seconds to verify just the validity of the last 10000 blocks. It will take about 8 hours to verify 565947 blocks. I think that is do-able. So the next time I restart my bitcoind, I will use the following command to check the validity of all block files.
Code:
bitcoind -checkblocks=0 -checklevel=1

I think that would be enough. But that means I assume that everything must be valid as all block files are valid. I really don't like that kind of assumption though.

I am wondering how many people (who run full nodes obviously) check all block files (checkblocks=0) with checklevel=4. Is there anybody who does that regularly?

There must be better way than that, as otherwise it will take a lot longer time next year when we reach above 1 million blocks. For instance, applying some kind of a process to notify the peer that the block which it sent out is invalid so that the peer can update its block file based on the data on its outbound peers
 

Jair

Member
I think that would be enough. But that means I assume that everything must be valid as all block files are valid. I really don't like that kind of assumption though.
Take the binary file editor, open one of the oldest files ( for example blk00005.dat )
and change some data in it. I am quite sure that the bitcoind on your node will not
discover any problems.

Next time I will ask you to write a small program which intercepts network traffic
from your node to 8333 ports and puts some garbage in it.
Grin
 

Ahmir

Member
Next time I will ask you to write a small program which intercepts network traffic
from your node to 8333 ports and puts some garbage in it.
Grin
Why do you think I would do that? Even if I were a top Bitcoin programmer, that would defy my own personal principle. I have already spent a lot of my own personal efforts and (some) money to support Bitcoin network since version 0.9.x as I like the idea and the objective. I hate the change of its name to "Bitcoin Core" though, just because of some stupid people tried to disrupt it.
Take the binary file editor, open one of the oldest files ( for example blk00005.dat )
and change some data in it. I am quite sure that the bitcoind on your node will not
discover any problems.
I believe this would only happen on a crappy in-house developed software and only maintained by 1 or 2 programmers. There are a lot of companies involve as this becomes an industry with huge market valuation and supported by thousands of developers around the globe. So I really doubt that we (including me) are so stupid to trust software that has no mechanism to maintain the integrity of the database.

What I asked in this topic is how to do manual check which is better and faster than using "bitcoin-cli verifychain" command. Maybe there is no other way to do manual check. But I believe there must be a mechanism within bitcoind (and bitcoin-qt) software to make sure the integrity of blockchain data is properly maintained. I asked about that mechanism is for me to understand the impact on my full node. A few years back when the blockchain data on my full node was corrupted, I had to do reindex resulting it to re-download the entire blocks from blk00000.dat.

But you pointed out something that I have to double check myself. I think I will intentionally corrupt the 2nd last blk*.dat file and run my full node on Bitcoin testnet to see what will happen
 
Top