QuarkDB 0.4.2 has been released

Releases
1

Dear all,

QuarkDB 0.4.2 has been released - notable changes:

  1. Bug fixes

    • Under complicated conditions (follower is very far behind leader + network instabilities), replication towards a particular follower could become stuck. (to workaround, restart leader node)
    • Running DEL on a lease key would cause all nodes in a cluster to crash with an assertion. DEL will now simply release the given lease, as if lease-release had been called.

  2. New features

    • Implement command quarkdb-verify-checksum for manually running a full checksum scan.
    • Addition of quarkdb-validate-checkpoint tool for ensuring that a given checkpoint is valid – useful to run in backup scripts before streaming a given checkpoint for long-term storage.

  3. Improvements

    • Security hardening of the redis parser for unauthenticated clients.
    • Package and distribute quarkdb-ldb tool based on the one provided by RocksDB.
    • Attempt to detect potential MANIFEST corruption early by measuring mtime lag compared to newest SST file.

Many thanks to Crystal Chua (@crystal, AARNet) for the bug report and all support offered related to RocksDB’s MANIFEST corruption issue, as well as to Pete Eby (@peby, ORNL) for finding and reporting the bug causing replication to become stuck.

Full release notes can be found here, packages here, and documentation on the optimal way to upgrade here.

Risk of upgrade: Low.

Cheers,
Georgios

2

Hi @gbitzes, thank you for this update.

One question about the quarkdb-validate-checkpoint command that I have tried. I often see when running it that the following message appears

[1584351256766] ERROR: Potential MANIFEST corruption for DB at /var/lib/quarkdb/backup/test/current/state-machine(1960873871 sec)

And it seems that the MANIFEST file in the checkpoint is changed after this run (it takes a new suffix number).
Is that an expected behavior ?

3

Hi Franck,

Interesting, thanks for reporting. It’s a race condition when ParanoidManifestChecker starts up, you can ignore unreasonable time values, this one equates to 61 years.

It’s OK if the MANIFEST file changes, but this is probably because we open RocksDB in r/w mode in quarkdb-validate-checkpoint, I’ll switch to read-only so it doesn’t happen.

Until then, feel free to ignore the error.

Cheers,
Georgios

4

Thank you for the answer.

OK, I get it : if the file is changed, probably the check is done at a moment were the checked file is deleted, so the information of the timestamp is absent.

Since the checkpoint is based on hard links to files of main DB, no risk that opening the checkpoint DB in r/w mode might affect the running instance ? In fact, also other files are changed after running a validation.

5

Hi Franck,

Indeed there’s no risk, only SST files are hard-linked, which are strictly immutable. RocksDB will never change the contents of an SST file, only delete / replace it entirely. Writing to a snapshot which still contains hard-links to a different DB is safe, as the SSTs never change.

(quarkdb-validate-checkpoint will not issue any writes)

Cheers,
Georgios

6

OK, thank you for your clarifications !

7

Hi Franck,

I committed a fix to make ParanoidManifestChecker more reliable: If RocksDB updates the MANIFEST file in the background, the manifest checker might miss it and use an uninitialized variable as a timestamp, erroneously reporting there’s problem.

It should land in 0.4.3, thanks for the bug report! In any case the manifest checker is experimental, and a last resort for early detection of an odd corruption we saw once in RocksDB. It should only cause worry if the message above is seen repeatedly in production (every 5 minutes), in such case HEALTH will turn red.

Cheers,
Georgios