We are running quarkdb-0.4.2 and xrootd 4.12.8 and have recently begun receiving messages:
[1624226401828] ERROR: Potential MANIFEST corruption for DB at /quarkdb/checkpoints/checkpoint_2021-06-21/current/state-machine(1783322963 sec)
Do the 0.4.3 release notes regarding the potential manifest corruption apply to the above, or is there a way to determine if there is a valid issue?
Did I understand correctly that quardkdb 0.4.3 packages built against xrootd 5 and removed from the storage-ci.web.cern.ch install repo but not replaced with ones built against xrootd 4? I do not see quark 0.4.3 packages there, and the linuxsoft.cern.ch repo in release notes returns 403.
Cheers,
Pete
After writing above I saw this issue was reported in the 0.4.2 thread and that the behavior we are seeing is the similar with timestamp 1783322963 sec
[root@warp-ornl-cern-05 ~]# quarkdb-validate-checkpoint --path /quarkdb/checkpoints/checkpoint_2021-06-21/
[1624296403520] INFO: Attempting to open ShardDirectory...
[1624296403521] INFO: --- OK!
[1624296403521] INFO: Attempting to open StateMachine...
[1624296403523] INFO: Openning state machine '/quarkdb/checkpoints/checkpoint_2021-06-21/current/state-machine'.
[1624296403599] INFO: --- OK! LAST-APPLIED: 582451415
[1624296403600] INFO: Attempting to open RaftJournal...
[1624296403600] INFO: Opening raft journal '/quarkdb/checkpoints/checkpoint_2021-06-21/current/raft-journal'
[1624296403600] ERROR: Potential MANIFEST corruption for DB at /quarkdb/checkpoints/checkpoint_2021-06-21/current/state-machine(1783322963 sec)
[1624296404064] INFO: --- OK! LOG-SIZE: 582451416, COMMIT-INDEX: 582451415, LOG-START: 532000000
[1624296404065] INFO: Closing state machine '/quarkdb/checkpoints/checkpoint_2021-06-21/current/state-machine'
[1624296404081] INFO: Closing raft journal '/quarkdb/checkpoints/checkpoint_2021-06-21/current/raft-journal'
We will ignore errors with uninitialized var timestamp until move to 0.4.3 and xrootd 5