Hi Jean-Michel,
To disable fsck you run:
eos fsck disable
Should we consider delaying the migration to having the namespace in QuarkDB until this is fixed ?
It’s hard for me to answer this question. It depends on your constraints. If you reach the limits of the memory of the MGM, or the reboot time is unacceptable for you, then you should move.
I read in the help pages that the default is 14 days, maybe it is no longer true. I set it to 1 hour.
I don’t recommend setting the scaninterval value to one hour since then the scanner will actually rescan all the files at reach run which is very heavy for the disks and the FSTs. I would leave the default of 7 days. Here is the code which set’s it to this value:
https://gitlab.cern.ch/dss/eos/blob/master/mgm/FsView.cc#L873
Is this true also for RAID6 chunks ? How can I test it ?
The current fsck does not do proper accounting for RAIN errors so you should not rely on it for RAIN layouts.
Errors that have been detected by the scandir on the FST ? How can I correct d_mem_sz_diff errors, I have several of them reported and I do not even know if this is real. How can I assess the result of repairement ? (I have also some d_cx_diff errors).
Yes, fsck reports errors detected by the scandir on the FST. Since EOS 4.4.45 there is a new tool that helps you inspect the local contents of the LevelDB on the FSTs and also the flags that mark the different errors reported by FSCK. For example you can do:
[esindril@esdss000 build_ninja]$ sudo eos fileinfo /eos/dev/replica/test.dat
File: '/eos/dev/replica/test.dat' Flags: 0640
Size: 3314
Modify: Fri Jul 26 14:55:50 2019 Timestamp: 1564145750.692870000
Change: Fri Jul 26 14:55:50 2019 Timestamp: 1564145750.670573117
Birth : Fri Jul 26 14:55:50 2019 Timestamp: 1564145750.670573117
CUid: 58602 CGid: 1028 Fxid: 0000b16b Fid: 45419 Pid: 11 Pxid: 0000000b
XStype: adler XS: 74 d7 7c 3a ETAGs: "12192069976064:74d77c3a"
Layout: replica Stripes: 2 Blocksize: 4k LayoutId: 00100112
#Rep: 2
┌───┬──────┬────────────────────────┬────────────────┬────────────────────────────┬──────────┬──────────────┬────────────┬────────┬────────────────────────┐
│no.│ fs-id│ host│ schedgroup│ path│ boot│ configstatus│ drain│ active│ geotag│
└───┴──────┴────────────────────────┴────────────────┴────────────────────────────┴──────────┴──────────────┴────────────┴────────┴────────────────────────┘
0 2 esdss000.cern.ch default.0 /home/esindril/Eos_data/fst2 booted rw nodrain online elvin
1 3 esdss000.cern.ch default.0 /home/esindril/Eos_data/fst3 booted rw nodrain online elvin
*******
[esindril@esdss000 build_ninja]$ sudo eos-leveldb-inspect --dbpath /var/eos/md/fmd.0002.LevelDB --fid 45419
fxid=b16b id=45419 cid=11 fsid=2 ctime=1564145750 ctime_ns=681283000 mtime=1564145750 mtime_ns=692870000 atime=1564145750 atime_ns=692870000 size=3314 disksize=3314 mgmsize=281474976710641 lid=0x100112 uid=58602 gid=1028 filecxerror=0x0 blockcxerror=0x0 layouterror=0x0 checksum=74d77c3a diskchecksum=74d77c3a mgmchecksum=none locations=none
I think this tool will also work if you just copy it on one of the FSTs, you don’t need necessarily to update to the EOS version. With this you can understand where the error comes. To be honest the easiest would be to drop the broken replica and trigger an adjust replica (for replica layout) or to rewrite the entire file (for RAIN layouts) by using “eos file convert --rewrite”. Again, many of these will either change or go away with the new FSCK implementation which will handle all these operations on its own in the background.
You mean the MGM namespace will be updated with what is found on the FST ?
No, the other way around. All the info from the MGM will be dumped (essentially an fs dumpmd is happening during this operation) and the info in the local db of the FST (see the the previous answer) is updated so that the scanner can spot inconsistencies between what is in the namespace and what is on disk.
Then I should maybe stop trying to understand the output of fsck report for files having raid6 as a replica type, right ?
Yes, exactly.
I would have to format the disk in xfs for example, mount it on the disk server and register it with eosfsregister ? Can I reuse the same fs id ?
Yes, you could reuse the same fsid, but there is no advantage to this. Just make sure you drain it properly and remove it using the fs rm command and then you can add it back with the same id if you wish so.
Cheers,
Elvin