Errors when compacting the (file based) namespace

Site Administrators
1

Hello,

I am preparing to move to QuarkDB (operation scheduled tomorrow) and I tried to force manually a compactification of the namespace doing:

eos -b ns compact on

There are errors and I find it worrying just before moving to QuarkDB:

[root@naneosmgr01(EOSMASTER) ~]#eos ns master
[...]
191113 09:26:20 time=1573633580.037309 func=ScheduleOnlineCompacting level=INFO  logid=static.............................. unit=mgm@naneosmgr01.in2p3.fr:1094 tid=00007feaa9efd700 source=Master:589                     tident= sec=(null) uid=99 gid=99 name=- geo="" msg="scheduling online compacting" starttime=1573633580 interval=0
191113 09:26:30 time=1573633590.155544 func=Compacting               level=INFO  logid=9af38aea-d92d-11e9-ad3f-14187764b113 unit=mgm@naneosmgr01.in2p3.fr:1094 tid=00007feac03fe700 source=Master:669                     tident=<service> sec=      uid=0 gid=0 name= geo="" archive(file)=/var/eos/md/files.naneosmgr01.in2p3.fr.mdlog.1573633590 oc=/var/eos/md/files.naneosmgr01.in2p3.fr.mdlog.oc
191113 09:26:30 time=1573633590.155585 func=Compacting               level=INFO  logid=9af38aea-d92d-11e9-ad3f-14187764b113 unit=mgm@naneosmgr01.in2p3.fr:1094 tid=00007feac03fe700 source=Master:710                     tident=<service> sec=      uid=0 gid=0 name= geo="" msg="compact prepare"
191113 09:26:31 time=1573633591.030682 func=Compacting               level=INFO  logid=9af38aea-d92d-11e9-ad3f-14187764b113 unit=mgm@naneosmgr01.in2p3.fr:1094 tid=00007feac03fe700 source=Master:723                     tident=<service> sec=      uid=0 gid=0 name= geo="" msg="compacting"
191113 09:28:13 time=1573633693.972736 func=Compacting               level=INFO  logid=9af38aea-d92d-11e9-ad3f-14187764b113 unit=mgm@naneosmgr01.in2p3.fr:1094 tid=00007feac03fe700 source=Master:736                     tident=<service> sec=      uid=0 gid=0 name= geo="" msg="compact commit"
191113 09:28:14 time=1573633694.033290 func=Compacting               level=CRIT  logid=9af38aea-d92d-11e9-ad3f-14187764b113 unit=mgm@naneosmgr01.in2p3.fr:1094 tid=00007feac03fe700 source=Master:824                     tident=<service> sec=      uid=0 gid=0 name= geo="" online-compacting returned ec=5 error: Changelog file has corruption - autorepair is disabled
191113 09:28:15 time=1573633695.033467 func=Compacting               level=CRIT  logid=9af38aea-d92d-11e9-ad3f-14187764b113 unit=mgm@naneosmgr01.in2p3.fr:1094 tid=00007feac03fe700 source=Master:864                     tident=<service> sec=      uid=0 gid=0 name= geo="" failed online compactification
191113 11:08:33 time=1573639713.859389 func=ScheduleOnlineCompacting level=INFO  logid=static.............................. unit=mgm@naneosmgr01.in2p3.fr:1094 tid=00007feaadafd700 source=Master:589                     tident= sec=(null) uid=99 gid=99 name=- geo="" msg="scheduling online compacting" starttime=0 interval=0
191113 11:14:13 time=1573640053.645170 func=GetLog                   level=ERROR logid=9af38aea-d92d-11e9-ad3f-14187764b113 unit=mgm@naneosmgr01.in2p3.fr:1094 tid=00007fe6fa2fe700 source=Master:2331                    tident=<service> sec=      uid=0 gid=0 name= geo="" error: corruption in file changelog at offset a3944dec

=> What should I do ?

The compactification is still running (apparently):

eos -b ns | grep -i compac
ALL      Files                            15366945 [compacting] (67s)
ALL      Compactification                 status=compacting waitstart=0 ratio-file=0.0:1 ratio-dir=0.0:1
2

Another piece of information:

tail -f /var/log/eos/mgm/Master.log
191113 09:26:30 NOTE  Master:657                     msg="starting online compaction" 
191113 09:26:30 INFO  Master:669                     archive(file)=/var/eos/md/files.naneosmgr01.in2p3.fr.mdlog.1573633590 oc=/var/eos/md/files.naneosmgr01.in2p3.fr.mdlog.oc 
191113 09:26:30 INFO  Master:710                     msg="compact prepare" 
191113 09:26:31 INFO  Master:723                     msg="compacting" 
191113 09:28:13 INFO  Master:736                     msg="compact commit" 
191113 09:28:14 CRIT  Master:824                     online-compacting returned ec=5 error: Changelog file has corruption - autorepair is disabled 
191113 09:28:15 CRIT  Master:864                     failed online compactification 
191113 11:08:33 INFO  Master:589                     msg="scheduling online compacting" starttime=0 interval=0 
191113 11:14:13 ERROR Master:2331                    error: corruption in file changelog at offset a3944dec

JM

3

You can just run eos-log-repair with the namespace files to see what is wrong/corrupted … if that works, before converting the files you do this offline during the intervention.

Cheers Andreas.

4

Thank you Andreas,

I did eos-log-repair on the files of the namespace (files+directories) before doing the eos-ns-convert and it worked well. I did not run offline compactification in addition, I suppose it was enough with eos-log-repair and anyway, it worked.

JM