Cant delete dirs because of hidden files

1

Hi.

When I try to delete an dir that appears to be empty, EOS returns an error saying that it is not empty

eos rmdir /eos/antaresdev/prod/eudat/trash/home/nagios/
error: unable to remove directory (errc=39) (Directory not empty)

There are hidden files in this dir. Getting the first of these

[root@antares-eos14 george]# eos-ns-inspect scan --members antares-eos14.scd.rl.ac.uk:9999 --password-file eos.keytab --no-dirs --path /eos/antaresdev/prod/eudat/trash/home/nagios/ | head -1

[QCLIENT - INFO - getNext:57] Received redirection to antares-eos99:9999
ctime=1537404549.0 fid=8597916 flags=100600 gid=804 layout_id=1048594 link_name= locations=65535 mtime=1537404546.0 name=tmp.0rMOmwvUS2 path=/eos/antaresdev/prod/eudat/trash/home/nagios/tmp.0rMOmwvUS2 pid=6828169 size=30 stime=0.0 uid=804 unlink_locations= xattr.eos.btime=1537404540 xattr.sys.archive.file_id=8597916 xattr.sys.archive.storage_class=eudat_tape1 xs=794c0713

but when I try to delete the file that is in the path, EOS cannot find it

eos rm /eos/antaresdev/prod/eudat/trash/home/nagios/tmp.0rMOmwvUS2
error: no such file or directory with path ‘/eos/antaresdev/prod/eudat/trash/home/nagios/tmp.0rMOmwvUS2’

Can you please give me an idea what I need to do?

Many thanks.

George

2

Hi George,

it can be that the file is not correctly linked to it’s container… but that case the scan shouldn’t be able to find it!?

Have you tried deleting it though f[x]id instead of path?

In any case can you share the output of:

eos-ns-inspect print --fid 8597916 --members antares-eos14.scd.rl.ac.uk:9999 --password-file eos.keytab

and

eos-ns-inspect print --cid 6828169 --members antares-eos14.scd.rl.ac.uk:9999 --password-file eos.keytab

Cheers,
Manuel

3

Hi Manuel,

Thank you so much for your reply! If I try to do: eos rm fid:8597916

I get

error: File #8597916 does not exist (found deletion tombstone)

Here is the output you asked

eos-ns-inspect print --fid 8597916 --members antares-eos14.scd.rl.ac.uk:9999 --password-file eos.keytab [QCLIENT - INFO - getNext:57] Received redirection to antares-eos99:9999
ID: 8597916
Name: tmp.0rMOmwvUS2
Link name:
Container ID: 6828169
uid: 804, gid: 804
Size: 30
Modify: Thu Sep 20 01:49:06 2018 Timestamp: 1537404546.0
Change: Thu Sep 20 01:49:09 2018 Timestamp: 1537404549.0
Flags: 100600
Checksum type: adler, checksum bytes: 794c0713
Expected number of replicas / stripes: 1
Etag: “2307985502109696:794c0713”
Locations: [65535]
Unlinked locations: []
Extended attributes (3):
eos.btime=1537404540
sys.archive.storage_class=eudat_tape1
sys.archive.file_id=8597916
Full path: /eos/antaresdev/prod/eudat/trash/home/nagios/tmp.0rMOmwvUS2

eos-ns-inspect print --cid 6828169 --members antares-eos14.scd.rl.ac.uk:9999 --password-file eos.keytab [QCLIENT - INFO - getNext:57] Received redirection to antares-eos99:9999
ID: 6828169
Parent ID: 6828168
Name: nagios
uid: 804, gid: 804
ctime: Tue Aug 20 06:21:53 2019 Timestamp: 1566278513.0
mtime: Fri Nov 25 14:02:12 2022 Timestamp: 1669384932.96679185
stime: Thu Nov 24 15:22:52 2022 Timestamp: 1669303372.436553389
Tree size: 1514
Mode: 16872
Flags: 0
Extended attributes (5):
sys.forced.checksum=adler
sys.eos.btime=1669303372.436570598
sys.forced.blocksize=4k
sys.forced.layout=replica
sys.forced.nstripes=1
Full path: /eos/antaresdev/prod/eudat/trash/home/nagios/

FileMap:
tmp.IceJitfihL: 8970958
tmp.Pq8RyOfF0P: 8797432
tmp.Ec3ucNabtr: 8404167
tmp.Z1eZrdUvkW: 8449476
tmp.edaa3n2R3B: 8799606
tmp.fbcBTJPZvv: 6865211
tmp.vKbYwQT0i3: 8868994
tmp.hXd4OStIki: 8804108
tmp.b8EA4KfaNg: 8263686
tmp.u1dhNDYaDD: 8641979
tmp.SRvmJ3hx3d: 8809144
tmp.lgG3JwYWft: 8800409
tmp.zkTo06jnG9: 8976451
tmp.rDAYYRC3Z1: 9820458
tmp.HkTqYfgull: 8808268
tmp.ucgDQ0Aoii: 6953453
tmp.tnhJSGaYEL: 8817823
tmp.PX1J68Z9n1: 8264480
tmp.qktn9jQCGa: 9764418
tmp.wm3sZ3R7Sx: 8604753
tmp.rCHUCqwU9U: 8820998
tmp.8oxQ29NXEi: 8232980
tmp.JQ5m1PiVhS: 8808728
tmp.e3II7shOO2: 8598039
tmp.BXmggHBPUJ: 8872331
tmp.zgN56s4GxI: 9726288
tmp.pPf9xDFFnT: 8459508
tmp.MaPFLdG58F: 8807656
tmp.g9HnXAZzRF: 8410009
tmp.yUYfN9VbW0: 8798219
tmp.OU7OTjEEfn: 8911789
tmp.4lYXqXWfFj: 8960893
tmp.G0cSRNhmpv: 8959474
tmp.adnbhTUvE6: 8243973
tmp.NRnIZH4z1Q: 8979886
tmp.3wrCGb2f02: 9854302
tmp.vXedLT72S7: 8255963
tmp.PywId1WeN0: 9744261
tmp.q21dKlKm1m: 8799197
tmp.NoQvpLnq9p: 8805693
tmp.wktjxHMEXh: 9915045
tmp.tsKCJDzePf: 8451134
tmp.22pZsCta7S: 8257184
tmp.YzNWLnedEv: 8809909
tmp.9f3BNfrmj2: 8415202
tmp.0rMOmwvUS2: 8597916
tmp.M2TUDeUp3C: 9734604
tmp.5Ph1jn2gTZ: 9740136
tmp.LFCSbXfqkP: 8809282
tmp.9LPBta6jft: 8665667
tmp.AOIn8vdIkA: 8804920

4

I have tried already to do a compaction of quarkDB to remove the deletion tombstones
but for some reason there are still there. Unless compaction is not the way to remove them.

George

5

Attempted again the compaction

quarkdb-sst-inspect --file=/var/lib/quarkdb/node-1/current/state-machine/ --command=check --show_properties

and this time I could manage to delete by fid/fxid.

Thanks again for your help!

1 Like