Cant delete dirs because of hidden files

Hi.

When I try to delete an dir that appears to be empty, EOS returns an error saying that it is not empty

eos rmdir /eos/antaresdev/prod/eudat/trash/home/nagios/
error: unable to remove directory (errc=39) (Directory not empty)

There are hidden files in this dir. Getting the first of these

[root@antares-eos14 george]# eos-ns-inspect scan --members antares-eos14.scd.rl.ac.uk:9999 --password-file eos.keytab --no-dirs --path /eos/antaresdev/prod/eudat/trash/home/nagios/ | head -1

[QCLIENT - INFO - getNext:57] Received redirection to antares-eos99:9999
ctime=1537404549.0 fid=8597916 flags=100600 gid=804 layout_id=1048594 link_name= locations=65535 mtime=1537404546.0 name=tmp.0rMOmwvUS2 path=/eos/antaresdev/prod/eudat/trash/home/nagios/tmp.0rMOmwvUS2 pid=6828169 size=30 stime=0.0 uid=804 unlink_locations= xattr.eos.btime=1537404540 xattr.sys.archive.file_id=8597916 xattr.sys.archive.storage_class=eudat_tape1 xs=794c0713

but when I try to delete the file that is in the path, EOS cannot find it

eos rm /eos/antaresdev/prod/eudat/trash/home/nagios/tmp.0rMOmwvUS2
error: no such file or directory with path ‘/eos/antaresdev/prod/eudat/trash/home/nagios/tmp.0rMOmwvUS2’

Can you please give me an idea what I need to do?

Many thanks.

George

Hi George,

it can be that the file is not correctly linked to it’s container… but that case the scan shouldn’t be able to find it!?

Have you tried deleting it though f[x]id instead of path?

In any case can you share the output of:

eos-ns-inspect print --fid 8597916 --members antares-eos14.scd.rl.ac.uk:9999 --password-file eos.keytab

and

eos-ns-inspect print --cid 6828169 --members antares-eos14.scd.rl.ac.uk:9999 --password-file eos.keytab

Cheers,
Manuel

Hi Manuel,

Thank you so much for your reply! If I try to do: eos rm fid:8597916

I get

error: File #8597916 does not exist (found deletion tombstone)

Here is the output you asked

eos-ns-inspect print --fid 8597916 --members antares-eos14.scd.rl.ac.uk:9999 --password-file eos.keytab [QCLIENT - INFO - getNext:57] Received redirection to antares-eos99:9999
ID: 8597916
Name: tmp.0rMOmwvUS2
Link name:
Container ID: 6828169
uid: 804, gid: 804
Size: 30
Modify: Thu Sep 20 01:49:06 2018 Timestamp: 1537404546.0
Change: Thu Sep 20 01:49:09 2018 Timestamp: 1537404549.0
Flags: 100600
Checksum type: adler, checksum bytes: 794c0713
Expected number of replicas / stripes: 1
Etag: “2307985502109696:794c0713”
Locations: [65535]
Unlinked locations: []
Extended attributes (3):
eos.btime=1537404540
sys.archive.storage_class=eudat_tape1
sys.archive.file_id=8597916
Full path: /eos/antaresdev/prod/eudat/trash/home/nagios/tmp.0rMOmwvUS2

eos-ns-inspect print --cid 6828169 --members antares-eos14.scd.rl.ac.uk:9999 --password-file eos.keytab [QCLIENT - INFO - getNext:57] Received redirection to antares-eos99:9999
ID: 6828169
Parent ID: 6828168
Name: nagios
uid: 804, gid: 804
ctime: Tue Aug 20 06:21:53 2019 Timestamp: 1566278513.0
mtime: Fri Nov 25 14:02:12 2022 Timestamp: 1669384932.96679185
stime: Thu Nov 24 15:22:52 2022 Timestamp: 1669303372.436553389
Tree size: 1514
Mode: 16872
Flags: 0
Extended attributes (5):
sys.forced.checksum=adler
sys.eos.btime=1669303372.436570598
sys.forced.blocksize=4k
sys.forced.layout=replica
sys.forced.nstripes=1
Full path: /eos/antaresdev/prod/eudat/trash/home/nagios/

FileMap:
tmp.IceJitfihL: 8970958
tmp.Pq8RyOfF0P: 8797432
tmp.Ec3ucNabtr: 8404167
tmp.Z1eZrdUvkW: 8449476
tmp.edaa3n2R3B: 8799606
tmp.fbcBTJPZvv: 6865211
tmp.vKbYwQT0i3: 8868994
tmp.hXd4OStIki: 8804108
tmp.b8EA4KfaNg: 8263686
tmp.u1dhNDYaDD: 8641979
tmp.SRvmJ3hx3d: 8809144
tmp.lgG3JwYWft: 8800409
tmp.zkTo06jnG9: 8976451
tmp.rDAYYRC3Z1: 9820458
tmp.HkTqYfgull: 8808268
tmp.ucgDQ0Aoii: 6953453
tmp.tnhJSGaYEL: 8817823
tmp.PX1J68Z9n1: 8264480
tmp.qktn9jQCGa: 9764418
tmp.wm3sZ3R7Sx: 8604753
tmp.rCHUCqwU9U: 8820998
tmp.8oxQ29NXEi: 8232980
tmp.JQ5m1PiVhS: 8808728
tmp.e3II7shOO2: 8598039
tmp.BXmggHBPUJ: 8872331
tmp.zgN56s4GxI: 9726288
tmp.pPf9xDFFnT: 8459508
tmp.MaPFLdG58F: 8807656
tmp.g9HnXAZzRF: 8410009
tmp.yUYfN9VbW0: 8798219
tmp.OU7OTjEEfn: 8911789
tmp.4lYXqXWfFj: 8960893
tmp.G0cSRNhmpv: 8959474
tmp.adnbhTUvE6: 8243973
tmp.NRnIZH4z1Q: 8979886
tmp.3wrCGb2f02: 9854302
tmp.vXedLT72S7: 8255963
tmp.PywId1WeN0: 9744261
tmp.q21dKlKm1m: 8799197
tmp.NoQvpLnq9p: 8805693
tmp.wktjxHMEXh: 9915045
tmp.tsKCJDzePf: 8451134
tmp.22pZsCta7S: 8257184
tmp.YzNWLnedEv: 8809909
tmp.9f3BNfrmj2: 8415202
tmp.0rMOmwvUS2: 8597916
tmp.M2TUDeUp3C: 9734604
tmp.5Ph1jn2gTZ: 9740136
tmp.LFCSbXfqkP: 8809282
tmp.9LPBta6jft: 8665667
tmp.AOIn8vdIkA: 8804920

I have tried already to do a compaction of quarkDB to remove the deletion tombstones
but for some reason there are still there. Unless compaction is not the way to remove them.

George

Attempted again the compaction

quarkdb-sst-inspect --file=/var/lib/quarkdb/node-1/current/state-machine/ --command=check --show_properties

and this time I could manage to delete by fid/fxid.

Thanks again for your help!