Good morning,
We had another set of such files with many “undeleted” replicas, and none valid, so the files result without any usable content, but seem to be considered created by the MGM.
This correspond to a relatively low number of files, but disturbs a lot our processes because the clients seem to not detect any issue while writing the files.
We can provide more information about what is going on. Some extract here, but maybe could we send them by opening an official ticket ?
This time, none of these “No more free SIDs” errors mentioned earlier in this thread is observed.
Maybe the reason why the file remains is because the MGM requests the FSTs to drop their replica, but they fail doing so, considering they do not have it, and we have a ghost, or zombie file :
File: '/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl' Flags: 0644 Clock: 12e4dfac3c
Size: 0
Modify: Mon Aug 27 18:04:19 2018 Timestamp: 1535385859.87855596
Change: Mon Aug 27 18:04:19 2018 Timestamp: 1535385859.87855596
CUid: 47000 CGid: 40500 Fxid: 1926899b Fid: 421955995 Pid: 90523775 Pxid: 0565487f
XStype: adler XS: 00 00 00 01 ETAG: 113267949929758720:00000001
replica Stripes: 2 Blocksize: 4k LayoutId: 00600112
#Rep: 0
(undeleted) $ 1168
(undeleted) $ 928
(undeleted) $ 1120
(undeleted) $ 1024
(undeleted) $ 784
(undeleted) $ 1072
(undeleted) $ 880
(undeleted) $ 928
(undeleted) $ 1024
(undeleted) $ 688
(undeleted) $ 880
(undeleted) $ 928
(undeleted) $ 1024
(undeleted) $ 688
(undeleted) $ 928
(undeleted) $ 880
(undeleted) $ 1024
(undeleted) $ 976
(undeleted) $ 688
(undeleted) $ 880
(undeleted) $ 928
(undeleted) $ 1024
(undeleted) $ 688
*******
Analysing the logs for a specific file, we identified this pattern : MGM contacts first FST, then a second one that fails (triedrc=fserr
) passes to another, etc… until it aborts, send remove requests to all tried FSTs, but they do not, claiming they do not know about the file, so they remain “undeleted” for the MGM.
The initiating FST error in func=fileOpen
is sometimes msg=[ERROR] Error response: Input/output error
, other times msg=[FATAL] Socket timeout
, but the result is the same for the numerous undeleted replicas.
We wonder if there can be some settings problem causing this under some conditions, or an issue in xrootd ? But anyway there is some inconsistency on eos level, leaving such unusable files after unforeseen error.
mgm s-jrciprjeos143p: 180827 18:04:20 time=1535385860.349331 func=open level=INFO logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5b2dfa700 source=XrdMgmOfsFile:2401 tident=jeodata.27151:888@s-jrciprcid03p sec=krb5 uid=47000 gid=40500 name=jeodata geo="JRC" op=write path=/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl info=eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919 target[0]=(s-jrciprjeos231p.cidsn.jrc.it,880) target[1]=(s-jrciprjeos237p.cidsn.jrc.it,1168) redirection=s-jrciprjeos237p.cidsn.jrc.it?&cap.sym=<...>&cap.msg=<...>&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.replicaindex=1&mgm.replicahead=1&mgm.id=1926899b&mgm.mtime=0:1095
mgm s-jrciprjeos143p: 180827 18:04:20 time=1535385860.349352 func=open level=INFO logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5b2dfa700 source=XrdMgmOfsFile:2409 tident=jeodata.27151:888@s-jrciprcid03p sec=krb5 uid=47000 gid=40500 name=jeodata geo="JRC" info="redirection" hostport=s-jrciprjeos237p.cidsn.jrc.it?&cap.sym=<...>&cap.msg=<...>&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.replicaindex=1&mgm.replicahead=1&mgm.id=1926899b&mgm.mtime=0:1095
fst s-jrciprjeos237p: 180827 18:04:20 time=1535385860.350448 func=open level=INFO logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=XrdFstOfsFile:189 tident=jeodata.27151:258@s-jrciprcid03p sec=(null) uid=99 gid=99 name=(null) geo="" path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl info=cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=1&mgm.replicaindex=1 isRW=0 open_mode=2
fst s-jrciprjeos237p: 180827 18:04:20 time=1535385860.350701 func=ProcessCapOpaque level=INFO logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=XrdFstOfsFile:2746 tident=jeodata.27151:258@s-jrciprcid03p sec=(null) uid=99 gid=99 name=(null) geo="" capability=&mgm.access=update&mgm.ruid=47000&mgm.rgid=40500&mgm.uid=47001&mgm.gid=99&mgm.path=/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl&mgm.manager=s-jrciprjeos143p.cidsn.jrc.it:1094&mgm.fid=1926899b&mgm.cid=90523775&mgm.sec=krb5|jeodata|s-jrciprcid03p.cidsn.jrc.it|||||fuse&mgm.lid=1048850&mgm.bookingsize=0&mgm.fsid=1168&mgm.url0=root://s-jrciprjeos231p.cidsn.jrc.it:1095//&mgm.fsid0=880&mgm.url1=root://s-jrciprjeos237p.cidsn.jrc.it:1095//&mgm.fsid1=1168&cap.valid=1535389460
fst s-jrciprjeos237p: 180827 18:04:20 time=1535385860.350795 func=open level=INFO logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=XrdFstOfsFile:313 tident=jeodata.27151:258@s-jrciprcid03p sec=(null) uid=47000 gid=40500 name=nobody geo="" fstpath=/data22/0000a4d3/1926899b
fst s-jrciprjeos237p: 180827 18:04:20 time=1535385860.350862 func=open level=INFO logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=XrdFstOfsFile:438 tident=jeodata.27151:258@s-jrciprcid03p sec=(null) uid=47000 gid=40500 name=nobody geo="" fstpath=/data22/0000a4d3/1926899b
fst s-jrciprjeos237p: 180827 18:04:20 time=1535385860.351140 func=open level=INFO logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=XrdFstOfsFile:481 tident=jeodata.27151:258@s-jrciprcid03p sec=(null) uid=47000 gid=40500 name=nobody geo="" fstpath=/data22/0000a4d3/1926899b open-mode=102 create-mode=41a4 layout-name=replica
mgm s-jrciprjeos143p: 180827 18:05:20 time=1535385920.414236 func=open level=INFO logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a6cf0700 source=XrdMgmOfsFile:186 tident=jeodata.27151:888@s-jrciprcid03p sec=krb5 uid=47001 gid=40501 name=jeodata geo="JRC" op=write trunc=0 path=fxid:1926899b info=cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=1&mgm.replicaindex=1&tried=s-jrciprjeos237p.cidsn.jrc.it
mgm s-jrciprjeos143p: 180827 18:05:20 time=1535385920.414311 func=open level=INFO logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a6cf0700 source=XrdMgmOfsFile:234 tident=jeodata.27151:888@s-jrciprcid03p sec=krb5 uid=47001 gid=40501 name=jeodata geo="JRC" msg="access by inode" ino=fxid:1926899b path=/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl
mgm s-jrciprjeos143p: 180827 18:05:20 time=1535385920.414792 func=open level=INFO logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a6cf0700 source=XrdMgmOfsFile:2401 tident=jeodata.27151:888@s-jrciprcid03p sec=krb5 uid=47000 gid=40500 name=jeodata geo="JRC" op=write path=/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl info=cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=1&mgm.replicaindex=1&tried=s-jrciprjeos237p.cidsn.jrc.it target[0]=(s-jrciprjeos234p.cidsn.jrc.it,1024) target[1]=(s-jrciprjeos233p.cidsn.jrc.it,976) redirection=s-jrciprjeos234p.cidsn.jrc.it?&cap.sym=<...>&cap.msg=<...>&mgm.logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a&mgm.replicaindex=0&mgm.replicahead=0&mgm.id=1926899b&mgm.mtime=0:1095
mgm s-jrciprjeos143p: 180827 18:05:20 time=1535385920.414807 func=open level=INFO logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a6cf0700 source=XrdMgmOfsFile:2409 tident=jeodata.27151:888@s-jrciprcid03p sec=krb5 uid=47000 gid=40500 name=jeodata geo="JRC" info="redirection" hostport=s-jrciprjeos234p.cidsn.jrc.it?&cap.sym=<...>&cap.msg=<...>&mgm.logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a&mgm.replicaindex=0&mgm.replicahead=0&mgm.id=1926899b&mgm.mtime=0:1095
fst s-jrciprjeos234p: 180827 18:05:20 time=1535385920.415853 func=open level=INFO logid==ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a unit=fst@s-jrciprjeos234p.cidsn.jrc.it:1095 tid=00007ff947cf9700 source=XrdFstOfsFile:189 tident=jeodata.27151:260@s-jrciprcid03p sec=(null) uid=99 gid=99 name=(null) geo="" path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl info=cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=0&mgm.replicaindex=0&tried=s-jrciprjeos237p.cidsn.jrc.it isRW=0 open_mode=2
mgm s-jrciprjeos143p: 180827 18:05:20 time=1535385920.416806 func=open level=INFO logid=ff4cf3dc-aa12-11e8-bbab-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a6cf0700 source=XrdMgmOfsFile:186 tident=jeodata.27151:888@s-jrciprcid03p sec=krb5 uid=47001 gid=40501 name=jeodata geo="JRC" op=write trunc=0 path=fxid:1926899b info=cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=0&mgm.replicaindex=0&tried=s-jrciprjeos237p.cidsn.jrc.it,s-jrciprjeos234p.cidsn.jrc.it&triedrc=fserr
[...]
mgm s-jrciprjeos143p: 180827 18:05:20 time=1535385920.446334 func=open level=INFO logid=ff516610-aa12-11e8-bbab-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a6cf0700 source=XrdMgmOfsFile:2409 tident=jeodata.27151:888@s-jrciprcid03p sec=krb5 uid=47000 gid=40500 name=jeodata geo="JRC" info="redirection" hostport=s-jrciprjeos227p.cidsn.jrc.it?&cap.sym=<...>&cap.msg=<...>&mgm.logid=ff516610-aa12-11e8-bbab-6c0b84b7fa3a&mgm.replicaindex=0&mgm.replicahead=0&mgm.id=1926899b&mgm.mtime=0:1095
fst s-jrciprjeos237p: 180827 18:05:38 16682 FstOfs_ReplicaParOpen: jeodata.27151:258@s-jrciprcid03p Unable to open stripes - remote open failed root://s-jrciprjeos231p.cidsn.jrc.it:1095///#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl?&cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=1&mgm.replicaindex=0&mgm.path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl; remote I/O error
fst s-jrciprjeos231p: 180827 18:05:38 time=1535385938.253367 func=open level=INFO logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos231p.cidsn.jrc.it:1095 tid=00007fd8eb1f8700 source=XrdFstOfsFile:189 tident=daemon.10254:228@s-jrciprjeos237p sec=(null) uid=99 gid=99 name=(null) geo="" path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl info=cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.bookingsize=0&mgm.id=1926899b&mgm.lid=1048850&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl&mgm.replicahead=1&mgm.replicaindex=0 isRW=0 open_mode=200
fst s-jrciprjeos237p: 180827 18:05:38 time=1535385938.254119 func=fileOpen level=ERROR logid=db7fb124-aa12-11e8-8c52-0894ef4d3bf6 unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=XrdIo:235 tident=<service> sec= uid=0 gid=0 name= geo="" error= "open failed url=root://s-jrciprjeos231p.cidsn.jrc.it:1095///#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl?cap.msg=qPFuqcArwW0OMB5N8IJ7+ycmP1zAR9lwoTU5Lt3JQxi+ggEvgeS5TvzjCdO5myb6/PRnoLaL1KiedJ/hNfjdaxvXCe6xjUQNCKNonAeXU+kFAWwyiBj+7GZWlsrf46HdQixZtSogHfEXLHRL/7+TFcZAgzxN+36UaXWSMcpcbQfASN62onij2BKNpmc1NfAWCgZ6B+F+G3Na55omu7whc7A8/akrV99PzEfvW0gJCzbOueTASG553eGQqzZMEG7DOQAfcROU1/wkAMMYCxOwcqphE2MskfZspynPc5GpuUXqlLR7kyybxMkOnYHD7z/7D6OjziRCM64KmZKQ7Gv03B68wODuJajAotWnBy/l5cGGvqSvNPp66RoTpzDBctDXpYduSeObb6o1MSwS6CRpapJBdweW4A3v/Zsc0hsTjBSmEHLVQEyKdZa1P9uLDUBXX+CMr0NQanuLVZc2HIwrSGYBZ6rqRQeOcuN97qU9kGr11p7pLA76fHp6W5tMFL4TTkm9lCM8Mmjxwk1HvvPBjlqOH+ZINHPz4qenijltarwFatnAm5fv0I9OM9Rbt3yC/7FzUB2X1z9g26EisZFgBhrnr1DK2/qIA6FE+ayytj5ckzO3QlFZdxsyF4KvMSPpuMuOuB7LWTrR/kem72tHPamjRPI0fkYP+8NALMH92ecBphtfCecsivrGpelPAQx8NOdmY7if+0w++X/mjP12MSZXGJD5E2wP5WAWLnyhR4KWFEwItcAJ7mKX53Qr5PHpuumdbir5+ujWpYqVlG5ilg==&cap.sym=nq9z3fi7CoSgkW2BJuOkw28sFZo=&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.bookingsize=0&mgm.id=1926899b&mgm.lid=1048850&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl&mgm.replicahead=1&mgm.replicaindex=0, errno=3005, errc=400, msg=[ERROR] Error response: Input/output error"
fst s-jrciprjeos237p: 180827 18:05:38 time=1535385938.254214 func=Open level=ERROR logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=ReplicaParLayout:212 tident=jeodata.27151:258@s-jrciprcid03p sec=unix uid=0 gid=0 name=jeodata geo="" Failed to open stripes - remote open failed on root://s-jrciprjeos231p.cidsn.jrc.it:1095///#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl?&cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=1&mgm.replicaindex=0&mgm.path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl
mgm s-jrciprjeos143p: 180827 18:06:11 time=1535385971.089786 func=FSctl level=INFO logid=static.............................. unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007f9f237df700 source=Schedule2Delete:93 tident= sec=(null) uid=99 gid=99 name=- geo="" msg="add to deletion message" fxid=1926899b fsid=1168
mgm s-jrciprjeos143p: 180827 18:06:12 time=1535385972.002677 func=FSctl level=INFO logid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007f9f237df700 source=Drop:39 tident=daemon.10254:5489@s-jrciprjeos237p sec=sss uid=2 gid=2 name=daemon geo="JRC" drop request for &mgm.pcmd=drop&mgm.fsid=1168&mgm.fid=1926899b&mgm.localprefix=/data22
fst s-jrciprjeos237p: 180827 18:06:12 time=1535385972.002755 func=_rem level=INFO logid=1c62d27e-9bc8-11e8-93bb-0894ef4d3bf6 unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2459afe700 source=XrdFstOfs:1238 tident=<service> sec= uid=0 gid=0 name= geo="" fstpath=/data22/0000a4d3/1926899b
mgm s-jrciprjeos143p: 180827 18:06:23 time=1535385983.071804 func=FSctl level=INFO logid=static.............................. unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a76fa700 source=Schedule2Delete:93 tident= sec=(null) uid=99 gid=99 name=- geo="" msg="add to deletion message" fxid=1926899b fsid=736
mgm s-jrciprjeos143p: 180827 18:06:24 time=1535385984.976527 func=FSctl level=INFO logid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a76fa700 source=Drop:39 tident=daemon.2896:4124@s-jrciprjeos228p sec=sss uid=2 gid=2 name=daemon geo="JRC" drop request for &mgm.pcmd=drop&mgm.fsid=736&mgm.fid=1926899b&mgm.localprefix=/data22
fst s-jrciprjeos228p: 180827 18:06:24 time=1535385984.976527 func=_rem level=INFO logid=37656fea-9bb5-11e8-bef1-0894ef4d41d6 unit=fst@s-jrciprjeos228p.cidsn.jrc.it:1095 tid=00007ffafaafe700 source=XrdFstOfs:1238 tident=<service> sec= uid=0 gid=0 name= geo="" fstpath=/data22/0000a4d3/1926899b
fst s-jrciprjeos228p: 180827 18:06:24 time=1535385984.976748 func=Remover level=WARN logid=static.............................. unit=fst@s-jrciprjeos228p.cidsn.jrc.it:1095 tid=00007ffafaafe700 source=Remover:63 tident= sec=(null) uid=99 gid=99 name=- geo="" unable to remove fid 1926899b fsid 736 localprefix=/data22
[...]
MGM is 4.2.28, FSTs are mix of 4.2.28 and 4.2.29, the client is fuse mount 4.2.28.
Observation shows that that some other older clients (like 4.2.4) sometimes have the same Input/output error, but don’t generate such ghost files, i.e. there is a readable content after this, sometimes with some undeleted replicas as well, but that might be a mere statistical fact.