Empty files with "undeleted" replicas

franck-jrc · July 24, 2018, 5:18pm

Good evening,

Under rare and undetermined circumstances, fuse clients create some empty files (by either copying or unzipping them, without getting any error) that then has the following output for eos file info command :

  Size: 0
Modify: Tue Oct 10 13:02:29 2017 Timestamp: 1507633349.0
Change: Tue Oct 10 13:02:29 2017 Timestamp: 1507633349.0
  CUid: 47305 CGid: 36203  Fxid: 16ead502 Fid: 384488706    Pid: 83669812   Pxid: 04fcb334
XStype: adler    XS: 00 00 00 01     ETAG: 103210401121959936:00000001
replica Stripes: 2 Blocksize: 4k LayoutId: 00600112
  #Rep: 0
(undeleted) $ 652
(undeleted) $ 556
(undeleted) $ 460
(undeleted) $ 508
(undeleted) $ 652
(undeleted) $ 556
(undeleted) $ 508
(undeleted) $ 460
*******

What could explain such situation ?
This has been observed with MGM v4.2.20, client v4.2.20

franck-jrc · July 25, 2018, 2:17pm

I identified this kind of messages in the FST logs where the files should be located :
errno=0, errc=301, msg=[ERROR] No more free SIDs

What could that mean ?

luca.mascetti · August 3, 2018, 2:09pm

Hi Frank,

this errors comes from xrootd, we had this at CERN as well,
basically the xrootd connection has a certain number of packets
and if there are network issues of errors pile up the connection
might lose track of messages end exhaust this sessions IDs
and break the connection.

A solution is to restart the fst daemon to reset the xrootd connections.

xrootd versions before 4.8.3-1 had this problem.

I suggest to update all the fst to the latest xrootd package (from 4.8.3-1 included and above).

Cheers,
Luca

franck-jrc · August 6, 2018, 8:16am

Hi Luca,

Thank you for your input. OK, that makes sense, so we will go on with upgrades.

Can these errors really explain these ghost files we have with undeleted replicas ? It basically seems that they were committed (sometimes also with a size and a checksum) into the MGM, but the replicas failed to be created on FSTs (and also to be deleted), letting think the client that the files were correctly created.

@esindril, I tried to boot some fs with synmgm, but this doesn’t remove the entries. Anyway, most of these files need to be removed, because they don’t have any content.

luca.mascetti · August 6, 2018, 10:02am

yes it is probably related with this connection issues between FSTs if the files were written by fuse

franck-jrc · August 6, 2018, 2:36pm

OK, thank you!
It seems many orphan replicas also got created during this error period, did you also observe such a thing ?
Is fsck repair --unlink-orphans completely safe to get rid of them, even with a large number (eos fsck reports numbers that vary from 0.5M to 3M, but when summing stat.fsck.orphans_n of eos fs status outputs, we find 5M) ?

franck-jrc · August 10, 2018, 9:30am

@esindril : following our discussion of yesterday, I confirm that most of the orphan replicas reported are now real orphans.
Probably some left over of previous (group)balancing, but most of them seem caused by the above xrootd errors that occurred a lot during last month (July)

There also very few (350) reported orphan files that have a path, and they seem to correspond in fact unregistered replicas, so I was wondering if running the fsck repair function could damage them (of course, we still have the possibility to backup them before hand, since we know them).

However, as I told you, since number of files is too big, the output we get from fsck report is incomplete (number of orphan replicas 2M is smaller than the sums of fsck.stat.orphans_n of all fs 5M, it seems that fsck report -a doesn’t hold information about all FS).

franck-jrc · August 28, 2018, 9:10am

Good morning,

We had another set of such files with many “undeleted” replicas, and none valid, so the files result without any usable content, but seem to be considered created by the MGM.
This correspond to a relatively low number of files, but disturbs a lot our processes because the clients seem to not detect any issue while writing the files.

We can provide more information about what is going on. Some extract here, but maybe could we send them by opening an official ticket ?

This time, none of these “No more free SIDs” errors mentioned earlier in this thread is observed.
Maybe the reason why the file remains is because the MGM requests the FSTs to drop their replica, but they fail doing so, considering they do not have it, and we have a ghost, or zombie file :

  File: '/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl'  Flags: 0644  Clock: 12e4dfac3c
  Size: 0
Modify: Mon Aug 27 18:04:19 2018 Timestamp: 1535385859.87855596
Change: Mon Aug 27 18:04:19 2018 Timestamp: 1535385859.87855596
  CUid: 47000 CGid: 40500  Fxid: 1926899b Fid: 421955995    Pid: 90523775   Pxid: 0565487f
XStype: adler    XS: 00 00 00 01     ETAG: 113267949929758720:00000001
replica Stripes: 2 Blocksize: 4k LayoutId: 00600112
  #Rep: 0
(undeleted) $ 1168
(undeleted) $ 928
(undeleted) $ 1120
(undeleted) $ 1024
(undeleted) $ 784
(undeleted) $ 1072
(undeleted) $ 880
(undeleted) $ 928
(undeleted) $ 1024
(undeleted) $ 688
(undeleted) $ 880
(undeleted) $ 928
(undeleted) $ 1024
(undeleted) $ 688
(undeleted) $ 928
(undeleted) $ 880
(undeleted) $ 1024
(undeleted) $ 976
(undeleted) $ 688
(undeleted) $ 880
(undeleted) $ 928
(undeleted) $ 1024
(undeleted) $ 688
*******

Analysing the logs for a specific file, we identified this pattern : MGM contacts first FST, then a second one that fails (triedrc=fserr) passes to another, etc… until it aborts, send remove requests to all tried FSTs, but they do not, claiming they do not know about the file, so they remain “undeleted” for the MGM.
The initiating FST error in func=fileOpen is sometimes msg=[ERROR] Error response: Input/output error, other times msg=[FATAL] Socket timeout, but the result is the same for the numerous undeleted replicas.

We wonder if there can be some settings problem causing this under some conditions, or an issue in xrootd ? But anyway there is some inconsistency on eos level, leaving such unusable files after unforeseen error.

mgm s-jrciprjeos143p: 180827 18:04:20 time=1535385860.349331 func=open                     level=INFO  logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5b2dfa700 source=XrdMgmOfsFile:2401             tident=jeodata.27151:888@s-jrciprcid03p sec=krb5  uid=47000 gid=40500 name=jeodata geo="JRC" op=write path=/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl info=eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919 target[0]=(s-jrciprjeos231p.cidsn.jrc.it,880) target[1]=(s-jrciprjeos237p.cidsn.jrc.it,1168)  redirection=s-jrciprjeos237p.cidsn.jrc.it?&cap.sym=<...>&cap.msg=<...>&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.replicaindex=1&mgm.replicahead=1&mgm.id=1926899b&mgm.mtime=0:1095
mgm s-jrciprjeos143p: 180827 18:04:20 time=1535385860.349352 func=open                     level=INFO  logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5b2dfa700 source=XrdMgmOfsFile:2409             tident=jeodata.27151:888@s-jrciprcid03p sec=krb5  uid=47000 gid=40500 name=jeodata geo="JRC" info="redirection" hostport=s-jrciprjeos237p.cidsn.jrc.it?&cap.sym=<...>&cap.msg=<...>&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.replicaindex=1&mgm.replicahead=1&mgm.id=1926899b&mgm.mtime=0:1095
fst s-jrciprjeos237p: 180827 18:04:20 time=1535385860.350448 func=open                     level=INFO  logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=XrdFstOfsFile:189              tident=jeodata.27151:258@s-jrciprcid03p sec=(null) uid=99 gid=99 name=(null) geo="" path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl info=cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=1&mgm.replicaindex=1 isRW=0 open_mode=2
fst s-jrciprjeos237p: 180827 18:04:20 time=1535385860.350701 func=ProcessCapOpaque         level=INFO  logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=XrdFstOfsFile:2746             tident=jeodata.27151:258@s-jrciprcid03p sec=(null) uid=99 gid=99 name=(null) geo="" capability=&mgm.access=update&mgm.ruid=47000&mgm.rgid=40500&mgm.uid=47001&mgm.gid=99&mgm.path=/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl&mgm.manager=s-jrciprjeos143p.cidsn.jrc.it:1094&mgm.fid=1926899b&mgm.cid=90523775&mgm.sec=krb5|jeodata|s-jrciprcid03p.cidsn.jrc.it|||||fuse&mgm.lid=1048850&mgm.bookingsize=0&mgm.fsid=1168&mgm.url0=root://s-jrciprjeos231p.cidsn.jrc.it:1095//&mgm.fsid0=880&mgm.url1=root://s-jrciprjeos237p.cidsn.jrc.it:1095//&mgm.fsid1=1168&cap.valid=1535389460
fst s-jrciprjeos237p: 180827 18:04:20 time=1535385860.350795 func=open                     level=INFO  logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=XrdFstOfsFile:313              tident=jeodata.27151:258@s-jrciprcid03p sec=(null) uid=47000 gid=40500 name=nobody geo="" fstpath=/data22/0000a4d3/1926899b
fst s-jrciprjeos237p: 180827 18:04:20 time=1535385860.350862 func=open                     level=INFO  logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=XrdFstOfsFile:438              tident=jeodata.27151:258@s-jrciprcid03p sec=(null) uid=47000 gid=40500 name=nobody geo="" fstpath=/data22/0000a4d3/1926899b
fst s-jrciprjeos237p: 180827 18:04:20 time=1535385860.351140 func=open                     level=INFO  logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=XrdFstOfsFile:481              tident=jeodata.27151:258@s-jrciprcid03p sec=(null) uid=47000 gid=40500 name=nobody geo="" fstpath=/data22/0000a4d3/1926899b open-mode=102 create-mode=41a4 layout-name=replica
mgm s-jrciprjeos143p: 180827 18:05:20 time=1535385920.414236 func=open                     level=INFO  logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a6cf0700 source=XrdMgmOfsFile:186              tident=jeodata.27151:888@s-jrciprcid03p sec=krb5  uid=47001 gid=40501 name=jeodata geo="JRC" op=write trunc=0 path=fxid:1926899b info=cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=1&mgm.replicaindex=1&tried=s-jrciprjeos237p.cidsn.jrc.it
mgm s-jrciprjeos143p: 180827 18:05:20 time=1535385920.414311 func=open                     level=INFO  logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a6cf0700 source=XrdMgmOfsFile:234              tident=jeodata.27151:888@s-jrciprcid03p sec=krb5  uid=47001 gid=40501 name=jeodata geo="JRC" msg="access by inode" ino=fxid:1926899b path=/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl
mgm s-jrciprjeos143p: 180827 18:05:20 time=1535385920.414792 func=open                     level=INFO  logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a6cf0700 source=XrdMgmOfsFile:2401             tident=jeodata.27151:888@s-jrciprcid03p sec=krb5  uid=47000 gid=40500 name=jeodata geo="JRC" op=write path=/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl info=cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=1&mgm.replicaindex=1&tried=s-jrciprjeos237p.cidsn.jrc.it target[0]=(s-jrciprjeos234p.cidsn.jrc.it,1024) target[1]=(s-jrciprjeos233p.cidsn.jrc.it,976)  redirection=s-jrciprjeos234p.cidsn.jrc.it?&cap.sym=<...>&cap.msg=<...>&mgm.logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a&mgm.replicaindex=0&mgm.replicahead=0&mgm.id=1926899b&mgm.mtime=0:1095
mgm s-jrciprjeos143p: 180827 18:05:20 time=1535385920.414807 func=open                     level=INFO  logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a6cf0700 source=XrdMgmOfsFile:2409             tident=jeodata.27151:888@s-jrciprcid03p sec=krb5  uid=47000 gid=40500 name=jeodata geo="JRC" info="redirection" hostport=s-jrciprjeos234p.cidsn.jrc.it?&cap.sym=<...>&cap.msg=<...>&mgm.logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a&mgm.replicaindex=0&mgm.replicahead=0&mgm.id=1926899b&mgm.mtime=0:1095
fst s-jrciprjeos234p: 180827 18:05:20 time=1535385920.415853 func=open                     level=INFO  logid==ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a unit=fst@s-jrciprjeos234p.cidsn.jrc.it:1095 tid=00007ff947cf9700 source=XrdFstOfsFile:189              tident=jeodata.27151:260@s-jrciprcid03p sec=(null) uid=99 gid=99 name=(null) geo="" path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl info=cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=0&mgm.replicaindex=0&tried=s-jrciprjeos237p.cidsn.jrc.it isRW=0 open_mode=2
mgm s-jrciprjeos143p: 180827 18:05:20 time=1535385920.416806 func=open                     level=INFO  logid=ff4cf3dc-aa12-11e8-bbab-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a6cf0700 source=XrdMgmOfsFile:186              tident=jeodata.27151:888@s-jrciprcid03p sec=krb5  uid=47001 gid=40501 name=jeodata geo="JRC" op=write trunc=0 path=fxid:1926899b info=cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=ff4c8ef6-aa12-11e8-bbab-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=0&mgm.replicaindex=0&tried=s-jrciprjeos237p.cidsn.jrc.it,s-jrciprjeos234p.cidsn.jrc.it&triedrc=fserr
[...]
mgm s-jrciprjeos143p: 180827 18:05:20 time=1535385920.446334 func=open                     level=INFO  logid=ff516610-aa12-11e8-bbab-6c0b84b7fa3a unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a6cf0700 source=XrdMgmOfsFile:2409             tident=jeodata.27151:888@s-jrciprcid03p sec=krb5  uid=47000 gid=40500 name=jeodata geo="JRC" info="redirection" hostport=s-jrciprjeos227p.cidsn.jrc.it?&cap.sym=<...>&cap.msg=<...>&mgm.logid=ff516610-aa12-11e8-bbab-6c0b84b7fa3a&mgm.replicaindex=0&mgm.replicahead=0&mgm.id=1926899b&mgm.mtime=0:1095
fst s-jrciprjeos237p: 180827 18:05:38 16682 FstOfs_ReplicaParOpen: jeodata.27151:258@s-jrciprcid03p Unable to open stripes - remote open failed  root://s-jrciprjeos231p.cidsn.jrc.it:1095///#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl?&cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=1&mgm.replicaindex=0&mgm.path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl; remote I/O error
fst s-jrciprjeos231p: 180827 18:05:38 time=1535385938.253367 func=open                     level=INFO  logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos231p.cidsn.jrc.it:1095 tid=00007fd8eb1f8700 source=XrdFstOfsFile:189              tident=daemon.10254:228@s-jrciprjeos237p sec=(null) uid=99 gid=99 name=(null) geo="" path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl info=cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.bookingsize=0&mgm.id=1926899b&mgm.lid=1048850&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl&mgm.replicahead=1&mgm.replicaindex=0 isRW=0 open_mode=200
fst s-jrciprjeos237p: 180827 18:05:38 time=1535385938.254119 func=fileOpen                 level=ERROR logid=db7fb124-aa12-11e8-8c52-0894ef4d3bf6 unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=XrdIo:235                      tident=<service> sec=      uid=0 gid=0 name= geo="" error= "open failed url=root://s-jrciprjeos231p.cidsn.jrc.it:1095///#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl?cap.msg=qPFuqcArwW0OMB5N8IJ7+ycmP1zAR9lwoTU5Lt3JQxi+ggEvgeS5TvzjCdO5myb6/PRnoLaL1KiedJ/hNfjdaxvXCe6xjUQNCKNonAeXU+kFAWwyiBj+7GZWlsrf46HdQixZtSogHfEXLHRL/7+TFcZAgzxN+36UaXWSMcpcbQfASN62onij2BKNpmc1NfAWCgZ6B+F+G3Na55omu7whc7A8/akrV99PzEfvW0gJCzbOueTASG553eGQqzZMEG7DOQAfcROU1/wkAMMYCxOwcqphE2MskfZspynPc5GpuUXqlLR7kyybxMkOnYHD7z/7D6OjziRCM64KmZKQ7Gv03B68wODuJajAotWnBy/l5cGGvqSvNPp66RoTpzDBctDXpYduSeObb6o1MSwS6CRpapJBdweW4A3v/Zsc0hsTjBSmEHLVQEyKdZa1P9uLDUBXX+CMr0NQanuLVZc2HIwrSGYBZ6rqRQeOcuN97qU9kGr11p7pLA76fHp6W5tMFL4TTkm9lCM8Mmjxwk1HvvPBjlqOH+ZINHPz4qenijltarwFatnAm5fv0I9OM9Rbt3yC/7FzUB2X1z9g26EisZFgBhrnr1DK2/qIA6FE+ayytj5ckzO3QlFZdxsyF4KvMSPpuMuOuB7LWTrR/kem72tHPamjRPI0fkYP+8NALMH92ecBphtfCecsivrGpelPAQx8NOdmY7if+0w++X/mjP12MSZXGJD5E2wP5WAWLnyhR4KWFEwItcAJ7mKX53Qr5PHpuumdbir5+ujWpYqVlG5ilg==&cap.sym=nq9z3fi7CoSgkW2BJuOkw28sFZo=&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.bookingsize=0&mgm.id=1926899b&mgm.lid=1048850&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl&mgm.replicahead=1&mgm.replicaindex=0, errno=3005, errc=400, msg=[ERROR] Error response: Input/output error"
fst s-jrciprjeos237p: 180827 18:05:38 time=1535385938.254214 func=Open                     level=ERROR logid==db7f540e-aa12-11e8-929c-6c0b84b7fa3a unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2428243700 source=ReplicaParLayout:212           tident=jeodata.27151:258@s-jrciprcid03p sec=unix  uid=0 gid=0 name=jeodata geo="" Failed to open stripes - remote open failed on root://s-jrciprjeos231p.cidsn.jrc.it:1095///#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl?&cap.msg=<...>&cap.sym=<...>&eos.app=fuse&eos.bookingsize=0&eos.encodepath=1&eos.lfn=fxid:1926899b&fst.blocksize=1048576&fst.readahead=true&fst.valid=1535385919&mgm.id=1926899b&mgm.logid=db7f540e-aa12-11e8-929c-6c0b84b7fa3a&mgm.mtime=0&mgm.replicahead=1&mgm.replicaindex=0&mgm.path=/#curl#/eos/jeodpp/data/SRS/Copernicus/S2/scenes/source/L1C/2018/08/27/088/S2B_MSIL1C_20180827T011719_N0206_R088_T53LNK_20180827T024558.SAFE/HTML/UserProduct_index.xsl
mgm s-jrciprjeos143p: 180827 18:06:11 time=1535385971.089786 func=FSctl                    level=INFO  logid=static.............................. unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007f9f237df700 source=Schedule2Delete:93             tident= sec=(null) uid=99 gid=99 name=- geo="" msg="add to deletion message" fxid=1926899b fsid=1168
mgm s-jrciprjeos143p: 180827 18:06:12 time=1535385972.002677 func=FSctl                    level=INFO  logid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007f9f237df700 source=Drop:39                        tident=daemon.10254:5489@s-jrciprjeos237p sec=sss   uid=2 gid=2 name=daemon geo="JRC" drop request for &mgm.pcmd=drop&mgm.fsid=1168&mgm.fid=1926899b&mgm.localprefix=/data22
fst s-jrciprjeos237p: 180827 18:06:12 time=1535385972.002755 func=_rem                     level=INFO  logid=1c62d27e-9bc8-11e8-93bb-0894ef4d3bf6 unit=fst@s-jrciprjeos237p.cidsn.jrc.it:1095 tid=00007f2459afe700 source=XrdFstOfs:1238                 tident=<service> sec=      uid=0 gid=0 name= geo="" fstpath=/data22/0000a4d3/1926899b
mgm s-jrciprjeos143p: 180827 18:06:23 time=1535385983.071804 func=FSctl                    level=INFO  logid=static.............................. unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a76fa700 source=Schedule2Delete:93             tident= sec=(null) uid=99 gid=99 name=- geo="" msg="add to deletion message" fxid=1926899b fsid=736
mgm s-jrciprjeos143p: 180827 18:06:24 time=1535385984.976527 func=FSctl                    level=INFO  logid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx unit=mgm@s-jrciprjeos143p.cidsn.jrc.it:1094 tid=00007fd5a76fa700 source=Drop:39                        tident=daemon.2896:4124@s-jrciprjeos228p sec=sss   uid=2 gid=2 name=daemon geo="JRC" drop request for &mgm.pcmd=drop&mgm.fsid=736&mgm.fid=1926899b&mgm.localprefix=/data22
fst s-jrciprjeos228p: 180827 18:06:24 time=1535385984.976527 func=_rem                     level=INFO  logid=37656fea-9bb5-11e8-bef1-0894ef4d41d6 unit=fst@s-jrciprjeos228p.cidsn.jrc.it:1095 tid=00007ffafaafe700 source=XrdFstOfs:1238                 tident=<service> sec=      uid=0 gid=0 name= geo="" fstpath=/data22/0000a4d3/1926899b
fst s-jrciprjeos228p: 180827 18:06:24 time=1535385984.976748 func=Remover                  level=WARN  logid=static.............................. unit=fst@s-jrciprjeos228p.cidsn.jrc.it:1095 tid=00007ffafaafe700 source=Remover:63                     tident= sec=(null) uid=99 gid=99 name=- geo="" unable to remove fid 1926899b fsid 736 localprefix=/data22
[...]

MGM is 4.2.28, FSTs are mix of 4.2.28 and 4.2.29, the client is fuse mount 4.2.28.
Observation shows that that some other older clients (like 4.2.4) sometimes have the same Input/output error, but don’t generate such ghost files, i.e. there is a readable content after this, sometimes with some undeleted replicas as well, but that might be a mere statistical fact.

CERN Accelerating science

Empty files with "undeleted" replicas