Problem with RAIN layouts

anfeng · November 8, 2023, 9:45am

Hi,

We encountered an issue when using RAIN layouts.

We have an EOS instance with currently 20 JBOD disks on a single node

➜  eos fs ls
┌─────────────────────────────┬────┬──────┬────────────────────────────────┬────────────────┬────────────────┬────────────┬──────────────┬────────────┬──────┬────────┬────────────────┐
│host                         │port│    id│                            path│      schedgroup│          geotag│        boot│  configstatus│       drain│ usage│  active│          health│
└─────────────────────────────┴────┴──────┴────────────────────────────────┴────────────────┴────────────────┴────────────┴──────────────┴────────────┴──────┴────────┴────────────────┘
 hepfarm41.hep.tsinghua.edu.cn 1095      1               /mnt/eosfst/01/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095      2               /mnt/eosfst/02/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095      3               /mnt/eosfst/03/eos        default.0       local::geo       booted             rw      nodrain   0.01   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095      4               /mnt/eosfst/04/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095      5               /mnt/eosfst/05/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095      6               /mnt/eosfst/06/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095      7               /mnt/eosfst/07/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095      8               /mnt/eosfst/08/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095      9               /mnt/eosfst/09/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095     10               /mnt/eosfst/10/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095     11               /mnt/eosfst/11/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095     12               /mnt/eosfst/12/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095     13               /mnt/eosfst/13/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095     14               /mnt/eosfst/14/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095     15               /mnt/eosfst/15/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095     16               /mnt/eosfst/16/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095     17               /mnt/eosfst/17/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095     18               /mnt/eosfst/18/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095     19               /mnt/eosfst/19/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl
 hepfarm41.hep.tsinghua.edu.cn 1095     20               /mnt/eosfst/20/eos        default.0       local::geo       booted             rw      nodrain   0.00   online      no smartctl

Consider a directory with RAIN layouts

➜  eos attr ls /eos/lhcb/laf/qrain_test
sys.eos.btime="1699426201.879999430"
sys.forced.blockchecksum="crc32c"
sys.forced.blocksize="1M"
sys.forced.checksum="adler"
sys.forced.layout="qrain"
sys.forced.nstripes="12"
sys.forced.space="default"
sys.recycle="/eos/dev/proc/recycle/"

Doing git clone, e.g. cloning our EOS project, in this directory will keep failing like the following

➜  qrain_test git clone https://github.com/cern-eos/eos.git
Cloning into 'eos'...
remote: Enumerating objects: 154646, done.
remote: Counting objects: 100% (11537/11537), done.
remote: Compressing objects: 100% (3181/3181), done.
remote: Total 154646 (delta 8759), reused 10742 (delta 8282), pack-reused 143109
Receiving objects: 100% (154646/154646), 170.25 MiB | 1.85 MiB/s, done.
error: inflate: data stream error (unknown compression method)
error: inflate: data stream error (unknown compression method)
error: inflate: data stream error (unknown compression method)
error: inflate: data stream error (unknown compression method)
error: inflate: data stream error (unknown compression method)
error: inflate: data stream error (unknown compression method)
fatal: serious inflate inconsistency
error: inflate: data stream error (unknown compression method)
warning: die() called many times. Recursion error or racy threaded death!
fatal: serious inflate inconsistency
error: inflate: data stream error (unknown compression method)
error: inflate: data stream error (unknown compression method)
fatal: fetch-pack: invalid index-pack output

and many warnings of "failed reading header" are generated in /var/log/eos/fst/xrdlog.fst. The following logs are a part of them

➜  qrain_test grep -v INFO /var/log/eos/fst/xrdlog.fst
231108 17:05:34 time=1699434334.304686 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0787fe700 source=RainMetaLayout:3
18             tident=daemon.400874:57@hepfarm41 sec=unix  uid=0 gid=0 name=daemon geo="" msg="failed reading header" url="/mnt/eosfst/20/eos/00000005/0000e032?&mgm.lid=1080298322&mgm.bookingsize=0&fst.readahead=
true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.306487 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0787fe700 source=RainMetaLayout:3
18             tident=daemon.400874:57@hepfarm41 sec=unix  uid=0 gid=0 name=daemon geo="" msg="failed reading header" url="/mnt/eosfst/09/eos/00000005/0000e032?&mgm.lid=1080298322&mgm.bookingsize=0&fst.readahead=
true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.308123 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0787fe700 source=RainMetaLayout:3
18             tident=daemon.400874:57@hepfarm41 sec=unix  uid=0 gid=0 name=daemon geo="" msg="failed reading header" url="/mnt/eosfst/12/eos/00000005/0000e032?&mgm.lid=1080298322&mgm.bookingsize=0&fst.readahead=
true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.308552 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0debfd700 source=RainMetaLayout:3
18             tident=_AfvAlkA.127234:59@hepfarm41 sec=unix  uid=0 gid=0 name=root geo="" msg="failed reading header" url="/mnt/eosfst/06/eos/00000005/0000e032?&mgm.lid=1080298322&mgm.bookingsize=0&fst.readahead=
true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.310744 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0787fe700 source=RainMetaLayout:3
18             tident=daemon.400874:57@hepfarm41 sec=unix  uid=0 gid=0 name=daemon geo="" msg="failed reading header" url="/mnt/eosfst/17/eos/00000005/0000e032?&mgm.lid=1080298322&mgm.bookingsize=0&fst.readahead=
true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.312418 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0787fe700 source=RainMetaLayout:3
18             tident=daemon.400874:57@hepfarm41 sec=unix  uid=0 gid=0 name=daemon geo="" msg="failed reading header" url="/mnt/eosfst/05/eos/00000005/0000e032?&mgm.lid=1080298322&mgm.bookingsize=0&fst.readahead=
true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.313827 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0787fe700 source=RainMetaLayout:3
18             tident=daemon.400874:57@hepfarm41 sec=unix  uid=0 gid=0 name=daemon geo="" msg="failed reading header" url="/mnt/eosfst/04/eos/00000005/0000e032?&mgm.lid=1080298322&mgm.bookingsize=0&fst.readahead=
true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.315155 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0787fe700 source=RainMetaLayout:3
18             tident=daemon.400874:57@hepfarm41 sec=unix  uid=0 gid=0 name=daemon geo="" msg="failed reading header" url="/mnt/eosfst/08/eos/00000005/0000e032?&mgm.lid=1080298322&mgm.bookingsize=0&fst.readahead=
true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.316505 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0787fe700 source=RainMetaLayout:3
18             tident=daemon.400874:57@hepfarm41 sec=unix  uid=0 gid=0 name=daemon geo="" msg="failed reading header" url="/mnt/eosfst/11/eos/00000005/0000e032?&mgm.lid=1080298322&mgm.bookingsize=0&fst.readahead=
true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.317896 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0787fe700 source=RainMetaLayout:3
18             tident=daemon.400874:57@hepfarm41 sec=unix  uid=0 gid=0 name=daemon geo="" msg="failed reading header" url="/mnt/eosfst/13/eos/00000005/0000e032?&mgm.lid=1080298322&mgm.bookingsize=0&fst.readahead=
true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.319251 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0787fe700 source=RainMetaLayout:3
18             tident=daemon.400874:57@hepfarm41 sec=unix  uid=0 gid=0 name=daemon geo="" msg="failed reading header" url="/mnt/eosfst/03/eos/00000005/0000e032?&mgm.lid=1080298322&mgm.bookingsize=0&fst.readahead=true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.320610 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0787fe700 source=RainMetaLayout:318             tident=daemon.400874:57@hepfarm41 sec=unix  uid=0 gid=0 name=daemon geo="" msg="failed reading header" url="/mnt/eosfst/14/eos/00000005/0000e032?&mgm.lid=1080298322&mgm.bookingsize=0&fst.readahead=true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.321195 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0debfd700 source=RainMetaLayout:3
18             tident=_AfvAlkA.127234:59@hepfarm41 sec=unix  uid=0 gid=0 name=root geo="" msg="failed reading header" url="root://hepfarm41.hep.tsinghua.edu.cn:1095///fusex-open?&cap.msg=ZrG14I29ItdK2N+NAkfmxAscX
BQS0tsdTMZMWbfNny9vWXejTLmZehvAv9wm45yafk0Egfoqh28mmP39SvIsIwr7uXd2uv/OYHb7DN44BZ1Ck85JWM0hKsiSC2CdvAz2PTuD752rjOUYBpMcRjVYryOZmDvLdpGE0Nc3zaMcDGvzZPWuLrqNehA778G1s167XEL1zMm8WpMli+U+iNh7SWzweOlQ8WLkvYb+N3dn1ivq+
ajWQJVXrA9+tPJTlqh22AAqhsUlHHOWfpdEgWNv5IPbHln/qJt7tUllnO0mcjWph2DFvzPCgFvn9kgqgo0I2NbZNkRdtDclsBgdYnuVoyuHcNsNPL1dGn6O2kh3gqW1jCpsb06K1TyztYEynidOKThqcsBq+VOcyuYmruT35mXnuDIhV+4vJWb25uTTwDjJDflr4Kvhs9o9Ox0twYqQi
Y97nccf9kZl/MQ1gMoUZVX0x7JZzJUcQvqXrN/OBoOxTH+/ljYak2MfT0hqnaQEMlUSwcUD9Scu3gYtA+f+9W+DceBUiuMLo/gx9rUtT5Hw/LlUTpxJJ2dF4Ehq0yDva44VThQRBAwGyYdrTLi3+anTwWZ1QE23KXlt8O9UOImcu9uWoqFCMvT73FOwC1Rv83bUi2q40vvaugJwTVIH4
fJ7AvgLE0cCNl9QZyx8sZMFc0Wqdx2flnlh+hUgtNKkmGfA5Ymp/5uJiDo299zTjkE75m6Q3304EeiHibgu1P89RKtdjyYNTGxfvtwiVktZ2ctGExZowm9vlzW0X5ScHukvC+1CPX0+5JWETz9HWynldjrDf7TlymkZdM6e6pMDYaAnWZLeZK3b4yhPEx1h/OdiMt9gOJ/bTXqQ2U3dE
cbP5P3kUv4cGR94mvySod4Tds4YBCTv2vCAolxNTIq/JrjlkQFkdCkcmkZn2S9yzcH1drpD++iD3otr5HVPkg9Zte1l2zV+HlELTwl6PAgzZPb+Q/RtEGbZEE5rYvNY79Q3LzTEwLHYDfWTLWVQhK306+f7FZnWsIhFjnrHHFTi9+ysuUs8x405Oeu6050Aoz+bpHSsmGSM6PNyaZcMn
SGMz/RmJtnN1j00CXOKBnXFPhFkOFd7xKCC+f9A6MieDh5cxI1617VMVaSc9qjbkHwPvH3YHa51FhGvt73ogXZUqVxUA/BhoJ7FdqaSZrpQrUqrjeI3DJQfVLM7UQRJc28LPqBvC5h3rp2Z4osa7iDZbeG2uNxe0fVq28zuKswsJZ5Pj/YSv7d9PUtj7UL/4x6FHguEVPQ2uo+6PPSoH
y31/pnf8gM9vJn24yXxR7PihB4u9JsdeF4rqglZETA+V2m4ucm+QsS0r5JCRyRGT74HWnmKyMsqzDwbOsQtOLgtY7XF79GNDivi9U/BMYeMvoV1VeezHy7fdUDgzqpazg6flxTBBlWy3lfJT0QpDrwLbv/2RMT0Sgk2Gw4DOkYBagPSD2sOxmIF7hSO/i6ujwWV3vhrvOoohL30hXisp
a2dae8=&cap.sym=KIzjy/ewPSG2R/GHfJ+ogGWGs3Y=&eos.app=fuse&eos.bookingsize=0&eos.checksum=ignore&eos.clientinfo=zbase64:MDAwMDAwNzF4nBXIUQ4CIQxF0a24gmYw+iFJF8PA69BEgXRKjLt3+Dv39oEWzflOIdy0x/3L2mTpzBWFt0X/DfA+RWAoa
wzTzgWS5ttXX1sN2bliSLLPI9Al8lPbUWcilEm5xbC9nn/hKyf4&eos.lfn=ino:e0320000000&fuse.exe=/usr/bin/git&fuse.gid=601&fuse.pid=180685&fuse.uid=2031&fuse.ver=5.2.0&mgm.checksum=ignore&mgm.fusex=1&mgm.id=0000e032&mgm.logid=f9a72916-7e15-11ee-bfaf-908d6ec41898&mgm.mtime=0&mgm.replicahead=7&mgm.replicaindex=0&xrd.wantprot=unix&mgm.path=/fusex-open&fst.readahead=true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.321710 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0debfd700 source=RainMetaLayout:318             tident=_AfvAlkA.127234:59@hepfarm41 sec=unix  uid=0 gid=0 name=root geo="" msg="failed reading header" url="root://hepfarm41.hep.tsinghua.edu.cn:1095///fusex-open?&cap.msg=ZrG14I29ItdK2N+NAkfmxAscXBQS0tsdTMZMWbfNny9vWXejTLmZehvAv9wm45yafk0Egfoqh28mmP39SvIsIwr7uXd2uv/OYHb7DN44BZ1Ck85JWM0hKsiSC2CdvAz2PTuD752rjOUYBpMcRjVYryOZmDvLdpGE0Nc3zaMcDGvzZPWuLrqNehA778G1s167XEL1zMm8WpMli+U+iNh7SWzweOlQ8WLkvYb+N3dn1ivq+ajWQJVXrA9+tPJTlqh22AAqhsUlHHOWfpdEgWNv5IPbHln/qJt7tUllnO0mcjWph2DFvzPCgFvn9kgqgo0I2NbZNkRdtDclsBgdYnuVoyuHcNsNPL1dGn6O2kh3gqW1jCpsb06K1TyztYEynidOKThqcsBq+VOcyuYmruT35mXnuDIhV+4vJWb25uTTwDjJDflr4Kvhs9o9Ox0twYqQiY97nccf9kZl/MQ1gMoUZVX0x7JZzJUcQvqXrN/OBoOxTH+/ljYak2MfT0hqnaQEMlUSwcUD9Scu3gYtA+f+9W+DceBUiuMLo/gx9rUtT5Hw/LlUTpxJJ2dF4Ehq0yDva44VThQRBAwGyYdrTLi3+anTwWZ1QE23KXlt8O9UOImcu9uWoqFCMvT73FOwC1Rv83bUi2q40vvaugJwTVIH4fJ7AvgLE0cCNl9QZyx8sZMFc0Wqdx2flnlh+hUgtNKkmGfA5Ymp/5uJiDo299zTjkE75m6Q3304EeiHibgu1P89RKtdjyYNTGxfvtwiVktZ2ctGExZowm9vlzW0X5ScHukvC+1CPX0+5JWETz9HWynldjrDf7TlymkZdM6e6pMDYaAnWZLeZK3b4yhPEx1h/OdiMt9gOJ/bTXqQ2U3dEcbP5P3kUv4cGR94mvySod4Tds4YBCTv2vCAolxNTIq/JrjlkQFkdCkcmkZn2S9yzcH1drpD++iD3otr5HVPkg9Zte1l2zV+HlELTwl6PAgzZPb+Q/RtEGbZEE5rYvNY79Q3LzTEwLHYDfWTLWVQhK306+f7FZnWsIhFjnrHHFTi9+ysuUs8x405Oeu6050Aoz+bpHSsmGSM6PNyaZcMnSGMz/RmJtnN1j00CXOKBnXFPhFkOFd7xKCC+f9A6MieDh5cxI1617VMVaSc9qjbkHwPvH3YHa51FhGvt73ogXZUqVxUA/BhoJ7FdqaSZrpQrUqrjeI3DJQfVLM7UQRJc28LPqBvC5h3rp2Z4osa7iDZbeG2uNxe0fVq28zuKswsJZ5Pj/YSv7d9PUtj7UL/4x6FHguEVPQ2uo+6PPSoHy31/pnf8gM9vJn24yXxR7PihB4u9JsdeF4rqglZETA+V2m4ucm+QsS0r5JCRyRGT74HWnmKyMsqzDwbOsQtOLgtY7XF79GNDivi9U/BMYeMvoV1VeezHy7fdUDgzqpazg6flxTBBlWy3lfJT0QpDrwLbv/2RMT0Sgk2Gw4DOkYBagPSD2sOxmIF7hSO/i6ujwWV3vhrvOoohL30hXispa2dae8=&cap.sym=KIzjy/ewPSG2R/GHfJ+ogGWGs3Y=&eos.app=fuse&eos.bookingsize=0&eos.checksum=ignore&eos.clientinfo=zbase64:MDAwMDAwNzF4nBXIUQ4CIQxF0a24gmYw+iFJF8PA69BEgXRKjLt3+Dv39oEWzflOIdy0x/3L2mTpzBWFt0X/DfA+RWAoawzTzgWS5ttXX1sN2bliSLLPI9Al8lPbUWcilEm5xbC9nn/hKyf4&eos.lfn=ino:e0320000000&fuse.exe=/usr/bin/git&fuse.gid=601&fuse.pid=180685&fuse.uid=2031&fuse.ver=5.2.0&mgm.checksum=ignore&mgm.fusex=1&mgm.id=0000e032&mgm.logid=f9a72916-7e15-11ee-bfaf-908d6ec41898&mgm.mtime=0&mgm.replicahead=7&mgm.replicaindex=1&xrd.wantprot=unix&mgm.path=/fusex-open&fst.readahead=true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.322197 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0debfd700 source=RainMetaLayout:318             tident=_AfvAlkA.127234:59@hepfarm41 sec=unix  uid=0 gid=0 name=root geo="" msg="failed reading header" url="root://hepfarm41.hep.tsinghua.edu.cn:1095///fusex-open?&cap.msg=ZrG14I29ItdK2N+NAkfmxAscXBQS0tsdTMZMWbfNny9vWXejTLmZehvAv9wm45yafk0Egfoqh28mmP39SvIsIwr7uXd2uv/OYHb7DN44BZ1Ck85JWM0hKsiSC2CdvAz2PTuD752rjOUYBpMcRjVYryOZmDvLdpGE0Nc3zaMcDGvzZPWuLrqNehA778G1s167XEL1zMm8WpMli+U+iNh7SWzweOlQ8WLkvYb+N3dn1ivq+ajWQJVXrA9+tPJTlqh22AAqhsUlHHOWfpdEgWNv5IPbHln/qJt7tUllnO0mcjWph2DFvzPCgFvn9kgqgo0I2NbZNkRdtDclsBgdYnuVoyuHcNsNPL1dGn6O2kh3gqW1jCpsb06K1TyztYEynidOKThqcsBq+VOcyuYmruT35mXnuDIhV+4vJWb25uTTwDjJDflr4Kvhs9o9Ox0twYqQiY97nccf9kZl/MQ1gMoUZVX0x7JZzJUcQvqXrN/OBoOxTH+/ljYak2MfT0hqnaQEMlUSwcUD9Scu3gYtA+f+9W+DceBUiuMLo/gx9rUtT5Hw/LlUTpxJJ2dF4Ehq0yDva44VThQRBAwGyYdrTLi3+anTwWZ1QE23KXlt8O9UOImcu9uWoqFCMvT73FOwC1Rv83bUi2q40vvaugJwTVIH4fJ7AvgLE0cCNl9QZyx8sZMFc0Wqdx2flnlh+hUgtNKkmGfA5Ymp/5uJiDo299zTjkE75m6Q3304EeiHibgu1P89RKtdjyYNTGxfvtwiVktZ2ctGExZowm9vlzW0X5ScHukvC+1CPX0+5JWETz9HWynldjrDf7TlymkZdM6e6pMDYaAnWZLeZK3b4yhPEx1h/OdiMt9gOJ/bTXqQ2U3dEcbP5P3kUv4cGR94mvySod4Tds4YBCTv2vCAolxNTIq/JrjlkQFkdCkcmkZn2S9yzcH1drpD++iD3otr5HVPkg9Zte1l2zV+HlELTwl6PAgzZPb+Q/RtEGbZEE5rYvNY79Q3LzTEwLHYDfWTLWVQhK306+f7FZnWsIhFjnrHHFTi9+ysuUs8x405Oeu6050Aoz+bpHSsmGSM6PNyaZcMnSGMz/RmJtnN1j00CXOKBnXFPhFkOFd7xKCC+f9A6MieDh5cxI1617VMVaSc9qjbkHwPvH3YHa51FhGvt73ogXZUqVxUA/BhoJ7FdqaSZrpQrUqrjeI3DJQfVLM7UQRJc28LPqBvC5h3rp2Z4osa7iDZbeG2uNxe0fVq28zuKswsJZ5Pj/YSv7d9PUtj7UL/4x6FHguEVPQ2uo+6PPSoH
y31/pnf8gM9vJn24yXxR7PihB4u9JsdeF4rqglZETA+V2m4ucm+QsS0r5JCRyRGT74HWnmKyMsqzDwbOsQtOLgtY7XF79GNDivi9U/BMYeMvoV1VeezHy7fdUDgzqpazg6flxTBBlWy3lfJT0QpDrwLbv/2RMT0Sgk2Gw4DOkYBagPSD2sOxmIF7hSO/i6ujwWV3vhrvOoohL30hXispa2dae8=&cap.sym=KIzjy/ewPSG2R/GHfJ+ogGWGs3Y=&eos.app=fuse&eos.bookingsize=0&eos.checksum=ignore&eos.clientinfo=zbase64:MDAwMDAwNzF4nBXIUQ4CIQxF0a24gmYw+iFJF8PA69BEgXRKjLt3+Dv39oEWzflOIdy0x/3L2mTpzBWFt0X/DfA+RWAoawzTzgWS5ttXX1sN2bliSLLPI9Al8lPbUWcilEm5xbC9nn/hKyf4&eos.lfn=ino:e0320000000&fuse.exe=/usr/bin/git&fuse.gid=601&fuse.pid=180685&fuse.uid=2031&fuse.ver=5.2.0&mgm.checksum=ignore&mgm.fusex=1&mgm.id=0000e032&mgm.logid=f9a72916-7e15-11ee-bfaf-908d6ec41898&mgm.mtime=0&mgm.replicahead=7&mgm.replicaindex=2&xrd.wantprot=unix&mgm.path=/fusex-open&fst.readahead=true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.322503 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0debfd700 source=RainMetaLayout:3
18             tident=_AfvAlkA.127234:59@hepfarm41 sec=unix  uid=0 gid=0 name=root geo="" msg="failed reading header" url="root://hepfarm41.hep.tsinghua.edu.cn:1095///fusex-open?&cap.msg=ZrG14I29ItdK2N+NAkfmxAscX
BQS0tsdTMZMWbfNny9vWXejTLmZehvAv9wm45yafk0Egfoqh28mmP39SvIsIwr7uXd2uv/OYHb7DN44BZ1Ck85JWM0hKsiSC2CdvAz2PTuD752rjOUYBpMcRjVYryOZmDvLdpGE0Nc3zaMcDGvzZPWuLrqNehA778G1s167XEL1zMm8WpMli+U+iNh7SWzweOlQ8WLkvYb+N3dn1ivq+
ajWQJVXrA9+tPJTlqh22AAqhsUlHHOWfpdEgWNv5IPbHln/qJt7tUllnO0mcjWph2DFvzPCgFvn9kgqgo0I2NbZNkRdtDclsBgdYnuVoyuHcNsNPL1dGn6O2kh3gqW1jCpsb06K1TyztYEynidOKThqcsBq+VOcyuYmruT35mXnuDIhV+4vJWb25uTTwDjJDflr4Kvhs9o9Ox0twYqQi
Y97nccf9kZl/MQ1gMoUZVX0x7JZzJUcQvqXrN/OBoOxTH+/ljYak2MfT0hqnaQEMlUSwcUD9Scu3gYtA+f+9W+DceBUiuMLo/gx9rUtT5Hw/LlUTpxJJ2dF4Ehq0yDva44VThQRBAwGyYdrTLi3+anTwWZ1QE23KXlt8O9UOImcu9uWoqFCMvT73FOwC1Rv83bUi2q40vvaugJwTVIH4
fJ7AvgLE0cCNl9QZyx8sZMFc0Wqdx2flnlh+hUgtNKkmGfA5Ymp/5uJiDo299zTjkE75m6Q3304EeiHibgu1P89RKtdjyYNTGxfvtwiVktZ2ctGExZowm9vlzW0X5ScHukvC+1CPX0+5JWETz9HWynldjrDf7TlymkZdM6e6pMDYaAnWZLeZK3b4yhPEx1h/OdiMt9gOJ/bTXqQ2U3dE
cbP5P3kUv4cGR94mvySod4Tds4YBCTv2vCAolxNTIq/JrjlkQFkdCkcmkZn2S9yzcH1drpD++iD3otr5HVPkg9Zte1l2zV+HlELTwl6PAgzZPb+Q/RtEGbZEE5rYvNY79Q3LzTEwLHYDfWTLWVQhK306+f7FZnWsIhFjnrHHFTi9+ysuUs8x405Oeu6050Aoz+bpHSsmGSM6PNyaZcMn
SGMz/RmJtnN1j00CXOKBnXFPhFkOFd7xKCC+f9A6MieDh5cxI1617VMVaSc9qjbkHwPvH3YHa51FhGvt73ogXZUqVxUA/BhoJ7FdqaSZrpQrUqrjeI3DJQfVLM7UQRJc28LPqBvC5h3rp2Z4osa7iDZbeG2uNxe0fVq28zuKswsJZ5Pj/YSv7d9PUtj7UL/4x6FHguEVPQ2uo+6PPSoH
y31/pnf8gM9vJn24yXxR7PihB4u9JsdeF4rqglZETA+V2m4ucm+QsS0r5JCRyRGT74HWnmKyMsqzDwbOsQtOLgtY7XF79GNDivi9U/BMYeMvoV1VeezHy7fdUDgzqpazg6flxTBBlWy3lfJT0QpDrwLbv/2RMT0Sgk2Gw4DOkYBagPSD2sOxmIF7hSO/i6ujwWV3vhrvOoohL30hXisp
a2dae8=&cap.sym=KIzjy/ewPSG2R/GHfJ+ogGWGs3Y=&eos.app=fuse&eos.bookingsize=0&eos.checksum=ignore&eos.clientinfo=zbase64:MDAwMDAwNzF4nBXIUQ4CIQxF0a24gmYw+iFJF8PA69BEgXRKjLt3+Dv39oEWzflOIdy0x/3L2mTpzBWFt0X/DfA+RWAoa
wzTzgWS5ttXX1sN2bliSLLPI9Al8lPbUWcilEm5xbC9nn/hKyf4&eos.lfn=ino:e0320000000&fuse.exe=/usr/bin/git&fuse.gid=601&fuse.pid=180685&fuse.uid=2031&fuse.ver=5.2.0&mgm.checksum=ignore&mgm.fusex=1&mgm.id=0000e032&mgm.logi
d=f9a72916-7e15-11ee-bfaf-908d6ec41898&mgm.mtime=0&mgm.replicahead=7&mgm.replicaindex=3&xrd.wantprot=unix&mgm.path=/fusex-open&fst.readahead=true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.323016 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0debfd700 source=RainMetaLayout:3
18             tident=_AfvAlkA.127234:59@hepfarm41 sec=unix  uid=0 gid=0 name=root geo="" msg="failed reading header" url="root://hepfarm41.hep.tsinghua.edu.cn:1095///fusex-open?&cap.msg=ZrG14I29ItdK2N+NAkfmxAscX
BQS0tsdTMZMWbfNny9vWXejTLmZehvAv9wm45yafk0Egfoqh28mmP39SvIsIwr7uXd2uv/OYHb7DN44BZ1Ck85JWM0hKsiSC2CdvAz2PTuD752rjOUYBpMcRjVYryOZmDvLdpGE0Nc3zaMcDGvzZPWuLrqNehA778G1s167XEL1zMm8WpMli+U+iNh7SWzweOlQ8WLkvYb+N3dn1ivq+
ajWQJVXrA9+tPJTlqh22AAqhsUlHHOWfpdEgWNv5IPbHln/qJt7tUllnO0mcjWph2DFvzPCgFvn9kgqgo0I2NbZNkRdtDclsBgdYnuVoyuHcNsNPL1dGn6O2kh3gqW1jCpsb06K1TyztYEynidOKThqcsBq+VOcyuYmruT35mXnuDIhV+4vJWb25uTTwDjJDflr4Kvhs9o9Ox0twYqQi
Y97nccf9kZl/MQ1gMoUZVX0x7JZzJUcQvqXrN/OBoOxTH+/ljYak2MfT0hqnaQEMlUSwcUD9Scu3gYtA+f+9W+DceBUiuMLo/gx9rUtT5Hw/LlUTpxJJ2dF4Ehq0yDva44VThQRBAwGyYdrTLi3+anTwWZ1QE23KXlt8O9UOImcu9uWoqFCMvT73FOwC1Rv83bUi2q40vvaugJwTVIH4
fJ7AvgLE0cCNl9QZyx8sZMFc0Wqdx2flnlh+hUgtNKkmGfA5Ymp/5uJiDo299zTjkE75m6Q3304EeiHibgu1P89RKtdjyYNTGxfvtwiVktZ2ctGExZowm9vlzW0X5ScHukvC+1CPX0+5JWETz9HWynldjrDf7TlymkZdM6e6pMDYaAnWZLeZK3b4yhPEx1h/OdiMt9gOJ/bTXqQ2U3dE
cbP5P3kUv4cGR94mvySod4Tds4YBCTv2vCAolxNTIq/JrjlkQFkdCkcmkZn2S9yzcH1drpD++iD3otr5HVPkg9Zte1l2zV+HlELTwl6PAgzZPb+Q/RtEGbZEE5rYvNY79Q3LzTEwLHYDfWTLWVQhK306+f7FZnWsIhFjnrHHFTi9+ysuUs8x405Oeu6050Aoz+bpHSsmGSM6PNyaZcMn
SGMz/RmJtnN1j00CXOKBnXFPhFkOFd7xKCC+f9A6MieDh5cxI1617VMVaSc9qjbkHwPvH3YHa51FhGvt73ogXZUqVxUA/BhoJ7FdqaSZrpQrUqrjeI3DJQfVLM7UQRJc28LPqBvC5h3rp2Z4osa7iDZbeG2uNxe0fVq28zuKswsJZ5Pj/YSv7d9PUtj7UL/4x6FHguEVPQ2uo+6PPSoH
y31/pnf8gM9vJn24yXxR7PihB4u9JsdeF4rqglZETA+V2m4ucm+QsS0r5JCRyRGT74HWnmKyMsqzDwbOsQtOLgtY7XF79GNDivi9U/BMYeMvoV1VeezHy7fdUDgzqpazg6flxTBBlWy3lfJT0QpDrwLbv/2RMT0Sgk2Gw4DOkYBagPSD2sOxmIF7hSO/i6ujwWV3vhrvOoohL30hXisp
a2dae8=&cap.sym=KIzjy/ewPSG2R/GHfJ+ogGWGs3Y=&eos.app=fuse&eos.bookingsize=0&eos.checksum=ignore&eos.clientinfo=zbase64:MDAwMDAwNzF4nBXIUQ4CIQxF0a24gmYw+iFJF8PA69BEgXRKjLt3+Dv39oEWzflOIdy0x/3L2mTpzBWFt0X/DfA+RWAoa
wzTzgWS5ttXX1sN2bliSLLPI9Al8lPbUWcilEm5xbC9nn/hKyf4&eos.lfn=ino:e0320000000&fuse.exe=/usr/bin/git&fuse.gid=601&fuse.pid=180685&fuse.uid=2031&fuse.ver=5.2.0&mgm.checksum=ignore&mgm.fusex=1&mgm.id=0000e032&mgm.logi
d=f9a72916-7e15-11ee-bfaf-908d6ec41898&mgm.mtime=0&mgm.replicahead=7&mgm.replicaindex=4&xrd.wantprot=unix&mgm.path=/fusex-open&fst.readahead=true&fst.blocksize=1048576"
231108 17:05:34 time=1699434334.323407 func=Open                     level=WARN  logid=f9a72916-7e15-11ee-bfaf-908d6ec41898 unit=fst@hepfarm41.hep.tsinghua.edu.cn:1095 tid=00007fa0debfd700 source=RainMetaLayout:3
18             tident=_AfvAlkA.127234:59@hepfarm41 sec=unix  uid=0 gid=0 name=root geo="" msg="failed reading header" url="root://hepfarm41.hep.tsinghua.edu.cn:1095///fusex-open?&cap.msg=ZrG14I29ItdK2N+NAkfmxAscX
BQS0tsdTMZMWbfNny9vWXejTLmZehvAv9wm45yafk0Egfoqh28mmP39SvIsIwr7uXd2uv/OYHb7DN44BZ1Ck85JWM0hKsiSC2CdvAz2PTuD752rjOUYBpMcRjVYryOZmDvLdpGE0Nc3zaMcDGvzZPWuLrqNehA778G1s167XEL1zMm8WpMli+U+iNh7SWzweOlQ8WLkvYb+N3dn1ivq+
ajWQJVXrA9+tPJTlqh22AAqhsUlHHOWfpdEgWNv5IPbHln/qJt7tUllnO0mcjWph2DFvzPCgFvn9kgqgo0I2NbZNkRdtDclsBgdYnuVoyuHcNsNPL1dGn6O2kh3gqW1jCpsb06K1TyztYEynidOKThqcsBq+VOcyuYmruT35mXnuDIhV+4vJWb25uTTwDjJDflr4Kvhs9o9Ox0twYqQi
Y97nccf9kZl/MQ1gMoUZVX0x7JZzJUcQvqXrN/OBoOxTH+/ljYak2MfT0hqnaQEMlUSwcUD9Scu3gYtA+f+9W+DceBUiuMLo/gx9rUtT5Hw/LlUTpxJJ2dF4Ehq0yDva44VThQRBAwGyYdrTLi3+anTwWZ1QE23KXlt8O9UOImcu9uWoqFCMvT73FOwC1Rv83bUi2q40vvaugJwTVIH4
fJ7AvgLE0cCNl9QZyx8sZMFc0Wqdx2flnlh+hUgtNKkmGfA5Ymp/5uJiDo299zTjkE75m6Q3304EeiHibgu1P89RKtdjyYNTGxfvtwiVktZ2ctGExZowm9vlzW0X5ScHukvC+1CPX0+5JWETz9HWynldjrDf7TlymkZdM6e6pMDYaAnWZLeZK3b4yhPEx1h/OdiMt9gOJ/bTXqQ2U3dE
cbP5P3kUv4cGR94mvySod4Tds4YBCTv2vCAolxNTIq/JrjlkQFkdCkcmkZn2S9yzcH1drpD++iD3otr5HVPkg9Zte1l2zV+HlELTwl6PAgzZPb+Q/RtEGbZEE5rYvNY79Q3LzTEwLHYDfWTLWVQhK306+f7FZnWsIhFjnrHHFTi9+ysuUs8x405Oeu6050Aoz+bpHSsmGSM6PNyaZcMn
SGMz/RmJtnN1j00CXOKBnXFPhFkOFd7xKCC+f9A6MieDh5cxI1617VMVaSc9qjbkHwPvH3YHa51FhGvt73ogXZUqVxUA/BhoJ7FdqaSZrpQrUqrjeI3DJQfVLM7UQRJc28LPqBvC5h3rp2Z4osa7iDZbeG2uNxe0fVq28zuKswsJZ5Pj/YSv7d9PUtj7UL/4x6FHguEVPQ2uo+6PPSoH
y31/pnf8gM9vJn24yXxR7PihB4u9JsdeF4rqglZETA+V2m4ucm+QsS0r5JCRyRGT74HWnmKyMsqzDwbOsQtOLgtY7XF79GNDivi9U/BMYeMvoV1VeezHy7fdUDgzqpazg6flxTBBlWy3lfJT0QpDrwLbv/2RMT0Sgk2Gw4DOkYBagPSD2sOxmIF7hSO/i6ujwWV3vhrvOoohL30hXisp
a2dae8=&cap.sym=KIzjy/ewPSG2R/GHfJ+ogGWGs3Y=&eos.app=fuse&eos.bookingsize=0&eos.checksum=ignore&eos.clientinfo=zbase64:MDAwMDAwNzF4nBXIUQ4CIQxF0a24gmYw+iFJF8PA69BEgXRKjLt3+Dv39oEWzflOIdy0x/3L2mTpzBWFt0X/DfA+RWAoa
wzTzgWS5ttXX1sN2bliSLLPI9Al8lPbUWcilEm5xbC9nn/hKyf4&eos.lfn=ino:e0320000000&fuse.exe=/usr/bin/git&fuse.gid=601&fuse.pid=180685&fuse.uid=2031&fuse.ver=5.2.0&mgm.checksum=ignore&mgm.fusex=1&mgm.id=0000e032&mgm.logi
d=f9a72916-7e15-11ee-bfaf-908d6ec41898&mgm.mtime=0&mgm.replicahead=7&mgm.replicaindex=5&xrd.wantprot=unix&mgm.path=/fusex-open&fst.readahead=true&fst.blocksize=1048576"

We did many tries, including changing the layout to other RAIN layout like archive, raid6 or raid5, changing the sys.forced.nstripes to 6 or 20, and changing the sys.forced.blocksize from 1M to 4k, but the problem always exists. Changing the layout to replica can solve this problem, so we are temporarily using replica layout.

anfeng · November 8, 2023, 5:21pm

I tried git clone on a different EOS instance, which is on virtual machines with virtual disks and RAIN layout, and still cannot clone successfully, with similar warning logs.

esindril · November 10, 2023, 8:29am

Hi Anfeng,

The RAIN/Errasure coded layouts are write-once/read-many and currently there is no support for updates for RAIN files. Therefore, a git repository is definitely not a good candidate to put in RAIN layout - first of all because of the lack of update functionality and secondly due to the waste of disk space / overhead of storing small files in RAIN layout.

The RAIN layout in general is very efficient and good for files which are more than 100MB. The way we use RAIN at CERN is only to enable it for data that comes from experiments where we know for sure the size of the files is big, usually 2GB, or we enable it by converting existing files which are more than 100MB.

So it is not a surprize that a git clone on a RAIN layout does not work and there are good reason why this is the case. As you already noticed, for such cases you can use a replica layout or even a simple plain layout (as you probably already have a backup of the repository somewhere else).

Hope this helps!
Cheers,
Elvin

telzhov · November 10, 2023, 1:54pm

Hi esindril,

You’re saying “updates” here so it means it’s about opening EOS files through FUSE layer in ‘append’ mode, doesn’t it? And this has nothing common with rewriting file on EOS via eos cp (which should erase the existing file, and then create a new one), that’s correct?

Thank you!

anfeng · November 10, 2023, 6:30pm

Hi Elvin,

Thank you very much for your explanations. This explains several problems that I met with RAIN layouts. It would be better if you can write this in the documentation, or it is possible to implement the updating of RAIN files by just rewriting the whole file or like some way in a traditional RAID array? This can at least provide full filesystem functionalities in RAIN layouts despite some performance loss.

It’s a pity that we can only use replica for full filesystem functionalities with redundancy, since replica:2 layout doubles the space usage and slows the writing speed (This is actually another question that I want to ask. We observed just 1/2 and 1/4 of the single disk writing speed when writing with replica:2 and replica:4 layouts, which should not be so slow if disks can be written simultaneously).

As for mixed layouts, is there a way for us to enable automatic conversion of existing file which are more than like 100MB from replica layout to RAIN layout? (or I missed the configuration in the docs?)

Thank you!
Anfeng

esindril · November 13, 2023, 7:54am

Hi Timur,

Yes, you are absolutely right! Appending or modifying the contents is forbidden but overwriting the full file works as this is just a new file (the old one is deleted).

Cheers,
Elvin

esindril · November 13, 2023, 8:12am

Hi Anfeng,

We have plans to add support for RAIN updates but this relies on a feature that will be first implemented in the XRootD framework - namely copy-on-write functionality, that would allow us to add this feature without the risk of corrupting the data if something goes wrong mid-flight. ETA probably end of next year. I will add this limitation to the documentation.

You can improve the write performance by setting the following env variable on all FSTs so that writes are done asynchronously between FSTs:
EOS_FST_REPLICA_ASYNC_WRITE=1

This has an important impact especially in setups where there is considerable latency between the FSTs or in case the disks are slow. I am sure this will help in your case.

Yes, you can set up the LRU engine to scan regularly a certain sub-directory and convert the files from the existing layout to a new one, satisfying a certain size constraint. You can find documentation about this feature here:
https://eos-docs.web.cern.ch/diopside/manual/microservices.html#lru-engine

The easiest way to fine the hex value of the new layout is to look at an existing file that is already in the desired layout with the eos fileinfo command and use the LayoutId field.

Cheers,
Elvin

CERN Accelerating science

Problem with RAIN layouts