Can not fsck repair for RAIN files

GeonmoRyu · May 14, 2025, 6:32am

Hello, everyone.

After the EOS 5.3 version upgrade, when I repair RAIN files with fsck repair, it doesn’t seem to do the conversion.

Even for files with all stripes with nrep=16, when I run fsck repair, it doesn’t show as normal or successful, it shows as failed.

Even when I tried fsck repair on a file with 15 stripes, sometimes it would say it was successful, and I would check the file and see that the stripe numbers were the same, but the old stripes had been moved to the new FST.

I was wondering if you could help me with this?

EOS Console [root://localhost] |/eos/gsdc/proc/conversion/f5/> file check fxid:0088e14b                                                                                                                           
path="/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998" fxid="0088e14b" size="1967736" nrep="15" checksumtype="adler" checksum="ace9596000000000000000000000000000000000000000000000000000000000"     
nrep="00" fsid="1357" host="jbod-mgmt-09.sdfarm.kr:1095" fstpath="/jbod/box_17_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                   
nrep="01" fsid="517" host="jbod-mgmt-04.sdfarm.kr:1095" fstpath="/jbod/box_07_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                    
nrep="02" fsid="13" host="jbod-mgmt-01.sdfarm.kr:1095" fstpath="/jbod/box_01_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                     
nrep="03" fsid="601" host="jbod-mgmt-04.sdfarm.kr:1096" fstpath="/jbod/box_08_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                    
nrep="04" fsid="349" host="jbod-mgmt-03.sdfarm.kr:1095" fstpath="/jbod/box_05_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                    
nrep="05" fsid="853" host="jbod-mgmt-06.sdfarm.kr:1095" fstpath="/jbod/box_11_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                    
nrep="06" fsid="1189" host="jbod-mgmt-08.sdfarm.kr:1095" fstpath="/jbod/box_15_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                   
nrep="07" fsid="1021" host="jbod-mgmt-07.sdfarm.kr:1095" fstpath="/jbod/box_13_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                   
nrep="08" fsid="769" host="jbod-mgmt-05.sdfarm.kr:1096" fstpath="/jbod/box_10_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                    
nrep="09" fsid="97" host="jbod-mgmt-01.sdfarm.kr:1096" fstpath="/jbod/box_02_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                     
nrep="10" fsid="1273" host="jbod-mgmt-08.sdfarm.kr:1096" fstpath="/jbod/box_16_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                   
nrep="11" fsid="685" host="jbod-mgmt-05.sdfarm.kr:1095" fstpath="/jbod/box_09_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                    
nrep="12" fsid="22012" host="jbod-mgmt-11.sdfarm.kr:1096" fstpath="/jbod/box_22_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                  
nrep="13" fsid="265" host="jbod-mgmt-02.sdfarm.kr:1096" fstpath="/jbod/box_04_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"                                                    
nrep="14" fsid="433" host="jbod-mgmt-03.sdfarm.kr:1096" fstpath="/jbod/box_06_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"

EOS Console [root://localhost] |/eos/gsdc/proc/conversion/f5/> fsck repair --fxid 0088e14b                
msg="repair successful"

EOS Console [root://localhost] |/eos/gsdc/proc/conversion/f5/> file check fxid:0088e14b
path="/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998" fxid="0088e14b" size="1967736" nrep="15" checksumtype="adler" checksum="ace9596000000000000000000000000000000000000000000000000000000000"
nrep="00" fsid="517" host="jbod-mgmt-04.sdfarm.kr:1095" fstpath="/jbod/box_07_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="01" fsid="13" host="jbod-mgmt-01.sdfarm.kr:1095" fstpath="/jbod/box_01_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="02" fsid="601" host="jbod-mgmt-04.sdfarm.kr:1096" fstpath="/jbod/box_08_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="03" fsid="349" host="jbod-mgmt-03.sdfarm.kr:1095" fstpath="/jbod/box_05_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="04" fsid="853" host="jbod-mgmt-06.sdfarm.kr:1095" fstpath="/jbod/box_11_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="05" fsid="1189" host="jbod-mgmt-08.sdfarm.kr:1095" fstpath="/jbod/box_15_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="06" fsid="1021" host="jbod-mgmt-07.sdfarm.kr:1095" fstpath="/jbod/box_13_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="07" fsid="769" host="jbod-mgmt-05.sdfarm.kr:1096" fstpath="/jbod/box_10_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="08" fsid="97" host="jbod-mgmt-01.sdfarm.kr:1096" fstpath="/jbod/box_02_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="09" fsid="1273" host="jbod-mgmt-08.sdfarm.kr:1096" fstpath="/jbod/box_16_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="10" fsid="685" host="jbod-mgmt-05.sdfarm.kr:1095" fstpath="/jbod/box_09_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="11" fsid="22012" host="jbod-mgmt-11.sdfarm.kr:1096" fstpath="/jbod/box_22_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="12" fsid="265" host="jbod-mgmt-02.sdfarm.kr:1096" fstpath="/jbod/box_04_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="13" fsid="433" host="jbod-mgmt-03.sdfarm.kr:1096" fstpath="/jbod/box_06_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
nrep="14" fsid="937" host="jbod-mgmt-06.sdfarm.kr:1096" fstpath="/jbod/box_12_disk_012/00000381/0088e14b" size="1967736" statsize="1052672" error_label="none"
'''

esindril · May 14, 2025, 7:15am

Hi Geonmo,

Could you tell me exactly what version you are running in your cluster?
eos version and do all the FSTs run the same version?

In general, it’s always good to specify the type of error that the FSCK mechanism needs to fix otherwise it might have a hard time deciding which repair procedure to apply.

For this particular case can you also print the following information:
eos fileinfo fxid:0088e14b
Double check how many file systems are available in the group that the stripes of this file belong to. If there are enough for a repair to happen then issue the following command:
eos fsck repair --fixd 0088e14b --fsid 433 --error rep_diff_n

If you still get an error, or a successful message without any modifications of the file, please attach the log of the MGM during that period and I will check it out.

Thanks,
Elvin

GeonmoRyu · May 29, 2025, 6:38am

Hello, Elvin.

This did not resolve the issue, so we have now upgraded all MGMs and FSTs on our system to 5.3.13 and retested, and are seeing the same error.

EOS Console [root://localhost] |/eos/gsdc/grid/> file verify file-10mb.add
success: sending verify to fsid=1450 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=1366 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=778 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=274 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=106 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=862 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=946 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=190 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=358 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=442 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=1282 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=526 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=1198 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=610 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=1114 for path=/eos/gsdc/grid/file-10mb.add
success: sending verify to fsid=21021 for path=/eos/gsdc/grid/file-10mb.add
EOS Console [root://localhost] |/eos/gsdc/grid/> file check file-10mb.add
path=“/eos/gsdc/grid/file-10mb.add” fxid=“0175788a” size=“10459447” nrep=“16” checksumtype=“adler” checksum=“27d20a5c00000000000000000000000000000000000000000000000000000000”
nrep=“00” fsid=“1450” host=“jbod-mgmt-09.sdfarm.kr:1096” fstpath=“/jbod/box_18_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“01” fsid=“1366” host=“jbod-mgmt-09.sdfarm.kr:1095” fstpath=“/jbod/box_17_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“02” fsid=“778” host=“jbod-mgmt-05.sdfarm.kr:1096” fstpath=“/jbod/box_10_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“03” fsid=“274” host=“jbod-mgmt-02.sdfarm.kr:1096” fstpath=“/jbod/box_04_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“04” fsid=“106” host=“jbod-mgmt-01.sdfarm.kr:1096” fstpath=“/jbod/box_02_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“05” fsid=“862” host=“jbod-mgmt-06.sdfarm.kr:1095” fstpath=“/jbod/box_11_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“06” fsid=“946” host=“jbod-mgmt-06.sdfarm.kr:1096” fstpath=“/jbod/box_12_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“07” fsid=“190” host=“jbod-mgmt-02.sdfarm.kr:1095” fstpath=“/jbod/box_03_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“08” fsid=“358” host=“jbod-mgmt-03.sdfarm.kr:1095” fstpath=“/jbod/box_05_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“09” fsid=“442” host=“jbod-mgmt-03.sdfarm.kr:1096” fstpath=“/jbod/box_06_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“10” fsid=“1282” host=“jbod-mgmt-08.sdfarm.kr:1096” fstpath=“/jbod/box_16_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“11” fsid=“526” host=“jbod-mgmt-04.sdfarm.kr:1095” fstpath=“/jbod/box_07_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“12” fsid=“1198” host=“jbod-mgmt-08.sdfarm.kr:1095” fstpath=“/jbod/box_15_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“13” fsid=“610” host=“jbod-mgmt-04.sdfarm.kr:1096” fstpath=“/jbod/box_08_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“14” fsid=“1114” host=“jbod-mgmt-07.sdfarm.kr:1096” fstpath=“/jbod/box_14_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
nrep=“15” fsid=“21021” host=“jbod-mgmt-11.sdfarm.kr:1095” fstpath=“/jbod/box_21_disk_021/0000098f/0175788a” size=“10459447” statsize=“1052672” error_label=“none”
EOS Console [root://localhost] |/eos/gsdc/grid/> fsck repair --fxid 0175788a
msg=“repair job failed”

The error log level is a bit high as “notice”, but I was able to see the logs below.

14812525:250529 06:23:21 time=1748499801.798255 func=ReportError level=ERROR logid=6b3fe866-3c55-11f0-b0b3-b8599f9c4330 unit=mgm@jbod-mgmt-04.sdfarm.kr:1094 tid=00007efd36bfd640 source=DrainTransferJob:66 tident= sec= uid=0 gid=0 name= geo=“” xt=“” ob=“” msg=“fxid=0175788a rain reconstruct already failed”
14812527:250529 06:23:21 time=1748499801.798601 func=RepairFstXsSzDiff level=ERROR logid=6b05ed0a-3c55-11f0-b0b3-b8599f9c4330 unit=mgm@jbod-mgmt-04.sdfarm.kr:1094 tid=00007efd36bfd640 source=FsckEntry:657 tident= sec= uid=0 gid=0 name= geo=“” xt=“” ob=“” msg=“fst xs/size repair failed” fxid=0175788a bad_fsid=0

Let us know if there’s anything else we can help you with.

esindril · June 2, 2025, 6:40am

Hi Geonmo,

Can you please provide the info that I requested in the previous post?
Also check the fsck command that I suggested when trying to repair under-replicated files.

The file verify command that you issued is not supposed to fix anything, so there is no surprise this did not alter the state of the file. I would be helpful to concentrate on one particular file and figure out if there is actually a problem with it and if the repair procedure can fix it or not.

Cheers,
Elvin

GeonmoRyu · June 4, 2025, 5:41am

Hello, Elvin.

I tried to repair previous fxid=0088e14b file.

EOS Console [root://localhost] |/eos/gsdc/grid/> fileinfo fxid:0088e14b
  File: '/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998'  Flags: 0600  Clock: 1845bfee7fb61cc4
  Size: 1967736
Status: locations::incomplete
Modify: Mon May 30 11:54:31 2022 Timestamp: 1653911671.820990000
Change: Mon May 30 11:54:25 2022 Timestamp: 1653911665.496800666
Access: Thu Jan  1 00:00:00 1970 Timestamp: 0.000000000
 Birth: Mon May 30 11:54:25 2022 Timestamp: 1653911665.496800666
  CUid: 10367 CGid: 1395 Fxid: 0088e14b Fid: 8970571 Pid: 17810 Pxid: 00004592
XStype: adler    XS: ac e9 59 60    ETAGs: "2408019316965376:ace95960"
Layout: qrain Stripes: 16 Blocksize: 1M LayoutId: 40640f52 Redundancy: d4::t0 
  #Rep: 15
┌───┬──────┬────────────────────────┬────────────────┬─────────────────────┬──────────┬──────────────┬────────────┬────────┬────────────────────────┐
│no.│ fs-id│                    host│      schedgroup│                 path│      boot│  configstatus│       drain│  active│                  geotag│
└───┴──────┴────────────────────────┴────────────────┴─────────────────────┴──────────┴──────────────┴────────────┴────────┴────────────────────────┘
 0      517   jbod-mgmt-04.sdfarm.kr       default.12 /jbod/box_07_disk_012     booted             rw      nodrain   online         kisti::gsdc::g02 
 1       13   jbod-mgmt-01.sdfarm.kr       default.12 /jbod/box_01_disk_012     booted             rw      nodrain   online         kisti::gsdc::g01 
 2      601   jbod-mgmt-04.sdfarm.kr       default.12 /jbod/box_08_disk_012     booted             rw      nodrain   online         kisti::gsdc::g02 
 3      349   jbod-mgmt-03.sdfarm.kr       default.12 /jbod/box_05_disk_012     booted             rw      nodrain   online         kisti::gsdc::g01 
 4      853   jbod-mgmt-06.sdfarm.kr       default.12 /jbod/box_11_disk_012     booted             rw      nodrain   online         kisti::gsdc::g02 
 5     1189   jbod-mgmt-08.sdfarm.kr       default.12 /jbod/box_15_disk_012     booted             rw      nodrain   online         kisti::gsdc::g03 
 6     1021   jbod-mgmt-07.sdfarm.kr       default.12 /jbod/box_13_disk_012     booted             rw      nodrain   online         kisti::gsdc::g03 
 7       97   jbod-mgmt-01.sdfarm.kr       default.12 /jbod/box_02_disk_012     booted             rw      nodrain   online         kisti::gsdc::g01 
 8     1273   jbod-mgmt-08.sdfarm.kr       default.12 /jbod/box_16_disk_012     booted             rw      nodrain   online         kisti::gsdc::g03 
 9      685   jbod-mgmt-05.sdfarm.kr       default.12 /jbod/box_09_disk_012     booted             rw      nodrain   online         kisti::gsdc::g02 
 10   22012   jbod-mgmt-11.sdfarm.kr       default.12 /jbod/box_22_disk_012     booted             rw      nodrain   online         kisti::gsdc::e01 
 11     265   jbod-mgmt-02.sdfarm.kr       default.12 /jbod/box_04_disk_012     booted             rw      nodrain   online         kisti::gsdc::g01 
 12     433   jbod-mgmt-03.sdfarm.kr       default.12 /jbod/box_06_disk_012     booted             rw      nodrain   online         kisti::gsdc::g01 
 13     937   jbod-mgmt-06.sdfarm.kr       default.12 /jbod/box_12_disk_012     booted             rw      nodrain   online         kisti::gsdc::g02 
 14   23012   jbod-mgmt-12.sdfarm.kr       default.12 /jbod/box_23_disk_012     booted             rw      nodrain   online         kisti::gsdc::e01 

*******

EOS Console [root://localhost] |/eos/gsdc/grid/> fsck repair --fxid 0088e14b --error rep_diff_n

msg="repair job failed"

[root@jbod-mgmt-04 ~]# cat xrootd.mgm.failed.fsck.log | egrep "0088e14b|ad59e1c1-db70-11ec-b821-3cecef03e998|tid=00007effc75df640"
250604 05:30:47 time=1749015047.103887 func=DoIt                     level=ERROR logid=bb2af79a-4104-11f0-a985-b8599f9c4330 unit=mgm@jbod-mgmt-04.sdfarm.kr:1094 tid=00007effc75df640 source=DrainTransferJob:186           tident=<service> sec=      uid=0 gid=0 name= geo="" xt="" ob="" src=root://jbod-mgmt-04.sdfarm.kr:1094//#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998 dst=root://jbod-mgmt-07.sdfarm.kr:1096//replicate:0 logid=bb2b191e-4104-11f0-a985-b8599f9c4330 tpc_err=[ERROR] Server responded with an error: [3005] sync - TPC failed source close src_url=root://211@jbod-mgmt-04.sdfarm.kr:1096//#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998?fst.readahead=true src_err=[ERROR] Socket timeout
250604 05:30:47 time=1749015047.104464 func=ReportError              level=ERROR logid=bb2af79a-4104-11f0-a985-b8599f9c4330 unit=mgm@jbod-mgmt-04.sdfarm.kr:1094 tid=00007effc75df640 source=DrainTransferJob:66            tident=<service> sec=      uid=0 gid=0 name= geo="" xt="" ob="" msg="fxid=0088e14b rain reconstruct already failed"
250604 05:30:47 time=1749015047.104930 func=DoIt                     level=ERROR logid=static.............................. unit=mgm@jbod-mgmt-04.sdfarm.kr:1094 tid=00007effc75df640 source=DrainTransferJob:145           tident= sec=(null) uid=0 gid=0 name=- geo="" xt="" ob="" msg="url invalid" src="" dst="root://daemon@jbod-mgmt-07.sdfarm.kr:1096//replicate:0?cap.msg=kKJytp6YGplT6O/TG1CdNsUDzr4BWSXtOOpKO1jPQUAaHEQJVq4j6VWg7bFINxG6u3VzRO8HArAenv2GIQTixBPIglx9LUvAaeAV+U6pKr6ACRN3MDACq6bp8RL2TsvBfvHRV4Hx9FqbgELfKguRRdno81lCqk3kdd/Cg8BIuuqgypJ4s/KxlDVqWFWnmV4A+ojWSjEcv/y4Nzp6vr4qwTCAETjj8qL/zcz1OgekpnHt37yEcv5rG/2Y9/wFzrqFI9/bMBdiNzyVLW9LliPfeMsyYmZwZKHLoOoxv8+ux7qD8CRQKAobDw==&cap.sym=T3YbdYkv+rl8YFJhtEugBjOFPt0=&mgm.logid=11664d76-4105-11f0-a985-b8599f9c4330"
250604 05:30:47 time=1749015047.104986 func=RepairRainInconsistencies level=ERROR logid=baba8528-4104-11f0-a985-b8599f9c4330 unit=mgm@jbod-mgmt-04.sdfarm.kr:1094 tid=00007effc75df640 source=FsckEntry:871                  tident=<service> sec=      uid=0 gid=0 name= geo="" xt="" ob="" msg="stripe inconsistency repair failed" fxid=0088e14b src_fsid=0

Based on the above message, it looks like the TPC transfer is not being processed because of a socket timeout.

We’d appreciate it if you could provide a workaround for this issue.

Regards,

– Geonmo

esindril · June 4, 2025, 9:13am

Hi Geonmo,

Can you check the FST logs of these two machines
jbod-mgmt-07.sdfarm.kr:1096
jbod-mgmt-04.sdfarm.kr:1096
looking for the follwoing logid bb2b191e-4104-11f0-a985-b8599f9c4330 and post the information so that I can understand what is the reason for the failiure?

Thanks,
Elvin

GeonmoRyu · June 4, 2025, 9:40am

Hello, Elvin.

I attached FST logs.

jbod-mgmt-04.sdfarm.kr:1096

[root@jbod-mgmt-04 fst]# cat xrootd.fst.log| grep bb2c4e60-4104-11f0-9621-b8599f9c4330
250604 05:28:22 time=1749014902.469497 func=open                     level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fb01c2c4640 source=XrdFstOfsFile:183              tident=211.35:5408@jbod-mgmt-04 sec=sss   uid=0 gid=0 name=daemon geo="" xt="" ob="" path=/#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998 info=cap.msg=<...>&cap.sym=<...>&eos.clientinfo=zbase64:MDAwMDAwNmF4nBXIbQ5AMAyA4au4gGU+IrGkhzFtGWZLVcTt2b/nfVOm04lCa7quCsn5B8LJRde8EoIt1DcT+JuZhLCMLCEBEk/3oaX/HYRmhc0nrOMStba9uZAniWYX19hx+ADHLiUW&eos.encodepath=curl&eos.pio.action=reconstruct&eos.pio.recfs=0&mgm.id=0088e14b&mgm.logid=bb2c4e60-4104-11f0-9621-b8599f9c4330&mgm.mtime=1653911671&mgm.replicahead=2&mgm.replicaindex=2&tpc.stage=placement open_mode=0
250604 05:28:22 time=1749014902.469966 func=ProcessCapOpaque         level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fb01c2c4640 source=XrdFstOfsFile:2543             tident=211.35:5408@jbod-mgmt-04 sec=(null) uid=65534 gid=65534 name=(null) geo="" xt="" ob="" capability=&mgm.access=update&mgm.ruid=2&mgm.rgid=2&mgm.uid=99&mgm.gid=99&mgm.path=/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998&mgm.manager=jbod-mgmt-04.sdfarm.kr:1094&mgm.fid=0088e14b&mgm.cid=17810&mgm.sec=sss|daemon|jbod-mgmt-04.sdfarm.kr||daemon|||xrootd&mgm.lid=1080299346&mgm.bookingsize=0&mgm.rain.store=1&mgm.rain.size=1967736&mgm.fsid=601&mgm.url0=root://jbod-mgmt-04.sdfarm.kr:1095//&mgm.fsid0=517&mgm.url1=root://jbod-mgmt-01.sdfarm.kr:1095//&mgm.fsid1=13&mgm.url2=root://jbod-mgmt-04.sdfarm.kr:1096//&mgm.fsid2=601&mgm.url3=root://jbod-mgmt-03.sdfarm.kr:1095//&mgm.fsid3=349&mgm.url4=root://jbod-mgmt-06.sdfarm.kr:1095//&mgm.fsid4=853&mgm.url5=root://jbod-mgmt-08.sdfarm.kr:1095//&mgm.fsid5=1189&mgm.url6=root://jbod-mgmt-07.sdfarm.kr:1095//&mgm.fsid6=1021&mgm.url7=root://jbod-mgmt-01.sdfarm.kr:1096//&mgm.fsid7=97&mgm.url8=root://jbod-mgmt-08.sdfarm.kr:1096//&mgm.fsid8=1273&mgm.url9=root://jbod-mgmt-05.sdfarm.kr:1095//&mgm.fsid9=685&mgm.url10=root://jbod-mgmt-11.sdfarm.kr:1096//&mgm.fsid10=22012&mgm.url11=root://jbod-mgmt-02.sdfarm.kr:1096//&mgm.fsid11=265&mgm.url12=root://jbod-mgmt-03.sdfarm.kr:1096//&mgm.fsid12=433&mgm.url13=root://jbod-mgmt-06.sdfarm.kr:1096//&mgm.fsid13=937&mgm.url14=root://jbod-mgmt-12.sdfarm.kr:1095//&mgm.fsid14=23012&mgm.url15=root://jbod-mgmt-07.sdfarm.kr:1096//&mgm.fsid15=1105&cap.valid=1749018502
250604 05:28:22 time=1749014902.470141 func=open                     level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fb01c2c4640 source=XrdFstOfsFile:226              tident=211.35:5408@jbod-mgmt-04 sec=(null) uid=2 gid=2 name=nobody geo="" xt="" ob="" msg="enabling RAIN store recovery" fxid=0088e14b
250604 05:28:22 time=1749014902.496688 func=open                     level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fb01c2c4640 source=XrdFstOfsFile:303              tident=211.35:5408@jbod-mgmt-04 sec=(null) uid=2 gid=2 name=nobody geo="" xt="" ob="" msg="kTpcSrcSetup return SFS_OK" fxid=0088e14b
250604 05:28:22 time=1749014902.554842 func=_close                   level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fb01c2c4640 source=XrdFstOfsFile:1530             tident=211.35:5408@jbod-mgmt-04 sec=      uid=2 gid=2 name=nobody geo="" xt="" ob="" msg="close already done" fxid=0088e14b close_rc=0
250604 05:28:23 time=1749014903.616871 func=open                     level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fb01c2c4640 source=XrdFstOfsFile:183              tident=211.35:5408@jbod-mgmt-04 sec=sss   uid=0 gid=0 name=daemon geo="" xt="" ob="" path=/#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998 info=cap.msg=<...>&cap.sym=<...>&eos.clientinfo=zbase64:MDAwMDAwNmF4nBXIbQ5AMAyA4au4gGU+IrGkhzFtGWZLVcTt2b/nfVOm04lCa7quCsn5B8LJRde8EoIt1DcT+JuZhLCMLCEBEk/3oaX/HYRmhc0nrOMStba9uZAniWYX19hx+ADHLiUW&eos.encodepath=curl&eos.pio.action=reconstruct&eos.pio.recfs=0&mgm.id=0088e14b&mgm.logid=bb2c4e60-4104-11f0-9621-b8599f9c4330&mgm.mtime=1653911671&mgm.replicahead=2&mgm.replicaindex=2&tpc.dst=jbod-mgmt-07.sdfarm.kr&tpc.key=239de0d200070023683fd976&tpc.stage=copy open_mode=0
250604 05:28:23 time=1749014903.660160 func=ProcessTpcOpaque         level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fb01c2c4640 source=XrdFstOfsFile:2880             tident=211.35:5408@jbod-mgmt-04 sec=sss   uid=0 gid=0 name=daemon geo="" xt="" ob="" msg="tpc src session" key=239de0d200070023683fd976, org=211.35@jbod-mgmt-04.sdfarm.kr, dst=jbod-mgmt-07.sdfarm.kr path=/#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998 expires=1749015023
250604 05:28:23 time=1749014903.660394 func=ProcessCapOpaque         level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fb01c2c4640 source=XrdFstOfsFile:2543             tident=211.35:5408@jbod-mgmt-04 sec=(null) uid=65534 gid=65534 name=(null) geo="" xt="" ob="" capability=&mgm.access=update&mgm.ruid=2&mgm.rgid=2&mgm.uid=99&mgm.gid=99&mgm.path=/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998&mgm.manager=jbod-mgmt-04.sdfarm.kr:1094&mgm.fid=0088e14b&mgm.cid=17810&mgm.sec=sss|daemon|jbod-mgmt-04.sdfarm.kr||daemon|||xrootd&mgm.lid=1080299346&mgm.bookingsize=0&mgm.rain.store=1&mgm.rain.size=1967736&mgm.fsid=601&mgm.url0=root://jbod-mgmt-04.sdfarm.kr:1095//&mgm.fsid0=517&mgm.url1=root://jbod-mgmt-01.sdfarm.kr:1095//&mgm.fsid1=13&mgm.url2=root://jbod-mgmt-04.sdfarm.kr:1096//&mgm.fsid2=601&mgm.url3=root://jbod-mgmt-03.sdfarm.kr:1095//&mgm.fsid3=349&mgm.url4=root://jbod-mgmt-06.sdfarm.kr:1095//&mgm.fsid4=853&mgm.url5=root://jbod-mgmt-08.sdfarm.kr:1095//&mgm.fsid5=1189&mgm.url6=root://jbod-mgmt-07.sdfarm.kr:1095//&mgm.fsid6=1021&mgm.url7=root://jbod-mgmt-01.sdfarm.kr:1096//&mgm.fsid7=97&mgm.url8=root://jbod-mgmt-08.sdfarm.kr:1096//&mgm.fsid8=1273&mgm.url9=root://jbod-mgmt-05.sdfarm.kr:1095//&mgm.fsid9=685&mgm.url10=root://jbod-mgmt-11.sdfarm.kr:1096//&mgm.fsid10=22012&mgm.url11=root://jbod-mgmt-02.sdfarm.kr:1096//&mgm.fsid11=265&mgm.url12=root://jbod-mgmt-03.sdfarm.kr:1096//&mgm.fsid12=433&mgm.url13=root://jbod-mgmt-06.sdfarm.kr:1096//&mgm.fsid13=937&mgm.url14=root://jbod-mgmt-12.sdfarm.kr:1095//&mgm.fsid14=23012&mgm.url15=root://jbod-mgmt-07.sdfarm.kr:1096//&mgm.fsid15=1105&cap.valid=1749018502
250604 05:28:23 time=1749014903.660522 func=open                     level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fb01c2c4640 source=XrdFstOfsFile:226              tident=211.35:5408@jbod-mgmt-04 sec=(null) uid=2 gid=2 name=nobody geo="" xt="" ob="" msg="enabling RAIN store recovery" fxid=0088e14b
250604 05:28:23 time=1749014903.661191 func=open                     level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fb01c2c4640 source=XrdFstOfsFile:303              tident=211.35:5408@jbod-mgmt-04 sec=(null) uid=2 gid=2 name=nobody geo="" xt="" ob="" msg="kTpcSrcSetup return SFS_OK" fxid=0088e14b
250604 05:28:24 time=1749014904.489072 func=ProcessCapOpaque         level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=XrdFstOfsFile:2543             tident=154.7:4124@jbod-mgmt-07 sec=(null) uid=65534 gid=65534 name=(null) geo="" xt="" ob="" capability=&mgm.access=update&mgm.ruid=2&mgm.rgid=2&mgm.uid=99&mgm.gid=99&mgm.path=/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998&mgm.manager=jbod-mgmt-04.sdfarm.kr:1094&mgm.fid=0088e14b&mgm.cid=17810&mgm.sec=sss|daemon|jbod-mgmt-04.sdfarm.kr||daemon|||xrootd&mgm.lid=1080299346&mgm.bookingsize=0&mgm.rain.store=1&mgm.rain.size=1967736&mgm.fsid=601&mgm.url0=root://jbod-mgmt-04.sdfarm.kr:1095//&mgm.fsid0=517&mgm.url1=root://jbod-mgmt-01.sdfarm.kr:1095//&mgm.fsid1=13&mgm.url2=root://jbod-mgmt-04.sdfarm.kr:1096//&mgm.fsid2=601&mgm.url3=root://jbod-mgmt-03.sdfarm.kr:1095//&mgm.fsid3=349&mgm.url4=root://jbod-mgmt-06.sdfarm.kr:1095//&mgm.fsid4=853&mgm.url5=root://jbod-mgmt-08.sdfarm.kr:1095//&mgm.fsid5=1189&mgm.url6=root://jbod-mgmt-07.sdfarm.kr:1095//&mgm.fsid6=1021&mgm.url7=root://jbod-mgmt-01.sdfarm.kr:1096//&mgm.fsid7=97&mgm.url8=root://jbod-mgmt-08.sdfarm.kr:1096//&mgm.fsid8=1273&mgm.url9=root://jbod-mgmt-05.sdfarm.kr:1095//&mgm.fsid9=685&mgm.url10=root://jbod-mgmt-11.sdfarm.kr:1096//&mgm.fsid10=22012&mgm.url11=root://jbod-mgmt-02.sdfarm.kr:1096//&mgm.fsid11=265&mgm.url12=root://jbod-mgmt-03.sdfarm.kr:1096//&mgm.fsid12=433&mgm.url13=root://jbod-mgmt-06.sdfarm.kr:1096//&mgm.fsid13=937&mgm.url14=root://jbod-mgmt-12.sdfarm.kr:1095//&mgm.fsid14=23012&mgm.url15=root://jbod-mgmt-07.sdfarm.kr:1096//&mgm.fsid15=1105&cap.valid=1749018502
250604 05:28:24 time=1749014904.645936 func=open                     level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=XrdFstOfsFile:226              tident=154.7:4124@jbod-mgmt-07 sec=(null) uid=2 gid=2 name=nobody geo="" xt="" ob="" msg="enabling RAIN store recovery" fxid=0088e14b
250604 05:28:24 time=1749014904.646815 func=open                     level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=XrdFstOfsFile:541              tident=154.7:4124@jbod-mgmt-07 sec=(null) uid=2 gid=2 name=nobody geo="" xt="" ob="" path=/jbod/box_08_disk_012/00000381/0088e14b open-mode=2 create-mode=180 layout-name=qrain oss-opaque=&mgm.lid=1080299346&mgm.bookingsize=0
250604 05:28:50 time=1749014930.280006 func=Open                     level=WARN  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=RainMetaLayout:323             tident=154.7:4124@jbod-mgmt-07 sec=unix  uid=0 gid=0 name=daemon geo="" xt="" ob="" msg="failed reading header" url="root://jbod-mgmt-07.sdfarm.kr:1096///#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998?&cap.msg=kKJytp6YGpmHwSlkLE/sFJRIlvZjbP+C6m/CGEPTbpPbMcaYybstiRocFLum4qKh3ni4QPxozareHmwhfhxU1/w8Izer1oTuXA9mT7EwnV5rN6CEkLzbMkqPxVTLYtTTAvWU45ARgiGDHeqlAP51m/Pka+sCXmHYX8orJz90ORlHxtU1QZ3LWL5nxR8Wsymad65Ju9fXQ9V4IKfwn0j7H6vbwa1ThTHA6DxD2xOI6BXNXWqgnsJG4+REx9q8UJ3Z0GGEmLW715R0aUpZMiI6bFmbdkl5exsV1BXp2LDfv6cRhD059vUu2tP06oN3QuOsMflW/FNN40mDg1QVy4xZWhS8ducZlGAbLpXjJICAXokv2IYJ9kFXvharmHK42hYExS37+oGUYNgvFadcECHariPgf3vrH3Ue8XDhi8NcMY+Thl8Cye/jDJsg/pb0dUi6mOWrSM5mKUPjdjoTxTpwF6glP31jKXkxdnyqW0ClXa7eaUHQoRcC8aXS4vMvE7rpumbFNrs6lszleGDEi+YsE9uVEsZu5yBMPLU4PRI4PzMbWy9OQG/JHiqF4eeT8YTxzQhfrKTqJsP0c5uTIy1DES0jivjGm8hEM8r/wyuw05BlfACJz4Eu0ktWEP6sZiCXiIDgAp2mEguZKb29+XH+C5cNkahQI1q+zFxdKjZy8c0K/oNRZid2XqywU9fBHnxWSip2k+x4cyAD8h3kk9FqPlZrgGh+p20tXI1+aLKv8oGLtXt998ODlKXOeTJsL+Dz+PBpA0jcGtOCmVq17KF4ERQYyKcu28rG0f4N1lbIgZYOnapLeZhdIj4jLVHTqnwRszzSH4zclzASEdDgI0/EPJLNS3ReMg5Olq3EGYaQDISHixHUwNfs+Sp6Rl5Yt5r5vz310pLsm9VKAo/0XTM5suGJ8+E3Nl5F4YQGHY/vFP72t9gZprVmxeTi1peVJRL2AbFb+iynhN8Z0mHZZuG78aQHBXPhTfVE1tMjxUcnOkQ5kPhhYOxH6HsCGn8hqTIcsE6XvZ9kKJIXX35G6owqD4dBZzvhCNAUQPGcP6cVs9tB8RMl/CV8d6EDaXwTk3t+6zzsl6LlR8ve7buDgUb5kbM4XnGglocBqOm+iI3XaV0UtQdSyqdVP2TKJ1GTL8ppym1OCFiweMu82+3gRGjHK2F9V3wd0GhhJXaMnPLrEA+VCwbsQcWeKiBnBLUz9b6USNB9BeOb8MoVCX+QsXr1/p3mauRk/8iLGBWuPNKasKyCHiZnx3W2MCi9aQYDLRkEqqDs0hylbo6A2nneuRJAuNW52ETW6wpEeMJGnIHftjA+Xg1GrpYxh8YwposDHsZ+BR9yCrwvbKXCm6kunMkoUpF8jacYeyVNAysI1qkxEX7OH3V4mzc5srt0JOrhGFiLqduj86jRGbRXahQQ3taA5QH5yP2SkmhioehQuqov7dNoBSCA0wpnU1lUwRwoWmeLKzSsUT67uQ5N4sIi1Iix/rHto9WBbxYENHkFhyk3p65/+0DF7vvsJ83B/E9dXmAzfvjTIqgXdS1SPwUzbjrEI9z/X7ULLfMXRHDtFxE0ybHYDiMm9UaCTbCFP9KdbCiuEjC9GRQ2hxUtzTSlPJoeGpoTdaI0lntrE1q1DNEOPdVDY0cHYYQ+Ov89BSn+fSPzIxRqdvOFMxdNkQ2QZqllbFX3/mViiYCUQpwnTeueCpIYZ9CYe2EBRhkEqFSdr7hoUupXQxik6f6OlzNkFVkp7PzjRdgGJ/5Dk1p8gspC7466vG9n5fEEcMFxVVz05+QGjt1rY9RFmks=&cap.sym=T3YbdYkv+rl8YFJhtEugBjOFPt0=&eos.clientinfo=zbase64:MDAwMDAwNmF4nBXIbQ5AMAyA4au4gGU+IrGkhzFtGWZLVcTt2b/nfVOm04lCa7quCsn5B8LJRde8EoIt1DcT+JuZhLCMLCEBEk/3oaX/HYRmhc0nrOMStba9uZAniWYX19hx+ADHLiUW&eos.encodepath=curl&eos.pio.action=reconstruct&eos.pio.recfs=0&mgm.id=0088e14b&mgm.logid=bb2c4e60-4104-11f0-9621-b8599f9c4330&mgm.mtime=1653911671&mgm.replicahead=2&mgm.replicaindex=15&mgm.path=/#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998&fst.readahead=true&fst.blocksize=1048576"
250604 05:28:50 time=1749014930.280072 func=ValidateHeader           level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=RainMetaLayout:660             tident=154.7:4124@jbod-mgmt-07 sec=unix  uid=0 gid=0 name=daemon geo="" xt="" ob="" msg="recovered header for stripe 2"
250604 05:28:51 time=1749014931.772478 func=open                     level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=XrdFstOfsFile:728              tident=154.7:4124@jbod-mgmt-07 sec=(null) uid=2 gid=2 name=nobody geo="" xt="" ob="" msg="layout size" fxid=0088e14b disk_size=1967736 db_size= 1967736
250604 05:28:51 time=1749014931.772561 func=open                     level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=XrdFstOfsFile:745              tident=154.7:4124@jbod-mgmt-07 sec=(null) uid=2 gid=2 name=nobody geo="" xt="" ob="" msg="checksum reset init" fxid=0088e14b file-xs=ace95960
250604 05:28:51 time=1749014931.772829 func=open                     level=ERROR logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=XrdFstOfsFile:812              tident=154.7:4124@jbod-mgmt-07 sec=(null) uid=2 gid=2 name=nobody geo="" xt="" ob="" msg="slow open operation" open-duration=27548.072ms fxid=0088e14b path="/#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998" path::print=421.235ms creation::barrier=0.809ms layout::exists=0.009ms clone::fst=0.028ms layout::open=0.028ms layout::opened=27125.456ms get::localfmd=0.007ms resync::localfmd=0.158ms layout::stat=0.001ms layout::stat=0.001ms fileio::object=0.259ms open::accounting=0.067ms end=0.014ms open=27548.072ms
250604 05:28:51 time=1749014931.772876 func=open                     level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=XrdFstOfsFile:817              tident=154.7:4124@jbod-mgmt-07 sec=(null) uid=2 gid=2 name=nobody geo="" xt="" ob="" open-duration=27548.072ms path="/#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998" fxid=0088e14b path::print=421.235ms creation::barrier=0.809ms layout::exists=0.009ms clone::fst=0.028ms layout::open=0.028ms layout::opened=27125.456ms get::localfmd=0.007ms resync::localfmd=0.158ms layout::stat=0.001ms layout::stat=0.001ms fileio::object=0.259ms open::accounting=0.067ms end=0.014ms open=27548.072ms
250604 05:28:54 time=1749014934.770989 func=ReadForceRecovery        level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=RainMetaLayout:860             tident=154.7:4124@jbod-mgmt-07 sec=unix  uid=0 gid=0 name=daemon geo="" xt="" ob="" msg="recover group index" grp_indx=0
250604 05:29:46 time=1749014986.033930 func=TpcCleanup               level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=XrdFstOfsFile:3755             tident=154.7:4124@jbod-mgmt-07 sec=      uid=2 gid=2 name=nobody geo="" xt="" ob="" msg="remove tpc key" key=239de0d200070023683fd976
250604 05:29:46 time=1749014986.224862 func=_close_wr                level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=XrdFstOfsFile:1654             tident=154.7:4124@jbod-mgmt-07 sec=      uid=2 gid=2 name=nobody geo="" xt="" ob="" viaDelete=0 writeDelete=0 mIsCreation=0
250604 05:29:46 time=1749014986.228745 func=Read                     level=WARN  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=RainMetaLayout:712             tident=154.7:4124@jbod-mgmt-07 sec=unix  uid=0 gid=0 name=daemon geo="" xt="" ob="" msg="read too big resizing the read length" end_offset=2097152 file_size=1967736
250604 05:29:46 time=1749014986.229447 func=VerifyChecksum           level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=XrdFstOfsFile:3383             tident=154.7:4124@jbod-mgmt-07 sec=      uid=2 gid=2 name=nobody geo="" xt="" ob="" info="rescanned checksum" size=1.97 MB time=4.40 ms rate=447.52 MB/s 42624dae
250604 05:29:46 time=1749014986.229477 func=VerifyChecksum           level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=XrdFstOfsFile:3410             tident=154.7:4124@jbod-mgmt-07 sec=      uid=2 gid=2 name=nobody geo="" xt="" ob="" (write) checksum type="adler" checksum hex="42624dae" requested-checksum hex="-none-"
250604 05:29:48 time=1749014988.463870 func=Truncate                 level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=RainMetaLayout:1684            tident=154.7:4124@jbod-mgmt-07 sec=unix  uid=0 gid=0 name=daemon geo="" xt="" ob="" msg="done truncate" async_req=102.243ms end=2106.429ms truncate=2208.672ms
250604 05:29:48 time=1749014988.477830 func=StartParityThread        level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fb27e961640 source=RainMetaLayout:2056            tident=154.7:4124@jbod-mgmt-07 sec=unix  uid=0 gid=0 name=daemon geo="" xt="" ob="" msg="parity thread exiting"
250604 05:30:13 time=1749015013.879881 func=PrepareStripeChecksum    level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=RainMetaLayout:1711            tident=154.7:4124@jbod-mgmt-07 sec=unix  uid=0 gid=0 name=daemon geo="" xt="" ob="" msg="rescanned unit checksum" fxid=0088e14b size=1.05 MB time=60.26 ms rate=17.40 MB/s 00f00001
250604 05:30:46 time=1749015046.789035 func=_close                   level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fb3cc9b0640 source=XrdFstOfsFile:1530             tident=211.35:5408@jbod-mgmt-04 sec=      uid=2 gid=2 name=nobody geo="" xt="" ob="" msg="close already done" fxid=0088e14b close_rc=0
250604 05:31:01 time=1749015061.968892 func=_close                   level=INFO  logid=static.............................. unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=XrdFstOfsFile:1575             tident= sec=(null) uid=0 gid=0 name=- geo="" xt="" ob="" msg="log=bb2c4e60-4104-11f0-9621-b8599f9c4330&path=/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998&fstpath=/jbod/box_08_disk_012/00000381/0088e14b&ruid=2&rgid=2&td=154.7:4124@jbod-mgmt-07&host=jbod-mgmt-04.sdfarm.kr&lid=1080299346&fid=8970571&fsid=601&ots=1749014904&otms=224&cts=1749015061&ctms=968&nrc=2&nwc=0&rb=1967736&rb_min=919160&rb_max=1048576&rb_sigma=64708.00&rv_op=0&rvb_min=0&rvb_max=0&rvb_sum=0&rvb_sigma=0.00&rs_op=0&rsb_min=0&rsb_max=0&rsb_sum=0&rsb_sigma=0.00&rc_min=0&rc_max=0&rc_sum=0&rc_sigma=0.00&wb=0&wb_min=0&wb_max=0&wb_sigma=0.00&sfwdb=4096&sbwdb=0&sxlfwdb=0&sxlbwdb=0&nfwds=1&nbwds=0&nxlfwds=0&nxlbwds=0&usage=95.36&iot=157744.016&idt=7315.499&lrt=46945.576&lrvt=0.000&lwt=0.000&ot=27548.072&ct=75934.869&rt=19.57&rvt=0.00&wt=0.00&osize=1967736&csize=1967736&delete_on_close=0&prio_c=2&prio_l=4&prio_d=1&forced_bw=0&ms_sleep=0&ior_err=0&iow_err=0&sec.prot=sss&sec.name=daemon&sec.host=jbod-mgmt-04.sdfarm.kr&sec.vorg=&sec.grps=daemon&sec.role=&sec.info=&sec.app=xrootd&tpc.dst=jbod-mgmt-07.sdfarm.kr&tpc.src_lfn=/#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998"
250604 05:31:01 time=1749015061.968935 func=_close                   level=INFO  logid=bb2c4e60-4104-11f0-9621-b8599f9c4330 unit=fst@jbod-mgmt-04.sdfarm.kr:1096 tid=00007fac87484640 source=XrdFstOfsFile:1598             tident=154.7:4124@jbod-mgmt-07 sec=      uid=2 gid=2 name=nobody geo="" xt="" ob="" msg="done close" rc=0 errc=328404

jbod-mgmt-07.sdfarm.kr:1096

[root@jbod-mgmt-07 fst]# cat xrootd.fst.log| grep "bb2b191e-4104-11f0-a985-b8599f9c4330"
250604 05:28:22 time=1749014902.605881 func=open                     level=INFO  logid=bb2b191e-4104-11f0-a985-b8599f9c4330 unit=fst@jbod-mgmt-07.sdfarm.kr:1096 tid=00007fba2af0b640 source=XrdFstOfsFile:183              tident=3.35:4334@jbod-mgmt-04 sec=sss   uid=0 gid=0 name=daemon geo="" xt="" ob="" path=/replicate:0 info=cap.msg=<...>&cap.sym=<...>&mgm.logid=bb2b191e-4104-11f0-a985-b8599f9c4330&oss.asize=1967736&tpc.dlg=211@jbod-mgmt-04.sdfarm.kr:1094&tpc.dlgon=0&tpc.key=239de0d200070023683fd976&tpc.lfn=/#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998&tpc.spr=root&tpc.src=211@jbod-mgmt-04.sdfarm.kr:1096&tpc.stage=copy&tpc.tpr=root open_mode=200
250604 05:28:22 time=1749014902.606373 func=ProcessTpcOpaque         level=INFO  logid=bb2b191e-4104-11f0-a985-b8599f9c4330 unit=fst@jbod-mgmt-07.sdfarm.kr:1096 tid=00007fba2af0b640 source=XrdFstOfsFile:2852             tident=3.35:4334@jbod-mgmt-04 sec=sss   uid=0 gid=0 name=daemon geo="" xt="" ob="" msg="tpc dst session" key=239de0d200070023683fd976, org=3.35@jbod-mgmt-04.sdfarm.kr, src=211@jbod-mgmt-04.sdfarm.kr:1096 path=/replicate:0 lfn=/#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998 expires=1749018442
250604 05:28:22 time=1749014902.606712 func=ProcessCapOpaque         level=INFO  logid=bb2b191e-4104-11f0-a985-b8599f9c4330 unit=fst@jbod-mgmt-07.sdfarm.kr:1096 tid=00007fba2af0b640 source=XrdFstOfsFile:2543             tident=3.35:4334@jbod-mgmt-04 sec=(null) uid=65534 gid=65534 name=(null) geo="" xt="" ob="" capability=&mgm.access=write&mgm.ruid=1&mgm.rgid=1&mgm.uid=1&mgm.gid=1&mgm.fid=0&mgm.lid=6557442&mgm.cid=17810&mgm.manager=jbod-mgmt-04.sdfarm.kr:1094&mgm.fsid=1105&mgm.sec=sss|eos|eos|-|-|-|-|eos/fsck&eos.app=fsck&cap.valid=1749018502
250604 05:28:22 time=1749014902.606845 func=open                     level=INFO  logid=bb2b191e-4104-11f0-a985-b8599f9c4330 unit=fst@jbod-mgmt-07.sdfarm.kr:1096 tid=00007fba2af0b640 source=XrdFstOfsFile:265              tident=3.35:4334@jbod-mgmt-04 sec=(null) uid=1 gid=1 name=nobody geo="" xt="" ob="" msg="file fxid=0 acting as a sink i.e. /dev/null"
250604 05:28:22 time=1749014902.623559 func=sync                     level=INFO  logid=bb2b191e-4104-11f0-a985-b8599f9c4330 unit=fst@jbod-mgmt-07.sdfarm.kr:1096 tid=00007fba2af0b640 source=XrdFstOfsFile:1323             tident=3.35:4334@jbod-mgmt-04 sec=      uid=1 gid=1 name=nobody geo="" xt="" ob="" msg="tpc enabled -> 1st sync"
250604 05:28:23 time=1749014903.606661 func=DoTpcTransfer            level=INFO  logid=bb2b191e-4104-11f0-a985-b8599f9c4330 unit=fst@jbod-mgmt-07.sdfarm.kr:1096 tid=00007fc0d5eff640 source=XrdFstOfsFile:3569             tident=3.35:4334@jbod-mgmt-04 sec=      uid=1 gid=1 name=nobody geo="" xt="" ob="" msg="tpc now running - 1st sync"
250604 05:28:23 time=1749014903.607231 func=DoTpcTransfer            level=INFO  logid=bb2b191e-4104-11f0-a985-b8599f9c4330 unit=fst@jbod-mgmt-07.sdfarm.kr:1096 tid=00007fc0d5eff640 source=XrdFstOfsFile:3601             tident=3.35:4334@jbod-mgmt-04 sec=      uid=1 gid=1 name=nobody geo="" xt="" ob="" sync-url=root://211@jbod-mgmt-04.sdfarm.kr:1096//#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998?fst.readahead=true sync-cgi=tpc.key=239de0d200070023683fd976&tpc.org=3.35@jbod-mgmt-04.sdfarm.kr&tpc.stage=copy
250604 05:28:23 time=1749014903.607863 func=fileOpen                 level=INFO  logid=bb2b191e-4104-11f0-a985-b8599f9c4330 unit=fst@jbod-mgmt-07.sdfarm.kr:1096 tid=00007fc0d5eff640 source=XrdIo:311                      tident=<service> sec=      uid=0 gid=0 name= geo="" xt="" ob="" xrd_connection_id=154@jbod-mgmt-04.sdfarm.kr:1096
250604 05:28:23 time=1749014903.672468 func=sync                     level=INFO  logid=bb2b191e-4104-11f0-a985-b8599f9c4330 unit=fst@jbod-mgmt-07.sdfarm.kr:1096 tid=00007fba2af0b640 source=XrdFstOfsFile:1345             tident=3.35:4334@jbod-mgmt-04 sec=      uid=1 gid=1 name=nobody geo="" xt="" ob="" msg="tpc running -> 2nd sync"
250604 05:28:52 time=1749014932.538531 func=DoTpcTransfer            level=INFO  logid=bb2b191e-4104-11f0-a985-b8599f9c4330 unit=fst@jbod-mgmt-07.sdfarm.kr:1096 tid=00007fc0d5eff640 source=XrdFstOfsFile:3634             tident=3.35:4334@jbod-mgmt-04 sec=      uid=1 gid=1 name=nobody geo="" xt="" ob="" msg="tpc pull" 
250604 05:30:46 time=1749015046.015646 func=DoTpcTransfer            level=INFO  logid=bb2b191e-4104-11f0-a985-b8599f9c4330 unit=fst@jbod-mgmt-07.sdfarm.kr:1096 tid=00007fc0d5eff640 source=XrdFstOfsFile:3726             tident=3.35:4334@jbod-mgmt-04 sec=      uid=1 gid=1 name=nobody geo="" xt="" ob="" msg="done tpc transfer, close remote file" is_ok=false src_url=root://211@jbod-mgmt-04.sdfarm.kr:1096//#curl#/eos/gsdc/grid/03/25446/ad59e1c1-db70-11ec-b821-3cecef03e998?fst.readahead=true
250604 05:30:46 time=1749015046.869458 func=TpcCleanup               level=INFO  logid=bb2b191e-4104-11f0-a985-b8599f9c4330 unit=fst@jbod-mgmt-07.sdfarm.kr:1096 tid=00007fc03e12b640 source=XrdFstOfsFile:3755             tident=3.35:4334@jbod-mgmt-04 sec=      uid=1 gid=1 name=nobody geo="" xt="" ob="" msg="remove tpc key" key=239de0d200070023683fd976

CERN Accelerating science

Can not fsck repair for RAIN files