HTTP in EOS 5.1.28

Hello,

We are getting an 500 error when attempting a HTTP transfer with EOS 5.1.28

gfal-copy random_650MB https://antares-preprod.stfc.ac.uk:9000//eos/antarespreprodtier1/dteam/random_650MB
Copying file:///home/tier1/georgep/random_650MB [FAILED] after 0s
gfal-copy error: 256 (Unknown error 256) - TRANSFER ERROR: Copy failed (streamed). Last attempt: HTTP 500 : Unexpected server error: 500 (destination)

it looks like the FST node throws this error (there was no reference of it in the MGM logs) - see bellow the pasted lines from the FST log (zcat /var/log/eos/fst/xrdlog.fst-20240104.gz | grep “240104 12:5” | grep -i http)

Do you know what could cause this error? Many thanks

240104 12:51:44 20991 anon.0:43@lcgui06.gridpp.rl.ac.uk http_Protocol: received dlen: 16
240104 12:51:44 20991 anon.0:43@lcgui06.gridpp.rl.ac.uk http_Protocol: received dump: 80 85 84 32 47 101 111 115 47 97 110 116 97 114 101 00
240104 12:51:44 20991 anon.0:43@lcgui06.gridpp.rl.ac.uk http_Protocol: Protocol matched. https: False
240104 12:51:44 20991 anon.0:43@lcgui06.gridpp.rl.ac.uk http_Protocol: Process. lp:0x7f8cc80183e0 reqstate: 0
240104 12:51:44 20991 anon.0:43@lcgui06.gridpp.rl.ac.uk http_Protocol: Setting host: [::ffff:130.246.183.70]
240104 12:51:44 20991 http_Protocol: getDataOneShot BuffAvailable: 1048576 maxread: 1048576
240104 12:51:44 20991 http_Protocol: read 2216 of 1048576 bytes
240104 12:51:44 20991 http_Protocol: rc:1455 got hdr line: PUT /eos/antarespreprodtier1/dteam/random_650MB?encURI=%26cap.sym%3DdmKk%2FF%2BmT4y%2F% 2FVhGRVCjm0JLsgg%3D%26cap.msg%3D3QlHHAZ593GHfasqBy8ybo8a35IFnZXhenwymYR0YKDnXrg38CmvvrAFCl9TzmM0gEf1wQNsr67B65WeH5EAsE3N8Y%2BZJahDGg7LqW6kVT9DB2Xq9 61Vl87c6%2BpfYAI2AkrURUteNFHZaXPvDx1SadHx6tYEUHj563Jc4UEp093zPzEb9ZsSW4e2IOmueT17fTrbPaTF0SWU3bzGhcAtcG8Vh%2FBLdAD1oWepvlaoj8uUhzazEKYTHsiKalyh861q Yl9SbTJrq4edXGkSw3joNQNh%2FD7dEUag7qgHsLeOka69YODuZ9VNL6Goqr4UPVyhMpev%2Bi%2Fwc3Asysm0rThgtMZ9%2FiG0CnP%2B86Mu3anr3CU3aDbqFIJdQN6K4Qxn9Ory9IMx3DK%2 BzF3wgU%2F64O47fJ4OoG3wvwlFucbwfelVwjpWVnv1dZGCyQMOj89OAxFA2IHMaMmpimz%2FCl79sdlBc3I5cyXok1QhwUcbRtUx7rzXU%2FxDcapZrZBtP58cDpY1nx8%2FcR3KUiMfoo5aGm mw6hbtZqrpRz3BaH%2B6quOvlYoBsNEysSdhluYLEi1kwHfb2F8mIfK1U0E%3D%26mgm.logid%3D03805fd6-ab00-11ee-be35-e8ebd34116f4%26mgm.replicaindex%3D0%26mgm.repl icahead%3D0%26mgm.etag%3D%22214748489622487040:00000000%22%26mgm.id%3D2faf09d1%26mgm.event%3Dsync::closew%26mgm.workflow%3Ddefault%26mgm.instance%3 Deosantarespreprodtier1%26mgm.owner_uid%3D36300%26mgm.owner_gid%3D24311%26mgm.requestor%3Ddteam001%26mgm.requestorgroup%3Ddteam%26mgm.attributes%3D c3lzLmFyY2hpdmUuZmlsZV9pZD00Mjk1MDU1NTEzOzs7c3lzLmFyY2hpdmUuc3RvcmFnZV9jbGFzcz1kdGVhbV90YXBldGVzdDs7O3N5cy5lb3MuYnRpbWU9MTcwNDM3MjcwNC4yMTcxNjU2MzY %3D%26eos.clientinfo%3Dzbase64:MDAwMDAwNmZ4nBXIYQqEIBCG4at0gR2sCDZhDmP6ycqGyjgS3b7897xvqchWlOeVln1KxR4XpxyHmv8hsBnUu4KPHiMEYYwqqXBAdP3U0e9OAq%2Fssj pB%2B6C0faPmA8lJzlP%2F268x8wOAYybC HTTP/1.1
240104 12:51:44 20991 http_Protocol: Parsing first line: PUT /eos/antarespreprodtier1/dteam/random_650MB?encURI=%26cap.sym%3DdmKk%2FF%2BmT4y%2F%2F VhGRVCjm0JLsgg%3D%26cap.msg%3D3QlHHAZ593GHfasqBy8ybo8a35IFnZXhenwymYR0YKDnXrg38CmvvrAFCl9TzmM0gEf1wQNsr67B65WeH5EAsE3N8Y%2BZJahDGg7LqW6kVT9DB2Xq961 Vl87c6%2BpfYAI2AkrURUteNFHZaXPvDx1SadHx6tYEUHj563Jc4UEp093zPzEb9ZsSW4e2IOmueT17fTrbPaTF0SWU3bzGhcAtcG8Vh%2FBLdAD1oWepvlaoj8uUhzazEKYTHsiKalyh861qYl 9SbTJrq4edXGkSw3joNQNh%2FD7dEUag7qgHsLeOka69YODuZ9VNL6Goqr4UPVyhMpev%2Bi%2Fwc3Asysm0rThgtMZ9%2FiG0CnP%2B86Mu3anr3CU3aDbqFIJdQN6K4Qxn9Ory9IMx3DK%2Bz F3wgU%2F64O47fJ4OoG3wvwlFucbwfelVwjpWVnv1dZGCyQMOj89OAxFA2IHMaMmpimz%2FCl79sdlBc3I5cyXok1QhwUcbRtUx7rzXU%2FxDcapZrZBtP58cDpY1nx8%2FcR3KUiMfoo5aGmmw 6hbtZqrpRz3BaH%2B6quOvlYoBsNEysSdhluYLEi1kwHfb2F8mIfK1U0E%3D%26mgm.logid%3D03805fd6-ab00-11ee-be35-e8ebd34116f4%26mgm.replicaindex%3D0%26mgm.replic ahead%3D0%26mgm.etag%3D%22214748489622487040:00000000%22%26mgm.id%3D2faf09d1%26mgm.event%3Dsync::closew%26mgm.workflow%3Ddefault%26mgm.instance%3De osantarespreprodtier1%26mgm.owner_uid%3D36300%26mgm.owner_gid%3D24311%26mgm.requestor%3Ddteam001%26mgm.requestorgroup%3Ddteam%26mgm.attributes%3Dc3 lzLmFyY2hpdmUuZmlsZV9pZD00Mjk1MDU1NTEzOzs7c3lzLmFyY2hpdmUuc3RvcmFnZV9jbGFzcz1kdGVhbV90YXBldGVzdDs7O3N5cy5lb3MuYnRpbWU9MTcwNDM3MjcwNC4yMTcxNjU2MzY%3 D%26eos.clientinfo%3Dzbase64:MDAwMDAwNmZ4nBXIYQqEIBCG4at0gR2sCDZhDmP6ycqGyjgS3b7897xvqchWlOeVln1KxR4XpxyHmv8hsBnUu4KPHiMEYYwqqXBAdP3U0e9OAq%2FssjpB %2B6C0faPmA8lJzlP%2F268x8wOAYybC HTTP/1.1
240104 12:51:44 20991 http_Protocol: rc:55 got hdr line: User-Agent: gfal2-util/1.8.0 gfal2/2.21.5 neon/0.0.29
240104 12:51:44 20991 http_Protocol: rc:14 got hdr line: Keep-Alive:
240104 12:51:44 20991 http_Protocol: rc:24 got hdr line: Connection: Keep-Alive
240104 12:51:44 20991 http_Protocol: rc:14 got hdr line: TE: trailers
240104 12:51:44 20991 http_Protocol: rc:39 got hdr line: Host: antares-eos95.scd.rl.ac.uk:8001
240104 12:51:44 20991 http_Protocol: rc:564 got hdr line: Authorization: Bearer MDAzOWxvY2F0aW9uIGNlcm5fdGFwZV9hcmNoaXZlX3JhbF9hbnRhcmVzLXByZXByb2 QtdGllcjEKMDAzNGlkZW50aWZpZXIgNDdjNDE4MTYtYzhiNi00MjQ4LWEyYmQtMTQ4NjQyNTFiZDQzCjAwMTZjaWQgbmFtZTpkdGVhbTAwMQowMDUyY2lkIGFjdGl2aXR5OlJFQURfTUVUQURBV EEsVVBMT0FELERPV05MT0FELERFTEVURSxNQU5BR0UsVVBEQVRFX01FVEFEQVRBLExJU1QKMDAzNGNpZCBhY3Rpdml0eTpMSVNULERPV05MT0FELE1BTkFHRSxVUExPQUQsREVMRVRFCjAwMzlj aWQgcGF0aDovZW9zL2FudGFyZXNwcmVwcm9kdGllcjEvZHRlYW0vcmFuZG9tXzY1ME1CCjAwMjRjaWQgYmVmb3JlOjIwMjQtMDEtMDRUMTU6NTE6NDRaCjAwMmZzaWduYXR1cmUgmz0K5BjaQZ4 EYRf200iJz20MDfnVadKxstUtPo3dHUIK
240104 12:51:44 20991 http_Protocol: rc:27 got hdr line: Content-Length: 681574400
240104 12:51:44 20991 http_Protocol: rc:22 got hdr line: Expect: 100-continue
240104 12:51:44 20991 http_Protocol: rc:2 got hdr line:
240104 12:51:44 20991 http_Protocol: rc:2 detected header end.
240104 12:51:44 20991 unknown.201:43@lcgui06.gridpp.rl.ac.uk http_Protocol: Process. lp:0x7f8cc80183e0 reqstate: 0
240104 12:51:44 20991 unknown.201:43@lcgui06.gridpp.rl.ac.uk http_Protocol: Process is exiting rc:1
240104 12:51:44 time=1704372704.234106 func=open level=INFO logid=unknown unit=fst@antares-eos95.scd.rl.ac.uk:1095 tid=00007f8 db3014700 source=XrdFstOfsFile:177 tident=unknown.201:43@lcgui06.gridpp.rl.ac.uk sec=http uid=0 gid=0 name= geo=“” path=/eos/antaresp reprodtier1/dteam/random_650MB info=encURI=%26cap.sym%3DdmKk%2FF%2BmT4y%2F%2FVhGRVCjm0JLsgg%3D%26cap.msg%3D3QlHHAZ593GHfasqBy8ybo8a35IFnZXhenwymYR0 YKDnXrg38CmvvrAFCl9TzmM0gEf1wQNsr67B65WeH5EAsE3N8Y%2BZJahDGg7LqW6kVT9DB2Xq961Vl87c6%2BpfYAI2AkrURUteNFHZaXPvDx1SadHx6tYEUHj563Jc4UEp093zPzEb9ZsSW4e 2IOmueT17fTrbPaTF0SWU3bzGhcAtcG8Vh%2FBLdAD1oWepvlaoj8uUhzazEKYTHsiKalyh861qYl9SbTJrq4edXGkSw3joNQNh%2FD7dEUag7qgHsLeOka69YODuZ9VNL6Goqr4UPVyhMpev%2 Bi%2Fwc3Asysm0rThgtMZ9%2FiG0CnP%2B86Mu3anr3CU3aDbqFIJdQN6K4Qxn9Ory9IMx3DK%2BzF3wgU%2F64O47fJ4OoG3wvwlFucbwfelVwjpWVnv1dZGCyQMOj89OAxFA2IHMaMmpimz%2 FCl79sdlBc3I5cyXok1QhwUcbRtUx7rzXU%2FxDcapZrZBtP58cDpY1nx8%2FcR3KUiMfoo5aGmmw6hbtZqrpRz3BaH%2B6quOvlYoBsNEysSdhluYLEi1kwHfb2F8mIfK1U0E%3D%26mgm.log id%3D03805fd6-ab00-11ee-be35-e8ebd34116f4%26mgm.replicaindex%3D0%26mgm.replicahead%3D0%26mgm.etag%3D%22214748489622487040:00000000%22%26mgm.id%3D2f af09d1%26mgm.event%3Dsync::closew%26mgm.workflow%3Ddefault%26mgm.instance%3Deosantarespreprodtier1%26mgm.owner_uid%3D36300%26mgm.owner_gid%3D24311% 26mgm.requestor%3Ddteam001%26mgm.requestorgroup%3Ddteam%26mgm.attributes%3Dc3lzLmFyY2hpdmUuZmlsZV9pZD00Mjk1MDU1NTEzOzs7c3lzLmFyY2hpdmUuc3RvcmFnZV9j bGFzcz1kdGVhbV90YXBldGVzdDs7O3N5cy5lb3MuYnRpbWU9MTcwNDM3MjcwNC4yMTcxNjU2MzY%3D%26eos.clientinfo%3Dzbase64:MDAwMDAwNmZ4nBXIYQqEIBCG4at0gR2sCDZhDmP6y cqGyjgS3b7897xvqchWlOeVln1KxR4XpxyHmv8hsBnUu4KPHiMEYYwqqXBAdP3U0e9OAq%2FssjpB%2B6C0faPmA8lJzlP%2F268x8wOAYybC open_mode=200
240104 12:51:44 20991 http_Req: XrdHttpReq::Error
240104 12:51:44 20991 unknown.201:43@lcgui06.gridpp.rl.ac.uk http_Req: PostProcessHTTPReq req: 4 reqstate: 0
240104 12:51:44 20991 unknown.201:43@lcgui06.gridpp.rl.ac.uk http_Req: PostProcessHTTPReq mapping Xrd error [3000] to status code [500]
240104 12:51:44 20991 unknown.201:43@lcgui06.gridpp.rl.ac.uk http_Protocol: Sending resp: 500 header len:105
240104 12:51:44 20991 http_Protocol: Sending 105 bytes
240104 12:51:44 time=1704372704.234345 func=open level=ERROR logid=unknown unit=fst@antares-eos95.scd.rl.ac.uk:1095 tid=00007f8 db3014700 source=XrdFstOfsFile:192 tident=unknown.201:43@lcgui06.gridpp.rl.ac.uk sec=http uid=0 gid=0 name= geo=“” msg=“failed while processing TPC/open opaque”
240104 12:51:44 20991 http_Protocol: Sending 98 bytes
240104 12:51:44 20991 http_Req: XrdHttpReq request ended.
240104 12:51:44 20991 http_Protocol: Cleanup
240104 12:51:44 20991 http_Protocol: Reset
240104 12:51:44 20991 http_Req: XrdHttpReq request ended.
240104 12:51:44 time=1704372704.234513 func=_close level=INFO logid=unknown unit=fst@antares-eos95.scd.rl.ac.uk:1095 tid=00007f8 db3014700 source=XrdFstOfsFile:2267 tident=unknown.201:43@lcgui06.gridpp.rl.ac.uk sec=http uid=0 gid=0 name= geo=“” msg=“done close” r c=0 errc=0
[root@antares-eos95 ~]#
[root@antares-eos95 ~]#
[root@antares-eos95 ~]# zcat /var/log/eos/fst/xrdlog.fst-20240104.gz | grep “240104 12:5” | grep -i http
240104 12:51:44 20991 anon.0:43@lcgui06.gridpp.rl.ac.uk http_Protocol: received dlen: 16
240104 12:51:44 20991 anon.0:43@lcgui06.gridpp.rl.ac.uk http_Protocol: received dump: 80 85 84 32 47 101 111 115 47 97 110 116 97 114 101 00
240104 12:51:44 20991 anon.0:43@lcgui06.gridpp.rl.ac.uk http_Protocol: Protocol matched. https: False
240104 12:51:44 20991 anon.0:43@lcgui06.gridpp.rl.ac.uk http_Protocol: Process. lp:0x7f8cc80183e0 reqstate: 0
240104 12:51:44 20991 anon.0:43@lcgui06.gridpp.rl.ac.uk http_Protocol: Setting host: [::ffff:130.246.183.70]
240104 12:51:44 20991 http_Protocol: getDataOneShot BuffAvailable: 1048576 maxread: 1048576
240104 12:51:44 20991 http_Protocol: read 2216 of 1048576 bytes
240104 12:51:44 20991 http_Protocol: rc:1455 got hdr line: PUT /eos/antarespreprodtier1/dteam/random_650MB?encURI=%26cap.sym%3DdmKk%2FF%2BmT4y%2F%2FVhGRVCjm0JLsgg%3D%26cap.msg%3D3QlHHAZ593GHfasqBy8ybo8a35IFnZXhenwymYR0YKDnXrg38CmvvrAFCl9TzmM0gEf1wQNsr67B65WeH5EAsE3N8Y%2BZJahDGg7LqW6kVT9DB2Xq961Vl87c6%2BpfYAI2AkrURUteNFHZaXPvDx1SadHx6tYEUHj563Jc4UEp093zPzEb9ZsSW4e2IOmueT17fTrbPaTF0SWU3bzGhcAtcG8Vh%2FBLdAD1oWepvlaoj8uUhzazEKYTHsiKalyh861qYl9SbTJrq4edXGkSw3joNQNh%2FD7dEUag7qgHsLeOka69YODuZ9VNL6Goqr4UPVyhMpev%2Bi%2Fwc3Asysm0rThgtMZ9%2FiG0CnP%2B86Mu3anr3CU3aDbqFIJdQN6K4Qxn9Ory9IMx3DK%2BzF3wgU%2F64O47fJ4OoG3wvwlFucbwfelVwjpWVnv1dZGCyQMOj89OAxFA2IHMaMmpimz%2FCl79sdlBc3I5cyXok1QhwUcbRtUx7rzXU%2FxDcapZrZBtP58cDpY1nx8%2FcR3KUiMfoo5aGmmw6hbtZqrpRz3BaH%2B6quOvlYoBsNEysSdhluYLEi1kwHfb2F8mIfK1U0E%3D%26mgm.logid%3D03805fd6-ab00-11ee-be35-e8ebd34116f4%26mgm.replicaindex%3D0%26mgm.replicahead%3D0%26mgm.etag%3D%22214748489622487040:00000000%22%26mgm.id%3D2faf09d1%26mgm.event%3Dsync::closew%26mgm.workflow%3Ddefault%26mgm.instance%3Deosantarespreprodtier1%26mgm.owner_uid%3D36300%26mgm.owner_gid%3D24311%26mgm.requestor%3Ddteam001%26mgm.requestorgroup%3Ddteam%26mgm.attributes%3Dc3lzLmFyY2hpdmUuZmlsZV9pZD00Mjk1MDU1NTEzOzs7c3lzLmFyY2hpdmUuc3RvcmFnZV9jbGFzcz1kdGVhbV90YXBldGVzdDs7O3N5cy5lb3MuYnRpbWU9MTcwNDM3MjcwNC4yMTcxNjU2MzY%3D%26eos.clientinfo%3Dzbase64:MDAwMDAwNmZ4nBXIYQqEIBCG4at0gR2sCDZhDmP6ycqGyjgS3b7897xvqchWlOeVln1KxR4XpxyHmv8hsBnUu4KPHiMEYYwqqXBAdP3U0e9OAq%2FssjpB%2B6C0faPmA8lJzlP%2F268x8wOAYybC HTTP/1.1
240104 12:51:44 20991 http_Protocol: Parsing first line: PUT /eos/antarespreprodtier1/dteam/random_650MB?encURI=%26cap.sym%3DdmKk%2FF%2BmT4y%2F%2FVhGRVCjm0JLsgg%3D%26cap.msg%3D3QlHHAZ593GHfasqBy8ybo8a35IFnZXhenwymYR0YKDnXrg38CmvvrAFCl9TzmM0gEf1wQNsr67B65WeH5EAsE3N8Y%2BZJahDGg7LqW6kVT9DB2Xq961Vl87c6%2BpfYAI2AkrURUteNFHZaXPvDx1SadHx6tYEUHj563Jc4UEp093zPzEb9ZsSW4e2IOmueT17fTrbPaTF0SWU3bzGhcAtcG8Vh%2FBLdAD1oWepvlaoj8uUhzazEKYTHsiKalyh861qYl9SbTJrq4edXGkSw3joNQNh%2FD7dEUag7qgHsLeOka69YODuZ9VNL6Goqr4UPVyhMpev%2Bi%2Fwc3Asysm0rThgtMZ9%2FiG0CnP%2B86Mu3anr3CU3aDbqFIJdQN6K4Qxn9Ory9IMx3DK%2BzF3wgU%2F64O47fJ4OoG3wvwlFucbwfelVwjpWVnv1dZGCyQMOj89OAxFA2IHMaMmpimz%2FCl79sdlBc3I5cyXok1QhwUcbRtUx7rzXU%2FxDcapZrZBtP58cDpY1nx8%2FcR3KUiMfoo5aGmmw6hbtZqrpRz3BaH%2B6quOvlYoBsNEysSdhluYLEi1kwHfb2F8mIfK1U0E%3D%26mgm.logid%3D03805fd6-ab00-11ee-be35-e8ebd34116f4%26mgm.replicaindex%3D0%26mgm.replicahead%3D0%26mgm.etag%3D%22214748489622487040:00000000%22%26mgm.id%3D2faf09d1%26mgm.event%3Dsync::closew%26mgm.workflow%3Ddefault%26mgm.instance%3Deosantarespreprodtier1%26mgm.owner_uid%3D36300%26mgm.owner_gid%3D24311%26mgm.requestor%3Ddteam001%26mgm.requestorgroup%3Ddteam%26mgm.attributes%3Dc3lzLmFyY2hpdmUuZmlsZV9pZD00Mjk1MDU1NTEzOzs7c3lzLmFyY2hpdmUuc3RvcmFnZV9jbGFzcz1kdGVhbV90YXBldGVzdDs7O3N5cy5lb3MuYnRpbWU9MTcwNDM3MjcwNC4yMTcxNjU2MzY%3D%26eos.clientinfo%3Dzbase64:MDAwMDAwNmZ4nBXIYQqEIBCG4at0gR2sCDZhDmP6ycqGyjgS3b7897xvqchWlOeVln1KxR4XpxyHmv8hsBnUu4KPHiMEYYwqqXBAdP3U0e9OAq%2FssjpB%2B6C0faPmA8lJzlP%2F268x8wOAYybC HTTP/1.1
240104 12:51:44 20991 http_Protocol: rc:55 got hdr line: User-Agent: gfal2-util/1.8.0 gfal2/2.21.5 neon/0.0.29
240104 12:51:44 20991 http_Protocol: rc:14 got hdr line: Keep-Alive:
240104 12:51:44 20991 http_Protocol: rc:24 got hdr line: Connection: Keep-Alive
240104 12:51:44 20991 http_Protocol: rc:14 got hdr line: TE: trailers
240104 12:51:44 20991 http_Protocol: rc:39 got hdr line: Host: antares-eos95.scd.rl.ac.uk:8001
240104 12:51:44 20991 http_Protocol: rc:564 got hdr line: Authorization: Bearer MDAzOWxvY2F0aW9uIGNlcm5fdGFwZV9hcmNoaXZlX3JhbF9hbnRhcmVzLXByZXByb2QtdGllcjEKMDAzNGlkZW50aWZpZXIgNDdjNDE4MTYtYzhiNi00MjQ4LWEyYmQtMTQ4NjQyNTFiZDQzCjAwMTZjaWQgbmFtZTpkdGVhbTAwMQowMDUyY2lkIGFjdGl2aXR5OlJFQURfTUVUQURBVEEsVVBMT0FELERPV05MT0FELERFTEVURSxNQU5BR0UsVVBEQVRFX01FVEFEQVRBLExJU1QKMDAzNGNpZCBhY3Rpdml0eTpMSVNULERPV05MT0FELE1BTkFHRSxVUExPQUQsREVMRVRFCjAwMzljaWQgcGF0aDovZW9zL2FudGFyZXNwcmVwcm9kdGllcjEvZHRlYW0vcmFuZG9tXzY1ME1CCjAwMjRjaWQgYmVmb3JlOjIwMjQtMDEtMDRUMTU6NTE6NDRaCjAwMmZzaWduYXR1cmUgmz0K5BjaQZ4EYRf200iJz20MDfnVadKxstUtPo3dHUIK
240104 12:51:44 20991 http_Protocol: rc:27 got hdr line: Content-Length: 681574400
240104 12:51:44 20991 http_Protocol: rc:22 got hdr line: Expect: 100-continue
240104 12:51:44 20991 http_Protocol: rc:2 got hdr line:
240104 12:51:44 20991 http_Protocol: rc:2 detected header end.
240104 12:51:44 20991 unknown.201:43@lcgui06.gridpp.rl.ac.uk http_Protocol: Process. lp:0x7f8cc80183e0 reqstate: 0
240104 12:51:44 20991 unknown.201:43@lcgui06.gridpp.rl.ac.uk http_Protocol: Process is exiting rc:1
240104 12:51:44 time=1704372704.234106 func=open level=INFO logid=unknown unit=fst@antares-eos95.scd.rl.ac.uk:1095 tid=00007f8db3014700 source=XrdFstOfsFile:177 tident=unknown.201:43@lcgui06.gridpp.rl.ac.uk sec=http uid=0 gid=0 name= geo=“” path=/eos/antarespreprodtier1/dteam/random_650MB info=encURI=%26cap.sym%3DdmKk%2FF%2BmT4y%2F%2FVhGRVCjm0JLsgg%3D%26cap.msg%3D3QlHHAZ593GHfasqBy8ybo8a35IFnZXhenwymYR0YKDnXrg38CmvvrAFCl9TzmM0gEf1wQNsr67B65WeH5EAsE3N8Y%2BZJahDGg7LqW6kVT9DB2Xq961Vl87c6%2BpfYAI2AkrURUteNFHZaXPvDx1SadHx6tYEUHj563Jc4UEp093zPzEb9ZsSW4e2IOmueT17fTrbPaTF0SWU3bzGhcAtcG8Vh%2FBLdAD1oWepvlaoj8uUhzazEKYTHsiKalyh861qYl9SbTJrq4edXGkSw3joNQNh%2FD7dEUag7qgHsLeOka69YODuZ9VNL6Goqr4UPVyhMpev%2Bi%2Fwc3Asysm0rThgtMZ9%2FiG0CnP%2B86Mu3anr3CU3aDbqFIJdQN6K4Qxn9Ory9IMx3DK%2BzF3wgU%2F64O47fJ4OoG3wvwlFucbwfelVwjpWVnv1dZGCyQMOj89OAxFA2IHMaMmpimz%2FCl79sdlBc3I5cyXok1QhwUcbRtUx7rzXU%2FxDcapZrZBtP58cDpY1nx8%2FcR3KUiMfoo5aGmmw6hbtZqrpRz3BaH%2B6quOvlYoBsNEysSdhluYLEi1kwHfb2F8mIfK1U0E%3D%26mgm.logid%3D03805fd6-ab00-11ee-be35-e8ebd34116f4%26mgm.replicaindex%3D0%26mgm.replicahead%3D0%26mgm.etag%3D%22214748489622487040:00000000%22%26mgm.id%3D2faf09d1%26mgm.event%3Dsync::closew%26mgm.workflow%3Ddefault%26mgm.instance%3Deosantarespreprodtier1%26mgm.owner_uid%3D36300%26mgm.owner_gid%3D24311%26mgm.requestor%3Ddteam001%26mgm.requestorgroup%3Ddteam%26mgm.attributes%3Dc3lzLmFyY2hpdmUuZmlsZV9pZD00Mjk1MDU1NTEzOzs7c3lzLmFyY2hpdmUuc3RvcmFnZV9jbGFzcz1kdGVhbV90YXBldGVzdDs7O3N5cy5lb3MuYnRpbWU9MTcwNDM3MjcwNC4yMTcxNjU2MzY%3D%26eos.clientinfo%3Dzbase64:MDAwMDAwNmZ4nBXIYQqEIBCG4at0gR2sCDZhDmP6ycqGyjgS3b7897xvqchWlOeVln1KxR4XpxyHmv8hsBnUu4KPHiMEYYwqqXBAdP3U0e9OAq%2FssjpB%2B6C0faPmA8lJzlP%2F268x8wOAYybC open_mode=200
240104 12:51:44 20991 http_Req: XrdHttpReq::Error
240104 12:51:44 20991 unknown.201:43@lcgui06.gridpp.rl.ac.uk http_Req: PostProcessHTTPReq req: 4 reqstate: 0
240104 12:51:44 20991 unknown.201:43@lcgui06.gridpp.rl.ac.uk http_Req: PostProcessHTTPReq mapping Xrd error [3000] to status code [500]
240104 12:51:44 20991 unknown.201:43@lcgui06.gridpp.rl.ac.uk http_Protocol: Sending resp: 500 header len:105
240104 12:51:44 20991 http_Protocol: Sending 105 bytes
240104 12:51:44 time=1704372704.234345 func=open level=ERROR logid=unknown unit=fst@antares-eos95.scd.rl.ac.uk:1095 tid=00007f8db3014700 source=XrdFstOfsFile:192 tident=unknown.201:43@lcgui06.gridpp.rl.ac.uk sec=http uid=0 gid=0 name= geo=“” msg=“failed while processing TPC/open opaque”
240104 12:51:44 20991 http_Protocol: Sending 98 bytes
240104 12:51:44 20991 http_Req: XrdHttpReq request ended.
240104 12:51:44 20991 http_Protocol: Cleanup
240104 12:51:44 20991 http_Protocol: Reset
240104 12:51:44 20991 http_Req: XrdHttpReq request ended.
240104 12:51:44 time=1704372704.234513 func=_close level=INFO logid=unknown unit=fst@antares-eos95.scd.rl.ac.uk:1095 tid=00007f8db3014700 source=XrdFstOfsFile:2267 tident=unknown.201:43@lcgui06.gridpp.rl.ac.uk sec=http uid=0 gid=0 name= geo=“” msg=“done close” rc=0 errc=0

Just to say that we managed to resolve this issue. We installed host certs on all FST nodes and added the following directives in the /etc/xrd.fst

xrd.tls /etc/grid-security/daemon/hostcert.pem /etc/grid-security/daemon/hostkey.pem
xrd.tlsca certdir /etc/grid-security/certificates/

following this thread Https/webdav on eos5 problem - #12 by esindril

George