Dear Experts,
recently, some of the FST nodes resist to establish connection with the other nodes. On the MGM, the regular commands (eos node ls
, eos fs ls
, eos health
etc) report everything as perfectly fine, with all the FSTs and filesystems online. However, there are some errors in the logs with Remote I/O error
and [FATAL] Connection error
messages, leading to quite a few nodes. And indeed, by executing the command below from the MGM, it never succeeds:
[root@eos-mgm1 ~]# env XRD_LOGLEVEL=Dump xrdfs eos-fst10.alice-af.wigner.hu:1095 stat /
[2023-02-07 15:04:00.921113 +0000][Debug ][Utility ] Unable to process user config file: [ERROR] OS Error: No such file or directory
[2023-02-07 15:04:00.922215 +0000][Debug ][PlugInMgr ] Initializing plug-in manager...
[2023-02-07 15:04:00.922228 +0000][Debug ][PlugInMgr ] No default plug-in, loading plug-in configs...
[2023-02-07 15:04:00.922238 +0000][Debug ][PlugInMgr ] Processing plug-in definitions in /etc/xrootd/client.plugins.d...
[2023-02-07 15:04:00.922313 +0000][Debug ][PlugInMgr ] Processing plug-in definitions in /root/.xrootd/client.plugins.d...
[2023-02-07 15:04:00.922330 +0000][Debug ][PlugInMgr ] Unable to process directory /root/.xrootd/client.plugins.d: [ERROR] OS Error: No such file or directory
[2023-02-07 15:04:00.922403 +0000][Dump ][Utility ] URL: eos-fst10.alice-af.wigner.hu:1095
[2023-02-07 15:04:00.922403 +0000][Dump ][Utility ] Protocol: root
[2023-02-07 15:04:00.922403 +0000][Dump ][Utility ] User Name:
[2023-02-07 15:04:00.922403 +0000][Dump ][Utility ] Password:
[2023-02-07 15:04:00.922403 +0000][Dump ][Utility ] Host Name: eos-fst10.alice-af.wigner.hu
[2023-02-07 15:04:00.922403 +0000][Dump ][Utility ] Port: 1095
[2023-02-07 15:04:00.922403 +0000][Dump ][Utility ] Path:
[2023-02-07 15:04:00.922448 +0000][Dump ][Utility ] URL: root://eos-fst10.alice-af.wigner.hu:1095/
[2023-02-07 15:04:00.922448 +0000][Dump ][Utility ] Protocol: root
[2023-02-07 15:04:00.922448 +0000][Dump ][Utility ] User Name:
[2023-02-07 15:04:00.922448 +0000][Dump ][Utility ] Password:
[2023-02-07 15:04:00.922448 +0000][Dump ][Utility ] Host Name: eos-fst10.alice-af.wigner.hu
[2023-02-07 15:04:00.922448 +0000][Dump ][Utility ] Port: 1095
[2023-02-07 15:04:00.922448 +0000][Dump ][Utility ] Path:
[2023-02-07 15:04:00.922480 +0000][Dump ][Utility ] URL: root://eos-fst10.alice-af.wigner.hu:1095/
[2023-02-07 15:04:00.922480 +0000][Dump ][Utility ] Protocol: root
[2023-02-07 15:04:00.922480 +0000][Dump ][Utility ] User Name:
[2023-02-07 15:04:00.922480 +0000][Dump ][Utility ] Password:
[2023-02-07 15:04:00.922480 +0000][Dump ][Utility ] Host Name: eos-fst10.alice-af.wigner.hu
[2023-02-07 15:04:00.922480 +0000][Dump ][Utility ] Port: 1095
[2023-02-07 15:04:00.922480 +0000][Dump ][Utility ] Path:
[2023-02-07 15:04:00.922512 +0000][Dump ][Utility ] URL: root://eos-fst10.alice-af.wigner.hu:1095/
[2023-02-07 15:04:00.922512 +0000][Dump ][Utility ] Protocol: root
[2023-02-07 15:04:00.922512 +0000][Dump ][Utility ] User Name:
[2023-02-07 15:04:00.922512 +0000][Dump ][Utility ] Password:
[2023-02-07 15:04:00.922512 +0000][Dump ][Utility ] Host Name: eos-fst10.alice-af.wigner.hu
[2023-02-07 15:04:00.922512 +0000][Dump ][Utility ] Port: 1095
[2023-02-07 15:04:00.922512 +0000][Dump ][Utility ] Path:
[2023-02-07 15:04:00.922559 +0000][Debug ][App ] Executing: stat /
[2023-02-07 15:04:00.922570 +0000][Dump ][App ] Param #00: 'stat'
[2023-02-07 15:04:00.922578 +0000][Dump ][App ] Param #01: '/'
[2023-02-07 15:04:00.922621 +0000][Dump ][FileSystem ] [0x1e8a070@eos-fst10.alice-af.wigner.hu:1095] Sending kXR_stat (path: /, flags: none)
[2023-02-07 15:04:00.922653 +0000][Debug ][Poller ] Available pollers: built-in
[2023-02-07 15:04:00.922661 +0000][Debug ][Poller ] Attempting to create a poller according to preference: built-in
[2023-02-07 15:04:00.922670 +0000][Debug ][Poller ] Creating poller: built-in
[2023-02-07 15:04:00.922683 +0000][Debug ][Poller ] Creating and starting the built-in poller...
[2023-02-07 15:04:00.922937 +0000][Debug ][Poller ] Using 1 poller threads
[2023-02-07 15:04:00.922951 +0000][Debug ][TaskMgr ] Starting the task manager...
[2023-02-07 15:04:00.922994 +0000][Debug ][TaskMgr ] Task manager started
[2023-02-07 15:04:00.923004 +0000][Debug ][JobMgr ] Starting the job manager...
[2023-02-07 15:04:00.923091 +0000][Debug ][JobMgr ] Job manager started, 3 workers
[2023-02-07 15:04:00.923104 +0000][Debug ][TaskMgr ] Registering task: "FileTimer task" to be run at: [2023-02-07 15:04:00 +0000]
[2023-02-07 15:04:00.923115 +0000][Dump ][XRootD ] [eos-fst10.alice-af.wigner.hu:1095] Sending message kXR_stat (path: /, flags: none)
[2023-02-07 15:04:00.923146 +0000][Debug ][ExDbgMsg ] [eos-fst10.alice-af.wigner.hu:1095] MsgHandler created: 0x1e8f450 (message: kXR_stat (path: /, flags: none) ).
[2023-02-07 15:04:00.923185 +0000][Dump ][Utility ] URL: eos-fst10.alice-af.wigner.hu:1095
[2023-02-07 15:04:00.923185 +0000][Dump ][Utility ] Protocol: root
[2023-02-07 15:04:00.923185 +0000][Dump ][Utility ] User Name:
[2023-02-07 15:04:00.923185 +0000][Dump ][Utility ] Password:
[2023-02-07 15:04:00.923185 +0000][Dump ][Utility ] Host Name: eos-fst10.alice-af.wigner.hu
[2023-02-07 15:04:00.923185 +0000][Dump ][Utility ] Port: 1095
[2023-02-07 15:04:00.923185 +0000][Dump ][Utility ] Path:
[2023-02-07 15:04:00.923222 +0000][Dump ][Utility ] URL: eos-fst10.alice-af.wigner.hu:1095
[2023-02-07 15:04:00.923222 +0000][Dump ][Utility ] Protocol: root
[2023-02-07 15:04:00.923222 +0000][Dump ][Utility ] User Name:
[2023-02-07 15:04:00.923222 +0000][Dump ][Utility ] Password:
[2023-02-07 15:04:00.923222 +0000][Dump ][Utility ] Host Name: eos-fst10.alice-af.wigner.hu
[2023-02-07 15:04:00.923222 +0000][Dump ][Utility ] Port: 1095
[2023-02-07 15:04:00.923222 +0000][Dump ][Utility ] Path:
[2023-02-07 15:04:00.923246 +0000][Debug ][PostMaster ] Creating new channel to: eos-fst10.alice-af.wigner.hu:1095 1 stream(s)
[2023-02-07 15:04:00.923277 +0000][Debug ][PostMaster ] [eos-fst10.alice-af.wigner.hu:1095 #0] Stream parameters: Network Stack: IPAuto, Connection Window: 120, ConnectionRetry: 5, Stream Error Window: 1800
[2023-02-07 15:04:00.923404 +0000][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: eos-fst10.alice-af.wigner.hu:1095" to be run at: [2023-02-07 15:04:15 +0000]
[2023-02-07 15:04:00.923421 +0000][Dump ][PostMaster ] [eos-fst10.alice-af.wigner.hu:1095 #0] Sending message kXR_stat (path: /, flags: none) (0x1e8a1a0) through substream 0 expecting answer at 0
[2023-02-07 15:04:00.923558 +0000][Debug ][PostMaster ] [eos-fst10.alice-af.wigner.hu:1095] Found 1 address(es): [::ffff:172.16.152.58]:1095
[2023-02-07 15:04:00.923600 +0000][Debug ][AsyncSock ] [eos-fst10.alice-af.wigner.hu:1095 #0.0] Attempting connection to [::ffff:172.16.152.58]:1095
[2023-02-07 15:04:00.923754 +0000][Debug ][Poller ] Adding socket 0x1e90200 to the poller
[2023-02-07 15:04:00.924084 +0000][Debug ][AsyncSock ] [eos-fst10.alice-af.wigner.hu:1095 #0.0] Async connection call returned
[2023-02-07 15:04:00.924152 +0000][Error ][AsyncSock ] [eos-fst10.alice-af.wigner.hu:1095 #0.0] Unable to connect: No route to host
[2023-02-07 15:04:00.924169 +0000][Debug ][AsyncSock ] [eos-fst10.alice-af.wigner.hu:1095 #0.0] Closing the socket
[2023-02-07 15:04:00.924188 +0000][Debug ][Poller ] <x><--><x> Removing socket from the poller
[2023-02-07 15:04:00.924240 +0000][Error ][PostMaster ] [eos-fst10.alice-af.wigner.hu:1095 #0] elapsed = 0, pConnectionWindow = 120 seconds.
[2023-02-07 15:04:00.924255 +0000][Info ][PostMaster ] [eos-fst10.alice-af.wigner.hu:1095 #0] Attempting reconnection in 120 seconds.
[2023-02-07 15:04:00.924269 +0000][Debug ][TaskMgr ] Registering task: "StreamConnectorTask for eos-fst10.alice-af.wigner.hu:1095 #0" to be run at: [2023-02-07 15:06:00 +0000]
[2023-02-07 15:04:01.923277 +0000][Dump ][TaskMgr ] Running task: "FileTimer task"
[2023-02-07 15:04:01.923344 +0000][Dump ][TaskMgr ] Will rerun task "FileTimer task" at [2023-02-07 15:04:16 +0000]
...
an example for a successful connection as well:
[root@eos-mgm1 ~]# env XRD_LOGLEVEL=Dump xrdfs eos-fst09.alice-af.wigner.hu:1095 stat /
[2023-02-07 15:06:12.963417 +0000][Debug ][Utility ] Unable to process user config file: [ERROR] OS Error: No such file or directory
[2023-02-07 15:06:12.963614 +0000][Debug ][PlugInMgr ] Initializing plug-in manager...
[2023-02-07 15:06:12.963626 +0000][Debug ][PlugInMgr ] No default plug-in, loading plug-in configs...
[2023-02-07 15:06:12.963636 +0000][Debug ][PlugInMgr ] Processing plug-in definitions in /etc/xrootd/client.plugins.d...
[2023-02-07 15:06:12.963707 +0000][Debug ][PlugInMgr ] Processing plug-in definitions in /root/.xrootd/client.plugins.d...
[2023-02-07 15:06:12.963722 +0000][Debug ][PlugInMgr ] Unable to process directory /root/.xrootd/client.plugins.d: [ERROR] OS Error: No such file or directory
[2023-02-07 15:06:12.963838 +0000][Dump ][Utility ] URL: eos-fst09.alice-af.wigner.hu:1095
[2023-02-07 15:06:12.963838 +0000][Dump ][Utility ] Protocol: root
[2023-02-07 15:06:12.963838 +0000][Dump ][Utility ] User Name:
[2023-02-07 15:06:12.963838 +0000][Dump ][Utility ] Password:
[2023-02-07 15:06:12.963838 +0000][Dump ][Utility ] Host Name: eos-fst09.alice-af.wigner.hu
[2023-02-07 15:06:12.963838 +0000][Dump ][Utility ] Port: 1095
[2023-02-07 15:06:12.963838 +0000][Dump ][Utility ] Path:
[2023-02-07 15:06:12.963883 +0000][Dump ][Utility ] URL: root://eos-fst09.alice-af.wigner.hu:1095/
[2023-02-07 15:06:12.963883 +0000][Dump ][Utility ] Protocol: root
[2023-02-07 15:06:12.963883 +0000][Dump ][Utility ] User Name:
[2023-02-07 15:06:12.963883 +0000][Dump ][Utility ] Password:
[2023-02-07 15:06:12.963883 +0000][Dump ][Utility ] Host Name: eos-fst09.alice-af.wigner.hu
[2023-02-07 15:06:12.963883 +0000][Dump ][Utility ] Port: 1095
[2023-02-07 15:06:12.963883 +0000][Dump ][Utility ] Path:
[2023-02-07 15:06:12.963915 +0000][Dump ][Utility ] URL: root://eos-fst09.alice-af.wigner.hu:1095/
[2023-02-07 15:06:12.963915 +0000][Dump ][Utility ] Protocol: root
[2023-02-07 15:06:12.963915 +0000][Dump ][Utility ] User Name:
[2023-02-07 15:06:12.963915 +0000][Dump ][Utility ] Password:
[2023-02-07 15:06:12.963915 +0000][Dump ][Utility ] Host Name: eos-fst09.alice-af.wigner.hu
[2023-02-07 15:06:12.963915 +0000][Dump ][Utility ] Port: 1095
[2023-02-07 15:06:12.963915 +0000][Dump ][Utility ] Path:
[2023-02-07 15:06:12.963949 +0000][Dump ][Utility ] URL: root://eos-fst09.alice-af.wigner.hu:1095/
[2023-02-07 15:06:12.963949 +0000][Dump ][Utility ] Protocol: root
[2023-02-07 15:06:12.963949 +0000][Dump ][Utility ] User Name:
[2023-02-07 15:06:12.963949 +0000][Dump ][Utility ] Password:
[2023-02-07 15:06:12.963949 +0000][Dump ][Utility ] Host Name: eos-fst09.alice-af.wigner.hu
[2023-02-07 15:06:12.963949 +0000][Dump ][Utility ] Port: 1095
[2023-02-07 15:06:12.963949 +0000][Dump ][Utility ] Path:
[2023-02-07 15:06:12.963996 +0000][Debug ][App ] Executing: stat /
[2023-02-07 15:06:12.964007 +0000][Dump ][App ] Param #00: 'stat'
[2023-02-07 15:06:12.964015 +0000][Dump ][App ] Param #01: '/'
[2023-02-07 15:06:12.964059 +0000][Dump ][FileSystem ] [0x1374070@eos-fst09.alice-af.wigner.hu:1095] Sending kXR_stat (path: /, flags: none)
[2023-02-07 15:06:12.964092 +0000][Debug ][Poller ] Available pollers: built-in
[2023-02-07 15:06:12.964101 +0000][Debug ][Poller ] Attempting to create a poller according to preference: built-in
[2023-02-07 15:06:12.964110 +0000][Debug ][Poller ] Creating poller: built-in
[2023-02-07 15:06:12.964124 +0000][Debug ][Poller ] Creating and starting the built-in poller...
[2023-02-07 15:06:12.964326 +0000][Debug ][Poller ] Using 1 poller threads
[2023-02-07 15:06:12.964340 +0000][Debug ][TaskMgr ] Starting the task manager...
[2023-02-07 15:06:12.964381 +0000][Debug ][TaskMgr ] Task manager started
[2023-02-07 15:06:12.964391 +0000][Debug ][JobMgr ] Starting the job manager...
[2023-02-07 15:06:12.964480 +0000][Debug ][JobMgr ] Job manager started, 3 workers
[2023-02-07 15:06:12.964492 +0000][Debug ][TaskMgr ] Registering task: "FileTimer task" to be run at: [2023-02-07 15:06:12 +0000]
[2023-02-07 15:06:12.964504 +0000][Dump ][XRootD ] [eos-fst09.alice-af.wigner.hu:1095] Sending message kXR_stat (path: /, flags: none)
[2023-02-07 15:06:12.964536 +0000][Debug ][ExDbgMsg ] [eos-fst09.alice-af.wigner.hu:1095] MsgHandler created: 0x1379450 (message: kXR_stat (path: /, flags: none) ).
[2023-02-07 15:06:12.964581 +0000][Dump ][Utility ] URL: eos-fst09.alice-af.wigner.hu:1095
[2023-02-07 15:06:12.964581 +0000][Dump ][Utility ] Protocol: root
[2023-02-07 15:06:12.964581 +0000][Dump ][Utility ] User Name:
[2023-02-07 15:06:12.964581 +0000][Dump ][Utility ] Password:
[2023-02-07 15:06:12.964581 +0000][Dump ][Utility ] Host Name: eos-fst09.alice-af.wigner.hu
[2023-02-07 15:06:12.964581 +0000][Dump ][Utility ] Port: 1095
[2023-02-07 15:06:12.964581 +0000][Dump ][Utility ] Path:
[2023-02-07 15:06:12.964619 +0000][Dump ][Utility ] URL: eos-fst09.alice-af.wigner.hu:1095
[2023-02-07 15:06:12.964619 +0000][Dump ][Utility ] Protocol: root
[2023-02-07 15:06:12.964619 +0000][Dump ][Utility ] User Name:
[2023-02-07 15:06:12.964619 +0000][Dump ][Utility ] Password:
[2023-02-07 15:06:12.964619 +0000][Dump ][Utility ] Host Name: eos-fst09.alice-af.wigner.hu
[2023-02-07 15:06:12.964619 +0000][Dump ][Utility ] Port: 1095
[2023-02-07 15:06:12.964619 +0000][Dump ][Utility ] Path:
[2023-02-07 15:06:12.964643 +0000][Debug ][PostMaster ] Creating new channel to: eos-fst09.alice-af.wigner.hu:1095 1 stream(s)
[2023-02-07 15:06:12.964676 +0000][Debug ][PostMaster ] [eos-fst09.alice-af.wigner.hu:1095 #0] Stream parameters: Network Stack: IPAuto, Connection Window: 120, ConnectionRetry: 5, Stream Error Window: 1800
[2023-02-07 15:06:12.964814 +0000][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: eos-fst09.alice-af.wigner.hu:1095" to be run at: [2023-02-07 15:06:27 +0000]
[2023-02-07 15:06:12.964833 +0000][Dump ][PostMaster ] [eos-fst09.alice-af.wigner.hu:1095 #0] Sending message kXR_stat (path: /, flags: none) (0x13741a0) through substream 0 expecting answer at 0
[2023-02-07 15:06:12.964969 +0000][Debug ][PostMaster ] [eos-fst09.alice-af.wigner.hu:1095] Found 1 address(es): [::ffff:172.16.152.57]:1095
[2023-02-07 15:06:12.965008 +0000][Debug ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Attempting connection to [::ffff:172.16.152.57]:1095
[2023-02-07 15:06:12.965064 +0000][Debug ][Poller ] Adding socket 0x137a200 to the poller
[2023-02-07 15:06:12.965311 +0000][Debug ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Async connection call returned
[2023-02-07 15:06:12.965412 +0000][Debug ][XRootDTransport ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Sending out the initial hand shake + kXR_protocol
[2023-02-07 15:06:12.965475 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Wrote a message: (0xe8000950), 44 bytes
[2023-02-07 15:06:12.965569 +0000][Dump ][XRootDTransport ] [msg: 0xe8000950] Expecting 8 bytes of message body
[2023-02-07 15:06:12.965583 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received message header, size: 8
[2023-02-07 15:06:12.965604 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received a message of 16 bytes
[2023-02-07 15:06:12.965623 +0000][Debug ][XRootDTransport ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Got the server hand shake response (type: server [], protocol version 400)
[2023-02-07 15:06:12.965641 +0000][Dump ][XRootDTransport ] [msg: 0xe8000950] Expecting 8 bytes of message body
[2023-02-07 15:06:12.965651 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received message header, size: 8
[2023-02-07 15:06:12.965664 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received a message of 16 bytes
[2023-02-07 15:06:12.965680 +0000][Debug ][XRootDTransport ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] kXR_protocol successful (type: server [], protocol version 400)
[2023-02-07 15:06:12.966133 +0000][Debug ][XRootDTransport ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Sending out kXR_login request, username: root, cgi: ?xrd.cc=hu&xrd.tz=0&xrd.appname=xrdfs&xrd.info=&xrd.hostname=eos-mgm1.alice-af.wigner.hu&xrd.rn=v4.12.7, dual-stack: false, private IPv4: false, private IPv6: false
[2023-02-07 15:06:12.966173 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Wrote a message: (0xe8000a60), 127 bytes
[2023-02-07 15:06:12.966252 +0000][Dump ][XRootDTransport ] [msg: 0xe8000950] Expecting 50 bytes of message body
[2023-02-07 15:06:12.966266 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received message header, size: 8
[2023-02-07 15:06:12.966279 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received a message of 58 bytes
[2023-02-07 15:06:12.966298 +0000][Debug ][XRootDTransport ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Logged in, session: 1e01000066f40000500000001f010000
[2023-02-07 15:06:12.966309 +0000][Debug ][XRootDTransport ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Authentication is required: &P=unix&P=sss,0.13:/etc/eos.keytab
[2023-02-07 15:06:12.966323 +0000][Debug ][XRootDTransport ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Sending authentication data
[2023-02-07 15:06:12.967028 +0000][Debug ][XRootDTransport ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Trying to authenticate using unix
[2023-02-07 15:06:12.967190 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Wrote a message: (0xe8001ed0), 39 bytes
[2023-02-07 15:06:12.967270 +0000][Dump ][XRootDTransport ] [msg: 0xe8001ed0] Expecting 0 bytes of message body
[2023-02-07 15:06:12.967282 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received message header, size: 8
[2023-02-07 15:06:12.967291 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received a message of 8 bytes
[2023-02-07 15:06:12.967304 +0000][Debug ][XRootDTransport ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Authenticated with unix.
[2023-02-07 15:06:12.967317 +0000][Debug ][PostMaster ] [eos-fst09.alice-af.wigner.hu:1095 #0] Stream 0 connected.
[2023-02-07 15:06:12.967330 +0000][Debug ][Utility ] Monitor library name not set. No monitoring
[2023-02-07 15:06:12.967367 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Wrote a message: kXR_stat (path: /, flags: none) (0x13741a0), 25 bytes
[2023-02-07 15:06:12.967377 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Successfully sent message: kXR_stat (path: /, flags: none) (0x13741a0).
[2023-02-07 15:06:12.967395 +0000][Dump ][XRootD ] [eos-fst09.alice-af.wigner.hu:1095] Message kXR_stat (path: /, flags: none) has been successfully sent.
[2023-02-07 15:06:12.967404 +0000][Debug ][ExDbgMsg ] [eos-fst09.alice-af.wigner.hu:1095] Moving MsgHandler: 0x1379450 (message: kXR_stat (path: /, flags: none) ) from out-queu to in-queue.
[2023-02-07 15:06:12.967426 +0000][Dump ][PostMaster ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] All messages consumed, disable uplink
[2023-02-07 15:06:12.967606 +0000][Dump ][XRootDTransport ] [msg: 0xe80008c0] Expecting 31 bytes of message body
[2023-02-07 15:06:12.967708 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received message header for 0xe80008c0 size: 8
[2023-02-07 15:06:12.967763 +0000][Dump ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received message 0xe80008c0 of 39 bytes
[2023-02-07 15:06:12.967783 +0000][Dump ][PostMaster ] [eos-fst09.alice-af.wigner.hu:1095 #0] Handling received message: 0xe80008c0.
[2023-02-07 15:06:12.968017 +0000][Dump ][XRootD ] [eos-fst09.alice-af.wigner.hu:1095] Got a kXR_ok response to request kXR_stat (path: /, flags: none)
[2023-02-07 15:06:12.968075 +0000][Debug ][ExDbgMsg ] [eos-fst09.alice-af.wigner.hu:1095] Calling MsgHandler: 0x1379450 (message: kXR_stat (path: /, flags: none) ) with status: [SUCCESS] .
[2023-02-07 15:06:12.968095 +0000][Dump ][XRootD ] [eos-fst09.alice-af.wigner.hu:1095] Parsing the response to kXR_stat (path: /, flags: none) as StatInfo: 11240347374 4096 19 1614856833
[2023-02-07 15:06:12.968132 +0000][Debug ][ExDbgMsg ] [eos-fst09.alice-af.wigner.hu:1095] Destroying MsgHandler: 0x1379450.
Path: /
Id: 11240347374
Size: 4096
MTime: 2021-03-04 11:20:33
Flags: 19 (XBitSet|IsDir|IsReadable)
[2023-02-07 15:06:12.968282 +0000][Debug ][JobMgr ] Stopping the job manager...
[2023-02-07 15:06:12.968304 +0000][Dump ][JobMgr ] Stopping worker #0...
[2023-02-07 15:06:12.968431 +0000][Dump ][JobMgr ] Worker #0 stopped
[2023-02-07 15:06:12.968443 +0000][Dump ][JobMgr ] Stopping worker #1...
[2023-02-07 15:06:12.968491 +0000][Dump ][JobMgr ] Worker #1 stopped
[2023-02-07 15:06:12.968502 +0000][Dump ][JobMgr ] Stopping worker #2...
[2023-02-07 15:06:12.968543 +0000][Dump ][JobMgr ] Worker #2 stopped
[2023-02-07 15:06:12.968553 +0000][Debug ][JobMgr ] Job manager stopped
[2023-02-07 15:06:12.968562 +0000][Debug ][TaskMgr ] Stopping the task manager...
[2023-02-07 15:06:12.968641 +0000][Debug ][TaskMgr ] Task manager stopped
[2023-02-07 15:06:12.968652 +0000][Debug ][Poller ] Stopping the poller...
[2023-02-07 15:06:12.968752 +0000][Debug ][TaskMgr ] Requesting unregistration of: "TickGeneratorTask for: eos-fst09.alice-af.wigner.hu:1095"
[2023-02-07 15:06:12.968769 +0000][Debug ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Closing the socket
[2023-02-07 15:06:12.968783 +0000][Debug ][Poller ] <[::ffff:172.16.152.16]:45400><--><[::ffff:172.16.152.57]:1095> Removing socket from the poller
[2023-02-07 15:06:12.968823 +0000][Debug ][PostMaster ] [eos-fst09.alice-af.wigner.hu:1095 #0] Destroying stream
[2023-02-07 15:06:12.968835 +0000][Debug ][AsyncSock ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Closing the socket
Practically there is no difference (that I’m aware of) between fst09
and fst10
, so I’m very much clueless. Do you have maybe any suggestion?
Thanks,
Gabor