FST connection timeout

Dear Experts,

recently, some of the FST nodes resist to establish connection with the other nodes. On the MGM, the regular commands (eos node ls, eos fs ls, eos health etc) report everything as perfectly fine, with all the FSTs and filesystems online. However, there are some errors in the logs with Remote I/O error and [FATAL] Connection error messages, leading to quite a few nodes. And indeed, by executing the command below from the MGM, it never succeeds:

[root@eos-mgm1 ~]# env XRD_LOGLEVEL=Dump xrdfs eos-fst10.alice-af.wigner.hu:1095 stat /
[2023-02-07 15:04:00.921113 +0000][Debug  ][Utility           ] Unable to process user config file: [ERROR] OS Error: No such file or directory
[2023-02-07 15:04:00.922215 +0000][Debug  ][PlugInMgr         ] Initializing plug-in manager...
[2023-02-07 15:04:00.922228 +0000][Debug  ][PlugInMgr         ] No default plug-in, loading plug-in configs...
[2023-02-07 15:04:00.922238 +0000][Debug  ][PlugInMgr         ] Processing plug-in definitions in /etc/xrootd/client.plugins.d...
[2023-02-07 15:04:00.922313 +0000][Debug  ][PlugInMgr         ] Processing plug-in definitions in /root/.xrootd/client.plugins.d...
[2023-02-07 15:04:00.922330 +0000][Debug  ][PlugInMgr         ] Unable to process directory /root/.xrootd/client.plugins.d: [ERROR] OS Error: No such file or directory
[2023-02-07 15:04:00.922403 +0000][Dump   ][Utility           ] URL: eos-fst10.alice-af.wigner.hu:1095
[2023-02-07 15:04:00.922403 +0000][Dump   ][Utility           ] Protocol:  root
[2023-02-07 15:04:00.922403 +0000][Dump   ][Utility           ] User Name: 
[2023-02-07 15:04:00.922403 +0000][Dump   ][Utility           ] Password:  
[2023-02-07 15:04:00.922403 +0000][Dump   ][Utility           ] Host Name: eos-fst10.alice-af.wigner.hu
[2023-02-07 15:04:00.922403 +0000][Dump   ][Utility           ] Port:      1095
[2023-02-07 15:04:00.922403 +0000][Dump   ][Utility           ] Path:      
[2023-02-07 15:04:00.922448 +0000][Dump   ][Utility           ] URL: root://eos-fst10.alice-af.wigner.hu:1095/
[2023-02-07 15:04:00.922448 +0000][Dump   ][Utility           ] Protocol:  root
[2023-02-07 15:04:00.922448 +0000][Dump   ][Utility           ] User Name: 
[2023-02-07 15:04:00.922448 +0000][Dump   ][Utility           ] Password:  
[2023-02-07 15:04:00.922448 +0000][Dump   ][Utility           ] Host Name: eos-fst10.alice-af.wigner.hu
[2023-02-07 15:04:00.922448 +0000][Dump   ][Utility           ] Port:      1095
[2023-02-07 15:04:00.922448 +0000][Dump   ][Utility           ] Path:      
[2023-02-07 15:04:00.922480 +0000][Dump   ][Utility           ] URL: root://eos-fst10.alice-af.wigner.hu:1095/
[2023-02-07 15:04:00.922480 +0000][Dump   ][Utility           ] Protocol:  root
[2023-02-07 15:04:00.922480 +0000][Dump   ][Utility           ] User Name: 
[2023-02-07 15:04:00.922480 +0000][Dump   ][Utility           ] Password:  
[2023-02-07 15:04:00.922480 +0000][Dump   ][Utility           ] Host Name: eos-fst10.alice-af.wigner.hu
[2023-02-07 15:04:00.922480 +0000][Dump   ][Utility           ] Port:      1095
[2023-02-07 15:04:00.922480 +0000][Dump   ][Utility           ] Path:      
[2023-02-07 15:04:00.922512 +0000][Dump   ][Utility           ] URL: root://eos-fst10.alice-af.wigner.hu:1095/
[2023-02-07 15:04:00.922512 +0000][Dump   ][Utility           ] Protocol:  root
[2023-02-07 15:04:00.922512 +0000][Dump   ][Utility           ] User Name: 
[2023-02-07 15:04:00.922512 +0000][Dump   ][Utility           ] Password:  
[2023-02-07 15:04:00.922512 +0000][Dump   ][Utility           ] Host Name: eos-fst10.alice-af.wigner.hu
[2023-02-07 15:04:00.922512 +0000][Dump   ][Utility           ] Port:      1095
[2023-02-07 15:04:00.922512 +0000][Dump   ][Utility           ] Path:      
[2023-02-07 15:04:00.922559 +0000][Debug  ][App               ] Executing: stat / 
[2023-02-07 15:04:00.922570 +0000][Dump   ][App               ] Param #00: 'stat'
[2023-02-07 15:04:00.922578 +0000][Dump   ][App               ] Param #01: '/'
[2023-02-07 15:04:00.922621 +0000][Dump   ][FileSystem        ] [0x1e8a070@eos-fst10.alice-af.wigner.hu:1095] Sending kXR_stat (path: /, flags: none)
[2023-02-07 15:04:00.922653 +0000][Debug  ][Poller            ] Available pollers: built-in
[2023-02-07 15:04:00.922661 +0000][Debug  ][Poller            ] Attempting to create a poller according to preference: built-in
[2023-02-07 15:04:00.922670 +0000][Debug  ][Poller            ] Creating poller: built-in
[2023-02-07 15:04:00.922683 +0000][Debug  ][Poller            ] Creating and starting the built-in poller...
[2023-02-07 15:04:00.922937 +0000][Debug  ][Poller            ] Using 1 poller threads
[2023-02-07 15:04:00.922951 +0000][Debug  ][TaskMgr           ] Starting the task manager...
[2023-02-07 15:04:00.922994 +0000][Debug  ][TaskMgr           ] Task manager started
[2023-02-07 15:04:00.923004 +0000][Debug  ][JobMgr            ] Starting the job manager...
[2023-02-07 15:04:00.923091 +0000][Debug  ][JobMgr            ] Job manager started, 3 workers
[2023-02-07 15:04:00.923104 +0000][Debug  ][TaskMgr           ] Registering task: "FileTimer task" to be run at: [2023-02-07 15:04:00 +0000]
[2023-02-07 15:04:00.923115 +0000][Dump   ][XRootD            ] [eos-fst10.alice-af.wigner.hu:1095] Sending message kXR_stat (path: /, flags: none)
[2023-02-07 15:04:00.923146 +0000][Debug  ][ExDbgMsg          ] [eos-fst10.alice-af.wigner.hu:1095] MsgHandler created: 0x1e8f450 (message: kXR_stat (path: /, flags: none) ).
[2023-02-07 15:04:00.923185 +0000][Dump   ][Utility           ] URL: eos-fst10.alice-af.wigner.hu:1095
[2023-02-07 15:04:00.923185 +0000][Dump   ][Utility           ] Protocol:  root
[2023-02-07 15:04:00.923185 +0000][Dump   ][Utility           ] User Name: 
[2023-02-07 15:04:00.923185 +0000][Dump   ][Utility           ] Password:  
[2023-02-07 15:04:00.923185 +0000][Dump   ][Utility           ] Host Name: eos-fst10.alice-af.wigner.hu
[2023-02-07 15:04:00.923185 +0000][Dump   ][Utility           ] Port:      1095
[2023-02-07 15:04:00.923185 +0000][Dump   ][Utility           ] Path:      
[2023-02-07 15:04:00.923222 +0000][Dump   ][Utility           ] URL: eos-fst10.alice-af.wigner.hu:1095
[2023-02-07 15:04:00.923222 +0000][Dump   ][Utility           ] Protocol:  root
[2023-02-07 15:04:00.923222 +0000][Dump   ][Utility           ] User Name: 
[2023-02-07 15:04:00.923222 +0000][Dump   ][Utility           ] Password:  
[2023-02-07 15:04:00.923222 +0000][Dump   ][Utility           ] Host Name: eos-fst10.alice-af.wigner.hu
[2023-02-07 15:04:00.923222 +0000][Dump   ][Utility           ] Port:      1095
[2023-02-07 15:04:00.923222 +0000][Dump   ][Utility           ] Path:      
[2023-02-07 15:04:00.923246 +0000][Debug  ][PostMaster        ] Creating new channel to: eos-fst10.alice-af.wigner.hu:1095 1 stream(s)
[2023-02-07 15:04:00.923277 +0000][Debug  ][PostMaster        ] [eos-fst10.alice-af.wigner.hu:1095 #0] Stream parameters: Network Stack: IPAuto, Connection Window: 120, ConnectionRetry: 5, Stream Error Window: 1800
[2023-02-07 15:04:00.923404 +0000][Debug  ][TaskMgr           ] Registering task: "TickGeneratorTask for: eos-fst10.alice-af.wigner.hu:1095" to be run at: [2023-02-07 15:04:15 +0000]
[2023-02-07 15:04:00.923421 +0000][Dump   ][PostMaster        ] [eos-fst10.alice-af.wigner.hu:1095 #0] Sending message kXR_stat (path: /, flags: none) (0x1e8a1a0) through substream 0 expecting answer at 0
[2023-02-07 15:04:00.923558 +0000][Debug  ][PostMaster        ] [eos-fst10.alice-af.wigner.hu:1095] Found 1 address(es): [::ffff:172.16.152.58]:1095
[2023-02-07 15:04:00.923600 +0000][Debug  ][AsyncSock         ] [eos-fst10.alice-af.wigner.hu:1095 #0.0] Attempting connection to [::ffff:172.16.152.58]:1095
[2023-02-07 15:04:00.923754 +0000][Debug  ][Poller            ] Adding socket 0x1e90200 to the poller
[2023-02-07 15:04:00.924084 +0000][Debug  ][AsyncSock         ] [eos-fst10.alice-af.wigner.hu:1095 #0.0] Async connection call returned
[2023-02-07 15:04:00.924152 +0000][Error  ][AsyncSock         ] [eos-fst10.alice-af.wigner.hu:1095 #0.0] Unable to connect: No route to host
[2023-02-07 15:04:00.924169 +0000][Debug  ][AsyncSock         ] [eos-fst10.alice-af.wigner.hu:1095 #0.0] Closing the socket
[2023-02-07 15:04:00.924188 +0000][Debug  ][Poller            ] <x><--><x> Removing socket from the poller
[2023-02-07 15:04:00.924240 +0000][Error  ][PostMaster        ] [eos-fst10.alice-af.wigner.hu:1095 #0] elapsed = 0, pConnectionWindow = 120 seconds.
[2023-02-07 15:04:00.924255 +0000][Info   ][PostMaster        ] [eos-fst10.alice-af.wigner.hu:1095 #0] Attempting reconnection in 120 seconds.
[2023-02-07 15:04:00.924269 +0000][Debug  ][TaskMgr           ] Registering task: "StreamConnectorTask for eos-fst10.alice-af.wigner.hu:1095 #0" to be run at: [2023-02-07 15:06:00 +0000]
[2023-02-07 15:04:01.923277 +0000][Dump   ][TaskMgr           ] Running task: "FileTimer task"
[2023-02-07 15:04:01.923344 +0000][Dump   ][TaskMgr           ] Will rerun task "FileTimer task" at [2023-02-07 15:04:16 +0000]
...

an example for a successful connection as well:

[root@eos-mgm1 ~]# env XRD_LOGLEVEL=Dump xrdfs eos-fst09.alice-af.wigner.hu:1095 stat /
[2023-02-07 15:06:12.963417 +0000][Debug  ][Utility           ] Unable to process user config file: [ERROR] OS Error: No such file or directory
[2023-02-07 15:06:12.963614 +0000][Debug  ][PlugInMgr         ] Initializing plug-in manager...
[2023-02-07 15:06:12.963626 +0000][Debug  ][PlugInMgr         ] No default plug-in, loading plug-in configs...
[2023-02-07 15:06:12.963636 +0000][Debug  ][PlugInMgr         ] Processing plug-in definitions in /etc/xrootd/client.plugins.d...
[2023-02-07 15:06:12.963707 +0000][Debug  ][PlugInMgr         ] Processing plug-in definitions in /root/.xrootd/client.plugins.d...
[2023-02-07 15:06:12.963722 +0000][Debug  ][PlugInMgr         ] Unable to process directory /root/.xrootd/client.plugins.d: [ERROR] OS Error: No such file or directory
[2023-02-07 15:06:12.963838 +0000][Dump   ][Utility           ] URL: eos-fst09.alice-af.wigner.hu:1095
[2023-02-07 15:06:12.963838 +0000][Dump   ][Utility           ] Protocol:  root
[2023-02-07 15:06:12.963838 +0000][Dump   ][Utility           ] User Name: 
[2023-02-07 15:06:12.963838 +0000][Dump   ][Utility           ] Password:  
[2023-02-07 15:06:12.963838 +0000][Dump   ][Utility           ] Host Name: eos-fst09.alice-af.wigner.hu
[2023-02-07 15:06:12.963838 +0000][Dump   ][Utility           ] Port:      1095
[2023-02-07 15:06:12.963838 +0000][Dump   ][Utility           ] Path:      
[2023-02-07 15:06:12.963883 +0000][Dump   ][Utility           ] URL: root://eos-fst09.alice-af.wigner.hu:1095/
[2023-02-07 15:06:12.963883 +0000][Dump   ][Utility           ] Protocol:  root
[2023-02-07 15:06:12.963883 +0000][Dump   ][Utility           ] User Name: 
[2023-02-07 15:06:12.963883 +0000][Dump   ][Utility           ] Password:  
[2023-02-07 15:06:12.963883 +0000][Dump   ][Utility           ] Host Name: eos-fst09.alice-af.wigner.hu
[2023-02-07 15:06:12.963883 +0000][Dump   ][Utility           ] Port:      1095
[2023-02-07 15:06:12.963883 +0000][Dump   ][Utility           ] Path:      
[2023-02-07 15:06:12.963915 +0000][Dump   ][Utility           ] URL: root://eos-fst09.alice-af.wigner.hu:1095/
[2023-02-07 15:06:12.963915 +0000][Dump   ][Utility           ] Protocol:  root
[2023-02-07 15:06:12.963915 +0000][Dump   ][Utility           ] User Name: 
[2023-02-07 15:06:12.963915 +0000][Dump   ][Utility           ] Password:  
[2023-02-07 15:06:12.963915 +0000][Dump   ][Utility           ] Host Name: eos-fst09.alice-af.wigner.hu
[2023-02-07 15:06:12.963915 +0000][Dump   ][Utility           ] Port:      1095
[2023-02-07 15:06:12.963915 +0000][Dump   ][Utility           ] Path:      
[2023-02-07 15:06:12.963949 +0000][Dump   ][Utility           ] URL: root://eos-fst09.alice-af.wigner.hu:1095/
[2023-02-07 15:06:12.963949 +0000][Dump   ][Utility           ] Protocol:  root
[2023-02-07 15:06:12.963949 +0000][Dump   ][Utility           ] User Name: 
[2023-02-07 15:06:12.963949 +0000][Dump   ][Utility           ] Password:  
[2023-02-07 15:06:12.963949 +0000][Dump   ][Utility           ] Host Name: eos-fst09.alice-af.wigner.hu
[2023-02-07 15:06:12.963949 +0000][Dump   ][Utility           ] Port:      1095
[2023-02-07 15:06:12.963949 +0000][Dump   ][Utility           ] Path:      
[2023-02-07 15:06:12.963996 +0000][Debug  ][App               ] Executing: stat / 
[2023-02-07 15:06:12.964007 +0000][Dump   ][App               ] Param #00: 'stat'
[2023-02-07 15:06:12.964015 +0000][Dump   ][App               ] Param #01: '/'
[2023-02-07 15:06:12.964059 +0000][Dump   ][FileSystem        ] [0x1374070@eos-fst09.alice-af.wigner.hu:1095] Sending kXR_stat (path: /, flags: none)
[2023-02-07 15:06:12.964092 +0000][Debug  ][Poller            ] Available pollers: built-in
[2023-02-07 15:06:12.964101 +0000][Debug  ][Poller            ] Attempting to create a poller according to preference: built-in
[2023-02-07 15:06:12.964110 +0000][Debug  ][Poller            ] Creating poller: built-in
[2023-02-07 15:06:12.964124 +0000][Debug  ][Poller            ] Creating and starting the built-in poller...
[2023-02-07 15:06:12.964326 +0000][Debug  ][Poller            ] Using 1 poller threads
[2023-02-07 15:06:12.964340 +0000][Debug  ][TaskMgr           ] Starting the task manager...
[2023-02-07 15:06:12.964381 +0000][Debug  ][TaskMgr           ] Task manager started
[2023-02-07 15:06:12.964391 +0000][Debug  ][JobMgr            ] Starting the job manager...
[2023-02-07 15:06:12.964480 +0000][Debug  ][JobMgr            ] Job manager started, 3 workers
[2023-02-07 15:06:12.964492 +0000][Debug  ][TaskMgr           ] Registering task: "FileTimer task" to be run at: [2023-02-07 15:06:12 +0000]
[2023-02-07 15:06:12.964504 +0000][Dump   ][XRootD            ] [eos-fst09.alice-af.wigner.hu:1095] Sending message kXR_stat (path: /, flags: none)
[2023-02-07 15:06:12.964536 +0000][Debug  ][ExDbgMsg          ] [eos-fst09.alice-af.wigner.hu:1095] MsgHandler created: 0x1379450 (message: kXR_stat (path: /, flags: none) ).
[2023-02-07 15:06:12.964581 +0000][Dump   ][Utility           ] URL: eos-fst09.alice-af.wigner.hu:1095
[2023-02-07 15:06:12.964581 +0000][Dump   ][Utility           ] Protocol:  root
[2023-02-07 15:06:12.964581 +0000][Dump   ][Utility           ] User Name: 
[2023-02-07 15:06:12.964581 +0000][Dump   ][Utility           ] Password:  
[2023-02-07 15:06:12.964581 +0000][Dump   ][Utility           ] Host Name: eos-fst09.alice-af.wigner.hu
[2023-02-07 15:06:12.964581 +0000][Dump   ][Utility           ] Port:      1095
[2023-02-07 15:06:12.964581 +0000][Dump   ][Utility           ] Path:      
[2023-02-07 15:06:12.964619 +0000][Dump   ][Utility           ] URL: eos-fst09.alice-af.wigner.hu:1095
[2023-02-07 15:06:12.964619 +0000][Dump   ][Utility           ] Protocol:  root
[2023-02-07 15:06:12.964619 +0000][Dump   ][Utility           ] User Name: 
[2023-02-07 15:06:12.964619 +0000][Dump   ][Utility           ] Password:  
[2023-02-07 15:06:12.964619 +0000][Dump   ][Utility           ] Host Name: eos-fst09.alice-af.wigner.hu
[2023-02-07 15:06:12.964619 +0000][Dump   ][Utility           ] Port:      1095
[2023-02-07 15:06:12.964619 +0000][Dump   ][Utility           ] Path:      
[2023-02-07 15:06:12.964643 +0000][Debug  ][PostMaster        ] Creating new channel to: eos-fst09.alice-af.wigner.hu:1095 1 stream(s)
[2023-02-07 15:06:12.964676 +0000][Debug  ][PostMaster        ] [eos-fst09.alice-af.wigner.hu:1095 #0] Stream parameters: Network Stack: IPAuto, Connection Window: 120, ConnectionRetry: 5, Stream Error Window: 1800
[2023-02-07 15:06:12.964814 +0000][Debug  ][TaskMgr           ] Registering task: "TickGeneratorTask for: eos-fst09.alice-af.wigner.hu:1095" to be run at: [2023-02-07 15:06:27 +0000]
[2023-02-07 15:06:12.964833 +0000][Dump   ][PostMaster        ] [eos-fst09.alice-af.wigner.hu:1095 #0] Sending message kXR_stat (path: /, flags: none) (0x13741a0) through substream 0 expecting answer at 0
[2023-02-07 15:06:12.964969 +0000][Debug  ][PostMaster        ] [eos-fst09.alice-af.wigner.hu:1095] Found 1 address(es): [::ffff:172.16.152.57]:1095
[2023-02-07 15:06:12.965008 +0000][Debug  ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Attempting connection to [::ffff:172.16.152.57]:1095
[2023-02-07 15:06:12.965064 +0000][Debug  ][Poller            ] Adding socket 0x137a200 to the poller
[2023-02-07 15:06:12.965311 +0000][Debug  ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Async connection call returned
[2023-02-07 15:06:12.965412 +0000][Debug  ][XRootDTransport   ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Sending out the initial hand shake + kXR_protocol
[2023-02-07 15:06:12.965475 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Wrote a message:  (0xe8000950), 44 bytes
[2023-02-07 15:06:12.965569 +0000][Dump   ][XRootDTransport   ] [msg: 0xe8000950] Expecting 8 bytes of message body
[2023-02-07 15:06:12.965583 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received message header, size: 8
[2023-02-07 15:06:12.965604 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received a message of 16 bytes
[2023-02-07 15:06:12.965623 +0000][Debug  ][XRootDTransport   ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Got the server hand shake response (type: server [], protocol version 400)
[2023-02-07 15:06:12.965641 +0000][Dump   ][XRootDTransport   ] [msg: 0xe8000950] Expecting 8 bytes of message body
[2023-02-07 15:06:12.965651 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received message header, size: 8
[2023-02-07 15:06:12.965664 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received a message of 16 bytes
[2023-02-07 15:06:12.965680 +0000][Debug  ][XRootDTransport   ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] kXR_protocol successful (type: server [], protocol version 400)
[2023-02-07 15:06:12.966133 +0000][Debug  ][XRootDTransport   ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Sending out kXR_login request, username: root, cgi: ?xrd.cc=hu&xrd.tz=0&xrd.appname=xrdfs&xrd.info=&xrd.hostname=eos-mgm1.alice-af.wigner.hu&xrd.rn=v4.12.7, dual-stack: false, private IPv4: false, private IPv6: false
[2023-02-07 15:06:12.966173 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Wrote a message:  (0xe8000a60), 127 bytes
[2023-02-07 15:06:12.966252 +0000][Dump   ][XRootDTransport   ] [msg: 0xe8000950] Expecting 50 bytes of message body
[2023-02-07 15:06:12.966266 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received message header, size: 8
[2023-02-07 15:06:12.966279 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received a message of 58 bytes
[2023-02-07 15:06:12.966298 +0000][Debug  ][XRootDTransport   ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Logged in, session: 1e01000066f40000500000001f010000
[2023-02-07 15:06:12.966309 +0000][Debug  ][XRootDTransport   ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Authentication is required: &P=unix&P=sss,0.13:/etc/eos.keytab
[2023-02-07 15:06:12.966323 +0000][Debug  ][XRootDTransport   ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Sending authentication data
[2023-02-07 15:06:12.967028 +0000][Debug  ][XRootDTransport   ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Trying to authenticate using unix
[2023-02-07 15:06:12.967190 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Wrote a message:  (0xe8001ed0), 39 bytes
[2023-02-07 15:06:12.967270 +0000][Dump   ][XRootDTransport   ] [msg: 0xe8001ed0] Expecting 0 bytes of message body
[2023-02-07 15:06:12.967282 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received message header, size: 8
[2023-02-07 15:06:12.967291 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received a message of 8 bytes
[2023-02-07 15:06:12.967304 +0000][Debug  ][XRootDTransport   ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Authenticated with unix.
[2023-02-07 15:06:12.967317 +0000][Debug  ][PostMaster        ] [eos-fst09.alice-af.wigner.hu:1095 #0] Stream 0 connected.
[2023-02-07 15:06:12.967330 +0000][Debug  ][Utility           ] Monitor library name not set. No monitoring
[2023-02-07 15:06:12.967367 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Wrote a message: kXR_stat (path: /, flags: none) (0x13741a0), 25 bytes
[2023-02-07 15:06:12.967377 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Successfully sent message: kXR_stat (path: /, flags: none) (0x13741a0).
[2023-02-07 15:06:12.967395 +0000][Dump   ][XRootD            ] [eos-fst09.alice-af.wigner.hu:1095] Message kXR_stat (path: /, flags: none) has been successfully sent.
[2023-02-07 15:06:12.967404 +0000][Debug  ][ExDbgMsg          ] [eos-fst09.alice-af.wigner.hu:1095] Moving MsgHandler: 0x1379450 (message: kXR_stat (path: /, flags: none) ) from out-queu to in-queue.
[2023-02-07 15:06:12.967426 +0000][Dump   ][PostMaster        ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] All messages consumed, disable uplink
[2023-02-07 15:06:12.967606 +0000][Dump   ][XRootDTransport   ] [msg: 0xe80008c0] Expecting 31 bytes of message body
[2023-02-07 15:06:12.967708 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received message header for 0xe80008c0 size: 8
[2023-02-07 15:06:12.967763 +0000][Dump   ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Received message 0xe80008c0 of 39 bytes
[2023-02-07 15:06:12.967783 +0000][Dump   ][PostMaster        ] [eos-fst09.alice-af.wigner.hu:1095 #0] Handling received message: 0xe80008c0.
[2023-02-07 15:06:12.968017 +0000][Dump   ][XRootD            ] [eos-fst09.alice-af.wigner.hu:1095] Got a kXR_ok response to request kXR_stat (path: /, flags: none)
[2023-02-07 15:06:12.968075 +0000][Debug  ][ExDbgMsg          ] [eos-fst09.alice-af.wigner.hu:1095] Calling MsgHandler: 0x1379450 (message: kXR_stat (path: /, flags: none) ) with status: [SUCCESS] .
[2023-02-07 15:06:12.968095 +0000][Dump   ][XRootD            ] [eos-fst09.alice-af.wigner.hu:1095] Parsing the response to kXR_stat (path: /, flags: none) as StatInfo: 11240347374 4096 19 1614856833
[2023-02-07 15:06:12.968132 +0000][Debug  ][ExDbgMsg          ] [eos-fst09.alice-af.wigner.hu:1095] Destroying MsgHandler: 0x1379450.
Path:   /
Id:     11240347374
Size:   4096
MTime:  2021-03-04 11:20:33
Flags:  19 (XBitSet|IsDir|IsReadable)
[2023-02-07 15:06:12.968282 +0000][Debug  ][JobMgr            ] Stopping the job manager...
[2023-02-07 15:06:12.968304 +0000][Dump   ][JobMgr            ] Stopping worker #0...
[2023-02-07 15:06:12.968431 +0000][Dump   ][JobMgr            ] Worker #0 stopped
[2023-02-07 15:06:12.968443 +0000][Dump   ][JobMgr            ] Stopping worker #1...
[2023-02-07 15:06:12.968491 +0000][Dump   ][JobMgr            ] Worker #1 stopped
[2023-02-07 15:06:12.968502 +0000][Dump   ][JobMgr            ] Stopping worker #2...
[2023-02-07 15:06:12.968543 +0000][Dump   ][JobMgr            ] Worker #2 stopped
[2023-02-07 15:06:12.968553 +0000][Debug  ][JobMgr            ] Job manager stopped
[2023-02-07 15:06:12.968562 +0000][Debug  ][TaskMgr           ] Stopping the task manager...
[2023-02-07 15:06:12.968641 +0000][Debug  ][TaskMgr           ] Task manager stopped
[2023-02-07 15:06:12.968652 +0000][Debug  ][Poller            ] Stopping the poller...
[2023-02-07 15:06:12.968752 +0000][Debug  ][TaskMgr           ] Requesting unregistration of: "TickGeneratorTask for: eos-fst09.alice-af.wigner.hu:1095"
[2023-02-07 15:06:12.968769 +0000][Debug  ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Closing the socket
[2023-02-07 15:06:12.968783 +0000][Debug  ][Poller            ] <[::ffff:172.16.152.16]:45400><--><[::ffff:172.16.152.57]:1095> Removing socket from the poller
[2023-02-07 15:06:12.968823 +0000][Debug  ][PostMaster        ] [eos-fst09.alice-af.wigner.hu:1095 #0] Destroying stream
[2023-02-07 15:06:12.968835 +0000][Debug  ][AsyncSock         ] [eos-fst09.alice-af.wigner.hu:1095 #0.0] Closing the socket

Practically there is no difference (that I’m aware of) between fst09 and fst10, so I’m very much clueless. Do you have maybe any suggestion?

Thanks,
Gabor

Any suggestions on this?

The log is clear:

[2023-02-07 15:04:00.923600 +0000][Debug  ][AsyncSock         ] [eos-fst10.alice-af.wigner.hu:1095 #0.0] Attempting connection to [::ffff:172.16.152.58]:1095
[2023-02-07 15:04:00.923754 +0000][Debug  ][Poller            ] Adding socket 0x1e90200 to the poller
[2023-02-07 15:04:00.924084 +0000][Debug  ][AsyncSock         ] [eos-fst10.alice-af.wigner.hu:1095 #0.0] Async connection call returned
[2023-02-07 15:04:00.924152 +0000][Error  ][AsyncSock         ] [eos-fst10.alice-af.wigner.hu:1095 #0.0] Unable to connect: No route to host
[2023-02-07 15:04:00.924169 +0000][Debug  ][AsyncSock         ] [eos-fst10.alice-af.wigner.hu:1095 #0.0] Closing the socket

You should check

  1. is the FST alive, can you connect to it from the fst itself? Is maybe the FST continuously crashing (check /var/log/eos/fst/ …)
  2. do you have forgotten to open port 1095 in the firewall on fst9 ?
  3. is the DNS entry correct? can you ssh from the mgm using the given IPV6 address?

Cheers Andreas.