georgep
November 17, 2021, 10:29am
1
Hello,
Occasionaly, we can’t issue eos client commands on the MGM node getting the following error message
error: MGM root://antares-eos01.scd.rl.ac.uk not online/reachable
Not sure if this poses a problem for the FSTs (if the rely on the MGM being up) but the FST GC service that we have set up to run with CTA cannot run, e.g.
2021/11/17 09:55:32.723000 antares-eos02.scd.rl.ac.uk ERROR cta-fst-gcd:LVL=“ERROR” PID=“70182” TID=“70182” MSG=“fsls: Failed to execute eos -r 0 0 root://antares-eos01.scd.rl.ac.uk fs ls -m: returncode=64 returncodestr=‘Machine is not on the network’ stderr=‘error: MGM root://antares-eos01.scd.rl.ac.uk not online/reachable’”
Restarting eos@mgm would temporarily fix this, but it is troubling that is happening. Do you have any ideas how to find our what is happening and how to resolve it on a permanent basis?
Thanks,
George
esindril
November 17, 2021, 2:02pm
2
Hi George,
This usually points to some networking problem with your instance. What the client command does it to first try to ping the MGM and if it gets not reply within 5 seconds then it declares the MGM not online.
So unless your MGM is not crashing regularly and getting restarted by systemd, which you can easily check if you connect to the MGM machine, this means some problem with the networking.
Cheers,
georgep
November 18, 2021, 1:13pm
3
Hi Elvin,
Thanks for this. It is not very likely that this is happening because of a network problem. We often see that the MGM becomes gradually slower in the response to EOS client commands (which succeed eventually) until the point when it becomes unresponsive as mentioned above.
Is there any error in the MGM log indicating that the MGM somehow “hangs”?
For info, we are running 4.8.45-1.
George
esindril
November 18, 2021, 1:51pm
4
Hi George,
It’s highly unlikely that the MGM can not deal with the number of requests. Unfortunately, given the available amount of information it’s very hard to say exactly what is the problem. Doing eos ns stat should already give you a pretty good view of the requests that it’s handling and the average response time per request. Hope this helps.
Cheers,
apeters
November 18, 2021, 2:09pm
5
Hi George,
┌────────┬───────┬────────┬───────┬──────┬─────────┬────────────────┐
│ uid│threads│sessions│ limit│stalls│stalltime│ status│
└────────┴───────┴────────┴───────┴──────┴─────────┴────────────────┘
0 1 4 65.54 K 0 1 user-OK
georgep
November 18, 2021, 3:03pm
6
Hi Elvin, Andreas
Thanks for the replies. In case it helps, I paste the requests with the highest (judged roughly by eye) number of sum of instances as I see them in the output"eos ns stat"
all AttrLs 187.18 K 16.50 8.81 3.81 14.18 0.11 ± 0.03
George
apeters
November 18, 2021, 3:23pm
7
Yeah, that looks all ‘peanuts’ … the last column shows you the average execution time … maybe you can post the full listing here to see if there is something sticking out.
georgep
November 18, 2021, 5:13pm
8
Thanks, here is the full listing
│who│command │ sum│ 5s│ 1min│ 5min│ 1h│exec(ms)│sigma(ms)│