We upgraded to 4.8.78 (from 4.8.45) on our EOS preprod instance. quarkDB (0.4.2-1) is running ok with a quorum as well as eos@mgm and eos@mq services. Unfortunately, when we try to issue EOS client commands on the MGM (such as eos ns), the MGM is experiencing a segmentation fault. You can see the var_log_messages and core_backtrace obtained by abrt-cli in the following links
Could you give me a bit more info? Do you have a coredump that I could look into? If not could you attach with gdb to the MGM process before it dies and send me a backtrace one it crashes? For example:
gdb -p <mgm_pid>
set pagination off
set logging on
continue
.....
<crash happens>
....
bt
info local
Thanks, I can now read the coredump. It seems that you MGM does not fully initialize and it gets a request before some of the internal data structures are ready. Could you please also send me the /var/log/eos/mgm/xrdlog.mgm file when such a crash is happening?
It seems that you have some process running on the same machine as the MGM which is trying to issue some command to it, while it’s not yet fully up and running. Can you try to stop any such process and restart the MGM? I would expect the MGM to start fine.
This is the user which is trying to execute some proc admin command:
#11 0x00007ff0e2762068 in XrdMgmOfsFile::open (this=0x7ff094002cb0, invid=0x0, inpath=0x7ff094002400 "/proc/admin/", open_mode=0, Mode=384, client=0x7ff094002ad8, ininfo=0x7ff09400240d "mgm.cmd.proto=GgQKAigB+AEB") at /root/rpmbuild/BUILD/eos-4.8.78-1/mgm/XrdMgmOfsFile.cc:396
more precisely root@antares-eos14^sss^antares-eos14.scd.rl.ac.uk
Then, one more intriguing thing is that in your coredump and logs I don’t see all the messages I would expect from the MGM initialization part.
Try out the first suggestion and see if things improve. I will have another look over the logs you provided maybe I find some further clues.
I think you are bitten by this part of the code - reading carefully the logs does help indeed …
if (mTapeEnabled && !getenv("EOS_HA_REDIRECT_READS")) {
std::ostringstream msg;
msg << "msg="
<< "Mgm node with tape-aware garbage collection must redirect all write/read traffic to master\"";
eos_crit(msg.str().c_str());
return 1;
}
So you need to set the env variable EOS_HA_REDIRECT_READS so that the configuration will work all the way to the end. This also explains why I could not see all the initialization bits. This is need to that all reads happen through the MGM master node and the garbage collector is properly notified about the files which are accessed for read. If reads were to be served by the slaves then this info would be lost.
Most probably the process you mention above should be from us when we try to issue admin commands (eos ns) to check if the MGM is responsive. We do this without knowing if the MGM initialization has been completed or not.
Not sure I understand the meaning of EOS_HA_REDIRECT_READS: in our EOS set up we don’t have a master and slave node but only a single MGM. Is this env var still relevant in our setup?
Yes, you need that value if you want your MGM which is tape enabled to start. This env variable is specific to MGM daemons which are supposed to run CTA - which I expect is you case. One can have multiple MGMs in an EOS cluster and this env variable was added so that we ensure no (slave) MGM processes any read requests otherwise the tape GC (which runs only at the MGM master) might not pick them up.
In normal circumstances, you should not care if the MGM is initialized or not, XRootD will take care of it, but these are not normal circumstances since the MGM fails the pass the configuration step due to the reason outlined in the first paragraph.
Setting EOS_HA_REDIRECT_READS=1 did indeed solve the issue; many thanks for this.
For my reference, has the above code snippet (which checks the content of the EOS_HA_REDIRECT_READS variable) been added in one of the EOS versions later than 4.8.45?