EOS MGM Fails to Start After Upgrade to 4.3.12

After upgrade of eos-server and eos-client from 4.2.24 to 4.3.12 the mgm fails to start, xrdlog.mgm below.

The FSTs restarted okay, and continued to serve after restart.

MGM failed to restart, regardless of using the pre-existing /etc/xrd.cf.mgm or the newly provided /etc/xrd.cf.mgm.rpmnew

Any suggestions on how to resolve?

Cheers,
Pete

xrdlog.mgm using pre-existing /etc/xrd.cf.mgm

17:24:46 # cat /var/log/eos/mgm/xrdlog.mgm              
181018 17:24:46 10859 Starting on Linux 2.6.32-696.30.1.el6.x86_64
Copr.  2004-2012 Stanford University, xrd version v4.8.3
++++++ xrootd mgm@alice-eos-01.ornl.gov initialization started.
Config using configuration file /etc/xrd.cf.mgm
=====> xrd.sched mint 8 maxt 256 idle 64
=====> xrd.network keepalive
=====> xrd.timeout idle 86400
Config maximum number of connections restricted to 65000
Copr.  2012 Stanford University, xrootd protocol 3.1.0 version v4.8.3
++++++ xrootd protocol initialization started.
=====> xrootd.fslib libXrdEosMgm.so
=====> xrootd.seclib libXrdSec.so
=====> xrootd.async off nosf
=====> xrootd.chksum adler32
=====> all.export /
Config exporting /
Plugin loaded 
++++++ Authentication system initialization started.
Plugin loaded 
=====> sec.protocol unix
Plugin loaded 
=====> sec.protocol sss -c /etc/eos.keytab -s /etc/eos.keytab
=====> sec.protbind localhost.localdomain sss unix
=====> sec.protbind localhost sss unix
=====> sec.protbind * only sss unix
=====> sec.protbind warp-ornl-cern-01 only sss
=====> sec.protbind warp-ornl-cern-01.ornl.gov only sss
=====> sec.protbind warp-ornl-cern-02 only sss
=====> sec.protbind warp-ornl-cern-02.ornl.gov only sss
=====> sec.protbind warp-ornl-cern-03 only sss
=====> sec.protbind warp-ornl-cern-03.ornl.gov only sss
=====> sec.protbind warp-ornl-cern-04 only sss
=====> sec.protbind warp-ornl-cern-04.ornl.gov only sss
Config 13 authentication directives processed in /etc/xrd.cf.mgm
------ Authentication system initialization completed.
++++++ Protection system initialization started.
Config warning: Security level is set to none; request protection disabled!
Config Local  protection level: none
Config Remote protection level: none
------ Protection system initialization completed.
Config Routing for alice-eos-01.ornl.gov: local pub4 prv4 
Config Route all4: alice-eos-01.ornl.gov Dest=[::192.188.182.18]:1094
Plugin No such file or directory loading fslib libXrdEosMgm-4.so
Config Falling back to using libXrdEosMgm.so
Plugin loaded 
++++++ (c) 2015 CERN/IT-DSS MgmOfs (meta data redirector) 4.3.12
=====> mgmofs enforces SSS authentication for XROOT clients
jemalloc is loaded!
jemalloc heap profiling is disabled
=====> mgmofs.hostname: alice-eos-01.ornl.gov
=====> mgmofs.hostpref: alice-eos-01
=====> mgmofs.managerid: alice-eos-01.ornl.gov:1094
=====> mgmofs.fs: /
=====> mgmofs.targetport: 1095
181018 17:24:46 10859 MgmOfs_Config: I cannot acccess you authorization library!
=====> mgmofs.authlib : 
=====> mgmofs.authorize : true
=====> mgmofs.instance : eosaliceornl
=====> mgmofs.metalog: /var/eos/md
=====> mgmofs.txdir:   /var/eos/tx
=====> mgmofs.authdir:   /var/eos/auth
=====> mgmofs.reportstorepath: /var/eos/report
=====> mgmofs.fstgw: someproxy.cern.ch:3001
=====> mgmofs.nslib : /usr/lib64/libEosNsInMemory.so
181018 17:24:46 10859 XrootdConfig: Unable to create file system object via libXrdEosMgm.so
181018 17:24:46 10859 XrootdConfig: Unable to load file system.
------ xrootd protocol initialization failed.
181018 17:24:46 10859 XrdProtocol: Protocol xrootd could not be loaded
------ xrootd mgm@alice-eos-01.ornl.gov:-1 initialization failed.

xrdlog.mgm using using newly provided xrd.cf.mgm.rpmnew

17:23:16 # cat /var/log/eos/mgm/xrdlog.mgm
181018 17:22:54 28007 Starting on Linux 2.6.32-696.30.1.el6.x86_64
Copr.  2004-2012 Stanford University, xrd version v4.8.3
++++++ xrootd mgm@alice-eos-01.ornl.gov initialization started.
Config using configuration file /etc/xrd.cf.mgm
=====> xrd.sched mint 8 maxt 256 idle 64
Config maximum number of connections restricted to 65000
Copr.  2012 Stanford University, xrootd protocol 3.1.0 version v4.8.3
++++++ xrootd protocol initialization started.
=====> xrootd.fslib libXrdEosMgm.so
=====> xrootd.seclib libXrdSec.so
=====> xrootd.async off nosf
=====> xrootd.chksum adler32
=====> all.export / nolock
Config exporting /
Plugin loaded 
++++++ Authentication system initialization started.
Plugin loaded 
=====> sec.protocol unix
Plugin loaded 
=====> sec.protocol sss -c /etc/eos.keytab -s /etc/eos.keytab
Plugin loaded 
Template for exports not set
Seckrb5: Unable to start sequence on the keytab file FILE:/etc/krb5.keytab; No such file or directory
=====> sec.protocol krb5 host/<host>@CERN.CH
Plugin loaded 
181018 17:22:54 28007 secgsi_InitOpts: *** ------------------------------------------------------------ ***
181018 17:22:54 28007 secgsi_InitOpts:  Mode: server
181018 17:22:54 28007 secgsi_InitOpts:  Debug: 0
181018 17:22:54 28007 secgsi_InitOpts:  CA dir: /etc/grid-security/certificates/
181018 17:22:54 28007 secgsi_InitOpts:  CA verification level: 1
181018 17:22:54 28007 secgsi_InitOpts:  CRL dir: /etc/grid-security/certificates/
181018 17:22:54 28007 secgsi_InitOpts:  CRL extension: .r0
181018 17:22:54 28007 secgsi_InitOpts:  CRL check level: 0
181018 17:22:54 28007 secgsi_InitOpts:  Certificate: /etc/grid-security/daemon/hostcert.pem
181018 17:22:54 28007 secgsi_InitOpts:  Key: /etc/grid-security/daemon/hostkey.pem
181018 17:22:54 28007 secgsi_InitOpts:  Proxy delegation option: 0
181018 17:22:54 28007 secgsi_InitOpts:  GRIDmap file: /etc/grid-security/grid-mapfile
181018 17:22:54 28007 secgsi_InitOpts:  GRIDmap option: 2
181018 17:22:54 28007 secgsi_InitOpts:  GRIDmap cache entries expiration (secs): 600
181018 17:22:54 28007 secgsi_InitOpts:  Client proxy availability in XrdSecEntity.endorsement: 0
181018 17:22:54 28007 secgsi_InitOpts:  VOMS option: 1
181018 17:22:54 28007 secgsi_InitOpts:  MonInfo option: 1
181018 17:22:54 28007 secgsi_InitOpts:  Crypto modules: ssl
181018 17:22:54 28007 secgsi_InitOpts:  Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc
181018 17:22:54 28007 secgsi_InitOpts:  MDigests: sha1:md5
181018 17:22:54 28007 secgsi_InitOpts: *** ------------------------------------------------------------ ***
181018 17:22:54 28007 secgsi_GetSrvCertEnt: problems loading srv cert: not EEC but: CA
181018 17:22:54 28007 secgsi_Init: problems loading srv cert
181018 17:22:54 28007 secgsi_XrdOucGMap: cannot access grid map file '/etc/grid-security/grid-mapfile'in read mode; errno: 2 - aborting
181018 17:22:54 28007 secgsi_Init: Secgsi: ErrError: error loading grid map file:: /etc/grid-security/grid-mapfile
Secgsi: ErrError: error loading grid map file:: /etc/grid-security/grid-mapfile
=====> sec.protocol gsi -crl:0 -cert:/etc/grid-security/daemon/hostcert.pem -key:/etc/grid-security/daemon/hostkey.pem -gridmap:/etc/grid-security/grid-mapfile -d:0 -gmapopt:2 -vomsat:1 -moninfo:1
=====> sec.protbind localhost.localdomain unix sss
=====> sec.protbind localhost unix sss
181018 17:22:54 28007 sec_Config: protbind krb5 protocol not previously defined.
=====> sec.protbind * only krb5
Config 7 authentication directives processed in /etc/xrd.cf.mgm
------ Authentication system initialization failed.
Config Unable to create security framework via libXrdSec.so
181018 17:22:54 28007 XrootdConfig: Unable to load security system.
------ xrootd protocol initialization failed.
181018 17:22:54 28007 XrdProtocol: Protocol xrootd could not be loaded
------ xrootd mgm@alice-eos-01.ornl.gov:-1 initialization failed.

Removing mgmofs references below from existing xrd.cf.mgm config appears to have allowed mgm to start.

#mgmofs.authlib /usr/lib64/libXrdAliceTokenAcc.so
#mgmofs.authorize 1

libXrdAliceTokenAcc.so seems to no longer exist, and both lines are commented out in provided xrd.cf.mgm.rpmnew

Hi Pete
the lib is available in the citrine-depend repos

http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/

i guess you have that lib installed before, do you know if it has been removed during the rpm upgrade?
cheers
Andrea

Hi Andrea,

I’m not sure if the lib was removed in the 4.3.12 upgrade, though I noted the reference to it was commented out in the xrd.cf.mgm.rpmnew – so commenting it out in our production xrd.cf.mgm allowed the mgm to start.

Cheers,
Pete