CERN Accelerating science

MGM authentication system initialization failed

Hi there. I’m trying to get EOS working with a FreeIPA client which provides kerberos on a Centos8 system but I’m running into security configuration setup issues.
I think there is something wrong with permissions or format of my keys, but I’m not sure how I would check it beyond the steps listed below.
I have kerberos working properly with FreeIPA because I can get tickets and use ksu to become root.
My kerberos domain is CS.RU.IS. The host’s name is archive.ru.is.
Of note, this is the test server that I was able to get the docker demo to run with help from another community member @esindril.

This is /etc/krb5.conf
#File modified by ipa-client-install

includedir /etc/krb5.conf.d/
includedir /var/lib/sss/pubconf/krb5.include.d/

[libdefaults]
  default_realm = CS.RU.IS
  dns_lookup_realm = true
  dns_lookup_kdc = true
  rdns = false
  dns_canonicalize_hostname = false
  ticket_lifetime = 24h
  forwardable = true
  udp_preference_limit = 0
  default_ccache_name = KEYRING:persistent:%{uid}


[realms]
  CS.RU.IS = {
    pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
    pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem

  }


[domain_realm]
  .cs.ru.is = CS.RU.IS
  cs.ru.is = CS.RU.IS
  archive.ru.is = CS.RU.IS
  .ru.is = CS.RU.IS
  ru.is = CS.RU.IS

Here is what I have in /etc/xrd.cf.mgm regarding authentication:

# UNIX authentication
sec.protocol unix
# SSS authentication
sec.protocol sss -c /etc/eos.keytab -s /etc/eos.keytab
# KRB  authentication
#sec.protocol krb5 -exptkn:/var/eos/auth/krb5#<uid> host/<host>@CS.RU.IS
sec.protocol krb5 host/archive.ru.is@CS.RU.IS

# GSI authentication
#sec.protocol gsi -crl:0 -cert:/etc/grid-security/daemon/hostcert.pem -key:/etc/grid-security/daemon/hostkey.pem$
#sec.protocol gsi -crl:0 -cert:/etc/grid-security/daemon/hostcert.pem -key:/etc/grid-security/daemon/hostkey.pem$

###########################################################
sec.protbind localhost.localdomain unix sss
sec.protbind localhost unix sss
#sec.protbind * only krb5 gsi sss unix
sec.protbind * only krb5 sss unix
###########################################################

I have copied my /etc/krb5.keytab to /etc/eos.keytab:

ls -la *\keytab
-r--------. 1 daemon daemon 158 Mar 30 10:20 eos.keytab
-rw-------. 1 root   root   158 Mar 30 10:22 krb5.keytab

And I think it is in the right formats with the latest KVNO

ktutil
ktutil:  rkt eos.keytab
ktutil:  l -p
l: usage: l [-t] [-k] [-e]
ktutil:  l -e
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    2              host/archive.ru.is@CS.RU.IS (aes256-cts-hmac-sha1-96)
   2    2              host/archive.ru.is@CS.RU.IS (aes128-cts-hmac-sha1-96)

But when I systemctl start eos I see these errors in systemctl:

 systemctl start eos
 systemctl status eos
● eos.service - EOS All Services
   Loaded: loaded (/usr/lib/systemd/system/eos.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

Apr 07 10:41:36 archive.ru.is systemd[1]: Starting EOS All Services...
Apr 07 10:41:36 archive.ru.is sh[407335]: Waiting for 407340 ...
Apr 07 10:41:40 archive.ru.is sh[407335]: Waiting for 407341 ...
Apr 07 10:41:41 archive.ru.is sh[407335]: Waiting for 407342 ...
Apr 07 10:41:41 archive.ru.is echo[407552]: For status of daemons, run 'journalctl -e' or 'systemctl status eos@>
Apr 07 10:41:41 archive.ru.is systemd[1]: eos.service: Succeeded.
Apr 07 10:41:41 archive.ru.is systemd[1]: Started EOS All Services.

in /var/log/eos/mgm/xrdlog.mgm

210407 10:42:47 410497 Starting on Linux 4.18.0-240.15.1.el8_3.x86_64
Copr.  2004-2012 Stanford University, xrd version v4.12.5
++++++ xrootd mgm@archive.ru.is initialization started.
Config using configuration file /etc/xrd.cf.mgm
=====> xrd.sched mint 8 maxt 256 idle 64
Config maximum number of connections restricted to 65000
Config maximum number of threads restricted to 7118
Copr.  2012 Stanford University, xrootd protocol 4.0.0 version v4.12.5
++++++ xrootd protocol initialization started.
=====> xrootd.fslib libXrdEosMgm.so
=====> xrootd.seclib libXrdSec.so
=====> xrootd.async off nosf
=====> xrootd.chksum adler32
=====> all.export / nolock
Config exporting /
Plugin loaded
++++++ Authentication system initialization started.
Plugin loaded
=====> sec.protocol unix
Plugin loaded
Secsss (getKeyTab): keytable format missing or unsupported; line 1 in /etc/eos.keytab
Secsss (getKeyTab): keytable format missing or unsupported; line 1 in /etc/eos.keytab
Invalid keytab file.
=====> sec.protocol sss -c /etc/eos.keytab -s /etc/eos.keytab
Plugin loaded
Template for exports not set
Seckrb5: Unable to start sequence on the keytab file FILE:/etc/krb5.keytab; Permission denied
=====> sec.protocol krb5 host/archive.ru.is@CS.RU.IS
210407 10:42:47 410497 sec_Config: protbind sss protocol not previously defined.
=====> sec.protbind localhost.localdomain unix sss
210407 10:42:47 410497 sec_Config: protbind sss protocol not previously defined.
=====> sec.protbind localhost unix sss
210407 10:42:47 410497 sec_Config: protbind krb5 protocol not previously defined.
=====> sec.protbind * only krb5
Config 6 authentication directives processed in /etc/xrd.cf.mgm
------ Authentication system initialization failed.
Config Unable to create security framework via libXrdSec.so
210407 10:42:47 410497 XrootdConfig: Unable to load security system.
------ xrootd protocol initialization failed.
210407 10:42:47 410497 XrdProtocol: Protocol xrootd could not be loaded
------ xrootd mgm@archive.ru.is:-1 initialization failed.

For comparison, I logged into the fst instance on the docker setup I got working previously to see what was up with the /etc/eos.keytab. This is what happened which seems really weird, especially since eos is working fine there and can be mounted:

docker exec -it eos-cli1 bash
[root@eos-cli1 /]# ktutil
ktutil:  rkt /etc/eos.keytab
rkt: Unsupported key table format version number while reading keytab "/etc/eos.keytab"
ktutil:  quit

Is there some interesting difference between a krb5.keytab and an eos.keytab? Why can’t ktutil read it?

Looking at the eos.keytab provided by the package eos-testkeytab, it is in a very different format:

 file eos.keytab
eos.keytab: ASCII text
 less eos.keytab
0 u:daemon g:daemon n:eos-test N:5584055516046295042 c:1300139240 e:0 k:a94721fa30d4eac6a6a4ff3425f6836e1607446dd7b268fe022f7448a3ed8d93

I reconfigured the mgm node closer to what was found on the docker instance and now it starts.
Apparently it is important to copy /etc/krb5.keytab to /etc/eos.krb5.keytab and set it owner daemon:daemon. That part of the installation instructions needs to be updated.
I’m also unclear on where one gets a sss /etc/eos.keytab other than the testkeytab provided by package eos-keytab.
Where would I find documentation on sss? (Which I now understand clearly is different than sssd, used by IPA)

More discovery: Mabe this relates to the Heimdal text format, not MIT binary format (the default for IPA) HeimdalKerberos5library: File formats
That said, the eos.keytab doesn’t quite follow that format.

Hi Joseph,

You can find documentation on the XRootD sss protocol here:
https://xrootd.slac.stanford.edu/doc/dev49/sec_config.htm#_Toc517294117

Looking at the error message in your first post it looked like there was some permission problem:

Seckrb5: Unable to start sequence on the keytab file FILE:/etc/krb5.keytab; Permission denied

I see that you figured that one out.
The eos.keytab has nothing to do with krb5 but only related to the sss protocol. There is the xrdsssadmin tool that can help you inspect and generate sss keytab.

Cheers,
Elvin

That’s a very helpful pointer on the XrootD sss elements. There’s a namespace collision here (mentally) because FreeIPA uses a system called sssd
This direction on the deployment configuration directions (Develop — EOS CITRINE documentation) I misinterpreted to mean that the keys and key formats were the same but now I realize that’s totally not the case.

Then you need security keys on your machine:

cp /etc/krb5.keytab /etc/eos.krb5.keytab