The logic of sss key operation

Dear community,

I would like to understand for myself how sss keys work.
This question is not much different from the one I asked before:User accessing via SSS authentication
I have an EOS installation (EOS_SERVER_VERSION=4.8.69) containing two MGMs and some FSTs:
Both MGM and FST machines have a keytab (/etc/eos.keytab) key containing one key:
for user daemon (service)
This is fine.

.
vm1-eos-mgm1.domain.com
     Number Len Date/Time Created Expires Keyname User & Group
     ------ --- --------- ------- -------- -------
          1 32 11/22/18 16:27:36 -------- eos daemon daemon
vm2-eos-mgm2.domain.com
     Number Len Date/Time Created Expires Keyname User & Group
     ------ --- --------- ------- -------- -------
          1 32 11/22/18 16:27:36 -------- eos daemon daemon
vm3-eos-fst0.domain.com
     Number Len Date/Time Created Expires Keyname User & Group
     ------ --- --------- ------- -------- -------
          1 32 11/22/18 16:27:36 -------- eos daemon daemon
...
vm10-eos-fst10.domain.com
     Number Len Date/Time Created Expires Keyname User & Group
     ------ --- --------- ------- -------- -------
          1 32 11/22/18 16:27:36 -------- eos daemon daemon
.
r--------. 1 daemon daemon 272 dec 10 11:10 //etc/eos.keytab

Then I want to add another key on both MGMs e.g. sss authorization for user eos_user.
On FST I did not add a user key.
I.e. now the following situation with the keys:

.
vm1-eos-mgm1.domain.com
     Number Len Date/Time Created Expires Keyname User & Group
     ------ --- --------- ------- -------- -------
          1 32 12/02/21 13:05:26 -------- eos eos_user eos_user
          1 32 11/22/18 16:27:36 -------- eos daemon daemon
vm2-eos-mgm2.domain.com
     Number Len Date/Time Created Expires Keyname User & Group
     ------ --- --------- ------- -------- -------
          1 32 12/02/21 13:05:26 -------- eos eos_user eos_user
          1 32 11/22/18 16:27:36 -------- eos daemon daemon
vm3-eos-fst0.domain.com
     Number Len Date/Time Created Expires Keyname User & Group
     ------ --- --------- ------- -------- -------
          1 32 11/22/18 16:27:36 -------- eos daemon daemon
...
vm10-eos-fst10.domain.com
     Number Len Date/Time Created Expires Keyname User & Group
     ------ --- --------- ------- -------- -------
          1 32 11/22/18 16:27:36 -------- eos daemon daemon
r--------. 1 daemon daemon 420 dec 14 13:16 //etc/eos.keytab

After a while, I mean as the key is read on all FST I see errors like:

211203 01:39:28 140870 XrootdXeq: User authentication failed; Decryption key not found.
211203 01:39:28 140870 XrootdXeq: daemon.2388:50@eos disc 0:00:00
.

Which says that there is now a key issue between MGM and FST.

Clearly if you add the user’s key to all FSTs the problem might go away, but in my opinion not logical if you draw the analogy with ssh keys.

Can you please explain why keys work this way?

Hi Aleksandr,

I think that in this particular setup the problem comes from the fact that both keys that you registered have the same name i.e. eos. Can you try out with keys having different names e.g eos and eos-cli?

Cheers,
Elvin

Dear Elvin,

Thank you for your response!
I generated a custom key with a different keyname.

     Number Len Date/Time Created Expires  Keyname User & Group
     ------ --- --------- ------- -------- -------
          1  32 10/20/21 03:09:51 -------- eos daemon daemon
          1  32 12/21/21 10:18:06 -------- eos-cli eos_user eos_user

It seems to work.
But I noticed another peculiarity in EOS behavior:
The file “eos.keytab” is not re-read by the eos@mgm process itself. I waited for 1 hour after adding the key. It is not enough to restart one process, the command “fs ls” shows that all the fs have “offline” status Therefore all eos@* processes need to be reloaded to work correctly. After restarting, the added key is read.
Is it ok?

Hi Aleksandr,

According to the documentation the refresh interval for the server keytab is 1 hour. Did you wait more than 60 minutes?
https://xrootd.slac.stanford.edu/doc/dev49/sec_config.htm#_Toc517294117
Also looking at the code this seems to be correct:
https://github.com/xrootd/xrootd/blob/master/src/XrdSecsss/XrdSecProtocolsss.cc#L937

If this is not the case, then I would recommend that you open a bug report on the XRootD tracker, as this functionality is not specific to EOS but comes from the base XRootD framework.

Cheers,
Elvin

I doubt that more time I waited. I’ll try the experiment with key again and take your advice if unsuccessful.

Thank you very much for your help!