EOS 5.5.0 MGM with WLCG token support fails to start

1

Hello,

I have upgraded one of our preprod instances to EOS 5.5.0 and it works fine with current auth config (x509/GSI). When I add to /etc/xrd.cf.mgm all the directives I think I need for WLCG token authentication the MGM service fails to start with the following message:

60909 12:06:53 1446208 sysEosMgmHttpHandler: Unable to Failed config of EosMgmHttpHandler; invalid argument
------ HTTP protocol initialization failed.
260909 12:06:53 1446208 XrdProtocol: Protocol XrdHttp could not be loaded
------ xrootd mgm@antares-eos94.scd.rl.ac.uk:-1 initialization failed.

I am not quite sure what exactly I am missing. Can you please help?

Thanks,

George

2

Hi George,

all the directives I think I need for WLCG token authentication

Which ones?

Our v5.5.1 MGM has this and it seems to work:

# grep -i http /etc/xrd.cf.mgm
##### Setup HTTP
http.exthandler EosMgmHttp libEosMgmHttp.so eos::mgm::http::redirect-to-https=0
http.exthandler xrdtpc libXrdHttpTPC.so
#http.gridmap /etc/grid-security/grid-mapfile
http.secxtractor libXrdVoms.so
http.trace all
xrd.protocol XrdHttp:8443 libXrdHttp.so

However the token-related config is a separate thing, we have this:

[root@eos-mgm-1 /]# grep -i sci /etc/xrd.cf.mgm
mgmofs.macaroonslib libXrdMacaroons.so libXrdAccSciTokens.so

[root@eos-mgm-1 /]# cat /etc/xrootd/scitokens.cfg 
[Global]
# If a token is not present or does not authorize the requested action, invoke the next configured authorization plugin.
#onmissing = passthrough
audience = https://wlcg.cern.ch/jwt/v1/any, https://eos-mgm.wlcg.uvic.ca:8443, roots://eos-mgm.wlcg.uvic.ca:1094, eos-mgm.wlcg.uvic.ca

[Issuer ATLAS]
issuer = https://atlas-auth.cern.ch/
base_path = /eos/wlcg.uvic.ca/data/atlas
map_subject = False
name_mapfile = /etc/eos.config/scitokens.map
default_user = nobody
authorization_strategy = capability

[Issuer DTeam]
issuer = https://dteam-auth.cern.ch/
base_path = /eos/wlcg.uvic.ca/data/dteam
map_subject = False
default_user = dteam

[root@eos-mgm-1 /]# cat /etc/eos.config/scitokens.map 
[
  {"path": "/atlasscratchdisk/",    "result": "atlas", "comment": "Owner of ATLASSCRATCHDISK"},
  {"path": "/atlasdatadisk/",       "result": "atprd", "comment": "Owner of ATLASDATADISK"},
  {"path": "/atlaslocalgroupdisk/", "result": "atcan", "comment": "Owner of ATLASLOCALGROUPDISK"}
]

Previously I tried to figure that out here: WLCG Token AuthN/Z for EOS but didn’t get any answers, anyway that is what I came up with.

Thanks.

3

Thanks Ryan.

We have the following directives for token support

sec.protocol ztn
sec.protbind * only ztn gsi unix
mgmofs.authlib ++ libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg
mgmofs.authlib ++ libXrdMacaroons.so

xrootd.tls capable all
http.header2cgi Authorization authz

xrd.tls /etc/grid-security/daemon/hostcert.pem /etc/grid-security/daemon/hostkey.pem
xrd.tlsca certdir /etc/grid-security/certificates/
scitokens.trace all

and the http block has not changed

if exec xrootd
  xrd.protocol XrdHttp:%5$s libXrdHttp.so
  http.cadir /etc/grid-security/certificates/
  http.cert /etc/grid-security/xrootd/hostcert.pem
  http.key /etc/grid-security/xrootd/hostkey.pem
  http.gridmap /etc/grid-security/http-grid-mapfile
  http.trace all
  http.exthandler xrdtpc libXrdHttpTPC.so
  http.exthandler EosMgmHttp libEosMgmHttp.so eos::mgm::http::redirect-to-https=0
  mgmofs.macaroonslib libXrdMacaroons.so
  macaroons.secretkey /etc/xrootd/macaroon-secret
  macaroons.trace all
fi

We are running 5.5.0 though (instead 5.5.1). I wonder if this makes any difference…

Best,

George

4

Sadly, the upgrade to 5.5.1 did not make a difference, The error remains.

5

Hi @georgep ,

This is what we have for the token authentication (xrd.cf.mgm):

mgmofs.macaroonslib libXrdMacaroons.so libXrdAccSciTokens.so
macaroons.secretkey /etc/eos.macaroon.secret
macaroons.trace all

we do not use mgmofs.authlib.

Can you please try?

6

Hi @ccaffy

It works now - many thanks for this!!

For the record, I used mgmofs.authlib because of what is mentioned here xrootd/src/XrdSciTokens at master · xrootd/xrootd · GitHub

7

How do you pass the “SciTokens Configuration File” (/etc/xrootd/scitokens.cfg) to the mgm config?

8

It is picked automatically by the plugin, we do not configure that path :slight_smile:

9

We get an ztn auth error when we try to stat a file via xrootd (we verified that the token is correct)

260924 16:11:45 2556400 XrootdXeq: User authentication failed; ztn required plugin (libXrdAccSciTokens.so) has not been loaded!

I dont understand why EOS thinks the library is not loaded only when an access request is made,

10

On the MGM process, can you do a lsof and tell me if indeed libXrdAccSciTokens.so is not loaded?

11

Thanks, it looks loaded

[root@antares-eos14 ~]# ps aux | grep mgm
daemon   2555782  1.8  0.4 19325360 839528 ?     SLsl 16:11   0:47 /opt/eos/xrootd/bin/xrootd -n mgm -c /etc/xrd.cf.mgm -l /var/log/eos/xrdlog.mgm -Rdaemon
root     2565017  0.0  0.0   6424  2280 pts/0    S+   16:53   0:00 grep --color=auto mgm
[root@antares-eos14 ~]#
[root@antares-eos14 ~]# lsof -p 2555782  | grep libXrdAccSci
xrootd  2555782 daemon  mem       REG               65,5    136872 201699570 /opt/eos/xrootd/lib64/libXrdAccSciTokens-6.so
[root@antares-eos14 ~]#
12

Thanks, could you please tell me what eos-xrootd version you are running `rpm -qa | grep eos-xrootd` ?

13

actually, maybe the problem is somewhere else. It would be great that you have a look at the startup logs. When you start the MGM, xrootd will print lots of logs from the config and what it configures. The answer maybe in those lines. If you manage to send me those logs, I can have a deeper look.

Thanks

14

Thanks for this.

I am running eos-xrootd 6.1.1-1

root@antares-eos14 ~]# rpm -qa | grep eos-xrootd
eos-xrootd-6.1.1-1.el9.x86_64

I put the MGM log lines that follow a restart of the service in http://www-public.gridpp.rl.ac.uk/tape_accounting/antares-eos14-xrdlog.mgm if you want to have look. I cannot see libXrdAccSciTokens being present in these or any other log lines.

15

Hi George,

OK can you please also share the current MGM configuration file?

I think the macaroonslib may be missing something…

Thanks!

16

Many thanks, please find it here: http://www-public.gridpp.rl.ac.uk/tape_accounting/xrd.cf.mgm

17

Hi George,

Thanks, it looks like this block does not contain the libXrdAccSciToken:

if exec xrootd
  xrd.protocol XrdHttp:9000 libXrdHttp.so
  http.cadir /etc/grid-security/certificates/
  http.cert /etc/grid-security/xrootd/hostcert.pem
  http.key /etc/grid-security/xrootd/hostkey.pem
  http.gridmap /etc/grid-security/http-grid-mapfile
  http.trace all
  http.exthandler xrdtpc libXrdHttpTPC.so
  http.exthandler EosMgmHttp libEosMgmHttp.so eos::mgm::http::redirect-to-https=0
  mgmofs.macaroonslib libXrdMacaroons.so
  macaroons.secretkey /etc/xrootd/macaroon-secret
  macaroons.trace all
fi

Try to add it and tell me what you get? Edit the line “mgmofs.macaroonslib”: mgmofs.macaroonslib libXrdMacaroons.so libXrdAccSciTokens.so

Thanks

18

Many thanks, adding the directive in the block results in the the loading of the library.

I think I missing something more (a vid mapping probably) because it looks that EOS is mapping me to the nobody user

261007 14:11:28 2198072 scitokens_Access: Grant authorization based on scopes for operation=stat, path=/eos/antaresdev/prod/dteam/random_900MB
261007 14:11:28 2198072 scitokens_Access: Request username
261007 14:11:28 time=1791378688.259928 func=IdMap                    level=INFO  logid=static.............................. unit=mgm@antares-eos14.scd.rl.ac.uk:1094 tid=00007f74aa7fc640 source=Mapping:1196                   tident= sec=(null) uid=0 gid=0 name=- geo="" xt="" ob="" sec.prot=ztn sec.name="d53473c0-abbd-402f-8ccb-20c3de9bdc09" sec.host="lcgui05.gridpp.rl.ac.uk" sec.vorg="" sec.grps="" sec.role="" sec.info="" sec.app="" sec.tident="georgep.1640850:400@lcgui05.gridpp.rl.ac.uk" vid.uid=65534 vid.gid=65534 sudo=0 gateway=0
261007 14:11:28 time=1791378688.259996 func=_stat                    level=ERROR logid=static.............................. unit=mgm@antares-eos14.scd.rl.ac.uk:1094 tid=00007f74aa7fc640 source=Stat.:193                      tident= sec=(null) uid=0 gid=0 name=- geo="" xt="" ob="" vid.uid=65534

261007 14:11:28 time=1791378688.260007 func=Emsg                     level=ERROR logid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx unit=mgm@antares-eos14.scd.rl.ac.uk:1094 tid=00007f74aa7fc640 source=XrdMgmOfs:1148                 tident=<single-exec> sec=      uid=0 gid=0 name= geo="" xt="" ob="" msg="Unable to access - public access level restriction: Permission denied" path="/eos/antaresdev/prod/dteam/random_900MB"

I have already done

eos vid tokensudo 3
19

Have you done in the past:

`eos vid enable ztn`

20

Yes, I have enabled ztn.

Also, tried to revert eos vid token sudo back to the default 0 (always)