Scitokens: authorization done but no username found

Dear Experts,

I’m struggling with the setup of my sandbox EOS with configured scitokens. I proceeded with the descriptions at HTTP(XrdHttp) and XRootD TPC with delegated credentials — EOS CITRINE documentation 7, but when I try to get ot put file from the EOS get an error the following message in log MGM:

220727 16:29:59 time=1658928599.122233 func=ProcessReq               level=INFO  logid=static.............................. unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=EosMgmHttpHandler:406          tident= sec=(null) 
uid=99 gid=99 name=- geo="" msg="(token) authorization done but no username found" client_prot=https
220727 16:29:59 time=1658928599.122502 func=IdMap                    level=INFO  logid=static.............................. unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=Mapping:1074                   tident= sec=(null) 
uid=99 gid=99 name=- geo="" sec.prot=https sec.name="" sec.host="[::ffff:10.220.16.45]" sec.vorg="" sec.grps="" sec.role="" sec.info="" sec.app="http" sec.tident="http" vid.uid=99 vid.gid=99
220727 16:29:59 time=1658928599.122573 func=XrdHttpHandler           level=INFO  logid=static.............................. unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=HttpServer:301                 tident= sec=(null) 
uid=99 gid=99 name=- geo="" request=GET client-real-ip= client-real-host= vid.name=nobody vid.uid=nobody vid.gid=nobody vid.host=[::ffff:10.220.16.45] vid.dn= vid.tident=http
220727 16:29:59 time=1658928599.122623 func=HandleRequest            level=INFO  logid=static.............................. unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=HttpHandler:71                 tident= sec=(null) 
uid=99 gid=99 name=- geo="" header:accept => */*
20727 16:29:59 time=1658928599.122646 func=HandleRequest            level=INFO  logid=static.............................. unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=HttpHandler:71                 tident= sec=(null) 
uid=99 gid=99 name=- geo="" header:authorization => Bearer eyJraWQiOiJyc2ExIiwiYWxnIjoiUlMyNTYifQ.eyJ3bGNnLnZlciI6IjEuMCIsInN1YiI6IjMyNWIyYTFhLTcwNTMtNDlhNy1hZjAwLWZiZmQ0YzFlOWIwMSIsImF1ZCI6Imh0dHBzOlwvXC93bGNnLmNlcm4uY2hcL2p3dFwvdjFcL2F
ueSIsIm5iZiI6MTY1ODkyNjY2MSwic2NvcGUiOiJzdG9yYWdlLnJlYWQ6XC9lb3NcL3VzZXJcL3RcL3RlbGVjYXN0XC8gb3BlbmlkIG9mZmxpbmVfYWNjZXNzIHByb2ZpbGUgZW1haWwgd2xjZyB3bGNnLmdyb3VwcyIsImlzcyI6Imh0dHBzOlwvXC92bTIyMS0yMjEuamluci5ydSIsImV4cCI6MTY1ODkzMDI2MSwi
aWF0IjoxNjU4OTI2NjYxLCJqdGkiOiIwM2RjMzNjYi1iN2MyLTQ2MGYtOGY3OS0yYmJmY2RmMGIyYWMiLCJjbGllbnRfaWQiOiI3ZmU4MmFjMS0wMzVmLTRhYTgtOTg3NS02MmMxYTJhMjFhZDEiLCJ3bGNnLmdyb3VwcyI6WyJcL0pJTlIiXX0.SqHGAa4UV7fTnn_4YbCUzviBgKnf-g10YbpFugEFyi98T02wlIKeZ
2GaHCl6buG6k1CFmXgGXvBQ6s6JULqDNY-kWMC3zTvdNYZ_gqHjdImr7YzetbV693dcZJ1D3Kzk9VEPhiyLbwdAtgPZJZq5dgKZzzGbyZ8WhRzTq_TeuHo
220727 16:29:59 time=1658928599.122818 func=_readlink                level=INFO  logid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=Link:303                       tident=<single-exec> sec=https uid=99 gid=99 name=nobody geo="ST::INST::LITVM" name=/eos/user/t/telecast/test
220727 16:29:59 time=1658928599.123240 func=IdMap                    level=INFO  logid=static.............................. unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=Mapping:1074                   tident= sec=(null) uid=99 gid=99 name=- geo="" sec.prot=https sec.name="nobody" sec.host="[::ffff:10.220.16.45]" sec.vorg="" sec.grps="" sec.role="" sec.info="" sec.app="http" sec.tident="http" vid.uid=99 vid.gid=99
220727 16:29:59 time=1658928599.123702 func=_access                  level=INFO  logid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=Access:158                     tident=<single-exec> sec=https uid=99 gid=99 name=nobody geo="ST::INST::LITVM" acl=0 r=0 w=0 wo=0 x=0 egroup=0 mutable=1
220727 16:29:59 time=1658928599.123753 func=Emsg                     level=ERROR logid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=XrdMgmOfs:841                  tident=<single-exec> sec=      uid=0 gid=0 name= geo="" Unable to access /eos/user/t/telecast/test; Operation not permitted

The interesting thing is that the same configuration worked successfully on EOS4. The installed version of EOS is 5.0.27 and xrootd is 5.4.3.

Any suggestion is welcome!

Thanks,
Alex

Hi Alexandr,

The issue here is that your token is not mapped to any local identity therefore, the request is assigned to user nobody i.e. uid=99. You can make use of a name_mapfile functionality to map the client identity that is present in your token to a local username that is recognized by your EOS instance and in allowed to read the data at this location: /eos/user/t/telecast/test.

You can find more info of how such a name_mapfile would look like here, towards the end of the page:
https://github.com/xrootd/xrootd/tree/master/src/XrdSciTokens

You also have the option to define a default username that tokens from a certain IAM provider are mapped to in the /etc/xrootd/scitokens.cfg file by using the default_user directive.

Concerning EOS4 can you send me a similar snippet of the logs from the MGM when such a request is being made? Also what are the permissions on this path in both EOS4 and 5 instances:
eos ls -lrta /eos/user/t/telecast/test

Cheers,
Elvin

Hi Alexandr,

Can you paste the scitoken configuration that you use for EOS 4? Is it any different from the one that you use for EOS 5? I’m interested in the file /etc/xrootd/scitokens.cfg.

Thanks,
Elvin

Hello. Elvin,

To roughly check the authentication functionality, use the onmissing=allow parameter in scitokens.cfg, and default_user = telecast

scitokens.cfg

[Global]
#audience = https://wlcg.cern.ch/jwt/v1/any
onmissing = allow

[Issuer OSG-Connect]
issuer = {-URL of IAM-}
base_path = /eos/user/t/telecast
map_subject = false
default_user = telecast
#name_mapfile = /etc/xrootd/mapfile

EOS refuses to start with mapfile, I must have made a mistake in it, I’ll try to figure it out.
I assume that to use the mapfile it would be necessary to change the value to passthrough?

Thanks for your help!
Alex

No, it’s the same

Hi Alexandr,

The onmissing configuration should make no difference in this case. The default_user should have been applied in this case. Could you please paste the following directive that you have in your /etc/xrd.cf.mgm/ file: mgmofs.macaroonslib?

Thanks,
Elvin

Thanks for explaining

can I post here all the settings that I have set for MGM,FST relevant to scitokens?

xrd.cf.mgm
####SciTokens                                                                                                                                                                                                                                
sec.level all none                                                                                                                                                                                                                           
                                                                                                                                                                                                                                             
ofs.authorize                                                                                                                                                                                                                                
ofs.authlib [++] /opt/eos/xrootd/lib64/libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg                                                                                                                                                
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
xrd.protocol XrdHttp:9000 /opt/eos/xrootd/lib64/libXrdHttp.so                                                                                                                                                                                
http.cadir /etc/grid-security/certificates                                                                                                                                                                                                   
http.cert /etc/grid-security/daemon/hostcert.pem                                                                                                                                                                                             
http.key /etc/grid-security/daemon/hostkey.pem                                                                                                                                                                                                                                                                                                                                                                                                
http.trace all debug                                                                                                                                                                                                                         
http.exthandler xrdtpc /opt/eos/xrootd/lib64/libXrdHttpTPC.so                                                                                                                                                                                
http.exthandler EosMgmHttp libEosMgmHttp.so eos::mgm::http::redirect-to-https=0                                                                                                                                                              
mgmofs.macaroonslib /opt/eos/xrootd/lib64/libXrdMacaroons.so /opt/eos/xrootd/lib64/libXrdAccSciTokens.so                                                                                                                                     
macaroons.secretkey /etc/eos.macaroon.secret                                                                                                                                                                                                 
all.sitename vm-eos.domain.com                                                                                                                                                                                                              
                                                                                                                                                                                                                                             
acc.authdb /opt/xrd/etc/Authfile                                                                                                                                                                                                                                                                                                                                                                                                                         
acc.audit deny grant                                                                                                                                                                                                                         
xrd.tlsca noverify
xrd.cf.fst
xrd.protocol XrdHttp:9001 libXrdHttp.so
http.exthandler EosFstHttp libEosFstHttp.so none
http.exthandler xrdtpc /usr/lib64/libXrdHttpTPC.so
http.cert /etc/grid-security/daemon/hostcert.pem
http.key /etc/grid-security/daemon/hostkey.pem
http.cafile /etc/grid-security/daemon/fullchain.pem

Thanks!

Hi Alexandr,

I see you have the following directive in your configuration:

ofs.authorize                                                                                                                                                                                                                                
ofs.authlib [++] /opt/eos/xrootd/lib64/libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg                                                                                                                                                
 

Do you actually stack some other authz plugin on the MGM? In principle, these two config directives you don’t need. Just as reference I attach one of our MGM config files for EOS 5.

xrootd.fslib  libXrdEosMgm.so
xrootd.seclib  libXrdSec.so
xrootd.async  off nosf
xrootd.chksum  adler32
xrootd.chksum  adler32
xrootd.chksum  adler32
xrd.sched  mint 64 maxt 4096 idle 300
xrd.timeout  idle 86400
all.export  / nolock
all.role  manager
oss.fdlimit  * max
sec.protocol  unix
sec.protocol  sss -c /etc/eos.keytab -s /etc/eos-archive.keytab
sec.protocol  krb5 /etc/krb5.keytab.eosatlas xrootd/eosatlas.cern.ch@CERN.CH
sec.protocol  gsi -crl:1 -moninfo:1 -cert:/etc/grid-security/daemon/hostcert.pem -key:/etc/grid-security/daemon/hostkey.pem -gridmap:/etc/grid-security/grid-mapfile -d:1 -gmapopt:2
ofs.tpc  redirect delegated eosatlassftp.cern.ch:1094
mgmofs.fs  /
mgmofs.targetport  1095
mgmofs.authdir  /var/eos/auth
mgmofs.trace  all debug
mgmofs.broker  root://localhost:1097//eos/
mgmofs.instance  eosatlas
mgmofs.configdir  /var/eos/config
mgmofs.metalog  /var/eos/md
mgmofs.reportnamespace  false
mgmofs.reportstore  true
mgmofs.reportstorepath  /var/eos/report
mgmofs.txdir  /var/eos/tx
mgmofs.autoloadconfig  default
mgmofs.autosaveconfig  true
mgmofs.alias  eosatlas.cern.ch
mgmofs.nslib  /usr/lib64/libEosNsQuarkdb.so
mgmofs.qdbcluster  eosatlas-qdb.cern.ch:7777
mgmofs.qdbpassword_file  /etc/eos.keytab
mgmofs.auththreads  8
mgmofs.authport  15555
mgmofs.cfgtype  quarkdb
sec.protbind  localhost.localdomain sss unix
sec.protbind  localhost sss unix
sec.protbind  * only krb5 gsi sss unix
xrd.protocol  XrdHttp:8444 libXrdHttp.so
http.trace  all -debug
http.secxtractor  libXrdVoms.so
http.exthandler  EosMgmHttp libEosMgmHttp.so eos::mgm::http::redirect-to-https=0
http.exthandler  xrdtpc libXrdHttpTPC.so
xrd.tls  /etc/grid-security/daemon/hostcert.pem /etc/grid-security/daemon/hostkey.pem
xrd.tlsca  certdir /etc/grid-security/certificates/
http.gridmap  /etc/grid-security/grid-mapfile
mgmofs.macaroonslib  libXrdMacaroons.so libXrdAccSciTokens.so
macaroons.secretkey  /etc/eos.macaroon.secret
macaroons.trace  all
all.sitename  eosatlas

The xrd.tls and xrd.tlsca can be used as replacement for the http.cadir/cert/key in EOS 5. But this is more of cosmetic change.

At this point, I don’t really understand why this does not work for you. Could you please restart the MGM daemon, do another GET request and send me (privately) all the MGM logs? You can use the following email: esindril at cern dot ch.

Thanks,
Elvin

Thanks Elvin, based on your config, I try corrected some of the parameters concerning scitokens in the MGM config
And now it looks like this:

xrd.cf.mgm
XrdSecDEBUG=6
xrootd.fslib libXrdEosMgm.so
xrootd.seclib libXrdSec.so
xrootd.async off nosf
xrootd.chksum adler32
xrd.sched mint 8 maxt 256 idle 64
all.export / nolock
all.role manager
oss.fdlimit 16384 32768
sec.protocol unix
sec.protocol sss -c /etc/eos.keytab -s /etc/eos.keytab
sec.protocol krb5 /etc/krb5.keytab  host/vm1-eos-mgm1.domain.com@domain.com
xrd.protocol XrdHttp:9000 libXrdHttp.so
xrd.tls /etc/grid-security/daemon/hostcert.pem /etc/grid-security/daemon/hostkey.pem
xrd.tlsca certdir /etc/grid-security/certificates
http.gridmap /etc/grid-security/grid-mapfile
http.secxtractor libXrdVoms.so
http.trace all -debug
http.exthandler xrdtpc libXrdHttpTPC.so
http.exthandler EosMgmHttp libEosMgmHttp.so eos::mgm::http::redirect-to-https=0
mgmofs.macaroonslib libXrdMacaroons.so libXrdAccSciTokens.so
macaroons.secretkey /etc/eos.macaroon.secret
macaroons.trace  all
all.sitename vm-eos.domain.com
sec.protbind localhost.localdomain unix sss
sec.protbind localhost unix sss
sec.protbind * only sss krb5 unix
mgmofs.fs /
mgmofs.targetport 1095
mgmofs.trace all debug
mgmofs.broker root://vm-eos.domain.com:1097//eos/
mgmofs.instance eosdev
mgmofs.configdir /var/eos/config
mgmofs.metalog /var/eos/md
mgmofs.txdir /var/eos/tx
mgmofs.authdir /var/eos/auth
mgmofs.archivedir /var/eos/archive
mgmofs.reportstorepath /var/eos/report
mgmofs.autoloadconfig default
mgmofs.cfgtype quarkdb
mgmofs.alias vm-eos.domain.com
mgmofs.fstgw vm3-eos-fst0.domain.com:3001
mgmofs.nslib /usr/lib64/libEosNsQuarkdb.so
mgmofs.qdbcluster vm1-eos-db1.domain.com:7777 vm2-eos-db2.domain.com:7777 vm3-eos-db3.domain.com:7777
mgmofs.qdbpassword_file /etc/eos.keytab-qdb
mgmofs.centraldrain true

You may also need to know some additional information:

# eos vid ls
geotag:"default" => "ST::INST::LITVM"
https:"<pwd>":gid => root
https:"<pwd>":uid => root
krb5:"<pwd>":gid => root
krb5:"<pwd>":uid => root
publicaccesslevel: => 1024
sss:"<pwd>":gid => root
sss:"<pwd>":uid => root
sss:"daemon":gid => root
sss:"daemon":uid => root
sudoer                 => uids()
tident:"*@vm1-eos-mgm1.domain.com":gid => root
tident:"*@vm1-eos-mgm1.domain.com":uid => root
tident:"*@vm2-eos-mgm2.domain.com":gid => root
tident:"*@vm2-eos-mgm2.domain.com":uid => root
tident:"*@vm3-eos-fst0.domain.com":gid => root
tident:"*@vm3-eos-fst0.domain.com":uid => root
tident:"*@vm4-ui1.domain.com":gid => root
tident:"*@vm4-ui1.domain.com":uid => root
tident:"daemon@vm223-1.domain.com":gid => root
tident:"daemon@vm223-1.domain.com":uid => root
# eos attr ls /eos/user/t/telecast
sys.acl="g:1002:rwcmxq"
sys.eos.btime="1640276406.127239116"
sys.forced.blocksize="4k"
sys.forced.checksum="adler"
sys.forced.layout="replica"
sys.forced.nstripes="2"
sys.forced.space="default"
sys.mask="770"
sys.owner.auth="telecast"

But the problem is still.
I sent to your email logs MGM which collected with enable option “XRD_LOGLEVEL=Dump”
In these logs, I tried to document some my attempts to download and upload the file to EOS via curl

Thanks for your help!

Thanks, I replied privately!

By changing the base_path to / (scitokens.cfg) authorization is working!
P.S. As I was able to understand by playing with scitokens, read or write access in EOS is managed through the scope in the token: storage.create or srorage.read, only (i.e. not via Authfile)

Thanks Elvin!