alexbaranov
(Aleksandr Baranov)
July 27, 2022, 2:02pm
1
Dear Experts,
I’m struggling with the setup of my sandbox EOS with configured scitokens. I proceeded with the descriptions at HTTP(XrdHttp) and XRootD TPC with delegated credentials — EOS CITRINE documentation 7, but when I try to get ot put file from the EOS get an error the following message in log MGM:
220727 16:29:59 time=1658928599.122233 func=ProcessReq level=INFO logid=static.............................. unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=EosMgmHttpHandler:406 tident= sec=(null)
uid=99 gid=99 name=- geo="" msg="(token) authorization done but no username found" client_prot=https
220727 16:29:59 time=1658928599.122502 func=IdMap level=INFO logid=static.............................. unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=Mapping:1074 tident= sec=(null)
uid=99 gid=99 name=- geo="" sec.prot=https sec.name="" sec.host="[::ffff:10.220.16.45]" sec.vorg="" sec.grps="" sec.role="" sec.info="" sec.app="http" sec.tident="http" vid.uid=99 vid.gid=99
220727 16:29:59 time=1658928599.122573 func=XrdHttpHandler level=INFO logid=static.............................. unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=HttpServer:301 tident= sec=(null)
uid=99 gid=99 name=- geo="" request=GET client-real-ip= client-real-host= vid.name=nobody vid.uid=nobody vid.gid=nobody vid.host=[::ffff:10.220.16.45] vid.dn= vid.tident=http
220727 16:29:59 time=1658928599.122623 func=HandleRequest level=INFO logid=static.............................. unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=HttpHandler:71 tident= sec=(null)
uid=99 gid=99 name=- geo="" header:accept => */*
20727 16:29:59 time=1658928599.122646 func=HandleRequest level=INFO logid=static.............................. unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=HttpHandler:71 tident= sec=(null)
uid=99 gid=99 name=- geo="" header:authorization => Bearer eyJraWQiOiJyc2ExIiwiYWxnIjoiUlMyNTYifQ.eyJ3bGNnLnZlciI6IjEuMCIsInN1YiI6IjMyNWIyYTFhLTcwNTMtNDlhNy1hZjAwLWZiZmQ0YzFlOWIwMSIsImF1ZCI6Imh0dHBzOlwvXC93bGNnLmNlcm4uY2hcL2p3dFwvdjFcL2F
ueSIsIm5iZiI6MTY1ODkyNjY2MSwic2NvcGUiOiJzdG9yYWdlLnJlYWQ6XC9lb3NcL3VzZXJcL3RcL3RlbGVjYXN0XC8gb3BlbmlkIG9mZmxpbmVfYWNjZXNzIHByb2ZpbGUgZW1haWwgd2xjZyB3bGNnLmdyb3VwcyIsImlzcyI6Imh0dHBzOlwvXC92bTIyMS0yMjEuamluci5ydSIsImV4cCI6MTY1ODkzMDI2MSwi
aWF0IjoxNjU4OTI2NjYxLCJqdGkiOiIwM2RjMzNjYi1iN2MyLTQ2MGYtOGY3OS0yYmJmY2RmMGIyYWMiLCJjbGllbnRfaWQiOiI3ZmU4MmFjMS0wMzVmLTRhYTgtOTg3NS02MmMxYTJhMjFhZDEiLCJ3bGNnLmdyb3VwcyI6WyJcL0pJTlIiXX0.SqHGAa4UV7fTnn_4YbCUzviBgKnf-g10YbpFugEFyi98T02wlIKeZ
2GaHCl6buG6k1CFmXgGXvBQ6s6JULqDNY-kWMC3zTvdNYZ_gqHjdImr7YzetbV693dcZJ1D3Kzk9VEPhiyLbwdAtgPZJZq5dgKZzzGbyZ8WhRzTq_TeuHo
220727 16:29:59 time=1658928599.122818 func=_readlink level=INFO logid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=Link:303 tident=<single-exec> sec=https uid=99 gid=99 name=nobody geo="ST::INST::LITVM" name=/eos/user/t/telecast/test
220727 16:29:59 time=1658928599.123240 func=IdMap level=INFO logid=static.............................. unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=Mapping:1074 tident= sec=(null) uid=99 gid=99 name=- geo="" sec.prot=https sec.name="nobody" sec.host="[::ffff:10.220.16.45]" sec.vorg="" sec.grps="" sec.role="" sec.info="" sec.app="http" sec.tident="http" vid.uid=99 vid.gid=99
220727 16:29:59 time=1658928599.123702 func=_access level=INFO logid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=Access:158 tident=<single-exec> sec=https uid=99 gid=99 name=nobody geo="ST::INST::LITVM" acl=0 r=0 w=0 wo=0 x=0 egroup=0 mutable=1
220727 16:29:59 time=1658928599.123753 func=Emsg level=ERROR logid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx unit=mgm@vm1-eos-mgm1.domain.com:1094 tid=00007f86975d1700 source=XrdMgmOfs:841 tident=<single-exec> sec= uid=0 gid=0 name= geo="" Unable to access /eos/user/t/telecast/test; Operation not permitted
The interesting thing is that the same configuration worked successfully on EOS4. The installed version of EOS is 5.0.27 and xrootd is 5.4.3.
Any suggestion is welcome!
Thanks,
Alex
esindril
(Elvin Alin Sindrilaru)
July 27, 2022, 2:31pm
2
Hi Alexandr,
The issue here is that your token is not mapped to any local identity therefore, the request is assigned to user nobody
i.e. uid=99. You can make use of a name_mapfile
functionality to map the client identity that is present in your token to a local username that is recognized by your EOS instance and in allowed to read the data at this location: /eos/user/t/telecast/test
.
You can find more info of how such a name_mapfile
would look like here, towards the end of the page:
https://github.com/xrootd/xrootd/tree/master/src/XrdSciTokens
You also have the option to define a default username that tokens from a certain IAM provider are mapped to in the /etc/xrootd/scitokens.cfg
file by using the default_user
directive.
Concerning EOS4 can you send me a similar snippet of the logs from the MGM when such a request is being made? Also what are the permissions on this path in both EOS4 and 5 instances:
eos ls -lrta /eos/user/t/telecast/test
Cheers,
Elvin
esindril
(Elvin Alin Sindrilaru)
July 27, 2022, 3:35pm
3
Hi Alexandr,
Can you paste the scitoken configuration that you use for EOS 4? Is it any different from the one that you use for EOS 5? I’m interested in the file /etc/xrootd/scitokens.cfg
.
Thanks,
Elvin
alexbaranov
(Aleksandr Baranov)
July 27, 2022, 3:38pm
4
Hello. Elvin,
To roughly check the authentication functionality, use the onmissing=allow parameter in scitokens.cfg, and default_user = telecast
scitokens.cfg
[Global]
#audience = https://wlcg.cern.ch/jwt/v1/any
onmissing = allow
[Issuer OSG-Connect]
issuer = {-URL of IAM-}
base_path = /eos/user/t/telecast
map_subject = false
default_user = telecast
#name_mapfile = /etc/xrootd/mapfile
EOS refuses to start with mapfile, I must have made a mistake in it, I’ll try to figure it out.
I assume that to use the mapfile it would be necessary to change the value to passthrough?
Thanks for your help!
Alex
esindril
(Elvin Alin Sindrilaru)
July 27, 2022, 3:57pm
6
Hi Alexandr,
The onmissing
configuration should make no difference in this case. The default_user
should have been applied in this case. Could you please paste the following directive that you have in your /etc/xrd.cf.mgm/
file: mgmofs.macaroonslib
?
Thanks,
Elvin
alexbaranov
(Aleksandr Baranov)
July 27, 2022, 6:54pm
7
Thanks for explaining
can I post here all the settings that I have set for MGM,FST relevant to scitokens?
xrd.cf.mgm
####SciTokens
sec.level all none
ofs.authorize
ofs.authlib [++] /opt/eos/xrootd/lib64/libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg
xrd.protocol XrdHttp:9000 /opt/eos/xrootd/lib64/libXrdHttp.so
http.cadir /etc/grid-security/certificates
http.cert /etc/grid-security/daemon/hostcert.pem
http.key /etc/grid-security/daemon/hostkey.pem
http.trace all debug
http.exthandler xrdtpc /opt/eos/xrootd/lib64/libXrdHttpTPC.so
http.exthandler EosMgmHttp libEosMgmHttp.so eos::mgm::http::redirect-to-https=0
mgmofs.macaroonslib /opt/eos/xrootd/lib64/libXrdMacaroons.so /opt/eos/xrootd/lib64/libXrdAccSciTokens.so
macaroons.secretkey /etc/eos.macaroon.secret
all.sitename vm-eos.domain.com
acc.authdb /opt/xrd/etc/Authfile
acc.audit deny grant
xrd.tlsca noverify
xrd.cf.fst
xrd.protocol XrdHttp:9001 libXrdHttp.so
http.exthandler EosFstHttp libEosFstHttp.so none
http.exthandler xrdtpc /usr/lib64/libXrdHttpTPC.so
http.cert /etc/grid-security/daemon/hostcert.pem
http.key /etc/grid-security/daemon/hostkey.pem
http.cafile /etc/grid-security/daemon/fullchain.pem
Thanks!
esindril
(Elvin Alin Sindrilaru)
July 28, 2022, 8:52am
8
Hi Alexandr,
I see you have the following directive in your configuration:
ofs.authorize
ofs.authlib [++] /opt/eos/xrootd/lib64/libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg
Do you actually stack some other authz plugin on the MGM? In principle, these two config directives you don’t need. Just as reference I attach one of our MGM config files for EOS 5.
xrootd.fslib libXrdEosMgm.so
xrootd.seclib libXrdSec.so
xrootd.async off nosf
xrootd.chksum adler32
xrootd.chksum adler32
xrootd.chksum adler32
xrd.sched mint 64 maxt 4096 idle 300
xrd.timeout idle 86400
all.export / nolock
all.role manager
oss.fdlimit * max
sec.protocol unix
sec.protocol sss -c /etc/eos.keytab -s /etc/eos-archive.keytab
sec.protocol krb5 /etc/krb5.keytab.eosatlas xrootd/eosatlas.cern.ch@CERN.CH
sec.protocol gsi -crl:1 -moninfo:1 -cert:/etc/grid-security/daemon/hostcert.pem -key:/etc/grid-security/daemon/hostkey.pem -gridmap:/etc/grid-security/grid-mapfile -d:1 -gmapopt:2
ofs.tpc redirect delegated eosatlassftp.cern.ch:1094
mgmofs.fs /
mgmofs.targetport 1095
mgmofs.authdir /var/eos/auth
mgmofs.trace all debug
mgmofs.broker root://localhost:1097//eos/
mgmofs.instance eosatlas
mgmofs.configdir /var/eos/config
mgmofs.metalog /var/eos/md
mgmofs.reportnamespace false
mgmofs.reportstore true
mgmofs.reportstorepath /var/eos/report
mgmofs.txdir /var/eos/tx
mgmofs.autoloadconfig default
mgmofs.autosaveconfig true
mgmofs.alias eosatlas.cern.ch
mgmofs.nslib /usr/lib64/libEosNsQuarkdb.so
mgmofs.qdbcluster eosatlas-qdb.cern.ch:7777
mgmofs.qdbpassword_file /etc/eos.keytab
mgmofs.auththreads 8
mgmofs.authport 15555
mgmofs.cfgtype quarkdb
sec.protbind localhost.localdomain sss unix
sec.protbind localhost sss unix
sec.protbind * only krb5 gsi sss unix
xrd.protocol XrdHttp:8444 libXrdHttp.so
http.trace all -debug
http.secxtractor libXrdVoms.so
http.exthandler EosMgmHttp libEosMgmHttp.so eos::mgm::http::redirect-to-https=0
http.exthandler xrdtpc libXrdHttpTPC.so
xrd.tls /etc/grid-security/daemon/hostcert.pem /etc/grid-security/daemon/hostkey.pem
xrd.tlsca certdir /etc/grid-security/certificates/
http.gridmap /etc/grid-security/grid-mapfile
mgmofs.macaroonslib libXrdMacaroons.so libXrdAccSciTokens.so
macaroons.secretkey /etc/eos.macaroon.secret
macaroons.trace all
all.sitename eosatlas
The xrd.tls
and xrd.tlsca
can be used as replacement for the http.cadir/cert/key
in EOS 5. But this is more of cosmetic change.
At this point, I don’t really understand why this does not work for you. Could you please restart the MGM daemon, do another GET request and send me (privately) all the MGM logs? You can use the following email: esindril at cern dot ch.
Thanks,
Elvin
alexbaranov
(Aleksandr Baranov)
July 28, 2022, 2:13pm
9
Thanks Elvin, based on your config, I try corrected some of the parameters concerning scitokens in the MGM config
And now it looks like this:
xrd.cf.mgm
XrdSecDEBUG=6
xrootd.fslib libXrdEosMgm.so
xrootd.seclib libXrdSec.so
xrootd.async off nosf
xrootd.chksum adler32
xrd.sched mint 8 maxt 256 idle 64
all.export / nolock
all.role manager
oss.fdlimit 16384 32768
sec.protocol unix
sec.protocol sss -c /etc/eos.keytab -s /etc/eos.keytab
sec.protocol krb5 /etc/krb5.keytab host/vm1-eos-mgm1.domain.com@domain.com
xrd.protocol XrdHttp:9000 libXrdHttp.so
xrd.tls /etc/grid-security/daemon/hostcert.pem /etc/grid-security/daemon/hostkey.pem
xrd.tlsca certdir /etc/grid-security/certificates
http.gridmap /etc/grid-security/grid-mapfile
http.secxtractor libXrdVoms.so
http.trace all -debug
http.exthandler xrdtpc libXrdHttpTPC.so
http.exthandler EosMgmHttp libEosMgmHttp.so eos::mgm::http::redirect-to-https=0
mgmofs.macaroonslib libXrdMacaroons.so libXrdAccSciTokens.so
macaroons.secretkey /etc/eos.macaroon.secret
macaroons.trace all
all.sitename vm-eos.domain.com
sec.protbind localhost.localdomain unix sss
sec.protbind localhost unix sss
sec.protbind * only sss krb5 unix
mgmofs.fs /
mgmofs.targetport 1095
mgmofs.trace all debug
mgmofs.broker root://vm-eos.domain.com:1097//eos/
mgmofs.instance eosdev
mgmofs.configdir /var/eos/config
mgmofs.metalog /var/eos/md
mgmofs.txdir /var/eos/tx
mgmofs.authdir /var/eos/auth
mgmofs.archivedir /var/eos/archive
mgmofs.reportstorepath /var/eos/report
mgmofs.autoloadconfig default
mgmofs.cfgtype quarkdb
mgmofs.alias vm-eos.domain.com
mgmofs.fstgw vm3-eos-fst0.domain.com:3001
mgmofs.nslib /usr/lib64/libEosNsQuarkdb.so
mgmofs.qdbcluster vm1-eos-db1.domain.com:7777 vm2-eos-db2.domain.com:7777 vm3-eos-db3.domain.com:7777
mgmofs.qdbpassword_file /etc/eos.keytab-qdb
mgmofs.centraldrain true
You may also need to know some additional information:
# eos vid ls
geotag:"default" => "ST::INST::LITVM"
https:"<pwd>":gid => root
https:"<pwd>":uid => root
krb5:"<pwd>":gid => root
krb5:"<pwd>":uid => root
publicaccesslevel: => 1024
sss:"<pwd>":gid => root
sss:"<pwd>":uid => root
sss:"daemon":gid => root
sss:"daemon":uid => root
sudoer => uids()
tident:"*@vm1-eos-mgm1.domain.com":gid => root
tident:"*@vm1-eos-mgm1.domain.com":uid => root
tident:"*@vm2-eos-mgm2.domain.com":gid => root
tident:"*@vm2-eos-mgm2.domain.com":uid => root
tident:"*@vm3-eos-fst0.domain.com":gid => root
tident:"*@vm3-eos-fst0.domain.com":uid => root
tident:"*@vm4-ui1.domain.com":gid => root
tident:"*@vm4-ui1.domain.com":uid => root
tident:"daemon@vm223-1.domain.com":gid => root
tident:"daemon@vm223-1.domain.com":uid => root
# eos attr ls /eos/user/t/telecast
sys.acl="g:1002:rwcmxq"
sys.eos.btime="1640276406.127239116"
sys.forced.blocksize="4k"
sys.forced.checksum="adler"
sys.forced.layout="replica"
sys.forced.nstripes="2"
sys.forced.space="default"
sys.mask="770"
sys.owner.auth="telecast"
But the problem is still.
I sent to your email logs MGM which collected with enable option “XRD_LOGLEVEL=Dump”
In these logs, I tried to document some my attempts to download and upload the file to EOS via curl
Thanks for your help!
esindril
(Elvin Alin Sindrilaru)
July 28, 2022, 2:31pm
10
Thanks, I replied privately!
alexbaranov
(Aleksandr Baranov)
July 28, 2022, 7:13pm
11
By changing the base_path to / (scitokens.cfg) authorization is working!
P.S. As I was able to understand by playing with scitokens, read or write access in EOS is managed through the scope in the token: storage.create or srorage.read, only (i.e. not via Authfile)
Thanks Elvin!