Fail auth ( with x509 map ) on xrdhttp native

1

Dear all
I can not authenticate correctly on tcp 9000 with HTTPS protocol

'210817 11:33:53 26327 ?:353@grid11 sysXrdHttp: received dlen: 16
210817 11:33:53 26327 ?:353@grid11 sysXrdHttp: received dump: 22 03 01 02 00 01 00 01 -04 03 03 99 -113 -34 -16 00
210817 11:33:53 26327 ?:353@grid11 sysXrdHttp: This does not look like http at pos 0
210817 11:33:53 26327 ?:353@grid11 sysXrdHttp: This may look like https
210817 11:33:53 26327 ?:353@grid11 sysXrdHttp: Protocol matched. https: 1
210817 11:33:53 26327 ?:353@grid11 sysXrdHttp: Process. lp:0x7ff94b8952b8 reqstate: 0
210817 11:33:53 26327 ?:353@grid11 sysXrdHttp: Setting host: [::ffff:134.158.72.161]
210817 11:33:53 26327 ?:353@grid11 sysXrdHttp: Entering SSL_accept…
210817 11:33:53 26327 ?:353@grid11 sysXrdHttp: SSL_accept returned :1
210817 11:33:53 26327 cryptossl_X509::CertType: certificate has 3 extensions
210817 11:33:53 26327 cryptossl_X509::CertType: Found RFC 382{0,1}compliant proxyCertInfo extension
210817 11:33:53 26327 cryptossl_X509::CertType: certificate has 9 extensions
210817 11:33:53 26327 ?:353@grid11 sysXrdHttp: Subject name is : ‘/O=GRID-FR/C=FR/O=CNRS/OU=LAL/CN=Emmanouil Vamvakopoulos’
210817 11:33:53 26327 ?:353@grid11 sysXrdHttp: Extracting auth info.
210817 11:33:53 26327 ?:353@grid11 sysXrdHttp: Mapping name: ‘/O=GRID-FR/C=FR/O=CNRS/OU=LAL/CN=Emmanouil Vamvakopoulos’ → dte
210817 11:33:53 26327 sysXrdHttp: getDataOneShot BuffAvailable: 1048576 maxread: 1048576
210817 11:33:53 26327 sysXrdHttp: getDataOneShot sslavail: 1048576
210817 11:33:53 26327 sysXrdHttp: read 197 of 1048576 bytes
210817 11:33:53 26327 sysXrdHttp: rc:21 got hdr line: HEAD //eos HTTP/1.1
210817 11:33:53 26327 sysXrdHttp: Parsing first line: HEAD //eos HTTP/1.1
210817 11:33:53 26327 sysXrdHttp: rc:55 got hdr line: User-Agent: gfal2-util/1.5.3 gfal2/2.18.1 neon/0.0.29
210817 11:33:53 26327 sysXrdHttp: rc:14 got hdr line: Keep-Alive:
210817 11:33:53 26327 sysXrdHttp: rc:24 got hdr line: Connection: Keep-Alive
210817 11:33:53 26327 sysXrdHttp: rc:14 got hdr line: TE: trailers
210817 11:33:53 26327 sysXrdHttp: rc:32 got hdr line: Host: grid23.lal.in2p3.fr:9000
210817 11:33:53 26327 sysXrdHttp: rc:35 got hdr line: Accept: application/metalink4+xml
210817 11:33:53 26327 sysXrdHttp: rc:2 got hdr line:
210817 11:33:53 26327 sysXrdHttp: rc:2 detected header end.
210817 11:33:53 time=1629192833.548073 func=MatchesPath level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007ff9c39dc700 source=EosMgmHttpHandler:324 tident= sec=(null) uid=99 gid=99 name=- geo="" verb=HEAD path=/eos
210817 11:33:53 time=1629192833.548098 func=MatchesPath level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007ff9c39dc700 source=EosMgmHttpHandler:324 tident= sec=(null) uid=99 gid=99 name=- geo="" verb=HEAD path=/eos
210817 11:33:53 time=1629192833.548113 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007ff9c39dc700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“Accept” value=“application/metalink4+xml”
210817 11:33:53 time=1629192833.548126 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007ff9c39dc700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“Connection” value=“Keep-Alive”
210817 11:33:53 time=1629192833.548135 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007ff9c39dc700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“Host” value=“grid23.lal.in2p3.fr:9000”
210817 11:33:53 time=1629192833.548143 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007ff9c39dc700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“Keep-Alive” value=""
210817 11:33:53 time=1629192833.548151 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007ff9c39dc700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“TE” value=“trailers”
210817 11:33:53 time=1629192833.548159 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007ff9c39dc700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“User-Agent” value=“gfal2-util/1.5.3 gfal2/2.18.1 neon/0.0.29”
210817 11:33:53 time=1629192833.548167 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007ff9c39dc700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“xrd-http-fullresource” value="//eos"
210817 11:33:53 26327 ?:353@grid11 sys210817 11:33:53 time=1629192833.548175 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007ff9c39dc700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“xrd-http-prot” value=“https”
210817 11:33:53 time=1629192833.548182 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007ff9c39dc700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“xrd-http-query” value=""
XrdHttp: 210817 11:33:53 time=1629192833.548205 func=ProcessReq level=ERROR logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007ff9c39dc700 source=EosMgmHttpHandler:399 tident= sec=(null) uid=99 gid=99 name=- geo="" msg="(token) authorization failed" path="/eos"
210817 11:33:53 26327 sysXrdHttp: Sending 89 bytes
210817 11:33:53 26327 sysXrdHttp: Sending 26 bytes
210817 11:33:53 26327 sysXrdHttp: XrdHttpReq request ended.
210817 11:33:53 26327 ?:353@grid11 sysXrdHttp: Process is exiting rc:1
210817 11:33:53 1415 ?:353@grid11 sysXrdHttp: Process. lp:0x7ff94b8952b8 reqstate: 0
210817 11:33:53 1415 sysXrdHttp: getDataOneShot BuffAvailable: 1048576 maxread: 1048576
210817 11:33:53 1415 sysXrdHttp: getDataOneShot sslavail: 1048576
210817 11:33:53 1415 sysXrdHttp: Cleanup
210817 11:33:53 1415 XrdLink: Unable to send to ?:353@grid11; connection reset by peer
210817 11:33:53 1415 sysXrdHttp: SSL_shutdown failed ’

I run at

eos-server-4.8.40-1.el7.cern.x86_64
eos-xrootd-4.12.8-1.el7.cern.x86_64

in a quarkdb master/slave conf
and follow the mini for for HTTPS acces at HTTP(XrdHttp) and XRootD TPC with delegated credentials — EOS CITRINE documentation

eos vid ls look ok
gsi:"":gid => root
gsi:"":uid => root
https:"/dteam:":gid => dteam
https:"/dteam:":uid => dte
https:"":gid => root
https:"":uid => root
publicaccesslevel: => 20
sss:"":gid => root
sss:"":uid => root
sudoer => uids()
unix:"":gid => nobody
unix:"":uid => nobody
voms:"/dteam:":gid => dteam
voms:"/dteam:":uid => dte
voms:"/dteam:lcgadmin":gid => dteam
voms:"/dteam:lcgadmin":uid => dtes
voms:"/dteam:production":gid => dteam
voms:"/dteam:production":uid => dtep
voms:"/vo.grif.fr:":gid => dteam
voms:"/vo.grif.fr:":uid => dte

any idea ?
thank you in advance
best
e.v.

p.s.
on port TCP 443 with nginx and microhttp thread, the access works fine

from the logs I have some x509/SSL mapping to a local account
but this is not propagated correctly to vid
therefore the UID stays 99

2

Hi Emmanouil,

How did you enable https in the vid mapping? The output of your vid map looks a bit strange. Doing a simple eos vid enable https should insert the following entry:

sudo eos vid ls | grep https
https:"<pwd>":gid => root
https:"<pwd>":uid => root

With this mapping, your client should be correctly mapped to the entry corresponding to your DN that you hopefully have set in /etc/grid-security/grid-mapfile.

Cheers,
Elvin

3

Hello Elvin
i correct the vid rules for https
but still the authedication failed
thank you for your hint
best
ev

4

What version of eos are you running on the MGM? Can you paste the log lines from the MGM connected to you request?

Thanks,
Elvin

5

I run on this stable version

the log are in the begin of the thread
thanks
e.v.

6

new one aftet the vid correction

`10817 15:35:56 25980 sysXrdHttp: XrdHttpReq request ended.
210817 15:35:56 26083 ?:389@grid11 sysXrdHttp: received dlen: 16
210817 15:35:56 26083 ?:389@grid11 sysXrdHttp: received dump: 22 03 01 02 00 01 00 01 -04 03 03 -19 -52 -59 -117 00
210817 15:35:56 26083 ?:389@grid11 sysXrdHttp: This does not look like http at pos 0
210817 15:35:56 26083 ?:389@grid11 sysXrdHttp: This may look like https
210817 15:35:56 26083 ?:389@grid11 sysXrdHttp: Protocol matched. https: 1
210817 15:35:56 26083 ?:389@grid11 sysXrdHttp: Process. lp:0x7f9d8a489d78 reqstate: 0
210817 15:35:56 26083 ?:389@grid11 sysXrdHttp: Setting host: [::ffff:134.158.72.161]
210817 15:35:56 26083 ?:389@grid11 sysXrdHttp: Entering SSL_accept…
210817 15:35:56 26083 ?:389@grid11 sysXrdHttp: SSL_accept returned :1
210817 15:35:56 26083 cryptossl_X509::CertType: certificate has 3 extensions
210817 15:35:56 26083 cryptossl_X509::CertType: Found RFC 382{0,1}compliant proxyCertInfo extension
210817 15:35:56 26083 cryptossl_X509::CertType: certificate has 9 extensions
210817 15:35:56 26083 ?:389@grid11 sysXrdHttp: Subject name is : ‘/O=GRID-FR/C=FR/O=CNRS/OU=LAL/CN=Emmanouil Vamvakopoulos’
210817 15:35:56 26083 ?:389@grid11 sysXrdHttp: Extracting auth info.
210817 15:35:56 26083 ?:389@grid11 sysXrdHttp: Mapping name: ‘/O=GRID-FR/C=FR/O=CNRS/OU=LAL/CN=Emmanouil Vamvakopoulos’ → dte
210817 15:35:56 26083 sysXrdHttp: getDataOneShot BuffAvailable: 1048576 maxread: 1048576
210817 15:35:56 26083 sysXrdHttp: getDataOneShot sslavail: 1048576
210817 15:35:56 26083 sysXrdHttp: read 214 of 1048576 bytes
210817 15:35:56 26083 sysXrdHttp: rc:38 got hdr line: HEAD //eos/grif/proc/whoami HTTP/1.1

210817 15:35:56 26083 sysXrdHttp: Parsing first line: HEAD //eos/grif/proc/whoami HTTP/1.1

210817 15:35:56 26083 sysXrdHttp: rc:55 got hdr line: User-Agent: gfal2-util/1.5.3 gfal2/2.18.1 neon/0.0.29

210817 15:35:56 26083 sysXrdHttp: rc:14 got hdr line: Keep-Alive:

210817 15:35:56 26083 sysXrdHttp: rc:24 got hdr line: Connection: Keep-Alive

210817 15:35:56 26083 sysXrdHttp: rc:14 got hdr line: TE: trailers

210817 15:35:56 26083 sysXrdHttp: rc:32 got hdr line: Host: grid23.lal.in2p3.fr:9000

210817 15:35:56 26083 sysXrdHttp: rc:35 got hdr line: Accept: application/metalink4+xml

210817 15:35:56 26083 sysXrdHttp: rc:2 got hdr line:

210817 15:35:56 26083 sysXrdHttp: rc:2 detected header end.
210817 15:35:56 time=1629207356.372845 func=MatchesPath level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007f9dc01ec700 source=EosMgmHttpHandler:324 tident= sec=(null) uid=99 gid=99 name=- geo="" verb=HEAD path=/eos/grif/proc/whoami
210817 15:35:56 time=1629207356.372867 func=MatchesPath level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007f9dc01ec700 source=EosMgmHttpHandler:324 tident= sec=(null) uid=99 gid=99 name=- geo="" verb=HEAD path=/eos/grif/proc/whoami
210817 15:35:56 time=1629207356.372881 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007f9dc01ec700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“Accept” value=“application/metalink4+xml”
210817 15:35:56 time=1629207356.372892 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007f9dc01ec700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“Connection” value=“Keep-Alive”
210817 15:35:56 time=1629207356.372901 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007f9dc01ec700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“Host” value=“grid23.lal.in2p3.fr:9000”
210817 15:35:56 time=1629207356.372909 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007f9dc01ec700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“Keep-Alive” value=""
210817 15:35:56 time=1629207356.372917 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007f9dc01ec700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“TE” value=“trailers”
210817 15:35:56 time=1629207356.372926 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007f9dc01ec700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“User-Agent” value=“gfal2-util/1.5.3 gfal2/2.18.1 neon/0.0.29”
210817 15:35:56 time=1629207356.372934 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007f9dc01ec700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“xrd-http-fullresource” value="//eos/grif/proc/whoami"
210817 15:35:56 26083 ?:389@grid11210817 15:35:56 time=1629207356.372942 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007f9dc01ec700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“xrd-http-prot” value=“https”
sysXrdHttp: Sending resp: 403 header len:89
210817 15:35:56 time=1629207356.372950 func=ProcessReq level=INFO logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007f9dc01ec700 source=EosMgmHttpHandler:372 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“normalize hdr” key=“xrd-http-query” value=""
210817 15:35:56 26083 sys210817 15:35:56 time=1629207356.372972 func=ProcessReq level=ERROR logid=static… unit=mgm@grid23.lal.in2p3.fr:1094 tid=00007f9dc01ec700 source=EosMgmHttpHandler:399 tident= sec=(null) uid=99 gid=99 name=- geo="" msg="(token) authorization failed" path="/eos/grif/proc/whoami"
XrdHttp: Sending 89 bytes
210817 15:35:56 26083 sysXrdHttp: Sending 26 bytes
210817 15:35:56 26083 sysXrdHttp: XrdHttpReq request ended.
210817 15:35:56 26083 ?:389@grid11 sysXrdHttp: Process is exiting rc:1
210817 15:35:56 25980 ?:389@grid11 sysXrdHttp: Process. lp:0x7f9d8a489d78 reqstate: 0
210817 15:35:56 25980 sysXrdHttp: getDataOneShot BuffAvailable: 1048576 maxread: 1048576
210817 15:35:56 25980 sysXrdHttp: getDataOneShot sslavail: 1048576
210817 15:35:56 25980 sysXrdHttp: Cleanup
210817 15:35:56 25980 XrdLink: Unable to send to ?:389@grid11; connection reset by peer
210817 15:35:56 25980 sysXrdHttp: SSL_shutdown failed
210817 15:35:56 25980 sysXrdHttp: Reset
210817 15:35:56 25980 sysXrdHttp: XrdHttpReq request ended`

7

I have master/slave quarkdb conf with 2 MGM ( and MQ )

8

hello Elvin

if I switch off the following lines in HTTPS conf at MGMs

#mgmofs.macaroonslib /usr/lib64/libXrdMacaroons.so /opt/eos/lib64/libXrdAccSciTokens.so
#macaroons.secretkey /etc/eos.macaroon.secret
#macaroons.trace all
without the macaroos support

https authentication ( and upload) it works normal on port 9000 with https protocol

any idea ?

FYI
e.v.
PS
the istance version is 4.8.40

9

Hi Emmanouil,

Can you check if the following command works for you?

curl -L -v --capath /etc/grid-security/certificates --cert ~/.globus/usercert.pem --cacert ~/.globus/usercert.pem --key ~/.globus/userkey.pem https://esdss000.cern.ch:9000//eos/dev/proc/whoami -o /tmp/dump

Please replace the https endpoint with the name of your machine. This command works perfectly fine for me and returns the expected result. Can you also paste the command that you issue currently using gfal(?!) ?

Cheers,
Elvin

10

hello Elvin
the curl command that you propose
without the macaroons support, it works
with macarons, support does not work
FYI
e.v.

11

That’s a bit surprising. Are 100% you are running a version >= 4.8.40?
Can you paste again the MGM logs that you get when executing the curl command that fails?

Thanks,
Elvin

12

As an alternative can you install the 4.8.60 version from this repo and try again the curl command?
https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64/

Cheers,
Elvin

13

hello Elvin
the access on 9000 tcp failed due to the fact that the /etc/xrootd/scitokens.cfg was empty.
if I configure according to the miniguide (on both eos-server version 4.8.40 and 4.8.60) the port reply normaly.
thank you for your help
best
e.v.

14

Great! Glad to hear that!

Cheers,
Elvin