Hi Elvin,
I updated to version 5.0.31, tested, now it works without error.
Thank you!
Hi Elvin,
It turned out that there is one problem in HTTPS.
Using:
{{{
gfal-copy --copy-mode push …
}}}
copying does not work even from local to local EOS.
I have now tested the versions:
{{{
dvl-eos-m02:~ # rpm -qa eos-* xrootd-* | sort
eos-client-5.1.1-1.el7.cern.x86_64
eos-folly-2019.11.11.00-1.el7.cern.x86_64
eos-folly-deps-2019.11.11.00-1.el7.cern.x86_64
eos-fusex-5.1.1-1.el7.cern.x86_64
eos-fusex-core-5.1.1-1.el7.cern.x86_64
eos-fusex-selinux-5.1.1-1.el7.cern.x86_64
eos-grpc-1.41.0-1.el7.x86_64
eos-grpc-devel-1.41.0-1.el7.x86_64
eos-libmicrohttpd-0.9.38-eos.el7.cern.x86_64
eos-librichacl-1.12-14.el7.cern.x86_64
eos-ns-inspect-5.1.1-1.el7.cern.x86_64
eos-protobuf3-3.17.3-1.el7.cern.eos.x86_64
eos-quarkdb-5.1.1-1.el7.cern.x86_64
eos-richacl-1.12-14.el7.cern.x86_64
eos-rocksdb-6.2.4-1.el7.cern.x86_64
eos-server-5.1.1-1.el7.cern.x86_64
eos-xrootd-5.5.1-1.el7.cern.x86_64
xrootd-client-libs-5.5.0-1.el7.x86_64
xrootd-libs-5.5.0-1.el7.x86_64
xrootd-scitokens-5.5.0-1.el7.x86_64
xrootd-server-5.5.0-1.el7.x86_64
xrootd-server-libs-5.5.0-1.el7.x86_64
xrootd-voms-5.5.0-1.el7.x86_64
}}}
We want to work without using /etc/grid-security/grid-mapfile, only with VOMS. But copying with “push” doesn’t work like that,
I get an error:
{{{
Copy failed (3rd push). Last attempt: [gfal_http_third_party_copy] Transfer failure: Remote side failed with status code 403
}}}
With “pull” everything works.
The xrootd protocol works with both “push” and “pull”.
When using /etc/grid-security/grid-mapfile, https works with “push” as well.
Looks like some settings are missing?
Hi Valeri,
Could you please show me the contents of your certificate including the VOMS group information that you are using when doing the transfer?
Also could you please paste the eos vid rules that you are using to enforce the VOMS mapping?
Thanks,
Elvin
Hi Elvin,
here is the proxy I’m using:
{{{
dvl-ui01:~ > voms-proxy-info --all
subject : /C=RU/O=RDIG/OU=users/OU=jinr.ru/CN=Valery Mitsyn/CN=2047955353
issuer : /C=RU/O=RDIG/OU=users/OU=jinr.ru/CN=Valery Mitsyn
identity : /C=RU/O=RDIG/OU=users/OU=jinr.ru/CN=Valery Mitsyn
type : RFC3820 compliant impersonation proxy
strength : 2048
path : /tmp/x509up_u8142
timeleft : 99:59:56
key usage : Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment, Key Agreement
=== VO cms extension information ===
VO : cms
subject : /C=RU/O=RDIG/OU=users/OU=jinr.ru/CN=Valery Mitsyn
issuer : /DC=ch/DC=cern/OU=computers/CN=lcg-voms2.cern.ch
attribute : /cms/Role=NULL/Capability=NULL
timeleft : 99:59:56
uri : lcg-voms2.cern.ch:15002
}}}
and related definitions for vid:
{{{
dvl-eos-m02:~ # eos vid ls | grep cms
voms:“/cms:”:gid => lcms
voms:“/cms:”:uid => cms001
}}}
Hi Valeri,
I tried configuring our pre-production instance to skip the grid-map file for HTTPS requests and this works as expected for me in the sense that the vid mapping is respected.
The only modification I have done in the /etc/xrd.cf.mgm configuration is to remove/comment the following line:
#http.gridmap /etc/grid-security/grid-mapfile
Then using the command below:
curl -L -v --capath /etc/grid-security/certificates --cert /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) https://eospps.cern.ch/eos/pps/opstest/egi/file1.dat --upload-file /etc/passwd
The upload was successful for me. Can you send me the full /etc/xrd.cf.mgm and the MGM logs that you have once you issue such a request?
Thanks,
Elvin
Hi Elvin,
just uploading files works without problems.
TPC in pull mode also works without problems.
Doesn’t work without authorization via grid-mapfile only TPC in push mode,
this mode only works with my certificate in grid-mapfile.
{{{
dvl-ui01:~ > gfal-copy --copy-mode pull
davs://se-wbdv.jinr-t1.ru:2880//pnfs/jinrt1.ru/data/cms/vvm-test-01/5GB-000
davs://dvl-eos.jinr.ru:8443//eos/tests/cms/5GB-020
Copying davs://se-wbdv.jinr-t1.ru:2880//pnfs/jinr-t1.ru/data/cms/vvm-test-01/5GB-000 [DONE] after 57s
dvl-ui01:~ >
dvl-ui01:~ > gfal-copy --copy-mode push
davs://se-wbdv.jinr-t1.ru:2880//pnfs/jinr-t1.ru/data/cms/vvm-test-01/5GB-000
davs://dvl-eos.jinr.ru:8443//eos/tests/cms/5GB-021
Copying davs://se-wbdv.jinr-t1.ru:2880//pnfs/jinr-t1.ru/data/cms/vvm-test-01/5GB-000 [FAILED] after 0s
gfal-copy error: 5 (Input/output error) - TRANSFER ERROR: Copy failed (3rd push). Last attempt: Transfer failure: rejected PUT: 403 FORBIDDEN\n
}}}
And to exclude dCache on the other side,
below TPC pull and push transmissions on the same EOS:
{{{
dvl-ui01:~ > gfal-copy -f --copy-mode pull
davs://dvl-eos.jinr.ru:8443//eos/tests/cms/5GB-000
davs://dvl-eos.jinr.ru:8443//eos/tests/cms/5GB-001
Copying davs://dvl-eos.jinr.ru:8443//eos/tests/cms/5GB-000 [DONE] after 45s
dvl-ui01:~ > gfal-copy -f --copy-mode push
davs://dvl-eos.jinr.ru:8443//eos/tests/cms/5GB-000
davs://dvl-eos.jinr.ru:8443//eos/tests/cms/5GB-001
Copying davs://dvl-eos.jinr.ru:8443//eos/tests/cms/5GB-000 [FAILED] after 0s
gfal-copy error: 5 (Input/output error) - TRANSFER ERROR: Copy failed (3rd push). Last attempt: Transfer failure: Remote side failed with status code 403
}}}
Hi Valeri,
Conceptually, a simple HTTP push and a HTTP TPC push with EOS as the destination is exactly the same thing. So you should be able to reproduce your problem by doing a simple HTTP push using the same credentials that you are supplying to the HTTP TPC operation.
Without some more logs or debug info from your MGM, there is not much I can infer.
Cheers,
Elvin
Hi Elvin,
I put logs and config in:
/afs/cern.ch/user/v/vmitsyn/public/dvl-eos.jinr.ru/etc
/afs/cern.ch/user/v/vmitsyn/public/dvl-eos.jinr.ru/logs
Hi Valeri,
Thanks for all the info. First of all, looking at your configuration you still have the gridmap file enabled for the HTTPS protocol. Please review one of my previous comments if you really want to disable the gridmap functionality for HTTPS.
Then, looking at the MGM logs it’s not clear to me which transfer actually failed. I see some attempts for the file /eos/tests/cms/5GB-031 but these are actually done using Bearer tokens, so at this point I am not sure exactly what you are trying to test …
I think it would be much more effective to start from the simple curl example above and confirm that the basic behavior matches your expectations in terms of VOMS attributes mapping and then you can gradually increase the complexity by involving gfal-copy or other storage endpoints for the TPC transfers.
Cheers,
Elvin
Hi Elvin,
gridmap file enabled, but my certificate is missing in it.
I believe that in this case authorization through a voms proxy will work, right?
Excerpts from the log in this case about copying:
{{{
dvl-ui01:~ > date ; gfal-copy -f --copy-mode push davs://dvl-eos.jinr.ru:8443//eos/tests/cms/5GB-000 davs://dvl-eos.jinr.ru:8443//eos/tests/cms/5GB-031
Wed Sep 21 16:44:19 MSK 2022
Copying davs://dvl-eos.jinr.ru:8443//eos/tests/cms/5GB-000 [FAILED] after 0s
gfal-copy error: 5 (Input/output error) - TRANSFER ERROR: Copy failed (3rd push). Last attempt: Transfer failure: Remote side failed with status code 403
}}}
And of course I checked the example with curl . The copy worked without error.
{{{
dvl-ui01:~ > curl -L -v --capath /etc/grid-security/certificates --cert /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) https://dvl-eos.jinr.ru:8443/eos/tests/cms/file2.dat --upload-file /etc/passwd > curl.log 2>&1
}}}
I uploaded curl.log to:
/afs/cern.ch/user/v/vmitsyn/public/dvl-eos.jinr.ru/logs
Hi Elvin,
as for the token, apparently it is generated during the authorization process by library in directive:
mgmgmofs.macaroonslib /usr/lib64/libXrdMacaroons.so /usr/lib64libXrdAccSciTokens.so
I didn’t generate the macaroon myself.
Hi Valeri,
The token requests that is in your logs actually comes form the gfal-copy command which “behind your back” is trying to get a token for the TPC transfer. I think the default behavior for gfal-copy is to try and use tokens for the TPC transfer over HTTPS. Futhermore, when talking over HTTPS, I think this is the only supported option. Therefore, the issue actually comes from the token which is presented to the EOS endpoint. Inspecting the macaroon token will indicate the identity which is passed with the token and which is used later on by EOS to decide on the authorization of the request.
You can find some useful resources for inspecting the macaroon token here:
https://eos-docs.web.cern.ch/configuration/http_tpc.html
Cheers,
Elvin
Hi Elvin,
thanks for the explanation on creating and using macaroon token. Now I more or less understand the details.
Unfortunately, the examples of using “curl” to get and use macaroon token do not work for me, both pull and push, with my DN in /etc/grid-security/grid-mapfile and without it.
However, gfal-copy works in pull and push modes when my DN is in /etc/grid-security/grid-mapfile.
But it doesn’t work only in push mode when the DN is missing. In push, however, gfal-copy with the “-f” flag deletes the file if it’s already in EOS, which seems to indicate that the generated macaroon token has some permissions on destination side but no WRITE.
Hi Elvin,
we updated our EOS to the latest version 5.19 and now everything works without the grid-mapfile.
gfal-copy works in pull and push modes with voms authorization.
Thank you!
Hi Valeri,
Glad to hear things are working as expected!
Thanks also to Emmanouil for pressing on this issue!
Cheers,
Elvin