Http TPC transfer failed whth frontend redirection

Site Administrators
1

Hi,
I set the EOS_NGINX_REDIRECT_EXTERNALLY=1 /etc/sysconfig/nginx, and try to execute HTTP TPC PUSH transfer(whth macaroon token) from one EOS instance to another one, and the transfer failed.
The error message show “failure: Send failed since rewinding of the data stream failed”.
Here’s the FST and MGM confiuration

# xrd.cf.fst
xrd.protocol XrdHttp:8001 /usr/lib64/libXrdHttp-4.so
http.exthandler EosFstHttp /usr/lib64/libEosFstHttp.so none
http.exthandler xrdtpc /opt/eos/xrootd/lib64/libXrdHttpTPC-4.so
http.trace all

http.cert /etc/grid-security/daemon/hostcert.pem
http.key /etc/grid-security/daemon/hostkey.pem
http.cadir /etc/grid-security/certificates

```
# xrd.cf.mgm
xrd.protocol XrdHttp:8000 /usr/lib64/libXrdHttp.so
http.trace all
http.cadir /etc/grid-security/certificates/
http.cert /etc/grid-security/daemon/hostcert.pem
http.key /etc/grid-security/daemon/hostkey.pem
http.gridmap /etc/grid-security/grid-mapfile
http.secxtractor /opt/eos/xrootd/lib64/libXrdVoms.so
http.exthandler xrdtpc /opt/eos/xrootd/lib64/libXrdHttpTPC.so
http.exthandler EosMgmHttp /usr/lib64/libEosMgmHttp.so eos::mgm::http::redirect-to-https=0
mgmofs.macaroonslib /opt/eos/xrootd/lib64/libXrdMacaroons.so
macaroons.secretkey /etc/eos.macaroon.secret
macaroons.trace all
all.sitename eostest

Is there something I’m missing for https tpc?

2

Hi Chiende,

Doing HTTP TPC transfers with macaroons is not supported through NGINX so you should not involve at all any Nginx server that you might have in front of your EOS instance - this is supported natively in EOS.

For the configuration bits, you should always point to either the Xrootd libraries installed on your system in /usr/lib64/ or the ones provided by the eos-xrootd package which are located in /opt/eos/xrootd/lib64 but you should not mix them - otherwise the config looks good to me.

For example on the fst side the xrd.protocol should point to this location:
xrd.protocol XrdHttp:8001 /opt/eos/xrootd/lib64/libXrdHttp-4.so

Can you give me more details about how you are trying to transfer the data? The command line that you use, and where exactly does this error message come from? Can you also explain a bit what is you setup? What eos and xrootd versions are you running?

Thanks,
Elvin

3

Hi,
I have two eos testbed instance with same configuration. I install fst, mgm and mq on one node.

Here’s the version of xrootd and eos

[root@f-dpmp28 ~]# rpm -qa | grep eos
eos-folly-2019.11.11.00-1.el7.cern.x86_64
eos-server-4.8.40-1.el7.cern.x86_64
eos-client-4.8.40-1.el7.cern.x86_64
eos-folly-deps-2019.11.11.00-1.el7.cern.x86_64
eos-fuse-core-4.8.40-1.el7.cern.x86_64
eos-fuse-sysv-4.8.40-1.el7.cern.x86_64
libmicrohttpd-0.9.38-eos.yves.el7.cern.x86_64
eos-testkeytab-4.8.40-1.el7.cern.x86_64
eos-xrootd-4.12.8-1.el7.cern.x86_64
eos-fuse-4.8.40-1.el7.cern.x86_64
eos-nginx-1.9.9-5.x86_64
eos-scitokens-debuginfo-1.2.0-1.el7.cern.x86_64
eos-protobuf3-3.5.1-5.el7.cern.eos.x86_64
eos-scitokens-1.2.0-1.el7.cern.x86_64
[root@f-dpmp28 ~]# rpm -qa | grep xrootd
xrootd-selinux-4.12.8-1.el7.noarch
xrootd-4.12.8-1.el7.x86_64
eos-xrootd-4.12.8-1.el7.cern.x86_64
xrootd-libs-4.12.8-1.el7.x86_64
xrootd-server-4.12.8-1.el7.x86_64
xrootd-server-libs-4.12.8-1.el7.x86_64
xrootd-client-libs-4.12.8-1.el7.x86_64

I use the following script to transfer data(by HTTPs TPC PUSH) :

export SRC=https://f-dpmp35.grid.sinica.edu.tw//eos/testarea/test.4101
export DST=https://f-dpmp28.grid.sinica.edu.tw//eos/testarea/test.4101
export TSRC=$(curl --silent --cert /root/cernproxy.pem --key /root/cernproxy.pem --cacert /root/cernproxy.pem --capath /etc/grid-security/certificates -X POST -H 'Content-Type: application/macaroon-request' -d '{"caveats": ["activity:DOWNLOAD"], "validity": "PT3000M"}' "$SRC" | jq -r '.macaroon')
export TDST=$(curl --silent --cert /root/cernproxy.pem --key /root/cernproxy.pem --cacert /root/cernproxy.pem --capath /etc/grid-security/certificates -X POST -H 'Content-Type: application/macaroon-request' -d '{"caveats": ["activity:UPLOAD,DELETE,LIST"], "validity": "PT3000M"}' "$DST" | jq -r '.macaroon')
curl --capath /etc/grid-security/certificates -L -X COPY -H 'Secure-Redirection: 1' -H 'X-No-Delegate: 1' -H 'Credentials: none' -H "Authorization: Bearer $TSRC" -H "TransferHeaderAuthorization: Bearer $TDST" -H "Destination: $DST" "$SRC"

and I get the message like this

Perf Marker
Timestamp: 1621490340
Stripe Index: 0
Stripe Bytes Transferred: 0
Total Stripe Count: 1
End
failure: Send failed since rewinding of the data stream failed(base)

The transfer will success when I disable the frontend redirection by setting EOS_NGINX_REDIRECT_EXTERNALLY=0 on the destination instance.

BUT when I try to transfer data from our production DPM instance, the transfer will success…

export SRC=https://f-dpm000.grid.sinica.edu.tw//dpm/grid.sinica.edu.tw/home/atlas/atlasscratchdisk/test.40046
export DST=https://f-dpmp28.grid.sinica.edu.tw//eos/testarea/test.4101
export TSRC=$(curl --silent --cert /root/cernproxy.pem --key /root/cernproxy.pem --cacert /root/cernproxy.pem --capath /etc/grid-security/certificates -X POST -H 'Content-Type: application/macaroon-request' -d '{"caveats": ["activity:DOWNLOAD"], "validity": "PT3000M"}' "$SRC" | jq -r '.macaroon')
export TDST=$(curl --silent --cert /root/cernproxy.pem --key /root/cernproxy.pem --cacert /root/cernproxy.pem --capath /etc/grid-security/certificates -X POST -H 'Content-Type: application/macaroon-request' -d '{"caveats": ["activity:UPLOAD,DELETE,LIST"], "validity": "PT3000M"}' "$DST" | jq -r '.macaroon')
curl --capath /etc/grid-security/certificates -L -X COPY -H 'Secure-Redirection: 1' -H 'X-No-Delegate: 1' -H 'Credentials: none' -H "Authorization: Bearer $TSRC" -H "TransferHeaderAuthorization: Bearer $TDST" -H "Destination: $DST" "$SRC"

success: Created

Could you please help us to solve this?

Thaks,
Chien-De

4

Hi Chien-De,

You need to drop the nginx if you want the HTTP TPC transfers to work properly. Namely you need to use the port 8000 when attempting the transfers eg. https://f-dpmp28.grid.sinica.edu.tw:8000/ rather then the default 443 where are I assume you have the nginx gw running.

Cheers,
Elvin

1 Like