TPC (Third Party transfer) stall and Permission Error

Dear All,

It’s Prasun from Kolkata.

From yesterday we are going to move data from Kolkata::SE to Kolkata::EOS2 which was trigger by EOS manager. But till the date, no data are copied. The value of pending files are same as yesterday. I hear that moving of those data are based on Third Party Copy (TPC). As per my understand, we have already enable TPC variable on FST i.e.
EOS_FST_NO_SSS_ENFORCEMENT=1 (in eos_env) and ofs.tpc pgm /usr/bin/xrdcp (in xrd.cf.fst).

Currently EOS version is EOS 4.4.23 (CERN) and XROOTD version is v4.8.4.

The output of eos -b io stat -x are below:-
###########################
[root@eos-mgm ~]# eos -b io stat -x
┌───┬────────────────────────┬────────┬────────┬────────┬────────┐
│io │ application│ 1min│ 5min│ 1h│ 24h│
└───┴────────────────────────┴────────┴────────┴────────┴────────┘
out other 1.90 G 20.45 G 624.52 G 38.76 T
out tpc 0 0 0 0
in transfer-3rd 171.99 K 2.99 M 189.25 M 2.44 G
in other 2.68 M 16.45 M 1.46 G 57.75 G
###########################

And permissions are:-
[root@eos-mgm ~]# eos -b vid ls
https:"":gid => root
https:"":uid => root
https:"":uid => root
krb5:"":gid => root
krb5:"":uid => root
sss:"":gid => root
sss:"":uid => root
sudoer => uids()
tident:"@eos-mgm":gid => root
tident:"
@eos-mgm":uid => root
unix:"":gid => 1395
unix:"":uid => 10367
[root@eos-mgm ~]# ls -l /eos/alicekolkata/
total 0
drwxr-x— 1 10367 1395 0 May 14 18:59 grid
drwxr-xr-x 1 root root 0 Jan 1 1970 proc
drwxrwxrwx 1 root root 0 Jul 30 18:58 test
[root@eos-mgm ~]#
###################

When we tried to copy file from EOS to local disk, it’s failed with: [ERROR] Operation not supported.
Details is below:-
[root@eos-mgm ~]# xrdcp -f -d 3 --tpc only root://eoskolkata.tier2-kol.res.in:1094//eos/alicekolkata/grid/00/65278/ef1babe8-6f56-11e9-afbc-1318731e806f /tmp/.
[2019-08-23 18:23:57.293499 +0530][Dump ][App ] Chunk size: 16777216, parallel chunks 4, streams: 1
[2019-08-23 18:23:57.293809 +0530][Dump ][App ] Processing source entry: root://eoskolkata.tier2-kol.res.in:1094//eos/alicekolkata/grid/00/65278/ef1babe8-6f56-11e9-afbc-1318731e806f, type xroot, target file: file:///tmp/.
[2019-08-23 18:23:57.293934 +0530][Dump ][Utility ] URL: root://eoskolkata.tier2-kol.res.in:1094//eos/alicekolkata/grid/00/65278/ef1babe8-6f56-11e9-afbc-1318731e806f
[2019-08-23 18:23:57.293934 +0530][Dump ][Utility ] Protocol: root
[2019-08-23 18:23:57.293934 +0530][Dump ][Utility ] User Name:
[2019-08-23 18:23:57.293934 +0530][Dump ][Utility ] Password:
[2019-08-23 18:23:57.293934 +0530][Dump ][Utility ] Host Name: eoskolkata.tier2-kol.res.in
[2019-08-23 18:23:57.293934 +0530][Dump ][Utility ] Port: 1094
[2019-08-23 18:23:57.293934 +0530][Dump ][Utility ] Path: /eos/alicekolkata/grid/00/65278/ef1babe8-6f56-11e9-afbc-1318731e806f
[2019-08-23 18:23:57.294329 +0530][Dump ][Utility ] Adding job with properties: ‘checkSumMode’ = ‘none’, ‘checkSumPreset’ = ‘’, ‘checkSumType’ = ‘’, ‘chunkSize’ = ‘16777216’, ‘coerce’ = ‘0’, ‘dynamicSource’ = ‘0’, ‘force’ = ‘1’, ‘initTimeout’ = ‘600’, ‘makeDir’ = ‘0’, ‘parallelChunks’ = ‘4’, ‘posc’ = ‘0’, ‘source’ = ‘root://eoskolkata.tier2-kol.res.in:1094//eos/alicekolkata/grid/00/65278/ef1babe8-6f56-11e9-afbc-1318731e806f’, ‘target’ = ‘file:///tmp/.//ef1babe8-6f56-11e9-afbc-1318731e806f’, ‘thirdParty’ = ‘only’, ‘tpcTimeout’ = ‘1800’, ‘xcp’ = ‘0’, ‘xcpBlockSize’ = ‘134217728’, ‘zipArchive’ = ‘0’
[2019-08-23 18:23:57.294428 +0530][Debug ][Utility ] CopyProcess: 2 jobs to prepare
[2019-08-23 18:23:57.294494 +0530][Dump ][Utility ] URL: root://eoskolkata.tier2-kol.res.in:1094//eos/alicekolkata/grid/00/65278/ef1babe8-6f56-11e9-afbc-1318731e806f
[2019-08-23 18:23:57.294494 +0530][Dump ][Utility ] Protocol: root
[2019-08-23 18:23:57.294494 +0530][Dump ][Utility ] User Name:
[2019-08-23 18:23:57.294494 +0530][Dump ][Utility ] Password:
[2019-08-23 18:23:57.294494 +0530][Dump ][Utility ] Host Name: eoskolkata.tier2-kol.res.in
[2019-08-23 18:23:57.294494 +0530][Dump ][Utility ] Port: 1094
[2019-08-23 18:23:57.294494 +0530][Dump ][Utility ] Path: /eos/alicekolkata/grid/00/65278/ef1babe8-6f56-11e9-afbc-1318731e806f
[2019-08-23 18:23:57.294649 +0530][Dump ][Utility ] URL: file:///tmp/.//ef1babe8-6f56-11e9-afbc-1318731e806f
[2019-08-23 18:23:57.294649 +0530][Dump ][Utility ] Protocol: file
[2019-08-23 18:23:57.294649 +0530][Dump ][Utility ] User Name:
[2019-08-23 18:23:57.294649 +0530][Dump ][Utility ] Password:
[2019-08-23 18:23:57.294649 +0530][Dump ][Utility ] Host Name: localhost
[2019-08-23 18:23:57.294649 +0530][Dump ][Utility ] Port: 1094
[2019-08-23 18:23:57.294649 +0530][Dump ][Utility ] Path: /tmp/.//ef1babe8-6f56-11e9-afbc-1318731e806f
[2019-08-23 18:23:57.294810 +0530][Dump ][Utility ] URL: root://eoskolkata.tier2-kol.res.in:1094//eos/alicekolkata/grid/00/65278/ef1babe8-6f56-11e9-afbc-1318731e806f
[2019-08-23 18:23:57.294810 +0530][Dump ][Utility ] Protocol: root
[2019-08-23 18:23:57.294810 +0530][Dump ][Utility ] User Name:
[2019-08-23 18:23:57.294810 +0530][Dump ][Utility ] Password:
[2019-08-23 18:23:57.294810 +0530][Dump ][Utility ] Host Name: eoskolkata.tier2-kol.res.in
[2019-08-23 18:23:57.294810 +0530][Dump ][Utility ] Port: 1094
[2019-08-23 18:23:57.294810 +0530][Dump ][Utility ] Path: /eos/alicekolkata/grid/00/65278/ef1babe8-6f56-11e9-afbc-1318731e806f
[2019-08-23 18:23:57.294961 +0530][Dump ][Utility ] URL: file:///tmp/.//ef1babe8-6f56-11e9-afbc-1318731e806f
[2019-08-23 18:23:57.294961 +0530][Dump ][Utility ] Protocol: file
[2019-08-23 18:23:57.294961 +0530][Dump ][Utility ] User Name:
[2019-08-23 18:23:57.294961 +0530][Dump ][Utility ] Password:
[2019-08-23 18:23:57.294961 +0530][Dump ][Utility ] Host Name: localhost
[2019-08-23 18:23:57.294961 +0530][Dump ][Utility ] Port: 1094
[2019-08-23 18:23:57.294961 +0530][Dump ][Utility ] Path: /tmp/.//ef1babe8-6f56-11e9-afbc-1318731e806f
[2019-08-23 18:23:57.295103 +0530][Debug ][Utility ] Creating a third party fall back copy job, from root://eoskolkata.tier2-kol.res.in:1094//eos/alicekolkata/grid/00/65278/ef1babe8-6f56-11e9-afbc-1318731e806f to file://localhost/tmp/.//ef1babe8-6f56-11e9-afbc-1318731e806f
[2019-08-23 18:23:57.295196 +0530][Debug ][Utility ] Monitor library name not set. No monitoring
[2019-08-23 18:23:57.295258 +0530][Debug ][Utility ] Check if third party copy between root://eoskolkata.tier2-kol.res.in:1094//eos/alicekolkata/grid/00/65278/ef1babe8-6f56-11e9-afbc-1318731e806f and file://localhost/tmp/.//ef1babe8-6f56-11e9-afbc-1318731e806f is possible
[0B/0B][100%][==================================================][0B/s]
Run: [ERROR] Operation not supported
[root@eos-mgm ~]#

There has one frequent error in xrdlog.mgm is “190823 18:28:39 179002 XrootdXeq: User authentication failed; Decryption key not found.
190823 18:28:39 179122 XrootdXeq: User authentication failed; Decryption key not found.”

Another error has shown in xrdlog.mgm which has filter by SE manager (dcache-server) and SE client (storage01) i.e.:- (Occur in 2 times)
###############
[root@eos-mgm ~]# cat /var/log/eos/mgm/xrdlog.mgm | grep dcache-server
190823 12:56:29 time=1566545189.439775 func=open level=INFO logid=52ea8c68-c577-11e9-b728-e4434b6639ba unit=mgm@eos-mgm.tier2-kol.res.in:1094 tid=00007fe8eb3fd700 source=XrdMgmOfsFile:189 tident=jalien.23709:262@alientest03.cern.ch sec=unix uid=10367 gid=1395 name=jalien geo="" op=write trunc=512 path=/eos/alicekolkata/grid/03/48272/41e56794-26a4-11e8-9678-001f29eb8d2a info=authz=<…>&eos.app=transfer-3rd&oss.asize=0&tpc.dlg=dcache-server.tier2-kol.res.in:1094&tpc.key=14e537771d1d5c9d5d5f9525&tpc.lfn=/03/48272/41e56794-26a4-11e8-9678-001f29eb8d2a&tpc.spr=root&tpc.src=storage01.tier2-kol.res.in:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root
190823 12:56:29 time=1566545189.444662 func=open level=INFO logid=52ea8c68-c577-11e9-b728-e4434b6639ba unit=mgm@eos-mgm.tier2-kol.res.in:1094 tid=00007fe8eb3fd700 source=XrdMgmOfsFile:2457 tident=jalien.23709:262@alientest03.cern.ch sec=unix uid=10367 gid=1395 name=jalien geo="" op=write path=/eos/alicekolkata/grid/03/48272/41e56794-26a4-11e8-9678-001f29eb8d2a info=authz=<…>&eos.app=transfer-3rd&oss.asize=0&tpc.dlg=dcache-server.tier2-kol.res.in:1094&tpc.key=14e537771d1d5c9d5d5f9525&tpc.lfn=/03/48272/41e56794-26a4-11e8-9678-001f29eb8d2a&tpc.spr=root&tpc.src=storage01.tier2-kol.res.in:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root target[0]=(eos09.tier2-kol.res.in,77) target[1]=(eos07.tier2-kol.res.in,74) target[2]=(eos04.tier2-kol.res.in,71) target[3]=(eos06.tier2-kol.res.in,75) target[4]=(eos05.tier2-kol.res.in,73) target[5]=(eos08.tier2-kol.res.in,72) target[6]=(eos10.tier2-kol.res.in,76) redirection=eos09.tier2-kol.res.in?&cap.sym=<…>&cap.msg=<…>&mgm.logid=52ea8c68-c577-11e9-b728-e4434b6639ba&mgm.replicaindex=0&mgm.replicahead=0&mgm.id=00e1da64:1095
[root@eos-mgm ~]# cat /var/log/eos/mgm/xrdlog.mgm | grep storage01
190823 12:56:29 time=1566545189.439775 func=open level=INFO logid=52ea8c68-c577-11e9-b728-e4434b6639ba unit=mgm@eos-mgm.tier2-kol.res.in:1094 tid=00007fe8eb3fd700 source=XrdMgmOfsFile:189 tident=jalien.23709:262@alientest03.cern.ch sec=unix uid=10367 gid=1395 name=jalien geo="" op=write trunc=512 path=/eos/alicekolkata/grid/03/48272/41e56794-26a4-11e8-9678-001f29eb8d2a info=authz=<…>&eos.app=transfer-3rd&oss.asize=0&tpc.dlg=dcache-server.tier2-kol.res.in:1094&tpc.key=14e537771d1d5c9d5d5f9525&tpc.lfn=/03/48272/41e56794-26a4-11e8-9678-001f29eb8d2a&tpc.spr=root&tpc.src=storage01.tier2-kol.res.in:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root
190823 12:56:29 time=1566545189.444662 func=open level=INFO logid=52ea8c68-c577-11e9-b728-e4434b6639ba unit=mgm@eos-mgm.tier2-kol.res.in:1094 tid=00007fe8eb3fd700 source=XrdMgmOfsFile:2457 tident=jalien.23709:262@alientest03.cern.ch sec=unix uid=10367 gid=1395 name=jalien geo="" op=write path=/eos/alicekolkata/grid/03/48272/41e56794-26a4-11e8-9678-001f29eb8d2a info=authz=<…>&eos.app=transfer-3rd&oss.asize=0&tpc.dlg=dcache-server.tier2-kol.res.in:1094&tpc.key=14e537771d1d5c9d5d5f9525&tpc.lfn=/03/48272/41e56794-26a4-11e8-9678-001f29eb8d2a&tpc.spr=root&tpc.src=storage01.tier2-kol.res.in:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root target[0]=(eos09.tier2-kol.res.in,77) target[1]=(eos07.tier2-kol.res.in,74) target[2]=(eos04.tier2-kol.res.in,71) target[3]=(eos06.tier2-kol.res.in,75) target[4]=(eos05.tier2-kol.res.in,73) target[5]=(eos08.tier2-kol.res.in,72) target[6]=(eos10.tier2-kol.res.in,76) redirection=eos09.tier2-kol.res.in?&cap.sym=<…>&cap.msg=<…>&mgm.logid=52ea8c68-c577-11e9-b728-e4434b6639ba&mgm.replicaindex=0&mgm.replicahead=0&mgm.id=00e1da64:1095
[root@eos-mgm ~]#
####################
I don’t know if it is an authentication/permission error of if it is another problem.
I’m looking forward any comment or suggestion.

Regards
Prasun