Hi EOS Team,
We are constantly trying to make it usable, but some glitch is there and shows permission denied for copying any file. Still we could not catch it. Kindly help to resolve it.
As such ALICE::Kolkata ::EOS2 is up with MGM, Slave, QuarkDb and 8 FSTs with EOS Version 5.3.32. Below are EOS VID and permissions
===================
[root@eos-mgm ~]# eos root://eoskolkata.tier2-kol.res.in whoami
Virtual Identity: uid=65534 (65534) gid=65534 (65534) [authz:sss] host=eos-mgm.tier2-kol.res.in domain=tier2-kol.res.in
[root@eos-mgm ~]# env XrdSecPROTOCOL=sss eos root://localhost whoami
Virtual Identity: uid=0 (0,3,65534) gid=0 (0,4,65534) [authz:sss] sudo* host=localhost domain=localdomain
[root@eos-mgm ~]# env XrdSecPROTOCOL=unix eos root://localhost whoami
Virtual Identity: uid=0 (0,3,10367,65534) gid=0 (0,4,1395) [authz:unix] sudo* host=localhost domain=localdomain
[root@eos-mgm ~]# eos vid ls
https:“”:gid => root
https:“”:uid => root
publicaccesslevel: => 1024
sss:“”:gid => root
sss:“”:uid => root
sudoer => uids(daemon)
tokensudo => always
unix:“”:gid => alice
unix:“”:uid => aliprod
[root@eos-mgm ~]# less /etc/xrd.cf.mgm
[root@eos-mgm ~]# eos -b space ls
┌──────────┬────────────────┬────────────┬────────────┬──────┬─────────┬───────────────┬──────────────┬─────────────┬─────────────┬──────────────┬──────┬──────┬──────────┬───────────┬───────────┬──────┬────────┬───────────┬──────┬────────┬───────────┐
│type │ name│ groupsize│ groupmod│ N(fs)│ N(fs-rw)│ sum(usedbytes)│ sum(capacity)│ capacity(rw)│ nom.capacity│sched.capacity│ usage│ quota│ balancing│ threshold│ converter│ ntx│ active│ wfe│ ntx│ active│ intergroup│
└──────────┴────────────────┴────────────┴────────────┴──────┴─────────┴───────────────┴──────────────┴─────────────┴─────────────┴──────────────┴──────┴──────┴──────────┴───────────┴───────────┴──────┴────────┴───────────┴──────┴────────┴───────────┘
spaceview default 8 24 128 110 1.07 PB 1.27 PB 1.09 PB 0 B 161.63 TB 98.30 off on 1 ??? 0 0 off 1 0 on
[root@eos-mgm ~]#
====================
But in the xrdlog.mgm probably (XrootdXeq: User authentication failed; Decryption key not found) is the main problem.
The Errors i.e. “User authentication failed; Decryption key not found” has still seem in xrdlog.fst and xrdlog.mgm. Also, another error i.e. “Auth failed: No protocols left to try” are seem in xrdlog.mgm.
Output of xrdlog.mgm
tident= sec=(null) uid=0 gid=0 name=- geo="" xt="" ob="" msg="url invalid" src="" dst="root://daemon@eos05.tier2-kol.res.in:1095//replicate:0?cap.msg=IYL38HN/kZIjgPhIwgAFGdkBlzO7V5Xv9H1RVd/plYhSa7C6CWBdibm2G8kFBdi/1TRGvwc0+cwWHvf1+FWuLUCV1ApAYUvxAiwFcNBhnZMxWc5kwZxAuE25oI2dxQHg4BNSFHoiR+2yIHv0FjTPH/ydsx/SCUARhnMWuz/Xi8rlulYmZZG15HcVaMGfEhm8C9VJW9ufGqSm62DMRNe6PxSem+EmbPs/8m7X4r/+56AreRCJJoiLpjWRLfW/AWmd8LIxZurUDi8MFhCp30/CV6B+3BTp6BBJdHmH+Gt3VE+UOj0+PRRcpA==&cap.sym=r2VMTRnN8Hsy6ZVH7h1114leegI=&mgm.logid=bd089698-492e-11f1-8997-e4434b664554"
260506 15:04:11 time=1778060051.397734 func=DoIt level=ERROR logid=bd05742c-492e-11f1-b61a-e4434b664554 unit=mgm@eos-mgm.tier2-kol.res.in:1094 tid=00007f185f1ff640 source=DrainTransferJob:187 tident= sec= uid=0 gid=0 name= geo=“” xt=“” ob=“” src=root://eoskolkata.tier2-kol.res.in:1094//#curl#/eos/alicekolkata/grid/11/06148/7c3e97cb-1411-11ef-a4c1-b47af1a61b9a dst=root://eos05.tier2-kol.res.in:1095//replicate:0 logid=bd085c78-492e-11f1-86ad-e4434b664554 tpc_err=[ERROR] Server responded with an error: [3010] Unable to open file /eos/alicekolkata/grid/11/06148/7c3e97cb-1411-11ef-a4c1-b47af1a61b9a; Operation not permitted
260506 15:04:11 time=1778060051.397811 func=ReportError level=ERROR logid=bd05742c-492e-11f1-b61a-e4434b664554 unit=mgm@eos-mgm.tier2-kol.res.in:1094 tid=00007f185f1ff640 source=DrainTransferJob:67 tident= sec= uid=0 gid=0 name= geo=“” xt=“” ob=“” msg=“fxid=08ba9772 rain reconstruct already failed”
260506 15:04:11 time=1778060051.397968 func=DoIt level=ERROR logid=static… unit=mgm@eos-mgm.tier2-kol.res.in:1094 tid=00007f185f1ff640 source=DrainTransferJob:146 tident= sec=(null) uid=0 gid=0 name=- geo=“” xt=“” ob=“” msg=“url invalid” src=“” dst=“root://daemon@eos05.tier2-kol.res.in:1095//replicate:0?cap.msg=IYL38HN/kZIjgPhIwgAFGdkBlzO7V5Xv9H1RVd/plYhSa7C6CWBdibm2G8kFBdi/1TRGvwc0+cwWHvf1+FWuLUCV1ApAYUvxAiwFcNBhnZMxWc5kwZxAuKPWHiyxuCNODMCrHVwdUnobJHfMcKPUzTIR+UWMMvsJO0GemfkW8vEeEywcIX8nTnQ7oXX0geBH+sQ7ysRxT186RGbbV8ceBBYJfEJMzjvBiio9BcW+7vzZeBgLp5S5K9VPEAzDKb6YwtjiwEVFloD/bHeXdcZIvCB8GPQY3dy+Wly7prVnHsVeZ1mOpgJvwA==&cap.sym=r2VMTRnN8Hsy6ZVH7h1114leegI=&mgm.logid=bd089eb8-492e-11f1-86ad-e4434b664554”
260506 15:04:11 9978 XrootdXeq: User authentication failed; Decryption key not found.
260506 15:04:11 9978 XrootdXeq: daemon.2202:693@eos04 disc 0:00:00
260506 15:04:11 9993 XrootdXeq: User authentication failed; Decryption key not found.
260506 15:04:11 9993 XrootdXeq: daemon.180391:321@eos05 disc 0:00:00
260506 15:04:11 9918 XrootdXeq: User authentication failed; Decryption key not found.
260506 15:04:11 9918 XrootdXeq: daemon.180391:693@eos05 disc 0:00:00
Output of xrdlog.fst
260506 14:53:18 time=1778059398.383675 func=open level=INFO logid=3731c2f2-492d-11f1-af9b-e4434b664554 unit=fst@eos05.tier2-kol.res.in:1095 tid=00007f1926df8640 source=XrdFstOfsFile:544 tident=jalien.4098820:33@pcapiserv03.cern.ch sec=(null) uid=10367 gid=1395 name=nobody geo=“” xt=“” ob=“” path=/xdata15/0000473b/0ade76f5 open-mode=300 create-mode=41a4 layout-name=raid6 oss-opaque=&mgm.lid=543425858&mgm.bookingsize=4096
260506 14:53:18 time=1778059398.383738 func=fileOpen level=INFO logid=37ce9492-492d-11f1-94fd-246e96b4de7c unit=fst@eos05.tier2-kol.res.in:1095 tid=00007f1926df8640 source=LocalIo:70 tident= sec= uid=0 gid=0 name= geo=“” xt=“” ob=“” flags=302 path=/xdata15/0000473b/0ade76f5
260506 14:53:18 time=1778059398.609456 func=CallManager level=INFO logid=static… unit=fst@eos05.tier2-kol.res.in:1095 tid=00007f19271fa640 source=XrdFstOfs:1329 tident= sec=(null) uid=0 gid=0 name=- geo=“” xt=“” ob=“” msg=“retry query” query=“/?mgm.pcmd=drop&mgm.fid=0ade75fb&mgm.fsid=0&mgm.dropall=1”
260506 14:53:18 time=1778059398.610479 func=CallManager level=ERROR logid=static… unit=fst@eos05.tier2-kol.res.in:1095 tid=00007f19271fa640 source=XrdFstOfs:1279 tident= sec=(null) uid=0 gid=0 name=- geo=“” xt=“” ob=“” msg=“MGM query failed” opaque=“/?mgm.pcmd=drop&mgm.fid=0ade75fb&mgm.fsid=0&mgm.dropall=1”
260506 14:53:18 time=1778059398.610496 func=CallManager level=ERROR logid=static… unit=fst@eos05.tier2-kol.res.in:1095 tid=00007f19271fa640 source=XrdFstOfs:1322 tident= sec=(null) uid=0 gid=0 name=- geo=“” xt=“” ob=“” msg=“query error” status=3 code=204
260506 14:53:18 180426 XrootdXeq: User authentication failed; Decryption key not found.
260506 14:53:18 180426 XrootdXeq: daemon.1962:42@eos-mgm disc 0:00:00
260506 14:53:18 180427 XrootdXeq: User authentication failed; Decryption key not found.
260506 14:53:18 180427 XrootdXeq: daemon.1962:45@eos-mgm disc 0:00:00
EOS Keytab is the same on all the EOS Cluster nodes and the ALICE TkAuthz file is also okay.
============================
[root@eos-mgm ~]# cat /etc/grid-security/xrootd/TkAuthz.Authorization
EXPORT PATH:/ VO:* ACCESS:ALLOW CERT:*
RULE PATH:/eos/ AUTHZ:delete|read|write|write-once| NOAUTHZ:| VO:| CERT:IGNORE
KEY VO: PRIVKEY:/etc/grid-security/xrootd/privkey.pem PUBKEY:/etc/grid-security/xrootd/pubkey.pem
[root@eos-mgm ~]#
============================
If we try to copy a healthy file then it shows 3010 permission denied
============================
[root@eos-mgm ~]# eos -b file info /eos/alicekolkata/grid/00/65278/fbbef312-2658-11ec-ad5c-7b5f5f28be11
File: ‘/eos/alicekolkata/grid/00/65278/fbbef312-2658-11ec-ad5c-7b5f5f28be11’ Flags: 0664
Size: 40987342
Status: healthy
Modify: Wed Oct 6 09:24:17 2021 Timestamp: 1633492457.384080000
Change: Wed Oct 6 09:24:16 2021 Timestamp: 1633492456.920868148
Access: Thu Jan 1 05:30:00 1970 Timestamp: 0.000000000
Birth: Wed Oct 6 09:24:16 2021 Timestamp: 1633492456.920868148
CUid: 10367 CGid: 1395 Fxid: 040e057a Fid: 68027770 Pid: 12948 Pxid: 00003294
XStype: adler XS: a2 91 ce 6a ETAGs: “18261065460613120:a291ce6a”
Layout: raid6 Stripes: 6 Blocksize: 1M LayoutId: 20640542 Redundancy: d3::t0
#Rep: 6
┌───┬──────┬────────────────────────┬────────────────┬────────────────┬──────────┬──────────────┬────────────┬────────┬────────────────────────┐
│no.│ fs-id│ host│ schedgroup│ path│ boot│ configstatus│ drain│ active│ geotag│
└───┴──────┴────────────────────────┴────────────────┴────────────────┴──────────┴──────────────┴────────────┴────────┴────────────────────────┘
0 143 eos09.tier2-kol.res.in default.14 /xdata8 booted drain waiting online Kolkata::EOS2
1 102 eos07.tier2-kol.res.in default.14 /xdata8 booted rw nodrain online Kolkata::EOS2
2 100 eos08.tier2-kol.res.in default.14 /xdata8 booted rw nodrain online Kolkata::EOS2
3 99 eos04.tier2-kol.res.in default.14 /xdata8 booted rw nodrain online Kolkata::EOS2
4 101 eos05.tier2-kol.res.in default.14 /xdata8 booted rw nodrain online Kolkata::EOS2
5 103 eos06.tier2-kol.res.in default.14 /xdata8 booted rw nodrain online Kolkata::EOS2
[root@eos-mgm ~]# /opt/eos/xrootd/bin/xrdcp -f -d 1 --tpc only root://eoskolkata.tier2-kol.res.in:1094//eos/alicekolkata/grid/00/65278/fbbef312-2658-11ec-ad5c-7b5f5f28be11 /tmp/.
[0B/0B][100%][==================================================][0B/s]
Run: [ERROR] Operation not supported: Cannot do a third-party-copy for local file.
[root@eos-mgm ~]#
[root@eos-mgm ~]# /opt/eos/xrootd/bin/xrdcp -f -d 1 root://eoskolkata.tier2-kol.res.in:1094//eos/alicekolkata/grid/00/65278/fbbef312-2658-11ec-ad5c-7b5f5f28be11 /tmp/.
[0B/0B][100%][==================================================][0B/s]
Run: [ERROR] Server responded with an error: [3010] Unable to open file /eos/alicekolkata/grid/00/65278/fbbef312-2658-11ec-ad5c-7b5f5f28be11; Operation not permitted (source)
[root@eos-mgm ~]#
===================================
Kindly suggest how to resolve it.
Regards
Kolkata Team.