Dear All,
EOS storage at Kolkata Tier2 has failed due to eos.fst service is failed after restart eos in all 3 FST servers. The error show in xrdlog.fst are below:-
[root@eos01 ~]# service eos restart
Stopping xrootd: fst [FAILED]
Starting xrootd as fst with -n fst -c /etc/xrd.cf.fst -l /var/log/eos/xrdlog.fst -b -Rdaemon
[ OK ]
[root@eos01 ~]# tail -f /var/log/eos/fst/xrdlog.fst
180921 17:10:37 time=1537530037.594507 func=Communicator level=INFO logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df8bff700 source=Comunicator:106 tident= sec=(null) uid=99 gid=99 name=- geo="" FST shared object notification subject is /eos01.tier2-kol.res.in:1095/fst/gw/txqueue/txq
180921 17:10:37 time=1537530037.594528 func=Communicator level=INFO logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df8bff700 source=Comunicator:106 tident= sec=(null) uid=99 gid=99 name=- geo="" FST shared object notification subject is /eos/eos01.tier2-kol.res.in:1095/fst/
Config warning: asynchronous I/O has been disabled!
Config warning: sendfile I/O has been disabled!
Config warning: ‘xrootd.prepare logdir’ not specified; prepare tracking disabled.
------ xrootd protocol initialization completed.
------ xrootd fst@eos01.tier2-kol.res.in:1095 initialization completed.
180921 17:10:37 time=1537530037.609006 func=getFstNodeConfigQueue level=INFO logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df89fd700 source=Config:40 tident= sec=(null) uid=99 gid=99 name=- geo="" Waiting for config queue in Publish …
180921 17:10:38 time=1537530038.595262 func=Run level=NOTE logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df79fd700 source=HttpServer:135 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“starting http server” mode=“epoll” threads=32
180921 17:10:38 time=1537530038.597095 func=Run level=INFO logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df79fd700 source=HttpServer:179 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“start of micro httpd succeeded [port=8001]”
180921 17:10:42 time=1537530042.583184 func=getFstNodeConfigQueue level=INFO logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6dfa1fd700 source=Config:40 tident= sec=(null) uid=99 gid=99 name=- geo="" Waiting for config queue in Remover …
180921 17:10:42 time=1537530042.586091 func=getFstNodeConfigQueue level=INFO logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df7bff700 source=Config:40 tident= sec=(null) uid=99 gid=99 name=- geo="" Waiting for config queue in Balancer …
180921 17:10:42 time=1537530042.586665 func=getFstNodeConfigQueue level=INFO logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df7afe700 source=Config:40 tident= sec=(null) uid=99 gid=99 name=- geo="" Waiting for config queue in …
180921 17:10:42 time=1537530042.590807 func=getFstNodeConfigQueue level=INFO logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6fff700 source=Config:40 tident= sec=(null) uid=99 gid=99 name=- geo="" Waiting for config queue in Cleaner …
180921 17:10:42 time=1537530042.590882 func=Cleaner level=NOTE logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6fff700 source=Cleaner:41 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“cleaning transactions”
180921 17:10:42 time=1537530042.590951 func=Cleaner level=ERROR logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6fff700 source=Cleaner:66 tident= sec=(null) uid=99 gid=99 name=- geo="" msg=“don’t know the manager name”
180921 17:10:42 time=1537530042.591161 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:10:42 time=1537530042.683411 func=CallManager level=ERROR logid=25b8a7b2-bd93-11e8-aaa4-b083fece9f5f unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6dfa1fd700 source=XrdFstOfs:857 tident= sec= uid=0 gid=0 name= geo="" error=URL is not valid: root:////dummy?xrd.wantprot=sss
180921 17:10:42 time=1537530042.683484 func=Remover level=ERROR logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6dfa1fd700 source=Remover:111 tident= sec=(null) uid=99 gid=99 name=- geo="" manager returned errno=22
180921 17:10:47 time=1537530047.591326 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:10:52 time=1537530052.591500 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:10:57 time=1537530057.591685 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:11:02 time=1537530062.591858 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:11:07 time=1537530067.592012 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:11:12 time=1537530072.592162 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:11:17 time=1537530077.592336 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:11:22 time=1537530082.592492 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:11:27 time=1537530087.592655 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:11:32 time=1537530092.592835 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:11:37 time=1537530097.593004 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:11:42 time=1537530102.593162 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:11:43 time=1537530103.654291 func=Release level=WARN logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df89fd700 source=RWMutex:1367 tident= sec=(null) uid=99 gid=99 name=- geo="" WARNING - read lock held for 55984 milliseconds by this thread:
backward disabled
180921 17:11:43 time=1537530103.654358 func=Publish level=WARN logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df89fd700 source=Publish:476 tident= sec=(null) uid=99 gid=99 name=- geo="" Publisher cycle exceeded 11463 millisecons - took 55984 milliseconds
180921 17:11:47 time=1537530107.593322 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:11:52 time=1537530112.593501 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:11:57 time=1537530117.593689 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:12:02 time=1537530122.593850 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:12:07 time=1537530127.594027 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:12:12 time=1537530132.594189 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:12:17 time=1537530137.594349 func=MgmSyncer level=INFO logid=FstOfsStorage unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:63 tident= sec= uid=0 gid=0 name= geo="" msg=“waiting to know manager”
180921 17:12:17 time=1537530137.594422 func=MgmSyncer level=ALERT logid=static… unit=fst@eos01.tier2-kol.res.in:1095 tid=00007f6df6efe700 source=MgmSyncer:66 tident= sec=(null) uid=99 gid=99 name=- geo="" didn’t receive manager name, aborting
@@@@@@ 00:00:00 op=shutdown msg="shutdown timedout after 0 seconds, signal=1
@@@@@@ 00:00:00 op=shutdown status=forced-complete
We check xrd.cf.fst and eos file:-
++++
[root@eos01 ~]# cat /etc/sysconfig/eos
DAEMON_COREFILE_LIMIT=unlimited
test -e /usr/lib64/libjemalloc.so.1 && export LD_PRELOAD=/usr/lib64/libjemalloc.so.1
XRD_ROLES=“fst”
export EOS_BROKER_URL=root://eos.tier2-kol.res.in:1097//eos/
export EOS_MGM_ALIAS=eos.tier2-kol.res.in
export EOS_HTTP_THREADPOOL=epoll
export EOS_HTTP_THREADPOOL_SIZE=32
export EOS_HTTP_CONNECTION_MEMORY_LIMIT=65536
export EOS_MGM_ALIAS=eos.tier2-kol.res.in
export APMON_INSTANCE_NAME=ALICE::KOLKATA::EOS
export MONALISAHOST=grid01.tier2-kol.res.in
export APMON_STORAGEPATH=edata
export EOS_GEOTAG=tier2-kol.res.in
export EOS_FST_NO_SSS_ENFORCEMENT=1
++++++
Two day ago, we convert EOS version from Aquamarine to Citrine. Under new version MGM is running good. FSTs are failed. The error of fst i.e. " didn’t receive manager name, aborting" has occur in previous EOS version “Aquamarine” also. Network between 3 FST and MGM is good. OS on those eos are Scientific Linux 6.10.
PLEASE HELP what we do?