Debug level command does not seem to be transferred to FSTs

1

Hi,

It is supposed to be working with eos debug <debuglevel> <target> when we want to change debug level usually however, in recent new installation, the command does not seem to be transferred from MGM to FSTs.

For example, when we try to run eos debug notice /eos/*/fst it claims an error as follows:

sh-4.2# eos debug notice /eos/*/fst
error: requested endpoint(s) not existing or not online

In MGM logs, we do not have any errors but some info messages like:

sh-4.2# grep a315468a-6623-11eb-8ed4-0242a01c0c78 /var/log/eos/mgm/xrootd.mgm.log
210203 13:28:01 time=1612358881.090390 func=open                     level=INFO  logid=a315468a-6623-11eb-8ed4-0242a01c0c78 unit=mgm@jbod-mgmt-01.sdfarm.kr:1094 tid=00007f4fb2ee1700 source=XrdMgmOfsFile:500              tident=root.18520:283@localhost sec=sss   uid=0 gid=0 name=daemon geo="" op=read path=/proc/user/ info=eos.rgid=0&eos.ruid=0&mgm.cmd.proto=ahYKFAoKL2Vvcy8qL2ZzdBIGbm90aWNl
210203 13:28:01 time=1612358881.090569 func=open                     level=DEBUG logid=a315468a-6623-11eb-8ed4-0242a01c0c78 unit=mgm@jbod-mgmt-01.sdfarm.kr:1094 tid=00007f4fb2ee1700 source=XrdMgmOfsFile:700              tident=root.18520:283@localhost sec=sss   uid=0 gid=0 name=daemon geo="" mode=0 create=100 truncate=200
210203 13:28:01 time=1612358881.090625 func=HandleProtobufRequest    level=INFO  logid=a315468a-6623-11eb-8ed4-0242a01c0c78 unit=mgm@jbod-mgmt-01.sdfarm.kr:1094 tid=00007f4fb2ee1700 source=ProcInterface:202              tident=root.18520:283@localhost sec=sss   uid=0 gid=0 name=daemon geo="" cmd_proto={"debug":{"set":{"nodename":"/eos/*/fst","debuglevel":"notice"}}}

Strangely when we try to change debug level only on MGM the same error happens but the command was effective, e.g. (info → notice):

sh-4.2# eos debug info
error: requested endpoint(s) not existing or not online
sh-4.2# eos debug get
# ------------------------------------------------------------------------------------
# Debug log level
# ....................................................................................
/eos/jbod-mgmt-01.sdfarm.kr:1094/mgm := info
/eos/jbod-mgmt-01.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-01.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-02.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-02.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-03.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-03.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-04.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-04.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-05.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-05.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-06.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-06.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-07.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-07.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-08.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-08.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-09.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-09.sdfarm.kr:1096/fst := info
sh-4.2# eos debug notice
error: requested endpoint(s) not existing or not online
sh-4.2# eos debug get
# ------------------------------------------------------------------------------------
# Debug log level
# ....................................................................................
/eos/jbod-mgmt-01.sdfarm.kr:1094/mgm := note
/eos/jbod-mgmt-01.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-01.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-02.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-02.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-03.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-03.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-04.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-04.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-05.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-05.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-06.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-06.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-07.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-07.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-08.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-08.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-09.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-09.sdfarm.kr:1096/fst := info

Do you have any suggestions where to look for further investigation? This seems trivial but we need to suppress some level of logs when it comes to production service.

FYI, the EOS version is 4.8.31 (tag release)

Thank you in advance.

Best regards,
Sang-Un

2

Hi Sang-Un,

You can use for the moment eos debug notice * to put all daemons in notice and then put the MGM log level to info:

eos debug info /eos/jbod-mgmt-01.sdfarm.kr:1094/mgm

Thanks
Elvin

3

Hi Elvin,

Thanks a lot for the suggestion. However eos debug notice * does not work but only prints out the usage as follows:

sh-4.2# eos debug notice *
 usage:
debug get|this|<level> [node-queue] [--filter <unitlist>]
'[eos] debug ...' allows to get or set the verbosity of the EOS log files in MGM and FST services.

Options:

debug get : retrieve the current log level for the mgm and fsts node-queue

debug this : toggle EOS shell debug mode

debug  <level> [--filter <unitlist>] : set the MGM where the console is connected to into debug level <level>

debug  <level> <node-queue> [--filter <unitlist>] : set the <node-queue> into debug level <level>.
         - <node-queue> are internal EOS names e.g. '/eos/<hostname>:<port>/fst'
         - <unitlist> is a comma separated list of strings of software units which should be filtered out in the message log!

The default filter list is:
'Process,AddQuota,Update,UpdateHint,UpdateQuotaStatus,SetConfigValue,Deletion,GetQuota,PrintOut,RegisterNode,SharedHash,listenFsChange,placeNewReplicas,placeNewReplicasOneGroup,accessReplicas,accessReplicasOneGroup,accessHeadReplicaMultipleGroup,updateTreeInfo,updateAtomicPenalties,updateFastStructures,work'.

The allowed debug levels are:
debug,info,warning,notice,err,crit,alert,emerg

Examples:
         debug info *                         set MGM & all FSTs into debug mode 'info'

         debug err /eos/*/fst                 set all FSTs into debug mode 'info'

         debug crit /eos/*/mgm                set MGM into debug mode 'crit'

         debug debug --filter MgmOfsMessage   set MGM into debug mode 'debug' and filter only messages coming from unit 'MgmOfsMessage'.

This is strange… one of the examples suggests such as debug info * but this does not work.

Best regards,
Sang-Un

4

It works when I run debug notice * in eos shell.

sh-4.2# eos
# ---------------------------------------------------------------------------
# EOS  Copyright (C) 2011-2020 CERN/Switzerland
# This program comes with ABSOLUTELY NO WARRANTY; for details type `license'.
# This is free software, and you are welcome to redistribute it
# under certain conditions; type `license' for details.
# ---------------------------------------------------------------------------
EOS_INSTANCE=gsdc
EOS_SERVER_VERSION=4.8.31 EOS_SERVER_RELEASE=1
EOS_CLIENT_VERSION=4.8.31 EOS_CLIENT_RELEASE=1
EOS Console [root://localhost] |/> debug notice *
success: debug level is now <notice>success: switched to mgm.debuglevel=notice on nodes mgm.nodename=*
EOS Console [root://localhost] |/> debug get
# ------------------------------------------------------------------------------------
# Debug log level
# ....................................................................................
/eos/jbod-mgmt-01.sdfarm.kr:1094/mgm := note
/eos/jbod-mgmt-01.sdfarm.kr:1095/fst := note
/eos/jbod-mgmt-01.sdfarm.kr:1096/fst := note
/eos/jbod-mgmt-02.sdfarm.kr:1095/fst := note
/eos/jbod-mgmt-02.sdfarm.kr:1096/fst := note
/eos/jbod-mgmt-03.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-03.sdfarm.kr:1096/fst := note
/eos/jbod-mgmt-04.sdfarm.kr:1095/fst := note
/eos/jbod-mgmt-04.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-05.sdfarm.kr:1095/fst := note
/eos/jbod-mgmt-05.sdfarm.kr:1096/fst := note
/eos/jbod-mgmt-06.sdfarm.kr:1095/fst := note
/eos/jbod-mgmt-06.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-07.sdfarm.kr:1095/fst := note
/eos/jbod-mgmt-07.sdfarm.kr:1096/fst := info
/eos/jbod-mgmt-08.sdfarm.kr:1095/fst := note
/eos/jbod-mgmt-08.sdfarm.kr:1096/fst := note
/eos/jbod-mgmt-09.sdfarm.kr:1095/fst := info
/eos/jbod-mgmt-09.sdfarm.kr:1096/fst := info

Best regards,
Sang-Un

5

I got a hint.

eos debug info '*' works.

Thanks!!

Best regards,
Sang-Un

6

Yes, exactly you need to put the star in quotes otherwise bash might expand it to some strange thing.

Cheers,
Elvin