CERN Accelerating science

No output of "eos -b group ls" and "eos -b fs ls" after migrating to QuarkDB

Dear Expert,

We have migrate Memory Based-NameSpace to Quarkdb Based NameSpace for Kolkata:EOS2.

The version of eos and quarkdb are :-


EOS_SERVER_VERSION=4.8.40 EOS_SERVER_RELEASE=1
EOS_CLIENT_VERSION=4.8.40 EOS_CLIENT_RELEASE=1
QUARKDB-VERSION 0.4.2


Today we had started the QuarkDB with three hosts (i.e. eos-mgm, eos-slave and eos-qdb). It’s working fine. We also convert files.md and directories.md to QuarkDB. Then start eos@master and eos@slave with eos@mq and eos@mgm daemon in two hosts. (eos-mgm has Master role and eos-slave has SLAVE role).
It’s start fine and shown the all seven FST’s i.e.


[root@eos-mgm ~]# redis-cli -p 7777 raft-info

  1. TERM 192
  2. LOG-START 1111
  3. LOG-SIZE 12572
  4. LEADER eos-mgm.tier2-kol.res.in:7777
  5. CLUSTER-ID 8370db63-fdf7-4df9-807a-36a7c5692232
  6. COMMIT-INDEX 12571
  7. LAST-APPLIED 12571
  8. BLOCKED-WRITES 0
  9. LAST-STATE-CHANGE 8036 (2 hours, 13 minutes, 56 seconds)

  10. MYSELF eos-mgm.tier2-kol.res.in:7777
  11. VERSION 0.4.2
  12. STATUS LEADER
  13. NODE-HEALTH GREEN
  14. JOURNAL-FSYNC-POLICY sync-important-updates

  15. MEMBERSHIP-EPOCH 1111
  16. NODES eos-mgm.tier2-kol.res.in:7777,eos-slave.tier2-kol.res.in:7777,eos-qdb.tier2-kol.res.in:7777
  17. OBSERVERS
  18. QUORUM-SIZE 2

  19. REPLICA eos-qdb.tier2-kol.res.in:7777 | ONLINE | UP-TO-DATE | NEXT-INDEX 12572 | VERSION 0.4.2
  20. REPLICA eos-slave.tier2-kol.res.in:7777 | ONLINE | UP-TO-DATE | NEXT-INDEX 12572 | VERSION 0.4.2
    [root@eos-mgm ~]#

[root@eos-mgm ~]# eos -b node ls
┌──────────┬────────────────────────────────┬────────────────┬──────────┬────────────┬──────┬──────────┬────────┬────────┬────────────────┬─────┐
│type │ hostport│ geotag│ status│ activated│ txgw│ gw-queued│ gw-ntx│ gw-rate│ heartbeatdelta│ nofs│
└──────────┴────────────────────────────────┴────────────────┴──────────┴────────────┴──────┴──────────┴────────┴────────┴────────────────┴─────┘
nodesview eos04.tier2-kol.res.in:1095 Kolkata::EOS2 online ??? off 0 10 120 0 0
nodesview eos05.tier2-kol.res.in:1095 Kolkata::EOS2 online ??? off 0 10 120 2 0
nodesview eos06.tier2-kol.res.in:1095 Kolkata::EOS2 online ??? off 0 10 120 2 0
nodesview eos07.tier2-kol.res.in:1095 Kolkata::EOS2 online ??? off 0 10 120 1 0
nodesview eos08.tier2-kol.res.in:1095 Kolkata::EOS2 online ??? off 0 10 120 0 0
nodesview eos09.tier2-kol.res.in:1095 Kolkata::EOS2 online ??? off 0 10 120 2 0
nodesview eos10.tier2-kol.res.in:1095 Kolkata::EOS2 online ??? off 0 10 120 1 0

[root@eos-mgm ~]#


Here we notice that all the seven FST’s are not activated and then we had activate all seven FSTs manually by “eos node set eos04.tier2-kol.res.in on”.
Then they are activated i.e.
++++++++++++
[root@eos-mgm ~]# eos -b node ls
┌──────────┬────────────────────────────────┬────────────────┬──────────┬────────────┬──────┬──────────┬────────┬────────┬────────────────┬─────┐
│type │ hostport│ geotag│ status│ activated│ txgw│ gw-queued│ gw-ntx│ gw-rate│ heartbeatdelta│ nofs│
└──────────┴────────────────────────────────┴────────────────┴──────────┴────────────┴──────┴──────────┴────────┴────────┴────────────────┴─────┘
nodesview eos04.tier2-kol.res.in:1095 Kolkata::EOS2 online on off 0 10 120 1 0
nodesview eos05.tier2-kol.res.in:1095 Kolkata::EOS2 online on off 0 10 120 2 0
nodesview eos06.tier2-kol.res.in:1095 Kolkata::EOS2 online on off 0 10 120 2 0
nodesview eos07.tier2-kol.res.in:1095 Kolkata::EOS2 online on off 0 10 120 1 0
nodesview eos08.tier2-kol.res.in:1095 Kolkata::EOS2 online on off 0 10 120 1 0
nodesview eos09.tier2-kol.res.in:1095 Kolkata::EOS2 online on off 0 10 120 2 0
nodesview eos10.tier2-kol.res.in:1095 Kolkata::EOS2 online on off 0 10 120 2 0

[root@eos-mgm ~]#
+++++++++++++++

Now we have check the eos space , eos group and eos fs , but there are no output. It horrifying!

[root@eos-mgm ~]# eos -b group ls
[root@eos-mgm ~]# eos -b space ls
[root@eos-mgm ~]# eos -b fs ls
[root@eos-mgm ~]#

How we can get recover those group and fs?

==================

Also, in eos-slave (SLAVE), an activated column of “eos -b node ls” are shown as “???” for all FSTs.

[root@eos-slave ~]# eos node ls
┌──────────┬────────────────────────────────┬────────────────┬──────────┬────────────┬──────┬──────────┬────────┬────────┬────────────────┬─────┐
│type │ hostport│ geotag│ status│ activated│ txgw│ gw-queued│ gw-ntx│ gw-rate│ heartbeatdelta│ nofs│
└──────────┴────────────────────────────────┴────────────────┴──────────┴────────────┴──────┴──────────┴────────┴────────┴────────────────┴─────┘
nodesview eos04.tier2-kol.res.in:1095 Kolkata::EOS2 online ??? off 0 10 120 2 0
nodesview eos05.tier2-kol.res.in:1095 Kolkata::EOS2 online ??? off 0 10 120 1 0


This is showing that, changes in eos-mgm (manager) will not reflect to eos-slave (slave).

How did eos-mgm config reflect to eos-slave?

Kindly suggest.

Regards
Prasun

Hi Prasun,

Did you also convert he configuration to QuarkDB?
What is the output when executing this command on your QuarkDB leader node?
redis-cli -p 7777 hgetall eos-config:default

If you don’t have anything then please use the eos-config-inspect tool to export the file based config to QuarkDB:
https://eos-docs.web.cern.ch/configuration/master_quarkdb.html

I have enabled the config in QuarkDB on my instance for example:

[esindril@esdss000 build_ninja_asan]$ grep cfg /etc/xrd.cf.mgm
mgmofs.cfgtype quarkdb
#mgmofs.cfgtype file

Also can you confirm you have something along these lines in your configuration (my MGM is esdss000.cern.ch) :

[esindril@esdss000 build_ninja_asan]$ grep qdb /etc/xrd.cf.mgm
mgmofs.qdbcluster localhost:7777
mgmofs.qdbpassword_file /etc/eos.keytab
[esindril@esdss000 build_ninja_asan]$ grep MASTER /etc/sysconfig/eos_env
EOS_USE_QDB_MASTER=1
EOS_MGM_MASTER1=esdss000.cern.ch
EOS_MGM_MASTER2=esdss000.cern.ch

For the beginning just use one MGM daemon and you can start the rest of the slaves later on, when you have stable situation.

Cheers,
Elvin

Dear Elvin,

The output of redis-cli -p 7777 hgetall eos-config:default on QuarkDB leader node are below:-

[root@eos-mgm ~]# redis-cli -p 7777 hgetall eos-config:default

  1. “global:/config/eosalicekolkata/node/eos04.tier2-kol.res.in:1095#stat.hostport”
  2. “eos04.tier2-kol.res.in:1095”
  3. “global:/config/eosalicekolkata/node/eos04.tier2-kol.res.in:1095#status”
  4. “on”
  5. “global:/config/eosalicekolkata/node/eos05.tier2-kol.res.in:1095#stat.hostport”
  6. “eos05.tier2-kol.res.in:1095”
  7. “global:/config/eosalicekolkata/node/eos05.tier2-kol.res.in:1095#status”
  8. “on”
  9. “global:/config/eosalicekolkata/node/eos06.tier2-kol.res.in:1095#stat.hostport”
  10. “eos06.tier2-kol.res.in:1095”
  11. “global:/config/eosalicekolkata/node/eos06.tier2-kol.res.in:1095#status”
  12. “on”
  13. “global:/config/eosalicekolkata/node/eos07.tier2-kol.res.in:1095#stat.hostport”
  14. “eos07.tier2-kol.res.in:1095”
  15. “global:/config/eosalicekolkata/node/eos07.tier2-kol.res.in:1095#status”
  16. “on”
  17. “global:/config/eosalicekolkata/node/eos08.tier2-kol.res.in:1095#stat.hostport”
  18. “eos08.tier2-kol.res.in:1095”
  19. “global:/config/eosalicekolkata/node/eos08.tier2-kol.res.in:1095#status”
  20. “on”
  21. “global:/config/eosalicekolkata/node/eos09.tier2-kol.res.in:1095#stat.hostport”
  22. “eos09.tier2-kol.res.in:1095”
  23. “global:/config/eosalicekolkata/node/eos09.tier2-kol.res.in:1095#status”
  24. “on”
  25. “global:/config/eosalicekolkata/node/eos10.tier2-kol.res.in:1095#stat.hostport”
  26. “eos10.tier2-kol.res.in:1095”
  27. “global:/config/eosalicekolkata/node/eos10.tier2-kol.res.in:1095#status”
  28. “on”
    [root@eos-mgm ~]#

Config of QuarkDB in /etc/xrd.cf.mgm and Master in /etc/sysconfig/eos_env are :-

[root@eos-mgm ~]# grep cfg /etc/xrd.cf.mgm

mgmofs.qoscfg /var/eos/qos/qos.conf

mgmofs.cfgtype file

mgmofs.cfgtype quarkdb
[root@eos-mgm ~]#

[root@eos-mgm ~]# grep qdb /etc/xrd.cf.mgm
#mgmofs.qdbcluster localhost:7777
#mgmofs.qdbpassword_file /etc/eos.keytab
mgmofs.qdbcluster eos-mgm.tier2-kol.res.in:7777 eos-slave.tier2-kol.res.in:7777 eos-qdb.tier2-kol.res.in:7777
mgmofs.qdbpassword_file /etc/eos.keytab
[root@eos-mgm ~]#
[root@eos-mgm ~]# grep MASTER /etc/sysconfig/eos_env
EOS_MGM_MASTER1=eos-mgm.tier2-kol.res.in
EOS_MGM_MASTER2=eos-slave.tier2-kol.res.in
EOS_USE_QDB_MASTER=1
[root@eos-mgm ~]#

=========================================

As per “Export instance configuration to QuarkDB” in your mentioned link i.e. Master/Slave QuarkDB Configuration — EOS CITRINE documentation, use of “eos config export” or “eos-config-inspect” tool to export the configuration.

Example:-
eos-config-inspect export --source /var/eos/config/[fqdn_of_current_master]/default.eoscf --members cluster-qdb:7777 --password-file /etc/eos.keytab

I already run above tools, but it fails. Actually I am , may be , unable to define “–members cluste-qdb:7777” parameter. How I define cluster-qdb? As per my understanding , it has point to all 3 quarkdb hosts. Is it ?

Suggest accordingly.

Regards
Prasun

Dear Elvin,

We try to export the configuration of default.eoscf to quarkdb by “eos config export”. but it not run.

[root@eos-mgm ~]# eos config export -f /var/eos/config/eos-mgm.tier2-kol.res.in/default.eoscf
usage:
config changelog|dump|export|load|ls|reset|save [OPTIONS]
‘[eos] config’ provides the configuration interface to EOS.

Subcommands:
config changelog [-#lines] : show the last #lines from the changelog - default is 10

config dump [] : dump configuration with name or current one by default

config export [-f] : export a configuration stored on file to QuarkDB (you need to specify the full path!)
-f : overwrite existing config name and create a timestamped backup

config load : load config

config ls [-b|–backup] : list existing configurations
-b : show also backup & autosave files

config reset : reset all configuration to empty state

config save [-f] [-c|–comment “”] : save config under
-f : overwrite existing config name and create a timestamped backup
-c : add a comment entry to the config

[root@eos-mgm ~]#

Also, trying to export by eos-config-inspect, but it fails again after reset.

[root@eos-mgm ~]# eos-config-inspect export --source /var/eos/config/eos-mgm.tier2-kol.res.in/default.eoscf --members eos-mgm.tier2-kol.res.in:7777 --password-file /etc/eos.keytab
— Successfully parsed configuration file
ERROR: (22): There’s MGM configuration stored in QDB already – will not delete.
[root@eos-mgm ~]# eos config reset
success: configuration has been reset(cleaned)!
[root@eos-mgm ~]# eos-config-inspect export --source /var/eos/config/eos-mgm.tier2-kol.res.in/default.eoscf --members eos-mgm.tier2-kol.res.in:7777 --password-file /etc/eos.keytab
— Successfully parsed configuration file
ERROR: (22): There’s MGM configuration stored in QDB already – will not delete.
[root@eos-mgm ~]#

Kindly Suggest.

Regards
Prasun

Dear Elvin,

We do it.

[root@eos-mgm ~]# eos-config-inspect export --source /var/eos/config/eos-mgm.tier2-kol.res.in/default.eoscf --members eos-mgm.tier2-kol.res.in:7777 --password-file /etc/eos.keytab --overwrite
— Successfully parsed configuration file
— Operation successful - wrote configuration ‘default’ with 307 entries
[root@eos-mgm ~]#

[root@eos-mgm ~]# eos fs ls
┌────────────────────────┬────┬──────┬────────────────────────────────┬────────────────┬────────────────┬────────────┬──────────────┬────────────┬────────┬────────────────┐
│host │port│ id│ path│ schedgroup│ geotag│ boot│ configstatus│ drain│ active│ health│
└────────────────────────┴────┴──────┴────────────────────────────────┴────────────────┴────────────────┴────────────┴──────────────┴────────────┴────────┴────────────────┘
eos04.tier2-kol.res.in 1095 1 /xdata0 default.0 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 2 /xdata0 default.0 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 3 /xdata0 default.0 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 4 /xdata0 default.0 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 5 /xdata0 default.0 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 6 /xdata0 default.0 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 7 /xdata0 default.0 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 8 /xdata1 default.1 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 9 /xdata1 default.1 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 10 /xdata1 default.1 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 11 /xdata1 default.1 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 12 /xdata1 default.1 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 13 /xdata1 default.1 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 14 /xdata1 default.1 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 15 /xdata10 default.2 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 16 /xdata10 default.2 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 17 /xdata10 default.2 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 18 /xdata10 default.2 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 19 /xdata10 default.2 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 20 /xdata10 default.2 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 21 /xdata10 default.2 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 22 /xdata11 default.3 rw nodrain online
eos08.tier2-kol.res.in 1095 23 /xdata11 default.3 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 24 /xdata11 default.3 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 25 /xdata11 default.3 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 26 /xdata11 default.3 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 27 /xdata11 default.3 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 28 /xdata11 default.3 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 29 /xdata12 default.4 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 30 /xdata12 default.4 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 31 /xdata12 default.4 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 32 /xdata12 default.4 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 33 /xdata12 default.4 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 34 /xdata12 default.4 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 35 /xdata12 default.4 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 36 /xdata13 default.5 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 37 /xdata13 default.5 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 38 /xdata13 default.5 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 39 /xdata13 default.5 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 40 /xdata13 default.5 rw nodrain online
eos10.tier2-kol.res.in 1095 41 /xdata13 default.5 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 42 /xdata13 default.5 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 43 /xdata14 default.6 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 44 /xdata14 default.6 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 45 /xdata14 default.6 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 46 /xdata14 default.6 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 47 /xdata14 default.6 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 48 /xdata14 default.6 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 49 /xdata14 default.6 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 50 /xdata15 default.7 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 51 /xdata15 default.7 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 52 /xdata15 default.7 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 53 /xdata15 default.7 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 54 /xdata15 default.7 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 55 /xdata15 default.7 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 56 /xdata15 default.7 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 57 /xdata2 default.8 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 58 /xdata2 default.8 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 59 /xdata2 default.8 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 60 /xdata2 default.8 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 61 /xdata2 default.8 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 62 /xdata2 default.8 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 63 /xdata2 default.8 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 64 /xdata3 default.9 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 65 /xdata3 default.9 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 66 /xdata3 default.9 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 67 /xdata3 default.9 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 68 /xdata3 default.9 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 69 /xdata3 default.9 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 70 /xdata3 default.9 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 71 /xdata4 default.10 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 72 /xdata4 default.10 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 73 /xdata4 default.10 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 74 /xdata4 default.10 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 75 /xdata4 default.10 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 76 /xdata4 default.10 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 77 /xdata4 default.10 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 78 /xdata5 default.11 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 79 /xdata5 default.11 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 80 /xdata5 default.11 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 81 /xdata5 default.11 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 82 /xdata5 default.11 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 83 /xdata5 default.11 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 84 /xdata5 default.11 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 85 /xdata6 default.12 rw nodrain online
eos08.tier2-kol.res.in 1095 86 /xdata6 default.12 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 87 /xdata6 default.12 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 88 /xdata6 default.12 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 89 /xdata6 default.12 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 90 /xdata6 default.12 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 91 /xdata6 default.12 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 92 /xdata7 default.13 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 93 /xdata7 default.13 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 94 /xdata7 default.13 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 95 /xdata7 default.13 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 96 /xdata7 default.13 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 97 /xdata7 default.13 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 98 /xdata7 default.13 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 99 /xdata8 default.14 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 100 /xdata8 default.14 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 101 /xdata8 default.14 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 102 /xdata8 default.14 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 103 /xdata8 default.14 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 104 /xdata8 default.14 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 105 /xdata8 default.14 Kolkata::EOS2 booted rw nodrain online N/A
eos04.tier2-kol.res.in 1095 106 /xdata9 default.15 Kolkata::EOS2 booted rw nodrain online N/A
eos08.tier2-kol.res.in 1095 107 /xdata9 default.15 Kolkata::EOS2 booted rw nodrain online N/A
eos05.tier2-kol.res.in 1095 108 /xdata9 default.15 Kolkata::EOS2 booted rw nodrain online N/A
eos07.tier2-kol.res.in 1095 109 /xdata9 default.15 Kolkata::EOS2 booted rw nodrain online N/A
eos06.tier2-kol.res.in 1095 110 /xdata9 default.15 Kolkata::EOS2 booted rw nodrain online N/A
eos10.tier2-kol.res.in 1095 111 /xdata9 default.15 Kolkata::EOS2 booted rw nodrain online N/A
eos09.tier2-kol.res.in 1095 112 /xdata9 default.15 Kolkata::EOS2 booted rw nodrain online N/A

[root@eos-mgm ~]# eos group ls
┌──────────┬────────────────┬────────────┬──────┬────────────┬────────────┬────────────┬──────────┬──────────┐
│type │ name│ status│ N(fs)│ dev(filled)│ avg(filled)│ sig(filled)│ balancing│ bal-shd│
└──────────┴────────────────┴────────────┴──────┴────────────┴────────────┴────────────┴──────────┴──────────┘
groupview default.0 on 7 0.57 29.61 0.28 idle 0
groupview default.1 on 7 0.51 29.32 0.27 idle 0
groupview default.10 on 7 0.62 29.26 0.30 idle 0
groupview default.11 on 7 0.55 29.21 0.28 idle 0
groupview default.12 on 7 0.45 29.29 0.22 idle 0
groupview default.13 on 7 0.58 29.14 0.30 idle 0
groupview default.14 on 7 0.24 29.22 0.12 idle 0
groupview default.15 on 7 0.65 29.22 0.31 idle 0
groupview default.2 on 7 0.45 29.31 0.25 idle 0
groupview default.3 on 7 0.30 29.55 0.16 idle 0
groupview default.4 on 7 0.40 29.44 0.23 idle 0
groupview default.5 on 7 0.50 20.19 0.27 idle 0
groupview default.6 on 7 0.56 29.43 0.26 idle 0
groupview default.7 on 7 15.54 26.86 6.35 idle 10
groupview default.8 on 7 0.59 29.45 0.28 idle 0
groupview default.9 on 7 0.60 29.26 0.28 idle 0

[root@eos-mgm ~]#

Now it show the output of “eos group ls” and “eos fs ls”. It show some errors , Let me check to those errors.

Thank for your great help.

Regards
Prasun

Hi Prasun,

Great job! Happy to hear that!
Let me know if there is anything I can help out with.

Cheers,
Elvin

Dear Elvin,

Your guide is appreciated.

I have still working on Master-Slave replication. Earlier we use EOS_MGM_HOST_TARGET and EOS_MGM_HOST parameters in eos_env and sync role for Master-Slave replication.

After moved to QuarkDB, we had comment out EOS_MGM_HOST_TARGET and remove sync role from eos_env Only mgm and mq role are defined. Also, reconfigure xrd.cf.mgm and xrd.cf.mq with quarkdb cluster. But replication between Master and Slave are failed. The output of “eos group ls” and “eos fs ls” are shown in eos-mgm (Master) but not shown in eos-slave (SLAVE).

We also follow Master-Slave document in EOS website. But, something I have miss.

So help me to resolve this issue.

Regards
Prasun

Hi Prasun,

Sorry for the late reply, I was sick last week. So the current status is that you have the master MGM working properly but the slave MGM does not properly show the groups and file systems.

Can you check that you also have the following configuration for the MQ daemons on both the master and the slave?

[esindril@esdss000 build_ninja_asan]$ less /etc/xrd.cf.mq | grep qdb
mq.qdbcluster esdss000.cern.ch:7777
mq.qdbpassword_file /etc/eos.keytab

This bit is needed to have the synchronization properly working between the two. Otherwise, let me know what is still not working and we can take one by one.

Cheers,
Elvin