Newly added FSTs not available from slave


We recently added some FST nodes to our instance, and we realize that these nodes are not seen activated on the slave.
We ran the eos node set host.domain on from the master, and it correctly works. But on the slave, eos node ls shows the new nodes, but nothing is present in status column. When running eos node set host.domain on on the slave, it says :

error: nodes can only be configured as 'root' or from the node itself them using sss protocol
 (errc=1) (Operation not permitted)

We have connected some fuse clients to the slave, and they can’t read any file whose replica are all on these nodes (the majority of recently created files…) because the FS are considered offline.

Is there another way than rebooting the slave (it takes a lot of time on our instance) to have these nodes correctly activated ?

If you add nodes the slave has to reload the configuration file. New nodes are not propagated…

Hi Andreas,

Thank you for your answer.

Do you mean that we should run eos config load default on the slave ? We seldom used the config interface. Is it safe to run it or are there some drawbacks, or precaution ? I tried this on a slave node of a test instance, and the MGM became unresponsive, I had to restart it…

Is your slave serving requests? Then it is not a good idea. You could stall clients while you do this, but it is sort of ‘complicated’ operation which can interfere with notifications coming from the master and the FSTs while it reloads.

Yes, we have few clients connected to the slave of our production instance, they need only read access, and the application they serve does not cope very well when the master is busy with heavy metadata operations. Anyway, we will switch them to the master for now.

But the test instance do not have any client connected to it, and config load operation never works, the command hangs, and many further commands (space ls, node ls, fs ls, file info, file check) are unresponsive. eos ns, eos io do return results. I wonder if it doesn’t go in a deadlock situation… In case it helps, the last line of xrdlog.mgm file when doing this is :

190228 16:35:59 time=1551368159.637727 func=FsConfigListener         level=INFO  logid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx tid=00
007fe6d5628700 source=FsConfigListener:154           tident=<single-exec> sec=      uid=0 gid=0 name= geo="" Calling Apply for iostat::collect true

We will let you know what happens if we do this on the production slave.

Good morning, the eos load config default command also hanged on the production slave. When restarted the service, it correctly loaded the FST status.