FST log complains that key=stat.sys.xrootd.version uses an empty value! and fsctl function does not work

1

Dear Experts,

Another key seems missing in our set of FST and the logs say that

key=stat.sys.xrootd.version uses an empty value!

The list of installed package related to xrootd is following:

sh-5.1# dnf list installed \*xrootd\*
Installed Packages
eos-xrootd.x86_64                               5.6.6-2.el9                          @storage-ci.web.cern.ch_storage-ci_eos_diopside-depend_el-9_x86_64_
xrootd-client.x86_64                            1:5.6.4-1.el9                        @epel
xrootd-client-libs.x86_64                       1:5.6.4-1.el9                        @epel
xrootd-libs.x86_64                              1:5.6.4-1.el9                        @epel
xrootd-server-libs.x86_64                       1:5.6.4-1.el9                        @epel

Would you kindly let me know which essential pkgs are missing?

And while we are looking at FST logs, sometimes it complains that

240131 02:18:54 023 FstOfs_fsctl: root.2364:56@jbod-mgmt-10 Unable to execute fsctl function ; operation not permitted

It will be helpful to know the root cause of this.

Thank you.

Best regards,
Sang-Un

2

Hi Sang-Un,

There is nothing wrong with your deployment when it comes to the xrootd version, it was just a bug on your side that will be fixed in the next release as it’s already addressed by this commit:
https://gitlab.cern.ch/dss/eos/-/commit/840f5a88db74fd5d0921edea2c8d964b56e09b03

For the log line related to fsctl function this is normally the result of something trying to do an XRootD query againt the FST daemon. We only support a limited set of query commands that the FST can reply to. For example this query would trigger the same log signature that you have:

 sudo xrdfs root://localhost:2001 query space /eos
[ERROR] Server responded with an error: [3010] Unable to execute fsctl function ; operation not permitted

This leaves the following sigature in the logs:

240131 09:09:02 24285 XrootdXeq: root.27136:35@localhost pvt IP64 login as root
240131 09:09:02 24285 FstOfs_fsctl: root.27136:35@localhost Unable to execute fsctl function ; operation not permitted
240131 09:09:02 24285 XrootdXeq: root.27136:35@localhost disc 0:00:00

I have pushed the following commit to actually display the command and the arguments passed to this call so we can more easily identify in the next release the exact command.
https://gitlab.cern.ch/dss/eos/-/commit/b1962bc644043abcfbd2e50ad2d74f42ebe6e291

Using this the output in the logs looks like this:

240131 09:17:00 31857 XrootdXeq: root.988:60@localhost pvt IP64 login as root
240131 09:17:00 31857 FstOfs_fsctl: root.988:60@localhost Unable to execute fsctl function cmd=3 args="/eos"; operation not permitted
240131 09:17:00 31857 XrootdXeq: root.988:60@localhost disc 0:00:00

Once you’ll have the next release in production (5.2.9) we can more easily identify who/what is calling this API.

Cheers,
Elvin

1 Like
3

Hi Elvin,

Thanks a lot for kind explanation, fix and update. I will try it as soon as it is released.

Best regards,
Sang-Un