Stat.statfs.freebytes has not yet been defined

1

eos experts, perhaps you can assist us in determining the root cause possible solutions for this new install issue.

We installed two new eos-server-5.1.23 FSTs with identical hardware. One has been operating fine. However, since install the other has had an issue that when the eos@fst service is started /var/log/eos/fst/xrdlog.fst logs msg=“stat.statfs.freebytes has not yet been defined, not setting file system fill status”

If eos@fst is left running on this FST it will cause the mgm to become quite unresponsive, and at times has cause the eos@mgm service on the mgm node to fail, though nothing was logged (outside the prior func=GetStatfs messages)

The reason why the systfs call against the fsids is failing is curious, given EOS runs as user daemon and that user can statfs() the mounted fsids when tested for example with a simple C call to statfs(“/fsidmnt/11000/”, &buf); which returns:

[root@eos-fst-11 ~]# su -c ‘./statfs.test’ daemon
File system size: 17998070939648
Free space: 17798225084416

Similarly with python os.statvfs(f’/fsidmnt/{i}') on the fsids, this also succeeds.

Both FSTs are deployed with ansible with configs identical.
The hardware, OS, eos versions, etc. are also identical.

Any suggestions on what might be causing eos to statfs() calls to fail?
We could nuke it and reinstall fresh but wanted to inquire before such.

Thank you,
Pete

2

Hi Pete,

Can you restart this problematic FST, then put it in debug mode, leave it running for 2 min and then send me the full FST log since the restart?

Thanks,
Elvin

3

Hi Elvin, thank you for the assistance. debug level log emailed to you.

I see the fsids all appear to boot okay and log reports statfs events with both succeed and fail:

XrdMqSharedObjectManager::Set=>[/eos/eos-fst-11.ornl.gov:1095/fst/fsidmnt/11011:stat.statfs.bfree]=>4352086832 notified
… and later…
fst-11.ornl.gov:1095 tid=00007f0de2df6700 source=ErrorReport:82 tident= sec= uid=0 gid=0 name= geo=“” broadcasting errorreport message: 230725 11:37:16 time=1690299436.356674 func=GetStatfs level=ERROR logid=c723136e-2b00-11ee-9049-5c6f699f25b0 unit=fst@eos-fst-11.ornl.gov:1095 tid=00007f08a8ff9700 source=FileSystem:129 tident= sec= uid=0 gid=0 name= geo=“” cannot statfs

Thank you,
Pete

4

@peby I don’t suppose you found the solution to this?

5

Hi Ryan,

Make sure that the FST process has execute/search permissions on all parent directories leading to the concerned directory.

Cheers,
Elvin

6

Although we have the same message:

260616 21:49:36 time=1781639376.010081 func=CheckFilesystemFullness  level=INFO  logid=static.............................. unit=fst@eos-fst-2.eos-fst.eos.svc.kermes-dev.local:1095 tid=00007f93bf7fe640 source=Storage:1224                   tident= sec=(null) uid=0 gid=0 name=- geo="" xt="" ob="" msg="stat.statfs.freebytes has not yet been defined, not setting file system fill status" fsid=3

it only seems to happen once, on FST startup. And

# eos fs status 1 |grep freebytes
stat.statfs.freebytes            := 1019245232128

so I guess this message must be normal in some cases.