Cephfs space is added up for each FST

Site Administrators
1

Hello,

I have 3 FSTs on CephFS. They each mount a separate subdirectory of the same filesystem and see something like this:

bash-4.2# df -B TB /cephfs
Filesystem                                                                                                      1TB-blocks  Used Available Use% Mounted on
10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789:/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2/test1     4412TB 304TB    4109TB   7% /cephfs

However:

# eos space ls                                       
┌──────────┬────────────────┬────────────┬────────────┬──────┬─────────┬───────────────┬──────────────┬─────────────┬─────────────┬──────────────┬──────┬──────────┬───────────┬───────────┬──────┬────────┬───────────┬──────┬────────┬───────────┐
│type      │            name│   groupsize│    groupmod│ N(fs)│ N(fs-rw)│ sum(usedbytes)│ sum(capacity)│ capacity(rw)│ nom.capacity│sched.capacity│ quota│ balancing│  threshold│  converter│   ntx│  active│        wfe│   ntx│  active│ intergroup│
└──────────┴────────────────┴────────────┴────────────┴──────┴─────────┴───────────────┴──────────────┴─────────────┴─────────────┴──────────────┴──────┴──────────┴───────────┴───────────┴──────┴────────┴───────────┴──────┴────────┴───────────┘
 spaceview           default            0           24      3         3       909.90 TB       13.24 PB      13.24 PB           0 B       12.33 PB    off        off          20         off      2        0         off      1        0         off

The space reported by MGM is equal to 3x the real available and used storage amounts. It must be adding up what is reported by each FST.

4412 TB * 3 = 13.236 PB
303319 GB * 3 = 909.9 TB

Is there a way to configure the FSTs so I can tell them they are all seeing the same filesystem on CephFS?

(Actually the storage reported by df on the FSTs is also incorrect but that is a separate issue.)

Thanks!

2

Actually I think this is, at least in part, a side effect of Bug #55090: mounting subvolume shows size/used bytes for entire fs, not subvolume - Linux kernel client - Ceph

But even when that is fixed, I’m not sure if it would allow us to use a single CephFS share for all the FSTs, each with their own separate subdir mounts.

3

I added this feature now, because statfs will report all files, not only the configured subtree.
The code is in the master branch and will appear with the next release.
Essentially you can then label your filesystem like:

eos fs config 1 sharedfs=cephfs1
eos fs config 2 sharedfs=cephfs1

And filesystems sharing the same labe; will not be counted several times.

See:
https://its.cern.ch/jira/browse/EOS-5281

1 Like
4

Thanks Andreas that’s great! I made some progress on the linked cephfs issue. With that and the new EOS feature it might open the way to proceed again.