Are there any thoughts about perhaps making FSTs fungible and redundant in the future, so that they can all share filesystems and handle requests interchangeably?
For EOS on CephFS, this seems like something of a holy grail for load balancing and resiliency.
It would also allow a model that is easier to fit with kubernetes (a service + deployment for FSTs would be natural, which would make scaling up FSTs much simpler, and applying FST updates could be outage-free), as opposed to having unique members of a statefulset and manually enumerating special services, one for each member of the statefulset. I understand this might be a significant architectural change.
Does anyone know if this is feasible or being considered? @apeters@ebocchi
No problem if it is not planned at this time, but if it is it would be good to know as it would change the approach to running EOS on k8s quite a lot (it would be much easier and simpler).
We have now added the stateless FST meta-data by storing it as an extended attribute.
The code to create proxy groups and schedule to them is already in, so it should be doable with few code adjustments, to run just N FSTs serving all the same files from a shared backend.
However there is still the need to pin one FST as the managing FST for a given filesystem id for all the lifecycle management tasks (draining, deletion). All the other members of the proxy group are used for IO and sharded over file IDs to distribute the load.