Hi Andreas,
Yes #1 sounds like what I had in mind. Great! The whole cluster is for ATLAS only so it is fine for all the jobs to have RO access to all the EOS data. Glad to hear it can be done easily.
Interesting to learn that a xrootd client can natively mount CephFS, however it might be difficult or insecure to integrate that with our Manila CephFS provider on Openstack.
As for writing, I imagine it would be problematic for a client to write a file directly into the filesystem of a FST, which it should not have privileges for anyway. I think the main benefit to be gained is from reading the input directly, however I wonder if there is also a way to potentially improve writing too, compared to a network transfer to a FST to CephFS.
Suppose CephFS is not mounted RO, but instead there are simply two locations, one writeable by EOS FST servers and one writeable by ATLAS jobs.
drwxr-xr-x 3 eos eos 40 Jan 28 2021 FST-data
drwxr-xr-x 3 atlas atlas 40 Jan 28 2021 job-data
A job could write its output data in job-data/, then notify MGM that it wants to do a copy operation of the file into EOS. However if the FST knows that the data is already available on the same CephFS filesystem, then it only needs to do ‘cp -l’, and the stage-in can be instant via a hardlink. Just an idea, what do you think? We might need to finesse the job config a bit to set the output file location in CephFS, while keeping the rest of the job work dir on local disk of the compute node.