A few words about our layout: we have infiniband(IPoIB) + Ethernet networks at nodes. MGM have just Ethernet.
FQDN is resolving to IP-address in Ethernet network for general usage. And we don’t want to use infiniband by default for all activities.
The idea is to use host-name that will be resolved to infiniband address at nodes (and Ethernet address at MGM) in EOS, but keep FQDN that resolved to Ethernet address.
What you require does not work out of the box and there is no configuration setting that you can use to achieve this. This definitely requires modifications to the code and even then am I not 100% sure it will work.
Just to give you some info on how things work. The MGM will use the FQDN that the FSTs provide to redirect the clients to the FSTs - so the clients need to be able to resolve these. Therefore all access that happens on your cluster needs to be Ethernet based.
What kind of traffic are you trying to separate on the two network interfaces? Do you actually see any bottleneck by just using the Ethernet network for all activities? I am not very familiar with this setup but won’t you need the FST(XRootD) daemon to listen on both network interfaces? How do you plan on doing this?
We trying to avoid routing of ordinary traffic to Infiniband network just to avoid undetectable behavior.
At same time we want to have fast (40Gbit/s via Infniband Vs 1Gbit/s via ethernet) access to data.
To do it we have to modify /usr/sbin/eos_start.sh and /usr/lib/systemd/system/eos@.service
Service file now is using unshare to start EOS script ExecStart=/usr/bin/unshare -u /usr/sbin/eos_start.sh -n %i -c /etc/xrd.cf.%i -l /var/log/eos/xrdlog.%i -Rdaemon
And we set needed hostname in /usr/sbin/eos_start.sh
After this MGMT is seeing FST with Infiniband hostnames and clients goes to it via Infiniband for data and via Ethernet in all other cases.