EOS startup on slow networks: systemd drop-in fixes

Site Administrators
1

I wanted to share this (and maybe it could be integrated?).

We’re currently running one of our EOS instances on an InfiniBand network. The interface sometimes needs a minute or more to get fully ready. During this time, for example, the FST service fails during startup, because it can’t resolve the MGM / QDBs. Systemd tries to restart it. Eventually, this loop hits the rate-limits, causing systemd to stop attempting restarts.

Systemd (in cooperation with other services, like networkd) has proper dependency helpers for this.

We have added this on all nodes (here FST as an example):

fst:/etc/systemd/system/eos5-fst@.service.d$ cat network-deps.conf 
[Unit]
Wants = network-online.target
After = network.target network-online.target nss-lookup.target

Explanation:

  • network.target: Networking generally available. This is good practice for network services (services that listen for requests from the network).
  • network-online.target: Network access is ready. This is the right thing, if one needs access to the network (talk to QDB / MGM / etc).
  • nss-lookup.target: DNS lookups are ready. So that hostname lookups in the config are possible.
  • Wants only lists network-online.target, because that’s what the docs explain. The other targets should not be “pulled in” by consumers.