Xrd.sched ignored?

Site Administrators
1

In the past we have defined xrd.sched on FSTs within /etc/xrd.cf.fst which successfully throttled the number of concurrent handler threads.

In recent version of xrootd (4.8.3 / 4.8.4) with Citrine 4.2.24 this no longer appears to take effect:

19:13:34 # grep xrd.sch /etc/xrd.cf.fst
xrd.sched mint 8 maxt 256 idle 60

19:14:12 # netstat -tonp | grep ESTABLISHED | grep xrootd | wc -l
549

This occurs with xrootd versions 4.8.3 and 4.8.4 from the xrootd-stable repo http://xrootd.org/binaries/stable/slc/6/x86_64/

xrd.sch appears supported in 4.8 per http://xrootd.org/doc/dev48/xrd_config.htm#_Toc496911324

No fd is open for the config file (expected?) though one is for the logfile

19:36:41 # ps aux | grep eos
daemon 3012 77.0 0.2 1104568 289620 ? SLl 19:36 0:16 /usr//bin/xrootd -n fst -c /etc/xrd.cf.fst -l /var/log/eos/xrdlog.fst -b -Rdaemon
daemon 3053 0.0 0.0 356808 47212 ? S 19:36 0:00 /usr//bin/xrootd -n fst -c /etc/xrd.cf.fst -l /var/log/eos/xrdlog.fst -b -Rdaemon

19:38:39 # lsof -p 3012 | grep xrdlog
xrootd 3012 daemon 2w REG 0,23 1030207319 384 /eos/log/fst/xrdlog.fst

19:38:49 # lsof -p 3012 | grep xrd.cf
(null)

Any ideas why xrd.sch seems be be ignored, or is there another method that should be used for such throttling at the FST?

Cheers,
Pete

2

Are we misinterpreting the use of the xrd.sched directive with eos with these versions of xrootd?

Can another site validate if this directive is honored or not?

Thank you,
Pete

3

Hey Pete,

Quick question. Is this only occurring on FST nodes or are you seeing the same behavior on the MGM nodes as well?

I might be able to give this a try. I’m running xroot 4.8.4-1 and EOS 4.2.28 on a few test nodes. I’ll let you know.

–
Dan Szkola
FNAL

4

Hi Pete,
there are many more threads running in the EOS FST, not only the XRootD IO threads. You have to take a strack trace with GDB and have to count the one corresponding to the XRootD thread pool. There is already one thread per disk aso. … Cheers Andreas.

5

Hi Andreas,

Thanks for the input.

I enable xrootd summary monitoring on the FST’s /etc/xrd.cf.fst via

And started a listener on the mgm via: mpxstats -f flat -p 3333

I see that indeed the xrootd summary reported values for sched.threads does not exceed the values defined by xrd.sched, and link.num essentially matches the netstat connections:

So yep, the xrd.sched directive appears to be honored and I was merely counting the incorrect thing.

The xrootd summary monitoring is helpful, and looks good.

Thank you,
Pete

6

Thanks Dan, looks like I have it now.

Cheers,
Pete