Meaning of eth values in eos fs ls --io

Site Administrators
1

In the output below the values represented in the columns eth-MiB/s│ ethi-MiB│ etho-MiB seem odd:

  • One FST shows eth-MiB/s values of ‘119’ while the other FSTs all show ‘1192’
  • The ethi-MiB values for all fsids are either 0, 6, or 26 depending on the FST - seems rather odd.

What is the ‘eth-MiB/s’ value intended to represent, and what underlaying config difference between FSTs might account for the above variations (the hardware, EOS versions, and network configs seem identical.)

Cheers,
Pete

root@alice-eos-01.ornl.gov:~
18:00:23 # eos fs ls --io
┌────────────────────────────────┬──────┬────────────────┬────────────────┬──────────┬────────────┬────────────┬──────────┬──────────┬──────────┬──────┬──────┬────────────┬────────────┬────────────┬───────────┬──────────┬───────
│hostport                        │    id│      schedgroup│          geotag│  diskload│  diskr-MB/s│  diskw-MB/s│ eth-MiB/s│  ethi-MiB│  etho-MiB│ ropen│ wopen│  used-bytes│   max-bytes│  used-files│  max-files│   bal-shd│     dr
└────────────────────────────────┴──────┴────────────────┴────────────────┴──────────┴────────────┴────────────┴──────────┴──────────┴──────────┴──────┴──────┴────────────┴────────────┴────────────┴───────────┴──────────┴───────
 warp-ornl-cern-01.ornl.gov:1095      25        default.0       CADES_ah72       0.00         0.00         0.03        119          0          0     45      0     35.11 TB     37.38 TB      10.28 K      4.44 G          0        
 warp-ornl-cern-01.ornl.gov:1095      26        default.1       CADES_ah72       0.00         0.00         0.03        119          0          0     23      0     33.29 TB     37.38 TB       9.53 K      8.00 G          0        
 warp-ornl-cern-01.ornl.gov:1095      27        default.2       CADES_ah72       0.00         0.00         0.03        119          0          0     14      0     35.57 TB     37.38 TB      11.99 K      3.54 G          0        
 warp-ornl-cern-01.ornl.gov:1095      28        default.3       CADES_ah72       0.00         0.00         0.03        119          0          0      4      0     36.76 TB     37.38 TB      10.55 K      1.22 G          0        
 warp-ornl-cern-01.ornl.gov:1095      29        default.4       CADES_ah72       0.00         0.00         0.03        119          0          0      5      0     36.43 TB     37.38 TB      10.20 K      1.87 G          0        
 warp-ornl-cern-01.ornl.gov:1095      30        default.5       CADES_ah72       0.00         0.00         0.03        119          0          0      8      0     36.37 TB     37.38 TB      10.25 K      1.99 G          0        
 warp-ornl-cern-01.ornl.gov:1095      31        default.6       CADES_ah72       0.00         0.00         0.03        119          0          0      6      0     36.30 TB     37.38 TB       9.77 K      2.12 G          0        
 warp-ornl-cern-02.ornl.gov:1095      32        default.0       CADES_ah72       0.01         0.00         0.04       1192          6          0      0      0      1.24 TB     30.76 TB      31.70 K     57.67 G          0        
 warp-ornl-cern-02.ornl.gov:1095      33        default.1       CADES_ah72       0.01         0.00         0.04       1192          6          0      0      3     92.49 GB     30.76 TB       2.83 K     59.90 G          0        
 warp-ornl-cern-02.ornl.gov:1095      34        default.2       CADES_ah72       0.01         0.00         0.04       1192          6          0      0      0     71.20 GB     30.76 TB       1.83 K     59.95 G          0        
 warp-ornl-cern-02.ornl.gov:1095      35        default.3       CADES_ah72       0.01         0.00         0.04       1192          6          0      0      0    262.08 GB     30.76 TB       6.72 K     59.57 G          0        
 warp-ornl-cern-02.ornl.gov:1095      36        default.4       CADES_ah72       0.01         0.00         0.04       1192          6          0      0      0     57.34 KB     30.76 TB            0     60.09 G          0        
 warp-ornl-cern-02.ornl.gov:1095      37        default.5       CADES_ah72       0.01         0.00         0.04       1192          6          0      0      0     57.34 KB     30.76 TB            0     60.09 G          0        
 warp-ornl-cern-02.ornl.gov:1095      38        default.6       CADES_ah72       0.01         0.00         0.04       1192          6          0      0      0     57.34 KB     30.76 TB            0     60.09 G          0        
 warp-ornl-cern-02.ornl.gov:1095      39        default.7       CADES_ah72       0.01         0.00         0.04       1192          6          0      0      0     57.34 KB     30.76 TB            0     60.09 G          0        
 warp-ornl-cern-03.ornl.gov:1095      40        default.0       CADES_ah72       0.02         0.00         0.04       1192         26          0      1      0      1.20 TB     30.76 TB      29.23 K     57.75 G          0        
 warp-ornl-cern-03.ornl.gov:1095      41        default.1       CADES_ah72       0.02         0.00         0.04       1192         26          0      0      2     94.27 GB     30.76 TB       2.71 K     59.90 G          0        
 warp-ornl-cern-03.ornl.gov:1095      42        default.2       CADES_ah72       0.02         0.00         0.04       1192         26          0      0      0     74.79 GB     30.76 TB       1.78 K     59.94 G          0        
 warp-ornl-cern-03.ornl.gov:1095      43        default.3       CADES_ah72       0.02         0.00         0.04       1192         26          0      0      0    275.80 GB     30.76 TB       6.57 K     59.55 G          0        
 warp-ornl-cern-03.ornl.gov:1095      44        default.4       CADES_ah72       0.02         0.00         0.04       1192         26          0      0      0    262.68 MB     30.76 TB           61     60.08 G          0        
 warp-ornl-cern-03.ornl.gov:1095      45        default.5       CADES_ah72       0.02         0.00         0.04       1192         26          0      0      0    463.08 MB     30.76 TB           62     60.08 G          0        
 warp-ornl-cern-03.ornl.gov:1095      46        default.6       CADES_ah72       0.02         0.00         0.04       1192         26          0      0      0    963.32 MB     30.76 TB           67     60.08 G          0        
 warp-ornl-cern-03.ornl.gov:1095      47        default.7       CADES_ah72       0.02         0.00         0.04       1192         26          0      0      0     57.34 KB     30.76 TB            0     60.09 G          0
2

I’ll take a stab at this and CERN folk can correct my mistakes. :slight_smile:

I’d guess from looking at those numbers that warp-ornl-cern-01.ornl.gov has a one gigabit ethernet interface and warp-ornl-cern-02.ornl.gov and warp-ornl-cern-03.ornl.gov are 10 gigabit interfaces. My test nodes (all 1GbE) show 119, and my prod nodes (10 GbE) all show 1192. I think it is supposed to indicate the speed of the monitored interface in MiB.

The reason they are all 0, 6, or 26 is those are per node numbers so they will be the same for all FS on the same node. The ethi-MiB and etho-MiB represent inbound and outbound bytes on the primary interface of the node.

The warp-ornl-cern-01.ornl.gov numbers may be 0 if the node is not using eth0 as the primary interface. If not using eth0, you need to specify the primary interface name in the config on the FST or the monitoring will be wrong:

# Network interface to monitor
export EOS_FST_NETWORK_INTERFACE=“eth1”

Or that node could just be idle if the interface is, in fact, eth0.

–
Dan Szkola
FNAL

3

Hi Dan,

Thanks for pointing us in the right direction.

This may be either a bug, or a nuance of eos bonded interface detection / naming in CentOS 6.9.

The FST (warp-ornl-cern-01) that was reporting gigabit speed (eth-MiB/s value 119) in fact had a single br0 10G bond. The OS showed this to be 10G, and br0 was the only interface showing TX/RX activity. (The 1G interface is on a mgmt vlan, and not used by eos.)

We verified /etc/sysconfig/eos was using export EOS_FST_NETWORK_INTERFACE=“br0” but this still appeared to the mgm as gigabit. Eventually we just removed the bond, and presented one of the 10G interfaces directly and the mgm now shows an eth-MiB/s value of 1192

It would be nice to get bonded 10G working with eos detecting the correct bonded eth-MiB/s, as some of our newer FSTs with 10T drives host .5PB of data each, but we’ll revisit this once we are ready to re-implement bonding.

Cheers,
Pete

4

Hi Pete, Dan, and *,

Running eos-server-4.4.18 with 20 FSTs which has 10Gb/s ‘p2p1’ ethernet interfaces, I set export EOS_FST_NETWORK_INTERFACE="p2p1" in /etc/sysconfig/eos_env and I still have no value in diskload, diskr-MB/s, diskw-MB/s, ethi-MiB,etho-MiB columns :

This is the same problem for group ls --io, node ls --io, fs ls --io :

[root@np02eos1 ~]# eos node ls --io
┌────────────────────────────────┬────────────────┬──────────┬────────────┬────────────┬──────────┬──────────┬──────────┬──────┬──────┬────────────┬────────────┬────────────┬───────────┬──────────┬──────────┬──────────┬──────┬─────────┐
│hostport                        │          geotag│  diskload│  diskr-MB/s│  diskw-MB/s│ eth-MiB/s│  ethi-MiB│  etho-MiB│ ropen│ wopen│  used-bytes│   max-bytes│  used-files│  max-files│   bal-shd│ drain-shd│  gw-queue│  iops│       bw│
└────────────────────────────────┴────────────────┴──────────┴────────────┴────────────┴──────────┴──────────┴──────────┴──────┴──────┴────────────┴────────────┴────────────┴───────────┴──────────┴──────────┴──────────┴──────┴─────────┘
 np02ss00.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      1     12.74 TB     46.18 TB       7.42 K      4.51 G          0          0          0    142   1930 MB 
 np02ss01.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      2     12.72 TB     46.18 TB       7.38 K      4.51 G          0          0          0    119    640 MB 
 np02ss02.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      1     12.72 TB     46.18 TB       7.40 K      4.51 G          0          0          0    129   1971 MB 
 np02ss03.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      1     12.68 TB     46.18 TB       7.37 K      4.51 G          0          0          0    103    423 MB 
 np02ss04.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      1     12.69 TB     46.18 TB       7.38 K      4.51 G          0          0          0    115    871 MB 
 np02ss05.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      1     12.72 TB     46.18 TB       7.41 K      4.51 G          0          0          0    130   1051 MB 
 np02ss06.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      0     12.67 TB     46.18 TB       7.40 K      4.51 G          0          0          0    128    681 MB 
 np02ss07.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      2     12.69 TB     46.18 TB       7.38 K      4.51 G          0          0          0    110    500 MB 
 np02ss08.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      2     12.67 TB     46.18 TB       7.39 K      4.51 G          0          0          0    132   1310 MB 
 np02ss09.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      0     12.71 TB     46.18 TB       7.39 K      4.51 G          0          0          0    132   1374 MB 
 np02ss10.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      0     12.68 TB     46.18 TB       7.40 K      4.51 G          0          0          0    112    368 MB 
 np02ss11.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      2     12.68 TB     46.18 TB       7.40 K      4.51 G          0          0          0    109    410 MB 
 np02ss12.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      1     12.63 TB     46.18 TB       7.39 K      4.51 G          0          0          0    130   1275 MB 
 np02ss13.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      2     12.75 TB     46.18 TB       7.41 K      4.51 G          0          0          0    112    441 MB 
 np02ss14.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      0     12.69 TB     46.18 TB       7.40 K      4.51 G          0          0          0    113    452 MB 
 np02ss15.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      1     12.64 TB     46.18 TB       7.38 K      4.51 G          0          0          0    111    405 MB 
 np02ss16.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      2     12.70 TB     46.18 TB       7.38 K      4.51 G          0          0          0    118    551 MB 
 np02ss17.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0      0     12.65 TB     46.18 TB       7.37 K      4.51 G          0          0          0    110    536 MB 
 np02ss18.cern.ch:1095                    np02-daq       0.00            0            0       1192          0          0      0     11     12.97 TB     46.18 TB       7.53 K      4.51 G          0          0          0    113    670 MB

I think I have something wrong in my config, I don’t see what. I’m looking for your suggestions.
Best regards,
Denis

5

For /etc/sysconfig/eos:

# Network interface to monitor
export EOS_FST_NETWORK_INTERFACE=“eth1"

For /etc/sysconfig/eos_env:

# Network interface to monitor
EOS_FST_NETWORK_INTERFACE=“eth1"

Note the lack of the ‘export’ directive for the systemd version of the config file.

The lack of diskload stats is a bit concerning, the network interface variable shouldn’t affect that as far as I know, but maybe some CERN folks can comment.

—

Dan Szkola

FNAL

6

Hi Dan,

Thank you, my mistake was the export in the systemd /etc/sysconfig/eos_env. In my case, the :

EOS_FST_NETWORK_INTERFACE="p2p1"

allows the ethi-MiB, etho-MiB columns to be filled in the fs ls --io, node ls --io, group ls --io and space ls --io.

But I still see two problems :

  • the diskr-MB/s and the diskw-MB/s show zeros

  • the space ls --io numbers in the ethi-MiB and etho-MiB columns are not consistents with the numbers shows in the other (node|fs|group) ls --io commands

    [root@np02eos1 ~]# eos group ls --io ; eos space ls --io
    ┌────────────────┬──────────┬────────────┬────────────┬──────────┬──────────┬──────────┬──────┬──────┬────────────┬────────────┬────────────┬───────────┬──────────┬──────────┐
    │name │ diskload│ diskr-MB/s│ diskw-MB/s│ eth-MiB/s│ ethi-MiB│ etho-MiB│ ropen│ wopen│ used-bytes│ max-bytes│ used-files│ max-files│ bal-shd│ drain-shd│
    └────────────────┴──────────┴────────────┴────────────┴──────────┴──────────┴──────────┴──────┴──────┴────────────┴────────────┴────────────┴───────────┴──────────┴──────────┘

      default.0              0.00            0            0       2384        768          0      0      1     12.83 TB     46.18 TB       7.43 K      4.51 G          0          0 
  default.1              0.00            0            0       2384        768          0      0      1     12.83 TB     46.18 TB       7.43 K      4.51 G          0          0 
  default.10             0.00            0            0       2384        881          1      0      2     12.78 TB     46.18 TB       7.42 K      4.51 G          0          0 
  default.11             0.00            0            0       2384        881          1      0      1     12.78 TB     46.18 TB       7.44 K      4.51 G          0          0 
  default.12             0.00            0            0       2384        967          0      0      1     12.77 TB     46.18 TB       7.44 K      4.51 G          0          0 
  default.13             0.00            0            0       2384        967          0      0      1     12.81 TB     46.18 TB       7.43 K      4.51 G          0          0 
  default.14             0.00            0            0       2384        859          0      0      2     12.78 TB     46.18 TB       7.44 K      4.51 G          0          0 
  default.15             0.00            0            0       2384        859          0      0      1     12.76 TB     46.18 TB       7.41 K      4.51 G          0          0 
  default.16             0.00            0            0       2384        960          1      0      1     12.77 TB     46.18 TB       7.42 K      4.51 G          0          0 
  default.17             0.00            0            0       2384        960          1      0      1     12.78 TB     46.18 TB       7.40 K      4.51 G          0          0 
  default.18             0.00            0            0       1192        955          2      0      3      6.59 TB     23.09 TB       3.80 K      2.25 G          0          0 
  default.19             0.00            0            0       1192        955          2      0      3      6.58 TB     23.09 TB       3.79 K      2.25 G          0          0 
  default.2              0.00            0            0       2384        867          1      0      1     12.80 TB     46.18 TB       7.41 K      4.51 G          0          0 
  default.3              0.00            0            0       2384        867          1      0      1     12.80 TB     46.18 TB       7.43 K      4.51 G          0          0 
  default.4              0.00            0            0       2384        839          1      0      1     12.80 TB     46.18 TB       7.41 K      4.51 G          0          0 
  default.5              0.00            0            0       2384        839          1      0      0     12.81 TB     46.18 TB       7.44 K      4.51 G          0          0 
  default.6              0.00            0            0       2384        910          0      0      1     12.78 TB     46.18 TB       7.42 K      4.51 G          0          0 
  default.7              0.00            0            0       2384        910          0      0      1     12.79 TB     46.18 TB       7.43 K      4.51 G          0          0 
  default.8              0.00            0            0       2384        890          0      0      1     12.80 TB     46.18 TB       7.43 K      4.51 G          0          0 
  default.9              0.00            0            0       2384        890          0      0      1     12.79 TB     46.18 TB       7.41 K      4.51 G          0          0 


 ┌──────────┬──────────┬────────────┬────────────┬──────────┬──────────┬──────────┬──────┬──────┬────────────┬────────────┬────────────┬───────────┬──────────┬──────────┐
 │name      │  diskload│  diskr-MB/s│  diskw-MB/s│ eth-MiB/s│  ethi-MiB│  etho-MiB│ ropen│ wopen│  used-bytes│   max-bytes│  used-files│  max-files│   bal-shd│ drain-shd│
 └──────────┴──────────┴────────────┴────────────┴──────────┴──────────┴──────────┴──────┴──────┴────────────┴────────────┴────────────┴───────────┴──────────┴──────────┘
  default          0.00            0            0       1887        741          0      0     25    243.43 TB    877.33 TB     141.22 K     85.68 G          0          0

Denis