Dual stack citrine but no ipv4

1

Hi all

I have a perfectly functioning ipv4 based eos for ALICE running aquamarine.
I am now trying to get another dual stacked eos with citrine and centos7 up.
I have reinstalled from scratch multiple times and never seem to be able to get it to respond to ipv4, ipv6 it does.
Even when I disabled ipv6 (boot with ipv6.disable=1), it would then start with a zeromq error, but could have been a carry over from previous installs.

I did suspect my ipv6 config was messing with things, but now I know I have a properly setup with dns working properly for ipv6, yet it still refuses to listen on ipv4. sshd and postfix happily connect to ipv4 and ipv6.

I did additionally bring up an apache server as a test and it happily served pages on both 4 and 6.

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2532/master
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1551/sshd
tcp6 0 0 ::1:25 :::* LISTEN 2532/master
tcp6 0 0 :::8000 :::* LISTEN 27692/xrootd
tcp6 0 0 :::1094 :::* LISTEN 27692/xrootd
tcp6 0 0 :::1097 :::* LISTEN 27396/xrootd
tcp6 0 0 :::1100 :::* LISTEN 27692/xrootd
tcp6 0 0 :::22 :::* LISTEN 1551/sshd

I do have a private and a public ipv4 (seperate interfaces) and the ipv6 public interface is the same interface as the private ipv4.

This happens on the mgm and fst (one of each for the current setup).

Insights or pointers would be greatly appreciated.

Thanks
Sean

[root@grid-se ]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether b8:ca:3a:6c:79:e0 brd ff:ff:ff:ff:ff:ff
inet 172.20.100.6/20 brd 172.20.111.255 scope global noprefixroute em1
valid_lft forever preferred_lft forever
inet6 2001:4200:597:1:baca:3aff:fe6c:79e0/64 scope global mngtmpaddr dynamic
valid_lft 2591997sec preferred_lft 604797sec
inet6 2001:4200:597:1::6/64 scope global noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::883e:24de:e5aa:c384/64 scope link
valid_lft forever preferred_lft forever
3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether b8:ca:3a:6c:79:e1 brd ff:ff:ff:ff:ff:ff
inet 196.24.44.235/32 brd 196.24.44.235 scope global noprefixroute em2
valid_lft forever preferred_lft forever
inet6 fe80::263f:655f:ef84:f9d5/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4: em3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether b8:ca:3a:6c:79:e2 brd ff:ff:ff:ff:ff:ff
5: em4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether b8:ca:3a:6c:79:e3 brd ff:ff:ff:ff:ff:ff
6: p3p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether a0:36:9f:30:d3:c0 brd ff:ff:ff:ff:ff:ff
7: p3p2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether a0:36:9f:30:d3:c1 brd ff:ff:ff:ff:ff:ff

2

I am in the same situation here.

Total: 263 (kernel 0)
TCP:   16 (estab 7, closed 0, orphaned 0, synrecv 0, timewait 0/0), ports 0

Transport Total     IP        IPv6
*         0         -         -
RAW       1         0         1
UDP       2         1         1
TCP       16        6         10
INET      19        7         12
FRAG      0         0         0

State       Recv-Q Send-Q Local Address:Port               Peer Address:Port           
LISTEN      0      128             *:22                          *:*
LISTEN      0      100     127.0.0.1:25                          *:*
LISTEN      0      32           [::]:8000                     [::]:*
LISTEN      0      128          [::]:50051                    [::]:*
LISTEN      0      128          [::]:1094                     [::]:*
LISTEN      0      128          [::]:1097                     [::]:*
LISTEN      0      100          [::]:1100                     [::]:*
LISTEN      0      128          [::]:22                       [::]:*
LISTEN      0      100         [::1]:25                       [::]:*

$ sudo eos version
EOS_INSTANCE=eosalice
EOS_SERVER_VERSION=4.8.31 EOS_SERVER_RELEASE=1
EOS_CLIENT_VERSION=4.8.31 EOS_CLIENT_RELEASE=1

Note that disabling IPv6 using sysctl.conf does not help.

3

Hi Masanori,

I think that the output that you get does not mean XRootD is only listening on the IPv6 interface. As a simple test you can do the following:

  • check that the XRootD daemon is listening on both interfaces. For my machine for example:
[esindril@esdss000 eos]$ nc -z 128.141.194.200 1094
[esindril@esdss000 eos]$ echo $?
0
[esindril@esdss000 eos]$ nc -z 2001:1458:202:229::100:3f 1094                                                                                                                                                                                                                             
[esindril@esdss000 eos]$ echo $?
0
  • actually try a transfer having the client connect to a certain interface:
[esindril@esdss000 eos]$ XRD_NETWORKSTACK=IPv4 xrdcp -f /etc/group root://localhost//eos/dev/replica/file1.dat 
[1.227kB/1.227kB][100%][==================================================][1.227kB/s]  
[esindril@esdss000 eos]$ XRD_NETWORKSTACK=IPv6 xrdcp -f /etc/group root://localhost//eos/dev/replica/file1.dat 
[1.227kB/1.227kB][100%][==================================================][1.227kB/s]

This all seems to work as expcted. My output of netstat is:

[esindril@esdss000 eos]$ sudo netstat -polenta | grep 1094
tcp6       0      0 :::1094                 :::*                    LISTEN      2          43427379   30330/xrootd         off (0.00/0/0)
tcp6       0      0 2001:1458:202:229:42734 2001:1458:202:229::1094 ESTABLISHED 2          43423676   30195/xrootd         off (0.00/0/0)
tcp6       0      0 2001:1458:202:229:42742 2001:1458:202:229::1094 ESTABLISHED 2          43432755   30207/xrootd         off (0.00/0/0)
tcp6       0      0 2001:1458:202:229::1094 2001:1458:202:229:42754 ESTABLISHED 2          43428278   30330/xrootd         keepalive (1440.06/0/0)
tcp6       0      0 2001:1458:202:229:42740 2001:1458:202:229::1094 ESTABLISHED 2          43432751   30186/xrootd         off (0.00/0/0)
tcp6       0      0 2001:1458:202:229:42752 2001:1458:202:229::1094 ESTABLISHED 2          43427520   30268/xrootd         off (0.00/0/0)

Cheers,
Elvin

1 Like
4

Hello Elvin,

Thank you so much for your detailed advice!

I confirmed that servers actually listen also on IPv4 using nc.

While xrdcp -f /etc/group root://localhost//eos/alice/file1.dat (as I set EOS_INSTANCE_NAME=eosalice) fails with a Run: [ERROR] Server responded with an error: [3010] Unable to open file /eos/alice/file1.dat; Operation not permitted (destination) error, I guess that this is not a connection problem but authN/authZ problem or similar so it must be an off-topic.

Best,
Masanori

5

Hi Masanori,

Yes, exactly, from the connectivity point of view everything is fine, this is an authn/authz issue.

Cheers,
Elvin

1 Like