Eos cp fails

Site Administrators
1

Hi,

I have configured eos using eos-deploy.v7 script (http://alien.web.cern.ch/sites/alien.web.cern.ch/files/eos-deploy.v7) provided on alien website (https://alien.web.cern.ch/content/documentation/howto/site/eosinstallation). Installation went smoothly, but I cannot copy a file. It says there is not enough space, even though it detects rw space. Do you have idea where is the problem? Below some information from /var/log/eos/mgm/Clients.log and terminal.

The only error I get in logs is in /var/log/eos/mgm/Clients.log:

190412 15:40:51 ERROR [00000/00000] daemon ::Emsg Unable to get free physical space /eos/testarea/file.1; No space left on device

Info from terminal:

[root@a-se ~]# eos fs ls
┌────────────────────────┬────┬──────┬────────────────────────────────┬────────────────┬────────────────┬────────────┬──────────────┬────────────┬────────┬────────────────┐
│host │port│ id│ path│ schedgroup│ geotag│ boot│ configstatus│ drainstatus│ active│ health│
└────────────────────────┴────┴──────┴────────────────────────────────┴────────────────┴────────────────┴────────────┴──────────────┴────────────┴────────┴────────────────┘
a-se.grid.icm.edu.pl 1095 1 /alice/gpfs0 default.0 booted rw nodrain online N/A

[root@a-se ~]# eos space ls
┌──────────┬────────────────┬────────────┬────────────┬──────┬─────────┬───────────────┬──────────────┬─────────────┬─────────────┬──────┬──────────┬───────────┬───────────┬──────┬────────┬───────────┬──────┬────────┬───────────┐
│type │ name│ groupsize│ groupmod│ N(fs)│ N(fs-rw)│ sum(usedbytes)│ sum(capacity)│ capacity(rw)│ nom.capacity│ quota│ balancing│ threshold│ converter│ ntx│ active│ wfe│ ntx│ active│ intergroup│
└──────────┴────────────────┴────────────┴────────────┴──────┴─────────┴───────────────┴──────────────┴─────────────┴─────────────┴──────┴──────────┴───────────┴───────────┴──────┴────────┴───────────┴──────┴────────┴───────────┘
spaceview default 0 24 1 1 2.30 GB 25.23 GB 20.13 GB 0 B off off 20 off 2 0 off 1 0 off

[root@a-se ~]# eos node ls
┌──────────┬────────────────────────────────┬────────────────┬──────────┬────────────┬──────┬──────────┬────────┬────────┬────────────────┬─────┐
│type │ hostport│ geotag│ status│ status│ txgw│ gw-queued│ gw-ntx│ gw-rate│ heartbeatdelta│ nofs│
└──────────┴────────────────────────────────┴────────────────┴──────────┴────────────┴──────┴──────────┴────────┴────────┴────────────────┴─────┘
nodesview a-se.grid.icm.edu.pl:1095 online on off 0 10 120 2 1

[root@a-se ~]# eos mkdir /eos/testarea/
[root@a-se ~]# eos chmod 777 /eos/testarea/
success: mode of file/directory /eos/testarea/ is now ‘777’
[root@a-se ~]# eos cp /root/eos-deploy.v7 /eos/testarea/file.1
error: target file open failed - ernno=28 : No space left on device
error: failed copying path=root://localhost//eos/testarea/file.1
#WARNING [eos-cp] copied 0/1 files and 0 B in 0.08 seconds with 0 B/s

2

Hello,

If this is a brand new installation of EOS, can you try running eos space set default on command and see if it helps?

Otherwise, you can find more information in the log files (especially the MGM one).
The log files normally reside in ‘/var/log/eos/<role>/xrdlog.<role>’
[ where <role> may be one of mgm, fst, mq ]

Cheers,
Mihai

3

Update:

Alternatively, you may have a replica layout type, in which case, you don’t have enough Filesystems active (just 1 in this case).

To check the layout type, run eos attr ls /eos/testarea/.

Example:
An output like this indicates each file in that directory will have 2 replicas.

sys.forced.layout="replica"
sys.forced.nstripes="2"
4

Mihai,

thank you for your help.
Yes, this is brand new installation.

When I run eos attr ls /eos/testarea/ . I get:
[root@a-se ~]# eos attr ls /eos/testarea/.
sys.forced.checksum=“adler”

5

If this is a brand new installation of EOS, can you try running eos space set default on command and see if it helps?

It is not helping.

Otherwise, you can find more information in the log files (especially the MGM one).
The log files normally reside in ‘/var/log/eos/<role>/xrdlog.<role>’
[ where <role> may be one of mgm, fst, mq ]

I do not get any errors except:
190412 15:40:51 ERROR [00000/00000] daemon ::Emsg Unable to get free physical space /eos/testarea/file.1; No space left on device

When running:
eoscp -v -d -N $‘eos-deploy.v7’ $‘/root/eos-deploy.v7’ $‘root://localhost//eos/testarea/file.1?eos.targetsize=30962&eos.bookingsize=30962&eos.app=eoscp&eos.ruid=0&eos.rgid=0’

I get:
[root@a-se ~]# eoscp -v -d -N $‘eos-deploy.v7’ $‘/root/eos-deploy.v7’ $‘root://localhost//eos/testarea/file.1?eos.targetsize=30962&eos.bookingsize=30962&eos.app=eoscp&eos.ruid=0&eos.rgid=0’
[eoscp]: allocate copy buffer with 8388608 bytes
[eoscp] src<0>=/root/eos-deploy.v7 dst<0>=root://localhost//eos/testarea/file.1?eos.targetsize=30962&eos.bookingsize=30962&eos.app=eoscp&eos.ruid=0&eos.rgid=0
[eoscp]: copy protocol file:=>xroot:

[eoscp]: doing POSIX stat on /root/eos-deploy.v7
[eoscp]: POSIX is transparent for staging - nothing to check
[eoscp]: doing POSIX open to read /root/eos-deploy.v7
[eoscp]: doing XROOT open to write /eos/testarea/file.1?eos.targetsize=30962&eos.bookingsize=30962&eos.app=eoscp&eos.ruid=0&eos.rgid=0
190423 15:29:33 time=1556026173.593196 func=DumpConnectionPool level=DEBUG logid=d4e2ae6e-65cb-11e9-acfe-0a0102010122 unit=none tid=00007f0bc2b1e940 source=XrdIo:1677 tident= sec= uid=0 gid=0 name= geo=“” [connection-pool-dump]
190423 15:29:33 time=1556026173.607424 func=fileOpen level=ERROR logid=d4e2ae6e-65cb-11e9-acfe-0a0102010122 unit=none tid=00007f0bc2b1e940 source=XrdIo:235 tident= sec= uid=0 gid=0 name= geo=“” error= “open failed url=root://localhost:1094//eos/testarea/file.1?eos.app=eoscp&eos.bookingsize=30962&eos.rgid=0&eos.ruid=0&eos.targetsize=30962&fst.valid=1556026232, errno=3009, errc=400, msg=[ERROR] Error response: No space left on device”
error: target file open failed - ernno=28 : No space left on device

6

Check if your group is enabled (eos group ls ) or just do:

eos space set default on

Try to define the GEOTAG for the FST in /etc/sysconfig/eos_env

EOS_GEOTAG=“ASE::GPFS”
(restart the FST service)

and the default client geotag on the MGM using

eos vid set geotag default ASE::GPFS

You can change the geotag ASE::GPFS to something else if you like.

Cheers Andreas.

7

Thank you,

it works now.
What geotags are allowed?

Best,
Karol

8

You can define anything which has not more than 8 characters e.g.

or :: or ::::

e.g.
ALICE::ASE::GPFS
or you describe some rack node location, but the strings between :: and :: should not have more than 8 characters.

Cheers Andreas.