eos quota seems to believe there is 1.24PB used bytes, whereas eos space shows (accurately) 276T used.
Attempting to set a quota for .8PB therefor shows the quota is ‘exceeded’.
Setting gid:uid quotas are represented in the default.eoscf config. However, where does ‘eos quota’ obtain the values it represents as ‘used bytes’ and why does this value differ from what is acutal show as used in the space?
Pete
root@alice-eos-01.ornl.gov:~
22:29:43 # eos quota set -p /eos/aliceornl/ -u cern -i 100M -v .8P
success: updated volume quota for uid=602 for node /eos/aliceornl/
success: updated inode quota for uid=602 for node /eos/aliceornl/
root@alice-eos-01.ornl.gov:~
22:30:16 # eos quota ls
┏━> Quota Node: /eos/aliceornl/
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│user │used bytes│logi bytes│used files│aval bytes│aval logib│aval files│ filled[%]│vol-status│ino-status│
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
adm 0 B 0 B 0 0 B 0 B 0 100.00 % ignored ignored
cern 1.24 PB 1.24 PB 32.46 M 800.00 TB 800.00 TB 100.00 M 100.00 % exceeded ok
daemon 0 B 0 B 0 0 B 0 B 0 100.00 % ignored ignored
root 10.48 MB 10.48 MB 0 0 B 0 B 0 100.00 % ignored ignored
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│group │used bytes│logi bytes│used files│aval bytes│aval logib│aval files│ filled[%]│vol-status│ino-status│
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
adm 0 B 0 B 0 0 B 0 B 0 100.00 % ignored ignored
cern 1.24 PB 1.24 PB 32.46 M 0 B 0 B 0 100.00 % ignored ignored
daemon 0 B 0 B 0 0 B 0 B 0 100.00 % ignored ignored
nobody 0 B 0 B 0 0 B 0 B 0 100.00 % ignored ignored
root 10.48 MB 10.48 MB 0 0 B 0 B 0 100.00 % ignored ignored
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│summary │used bytes│logi bytes│used files│aval bytes│aval logib│aval files│ filled[%]│vol-status│ino-status│
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
All users 1.24 PB 1.24 PB 32.46 M 800.00 TB 800.00 TB 100.00 M 100.00 % exceeded ok
All groups 1.24 PB 1.24 PB 32.46 M 0 B 0 B 0 100.00 % ignored ignored
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
root@alice-eos-01.ornl.gov:~
22:30:28 # grep quota /var/eos/config/alice-eos-01.ornl.gov/default.eoscf
global:/config/eosaliceornl/space/2#quota => off
global:/config/eosaliceornl/space/4#quota => off
global:/config/eosaliceornl/space/7#quota => off
global:/config/eosaliceornl/space/default#quota => off
global:/config/eosaliceornl/space/default:0#quota => off
global:/config/eosaliceornl/space/recovery#quota => off
global:/config/eosaliceornl/space/spare#quota => off
global:/config/eosaliceornl/space/test#quota => off
quota:/eos/aliceornl/:uid=602:userbytes => 800000000000000
quota:/eos/aliceornl/:uid=602:userfiles => 100000000
root@alice-eos-01.ornl.gov:~
22:30:43 # eos space ls
┌──────────┬────────────────┬────────────┬────────────┬──────┬─────────┬───────────────┬──────────────┬─────────────┬─────────────┬──────┬──────────┬───────────┬───────────┬──────┬────────┬───────────┬──────┬────────┬───────────┐
│type │ name│ groupsize│ groupmod│ N(fs)│ N(fs-rw)│ sum(usedbytes)│ sum(capacity)│ capacity(rw)│ nom.capacity│ quota│ balancing│ threshold│ converter│ ntx│ active│ wfe│ ntx│ active│ intergroup│
└──────────┴────────────────┴────────────┴────────────┴──────┴─────────┴───────────────┴──────────────┴─────────────┴─────────────┴──────┴──────────┴───────────┴───────────┴──────┴────────┴───────────┴──────┴────────┴───────────┘
spaceview default 6 8 31 21 276.87 TB 999.99 TB 644.97 TB 0 B off on 20 on 20 8 0 0 on
Indeed, this looks odd. I’ll have a look at the code to check if quotas are accounted correctly, and try to reproduce what you’re seeing.
The in-memory namespace reconstructs the quota values on reboot when replaying the changelogs, so it would be interesting to see if they change then. (not suggesting to reboot just for this, but a thing to keep in mind when it does happen - please let us know if the numbers change)
By the way, Elvin and Andreas are on holiday this week, let’s wait to see if they know something more.
Hi Pete,
did you have some mixture between Beryl and CITRINE runing? What can be is, that there are files commited with a very large size but indeed they are just created by a truncate operation, so they don’t take space on disk.
In Beryl there was a 1TB truncate used to indicate a certain condition due to the lack of a plug-in in Xrootd 3 and if you mixed Beryl with CITRINE it was indeed doing a truncate to 1 TB when talking to a citrine FST.
The other option is, that you actually have more files in the namespace registered than in the storage nodes. You might try to dump the namespace with ‘find --size’ and sort by size to figure out, if the extra space is created by one or few files which are extremely large.