We have a situation where we have populated some 700TB of data in a directory before a quota was applied. The quota appears to only be applied to new data in said directory (data written after the quota). Is there a trick to get the whole dir fed into the quota accounting?
Hi John,
You can use the following command to recompute the quota node:
eos ns recompute_quotanode <path>|cid:<decimal_id>|cxid:<hex_id>
recompute the specified quotanode
Cheers,
Elvin
One question regarding this command. We do not use quota, but we use tree size information, and there is an equivalent command eos ns recompute_treesize, which is sometimes needed because the information is not always accurate.
In 2 cases, when we ran this command on a large hierarchy, this completely blocked the namespace while it was running, and it also seem that it was running way too long eveng considering the number of folders to be browsed. Note that we have still version 5.2.32 so it might be that it is fixed in newer versions, athough I didn’t see anything in the release notes about it.
But I wonder if it is normal that all namespace is locked while such a command is launched ? And if there might be an issue when running it on a large folder that it goes in some infitinite loop ?
In any case, if the lock is unavailable, maybe a maximum running time of the command could be enforced so that we do not risk to hang the whole instance when we make a mistake on the path ?
Hi Franck,
currently this is implemented like this:void
NsCmd::TreeSizeSubcmd(const eos::console::NsProto_TreeSizeProto& tree,
eos::console::ReplyProto& reply)
{
eos::common::RWMutexWriteLock ns_wr_lock(gOFS->eosViewRWMutex);
…
So it locks the whole namespace while it does it, because otherwise the resulting information is wrong if someone changes something in the meanwhile. So you cannot really run this on large trees during production.
One could refine now this function to lock only the individual containers involved, so only the recomputing tree is locked for new writes.
There are several ways to put a protection, we could estimate the run-time and then do a yes/no question, or add a timeout parameter, but then you could forget to put it and you have the same problem.
OK, thank you for confirming that it is a delicate operation in production.
I understand that rewriting the function refining the lock might be a hard work, so at least introducing a protection could be a helpful first step.
Estimating the run-time would also take a long time if the request is large, so maybe not a good option.
To my opinion, a good compromise could be to add a default reasonable timeout that can be overridden in case we really know what we do. Since the whole namespace is locked, and the operation would prevent any other access, so even a few seconds timeout makes sense, for me.
Or also simply change the default of this :
–depth : maximum depth for recomputation, default 0 i.e no limit
put it 1, to just do the first level by default and avoid mistakes
Hi folks,
I have another question about this topic.
We have two paths (/eos/alicelblaf/ and /eos/alicelblhpcs/) and there was only a quota set for one of them (/eos/alicelblaf/):
[root@alicemgm0.lbl.gov ~]# eos quota ls
┏━> Quota Node: /eos/alicelblaf/
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│user │used bytes│logi bytes│used files│aval bytes│aval logib│aval files│ filled[%]│vol-status│ino-status│
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
cern 1.07 PB 1.07 PB 1.69 M 0 B 0 B 0 100.00 % ignored ignored
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│group │used bytes│logi bytes│used files│aval bytes│aval logib│aval files│ filled[%]│vol-status│ino-status│
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
900 1.07 PB 1.07 PB 1.69 M 1.35 PB 1.35 PB 0 79.56 % ok ignored
nobody 0 B 0 B 0 0 B 0 B 0 100.00 % ignored ignored
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│summary │used bytes│logi bytes│used files│aval bytes│aval logib│aval files│ filled[%]│vol-status│ino-status│
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
All users 1.07 PB 1.07 PB 1.69 M 0 B 0 B 0 100.00 % ignored ignored
All groups 1.07 PB 1.07 PB 1.69 M 1.35 PB 1.35 PB 0 79.56 % ok ignored
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
The majority of data is under the other path (/eos/alicelblhpcs/):
[root@alicemgm0.lbl.gov ~]# eos df
┌──────────────┬────────┬────────┬────────┬───────────────┬────────────┬──────┬───────┬─────────────────┐
│Instance │ Size│ Used│ Files│ Directories│ PCR GB/TB*s│ Use%│ Vol-x│ Path│
└──────────────┴────────┴────────┴────────┴───────────────┴────────────┴──────┴───────┴─────────────────┘
alicelblhpcs 7.37 PiB 5.70 PiB 109.03 M 103.79 k 0.01 77% 1.00 /eos/alicelblhpcs
I’ve set a quota of 7.23PB for /eos/alicelblhpcs:
[root@alicemgm0.lbl.gov ~]# eos quota set -g 900 -v 7230000000000000 -p /eos/alicelblhpcs
updating quota using 7230000000000000 bytes (7230000000000000 raw bytes)
success: updated volume quota for gid=900 for node /eos/alicelblhpcs/
But recomputing the quota fails:
[root@alicemgm0.lbl.gov ~]# eos ns recompute_quotanode /eos/alicelblhpcs
error: errc=52 msg=“[ERROR] Operation expired”
And even while it was running the command would have taken a little over a year to complete.
eos quota ls /eos/alicelblhpcs now looks like this:
[root@alicemgm0.lbl.gov ~]# eos quota ls /eos/alicelblhpcs
┏━> Quota Node: /eos/alicelblhpcs/
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│user │used bytes│logi bytes│used files│aval bytes│aval logib│aval files│ filled[%]│vol-status│ino-status│
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
cern 6.42 PB 6.42 PB 107.15 M 0 B 0 B 0 100.00 % ignored ignored
root 20.48 KB 20.48 KB 715 0 B 0 B 0 100.00 % ignored ignored
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│group │used bytes│logi bytes│used files│aval bytes│aval logib│aval files│ filled[%]│vol-status│ino-status│
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
900 6.42 PB 6.42 PB 107.15 M 7.23 PB 7.23 PB 0 88.80 % ok ignored
nobody 0 B 0 B 0 0 B 0 B 0 100.00 % ignored ignored
root 20.48 KB 20.48 KB 715 0 B 0 B 0 100.00 % ignored ignored
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│summary │used bytes│logi bytes│used files│aval bytes│aval logib│aval files│ filled[%]│vol-status│ino-status│
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
All users 6.42 PB 6.42 PB 107.15 M 0 B 0 B 0 100.00 % ignored ignored
All groups 6.42 PB 6.42 PB 107.15 M 7.23 PB 7.23 PB 0 88.80 % ok ignored
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Is there anything else we could try to apply the new quota? Or are we just SOL with that amount of data?
Thanks,
Torben
Hi Torben,
The “operation expired” is an artifact of how the XRootD client works - the command still executed in the MGM server. You can actually see that it accounted for 6.42 PB. You can run the same command using a longer timeout like for example (10h timeout):
XRD_STREAMTIMEOUT=36000 eos ns recompute_quotanode ...
What makes you think that the quota recomputation would take “over an year to complete”?
By the looks of it, most of the data is already accounted in the quota node. Also you need to keep in mind that both the quota and the tree-size accounting (that is used as input for the df command) are not necessarily strongly consistent with the reality on the disk. You can also trigger an ns recompute_tree_size for a more accurate reading of the df result.
Cheers,
Elvin
Hi Elvin,
Thanks for the explanation.
When I had run the eos ns recompute_quotanode command and then checked with eos quota ls there was a value that was slowly increasing (maybe it was under used files) which I misinterpreted as the quota being applied.
Cheers,
Torben
@esindril
Another thing that we’re slightly confused about is that in our case / should be an alias for /eos/alicelblhpcs/grid/ :
[root@alicemgm0.lbl.gov ~]# xrdfs alicemgm0.lbl.gov:1094 stat /10/26418/8972e664-1129-11eb-9926-fb3c77303c52
Path: /10/26418/8972e664-1129-11eb-9926-fb3c77303c52
Id: 559423093781613311
Size: 27196
MTime: 2020-10-18 10:06:07
Flags: 16 (IsReadable)
[root@alicemgm0.lbl.gov ~]# xrdfs alicemgm0.lbl.gov:1094 stat /eos/alicelblhpcs/grid/10/26418/8972e664-1129-11eb-9926-fb3c77303c52
Path: /eos/alicelblhpcs/grid/10/26418/8972e664-1129-11eb-9926-fb3c77303c52
Id: 559423093781613311
Size: 27196
MTime: 2020-10-18 10:06:07
Flags: 16 (IsReadable)
However, when doing a space query we’re getting different results:
[root@alicemgm0.lbl.gov ~]# xrdfs ``alicemgm0.lbl.gov:1094`` query space /
oss.cgroup=default&oss.space=8297087615631360&oss.free=1611374287388672&oss.maxf=68719476736&oss.used=6685713328242688&oss.quota=8297087615631360
[root@alicemgm0.lbl.gov ~]# xrdfs ``alicemgm0.lbl.gov:1094`` query space /eos/alicelblhpcs/grid
oss.cgroup=default&oss.space=7200000000000000&oss.free=1595616889936266&oss.maxf=68719476736&oss.used=5604383110063734&oss.quota=7200000000000000
Do you have any idea why this is happening?
Thanks,
Torben
Hi Tobern,
The explanation is quite simple, the mapping is not applied in the code that is handling the query space command. I suspect this is happening because you are using the eos map functionality to create the mapping between the / and /eos/alicelblhpcs/grid directories. To double check what is the output of eos map ls in your instance?
For our ALICE instances at CERN we dropped this configuration in favour of the symlink functionality that is properly handled also for the query space command and achieves the same purpose. For example one of our ALICE instance has the following configuration:
# no mappings
[root@eosalice-ns-ip700 (mgm:master) ~]$ eos map ls
# but there are symlinks created for the required mappings
#[root@eosalice-ns-ip700 (mgm:master) ~]$ eos ls -lrt /
lrwxrwxrwx 1 root root 19 Jul 3 2025 00 → /eos/alice/grid/00/
lrwxrwxrwx 1 root root 19 Jul 3 2025 01 → /eos/alice/grid/01/
lrwxrwxrwx 1 root root 19 Jul 3 2025 02 → /eos/alice/grid/02/
lrwxrwxrwx 1 root root 19 Jul 3 2025 03 → /eos/alice/grid/03/
lrwxrwxrwx 1 root root 19 Jul 3 2025 04 → /eos/alice/grid/04/
lrwxrwxrwx 1 root root 19 Jul 3 2025 05 → /eos/alice/grid/05/
lrwxrwxrwx 1 root root 19 Jul 3 2025 06 → /eos/alice/grid/06/
lrwxrwxrwx 1 root root 19 Jul 3 2025 07 → /eos/alice/grid/07/
lrwxrwxrwx 1 root root 19 Jul 3 2025 08 → /eos/alice/grid/08/
lrwxrwxrwx 1 root root 19 Jul 3 2025 09 → /eos/alice/grid/09/
lrwxrwxrwx 1 root root 19 Jul 3 2025 10 → /eos/alice/grid/10/
lrwxrwxrwx 1 root root 19 Jul 3 2025 11 → /eos/alice/grid/11/
lrwxrwxrwx 1 root root 19 Jul 3 2025 12 → /eos/alice/grid/12/
lrwxrwxrwx 1 root root 19 Jul 3 2025 13 → /eos/alice/grid/13/
lrwxrwxrwx 1 root root 19 Jul 3 2025 14 → /eos/alice/grid/14/
lrwxrwxrwx 1 root root 19 Jul 3 2025 15 → /eos/alice/grid/15/
drwxrwxr-x 1 root root 75499572123698750 Apr 25 2024 eos
Cheers,
Elvin
Thanks for the explanation. Looks like we are indeed using eos map:
[root@alicemgm0.lbl.gov ~]# eos map ls
/00/ => /eos/alicelblhpcs/grid/00/
/01/ => /eos/alicelblhpcs/grid/01/
/02/ => /eos/alicelblhpcs/grid/02/
/03/ => /eos/alicelblhpcs/grid/03/
/04/ => /eos/alicelblhpcs/grid/04/
/05/ => /eos/alicelblhpcs/grid/05/
/06/ => /eos/alicelblhpcs/grid/06/
/07/ => /eos/alicelblhpcs/grid/07/
/08/ => /eos/alicelblhpcs/grid/08/
/09/ => /eos/alicelblhpcs/grid/09/
/10/ => /eos/alicelblhpcs/grid/10/
/11/ => /eos/alicelblhpcs/grid/11/
/12/ => /eos/alicelblhpcs/grid/12/
/13/ => /eos/alicelblhpcs/grid/13/
/14/ => /eos/alicelblhpcs/grid/14/
/15/ => /eos/alicelblhpcs/grid/15/
[root@alicemgm0.lbl.gov ~]# eos ls -lrt /
drwxrwsr-x 1 root root 6674592435992466 Oct 3 2025 eos
drwxr-sr-x 1 root root 0 Dec 31 1969 lost+found