We have a situation where we have populated some 700TB of data in a directory before a quota was applied. The quota appears to only be applied to new data in said directory (data written after the quota). Is there a trick to get the whole dir fed into the quota accounting?
Hi John,
You can use the following command to recompute the quota node:
eos ns recompute_quotanode <path>|cid:<decimal_id>|cxid:<hex_id>
recompute the specified quotanode
Cheers,
Elvin
One question regarding this command. We do not use quota, but we use tree size information, and there is an equivalent command eos ns recompute_treesize, which is sometimes needed because the information is not always accurate.
In 2 cases, when we ran this command on a large hierarchy, this completely blocked the namespace while it was running, and it also seem that it was running way too long eveng considering the number of folders to be browsed. Note that we have still version 5.2.32 so it might be that it is fixed in newer versions, athough I didn’t see anything in the release notes about it.
But I wonder if it is normal that all namespace is locked while such a command is launched ? And if there might be an issue when running it on a large folder that it goes in some infitinite loop ?
In any case, if the lock is unavailable, maybe a maximum running time of the command could be enforced so that we do not risk to hang the whole instance when we make a mistake on the path ?
Hi Franck,
currently this is implemented like this:void
NsCmd::TreeSizeSubcmd(const eos::console::NsProto_TreeSizeProto& tree,
eos::console::ReplyProto& reply)
{
eos::common::RWMutexWriteLock ns_wr_lock(gOFS->eosViewRWMutex);
…
So it locks the whole namespace while it does it, because otherwise the resulting information is wrong if someone changes something in the meanwhile. So you cannot really run this on large trees during production.
One could refine now this function to lock only the individual containers involved, so only the recomputing tree is locked for new writes.
There are several ways to put a protection, we could estimate the run-time and then do a yes/no question, or add a timeout parameter, but then you could forget to put it and you have the same problem.
OK, thank you for confirming that it is a delicate operation in production.
I understand that rewriting the function refining the lock might be a hard work, so at least introducing a protection could be a helpful first step.
Estimating the run-time would also take a long time if the request is large, so maybe not a good option.
To my opinion, a good compromise could be to add a default reasonable timeout that can be overridden in case we really know what we do. Since the whole namespace is locked, and the operation would prevent any other access, so even a few seconds timeout makes sense, for me.
Or also simply change the default of this :
–depth : maximum depth for recomputation, default 0 i.e no limit
put it 1, to just do the first level by default and avoid mistakes