Limit user activity, and control the log file output

Site Administrators
1

Last week, we had the an incident which was linked to a very high activity of a user (using fusex mounts, on a htcondor cluster, so many many processes), leading to a huge log being written on disk. More that the necessary space, the problem was it seems that logrotate wasn’t compressing fast enough to consume all log messages to free space.

Of course, the user launched the culprit process on Friday evening, so there was no one to intervene on this, and on Saturday evening the disk was full, while 300GB were available before leaving.

This time, it occurred that freeing up space wasn’t sufficient to restore the MGM service, it has to be restarted. After the restart, the previous activity of that user started again, with 3Khz of Open operations. Thinking it might be too much, we tried to limit it using eos acces set limit <rate> rate:use:*:Open (up to now we never really needed this feature).

However we observed that if the rate value is below it limits the rate observed in eos stat output to around 400Hz, but any value above doesn’t limit at all (or to a higher limit that what the user is actually doing). Is that a normal behaviour ?

In any case, since this activity actually doesn’t disturb the MGM (very very resilient on this :+1: , we were wondering if there would be a way to still have a useful log file (in case of issue, to analyse what is going on), in particular, we find useful to still have the open information in the logs.

The filter we use for the log file is this (was defined some years ago to remove useless, maybe not any mor valid for this version):

eos debug info --filter IdMap,Commit,_access,readlink,_readlink,BroadcastReleaseFromExternal,BroadcastRefreshFromExternal,BroadcastMD,OpSetFile,RefreshEntry,HandleMD,FillContainerCAP,Store,OpGetLs,OpGetCap,log_info,Imply,OpSetDirectory

With this, and the activity we observed these days, the xrdlog.mgm log file growth is above 8MB/s, which makes 30GB in one hour. In addition, the Clients.log file is almost as big.

The questions are :

  • How to correctly set rate limit to users ?
  • Does someone of you already had this log file increase issue, and what is the configuration you use for the log files, to still be able to analyse the logs ? Could it be possible to filter out messages only in case of high activity ? Or do you just mainly filter them a lot on big instances ?
  • What is causing the unavailability of the MGM when the /var/eos partition (where the logs are written) is full ? Is it the log writing, or something els (report, ns-queue) ? Would it be better to separated the logs from the /var/eos parition ? We put them there as in the past they were on the root partition, much smaller and this gave other types of issues… But maybe now we could think to do otherwise… The /var/eos partition used to be big to host the files hosting the in-memory namespace. But now the MD are on a qurkdb separated one…

Our server version : 5.2.32

2

Hi Franck,
in some instances with a lot of clients we do hourly compression of the log files on the MGM. There were several fixes in the rate limiting and since you run 5.2.x I don’t recall if they all were in that version. The general problem with it is, that it has to decide based on a 5s window. And if you have periods of spikes with larger spacing this might not work well. We have also changed some of the messages from info to debug in 5.3. Still if you have 3kHz of open, you will have a very fast growing log file. One can compress the log files better with zstd or bzip3 if you want more space, but bzip3 is significantly slower. Normally log files are in /var/log/eos not in /var/eos/ ???

3

Dear Andreas,

Thank you for your answer.

I read from your message the confirmation that 3kHz of open is a kind of exceptional peak, even for you. This is why we were thinking of limiting the rate. Regarding the access rules not really working, I noticed that if I add also a threads:username limit (higher than what is actually reported in eos ns), then the rate limit start to limit the rate more or less proportionally to the limit set, could that have some explanation ?

We do indeed already use hourly compression since a long time, for us it was necessary, but this time it wasn’t even sufficient.

Regarding the rate limiting of log files in newer versions, I will try to look for extra information about that, it might be useful when we migrate to 5.3.x (hopefully soon). I haven’t heard of this feature, so it might be new. A 5s window would be very fine for us, we had the problem because of sustained activity for several hours, so it would for sure have helped.

Regarding the partition, yes indeed as I tried to explain at the end of my message (I recalled it after writing) we do not use the normal scheme, as our original /var/log/eos partition wasn’t big enough and we used the large /var/eos (with a symlink from /var/log/eos to /var/eos/log) for that. But when introducing quarkdb namespace we should have thought of reviewing that… now this is a consequence. Since we are currently provisioning new hardware to replace the MGM servers, we can do it now.

Just can you confirm that if /var/log/eos only is full, do you know if the MGM will still serve (without logging) as long as /var/eos has free space, or is successful log writing mandatory ? I see there are only folders ns-queue and report which are still used.