Is there a way to stop the MGM from sending pending drop requests to a FST ?
It seems that it has many of them in queue (like millions), which are duplicated. I do not know why from some reason each file was “added to deletion message” several times, so now the FST is trying to delete files that are already deleted. This is annoying because it seems to block any further deletion on the disks that actually need deletion.
To clarify the context, we fully drained a FST node that had an hardware issue, then when we added it back to the instance, we expected the drained files to be deleted from the disks, so that space could be used again. We set the disks online (ro) few at once to monitor the deletion more accurately.
The first 4 disks were correctly emptied, but then for the next 6 disks, there has been a moment where each of their file was sent more than 50 times to deletion with “Schedule2Delete” operations, that are now being sent as “drop requests” to the FST, which answers with a failure (file does not exist). At the current rate of the drop request being rejected, it will take ages to exhaust the ~300 or 400M drop requests, so we would like to cancel them.
Each disk contains 1M file, and we have 48 disks on this FST.
Our instance is running eos version 4.8.15.
Maybe @esindril you have some suggestion.
Yes, we experienced similar problems in the past but now things are working fine for us. There were several things that were changed and improved when it comes to deletions.
If you don’t plan to upgrade then you could use the
eos fs dropdeletion <fsid> command to clear the list of pending deletions from the FileSystem view in the namespace. If these file systems were completely drained and are empty, you can proceed with cleaning up the file system mountpoints by hand.
Once you drop the deletions the drop requests and deletion messages should go down to reasonable levels that match the current deletions actually happening on the system.
Thank you for your answer.
Yes, we will hopefully manage to upgrade in the next months, but in the meantime we need to have things sorted.
How will this
dropdeletion command behave with a lot of files ? Does it have an execution time proportional to the number of deletions to be cancelled ? If yes, it might have an impact on the responsiveness of the MGM (we have that sometimes for intensive operations in our so large namespace)
And will the drop requests that are already queued be aborted ?
The is no performance penalty for doing the
dropdeletion command even with many files on the corresponding file system as this is basically cleaning up a set of file identifiers. Therefore, it does not impact the responsiveness of the MGM.
All the deletions pending for that file systems are dropped. Therefore, you can end up with orphan files that will later be picked up by fsck if you issue this command on a file system which was not supposed to be empty, and some legitimate deletions don’t reach the FST.
Sorry that I didn’t come back earlier about this.
I confirm that the use of
dropdeletion command was harmless and avoid to have the remaining disks receive tons of drop requests (the disks were empty before that, and formatted to actually remove any files). All the 48 disks could be put back online after being emptied and deletions dropped.
However, the ~350M duplicated drop requests that were already queued went on being sent to the FST, which is still answering error because the file are not present. This doesn’t harm, but it seems that at current rate this would probably last for at least one month in total.
The conclusion of this event is that when draining a full node (with a very large amount of files, in our case ~50M total) while it is offline, it is advisable that (once the disks are really empty -as configstatus) all their deletions are dropped, and they are physically cleaned from the OS (by removing all files, or formatting the disks, faster with millions of files) before putting back the node online. Or more radically, removing all the fs and inserting back the FST as new. At least with the EOS version we are running (4.5.15).
Thank you again for your support.