I did some testing. The orphans probably appear after a ‘fs boot --syncmgm’ because the timeout for retrieving the MGM meta data stream is 10 minutes. Can it be that you have many millions of files on a single filesystem?
The logic behind works like the following on a full sync:
clear the db
put all the files on disk in the DB and flag all as orphan
flag all the files reported from the MGM as not orphans
When the MGM syncing part times out you get a lot of files as fake orphans.
I also noticed, that the process in the FST is not correctly spawned, it uses ‘system’ which is very bad in a multi-threaded program because it uses fork and can give an OOM. We have to fix that.
We should probably also make this timeout reconfigurable or have a higher default.
Not really. Each FS has as average 1M files, oscillating roughly between 500K and 1.5M. From the last fsck export (before we disabled it as mentioned above) almost all FS that report orphans file report more than 600K, so it could be that we reach the timeout…
But we also observe such report on our test instance, currently on one FS out of 9, which reports only 1K files, and repair --resync does not clear them… nor does regular scan on FST (test instance scans them daily…)
They were not actual orphans (there was a path for them, and there was a replica entry for the FS reporting them as orphans), but I just rechecked and the resync action did eventually clear them, although not immediately. So on this small instance, orphans number dropped to 29, and they are actual orphans.
Regarding the production instance, during the last maintenance period, we launched a resync on the around 10M files reported in the fsck report. This took around 1 hour - and in the mean time fsck interface was not responding. But since we also restarted all the FSTs for upgrade, the number of files eventually grew again to a larger number, and we couldn’t tell if the previous ones were cleared or no. Now we have even more files.
After this was finished the reporting of orphans was “more reasonable”:
eos fs status 3870 | grep orphans
stat.fsck.orphans_n := 12
Bottom line, this might point to the scenario that Andreas suggested. There is now (already in 4.2.23) a timeout of 4 hours for the dumpmd command so this should go away. Together with the recent fixes such orphans should be handled properly.
What is the way to use this specific pipeline version ? Is there some package repository for it ? Unfortunately, I don’t know if I will be able to test this not official release on our production instance, which present the issue…
About the resync, in fact I realize that a FS takes much more time than yours to boot, between 30 minutes and one hour, and this happens sometimes when a FST starts, i.e. all FS boot at the same time, that slows down things for sure. So maybe if we boot them one y one sequentially, that might improves things. But still, I might want to wait we upgrade to V 4.2.23, because the previous 10 minutes timeout will for sure be exceeded.
The tests I’ve done were using 4.2.22. Therefore, I think you can go ahead and test it for one file system on one of the FSTs without doing any update. You don’t need any more the new version from the pipeline. Just follow the steps from my last post.
We’ve removed this full resync operation and this should not happen in the future, but this will only be available in 4.2.24.
I confirm that running eos fs boot <fsid> --syncmgm on the file system one by one, the number of reported orphans does reset to a small value.
We are wondering if doing such operation on a fs which reports non-zero ropen or wopen values has impact on the client accessing the files ? Or would that be transparent to the users ?
All updates to the local database are protected by locks so the only interference that I can think of is that there is a delay in replying to the users, and if this takes more than 1 min they will timeout.