Eos-ns-inspect use and results


Following the discussion about eos-ns-inspect command at last EOS Workshop, I wanted to summarize what we obtained while running it on out large instance at JRC (800M files, 170M directories). This commands helps in inspecting a EOS namespace without going through the MGM. IT is available in a separate yum package eos-ns-inspect.

In order to not bother the production at all (although this shouldn’t be a risk), the run was done on a haddock one node cluster running from a restored copy of a previous backup, like explained in the QDB documentation .

We ran all of the available check commands :

  check-naming-conflicts      Scan through the entire namespace looking for naming conflicts
  check-cursed-names          Scan through the namespace to find files / containers with invalid names
  stripediff                  Find files which have non-nominal number of stripes (replicas)
  one-replica-layout          Find all files whose layout asks for a single replica
  check-orphans               Find files and directories with invalid parents
  check-fsview-missing        Check which FileMDs have locations / unlinked locations not present in the filesystem view
  check-fsview-extra          Check whether there exist FsView entries without a corresponding FMD location
  check-shadow-directories    Check for naming conflicts between directories inside the same subdirectory
  check-simulated-hardlinks   Check for corruption in simulated hardlinks

Some of these commands are making only few reads (1-2) per seconds (stripediff, one-replica-layout, etc…) and finish in few hours, but others are more intrusive are make a lot of reads 20-40KHz (check-fsview-extra, check-fsview-missing, check-orphans) and complete up to 1 or 2 days on our instance.

By analyzing the output, it seems that nothing really critical occurred, however, it appears many things due to some left over items from deleted files, that come difficult to clean, because the corresponding files are not existing any more in the namespace. We were wondering if this could be cleaned, and how.

For instance, for orphans, many files have invalid-parent-id=0, or a number that doesn’t corresponf to an existing folder :

file-id=365946114 invalid-parent-id=188331425 size=989696 locations= unlinked-locations=558,462,654,558,462
file-id=343061649 invalid-parent-id=0 size=407 locations= unlinked-locations=549

$ eos-ns-inspect print --members localhost:8888 --cid 188331425
Error while fetching metadata for ContainerMD #188331425: Error while fetching ContainerMD #188331425 protobuf from QDB: Empty response

How can these files be deleted ? eos rm fid:343061649 fails by saying error: cannot retrieve file meta data

There are also many files with unlinked locations in check-fsview-missing. Some we caused to some issue with the FSTs some year ago, and others remain after a FS that was defect, drained, then reinserted. Wonder if these could also be cleaned by removing the reference to unexisting replicas.

Ex :

# eos file info fid:343061649
  File: 'fid:343061649'  Flags: 0644  Clock: 15f42e7bcf0c794b
  Size: 407
Modify: Thu Feb  4 03:29:58 2016 Timestamp: 1454552998.0
Change: Wed Mar 22 16:57:57 2017 Timestamp: 1490198277.689263039
Birth : Thu Jan  1 01:00:00 1970 Timestamp: 0.0
  CUid: 47000 CGid: 40500  Fxid: 1472b491 Fid: 343061649    Pid: 0   Pxid: 00000000
XStype: adler    XS: 10 41 7e 6d    ETAGs: "92089910185426944:10417e6d"
Layout: replica Stripes: 2 Blocksize: 4k LayoutId: 00100112
  #Rep: 0
(undeleted) $ 549
error: cannot retrieve file meta data - Container #0 not found (errc=0) (Success)

Some similar situation in stripediff test, with files with no missing location, no path ans no replicas :

# eos file info fid:862187607
  File: 'fid:862187607'  Flags: 0600  Clock: 15f42ea6bb813c35
  Size: 1572864
Modify: Mon Jul  8 17:41:57 2019 Timestamp: 1562600517.697016582
Change: Mon Jul  8 17:41:57 2019 Timestamp: 1562600517.516073734
Birth : Mon Jul  8 17:41:57 2019 Timestamp: 1562600517.516073734
  CUid: 35835 CGid: 41068  Fxid: 3363f057 Fid: 862187607    Pid: 0   Pxid: 00000000
XStype: adler    XS: 00 00 00 00    ETAGs: "231441723442593792:00000000"
Layout: replica Stripes: 2 Blocksize: 4k LayoutId: 00600112
  #Rep: 0
error: cannot retrieve file meta data - Container #0 not found (errc=0) (Success)

In naming-conflicts, many are reported have the same name, and parent-id=0, but few also have the same name in the same folder, I suspect they could have been created at the very same time by 2 different nodes :

Detected conflict for 'auth.pyc' in container 210612997, betewen files 1236222806 and 1236222808
Detected conflict for 'auth.pyc' in container 210612997, betewen files 1236222808 and 1236819888

Hi Franck,

Nicely done, you’re quite patient / curious for running all check commands. :slight_smile: A note that check-fsview-* are rather experimental, I should probably mark them as such, I implemented them a while ago but not sure how well they work.

Indeed, leftover invisible files with parent=0 seem to be a common issue in EOS, even with the old namespace. When a user issues rm, the underlying data on the FST are not deleted synchronously, but are scheduled for deletion in the future, while the file is marked with parent=0 thus making it invisible to end-users, even though it’s still part of the namespace. EOS has had issues with this mechanism, with many files persisting in this state forever.

I discovered this while playing with ns-inspect too. :slight_smile: We’re working on a solution for how to properly and permanently clear such files.

I believe these have been generated by early versions of FUSEX – bugs in the server would sometimes create such conflicting entries. These might only pose a problem during draining, we’re working on providing a user-friendly way of clearing them up.

Unfortunately such inconsistencies can happen due to the way the internal namespace API is used – we had to use the same API as the in-memory namespace, which does not allow for transactional operations. :neutral_face: Once we drop the in-memory namespace and I can improve the API, such issues should go away completely.


Hi Georgios,

Thank you for your explanations. OK, so you confirm that these situations shouldn’t really hurt, so for now we live with that until there are way to clean them.