Trying to upgrade from EOS 4.8.105 to 5.1.28 (dev, single-node, instance)
The yum errrors reported in another thread (Eos-xrootd vs xrootd) have been resolved and all required EOS5 packages have been (I think) installed. All previously installed xrootd 4.12.8 packages were removed to block the upgrade to xrootd 5.2 and allow the use of xrootd libs provided only by eos-xrootd-5.5.10 as suggested by the EOS developers
eos@quarkdb.service has started without a problem but the eos@mgm.service, eos@mq.service and eos@fst.service have failed to start because the the EOS libs cannot be located (see below). I can clearly see that all these libs have been installed via the package eos-server-5.1.28-1.el7.cern.x86_64 so I dont quite understand.
I have LD_LIBRARY_PATH=/opt/eos/xrootd/lib64 in /etc/sysconfig/eos_env
Any help would be much appreciated. Many thanks
George
Plugin fslib libXrdEosMgm-5.so not found; falling back to using libXrdEosMgm.so
Plugin No such file or directory loading fslib libXrdEosMgm.so
231121 15:57:21 27891 XrootdConfig: Unable to load file system via libXrdEosMgm.so
231121 15:57:21 27891 XrootdConfig: Unable to load base file system using libXrdEosMgm.so
…
Plugin fslib libXrdMqOfs-5.so not found; falling back to using libXrdMqOfs.so
Plugin No such file or directory loading fslib libXrdMqOfs.so
231121 15:57:40 28481 XrootdConfig: Unable to load file system via libXrdMqOfs.so
231121 15:57:40 28481 XrootdConfig: Unable to load base file system using libXrdMqOfs.so
…
Plugin fslib libXrdEosFst-5.so not found; falling back to using libXrdEosFst.so
Plugin No such file or directory loading fslib libXrdEosFst.so
231121 15:58:03 29226 XrootdConfig: Unable to load file system via libXrdEosFst.so
231121 15:58:03 29226 XrootdConfig: Unable to load base file system using libXrdEosFst.so
.
I had a similar issue when we upgraded our test environment. Check your /etc/xrd.cf.mgm file and make sure none of the lines that specify libraries contain the old xrood library paths.
Thanks for the replies. Yes, I miss the following dependency library (not sure why is listed more than once)
[root@antares-eos14 ~]# ldd /usr/lib64/libXrdEosMgm-5.so | grep “not found”
libprotobuf.so.3 => not found
libprotobuf.so.3 => not found
libprotobuf.so.3 => not found
libprotobuf.so.3 => not found
libprotobuf.so.3 => not found
I notet that the following has been linked to /usr/lib64/libXrdEosMgm-5.so
For my info, how does the missing dep prevent the EOS from locating /usr/lib64/libXrdEosMgm-5.so? Rather, it should not be able to run instead (even after finding it)
In the config files, I use the relative paths as per EOS team’s suggestions
appears to remove the missing library dependency - all EOS services appear to be running now. I can run eos client commands, list the filesystems, files etc; I will run end-to-end tests later.
For EOS 5.1.28 you need both the following packages installed: eos-protobuf3 and eos-grpc-1.41-1. This will be simplified in EOS 5.2.x, where you only need the eos-grpc-1.56 package.
If you have both these packages installed you don’t need to do the trick with the symbolic link. Normally, in the packages that we built for 5.1.x, you should have the following dependency:
The ssl errors might point to some issue with the CRLs and the certificate directories. We normally get everything properly set up from the WLCG point of view by installing the following package that brings in all the CA dependencies: ca-policy-egi-core-1.124-1.noarch
Also we use the following configuration at the MGM for the gsi auth plugin and HTTP:
Thanks again for the detailed reply. Both eos-grpc-1.41.0-2.el7.x86_64 and
eos-protobuf3-3.17.3-1.el7.cern.eos.x86_64 are installed on my host: the first one is explicitely included the in profile and the second is installed as a dependency. So, I am somewhat puzzled why ldd shows no missing libs in your case.
Removing the link breaks again eos@mgm
In my case, eos-protobuf3-3.17.3-1.el7.cern.eos.x86_64 provides /opt/eos/lib64/protobuf3/libprotobuf.so.28 and not the one you show in your ldd output
/opt/eos/lib64/protobuf3/libprotobuf.so.3
Maybe this one is coming from the earlier version of eos-protobuf3 package that I see in the diopside dep repo,
eos-protobuf3-3.17.3-1.el7.cern.eos.x86_64
So, I either have to version lock this particular eos-protobuf3
or create a link between (the real) /opt/eos/lib64/protobuf3/libprotobuf.so.28 and /opt/eos/lib64/protobuf3/libprotobuf.so.3
The last option doesnt look very promising. It removes the missing lib but eos@mgm keeps crashing
By the way, if I remove all EOS packages and try to install only
eos-protobuf3-3.5.1-5.el7.cern.eos.x86_64
yum says that
Package eos-protobuf3 is obsoleted by eos-grpc, trying to install eos-grpc-1.56.1-3.el7.x86_64 instead
Resolving Dependencies
→ Running transaction check
—> Package eos-grpc.x86_64 0:1.56.1-3.el7 will be installed
Trying to install eos-protobuf3-3.5.1-5.el7.cern.eos.x86_64 and eos-grpc-1.41.0-2.el7.x86_64 yum fails with “Error: Multilib version problems found”
Attempted again the upgrade from scratch, and version locking the earlier
eos-protobuf3-3.5.1-5.el7.cern.eos.x86_64 did work (was already instaled for eos4) did work, but the the dependency for libprotobuf.so.3 is still not satisfied.
From which package you get the following on your systems?
Thanks Elvin, and sorry for taxing your patience but we must be looking at a different package…Accorrding to what I see, this eos-protobuf3 package is not prot providing any libprotobuf.so.3 but only the following
What is the date on the file provided by the rpm? Are you using the package from a snapshot or from our dependency repos? Are you using the following: https://storage-ci.web.cern.ch/storage-ci/eos/diopside-depend/el-7/x86_64/eos-protobuf3-3.17.3-1.el7.cern.eos.x86_64.rpm
I am, indeed, using the https://storage-ci.web.cern.ch/storage-ci/eos/diopside-depend/el-7/x86_64/eos-protobuf3-3.17.3-1.el7.cern.eos.x86_64.rpm but taken from out of our local mirror of the repo storage-ci.web.cern.ch/storage-ci/eos/diopside-depend/el-7/x86_64
The date of the rpm is the same: 2023-01-12 13:53 (in our mirror) and
2023-01-12 14:53 in yours with exactly the same size, 838K
The date of the provided library is different that yours though
The library provided by eos-grpc-1.41 is more recent
[root@antares-eos14 ~]# ls -lrt /opt/eos/grpc/lib64/libprotobuf.so.3.17.3
-rwxr-xr-x 1 root root 3674608 Jun 12 09:19 /opt/eos/grpc/lib64/libprotobuf.so.3.17.3
[root@antares-eos14 ~]#
which explains why if I try to install the above eos-protobuf3 when there are no EOS packages at all, yum is trying to get eos-grpc-1.56 being obsoleted by it
Just a sanity check. While upgrading the QuarkDB nodes one by one, does it matter that there are different QuarkDB versions in the cluster? .E.g.
…
7) MEMBERSHIP-EPOCH 0
18) NODES antares-eos97:9999,antares-eos98:9999,antares-eos99:9999
19) OBSERVERS
20) QUORUM-SIZE 2
21) ---------- 22) REPLICA antares-eos98:9999 | ONLINE | UP-TO-DATE | NEXT-INDEX 8745851 | VERSION 0.4.2 23) REPLICA antares-eos99:9999 | ONLINE | UP-TO-DATE | NEXT-INDEX 8745851 | VERSION 5.1.2.5.1.28