Hello,
I am trying to drain a group, as I need to remove the disks from that group.
I’ve set the group in drain, but there are no drain jobs running. From the logs, it looks like
some jobs started, failed, then nothing was ever retried:
GroupDrainer:331 msg="group drainer scheduled job file="00000000004fdb19:default.23#20640b42^groupdrainer^" src_grp="default.30" dst_grp="default.23"
GroupDrainer:331 msg="group drainer scheduled job file="00000000004fad21:default.23#20640b42^groupdrainer^" src_grp="default.30" dst_grp="default.23"
GroupDrainer:331 msg="group drainer scheduled job file="00000000004fa573:default.23#20640b42^groupdrainer^" src_grp="default.30" dst_grp="default.23"
GroupDrainer:331 msg="group drainer scheduled job file="00000000004f888d:default.23#20640b42^groupdrainer^" src_grp="default.30" dst_grp="default.23"
.....
GroupDrainer:314 msg="Skipping scheduling of Tracked Transfer" fid=004fad21
GroupDrainer:314 msg="Skipping scheduling of Tracked Transfer" fid=004fa573
GroupDrainer:314 msg="Skipping scheduling of Tracked Transfer" fid=004f888d
GroupDrainer:314 msg="Skipping scheduling of Tracked Transfer" fid=004f7ca5
......
GroupDrainer:107 msg="Tracking failed transfer" fid=5221665 tag=00000000004fad21:default.23#20640b42^groupdrainer^
GroupDrainer:472 msg="Retrying failed transfers for" fsid=572, count=1 retry_count=0
GroupDrainer:107 msg="Tracking failed transfer" fid=5233433 tag=00000000004fdb19:default.23#20640b42^groupdrainer^
........
GroupDrainer:140 msg="Cannot pick, Empty source or target groups, check status if this is not expected", groupbalancer.groups_over_threshold=1 groupbalancer.groups_under_threshold=0
GroupDrainer:140 msg="Cannot pick, Empty source or target groups, check status if this is not expected", groupbalancer.groups_over_threshold=1 groupbalancer.groups_under_threshold=0
The log then loops in Cannot pick, Empty source or target groups forever.
I have enabled groupdrainer in the space config, and groupdrainer.threshold is set to 95.
The current groups that are relevant are pasted below (as all other groups are above the 95% threshold):
┌──────────┬────────────────┬────────────┬──────┬────────────┬────────────┬────────────┬──────────┐
│type │ name│ status│ N(fs)│ dev(filled)│ avg(filled)│ sig(filled)│ bal-shd│
└──────────┴────────────────┴────────────┴──────┴────────────┴────────────┴────────────┴──────────┘
groupview default.23 on 12 0.15 92.15 0.07 0
groupview default.30 drain 12 0.08 0.73 0.04 0
In the xrd log, I can see the drainjobs starting, followed by some error messages, but they don’t give me much information:
261026 07:29:11 time=1792999751.175209 func=DoIt level=INFO logid=static.............................. source=ConversionJob:240 [tpc]: ....:1094//eos/dev/proc/conversion/ca/00000000004e4cca:default.23#20640b42^groupdrainer^ prepare_msg=[SUCCESS]
261026 07:29:11 time=1792999751.232543 func=HandlePostJobRun
level=ERROR logid=static.............................. source=ConverterEngine:114 tident= sec=(null
) uid=0 gid=0 name=- geo="" xt="" ob="" msg="failed to delete conversion file" path="/eos/dev/proc/conversion/e9/00000000004e3de9:default.23#20640b42^groupdrainer^" err="Unable to remove /eos/dev/proc/conversion/e9/00000000004e3de9:defau
lt.23#20640b42^groupdrainer^; No such file or directory"
Is there any way to see the failure reason for the drain jobs, to retrigger the drain, or to manually move the files from this group to another (there are less than 30 files in the group I want to drain)?
EOS version is 5.3.34
Thank you.