Dear Experts,
I have encountered an issue regarding manual file conversion. A new conversion job of a file that its conversions were already failed or pending cannot be pushed to QuarkDB after clearing the failed or pending jobs.
The converter is enabled on the space:
sh-4.2# eos space status default | grep convert
converter := on
converter.ntx := 10
This is the target file information. It is stored on raid6 RAIN layout with 6 stripes as you can see below:
sh-4.2# eos file info /eos/gsdc/user/s/sahn/file-1g.raid6.3
File: '/eos/gsdc/user/s/sahn/file-1g.raid6.3' Flags: 0640
Size: 1073741824
Modify: Fri Sep 18 00:56:25 2020 Timestamp: 1600390585.738257000
Change: Fri Sep 18 00:56:21 2020 Timestamp: 1600390581.977760760
Birth: Fri Sep 18 00:56:21 2020 Timestamp: 1600390581.977760760
CUid: 556800006 CGid: 556800006 Fxid: 00000068 Fid: 104 Pid: 31 Pxid: 0000001f
XStype: adler XS: 33 34 ab 4b ETAGs: "27917287424:3334ab4b"
Layout: raid6 Stripes: 6 Blocksize: 1M LayoutId: 20640542 Redundancy: d3::t0
#Rep: 6
┌───┬──────┬────────────────────────┬────────────────┬─────────────────────┬──────────┬──────────────┬────────────┬────────┬────────────────────────┐
│no.│ fs-id│ host│ schedgroup│ path│ boot│ configstatus│ drain│ active│ geotag│
└───┴──────┴────────────────────────┴────────────────┴─────────────────────┴──────────┴──────────────┴────────────┴────────┴────────────────────────┘
0 1083 jbod-mgmt-07.sdfarm.kr default.74 /jbod/box_13_disk_074 booted rw nodrain online kisti::gsdc::g03
1 1419 jbod-mgmt-09.sdfarm.kr default.74 /jbod/box_17_disk_074 booted rw nodrain online kisti::gsdc::g03
2 495 jbod-mgmt-03.sdfarm.kr default.74 /jbod/box_06_disk_074 booted rw nodrain online kisti::gsdc::g01
3 663 jbod-mgmt-04.sdfarm.kr default.74 /jbod/box_08_disk_074 booted rw nodrain online kisti::gsdc::g02
4 411 jbod-mgmt-03.sdfarm.kr default.74 /jbod/box_05_disk_074 booted rw nodrain online kisti::gsdc::g01
5 915 jbod-mgmt-06.sdfarm.kr default.74 /jbod/box_11_disk_074 booted rw nodrain online kisti::gsdc::g02
*******
And this is the output of attr ls for the parent directory.
sh-4.2# eos attr ls /eos/gsdc/user/s/sahn/
sys.eos.btime="1600303554.621731370"
sys.forced.blockchecksum="crc32c"
sys.forced.blocksize="1M"
sys.forced.checksum="adler"
sys.forced.layout="raid6"
sys.forced.nstripes="6"
sys.forced.space="default"
sys.recycle="/eos/gsdc/proc/recycle/"
user.acl=""
This is the command that I run to convert the file into replica layout:
sh-4.2# eos file convert /eos/gsdc/user/s/sahn/file-1g.raid6.3 replica:2
info: conversion based layout+stripe arguments
success: pushed conversion job '0000000000000068:default#02650112' to QuarkDB
However for some reason the job went to pending and failed eventually.
sh-4.2# eos convert list
┌────────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────┐
│Conversion string │Failure │
├────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────┤
│0000000000000065:default.48#00650112 converted file replica number mismatch -- expected=2 actual=0 │
│0000000000000068:default.0#02650112 converted file replica number mismatch -- expected=2 actual=0 │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Then, I tried to clear the failed jobs and run another conversion job but the job was unable to be submitted.
sh-4.2# eos convert clear --failed
info: list cleared
sh-4.2# eos convert list
info: no failed conversions
sh-4.2# eos file convert /eos/gsdc/user/s/sahn/file-1g.raid6.3 raid6:8
info: conversion based layout+stripe arguments
error: unable to push conversion job '0000000000000068:default#22650742' to QuarkDB (errc=0) (Success)
Repeated tries turned out that conversion jobs can be pushed to QuarkDB only for newly created files. It seems that the cleared jobs are still there in somewhere else… Indeed, I found that there are files in /eos/gsdc/proc/conversion:
sh-4.2# eos ls -al /eos/gsdc/proc/conversion
drwxr-xr-+ 1 daemon daemon 2147483648 Sep 18 00:57 .
drwxr-xr-x 1 daemon daemon 2147483648 Sep 18 00:46 ..
-rw------- 2 daemon daemon 1073741824 Sep 18 00:46 0000000000000065:default.48#00650112
-rw------- 2 daemon daemon 1073741824 Sep 18 00:57 0000000000000068:default.0#02650112
By the way, the conversion jobs were not be able to run because the operation was not permitted on the directory /eos/gsdc/proc, which was owned only by root initially. After changing its ownership so that daemon can access, the jobs were able to run but the actual task, i.e. conversion, does not work…
And the last point, when the access to a user home directory is only allowed for that user, the conversion job (probably run by daemon) does not have access to the target file, e.g.
sh-4.2# eos convert list
┌───────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│Conversion string │Failure │
├───────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│000000000000006a:default.0#02650112 [ERROR] Server responded with an error: [3010] Unable to open file /eos/gsdc/user/s/sahn/file-1g.raid6.4; Operation not permitted;
-- tpc_src=root://jbod-mgmt-01.sdfarm.kr:1094//eos/gsdc/user/s/sahn/file-1g.raid6.4 tpc_dst=root://jbod-mgmt-01.sdfarm.kr:1094//eos/gsdc/proc/conversion/000000000000006a:default.0#02650112│
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
It would be very much appreciated if someone can point out where and what I have missed. Please just let me know if you need any further information.
Thank you.
Best regards,
Sang-Un