Eos log and logstash log format issues

Site Administrators
1

Dear all
eos xrdlog.fst| logs looks quite unstructured
I have 15 group of log length ( by columns) for the 29 different func of FST
… ( func=name # of colunms in log line)
func=LogMsg 20
func=Matches 20
func=MgmSyncer 17
func=MoveToOrphans 19
func=ProcessCapOpaque 24
func=ProcessReq 22
func=Put 20
func=ResyncFileFromQdb 32
func=ResyncMgm 21
func=RunDiskScan 32
func=VerifyChecksum 25
func=_close 24
…

do we have some idea how we could normalized this data ?

I do not think that I can match with efficient way more than 5 col for each log line
241126 15:42:44 time=1732632164.520784 func=_close level=INFO logid=b2289e30-ac04-11ef-a452-b0262874ab30

another issue is the timestamp if the 241126 15:42:44 is not the nomalized format
and the epoch timestamp do not appeared in all logs

any idea or current solution for the eos fst ( and mgm) log parsing is welcome
thank you in advance
best
e.v.

2

match => { “message” => “%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second} \btime=%{NUMBER:xrdtime} \bfunc=%{WORD:func} .?\blevel=%{WORD:level} .?\blogid=%{UUID:logid} %{GREEDYDATA:msg}”

match => { "message" => "%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second} %{INT} FstOfs_(?<func>[a-zA-Z]+): %{GREEDYDATA:msg}"

I conclude to two matching pattern of grok/logstash for unstructured data one for func=* event log and one for FstOfs_* ?
at least I have those in loglevel=info

FYI
e.v.