[parse_key_value] does not extract some logfiles correctly

[StefanSa]

Hi there,
have here this part of a syslog message:

set-cookie=\"MapiRouting=UlVNOjkwYzZlMGVmLTdiMTctNDA0ZC05YWEyLTE3ZWU0MmY2ZjBlODppBu888bjbCA==; path=/mapi/; secure; HttpOnly, MapiSequence=18-YtLoPw==; path=/mapi/emsmdb; secure; HttpOnly, MapiContext=MAPIAAAAAOin9NmQxOm/8q"

but i get with . = set!(., path: ["sophos", "utm"], data: parse_key_value!(.message)) only some parts of it extracted.

"set-cookie": "\"MapiRouting=UlVNOjkwYzZlMGVmLTdiMTctNDA0ZC05YWEyLTE3ZWU0MmY2ZjBlODppBu888bjbCA==;"
"HttpOnly,": true,
"secure;": true,
"MapiContext": "MAPIAAAAAOin9NmQxOm/8q",
"MapiSequence": "18-YtLoPw==;",

correct would be here:

set-cookie=\"MapiRouting=UlVNOjkwYzZlMGVmLTdiMTctNDA0ZC05YWEyLTE3ZWU0MmY2ZjBlODppBu888bjbCA==; path=/mapi/; secure; HttpOnly, MapiSequence=18-YtLoPw==; path=/mapi/emsmdb; secure; HttpOnly, MapiContext=MAPIAAAAAOin9NmQxOm/8q"

i think the problem is the key set-cookie with a content of ==;

Is there a chance to extract this key correctly with parse_key_value ?

thx for any help here
Stefan

[StefanSa]

@jszwedko , hi Jesse
first thank you for your time and vector.dev.
Do you have any idea how to proceed ?
Due to the partial not detection of the key by parse_key_value, new keys ones are created and also with special characters HttpOnly, or
secure; which is very difficult to delete later.
Or something like this...
winsocket_pr winsocket_proto winsocket_protocol

[StefanSa]

@jszwedko
on closer look this is more likely a parse_grok error. A part of the message is missing.

[transforms.remap_grok_sophos_utm]
type = "remap"
inputs = ["syslog_sophos_utm"]
source = '''
. = parse_grok!(.message, "%{SYSLOG5424PRI}%{DATA:timestamp} %{DATA:device} %{SYSLOGPROG}: %{GREEDYDATA:message}")
'''

[transforms.split_sophos_utm]
type = "remap"
inputs = ["remap_grok_sophos_utm"]
source = '''
. = set!(., path: ["sophos", "utm"], data: parse_key_value!(.message))
.sophos.utm.program = del(.program)
.sophos.utm.pid = del(.pid)
.sophos.utm.device = del(.device)
.event.ingested = now()
.event.original = .message
'''

This one is event.original actually %{GREEDYDATA:message}
As you can see a part of the message is cut off websocket_protoc should be websocket_protocol

"id=\"0299\" srcip=\"87.160.143.95\" localip=\"172.16.22.10\" size=\"110\" user=\"-\" host=\"87.160.143.95\" method=\"POST\" statuscode=\"200\" reason=\"-\" extra=\"-\" exceptions=\"SkipBlacklistDNSRBL, SkipBlacklistGeoIP, SkipAntiVirus, SkipTFT, SkipURLHardening, SkipFormHardening, SkipCookieSigning, SkipThreatsFilter\" time=\"40389\" url=\"/mapi/nspi/\" server=\"webmail.mpdv.com\" port=\"443\" query=\"[email protected]\" referer=\"-\" cookie=\"-\" set-cookie=\"MapiRouting=UlVNOmQyOGJiYjUzLTA1OWYtNDZkNy1hMmE4LWQ5MTAwMzVlMDcwZDpDng83tLrbCA==; path=/mapi/; secure; HttpOnly, MapiContext=MAPIAAAAAOin9NmQxOm/8qqbrYCwgaKQoJKhjLyFqJmggLCFv4+3jbmN1/TF8cn7yvzN9MH0+CMAAAAAAAA=; path=/mapi/nspi; secure; HttpOnly, MapiSequence=0-t2ZINQ==; path=/mapi/nspi; secure; HttpOnly, X-BackEndCookie=6d76401b-6bcf-4cee-852b-74d57a623a9e=u56Lnp2ejJqBzM/Gns3PzsvSypuZytLLzpya0seem8nSyM7NypnLnprMzZvNgYHNz83M0s7P0s3Oq87Kxc/KxczH; expires=Sat, 21-Oct-2023 15:05:38 GMT; path=/mapi; secure; HttpOnly\" websocket_scheme=\"-\" websocket_protoc",