Decompress message data from S3 source via SQS

[amarnathpv]

Question

I am attempting to decompress the data that is contained in the message payload. Despite the file being in gz format, I am unable to decompress the data using the decode_* option.

Any clue?

I tried converting below "message" payload to base64 conversion, but didn't help either.

Vector Config

[api]
enabled = true
address = "0.0.0.0:8686"

[sources.s3_logs]
type = "aws_s3"
region = "eu-west-2"
sqs.queue_url = "https://sqs.eu-west-2.amazonaws.com/xxxxxxxxxx/vector-s3-log-queue"

compression = "none"
decoding.codec = "bytes"

[sources.s3_logs.auth]
access_key_id = "xxxxxxxxx"
secret_access_key = "xxxxxxxxx"

[transforms.decompress_message]
type = "remap"
inputs = ["s3_logs"]
source = '''
  #.message_decoded, err = decode_zlib(.message)
  #.message_decoded, err = decode_zstd(.message)
  .message_decoded, err = decode_gzip(.message)
  if err != null { abort }
'''

[sinks.console]
type = "console"
inputs = ["decompress_message"]
encoding.codec = "json"

Vector Logs

{"bucket":"euw2a-s3-app-01","message":"�U�\u0007\u0014y�b��]zA�\u0007=�}HS�B 諼\u0010#\u0018�\u0002I���\u0002Y��]��\b�s3\u0010w���\u0005�\u001d���7q��ҩ�0��*�.�MpR","object":"clm/2025/10/22/20/euw2a-kinds-app-01-2-2025-10-22-20-24-57-752185ba-3e48-45de-b14f-045525c33064.gz","region":"eu-west-2","source_type":"aws_s3","timestamp":"2025-10-22T20:40:04Z"}
{"bucket":"euw2a-s3-app-01","message":"����$ކ(\u001b]�4IM��\u0019i��\"\u0003�\u0004���X�_a��\u001d����\u001aI�9��\u0018n��򂔿�=yE;5�DTC�W̹]{G7fƃ�.\u0012Ƒ\u0019,b_���\u0011q\u001f�\u0003_��qme\u0007�m�|�/���\u0017\tn�\u0005�w`\u001b��","object":"clm/2025/10/22/20/euw2a-kinds-app-01-2-2025-10-22-20-24-57-752185ba-3e48-45de-b14f-045525c33064.gz","region":"eu-west-2","source_type":"aws_s3","timestamp":"2025-10-22T20:40:04Z"}
{"bucket":"euw2a-s3-app-01","message":"�1h�\f��o�� �:�\r�I�<\\�\u001f�Ys�\u000e�]\u0006\u0001W\u0003\u0001�8S��V\u0000\u001c\\�\u0012��Bz4}�ɠ�\u0011=�!���4I�\u0015r�\u001e�$\u0015���D�\"`�S�'g��>�8��\u0019�ݪ�(X��Zd��d%\u001c���C�;C\u001f\u0017鯔��C���p��7�Ͻ�O�\u0010��2��-��\u000e\f=�\\����\\�jQZ��쬥`\u000f<��\u0019����Ǯ)^�\u001b�\u000b2","object":"clm/2025/10/22/20/euw2a-kinds-app-01-2-2025-10-22-20-24-57-752185ba-3e48-45de-b14f-045525c33064.gz","region":"eu-west-2","source_type":"aws_s3","timestamp":"2025-10-22T20:40:04Z"}
{"bucket":"euw2a-s3-app-01","message":"���","object":"clm/2025/10/22/20/euw2a-kinds-app-01-2-2025-10-22-20-24-57-752185ba-3e48-45de-b14f-045525c33064.gz","region":"eu-west-2","source_type":"aws_s3","timestamp":"2025-10-22T20:40:04Z"}

[thomasqueirozb]

Hi @amarnathpv, is there any reason why you are setting compression = "none"? The default is auto which should automatically decompress gzipped files. You could remove that line or you could also set compression = "gzip". You should also keep in mind that it is not possible to decode chunked gzip bytes, so if you are getting many events for one file decode_gzip won't work in the way you're using it.

Also please use ``` when pasting code/logs. The formatting vastly improves readability