Extra null bytes as start of protobuf message

[fbs]

I'm testing http/protobuf between vector instances and ran into some 400s I do not understand. It seems like a http sink in protobuf mode adds extra null bytes at the start of the payload.

As a reproducer I created a single instance to act as both the log generator, and sink, and source, with 2 pipelines i.e.

source demo_logs -> remap to fixed format -> sink http/protobuf -->
source http/protobuf -> sink console as json

config:

api:
  enabled: true

sources:
  generate_syslog:
    type: "demo_logs"
    format: "syslog"
    count: 10
    interval: 5

  accept_http:
    type: http_server
    address: 127.0.0.1:5555
    decoding:
      # codec: protobuf
      codec: json
      protobuf:
        desc_file: test.desc
        message_type: SimpleMessage

transforms:
  remap_custom:
    inputs: ["generate_syslog"]
    type: "remap"
    source: |
      . = {}
      .message = "a test message"
      .bliep = 2

sinks:
  emit_http:
    inputs: ['remap_custom']
    type: http
    uri: http://127.0.0.1:5555
    encoding:
      # codec: protobuf
      codec: json
      protobuf:
        desc_file: test.desc
        message_type: SimpleMessage
    request:
      headers:
        content-type: x-protobuf

  emit_console:
    type: console
    encoding:
      codec: json
    inputs:
    - accept_http
    - remap_custom

In json mode this work and the console shows 2 lines:

{"bliep":2,"message":"a test message","path":"/","source_type":"http_server","timestamp":"2025-01-30T18:53:03.513011Z"}
{"bliep":2,"message":"a test message"}

inspection in wireguard shows:

POST / HTTP/1.1
content-type: x-protobuf
accept-encoding: zstd,gzip,deflate,br
user-agent: Vector/0.44.0 (aarch64-apple-darwin 3cdc7c3 2025-01-13 21:26:04.735691656)
host: 127.0.0.1:5555
content-length: 40

[{"bliep":2,"message":"a test message"}]

HTTP/1.1 200 OK
content-length: 0
date: Thu, 30 Jan 2025 20:20:11 GMT

However when switching to protobufs using:

syntax = "proto3";

message SimpleMessage {
  int64 bliep = 1;
  string message = 2;
  int64 bla = 3;
}

POST / HTTP/1.1
content-type: x-protobuf
accept-encoding: zstd,gzip,deflate,br
user-agent: Vector/0.44.0 (aarch64-apple-darwin 3cdc7c3 2025-01-13 21:26:04.735691656)
host: 127.0.0.1:5555
content-length: 22

��������a test message

HTTP/1.1 400 Bad Request
content-type: application/json
content-length: 149
date: Thu, 30 Jan 2025 20:22:36 GMT

{"code":400,"message":"Failed decoding body: ParsingError(Error parsing protobuf: DecodeError { description: \"invalid tag value: 0\", stack: [] })"}

Note the 3 nulls at the start of the protobuf message. The full payload:

000000120802120e612074657374206d657373616765
AAAAEggCEg5hIHRlc3QgbWVzc2FnZQ==

If I post the proto and base64 string in https://www.protobufpal.com/ it does seem to decode it ok.

{
  "bliep": 2,
  "message": "a test message",
  "bla": 0
}

If I use that same tool to encode just the message:

{
  "message": "a test message"
}

-> Eg5hIHRlc3QgbWVzc2FnZQ==

and send that it does actually work:

curl localhost:5555 -XPOST -d "$(echo -n Eg5hIHRlc3QgbWVzc2FnZQ==|base64 -D)" -v
< HTTP/1.1 200 OK
< content-length: 0
< date: Thu, 30 Jan 2025 20:34:50 GMT

vector console shows:

{"message":"a test message","path":"/","source_type":"http_server","timestamp":"2025-01-30T20:34:50.871782Z"}

Same for adding bla: 123

curl localhost:5555 -XPOST -d "$(echo -n Eg5hIHRlc3QgbWVzc2FnZRh7|base64 -D)"
{"bla":123,"message":"a test message","path":"/","source_type":"http_server","timestamp":"2025-01-30T20:37:39.358163Z"}

[fbs]

Looks like its the framer that causes this:

000000120802120e612074657374206d657373616765

thats actually

[00 00 00 12] 0802120e612074657374206d657373616765

length_delimited	
Event data is prefixed with its length in bytes.

The prefix is a 32-bit unsigned integer, little endian

0x12 == 18 == len(0802120e612074657374206d657373616765)

[jszwedko]

I was just about to comment this :)

I believe the default, when using the protobuf codec, is for the length_delimited framing to be used (https://vector.dev/docs/reference/configuration/sinks/http/#framing.method). This causes the messages to be prefixed with their length. I believe the length is encoded as a 4 byte integer (little endian). I think that is introducing the padding you are seeing there.

How would you like to delimit the protobuf messages in the HTTP request body? If you don't want any delimiter you can set framing.method = "bytes", but I'm not sure if streaming protobuf messages can be decoded or not 🤔

[fbs]

I think the biggest surprise here is the mismatch in behaviour between sink&source. If I add this on both the sink and source it does work:

framing:
  method: length_delimited
  length_delimited: {}

(also, is having to specify length_delimited a bug? won't start without. All fields are optional)

[jszwedko]

I think the biggest surprise here is the mismatch in behaviour between sink&source. If I add this on both the sink and source it does work:

framing:
  method: length_delimited
  length_delimited: {}

Aha, you are right, the http_server sink defaults to no delimiter. I think this is a bug. It should match the sink here.

(also, is having to specify length_delimited a bug? won't start without. All fields are optional)

That also seems like a bug.

Do you mind opening bug reports for both of these issues? I think they have easy fixes.

[fbs]

Let's see if I can figure out how to fix it and send out a PR :)

[jszwedko]

That'd be fantastic :)

For the first bug, I think it is here:

vector/lib/codecs/src/decoding/mod.rs

Line 370 in 0431a07

DeserializerConfig::Protobuf(_) => FramingConfig::Bytes,

I believe this should match the encoding side:

vector/lib/codecs/src/encoding/mod.rs

Line 376 in 0431a07

| SerializerConfig::Protobuf(_) => {

For the second bug, I think this struct just needs a #[derivative(Default)]:

vector/lib/codecs/src/common/length_delimited.rs

Line 7 in 0431a07

pub struct LengthDelimitedCoderOptions {