[Feature] protobuf topic to message mapping

[alebar42]

💡 Motivation

There is typically a 1:1 mapping between MQTT topics and protobuf message definitions as any one topic can only have an expected protobuf payload (1 payload per channel). Currently this requires manual configuration on top of importing a proto spec. The idea is to unify the information entirely in the proto spec itself. The relationship between the proto message and the MQTT topic is baked in and can be used programatically in generated proto libraries/SDKs.

🏗️ Detailed design

MQTTX UI ➡️ Connection ➡️ Edit ➡️ Advanced ➡️ Protobuf Spec (additional options could be defined here e.g. MQTT Topic Mapping feature)

Support automatic mapping of protobuf messages to MQTT topics using MessageOptions e.g.

edition = "2023";

package mqtt.v5;

import "google/protobuf/descriptor.proto";

extend google.protobuf.MessageOptions {
  string topic = 50050;
  string response_topic = 50051;
  string response_message = 50052;
  uint32 qos = 50053;
  bool retain = 50054;
}

message Telemetry {
  option (mqtt.v5.topic) = "unit/+/telemetry";
  option (mqtt.v5.response_topic) = "unit/+/telemetry/ack";
  option (mqtt.v5.response_message) = "TelemetryResponse";
  option (mqtt.v5.retain) = true;
  option (mqtt.v5.qos) = 1;
  
  uint32 cpu_pct = 1;
}

✉️ Automatic Serialization and Deserialization

In the UI...

• All the topics will be available in the topic dropdown when publishing a message
• When selecting the protobuf mqtt_topic to publish, a default representation in JSON is shown in the payload area with default values e.g. uint32 = 0.

⚡ Zero Overhead

MessageOptions are metadata stored in the descriptor and do NOT Affect Message Serialization or Size

Custom options (including your MQTT metadata options) have ZERO impact on:

✅ Wire format - Options are NOT serialized when messages are sent over the network
✅ Message size - Options don't add bytes to your serialized messages
✅ Runtime performance - No serialization/deserialization overhead
✅ Bandwidth - No additional data transmitted

Alternatives

Annotations could be used, however this method is not machine-readable without custom parsing

/**
 * @Topic unit/+/telemetry
 */
message Telemetry { ... }

More detail (optional)

Further options could be standardized to satisfy other features e.g.

message Telemetry {
  option (metrics.v1.mqtt_topic) = "unit/+/get/request";
  option (metrics.v1.mqtt_response_topic) = "unit/+/get/response";
  option (metrics.v1.mqtt_response_message) = "GetResponse";
  option (metrics.v1.mqtt_retain) = true;
  option (metrics.v1.mqtt_qos) = 1;
  
  uint32 cpu_pct = 1;
}

[alebar42]

Relates to #1971 and #1371

[alebar42]

I will attempt a PR for this, if there are no objections to this idea?

[ysfscream]

@alebar42 welcome! Happy to discuss on the PR.

[alebar42]

@ysfscream great! do you have any feedback? Where would you put the configuration of the Proto file? Should it be on the connection level as proposed?

[ysfscream]

OK. I want to confirm — your idea is that with a single .proto file, we can define everything through metadata:
What the data looks like, where it should be sent, what type of message it is, and which topic it should be sent to.
So the system can automatically use this .proto file for encoding and decoding, right?
And the user only needs to import this .proto file in the connection page?

[alebar42]

OK. I want to confirm — your idea is that with a single .proto file, we can define everything through metadata: What the data looks like, where it should be sent, what type of message it is, and which topic it should be sent to. So the system can automatically use this .proto file for encoding and decoding, right? And the user only needs to import this .proto file in the connection page?

yes exactly 👍