Gatewayd is no longer stateless after adding LoadBalancer

[sinadarbouy]

Description:

After adding a LoadBalancer to Gatewayd, the application is no longer stateless. This change impacts the architecture and might require adjustments to the Helm chart and deployment strategy.

For example, if we use strategies like weighted load balancing or consistent hashing, data might be stored in the memory of one pod. If a second client request is routed to another pod, that pod will be unaware of the previous state, potentially leading to inconsistent behavior.

Proposal:

• We may need to switch from a Deployment to a StatefulSet to better manage the stateful nature of Gatewayd.
  • if we go with leader election, the order provided by the StatefulSet would be beneficial in ensuring proper sequencing and coordination among pods.
• Implementing leader election or gossip protocols within Gatewayd could help coordinate the nodes, ensuring high availability and consistency across instances.

Suggested Next Steps:

• Explore options for implementing leader election or gossip protocols within Gatewayd.

Any thoughts or feedback on this approach would be appreciated!

[mostafa]

This sounds good and I already wrote an epic ticket a while ago that relates to consensus here:

• High availability gatewayd#169

I'm thinking about implementing Raft, but let's continue the discussion on that ticket and possibly create more linked ticket to that one.