Skip to content

Commit dc25c72

Browse files
baburciubaburciu
authored
Chore: Add example of GAMMA HTTPRoute header-based traffic split with Cilium (#150)
Signed-off-by: Bogdan-Adrian Burciu <[email protected]>
1 parent bc8d387 commit dc25c72

File tree

9 files changed

+342
-4
lines changed

9 files changed

+342
-4
lines changed

‎docs/provider-status.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ For convenience we are including here a list of those actually tested with the p
1616
| Provider | Version | API Version | Plugin | Code |
1717
|------------|------------|-------------| ---------| ---------|
1818
| [Amazon VPC Lattice](https://www.gateway-api-controller.eks.aws.dev/latest//) | 1.1.2 | 1.3.0 | 0.6.0 | [Example](https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/tree/main/examples/aws-gateway-api-controller-lattice ) |
19-
| [Cilium](https://cilium.io/) | Unknown | 0.7.0 | 0.2.0 | [Example](https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/tree/main/examples/cilium) |
19+
| [Cilium](https://cilium.io/) | 1.18.2 | 1.2.0 | 0.8.0 | [Example](https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/tree/main/examples/cilium-header-based) |
2020
| [Envoy Gateway](https://gateway.envoyproxy.io/) | 0.5.0 | Unknown | 0.2.0 | [Example](https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/tree/main/examples/envoygateway) |
2121
| [Gloo Gateway](https://docs.solo.io/gloo-gateway/v2/) | 2.0.0-beta | 1.0 | 0.2.0 | [Example](https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/tree/main/examples/gloo-gateway) |
2222
| [Google Cloud](https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-api) | N/A | 0.7.0 | 0.2.0 | [Example](https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/tree/main/examples/google-cloud) |

‎examples/cilium-header-based/README.md‎

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
# Using Cilium with Argo Rollouts for header based traffic split
2+
3+
## Prerequisites
4+
5+
A Kubernetes cluster. If you do not have one, you can create one using [kind](https://kind.sigs.k8s.io/), [minikube](https://minikube.sigs.k8s.io/), or any other Kubernetes cluster. This guide will use Kind.
6+
7+
## Step 1 - Create a Kind cluster by running the following command
8+
9+
```shell
10+
kind create cluster --config ./kind-cluster.yaml
11+
```
12+
13+
## Step 2 - Install Cilium
14+
15+
I will use helm to install Cilium in the cluster, but before that we'll need to install Gateway API CRDs. You can also install Cilium using [cilium CLI](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli).
16+
17+
> [!NOTE]
18+
> Cilium `v1.18.2` supports Gateway API v1.2.0, per [docs](https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/).
19+
20+
```shell
21+
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
22+
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
23+
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
24+
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
25+
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.0/config/crd/standard/gateway.networking.k8s.io_grpcroutes.yaml
26+
```
27+
```shell
28+
helm repo add cilium https://helm.cilium.io/
29+
helm repo update
30+
helm install cilium cilium/cilium --version 1.18.2 \
31+
--namespace kube-system \
32+
--set image.pullPolicy=IfNotPresent \
33+
--set ipam.mode=kubernetes \
34+
--set cni.exclusive=false \
35+
--set kubeProxyReplacement=true \
36+
--set gatewayAPI.enabled=true \
37+
--wait
38+
cilium status --wait
39+
```
40+
41+
## Step 3 - Install Argo Rollouts and Argo Rollouts plugin to instruct Cilium to manage the traffic
42+
43+
```shell
44+
helm repo add argo https://argoproj.github.io/argo-helm
45+
helm repo update
46+
helm install argo-rollouts argo/argo-rollouts --version 2.40.4 \
47+
--namespace argo-rollouts \
48+
--create-namespace \
49+
--set 'controller.trafficRouterPlugins[0].location=https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/releases/download/v0.8.0/gatewayapi-plugin-linux-amd64' \
50+
--set 'controller.trafficRouterPlugins[0].name=argoproj-labs/gatewayAPI'
51+
```
52+
53+
## Step 4 - Create the services required for traffic split
54+
55+
Create three Services required for canary based rollout strategy
56+
57+
```shell
58+
kubectl apply -f service.yaml
59+
```
60+
61+
## Step 5 - Create HTTPRoute that defines a traffic split between two services
62+
63+
> [!IMPORTANT]
64+
> For Cilium the K8s Services refs need to use `group: ""`. This is different than Linkerd, where `group: "core"` could be used.
65+
```yaml
66+
apiVersion: gateway.networking.k8s.io/v1beta1
67+
kind: HTTPRoute
68+
spec:
69+
parentRefs:
70+
- group: ""
71+
name:
72+
kind: Service
73+
port:
74+
rules:
75+
- backendRefs:
76+
- group: ""
77+
name:
78+
kind: Service
79+
port:
80+
```
81+
82+
Create a GAMMA [producer `HTTPRoute`](https://gateway-api.sigs.k8s.io/concepts/glossary/#producer-route) resource and connect it to a parent K8s service (using a canary and stable K8s services as backends)
83+
84+
```shell
85+
kubectl apply -f httproute.yaml
86+
```
87+
88+
## Step 6 - Create an example Rollout
89+
90+
Deploy a rollout to get the initial version
91+
92+
```shell
93+
kubectl apply -f rollout.yaml
94+
```
95+
96+
## Step 7 - Patch the rollout to see the canary deployment
97+
```shell
98+
kubectl patch rollout rollouts-demo --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/env/0/value", "value": "1.1.0"}]'
99+
```
100+
101+
## Step 8 - Observe the rollout and HTTPRoute rule addition of [canary header matching rule](https://gateway-api.sigs.k8s.io/guides/traffic-splitting/#canary-traffic-rollout)
102+
103+
```shell
104+
$ kubectl argo rollouts promote rollouts-demo # promote to Rollout step 1
105+
$ kubectl argo rollouts get rollout rollouts-demo
106+
Name: rollouts-demo
107+
Namespace: default
108+
Status: ॥ Paused
109+
Message: CanaryPauseStep
110+
Strategy: Canary
111+
Step: 3/5
112+
SetWeight: 0
113+
ActualWeight: 0
114+
Images: hashicorp/http-echo:1.0 (canary, stable)
115+
Replicas:
116+
Desired: 5
117+
Current: 6
118+
Updated: 1
119+
Ready: 6
120+
Available: 6
121+
122+
NAME KIND STATUS AGE INFO
123+
⟳ rollouts-demo Rollout ॥ Paused 114s
124+
├──# revision:2
125+
│ └──⧉ rollouts-demo-7bd564d79f ReplicaSet ✔ Healthy 23s canary
126+
│ └──□ rollouts-demo-7bd564d79f-tshpg Pod ✔ Running 6s ready:1/1
127+
└──# revision:1
128+
└──⧉ rollouts-demo-784858d6db ReplicaSet ✔ Healthy 114s stable
129+
├──□ rollouts-demo-784858d6db-d799l Pod ✔ Running 114s ready:1/1
130+
├──□ rollouts-demo-784858d6db-hh44q Pod ✔ Running 114s ready:1/1
131+
├──□ rollouts-demo-784858d6db-nf2wh Pod ✔ Running 114s ready:1/1
132+
├──□ rollouts-demo-784858d6db-qn7dc Pod ✔ Running 114s ready:1/1
133+
└──□ rollouts-demo-784858d6db-ww2q5 Pod ✔ Running 114s ready:1/1
134+
$
135+
$ kubectl get httproute argo-rollouts-http-route -o yaml | yq .spec.rules
136+
- backendRefs:
137+
- group: ""
138+
kind: Service
139+
name: argo-rollouts-stable-service
140+
port: 80
141+
weight: 100
142+
- group: ""
143+
kind: Service
144+
name: argo-rollouts-canary-service
145+
port: 80
146+
weight: 0
147+
matches:
148+
- path:
149+
type: PathPrefix
150+
value: /
151+
- backendRefs:
152+
- group: ""
153+
kind: Service
154+
name: argo-rollouts-canary-service
155+
port: 80
156+
weight: 0
157+
matches:
158+
- headers:
159+
- name: X-Test
160+
type: Exact
161+
value: test
162+
path:
163+
type: PathPrefix
164+
value: /
165+
```
166+
```shell
167+
$ kubectl run -it --image nicolaka/netshoot:v0.13 network-test -- sh # run a pod to source curl tests
168+
~ #
169+
~ # # stable K8s service targets any of the 5 stable pods
170+
~ # seq 1 100 | xargs -P 10 -I {} bash -c 'curl -s http://argo-rollouts-stable-service' > pods.txt
171+
~ # sort pods.txt | uniq -c | sort -rn
172+
26 Hello from rollouts-demo-784858d6db-qn7dc
173+
20 Hello from rollouts-demo-784858d6db-hh44q
174+
19 Hello from rollouts-demo-784858d6db-ww2q5
175+
18 Hello from rollouts-demo-784858d6db-d799l
176+
17 Hello from rollouts-demo-784858d6db-nf2wh
177+
~ #
178+
~ #
179+
~ # # canary K8s service targets the one canary pod, created for the `setCanaryScale` step in the Rollout
180+
~ # seq 1 100 | xargs -P 10 -I {} bash -c 'curl -s http://argo-rollouts-canary-service' > pods.txt
181+
~ # sort pods.txt | uniq -c | sort -rn
182+
100 Hello from rollouts-demo-7bd564d79f-tshpg
183+
~ #
184+
~ #
185+
~ # # GAMMA-type HTTPRoute's `.spec.parentRefs` K8s service only targets stable pods since no `setWeight` step is used in the Rollout
186+
~ # seq 1 100 | xargs -P 10 -I {} bash -c 'curl -s http://argo-rollouts-service' > pods.txt
187+
~ # sort pods.txt | uniq -c | sort -rn
188+
22 Hello from rollouts-demo-784858d6db-d799l
189+
21 Hello from rollouts-demo-784858d6db-hh44q
190+
20 Hello from rollouts-demo-784858d6db-qn7dc
191+
19 Hello from rollouts-demo-784858d6db-ww2q5
192+
18 Hello from rollouts-demo-784858d6db-nf2wh
193+
~ #
194+
```
195+
196+
## Step 12 - Test the header-based routing with curl
197+
198+
You can test the header-based routing by sending requests with the specified header.
199+
With header it always goes to canary:
200+
201+
```shell
202+
$ kubectl exec -it network-test -- sh
203+
~ # seq 1 100 | xargs -P 10 -I {} bash -c 'curl -s -H "X-Test: test" http://argo-rollouts-service' > pods.txt
204+
~ # sort pods.txt | uniq -c | sort -rn
205+
100 Hello from rollouts-demo-7bd564d79f-tshpg
206+
~ #
207+
```

‎examples/cilium-header-based/httproute.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
apiVersion: gateway.networking.k8s.io/v1beta1
3+
kind: HTTPRoute
4+
metadata:
5+
name: argo-rollouts-http-route
6+
namespace: default
7+
spec:
8+
parentRefs:
9+
- group: ""
10+
name: argo-rollouts-service
11+
kind: Service
12+
port: 80
13+
rules:
14+
- backendRefs:
15+
- name: argo-rollouts-stable-service
16+
group: ""
17+
port: 80
18+
kind: Service
19+
- name: argo-rollouts-canary-service
20+
group: ""
21+
port: 80
22+
kind: Service

‎examples/cilium-header-based/kind-cluster.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
kind: Cluster
2+
apiVersion: kind.x-k8s.io/v1alpha4
3+
name: cilium-header-based
4+
nodes:
5+
- role: control-plane
6+
- role: worker
7+
networking:
8+
disableDefaultCNI: true

‎examples/cilium-header-based/rollout.yaml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
---
2+
apiVersion: argoproj.io/v1alpha1
3+
kind: Rollout
4+
metadata:
5+
name: rollouts-demo
6+
namespace: default
7+
spec:
8+
replicas: 5
9+
strategy:
10+
canary:
11+
canaryService: argo-rollouts-canary-service
12+
stableService: argo-rollouts-stable-service
13+
trafficRouting:
14+
plugins:
15+
argoproj-labs/gatewayAPI:
16+
httpRoutes:
17+
- name: argo-rollouts-http-route
18+
useHeaderRoutes: true
19+
namespace: default
20+
managedRoutes:
21+
- name: argo-rollouts
22+
steps:
23+
- pause: {}
24+
- setCanaryScale:
25+
replicas: 1
26+
- setHeaderRoute:
27+
name: argo-rollouts
28+
match:
29+
- headerName: X-Test
30+
headerValue:
31+
exact: test
32+
- pause: {}
33+
- setHeaderRoute: # remove header route
34+
name: argo-rollouts
35+
revisionHistoryLimit: 2
36+
selector:
37+
matchLabels:
38+
app: rollouts-demo
39+
template:
40+
metadata:
41+
labels:
42+
app: rollouts-demo
43+
spec:
44+
containers:
45+
- name: http-echo
46+
image: hashicorp/http-echo:1.0
47+
args:
48+
- "-text=Hello from $(POD_NAME)"
49+
ports:
50+
- name: http
51+
containerPort: 5678
52+
protocol: TCP
53+
env:
54+
- name: APP_VERSION
55+
value: "1.0.0"
56+
- name: POD_NAME
57+
valueFrom:
58+
fieldRef:
59+
fieldPath: metadata.name
60+
resources:
61+
requests:
62+
memory: 32Mi
63+
cpu: 5m

‎examples/cilium-header-based/service.yaml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
---
2+
apiVersion: v1
3+
kind: Service
4+
metadata:
5+
name: argo-rollouts-service
6+
namespace: default
7+
spec:
8+
ports:
9+
- port: 80
10+
targetPort: http
11+
selector:
12+
app: dummy # dummy selector needed for Cilium, see https://github.com/cilium/cilium/issues/38415
13+
---
14+
apiVersion: v1
15+
kind: Service
16+
metadata:
17+
name: argo-rollouts-canary-service
18+
namespace: default
19+
spec:
20+
ports:
21+
- port: 80
22+
targetPort: http
23+
selector:
24+
app: rollouts-demo
25+
---
26+
apiVersion: v1
27+
kind: Service
28+
metadata:
29+
name: argo-rollouts-stable-service
30+
namespace: default
31+
spec:
32+
ports:
33+
- port: 80
34+
targetPort: http
35+
selector:
36+
app: rollouts-demo

‎examples/linkerd-header-based/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ linkerd install | kubectl apply -f - && linkerd check
2929
linkerd viz install | kubectl apply -f - && linkerd check
3030
```
3131

32-
## Step 3 - Install Argo Rollouts and Argo Rollouts plugin to allow Linkerd to manage the traffic:
32+
## Step 3 - Install Argo Rollouts and Argo Rollouts plugin to instruct Linkerd to manage the traffic
3333

3434
```shell
3535
kubectl create namespace argo-rollouts

‎examples/linkerd-header-based/rollout.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ apiVersion: argoproj.io/v1alpha1
33
kind: Rollout
44
metadata:
55
name: rollouts-demo
6+
namespace: default
67
spec:
78
replicas: 5
89
strategy:

‎examples/linkerd-header-based/service.yaml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,17 @@ apiVersion: v1
33
kind: Service
44
metadata:
55
name: argo-rollouts-service
6+
namespace: default
67
spec:
78
ports:
89
- port: 80
910
targetPort: http
10-
selector:
11-
app: rollouts-demo
1211
---
1312
apiVersion: v1
1413
kind: Service
1514
metadata:
1615
name: argo-rollouts-canary-service
16+
namespace: default
1717
spec:
1818
ports:
1919
- port: 80
@@ -25,6 +25,7 @@ apiVersion: v1
2525
kind: Service
2626
metadata:
2727
name: argo-rollouts-stable-service
28+
namespace: default
2829
spec:
2930
ports:
3031
- port: 80

0 commit comments

Comments
 (0)