Update charts to support queue modes and queue depth scaling

chart/openfaas/README.md

@@ -77,7 +77,7 @@ Now decide how you want to expose the services and edit the `helm upgrade` comma
 #### Deploy OpenFaaS Community Edition (CE)
 
 > OpenFaaS Community Edition is meant exploration and development.
-> 
+>
 > OpenFaaS Pro has been tuned for production use including flexible auto-scaling, high-available deployments, durability, add-on features, and more.
 
 Deploy CE from the helm chart repo directly:
@@ -310,7 +310,7 @@ Use the following guide to setup TLS for the [Gateway and Dashboard](https://doc
 If you are using Ingress locally, for testing, then you can access the gateway by adding:
 
 ```yaml
-ingress: 
+ingress:
   enabled: true
 ```
 
@@ -446,7 +446,7 @@ Then delete the CRDs:
 kubectl delete crd -l app.kubernetes.io/name=openfaas
 ```
 
-If you have created additional namespaces for functions, delete those too, with `kubectl delete namespace <namespace>`. 
+If you have created additional namespaces for functions, delete those too, with `kubectl delete namespace <namespace>`.
 
 ## Kubernetes versioning
 
@@ -495,7 +495,6 @@ See [values.yaml](./values.yaml) for detailed configuration.
 | `openfaasImagePullPolicy` | Image pull policy for openfaas components, can change to `IfNotPresent` in offline env | `Always` |
 | `openfaasPro` | Deploy OpenFaaS Pro | `false` |
 | `psp` | Enable [Pod Security Policy](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) for OpenFaaS accounts | `false` |
-| `queueMode` | Set to `jetstream` to run the async system backed by NATS JetStream. By default the async system uses NATS Streaming|
 | `rbac` | Enable RBAC | `true` |
 | `registryPrefix` | Adds a prefix or replaces the server prefix for all images in chart i.e. `nats:2.11.6` becomes `registryPrefix/nats:2.11.6` | `""` |
 | `securityContext` | Give a `securityContext` template to be applied to each of the various containers in this chart, set to `{}` to disable, if required for Istio side-car injection.  | See values.yaml |
@@ -575,9 +574,11 @@ See [values.yaml](./values.yaml) for detailed configuration.
 
 | Parameter               | Description                           | Default                                                    |
 | ----------------------- | ----------------------------------    | ---------------------------------------------------------- |
-| `jetstreamQueueWorker.durableName` | Durable name used by JetStream consumers | `faas-workers` |
+| `jetstreamQueueWorker.mode` | Queue operation mode: `static` or `function` | `static` |
+| `jetstreamQueueWorker.durableName` | Deprecated: Durable name used by JetStream consumers | `faas-workers` |
 | `jetstreamQueueWorker.image` | Container image used for the queue-worker when the `queueMode` is `jetstream` | See [values.yaml](./values.yaml) |
-| `jetstreamQueueWorker.maxWaiting` | Configure the max waiting pulls for the queue-worker JetStream consumer. The value should be at least max_inflight * queue_worker.replicas. Note that this value can not be updated once the consumer is created. | `512` |
+| `jetstreamQueueWorker.consumer.inactiveThreshold` | If a function is inactive (has no invocations) for longer than this threshold its consumer will be removed to save resources | `30s` |
+| `jetstreamQueueWorker.consumer.pullMaxMessages` | PullMaxMessages limits the number of messages to be buffered per consumer. Leave empty to use optimized default for the selected queue mode | `` |
 | `jetstreamQueueWorker.logs.debug` | Log debug messages | `false` |
 | `jetstreamQueueWorker.logs.format` | Set the log format, supports `console` or `json` | `console` |
 | `nats.channel` | The name of the NATS Streaming channel or NATS JetStream stream to use for asynchronous function invocations | `faas-request` |
@@ -596,7 +597,6 @@ See [values.yaml](./values.yaml) for detailed configuration.
 | `queueWorker.queueGroup` | The name of the queue group used to process asynchronous function invocations | `faas` |
 | `queueWorkerPro.backoff` | The backoff algorithm used for retries. Must be one off `exponential`, `full` or `equal`| `exponential` |
 | `queueWorkerPro.httpRetryCodes` | Comma-separated list of HTTP status codes the queue-worker should retry | `408,429,500,502,503,504` |
-| `queueWorkerPro.image` | Container image used for the Pro version of the queue-worker | See [values.yaml](./values.yaml) |
 | `queueWorkerPro.initialRetryWait` | Time to wait for the first retry | `10s` |
 | `queueWorkerPro.insecureTLS` | Enable insecure TLS for callback invocations | `false` |
 | `queueWorkerPro.maxRetryAttempts` | Amount of times to try sending a message to a function before discarding it | `10` |
@@ -616,7 +616,7 @@ See [values.yaml](./values.yaml) for detailed configuration.
 | `iam.dashboardIssuer.clientSecret` | Name of the Kubernetes secret that contains the OAuth client secret for the dashboard | `""` |
 | `iam.dashboardIssuer.scopes` | OpenID Connect (OIDC) scopes for the dashboard | `[openid, email, profile]` |
 | `iam.kubernetesIssuer.create` | Create a JwtIssuer object for the kubernetes service account issuer | `true` |
-| `iam.kubernetesIssuer.tokenExpiry` | Expiry time of OpenFaaS access tokens exchanged for tokens issued by the Kubernetes issuer. | `2h` | 
+| `iam.kubernetesIssuer.tokenExpiry` | Expiry time of OpenFaaS access tokens exchanged for tokens issued by the Kubernetes issuer. | `2h` |
 | `iam.kubernetesIssuer.url` | URL for the Kubernetes service account issuer. | `https://kubernetes.default.svc.cluster.local` |
 
 ### Dashboard (OpenFaaS Pro)

chart/openfaas/templates/NOTES.txt

@@ -17,13 +17,6 @@ These should only be used for development. See: https://docs.openfaas.com/openfa
 
 {{- end}}
 
-{{- if and .Values.openfaasPro (ne .Values.queueMode "jetstream") }}
-
-Warning: Invalid queueMode when openfaasPro is set to true.
-Set queueMode to "jetstream", see values-pro.yaml for recommended settings.
-
-{{- end }}
-
 {{- if and .Values.openfaasPro (not .Values.async) }}
 
 Warning: It is not recommended to disable async for OpenFaaS Standard/Enterprise.

chart/openfaas/templates/gateway-dep.yaml

@@ -178,10 +178,6 @@ spec:
           value: "{{ .Values.nats.channel }}"
         {{- end }}
         {{- end }}
-        {{- if eq .Values.queueMode "jetstream" }}
-        - name: nats_jetstream
-          value: "true"
-        {{- end }}
         {{- if .Values.basic_auth }}
         - name: basic_auth
           value: "true"

chart/openfaas/templates/nats-dep.yaml

@@ -30,7 +30,7 @@ spec:
       - name:  nats
         resources:
           {{- .Values.nats.resources | toYaml | nindent 12 }}
-        {{- if eq .Values.queueMode "jetstream" }}
+        {{- if .Values.openfaasPro }}
         image: {{ include "openfaas.image" (dict "image" .Values.nats.image "registryPrefix" .Values.registryPrefix) }}
         {{- else }}
         image: {{ include "openfaas.image" (dict "image" .Values.stan.image "registryPrefix" .Values.registryPrefix) }}
@@ -39,11 +39,11 @@ spec:
         ports:
         - containerPort: 4222
           protocol: TCP
-        {{- if not (eq .Values.queueMode "jetstream") }}
+        {{- if not .Values.openfaasPro }}
         command: ["/nats-streaming-server"]
         {{- end }}
         args:
-          {{- if eq .Values.queueMode "jetstream" }}
+          {{- if .Values.openfaasPro }}
           - --js
           {{- else}}
           - --store

chart/openfaas/templates/prometheus-pro-cfg.yaml

@@ -142,7 +142,7 @@ data:
           regex: '(.*);(.*)'
           replacement: '${1}.${2}'
           target_label: "function_name"
-      
+
       - job_name: 'openfaas-function-pods'
         scrape_interval: 15s
         kubernetes_sd_configs:
@@ -181,7 +181,7 @@ data:
             regex: '^([0-9a-zA-Z-]+)+(-[0-9a-zA-Z]+-[0-9a-zA-Z]+)$'
             replacement: '$1'
             target_label: deployment_name
-          
+
         metric_relabel_configs:
         - source_labels: [deployment_name, kubernetes_namespace]
           separator: ";"
@@ -207,7 +207,12 @@ data:
         expr: ceil(sum(irate ( pod_cpu_usage_seconds_total{}[1m])*1000) by (function_name) * on (function_name) avg by (function_name) (gateway_service_target_load{scaling_type="cpu"}  > bool 1 ))
         labels:
           scaling_type: cpu
-
+
+      - record: job:function_current_load:sum
+        expr: avg(queue_worker_pending_messages{}) by (function_name) and on (function_name) avg by (function_name) (gateway_service_target_load{scaling_type="queue"})  > bool 1
+        labels:
+          scaling_type: queue
+
       {{- with .Values.prometheus.recordingRules }}
       {{ toYaml . | nindent 6 }}
       {{- end }}

chart/openfaas/templates/queueworker-ce-dep.yaml

@@ -0,0 +1,90 @@
+{{- if and .Values.async (not .Values.openfaasPro) }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    app: {{ template "openfaas.name" . }}
+    chart: {{ .Chart.Name }}-{{ .Chart.Version }}
+    component: queue-worker
+    heritage: {{ .Release.Service }}
+    release: {{ .Release.Name }}
+  name: queue-worker
+  namespace: {{ .Release.Namespace | quote }}
+spec:
+  replicas: {{ .Values.queueWorker.replicas }}
+  selector:
+    matchLabels:
+      app: queue-worker
+  template:
+    metadata:
+      annotations:
+        prometheus.io/scrape: "false"
+      labels:
+        app: queue-worker
+    spec:
+      {{- with .Values.imagePullSecrets }}
+      imagePullSecrets:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      volumes:
+      {{- if .Values.basic_auth }}
+      - name: auth
+        secret:
+          secretName: basic-auth
+      {{- end }}
+      containers:
+      - name: queue-worker
+        resources:
+          {{- .Values.queueWorker.resources | toYaml | nindent 12 }}
+        image: {{ include "openfaas.image" (dict "image" .Values.queueWorker.image "registryPrefix" .Values.registryPrefix) }}
+        imagePullPolicy: {{ .Values.openfaasImagePullPolicy }}
+        env:
+        {{- if .Values.nats.external.enabled }}
+        - name: faas_nats_address
+          value: "{{ .Values.nats.external.host }}"
+        - name: faas_nats_port
+          value: "{{ .Values.nats.external.port }}"
+        - name: faas_nats_cluster_name
+          value: "{{ .Values.nats.external.clusterName }}"
+        {{- else }}
+        - name: faas_nats_address
+          value: "nats.{{ .Release.Namespace }}.svc.{{ .Values.kubernetesDNSDomain }}"
+        {{- end}}
+        - name: faas_nats_channel
+          value: "{{ .Values.nats.channel }}"
+        - name: faas_nats_queue_group
+          value: "{{ .Values.queueWorker.queueGroup }}"
+        - name: faas_gateway_address
+          value: "gateway.{{ .Release.Namespace }}.svc.{{ .Values.kubernetesDNSDomain }}"
+        {{- if .Values.functionNamespace }}
+        - name: faas_function_suffix
+          value: ".{{ .Values.functionNamespace }}.svc.{{ .Values.kubernetesDNSDomain }}"
+        {{- end }}
+        - name: ack_wait
+          value: "{{ .Values.queueWorker.ackWait }}"
+        - name: max_inflight
+          value: "{{ or (.Values.queueWorker.maxInflight ) ("1") }}"
+
+        {{- if .Values.basic_auth }}
+        - name: secret_mount_path
+          value: "/var/secrets/gateway"
+        - name: basic_auth
+          value: "{{ .Values.basic_auth }}"
+        volumeMounts:
+        - name: auth
+          readOnly: true
+          mountPath: "/var/secrets/gateway"
+        {{- end }}
+    {{- with .Values.nodeSelector }}
+      nodeSelector:
+{{ toYaml . | indent 8 }}
+    {{- end }}
+    {{- with .Values.affinity }}
+      affinity:
+{{ toYaml . | indent 8 }}
+    {{- end }}
+    {{- with .Values.tolerations }}
+      tolerations:
+{{ toYaml . | indent 8 }}
+    {{- end }}
+{{- end }}