Build public Collector pool

[austinlparker]

We need a public pool of OpenTelemetry Collectors for a few reasons --

• Receiving GHA telemetry for public CI/CD dashboards
• Receiving telemetry from opentelemetry.io
• Receiving telemetry from other OpenTelemetry properties/SIGs (e.g., on-demand Collectors for other infra projects)

Currently, we don't have a normalized way to do this, and collectors for these purposes are being run/managed ad-hoc.

A few hard requirements --

• Collectors must run under opentelemetry.io host (to avoid CORS weirdness)
• Collectors must have WAF or other security/management layer, and it needs to be configurable per-route.
• Self-service deployment/config of Collectors (via PR workflows ideally)

[austinlparker]

Plan:

• Set up a ARM K8s cluster in OCI
• Set up Argo on that cluster and OTel Operator
• We will deploy cluster pools from k8s manifests in this repo
• Argo GitOps workflow against the manifest directory to handle deployments
• Set up meta-monitoring manifest
• Set up CF WAF and cloudflared
• Use TF + State in R2 for CF

@adrielp

[adrielp]

Sounds good. Will start with the bootstrap of the cluster & then move to Argo. I presume we'll continue to use this single repository where the Terraform for bootstrapping K8S + Argo will be in one directory, and the manifests will be ventured in another. We'll likely want to eventually add directory based semantic versioning so that things can be versioned separately but in the same repository.

I've went and associated the Project Infra GitHub project with this repo. Will break out this work and associate it with that board if no objections.