Add support to authenticate to GCP via GOOGLE_APPLICATION_CREDENTIALS file

[mrodm]

Add support in elastic-package to use GOOGLE_APPLICATION_CREDENTIALS environment variable to be able to authenticate against GCP provider (documentation).

This variable is set in CI via this Buildkite plugin:

https://github.com/mrodm/oblt-google-auth-buildkite-plugin/blob/2468fe096ba7f559b0b0b63e9ec17c7a2ab12fab/hooks/environment#L32

In order to be used by elastic-package in system tests, this variable must be added to:

• terraform container
  • required to create the expected resources
    

    elastic-package/internal/servicedeployer/_static/terraform_deployer.yml

    Lines 13 to 16 in a1cc1b4

    	volumes:
    	- ${TF_DIR}:/stage
    	- ${TF_OUTPUT_DIR}:/output
    	- ${SERVICE_LOGS_DIR}:/tmp/service_logs/

• elastic-agent container
  • elastic-agent must have access to the same files
    

    elastic-package/internal/stack/_static/docker-compose-stack.yml.tmpl

    Lines 138 to 162 in a1cc1b4

    	elastic-agent:
    	image: "${ELASTIC_AGENT_IMAGE_REF}"
    	depends_on:
    	fleet-server:
    	condition: service_healthy
    	healthcheck:
    	test: "elastic-agent status"
    	timeout: 2s
    	start_period: 360s
    	retries: 180
    	interval: 5s
    	hostname: docker-fleet-agent
    	env_file: "./elastic-agent.env"
    	cap_drop:
    	- ALL
    	ports: [{{ fact "agent_publish_ports" }}]
    	volumes:
    	- "../certs/ca-cert.pem:/etc/ssl/certs/elastic-package.pem"
    	- type: bind
    	source: ../../../tmp/service_logs/
    	target: /tmp/service_logs/
    	# Mount service_logs under /run too as a testing workaround for the journald input (see elastic-package#1235).
    	- type: bind
    	source: ../../../tmp/service_logs/
    	target: /run/service_logs/

According to that Buildkite plugin, it should also be mounted TOKEN_FILE. This file path can be obtained via in the Buildkite pre-command hook: "${BUILDKITE_OIDC_TMPDIR}"/token.json

Requisites:

• Test all packages in integrations repository
• Test scenarios where those environment variables are not defined (e.g. our own laptops)
• It does not affect to the current authentication to upload files to the private bucket.

As context, previouslyelastic-package was testing a package using GCP resources via the GOOGLE_CREDENTIALS environment variable:

• elastic-package/test/packages/parallel/gcp/data_stream/compute/_dev/deploy/tf/env.yml

  Line 7 in 364fd63

  - GOOGLE_CREDENTIALS=${GOOGLE_CREDENTIALS}

• elastic-package/test/packages/parallel/gcp/data_stream/compute/agent/stream/stream.yml.hbs

  Line 8 in 364fd63

  credentials_json: '{{credentials_json}}'

• elastic-package/test/packages/parallel/gcp/data_stream/compute/_dev/test/system/test-default-config.yml

  Line 4 in 364fd63

  credentials_json: '{{{GOOGLE_CREDENTIALS}}}'

• elastic-package/.buildkite/hooks/pre-command

  Lines 98 to 102 in 364fd63

  	ELASTIC_PACKAGE_GCP_CREDENTIALS_SECRET=$(retry 5 vault read -field credentials ${GCP_SERVICE_ACCOUNT_SECRET_PATH} | jq -c)
  	export ELASTIC_PACKAGE_GCP_CREDENTIALS_SECRET
  	# Environment variables required by the service deployer
  	export GOOGLE_CREDENTIALS=${ELASTIC_PACKAGE_GCP_CREDENTIALS_SECRET}

This package was removed in #1930

Relates

• [panw_cortex_xdr] Add event data stream and dashboards of incident and alert integrations#13680
• Panw cortex xdr enhancement test - DO NOT MERGE integrations#13948

[moxarth-rathod]

Hi, I’m encountering the same issue with the CI in this Netskope system test PRs - elastic/integrations#14836 and elastic/integrations#14887 for GCS which uses the Terraform deployer.

[chrisberkhout]

@mrodm I'm a little confused about this issue. Is this about fixing something that's broken, or just adding support for a different method of providing credentials in tests?

The input supports both:

• "json service account credentials string" (auth.credentials_json.account_key)
• "service account credentials file" (auth.credentials_file.path)

Can an system test in CI pass currently by reading GOOGLE_CREDENTIALS from the environment and setting that as auth.credentials_json.account_key in the input config?

[mrodm]

Hi @chrisberkhout ,

The main issue is that after switching the ephemeral tokens for Google resources (Buildkite plugin), the credentials file contains a reference to another file inside the JSON:

https://github.com/elastic/oblt-google-auth-buildkite-plugin/blob/2468fe096ba7f559b0b0b63e9ec17c7a2ab12fab/hooks/environment#L32-L41

And that forces to add support to include both files in the terraform and the elastic-agent containers to be able to perform the authentication/authorization.

[chrisberkhout]

@mrodm Okay, thanks. So using a service account key would work, but now we want to use the Workload Identity Federation method, at least in CI, so the change is needed for that.

I was looking into how Workload Identity Federation will work for the GCS input, and this is what I found:
We won't set either auth.credentials_json.account_key or auth.credentials_file.path. In that case, the input will use google.FindDefaultCredentials (here) which looks for credentials in several places, starting with a JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS environment variable. That can provide the information for Workload Identity Federation.

So if we use the (empty) config that makes the input find default credentials, it can find service account credentials in local test runs, and find Workload Identity Federation credentials in CI test runs.