fix(backend): Allow pod namespace configuration on env.go metadata-grpc service address #12428

alyssacgoins · 2025-11-10T18:16:21Z

Description of your changes:
Update backend/src/v2/metadata/env.go DefaultConfig() to return a metadata service config containing the pod-specific namespace. This also includes a change to backend/src/apiserver/common/config.go GetPodNamespace() that sets the default pod namespace to "kubeflow"

Checklist:

You have signed off your commits
The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

gmfrasca · 2025-11-10T20:43:26Z

backend/src/apiserver/common/config.go

 	CaBundleSecretName                      string = "CABUNDLE_SECRET_NAME"
 	RequireNamespaceForPipelines            string = "REQUIRE_NAMESPACE_FOR_PIPELINES"
 	CompiledPipelineSpecPatch               string = "COMPILED_PIPELINE_SPEC_PATCH"
+	DefaultPodNamespace                     string = "kubeflow"


Just for consistency, I think this should go in backend/src/apiserver/common/const.go

gmfrasca

/lgtm

gmfrasca · 2025-11-10T23:15:55Z

backend/metadata_writer/src/metadata_helpers.py


-
+# Returns the default_value if the environment variable is not found.
+def get_env_var_with_default(config_name: str, default_value: str) -> str:


this is essentially just renaming os.getenv, so we should probably just use that directly rather than create a new function

backend/src/v2/metadata/env.go

mprahl · 2025-11-11T01:14:27Z

backend/metadata_writer/src/metadata_helpers.py

-    metadata_service_host = os.environ.get("METADATA_GRPC_SERVICE_SERVICE_HOST", "metadata-grpc-service.kubeflow")
-    metadata_service_port = int(os.environ.get("METADATA_GRPC_SERVICE_SERVICE_PORT", 8080))
-    metadata_service_host = "metadata-grpc-service.kubeflow"
+    pod_namespace = get_env_var_with_default("POD_NAMESPACE", "kubeflow")


Could you check to see if metadawriter runs in the user's namespace in the multi-user mode or in the kubeflow namespace? If it's the former, then the POD_NAMESPACE will be incorrect similar to my comment on driver.

Pre-upstream TLS implementation, metadata-writer hard-coded the service host as "metadata-grpc-service.kubeflow", (https://github.com/kubeflow/pipelines/blob/e31038da5f124e309f5e71193bce0c0959acb7e0/backend/metadata_writer/src/metadata_helpers.py) and this env var is only set in the TLS manifests - so it looks like metadata-writer runs in the kubeflow namespace only

backend/metadata_writer/src/metadata_helpers.py

mprahl · 2025-11-12T17:50:33Z

backend/src/v2/driver/driver.go

-		fmt.Sprintf("$(%s)", component.EnvMetadataHost),
-		"--mlmd_server_port",
-		fmt.Sprintf("$(%s)", component.EnvMetadataPort),
+		"--mlmd_server_address", mlmdServerAddress,


We also need this change in addImporterTemplate.

mprahl · 2025-11-12T17:53:35Z

backend/src/v2/driver/driver.go

-		fmt.Sprintf("$(%s)", component.EnvMetadataHost),
-		"--mlmd_server_port",
-		fmt.Sprintf("$(%s)", component.EnvMetadataPort),
+		"--mlmd_server_address", mlmdServerAddress,


This will also need the equivalent change in addContainerExecutorTemplate. Please scan for other areas.

Because I added the mlmd address/host in driver.initPodSpecpatch(), those arguments are applied to the container executor pod - but does it make more sense to add those arguments via addContainerExecutorTemplate() instead?

Yeah, let's just let the API server own that to make it consistent.

Sounds good - updated. Removed these two flags from initPodSpecPatch(), and added them to addContainerExecutorTemplate()

mprahl

/approve
/lgtm

…space configurable. Signed-off-by: agoins <[email protected]> # Conflicts: # test_data/compiled-workflows/ray_job_integration_compiled.yaml

mprahl

/approve
/lgtm

google-oss-prow · 2025-11-13T20:51:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mprahl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mprahl]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow bot requested review from droctothorpe and rimolive November 10, 2025 18:16

google-oss-prow bot added the size/S label Nov 10, 2025

alyssacgoins changed the title ~~Update env.go~~ fix(backend): Allow pod namespace configuration on env.go metadata-grpc service address Nov 10, 2025

alyssacgoins force-pushed the update-metadata-env-config branch 3 times, most recently from a905fc7 to 3f769fb Compare November 10, 2025 19:43

gmfrasca reviewed Nov 10, 2025

View reviewed changes

alyssacgoins mentioned this pull request Nov 10, 2025

Revert changes to backend/src/v2/metadata/env.go. opendatahub-io/data-science-pipelines#213

Closed

2 tasks

alyssacgoins force-pushed the update-metadata-env-config branch from 3f769fb to bb58bbc Compare November 10, 2025 21:37

gmfrasca reviewed Nov 10, 2025

View reviewed changes

google-oss-prow bot assigned gmfrasca Nov 10, 2025

google-oss-prow bot added the lgtm label Nov 10, 2025

alyssacgoins force-pushed the update-metadata-env-config branch from bb58bbc to db5e7d6 Compare November 10, 2025 21:54

google-oss-prow bot removed the lgtm label Nov 10, 2025

gmfrasca reviewed Nov 10, 2025

View reviewed changes

mprahl reviewed Nov 11, 2025

View reviewed changes

backend/src/v2/metadata/env.go Show resolved Hide resolved

mprahl reviewed Nov 11, 2025

View reviewed changes

backend/src/v2/metadata/env.go Show resolved Hide resolved

mprahl reviewed Nov 11, 2025

View reviewed changes

google-oss-prow bot added size/M and removed size/S labels Nov 11, 2025

alyssacgoins force-pushed the update-metadata-env-config branch from 8c44353 to 5e7d656 Compare November 12, 2025 16:30

google-oss-prow bot added size/XXL and removed size/M labels Nov 12, 2025

mprahl reviewed Nov 12, 2025

View reviewed changes

backend/metadata_writer/src/metadata_helpers.py Show resolved Hide resolved

mprahl reviewed Nov 12, 2025

View reviewed changes

alyssacgoins force-pushed the update-metadata-env-config branch 3 times, most recently from 3b113ca to e8be142 Compare November 12, 2025 18:10

alyssacgoins force-pushed the update-metadata-env-config branch 4 times, most recently from 851ddd7 to 9036661 Compare November 12, 2025 19:45

google-oss-prow bot added size/XL and removed size/XXL labels Nov 12, 2025

alyssacgoins force-pushed the update-metadata-env-config branch from 9036661 to 7510781 Compare November 12, 2025 19:48

google-oss-prow bot added size/XXL and removed size/XL labels Nov 12, 2025

mprahl approved these changes Nov 12, 2025

View reviewed changes

google-oss-prow bot assigned mprahl Nov 12, 2025

google-oss-prow bot added lgtm approved labels Nov 12, 2025

alyssacgoins force-pushed the update-metadata-env-config branch from 7510781 to 0f0aced Compare November 12, 2025 21:09

google-oss-prow bot removed the lgtm label Nov 12, 2025

alyssacgoins force-pushed the update-metadata-env-config branch 2 times, most recently from c6aa124 to d9318d2 Compare November 13, 2025 19:03

Update metadata-grpc-service host/port configuration to make pod name…

faf43c6

…space configurable. Signed-off-by: agoins <[email protected]> # Conflicts: # test_data/compiled-workflows/ray_job_integration_compiled.yaml

alyssacgoins force-pushed the update-metadata-env-config branch from d9318d2 to faf43c6 Compare November 13, 2025 19:29

mprahl approved these changes Nov 13, 2025

View reviewed changes

google-oss-prow bot added the lgtm label Nov 13, 2025

google-oss-prow bot merged commit caecb1e into kubeflow:master Nov 13, 2025
190 of 193 checks passed



		# Returns the default_value if the environment variable is not found.
		def get_env_var_with_default(config_name: str, default_value: str) -> str:

fix(backend): Allow pod namespace configuration on env.go metadata-grpc service address #12428

fix(backend): Allow pod namespace configuration on env.go metadata-grpc service address #12428

Uh oh!

Conversation

alyssacgoins commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gmfrasca Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmfrasca left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mprahl left a comment

Choose a reason for hiding this comment

Uh oh!

mprahl left a comment

Choose a reason for hiding this comment

Uh oh!

google-oss-prow bot commented Nov 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alyssacgoins commented Nov 10, 2025 •

edited

Loading

gmfrasca Nov 10, 2025 •

edited

Loading