WIP: Add make targets for benchmark operation tooling #20769

kishen-v · 2025-10-07T16:54:16Z

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

Changes in this PR:

Support benchmarking other operations such as lease-keepalive, put, txn-mixed,watch-latency, watch through the benchmark_test.sh script.
Modularise benchmark_test.sh, with supporting logic to set up and teardown the etcd process and the associated DATA_DIR between runs.
Introduce a generic target to support a combination of tests through the make bench target.
Add the make help to document the usage of the benchmark related targets.
Switch to EtcdAPI_benchmark_<benchmark_operation>_<date>.json format to help distingush between files and the associated benchmark, from the earlier EtcdAPI_benchmark_<date>.json format.

Sample Result:

make bench-lease-keepalive bench-put bench-txn-mixed bench-watch-latency bench-watch bench-range-key
GO_BUILD_FLAGS=" -v -mod=readonly" ./scripts/build.sh
Running etcd_build
% 'rm' '-f' 'bin/etcd'
% (cd server && 'env' 'CGO_ENABLED=0' 'GO_BUILD_FLAGS= -v -mod=readonly' 'GOOS=darwin' 'GOARCH=arm64' 'go' 'build' '-v' '-mod=readonly' '-trimpath' '-installsuffix=cgo' '-ldflags=-X=go.etcd.io/etcd/api/v3/version.GitSHA=f2f8c70e7' '-o=../bin/etcd' '.')
stderr: go.etcd.io/etcd/server/v3
% 'rm' '-f' 'bin/etcdutl'
% (cd etcdutl && 'env' 'GO_BUILD_FLAGS= -v -mod=readonly' 'CGO_ENABLED=0' 'GO_BUILD_FLAGS= -v -mod=readonly' 'GOOS=darwin' 'GOARCH=arm64' 'go' 'build' '-v' '-mod=readonly' '-trimpath' '-installsuffix=cgo' '-ldflags=-X=go.etcd.io/etcd/api/v3/version.GitSHA=f2f8c70e7' '-o=../bin/etcdutl' '.')
stderr: go.etcd.io/etcd/etcdutl/v3
% 'rm' '-f' 'bin/etcdctl'
% (cd etcdctl && 'env' 'GO_BUILD_FLAGS= -v -mod=readonly' 'CGO_ENABLED=0' 'GO_BUILD_FLAGS= -v -mod=readonly' 'GOOS=darwin' 'GOARCH=arm64' 'go' 'build' '-v' '-mod=readonly' '-trimpath' '-installsuffix=cgo' '-ldflags=-X=go.etcd.io/etcd/api/v3/version.GitSHA=f2f8c70e7' '-o=../bin/etcdctl' '.')
stderr: go.etcd.io/etcd/etcdctl/v3
SUCCESS: etcd_build (GOARCH=arm64)
benchmark tool already installed...
Running benchmark: lease-keepalive
./scripts/benchmark_test.sh "lease-keepalive:"
Setup: Starting the etcd server...
Setup: etcd log file path set to ./_artifacts/etcd-lease-keepalive-20251007-213438.log. DATA_DIR set to /tmp/etcd/data-QJw5qG
Setup: Waiting for etcd to be healthy... (retry: 1/10)
Setup: Waiting for etcd to be healthy... (retry: 2/10)
Setup: etcd is healthy and running on pid 86335
.....
.....
  99.9% in 0.0005 secs.
Cleanup: Stopping etcd server - PID 86999
Cleanup: Deleted the DATA_DIR contents from /tmp/etcd/data-Mle18B related to benchmark test

k8s-ci-robot · 2025-10-07T16:54:22Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kishen-v
Once this PR has been reviewed and has the lgtm label, please assign spzala for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-10-07T16:54:26Z

Hi @kishen-v. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: Kishen Viswanathan <[email protected]>

kishen-v · 2025-10-07T16:56:44Z

pkg/report/perfdash.go

 	}

-	fileName := fmt.Sprintf("EtcdAPI_benchmark_%s.json", time.Now().UTC().Format(time.RFC3339))
+	fileName := fmt.Sprintf("EtcdAPI_benchmark_%s_%s.json", benchmarkOp, time.Now().UTC().Format(time.RFC3339))


This change should not affect the perfdash parser's config, and is done to help to find relevant files of data tied to a particular benchmark operation easily.

kishen-v · 2025-10-07T16:59:15Z

scripts/benchmark_test.sh

+  DATA_DIR=$(mktemp -d /tmp/etcd/data-XXXXXX)
+  ARTIFACTS_DIR="${ARTIFACTS:-./_artifacts}"
+  mkdir -p "$ARTIFACTS_DIR"
+  ETCD_LOG_FILE="${ARTIFACTS_DIR}/etcd-${bench}-$(date +%Y%m%d-%H%M%S).log"


Thought: Should it be good to have the logs uploaded to the GCS bucket along with the perfash report?

kishen-v · 2025-10-07T16:59:44Z

/cc @serathius

serathius · 2025-10-14T12:12:00Z

Makefile

 	./scripts/update_go_workspace.sh
+
+.PHONY: help
+help:


Please remove everything unrelated to the PR.

serathius · 2025-10-14T12:13:17Z

Makefile

-bench-put: build install-benchmark
-	@echo "Running benchmark: put $(ARGS)"
-	./scripts/benchmark_test.sh put $(ARGS)
+.PHONY: bench-lease-keepalive bench-put bench-txn-mixed bench-watch-latency bench-watch bench-range-key


I don't think we are ready yet to run all those benchmarks. Please focus on one.

serathius · 2025-10-14T12:13:43Z

scripts/benchmark_test.sh

 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and


This rewrite is too big and unreviewable.

k8s-ci-robot added needs-ok-to-test size/L labels Oct 7, 2025

Add make targets for benchmark tooling

cc29fc7

Signed-off-by: Kishen Viswanathan <[email protected]>

kishen-v force-pushed the bench-other-targets branch from f2f8c70 to cc29fc7 Compare October 7, 2025 16:54

kishen-v commented Oct 7, 2025

View reviewed changes

k8s-ci-robot requested a review from serathius October 7, 2025 16:59

serathius reviewed Oct 14, 2025

View reviewed changes

Makefile

./scripts/update_go_workspace.sh

.PHONY: help

help:

Copy link

Member

serathius Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove everything unrelated to the PR.

serathius reviewed Oct 14, 2025

View reviewed changes

kishen-v changed the title ~~Add make targets for benchmark operation tooling~~ WIP: Add make targets for benchmark operation tooling Oct 21, 2025

k8s-ci-robot added the do-not-merge/work-in-progress label Oct 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP: Add make targets for benchmark operation tooling #20769

WIP: Add make targets for benchmark operation tooling #20769

kishen-v commented Oct 7, 2025

Uh oh!

k8s-ci-robot commented Oct 7, 2025

Uh oh!

k8s-ci-robot commented Oct 7, 2025

Uh oh!

kishen-v Oct 7, 2025

Uh oh!

kishen-v Oct 7, 2025

Uh oh!

kishen-v commented Oct 7, 2025

Uh oh!

serathius Oct 14, 2025

Uh oh!

serathius Oct 14, 2025

Uh oh!

serathius Oct 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

WIP: Add make targets for benchmark operation tooling #20769

Are you sure you want to change the base?

WIP: Add make targets for benchmark operation tooling #20769

Conversation

kishen-v commented Oct 7, 2025

Uh oh!

k8s-ci-robot commented Oct 7, 2025

Uh oh!

k8s-ci-robot commented Oct 7, 2025

Uh oh!

kishen-v Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

kishen-v Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

kishen-v commented Oct 7, 2025

Uh oh!

serathius Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

serathius Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

serathius Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants