Align metering with namespace quotas #2007
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yu-shipit
reviewed
May 20, 2025
| val tags = Map( | ||
| "metric_ws" -> ws, | ||
| "metric_ns" -> ns, | ||
| "dataset" -> dsRef.dataset, |
Contributor
There was a problem hiding this comment.
when dsRef.dataset is null. Can you use "unknown" or "null" as the value.
|
|
||
| // Update query bytes scanned per minute based on active timeseries and configured max data per shard | ||
| val maxDataPerShardQuery = settings.config.getBytes("max-data-per-shard-query").longValue() | ||
| val avgBytesPerTs = maxDataPerShardQuery / 1000000 // Convert to MB for better readability |
Contributor
There was a problem hiding this comment.
not sure about this. The name is bytes but the value is MB?
alextheimer
reviewed
May 28, 2025
Contributor
alextheimer
left a comment
alextheimer
left a comment
There was a problem hiding this comment.
Left some comments; will discuss more offline, as well
| private val METRIC_LONGTERM = "tsdb_metering_longterm_timeseries" | ||
| // Quota-aligned metrics | ||
| private val METRIC_SAMPLES_INGESTED = "tsdb_metering_samples_ingested_per_min" | ||
| private val METRIC_QUERY_BYTES_SCANNED = "tsdb_metering_query_samples_scanned_per_min" |
Contributor
There was a problem hiding this comment.
Should this constant be named SAMPLES_SCANNED?
| // Quota-aligned metrics | ||
| private val METRIC_SAMPLES_INGESTED = "tsdb_metering_samples_ingested_per_min" | ||
| private val METRIC_QUERY_BYTES_SCANNED = "tsdb_metering_query_samples_scanned_per_min" | ||
| private val METRIC_RETAINED_TIMESERIES = "tsdb_metering_retained_timeseries" |
Contributor
There was a problem hiding this comment.
What does "retained" mean?
Comment on lines
+90
to
+93
| // Update retained timeseries count - directly maps to active timeseries | ||
| Kamon.gauge(METRIC_RETAINED_TIMESERIES) | ||
| .withTags(TagSet.from(tags)) | ||
| .update(data.counts.active.toDouble) |
Contributor
There was a problem hiding this comment.
Isn't this the same thing as METRIC_ACTIVE? Why are both needed?
Contributor
There was a problem hiding this comment.
I think these names are from the design doc. Need Alex Goodwin to confirm.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Current behavior : (link exiting issues here : https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests)
New behavior :
This PR enhances monitoring by adding quota-aligned metrics to track ingestion, query usage, and retained data:
New Metrics
tsdb_metering_samples_ingested_per_min: Estimated samples/min ingested (based on active timeseries +ingest-resolution-millis).tsdb_metering_query_samples_scanned_per_min: Estimated bytes/min scanned during queries (using active timeseries +max-data-per-shard-query).tsdb_metering_retained_timeseries: Active timeseries count (mirrors retained data).Configuration
Added
ingest-resolution-millisandmax-data-per-shard-queryto compute metrics.Improvements
_ws_and_ns_tags for better filtering.