Tracking cardinality count of write buffer

[sunng87]

What type of enhancement is this?

Tech debt reduction, Performance

What does the enhancement do?

We have seen some cases when user's data ingestion was blocked due to high-cardinality data. Due to lack of direct stats or metrics that tracks the cardinality, we need to manually check logs to understand the cause. And we also don't have an exact number of capacity of cardinality, if the write buffer performance is critical related to cardinality.

To address this, ideally there will be several steps:

1. Improve observability of cardinality: export metrics for human to monitor and understand it.
2. Keeping cardinality as internal statistics
3. Define some policy on cardinality count, for example, victoriametrics allows you to config a cardinality limit to drop additional time-series. This works really good in real world because even if cardinality goes high, it still keeps the system work and previous data ingested.
4. Some adaptive policy based on cardinality like auto switching memtable

Implementation challenges

No response

[evenyag]

We already have this in the memtable stats.

greptimedb/src/mito2/src/memtable.rs

Lines 147 to 148 in c2ff563

	/// Number of estimated timeseries in memtable.
	series_count: usize,

[sunng87]

Looks good, we should expose this as a metrics and display it on grafana dashboard. This will show how ingestion is related to series/cardinality count