The memory of the node0 abnormally increases periodically.

[tangyouzzz]

What type of bug is this?

Performance issue

What subsystems are affected?

Datanode

Minimal reproduce step

Run it normally for a period of time

What did you expect to see?

Memory usage is normal.

What did you see instead?

Memory surges around 0:00 every day

What operating system did you use?

linux

What version of GreptimeDB did you use?

0.17.2

Relevant log output and stack trace

# datanode config
[storage]
cache_capacity = "25G"

[[region_engine]]
[region_engine.mito]
write_cache_size = "25G"
sst_meta_cache_size = "5GB"
vector_cache_size = "2GB"
page_cache_size = "10GB"
selector_result_cache_size = "2GB"
max_concurrent_scan_files = 1000

[region_engine.mito.index]
content_cache_size = "3GB"

# vm
32C64G

[evenyag]

Could you help collect a heap profile file when the memory usage is high?
https://github.com/GreptimeTeam/greptimedb/blob/main/docs/how-to/how-to-profile-memory.md#dump-memory-profiling-data

If you are using the official docker image, it should already enabled the heap profiling feature.

[tangyouzzz]

This function was turned off before. I restarted the datanode three days ago and enabled it again. It needs to be updated for a few more days to reproduce.

This is the heap profile file before it was closed on October 31st

This is the heap profile file after the restart is enabled

[tangyouzzz]

20251111

[evenyag]

Thanks a lot! Could you please upload the svg file?

We will also try to reproduce the issue. By the way, could you try reducing the index cache size to a relative small value?

[region_engine.mito.index]
content_cache_size = "32MB"

I suspect the issue is related to the cache.

[tangyouzzz]

Thanks a lot! Could you please upload the svg file?

We will also try to reproduce the issue. By the way, could you try reducing the index cache size to a relative small value?

[region_engine.mito.index]
content_cache_size = "32MB"

I suspect the issue is related to the cache.

The file I uploaded is an svg file.

I will reduce the index cache size to a relatively small value and then observe for a period of time.

[evenyag]

The file I uploaded is an svg file.

Got it. I have to download it and rename it to a svg file.

You may also need to reduce the metadata_cache_size.

[region_engine.mito.index]
content_cache_size = "32MB"
metadata_cache_size = "64KB"

[tangyouzzz]

The file I uploaded is an svg file.

Got it. I have to download it and rename it to a svg file.

You may also need to reduce the metadata_cache_size.

[region_engine.mito.index]
content_cache_size = "32MB"
metadata_cache_size = "64KB"

All have decreased. I will keep observing and save the heap profile file

[tangyouzzz]

It seems there will still be problems.

2025-11-17

2025-11-14

[tangyouzzz]

I have a question. When no one uses it during the weekend holiday, the memory price keeps rising. Is it normal that the memory drops after I start using the monitoring system at work today, Monday?

[evenyag]

I have a question. When no one uses it during the weekend holiday, the memory price keeps rising. Is it normal that the memory drops after I start using the monitoring system at work today, Monday?

Based on the flamegraph, it is likely related to the cache issue we reproduced. I guess the memory drop may be caused by cache eviction. If we're lucky, #7229 might fixed this.

This nightly version contains the fix and you could have a try:
https://github.com/GreptimeTeam/greptimedb/releases/tag/v1.0.0-beta.1-nightly-20251117

You can observe the peak memory usage and check if it reduces. By the way, you can reduce the page cache size if you want a lower memory usage.

[tangyouzzz]

This is the memory usage after updating to version v1.0.0-beta.1-nightly-20251117. Although it will suddenly increase at 0:00, it will decrease around 9:00

2025-11-21

[evenyag]

Looks like the memory usage will increase a lot during 00:00 ~ 08:00. The peak usage may be caused by background compaction tasks and might be expected.

Do you have logs during that time? You can join our slack to send me the logs or email the logs to yingwen@greptime.com. We can check the compaction logs and maybe adjusting the compaction options can reduce the peak usage.

What's more, we can collect the profile after 00:00 to help confirm this. You can setup a crontab job like this to collect the hprof and svg during 00:00 ~ 08:00 every hour. Then you can archive them and send in this thread.

0 0-8 * * * curl -X POST localhost:4000/debug/prof/mem > "/path/to/greptime_$(date +\%Y-\%m-\%d_\%H_\%M).hprof"
0 0-8 * * * curl -X POST "localhost:4000/debug/prof/mem?output=flamegraph" > "/path/to/greptime_$(date +\%Y-\%m-\%d_\%H_\%M).svg"