feat(base): enhanced query-level memory management #17358

zhang2014 · 2025-01-23T08:51:50Z

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

feat(base): enhanced query-level memory management

introduced query-level memory tracking
added max_query_memory_usage setting to set per-query memory limits
- queries exceeding the specified limit will automatically terminate with OOM (Out-Of-Memory) errors

Tests

Unit Test
Logic Test
Benchmark Test
No Test - don't merged.

Type of change

Bug Fix (non-breaking change which fixes an issue)
New Feature (non-breaking change which adds functionality)
Breaking Change (fix or feature that could cause existing functionality not to work as expected)
Documentation Update
Refactoring
Performance Improvement
Other (please describe): only test

This change is

github-actions · 2025-01-23T09:22:25Z

Docker Image for PR

tag: pr-17358-d59cb28-1737624078

note: this image tag is only available for internal use,
please check the internal doc for more details.

github-actions · 2025-01-23T10:03:37Z

ClickBench Report

github-actions · 2025-01-23T14:03:27Z

Docker Image for PR

tag: pr-17358-b5b9ad5-1737640942

note: this image tag is only available for internal use,
please check the internal doc for more details.

github-actions · 2025-01-24T02:34:07Z

Docker Image for PR

tag: pr-17358-3bace8e-1737685980

note: this image tag is only available for internal use,
please check the internal doc for more details.

github-actions · 2025-02-14T18:54:08Z

Docker Image for PR

tag: pr-17358-be6808c-1739559177

note: this image tag is only available for internal use,
please check the internal doc for more details.

github-actions · 2025-02-25T02:46:35Z

Docker Image for PR

tag: pr-17358-b34987a-1740451468

note: this image tag is only available for internal use,
please check the internal doc for more details.

zhang2014 · 2025-02-25T03:15:51Z

valgrind results for 100k memory chaos tests(passed)

valgrind --tool=memcheck --leak-check=full --track-origins=yes /home/ubuntu/datafuse/target/debug/deps/databend_common_base-8212ca84429fbc04 'mem_allocator::tracker::tests::test_mixed_failure_scenarios'
==148789== Memcheck, a memory error detector
==148789== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==148789== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==148789== Command: /home/ubuntu/datafuse/target/debug/deps/databend_common_base-8212ca84429fbc04 mem_allocator::tracker::tests::test_mixed_failure_scenarios
==148789== 

running 1 test
test mem_allocator::tracker::tests::test_mixed_failure_scenarios has been running for over 60 seconds
test mem_allocator::tracker::tests::test_mixed_failure_scenarios ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 44 filtered out; finished in 322.73s

==148789== 
==148789== HEAP SUMMARY:
==148789==     in use at exit: 608 bytes in 2 blocks
==148789==   total heap usage: 675 allocs, 673 frees, 8,475,219 bytes allocated
==148789== 
==148789== 304 bytes in 1 blocks are possibly lost in loss record 1 of 2
==148789==    at 0x488C0AC: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==148789==    by 0x400F553: calloc (rtld-malloc.h:44)
==148789==    by 0x400F553: allocate_dtv (dl-tls.c:370)
==148789==    by 0x400FF5F: _dl_allocate_tls (dl-tls.c:629)
==148789==    by 0x4A76387: allocate_stack (allocatestack.c:429)
==148789==    by 0x4A76387: pthread_create@@GLIBC_2.34 (pthread_create.c:655)
==148789==    by 0x312ECB: _rjem_je_pthread_create_wrapper (background_thread.c:47)
==148789==    by 0x313E43: background_thread_create_signals_masked (background_thread.c:287)
==148789==    by 0x31574F: background_thread_create_locked (background_thread.c:495)
==148789==    by 0x31589F: _rjem_je_background_thread_create (background_thread.c:520)
==148789==    by 0x27DC6B: malloc_init_hard (jemalloc.c:2176)
==148789==    by 0x2ED59B: malloc_init (jemalloc.c:298)
==148789==    by 0x2ED59B: jemalloc_constructor (jemalloc.c:4322)
==148789==    by 0x4A18613: call_init (libc-start.c:145)
==148789==    by 0x4A18613: __libc_start_main@@GLIBC_2.34 (libc-start.c:347)
==148789==    by 0x13B52F: imalloc (tsd.h:344)
==148789==    by 0x13B52F: imalloc (jemalloc.c:3653)
==148789==    by 0x13B52F: (below main) (jemalloc.c:3242)
==148789== 
==148789== 304 bytes in 1 blocks are possibly lost in loss record 2 of 2
==148789==    at 0x488C0AC: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==148789==    by 0x400F553: calloc (rtld-malloc.h:44)
==148789==    by 0x400F553: allocate_dtv (dl-tls.c:370)
==148789==    by 0x400FF5F: _dl_allocate_tls (dl-tls.c:629)
==148789==    by 0x4A76387: allocate_stack (allocatestack.c:429)
==148789==    by 0x4A76387: pthread_create@@GLIBC_2.34 (pthread_create.c:655)
==148789==    by 0x312ECB: _rjem_je_pthread_create_wrapper (background_thread.c:47)
==148789==    by 0x313E43: background_thread_create_signals_masked (background_thread.c:287)
==148789==    by 0x314027: check_background_thread_creation (background_thread.c:332)
==148789==    by 0x314253: background_thread0_work (background_thread.c:370)
==148789==    by 0x3147DB: background_work (background_thread.c:412)
==148789==    by 0x3152DF: background_thread_entry (background_thread.c:444)
==148789==    by 0x4A7595B: start_thread (pthread_create.c:447)
==148789==    by 0x4ADB7DB: thread_start (clone.S:79)
==148789== 
==148789== LEAK SUMMARY:
==148789==    definitely lost: 0 bytes in 0 blocks
==148789==    indirectly lost: 0 bytes in 0 blocks
==148789==      possibly lost: 608 bytes in 2 blocks
==148789==    still reachable: 0 bytes in 0 blocks
==148789==         suppressed: 0 bytes in 0 blocks
==148789== 
==148789== For lists of detected and suppressed errors, rerun with: -s
==148789== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

github-actions · 2025-02-25T03:24:51Z

ClickBench Report

github-actions · 2025-02-28T03:15:20Z

Docker Image for PR

tag: pr-17358-b863ac1-1740712397

note: this image tag is only available for internal use,
please check the internal doc for more details.

github-actions · 2025-02-28T03:54:21Z

ClickBench Report

src/common/base/src/runtime/runtime_tracker.rs

github-actions · 2025-03-01T13:47:23Z

Docker Image for PR

tag: pr-17358-1203db8-1740836773

note: this image tag is only available for internal use,
please check the internal doc for more details.

github-actions · 2025-03-01T14:23:00Z

ClickBench Report

chore(base): test memory alloc bucket

eb1e9f4

zhang2014 added the ci-benchmark Benchmark: run all test label Jan 23, 2025

github-actions bot added the pr-chore this PR only has small changes that no need to record, like coding styles. label Jan 23, 2025

chore(base): embed tracker into large memory

f6f8c42

zhang2014 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Jan 23, 2025

zhang2014 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Jan 24, 2025

zhang2014 added 15 commits February 6, 2025 20:04

chore(base): embed tracker into large memory.

7661aec

chore(base): embed tracker into large memory.

a9e7f61

chore(base): embed tracker into large memory.

6eca024

chore(base): embed tracker into large memory.

df65f83

chore(base): embed tracker into large memory.

d859c6e

chore(base): embed tracker into large memory.

95946ae

chore(base): embed tracker into large memory.

1c553ef

chore(base): embed tracker into large memory.

30051bf

chore(base): embed tracker into large memory.

9f39538

chore(base): embed tracker into large memory.

477ac52

chore(base): embed tracker into large memory.

cf6a6c2

chore(base): embed tracker into large memory.

6106179

chore(base): embed tracker into large memory.

e73aed6

chore(base): embed tracker into large memory.

872053c

Merge branch 'main' into chore/test_alloc_bucket

ba753a3

zhang2014 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Feb 14, 2025

zhang2014 added the ci-benchmark Benchmark: run all test label Feb 25, 2025

zhang2014 added 5 commits February 25, 2025 17:11

chore(base): embed tracker into large memory.

21363f9

chore(base): embed tracker into large memory.

36166ec

chore(base): embed tracker into large memory.

dc9a6cd

Merge branch 'main' into chore/test_alloc_bucket

4e252c8

Merge branch 'main' into chore/test_alloc_bucket

08885a6

zhang2014 changed the title ~~feat(base): support query level memory tracking~~ feat(base): enhanced Query-Level Memory Management Feb 28, 2025

zhang2014 changed the title ~~feat(base): enhanced Query-Level Memory Management~~ feat(base): enhanced query-level memory management Feb 28, 2025

zhang2014 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Feb 28, 2025

chore(base): embed tracker into large memory.

540f7ff

zhang2014 added 2 commits February 28, 2025 12:04

chore(base): embed tracker into large memory.

0ded685

Merge branch 'main' into chore/test_alloc_bucket

1339d5f

zhang2014 marked this pull request as ready for review March 1, 2025 13:05

zhang2014 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Mar 1, 2025

zhang2014 requested review from dqhl76 and sundy-li March 1, 2025 13:11

sundy-li approved these changes Mar 1, 2025

View reviewed changes

dqhl76 approved these changes Mar 1, 2025

View reviewed changes

dqhl76 reviewed Mar 1, 2025

View reviewed changes

src/common/base/src/runtime/runtime_tracker.rs Outdated Show resolved Hide resolved

chore(base): embed tracker into large memory.

e8e3380

zhang2014 merged commit 1ad2acb into databendlabs:main Mar 1, 2025
76 checks passed

feat(base): enhanced query-level memory management #17358

feat(base): enhanced query-level memory management #17358

Uh oh!

Conversation

zhang2014 commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

Type of change

Uh oh!

github-actions bot commented Jan 23, 2025

Docker Image for PR

Uh oh!

github-actions bot commented Jan 23, 2025

ClickBench Report

Uh oh!

github-actions bot commented Jan 23, 2025

Docker Image for PR

Uh oh!

github-actions bot commented Jan 24, 2025

Docker Image for PR

Uh oh!

github-actions bot commented Feb 14, 2025

Docker Image for PR

Uh oh!

github-actions bot commented Feb 25, 2025

Docker Image for PR

Uh oh!

zhang2014 commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 25, 2025

ClickBench Report

Uh oh!

github-actions bot commented Feb 28, 2025

Docker Image for PR

Uh oh!

github-actions bot commented Feb 28, 2025

ClickBench Report

Uh oh!

Uh oh!

github-actions bot commented Mar 1, 2025

Docker Image for PR

Uh oh!

github-actions bot commented Mar 1, 2025

ClickBench Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zhang2014 commented Jan 23, 2025 •

edited

Loading

zhang2014 commented Feb 25, 2025 •

edited

Loading