Test test_change_region_compaction_window may fail occasionally #7135
Description
What type of bug is this?
Other
What subsystems are affected?
Other
Minimal reproduce step
Run make test
What did you expect to see?
All tests should pass
What did you see instead?
Test test_change_region_compaction_window failed.
The reason may be the checkpointer was updating the checkpoint in the background while we were reopening the region.
Need more logs to debug this.
What operating system did you use?
macOS
What version of GreptimeDB did you use?
0.18
Relevant log output and stack trace
2025-10-23T07:45:40.264206Z INFO mito2::worker::handle_compaction: Successfully scheduled compaction task for region: 4294967297(1, 1)
2025-10-23T07:45:40.284088Z INFO mito2::manifest::checkpointer: Start doing checkpoint for region 4294967297(1, 1), compacted version: [0, 10]
2025-10-23T07:45:40.284788Z INFO mito2::request: Successfully compacted region: 4294967297(1, 1)
2025-10-23T07:45:40.284866Z DEBUG mito2::compaction: Pick compaction strategy TwcsPicker { trigger_file_num: 4, time_window_seconds: Some(7200), max_output_file_size: Some(536870912), append_mode: false } for region: 4294967297(1, 1), ttl: Forever
2025-10-23T07:45:40.285239Z INFO mito2::worker: Stop region worker group
2025-10-23T07:45:40.289716Z INFO mito2::sst::file: Deleted 1 files for region 4294967297(1, 1): [FileId(ac317aba-309d-4000-bb0e-707296bbbe41)]
2025-10-23T07:45:40.289766Z INFO mito2::sst::file: Deleted 1 files for region 4294967297(1, 1): [FileId(4fa6fe25-2052-4173-8ed1-81d9e6e6a3ab)]
2025-10-23T07:45:40.289807Z INFO mito2::sst::file: Deleted 1 files for region 4294967297(1, 1): [FileId(66374161-3ad4-4ffa-ab63-75f33c7fec21)]
2025-10-23T07:45:40.289823Z INFO mito2::sst::file: Deleted 1 files for region 4294967297(1, 1): [FileId(25a26350-f63f-4bc2-b403-57f1c74fcc7c)]
2025-10-23T07:45:40.290151Z INFO mito2::worker: Stop region worker 0
2025-10-23T07:45:40.290180Z INFO mito2::worker: Stop region worker 1
2025-10-23T07:45:40.290197Z INFO mito2::worker: Stop region worker 2
2025-10-23T07:45:40.290208Z INFO mito2::worker: Stop region worker 3
2025-10-23T07:45:40.290216Z INFO mito2::worker: Stop region worker 4
2025-10-23T07:45:40.290230Z INFO mito2::worker: Exit region worker thread 1
2025-10-23T07:45:40.290226Z INFO mito2::worker: Exit region worker thread 0
2025-10-23T07:45:40.290235Z INFO mito2::region: Stopped region manifest manager, region_id: 4294967297(1, 1)
2025-10-23T07:45:40.290244Z INFO mito2::worker: Exit region worker thread 2
2025-10-23T07:45:40.290245Z INFO mito2::worker: Exit region worker thread 3
2025-10-23T07:45:40.290255Z INFO mito2::worker: Exit region worker thread 4
2025-10-23T07:45:40.290453Z INFO puffin::puffin_manager::stager::bounded_stager: Recovering the staging area, base_dir: "/var/folders/0g/nq9zjrj92rx8h465llxn9qch0000gn/T/jmuLEC/index_intermediate/staging"
2025-10-23T07:45:40.290646Z INFO puffin::puffin_manager::stager::bounded_stager: Recovered the staging area, num_entries: 0, num_bytes: 0, cost: 179.167µs
2025-10-23T07:45:40.290667Z INFO mito2::sst::index::intermediate: Initializing intermediate manager, aux_path: /var/folders/0g/nq9zjrj92rx8h465llxn9qch0000gn/T/jmuLEC/index_intermediate
2025-10-23T07:45:40.298945Z DEBUG mito2::manifest::storage: Save checkpoint in path: test/1_0000000001/manifest/_last_checkpoint, metadata: CheckpointMetadata { size: 2469, version: 10, checksum: Some(161525754), extend_metadata: {} }
2025-10-23T07:45:40.299502Z DEBUG mito2::cache::index: Building IndexCache with metadata size: 67108864, content size: 134217728, page size: 65536, index type: inverted_index
2025-10-23T07:45:40.299603Z DEBUG mito2::cache::index: Building IndexCache with metadata size: 67108864, content size: 134217728, page size: 65536, index type: bloom_filter_index
2025-10-23T07:45:40.299765Z DEBUG puffin::puffin_manager::cache: Building PuffinMetadataCache with capacity: 67108864
2025-10-23T07:45:40.299888Z INFO mito2::worker: Start region worker thread 0, init_check_delay: 27s
2025-10-23T07:45:40.299914Z INFO mito2::worker: Start region worker thread 2, init_check_delay: 166s
2025-10-23T07:45:40.299917Z INFO mito2::worker: Start region worker thread 3, init_check_delay: 148s
2025-10-23T07:45:40.299940Z INFO mito2::worker: Start region worker thread 1, init_check_delay: 138s
2025-10-23T07:45:40.299946Z INFO mito2::worker: Start region worker thread 4, init_check_delay: 102s
2025-10-23T07:45:40.300227Z INFO mito2::worker::handle_open: Try to open region 4294967297(1, 1), worker: 2
2025-10-23T07:45:40.322046Z DEBUG mito2::manifest::storage: Deleting 10 logs from manifest storage path test/1_0000000001/manifest/ until 10, checkpoint_version: None, paths: ["test/1_0000000001/manifest/00000000000000000007.json", "test/1_0000000001/manifest/00000000000000000006.json", "test/1_0000000001/manifest/00000000000000000001.json", "test/1_0000000001/manifest/00000000000000000000.json", "test/1_0000000001/manifest/00000000000000000003.json", "test/1_0000000001/manifest/00000000000000000002.json", "test/1_0000000001/manifest/00000000000000000009.json", "test/1_0000000001/manifest/00000000000000000005.json", "test/1_0000000001/manifest/00000000000000000004.json", "test/1_0000000001/manifest/00000000000000000008.json"]
2025-10-23T07:45:40.358402Z DEBUG mito2::manifest::storage: Unset file size: Delta(7), size: 516
2025-10-23T07:45:40.358427Z DEBUG mito2::manifest::storage: Unset file size: Delta(6), size: 517
2025-10-23T07:45:40.358433Z DEBUG mito2::manifest::storage: Unset file size: Delta(1), size: 507
2025-10-23T07:45:40.358437Z DEBUG mito2::manifest::storage: Unset file size: Delta(0), size: 782
2025-10-23T07:45:40.358441Z DEBUG mito2::manifest::storage: Unset file size: Delta(3), size: 516
2025-10-23T07:45:40.358446Z DEBUG mito2::manifest::storage: Unset file size: Delta(2), size: 515
2025-10-23T07:45:40.358450Z DEBUG mito2::manifest::storage: Unset file size: Delta(9), size: 516
2025-10-23T07:45:40.358454Z DEBUG mito2::manifest::storage: Unset file size: Delta(5), size: 1800
2025-10-23T07:45:40.358458Z DEBUG mito2::manifest::storage: Unset file size: Delta(4), size: 516
2025-10-23T07:45:40.358462Z DEBUG mito2::manifest::storage: Unset file size: Delta(8), size: 516
2025-10-23T07:45:40.358474Z INFO mito2::manifest::checkpointer: Checkpoint for region 4294967297(1, 1) success, version: 10
2025-10-23T07:45:40.859048Z INFO puffin::puffin_manager::stager::bounded_stager: The delete routine for the bounded stager is terminated.
2025-10-23T07:45:40.907656Z INFO puffin::puffin_manager::stager::bounded_stager: The delete routine for the bounded stager is terminated.
test engine::compaction_test::test_change_region_compaction_window ... FAILED
failures:
failures:
engine::compaction_test::test_change_region_compaction_window
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 609 filtered out; finished in 3.19s
--- STDERR: mito2 engine::compaction_test::test_change_region_compaction_window ---
thread 'engine::compaction_test::test_change_region_compaction_window' (41955463) panicked at src/mito2/src/engine/compaction_test.rs:886:10:
called `Result::unwrap()` on an `Err` value: 0: Failed to open region, at src/mito2/src/worker/handle_open.rs:147:54
1: Failed to ser/de json object, at src/mito2/src/manifest/storage.rs:737:36
2: Error("EOF while parsing a value", line: 1, column: 0)
stack backtrace:
0: __rustc::rust_begin_unwind
at /rustc/fa3155a644dd62e865825087b403646be01d4cef/library/std/src/panicking.rs:698:5
1: core::panicking::panic_fmt
at /rustc/fa3155a644dd62e865825087b403646be01d4cef/library/core/src/panicking.rs:78:14
2: core::result::unwrap_failed
at /rustc/fa3155a644dd62e865825087b403646be01d4cef/library/core/src/result.rs:1855:5
3: core::result::Result<T,E>::unwrap
at /Users/evenyag/.rustup/toolchains/nightly-2025-10-01-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:1226:23
4: mito2::engine::compaction_test::test_change_region_compaction_window_with_format::{{closure}}
at ./src/engine/compaction_test.rs:886:10
5: mito2::engine::compaction_test::test_change_region_compaction_window::{{closure}}
at ./src/engine/compaction_test.rs:757:61
6: <core::pin::Pin<P> as core::future::future::Future>::poll
at /Users/evenyag/.rustup/toolchains/nightly-2025-10-01-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:133:9
7: <core::pin::Pin<P> as core::future::future::Future>::poll
at /Users/evenyag/.rustup/toolchains/nightly-2025-10-01-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:133:9
8: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
9: tokio::task::coop::with_budget
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
10: tokio::task::coop::budget
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
11: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
12: tokio::runtime::scheduler::current_thread::Context::enter
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
14: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
15: tokio::runtime::context::scoped::Scoped<T>::set
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
16: tokio::runtime::context::set_scheduler::{{closure}}
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
17: std::thread::local::LocalKey<T>::try_with
at /Users/evenyag/.rustup/toolchains/nightly-2025-10-01-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/thread/local.rs:315:12
18: std::thread::local::LocalKey<T>::with
at /Users/evenyag/.rustup/toolchains/nightly-2025-10-01-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/thread/local.rs:279:20
19: tokio::runtime::context::set_scheduler
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
20: tokio::runtime::scheduler::current_thread::CoreGuard::enter
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
21: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
22: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
23: tokio::runtime::context::runtime::enter_runtime
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
24: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
25: tokio::runtime::runtime::Runtime::block_on_inner
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
26: tokio::runtime::runtime::Runtime::block_on
at /Users/evenyag/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
27: mito2::engine::compaction_test::test_change_region_compaction_window
at ./src/engine/compaction_test.rs:758:65
28: mito2::engine::compaction_test::test_change_region_compaction_window::{{closure}}
at ./src/engine/compaction_test.rs:756:48
29: core::ops::function::FnOnce::call_once
at /Users/evenyag/.rustup/toolchains/nightly-2025-10-01-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
30: core::ops::function::FnOnce::call_once
at /rustc/fa3155a644dd62e865825087b403646be01d4cef/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.