Skip to content

Conversation

@sundy-li
Copy link
Member

@sundy-li sundy-li commented Dec 19, 2024

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

feat(query): refactor vacuum temp files

  • VacuumTempOptions::QueryHook only vacuum temp files after the query finish

    1. It will vacuum files by hook limit
    2. If there are files remained, store ${query_id}_${index}.list` mark that there are remaining files to be cleaned in the future.
  • VacuumTempOptions::VacuumCommand is called via command VACUUM TEMPORARY FILES
    It will list the spill directory

    1. If it scans the meta file, it will clean the data by meta
    2. If it scans the directory, it will get the meta file firstly. If found then delete by meta else delete by list
    3. If it scans other files, delete by iterator.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Dec 19, 2024
@sundy-li sundy-li marked this pull request as ready for review December 20, 2024 15:20
@BohuTANG BohuTANG merged commit 0e12f28 into databendlabs:main Dec 23, 2024
72 of 73 checks passed
@sundy-li sundy-li added the ci-benchmark Benchmark: run all test label Dec 26, 2024
BohuTANG added a commit that referenced this pull request Dec 26, 2024
BohuTANG added a commit that referenced this pull request Dec 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-benchmark Benchmark: run all test pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants