fix: full compactions not scheduled under some circumstances (#26668)

Fix issue introduced with 1.12.1 that caused some TSM files to not be
scheduled for full compaction when they should be. The user visible
symptom was an unbounded increase in disk space for impacted shards.

This simplifies the compaction planning code by removing the
PT_SmartOptimize and PT_NoOptimize special cases. These are no longer
needed for performance thanks to the first block caching code that was
also added in #26432. Issues with the PT_SmartOptimize and PT_NoOptimize
cases could cause TSM files to be locked by the compaction planner as
in-use, preventing them from being compacted.

The optimized compaction hold-off is now handled entirely in
Engine.compact rather than by the compaction planner.

Update test cases as necessary for changes, plus add checks for issue
that caused #26667.

Closes: #26667

Author: gwossum

tsdb/engine/tsm1/compact.go

@@ -781,6 +781,14 @@ func (c *DefaultPlanner) Release(groups []CompactionGroup) {
 	}
 }
 
+// InUseCount returns the number of files currently locked as in-use.
+// This method is primarily useful for test validation.
+func (c *DefaultPlanner) InUseCount() int {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	return len(c.filesInUse)
+}
+
 // Compactor merges multiple TSM files into new files or
 // writes a Cache into 1 or more TSM files.
 type Compactor struct {