feat: Adds detection for orphaned TSM files while planning compaction #26766

devanbenz · 2025-08-30T01:20:22Z

We were seeing a state in which a shard would not perform full compactions leading to a build up of level 4 TSM files.

File state:

-rw-r--r--.  1 root root 2.1G Aug  5 20:59 000016684-000000007.tsm
-rw-r--r--.  1 root root 2.1G Aug  5 21:02 000016684-000000008.tsm
-rw-r--r--.  1 root root 2.1G Aug  5 21:04 000016684-000000009.tsm
-rw-r--r--.  1 root root 376M Aug  5 21:05 000016684-000000010.tsm
-rw-r--r--.  1 root root 2.1G Aug  5 18:00 000016812-000000004.tsm
-rw-r--r--.  1 root root 1.4G Aug  5 18:00 000016812-000000005.tsm
-rw-r--r--.  1 root root 1.3G Aug  5 21:21 000016844-000000002.tsm
-rw-r--r--.  1 root root 2.1G Aug  5 18:00 000016948-000000004.tsm
-rw-r--r--.  1 root root 1.4G Aug  5 18:00 000016948-000000005.tsm
-rw-r--r--.  1 root root 2.1G Aug  5 18:00 000017076-000000004.tsm

There is a rouge level 2 file packed within fully compacted files

-rw-r--r--.  1 root root 2.1G Aug  5 20:59 000016684-000000007.tsm
-rw-r--r--.  1 root root 2.1G Aug  5 21:02 000016684-000000008.tsm
-rw-r--r--.  1 root root 2.1G Aug  5 21:04 000016684-000000009.tsm
-rw-r--r--.  1 root root 376M Aug  5 21:05 000016684-000000010.tsm

and level 4 files

-rw-r--r--.  1 root root 2.1G Aug  5 18:00 000016948-000000004.tsm
-rw-r--r--.  1 root root 1.4G Aug  5 18:00 000016948-000000005.tsm
-rw-r--r--.  1 root root 2.1G Aug  5 18:00 000017076-000000004.tsm

The area of our code that would cause this state to be skipped would be here

influxdb/tsdb/engine/tsm1/compact.go

Lines 620 to 670 in 22bec4f

    
           // step is how may files to compact in a group.  We want to clamp it at 4 but also stil 
        
           // return groups smaller than 4. 
        
           step := 4 
        
           if step > end { 
        
           	step = end 
        
           } 
        
           // slice off the generations that we'll examine 
        
           generations = generations[start:end] 
        
           // Loop through the generations in groups of size step and see if we can compact all (or 
        
           // some of them as group) 
        
           groups := []tsmGenerations{} 
        
           for i := 0; i < len(generations); i += step { 
        
           	var skipGroup bool 
        
           	startIndex := i 
        
           	for j := i; j < i+step && j < len(generations); j++ { 
        
           		gen := generations[j] 
        
           		lvl := gen.level() 
        
           		// Skip compacting this group if there happens to be any lower level files in the 
        
           		// middle.  These will get picked up by the level compactors. 
        
           		if lvl <= 3 { 
        
           			skipGroup = true 
        
           			break 
        
           		} 
        
           		// Skip the file if it's over the max size and it contains a full block 
        
           		if gen.size() >= uint64(tsdb.MaxTSMFileSize) && gen.files[0].FirstBlockCount >= tsdb.DefaultMaxPointsPerBlock && !gen.hasTombstones() { 
        
           			startIndex++ 
        
           			continue 
        
           		} 
        
           	} 
        
           	if skipGroup { 
        
           		continue 
        
           	} 
        
           	endIndex := i + step 
        
           	if endIndex > len(generations) { 
        
           		endIndex = len(generations) 
        
           	} 
        
           	if endIndex-startIndex > 0 { 
        
           		groups = append(groups, generations[startIndex:endIndex]) 
        
           	} 
        
           } 
        
           if len(groups) == 0 { 
        
           	return nil, 0 
        
           }

We need to add some sort of escape mechanism that would allow for compactions to occur or simplify this logic.

This PR adds a test for the bugged file listing where we experience the issue. I'm still trying to find the way files get in to this state, but, this should resolve the issue if and when it occurs. I've added the check in PlanOptimize() as to not change any of the core logic in Plan().

Closes #

Describe your proposed changes here.

I've read the contributing section of the project README.
Signed CLA (if not already signed).

We were seeing a state in which a shard would not perform full compactions leading to a build up of level 4 TSM files. File state: ``` -rw-r--r--. 1 root root 2.1G Aug 5 20:59 000016684-000000007.tsm -rw-r--r--. 1 root root 2.1G Aug 5 21:02 000016684-000000008.tsm -rw-r--r--. 1 root root 2.1G Aug 5 21:04 000016684-000000009.tsm -rw-r--r--. 1 root root 376M Aug 5 21:05 000016684-000000010.tsm -rw-r--r--. 1 root root 2.1G Aug 5 18:00 000016812-000000004.tsm -rw-r--r--. 1 root root 1.4G Aug 5 18:00 000016812-000000005.tsm -rw-r--r--. 1 root root 1.3G Aug 5 21:21 000016844-000000002.tsm -rw-r--r--. 1 root root 2.1G Aug 5 18:00 000016948-000000004.tsm -rw-r--r--. 1 root root 1.4G Aug 5 18:00 000016948-000000005.tsm -rw-r--r--. 1 root root 2.1G Aug 5 18:00 000017076-000000004.tsm ``` There is a rouge level 2 file packed within fully compacted files ``` -rw-r--r--. 1 root root 2.1G Aug 5 20:59 000016684-000000007.tsm -rw-r--r--. 1 root root 2.1G Aug 5 21:02 000016684-000000008.tsm -rw-r--r--. 1 root root 2.1G Aug 5 21:04 000016684-000000009.tsm -rw-r--r--. 1 root root 376M Aug 5 21:05 000016684-000000010.tsm ``` and level 4 files ``` -rw-r--r--. 1 root root 2.1G Aug 5 18:00 000016948-000000004.tsm -rw-r--r--. 1 root root 1.4G Aug 5 18:00 000016948-000000005.tsm -rw-r--r--. 1 root root 2.1G Aug 5 18:00 000017076-000000004.tsm ``` The area of our code that would cause this state to be skipped would be here https://github.com/influxdata/influxdb/blob/22bec4f046a28e3f1fa815705362767151407e1b/tsdb/engine/tsm1/compact.go#L620-L670 We need to add some sort of escape mechanism that would allow for compactions to occur or simplify this logic. This PR adds a test for the bugged file listing where we experience the issue. I'm still trying to find the way files get in to this state, but, this should resolve the issue if and when it occurs. I've added the check in PlanOptimize() as to not change any of the core logic in Plan().

davidby-influx · 2025-08-30T01:37:41Z

The test fails without the second condition check? || cur.Len() < 2

devanbenz · 2025-08-30T01:50:22Z

The test fails without the second condition check? || cur.Len() < 2

Correct, I should have pushed up a commit with just the test but if you remove the check it fails.

davidby-influx · 2025-08-30T01:51:29Z

Great!

…

On Fri, Aug 29, 2025, 6:50 PM WeblWabl ***@***.***> wrote: *devanbenz* left a comment (influxdata/influxdb#26766) <#26766 (comment)> The test fails without the second condition check? || cur.Len() < 2 Correct, I should have pushed up a commit with just the test but if you remove the check it fails. — Reply to this email directly, view it on GitHub <#26766 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARIQHJAA3H5YUAJXYFTWWMT3QD7PJAVCNFSM6AAAAACFF6HDZOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMZYHA2DSNBXGM> . You are receiving this because you commented.Message ID: ***@***.***>

* 2 level 2 files * 3 level 2 files * 4 level 2 files * 5 level 2 files

…k at the files on disk a lot of these l4 files are not at the max points per block, just at the max file size of 2.1 GB.

…we don't have nested lower level files in our plans

devanbenz · 2025-09-02T21:23:20Z

tsdb/engine/tsm1/compact.go

 			// Skip compacting this group if there happens to be any lower level files in the
 			// middle.  These will get picked up by the level compactors.
-			if lvl <= 3 {
+			if lvl <= 3 && len(gen.files) > 4 {


I think that it would be better to ensure that the files are nested within high level files. For example, lets say we have the following levels listed

5 6 7 8 4 5 2 2 4 5 4 5

We can assume that the 2 level 2 files need to be compacted as a group with our high level files.

But if we have

5 6 7 8 4 5 2 2

since the 2 level 2 files are at the end of the entire list of files we should not compact them with our higher groups

I found the following test case:

data := []tsm1.FileStat{ { Path: "01-04.tsm1", Size: 64 * 1024 * 1024, }, { Path: "02-04.tsm1", Size: 64 * 1024 * 1024, }, { Path: "03-04.tsm1", Size: 64 * 1024 * 1024, }, { Path: "04-04.tsm1", Size: 64 * 1024 * 1024, }, { Path: "05-02.tsm1", Size: 2 * 1024 * 1024, }, }

Which is found here:

influxdb/tsdb/engine/tsm1/compact_test.go

Lines 2593 to 2635 in b647892

func TestDefaultPlanner_Plan_CompactsMiddleSteps(t *testing.T) {

data := []tsm1.FileStat{

{

Path: "01-04.tsm1",

Size: 64 * 1024 * 1024,

},

{

Path: "02-04.tsm1",

Size: 64 * 1024 * 1024,

},

{

Path: "03-04.tsm1",

Size: 64 * 1024 * 1024,

},

{

Path: "04-04.tsm1",

Size: 64 * 1024 * 1024,

},

{

Path: "05-02.tsm1",

Size: 2 * 1024 * 1024,

},

}

cp := tsm1.NewDefaultPlanner(

newFakeFileStore(withFileStats(data)),

tsdb.DefaultCompactFullWriteColdDuration,

)

expFiles := []tsm1.FileStat{data[0], data[1], data[2], data[3]}

tsm, pLen := cp.Plan(time.Now())

if exp, got := len(expFiles), len(tsm[0]); got != exp {

t.Fatalf("tsm file length mismatch: got %v, exp %v", got, exp)

} else if pLen != int64(len(tsm)) {

t.Fatalf("tsm file plan length mismatch: got %v, exp %v", pLen, exp)

}

for i, p := range expFiles {

if got, exp := tsm[0][i], p.Path; got != exp {

t.Fatalf("tsm file mismatch: got %v, exp %v", got, exp)

}

}

}

With this change does not pass. I would expect for the first four files to get compacted but the last one to be left out since its at the end of the file list.

Thoughts, @gwossum @davidby-influx?

Why not pick up the level 2 files at the end? They will get the benefits of the better compression from inclusion with higher level files. Are you concerned it will be extra work on a hot shard?

Retracting the above comment after discussion.

with higher level files if it is nested

davidby-influx · 2025-09-03T03:46:50Z

@devanbenz - I think we need to create a new test case file. In it, we should have many, many permutations of compaction levels and generations. Here are a few examples:

4,5,4,5,4

2,4,5,4,5,4

3,2,4,5,4,5,4

3,2,4,5,4,5,4,2

2,2,3,2,4,5,4,5,4,2

2,2,3,2,4,5,2,4,5,4,2

2,2,3,2,4,5,2,2,4,5,3,4,2

4,5,2,2,4,5,3,5,4,2,2,2

4,5,2,2,4,5,3,5,2,3,4,2,2,2,5,4,5,6

We want to test

leading low levels with runs from 1 to 9.
trailing low levels,
nested low levels (with varying run lengths from 1 to 9),
multiple nested low levels with varying run lengths
leading low levels and multiple nested low levels with varying run lengths
leading low levels and multiple nested low levels with varying run lengths and trailing low levels
etc.

Create compact_case_test.go, generate all the test cases (maybe statically with Claude, maybe programmatically at run time) with labels (don't forget to vary points per block and file size). This will be a large set of permutations, so test them as you go along with whatever algorithm choices you have made to the planner.

Claude generated the test cases and I modified the results as to what I would expect.

devanbenz · 2025-09-03T16:35:19Z

@devanbenz - I think we need to create a new test case file. In it, we should have many, many permutations of compaction levels and generations. Here are a few examples:
4,5,4,5,4
2,4,5,4,5,4
3,2,4,5,4,5,4
3,2,4,5,4,5,4,2
2,2,3,2,4,5,4,5,4,2
2,2,3,2,4,5,2,4,5,4,2
2,2,3,2,4,5,2,2,4,5,3,4,2
4,5,2,2,4,5,3,5,4,2,2,2
4,5,2,2,4,5,3,5,2,3,4,2,2,2,5,4,5,6
We want to test
* leading low levels with runs from 1 to 9.

* trailing low levels,

* nested low levels (with varying run lengths from 1 to 9),

* multiple nested low levels with varying run lengths

* leading low levels and multiple nested low levels with varying run lengths

* leading low levels and multiple nested low levels with varying run lengths and trailing low levels

* etc.
Create compact_case_test.go, generate all the test cases (maybe statically with Claude, maybe programmatically at run time) with labels (don't forget to vary points per block and file size). This will be a large set of permutations, so test them as you go along with whatever algorithm choices you have made to the planner.

I've found the static test file to work better this AM while working on this. I'm going to add the same tests duplicated but using different block counts to see how it impacts the planner + expected results.

tsdb/engine/tsm1/compact_test.go

tsdb/engine/tsm1/compact.go

philjb · 2025-09-05T22:06:27Z

tsdb/engine/tsm1/engine.go


 	var planner CompactionPlanner = NewDefaultPlanner(fs, time.Duration(opt.Config.CompactFullWriteColdDuration))
 	planner.SetAggressiveCompactionPointsPerBlock(int(opt.Config.AggressivePointsPerBlock))
+	planner.SetNestedCompactor(opt.Config.NestedCompactorEnabled)


perhaps log here if the config is set to true. I think that will be helpful confirmation. Sometimes we get logs but not a config file (although with Cloud1 we have direct access to both).

Absolutely log. Info level. Great suggestion.

philjb · 2025-09-05T22:24:21Z

tsdb/engine/tsm1/compact_case_static_test.go

+								"000016684-000000007.tsm",
+								"000016812-000000004.tsm",
+								"000016853-000000002.tsm",
+								"000016948-000000004.tsm",


I think this is wrong. This compaction group will merge these tsm files but that will produce incorrect overwrite semantics as the files in the group aren't adjacent.

The level 2 group looks right --- it pulled 4 level 2 files together into a group that are sequential. They will be compacted into a level 3 generation that will be correct for overwrites, producing a 000016844-3

But if the weighting of the scheduler picks this level4group to be run, it'll compact 684-7, 812-5, 853-2, 948-4 together and produce a 000016684-8 which will be older than the one from the level 2 group (16684 < 16844) and overwrites will be broken.

So you might need to check there isn't a run of non-fully compacted generations longer than 4 (which is what the level 2 & 3 compactions look for.

if i - lastHighLevelIndex > 3 { break }

philjb

I found an issue in one of the tests -- makes me wonder if other tests are incorrectly combining tsm files that aren't adjacent. I think each test case could have a check that given the input files/generations and the expected output, all generations in all groups are adjacent -- just to verify the test is accurate too.

…iles but it's less than 4 len AND we have multiple level < 4 files that can be compacted in full i.e. nested lower level files. We will compact them all together if that's the case. Please see our test compact_case_static_test.go/Mixed generations with 4 level 2 files

philjb · 2025-09-08T18:41:40Z

tsdb/engine/tsm1/compact.go

+		for i, cur := range groups {
+			if len(groups) > i+1 {
+				nextLevel := groups[i+1]
+				if cur.level() < nextLevel.level() {


I don't think this is handling the groups as we want.

concern these generations:
old 4, 4, 4, 2, 2, 2, 2, 1, 1, 1, 1, 4, 4, 4, 3, 3, 3, 2, 2, 1, 1, 1, 1, 1, 1 young
They will group into:
old G4, G2, G1, G4', G3, G2', G1 young
This conditional will "skip/continue" G1 but not G2 even though both G1 and G2 are nested within L4 generations.

I think you need to scan ahead. I don't think you can "only" look at the level of an immediately subsequent group.

Perhaps this code:

// Remove any groups in the wrong level var levelGroups []tsmGenerations if c.enableNestedCompactor { // search for runs of groups nested between level 4+ groups // e.g. G4, G4, G1, G4, G2 ... = G1 // e.g. G4, G4, G2, G4, G1, G4 ... = (G2), (G1) // e.g. G4, G4, G2, G1, G3, G1, G4, G1, G4 ... = (G2, G1, G3, G1), (G1) skip := make([]bool, len(groups)) lastG4 := 0 for i, cur := range groups { lvl := cur.level() if lvl >= 4 { if i > 0 && lastG4 != i - 1 { // we have returned to L4+ after seeing L3- // skip all the groups in between for j := lastG4+1; j < i; j++ { skip[j] = true } } lastG4 = i } if lvl != level { // normal (non-nested) behavior: group's level doesn't match the requested planning level skip[i] = true } } // process skip list for i, cur := range groups { if !skip[i] { levelGroups = append(levelGroups, cur) } } } else { for _, cur := range groups { if cur.level() == level { levelGroups = append(levelGroups, cur) } } }

The most important thing is that we don't change the ordering of input so this will stop that, as we check the index anyways 🤔

…-tsm-state-compact

This commit will add a check to ensure that we do not have non-adjacent files for our l4 and l5 planning. Since they can both plan and be comprised of similar files we need to ensure that all files for all groups are adjacent For example, lets say we have the following generations 1_2_3_4_5_6_7_8 our l4 plan picks up 3_4_5_6 our l5 plan picks up 1_2_7_8 There is now a non-adjacent run within l5 while the middle is picked up by l4 l5: 1_2_x_x_x_x_7_8 l4: x_x_3_4_5_6_x_x This will make it so if this is encountered we instead will get l5: 1_2_x_x_x_x_x_x l4: x_x_3_4_5_6_7_8 satisfying our adjacent file policy

devanbenz · 2025-10-31T20:26:39Z

Closing this PR per @davidby-influx's implementation is in master-1.x.

devanbenz marked this pull request as ready for review September 2, 2025 14:21

devanbenz requested review from davidby-influx and gwossum and removed request for davidby-influx September 2, 2025 14:21

devanbenz self-assigned this Sep 2, 2025

devanbenz added kind/bug 1.x area/tsm labels Sep 2, 2025

devanbenz added 4 commits September 2, 2025 12:49

feat: Adds 4 more tests

ff08f6e

* 2 level 2 files * 3 level 2 files * 4 level 2 files * 5 level 2 files

feat: Make sure tests are level4

b9a9667

feat: Let's use less then default max points per blocks, taking a loo…

b3f1b2d

…k at the files on disk a lot of these l4 files are not at the max points per block, just at the max file size of 2.1 GB.

feat: Sets the planning logic for Plan() to include checks to ensure …

2c178fd

…we don't have nested lower level files in our plans

devanbenz commented Sep 2, 2025

View reviewed changes

devanbenz added 2 commits September 2, 2025 18:25

feat: Adds logic where we will only compact lower level files

4425ded

with higher level files if it is nested

feat: Only push up tests to show failures

4ee4278

devanbenz mentioned this pull request Sep 3, 2025

feat: Only tests for PR 26766 #26772

Closed

devanbenz added 2 commits September 3, 2025 10:12

feat: Adds test generation file

86ad19a

feat: Adds static test case file

84db100

Claude generated the test cases and I modified the results as to what I would expect.

devanbenz marked this pull request as draft September 3, 2025 16:34

feat: Set up lower first block count tests

3020f03

devanbenz commented Sep 3, 2025

View reviewed changes

tsdb/engine/tsm1/compact_test.go Outdated Show resolved Hide resolved

devanbenz commented Sep 3, 2025

View reviewed changes

tsdb/engine/tsm1/compact.go Outdated Show resolved Hide resolved

feat: Move other tests in to static test file

0cbf752

devanbenz added 2 commits September 5, 2025 15:56

feat: Reword nested compactor config

0030fb0

feat: Run tests for both ff off and on

397236e

philjb reviewed Sep 5, 2025

View reviewed changes

devanbenz added 3 commits September 7, 2025 10:34

feat: Adjust logic so we will not pick up files out of order

97db9f3

feat: Simplify logic for nested PlanLevel files

d19d19d

philjb reviewed Sep 8, 2025

View reviewed changes

devanbenz added 19 commits September 8, 2025 13:45

feat: Trying to make sure generations are not compacted out of order

0c556e2

feat: Adds property based testing framework for compaction tests

065d359

feat: Working on property based testing framework

a697635

feat: Adds property based testing framework for compaction

2a73c97

feat: Adds common for compact tests

b708182

feat: Adjusts tests to remove file gaps

6896118

feat: Remove comments

4d03abe

feat: Update with results

8d241d9

feat: Adjust AdjacentFileProperty to improve algorithm

c14a416

feat: Adds a log instead of requirement for violations in adjacency

a149a65

feat: I DO NOT think that I need to sort

f3498c0

The most important thing is that we don't change the ordering of input so this will stop that, as we check the index anyways 🤔

feat: Use assert for testing

5ace584

feat: Merge branch 'db/compaction-property-based-testing' into db/bad…

40affaa

…-tsm-state-compact

feat: rename method to validateFileAdjacency

061ade9

feat: Merge branch 'db/compaction-property-based-testing' into db/bad…

cd61e6d

…-tsm-state-compact

feat: Just build the map and use that

36e8106

feat: Merge branch 'db/compaction-property-based-testing' into db/bad…

2e11b6f

…-tsm-state-compact

feat: cleanup

f1042fa

devanbenz closed this Oct 31, 2025

	// step is how may files to compact in a group. We want to clamp it at 4 but also stil
	// return groups smaller than 4.
	step := 4
	if step > end {
	step = end
	}

	// slice off the generations that we'll examine
	generations = generations[start:end]

	// Loop through the generations in groups of size step and see if we can compact all (or
	// some of them as group)
	groups := []tsmGenerations{}
	for i := 0; i < len(generations); i += step {
	var skipGroup bool
	startIndex := i

	for j := i; j < i+step && j < len(generations); j++ {
	gen := generations[j]
	lvl := gen.level()

	// Skip compacting this group if there happens to be any lower level files in the
	// middle. These will get picked up by the level compactors.
	if lvl <= 3 {
	skipGroup = true
	break
	}

	// Skip the file if it's over the max size and it contains a full block
	if gen.size() >= uint64(tsdb.MaxTSMFileSize) && gen.files[0].FirstBlockCount >= tsdb.DefaultMaxPointsPerBlock && !gen.hasTombstones() {
	startIndex++
	continue
	}
	}

	if skipGroup {
	continue
	}

	endIndex := i + step
	if endIndex > len(generations) {
	endIndex = len(generations)
	}
	if endIndex-startIndex > 0 {
	groups = append(groups, generations[startIndex:endIndex])
	}
	}

	if len(groups) == 0 {
	return nil, 0
	}

	func TestDefaultPlanner_Plan_CompactsMiddleSteps(t *testing.T) {
	data := []tsm1.FileStat{
	{
	Path: "01-04.tsm1",
	Size: 64 * 1024 * 1024,
	},
	{
	Path: "02-04.tsm1",
	Size: 64 * 1024 * 1024,
	},
	{
	Path: "03-04.tsm1",
	Size: 64 * 1024 * 1024,
	},
	{
	Path: "04-04.tsm1",
	Size: 64 * 1024 * 1024,
	},
	{
	Path: "05-02.tsm1",
	Size: 2 * 1024 * 1024,
	},
	}

	cp := tsm1.NewDefaultPlanner(
	newFakeFileStore(withFileStats(data)),
	tsdb.DefaultCompactFullWriteColdDuration,
	)

	expFiles := []tsm1.FileStat{data[0], data[1], data[2], data[3]}
	tsm, pLen := cp.Plan(time.Now())
	if exp, got := len(expFiles), len(tsm[0]); got != exp {
	t.Fatalf("tsm file length mismatch: got %v, exp %v", got, exp)
	} else if pLen != int64(len(tsm)) {
	t.Fatalf("tsm file plan length mismatch: got %v, exp %v", pLen, exp)
	}

	for i, p := range expFiles {
	if got, exp := tsm[0][i], p.Path; got != exp {
	t.Fatalf("tsm file mismatch: got %v, exp %v", got, exp)
	}
	}
	}

feat: Adds detection for orphaned TSM files while planning compaction #26766

feat: Adds detection for orphaned TSM files while planning compaction #26766

Conversation

devanbenz commented Aug 30, 2025

Uh oh!

davidby-influx commented Aug 30, 2025

Uh oh!

devanbenz commented Aug 30, 2025

Uh oh!

davidby-influx commented Aug 30, 2025 via email

Uh oh!

devanbenz Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devanbenz Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

davidby-influx Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

davidby-influx Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

davidby-influx commented Sep 3, 2025

Uh oh!

devanbenz commented Sep 3, 2025

Uh oh!

Uh oh!

Uh oh!

philjb Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

davidby-influx Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

philjb Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

philjb left a comment

Choose a reason for hiding this comment

Uh oh!

philjb Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

philjb Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

devanbenz commented Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

devanbenz Sep 2, 2025 •

edited

Loading

philjb Sep 5, 2025 •

edited

Loading

philjb Sep 8, 2025 •

edited

Loading