Performance 18254756429: Improve hash grouping aggregation parallelism

[alexowens90]

Reference Issues/PRs

18254756429

What does this implement or fix?

Poor quality hash implementations of integral types, including at least some implementations of std::hash are basically a static cast. e.g. std::hash<int64_t>{}(100) == 100. This is fast, but leads to poor distributions in our bucketing, where we mod the hash with the number of buckets. In particular, if performing a grouping hash on a timeseries where the time points are dates results in all of the rows being partitioned into bucket zero, which then results in no parallelism in the aggregation clause.

Swap to using a consistent hash function across all supported platforms with improved uniformity.

[jjerphan]

Memory allocation grows linearly here before the process is SIGKILLed after OOM.

Most of the time seems to be spent here:

[jjerphan]

The hard-coded constant influences memory allocation; if we decrease it (as shown bellow), the test passes for instance.

diff --git i/cpp/arcticdb/processing/test/test_clause.cpp w/cpp/arcticdb/processing/test/test_clause.cpp
index 3bbe66a00..f9f5add0a 100644
--- i/cpp/arcticdb/processing/test/test_clause.cpp
+++ w/cpp/arcticdb/processing/test/test_clause.cpp
@@ -84,7 +84,7 @@ TEST(Clause, PartitionHashQuality) {
             make_scalar_type(DataType::INT64), 0, AllocationType::DYNAMIC, Sparsity::NOT_PERMITTED
     );
     for (auto idx = 0; idx < num_rows; ++idx) {
-        col->set_scalar<int64_t>(idx, idx * 1'000'000'000);
+        col->set_scalar<int64_t>(idx, idx * 100'000);
     }
     seg.add_column("grouping", col);
     seg.set_row_id(num_rows - 1);

[jjerphan]

I can reproduce with gcc 11.2 from conda-forge (also used for the PyPI builds).

[alexowens90]

Yeh I just came to the same conclusion. The loop isn't exiting when idx hits 100.
auto is resolving to int, which is equivalent to int32_t here. Changing the auto to int64_t fixes the issue. Something about the idx * 1'000'000'000 overflowing the max representable value of int32_t is breaking something, which feels like a compiler bug, since both arguments to set_scalar are passed by value.

[jjerphan]

Yes, time to refresh and old but gold quiz.