Meaning of fold in the results parquet file

[puhsu]

Hi! I was trying to match some of my run results with the ones reported in tabarena and had a question about folds and repeats reporting in the provided results dataframe. I am using the following dataframe:

df = pd.read_parquet("https://tabarena.s3.us-west-2.amazonaws.com/results/df_results_leaderboard.parquet")

It has the following columns:

# df.columns -> returns
Index(['dataset', 'fold', 'method', 'metric_error', 'time_train_s',
       'time_infer_s', 'metric_error_val', 'config_selected', 'seed',
       'method_metadata', 'ensemble_weight', 'problem_type', 'metric',
       'method_type', 'method_subtype', 'config_type'],
      dtype='object')

There are up to 30 "folds" – I get that the paper used 3-fold outer validation for scoring with up to 10 repeats. Can you please help me understand how do dataset fold indexes and repeat indexes are converted to these 30 folds.

Basically, which of the two options is correct (do we first iterate over fold and then over repeat or is it the other way around):

# just an example
df_fold = 13

# option 1:
openml_fold = df_fold // 10
openml_repeat = df_fold % 10

# or option 2:
openml_fold = df_fold % 3
openml_repeat = df_fold // 3

---

To help you better understand where my question is coming from. I wanted to quickly check some results on a few different small dataset splits (e.g. fold=0, repeat in range(0,5) -- in openml terms when loading the data). And now I want to match results that I have to the dataframe.

[LennartPurucker]

Heyho @puhsu, thank you for the question! Sorry for the late reply!

It is option 1, we first iterate over fold (@Innixma correct me if I am wrong).

It is similar to the following scikit-learn logic:

for split_i, (train_idx, test_idx) in enumerate(splits_from_an_sklearn_spliter):
    repeat_i = split_i // n_splits
    fold_i = split_i % n_splits

Let us keep this issue open until we find a good spot to document it.

[Innixma]

As @LennartPurucker said, it is option 1, as the original logic was implemented prior to repeats being supported, and then we made it support repeats via the above method.

In future we hopefully can disambiguate this by renaming it to "split" while also providing fold and repeat directly as columns.