[TabArena] [Feat] Improve Getting Datasets Workflow

[LennartPurucker]

See TabArena/tabarena_benchmarking_examples#5

I think we could easily add a function or example scripts on how to use the datasets w/o TabArena code to people who want to use them.
How exactly is to be determined.

[LennartPurucker]

Rel.: https://github.com/TabArena/tabarena_benchmarking_examples/blob/main/tabarena_minimal_example/get_tabarena_data.py

[jediroberthatch]

Related(?): when using that get_tabarena_data.py script (with timing prints added), I am seeing it take about 8 minutes to get a single dataset:

Processing Task ID: 363613
WARNING:root:Received uncompressed content from OpenML for https://api.openml.org/data/v1/download/22125216/Amazon_employee_access.arff.
245.9502552999911 seconds to get task
Got task: 363613. Getting dataset from that task...
231.56049930000154 seconds to get dataset
Task ID: 363613, Dataset ID: 46905, Dataset Name: Amazon_employee_access, Number of Samples: 32769.0

And also often fails:
Processing Task ID: 363614
WARNING:root:Received uncompressed content from OpenML for https://api.openml.org/data/v1/download/22125217/anneal.arff.
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'RemoteDisconnected('Remote end closed connection without response')': /datasets/0004/46906/dataset_46906.pq
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'RemoteDisconnected('Remote end closed connection without response')': /datasets/0004/46906/dataset_46906.pq
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Is this a backend issue with OpenML? Is there any streamlined way to workaround this issue? If this issue is expected and/or persists, and assuming that providing a zip of all the datasets and 'task' data is not practical, it might be nice to have a variant of the above script that saves the data in subfolder of current working dir or in user-specified dir instead of hidden in a cache folder somewhere, since it takes so long to get the first time.

[LennartPurucker]

Yes, I think this was an issue with the backend of OpenML due to temporary server issues: openml/OpenML#1232

So far, there is no real workaround. But I would have an idea: we could host a backup of all the OpenML versions of all the TabArena datasets and tasks. That is, we can share an OpenML cache directory for users to download and then users would only need to point their OpenML cache path to this backup directory. We can only do this as long as the datasets are from the public domain (as they are for TabArena). @Innixma, thoughts?

it might be nice to have a variant of the above script that saves the data in subfolder of current working dir or in user-specified dir instead of hidden in a cache folder somewhere

You can change the cache path of OpenML to any dir you want to use, e.g.,:

import openml
openml.config.set_root_cache_directory(root_cache_directory=YOUR_DIR)

[LennartPurucker]

@jediroberthatch here you go: https://data.lennart-purucker.com/tabarena/openml_dataset_backup/

You have to download the file, unzip it, and point your OpenML cache to it

import openml
openml.config.set_root_cache_directory(root_cache_directory=`/path/to/.openml-cache`)

Note the path should point to the .openml-cache such that the next sub dir is org.

[jediroberthatch]

Thanks, that download worked great for me.

Re my original report: I found a secondary issue was an old version of openml 0.12.0. (The conda env I was using happened to have a package that forced an old version of openml).

With the correct openml version (0.15.1) the cache above worked great, and while the specifics are slightly different (get_task and get_dataset now work, but not dataset.get_data), I am still seeing slowness if trying to download from openml directly. Looks like I am hitting the same or similar to openml/OpenML#1231

[LennartPurucker]

Mhm, dataset.get_data should work. Maybe check that it uses the parquet files etc

[joaquinvanschoren]

Looks like an issue with the parquet files indeed. Looking into it.

[joaquinvanschoren]

This may take a little longer. I need some info from people who are on holidays this week. I'll get back to you asap.

[joaquinvanschoren]

Fixed now :)