[TabArena] Model Usage Workflow Brainstorming

[LennartPurucker]

Below are some thoughts on how we could create the optimal model usage workflow. The idea is to enable all models implemented as part of TabArena to be easily usable by anyone on new datasets, just like any other scikit-learn compatible model.

The current workflow is shown here: https://github.com/TabArena/tabarena_benchmarking_examples/blob/main/tabarena_minimal_example/run_tabarena_model.py

My ideal workflow would be something like this:

from sklearn.datasets import load_breast_cancer
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

# Get Data
X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42)

# Import a TabArena model
from tabarena.pipelines import RealMLP # or any other model

# Train TabArena Model
clf = RealMLP()
clf.fit(X=X_train, y=y_train)

# Predict and score
prediction_probabilities = clf.predict_proba(X=X_test)
print("ROC AUC:", roc_auc_score(y_test, prediction_probabilities))

Some requirements that would be nice to have:

• The pipeline has support for model-agnostic preprocessing (by default True)
• The model ideal should be sklearn compatible up to a certain degree. We might also need explicit classifier/regressor wrapper interfaces as a result. See my PHE code as an example for this https://github.com/PriorLabs/tabpfn-extensions/blob/main/src/tabpfn_extensions/post_hoc_ensembles/sklearn_interface.py
• One should be able to set per parameter if the model is standalone, a bagging model, or the full AG AutoML pipeline (i.e., the different options from here https://auto.gluon.ai/stable/tutorials/tabular/advanced/tabular-custom-model.html)
• Ideally, the model interface would have a docstring, parameters, and documentation for all relevant hyperparameters of the model and easy access to a default search space (like RealMLPPipeline.search_space)
• Ideally, we could have (by default) almost no output/logging from running the model or bagged versions.

[Innixma]

Fully agree with all requirements, but have a comment here:

We might also need explicit classifier/regressor wrapper interfaces as a result.

That will be a major decision we will need to make. I originally avoided this for the model code, but this was a decision I made a very long time ago for my own sake to have less boilerplate at that point in time (unsure if that was what actually ended up happening in the long run though). If this is simply wrapping the model code, this isn't a big deal, as it doesn't require a big refactor.

Separately, from an ease of use perspective, are we thinking of having the user specify the classifier/regressor models? Ex: from tabarena.pipelines import RealMLPRegressor, or alternatively

constructor = RealMLP()
model = constructor.fit(X=X_train, y=y_train)
# model is not a RealMLP object, instead it is `RealMLPRegressor` if regression, else `RealMLPClassifier` if classification. RealMLP is just a constructor class.

Another question will be, if we do make pipelines, will it be separate from/inherit from AbstractModel or will it replace/be AbstractModel.

One should be able to set per parameter if the model is standalone, a bagging model, or the full AG AutoML pipeline

If this is possible, we need to determine if we expect the model object to have the same methods regardless of which setting it is. Aka, bagging models have the models variable, but a standalone model doesn't. Or will we treat it such that to access the internal model we need to call clf.model, such that for bagging it would be clf.model.models for the internal list of fold models?

or the full AG AutoML pipeline

Would the grand vision be that TabularPredictor in some fashion inherits from AbstractPipeline? What separates TabularPredictor from tabarena.pipelines.RealMLP (ignoring the X, y input format)?

[Innixma]

Some brainstorming code for one way this could work without refactoring the AbstractModel code:

# tabarena.pipelines.py
from tabarena.models import RealMLPModel

# approach 1: adopt sklearn classifier/regressor interface
RealMLPClassifier = classifier_pipeline_factory(cls=RealMLPModel)
RealMLPRegressor = regressor_pipeline_factory(cls=RealMLPModel)

# approach 2: keep the AG approach of inferring problem type for the user
RealMLP = pipeline_factory(cls=RealMLPModel)

# approach 3: support both 1 and 2
# RealMLP simply returns RealMLPClassifier or RealMLPRegressor after the fit call.

[LennartPurucker]

One more thought, we could add the bagging/predictor idea to the factory, e.g..:

RealMLP = pipeline_factory(cls=RealMLPModel, cv_ensemble=True) # bagged model
RealMLP = pipeline_factory(cls=RealMLPModel, automl=True) # TabularPredictor version