predict_class and predict_probabilities

[soxofaan]

@JeroenVerstraelen is working on an implementation of CatBoost base ML in the VITO backend and while discussing details a couple of things came up:

• the predict_catboost process would be practically identical to predict_random_forest, except for some textual differences in title and descriptions. Turns out that it is not really necessary to define a dedicated predict_ process for each kind of machine learning model: all the model details are embedded in the ml-model object and you could just use a single predict(data: array, model: ml-model) for all kinds of ML models.
• for some use cases we want to predict the probability of each class instead of a single class prediction. We first considered adding a parameter to toggle between class output or probabilities output, but that would mean that the output type would change: scalar for class prediction and array for probability prediction. Moreover, the former has to be used in reduce_dimension and the other in apply_dimension. It felt error prone and confusing to let these two different patterns depend on a rather inconspicuous boolean parameter. It might be better to have a separate processes for class prediction and probabilities prediction

So with this background, the proposal is to introduce two generic ml prediction processes:

• predict_class(data: array, model: ml-model) -> number
• predict_probabilities(data: array, model: ml-model) -> array

both can be easily spec'ed based on current https://github.com/Open-EO/openeo-processes/blob/draft/proposals/predict_random_forest.json

[m-mohr]

Sounds reasonable. What do others think? @clausmichele @LukeWeidenwalker

[LukeWeidenwalker]

Some thoughts:

• predict_class() is conceptually just an argmax over the probabilities, so predict_probabilities() + something extra. Would it make sense to introduce an argmax process instead? I don't think it's asking too much of users to understand that for classification you need probabilities for each class.
• This process only makes sense for classification models. sklearn has this taxonomy for models (see Base classes -> Regressor, Classifier, Cluster, etc.). If we want to know exactly what the output from an inference process will be (scalar? array of probabilities? array of indices (in the case of clustering)?), we need to know what kind of model we're dealing with. Curious what others think, but I'd be reluctant to go down a path where openeo attempts to replicate significant pieces of sklearn.
• Without replicating/exposing large parts of already existing ML libraries (e.g. sklearn, pytorch/tensorflow, catboost), can openeo enable users to train quality models? E.g. what about model evaluation? Training data selection? Hyperparameter tuning? None of my usual ecosystem is available, so tbh I'd have a hard time building models whose performance I'm actually confident in.
• I don't have the historic context on this, but is the vision of openeo really to do 1) distributed 2) machine-learning 3) on EO data? Or is that an instance of feature creep?
• I'm a big fan of being able to interface with existing/pretrained models for inference from within an openeo process graph. The ml-model STAC extension already does most of the heavy lifting towards enabling that. I'd love to hear people's thoughts about a world where openeo only exposes models that have already been trained elsewhere (and evaluated + quality controlled!).

Sorry for the wall of text - I realize this is out of scope for this issue, feel free to redirect this discussion somewhere more suitable.

[soxofaan]

predict_class() is conceptually just an argmax over the probabilities, so predict_probabilities() + something extra.

In the VITO backend we had problems with getting the probabilities out of Spark's RandomForrest implementation, while getting the class was straightforward. So I think it's best to have predict_class independent from predict_probabilities. I also guess that predict_class usage will be more common than predict_probabilities, so I would not make the former more cumbersome to use than the latter.

This process only makes sense for classification models.

Indeed, this proposal is only for classification models. In that light, it might be more future proof to use predict_class_probabilities instead of predict_probabilities

but I'd be reluctant to go down a path where openeo attempts to replicate significant pieces of sklearn.

on the level of openEO process we would, at best, replicate the API of sklearn in some abstracted way, which doesn't seem like a bad thing to me.

Without replicating/exposing large parts of already existing ML libraries ..., can openeo enable users to train quality models? E.g. what about model evaluation? Training data selection? Hyperparameter tuning?

The long term goal is probably to do this, but at the moment we are just focused on training and inference as the core ML building blocks. All the rest (evaluation, tuning, ...) is now expected to be done client side. Which gives the user the most flexibility anyway.

I don't have the historic context on this, but is the vision of openeo really to do 1) distributed 2) machine-learning 3) on EO data? Or is that an instance of feature creep?

openEO won't stay relevant if ML/AI in some form isn't part of the offering I'd think.
In SRR3 we already did use cases with ML.

[m-mohr]

I'm +1 on the original proposal. What do others think? How do we proceed?

[clausmichele]

I also agree with the proposal. We should move forward with the ML processes to keep openEO attractive! I also partly agree with Lukas, but I would say that for more advanced users and models we could still use UDFs.