[EPIC] invoke inference UDF on GPU

[jdries]

An inference UDF, like for instance used by ESA WorldCereal, can execute an order of magnitude faster when a GPU is available.

The problem is that GPU nodes are more expensive, so it is not a good idea to keep them allocated for a long time.
At the same time, it would be cost effective if cpu based executors can be decommissioned when we switch to gpu.

In many cases, this inference can run at the end of the job, so that could be a potential constraint to limit scope.

There's multiple approaches that we can take here:

• spark RDD resource profiles: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.RDD.withResources.html
• using CWL (calrissian) to execute custom code
• send inference request to an ML serving API running on a dynamically scaled cluster of GPU nodes.

An extra design constraint is the ability to keep ML model weights private. A solution based on API or custom docker image might be beneficial in that regard. (But maybe there's also other approaches??)