Search before asking
Description
@EricGao888 has released a plugin earlier to implement batch job submission on Aliyun Serverless Spark. After a careful review, i think there are still something to be optimized. as listed below:
- retry for remote api call is missing now, which may cause inconsistence state when some temporary issue occur like network failure
- idempotent submitting is necessary especially for the same scheduled node in the same process instance
- termination should be ensured cuz a single call on CancelJobRun Api can only turn a remote job into cancelling state, but not really canceled
Are you willing to submit a PR?
Code of Conduct