More exhaustive error catching job manager

[HansVRP]

uncaught 404s can still stop the runtime of the job manager:

openeo.rest.OpenEoApiError: [404] JobNotFound: The batch job 'j-2508280309394033a7b090abae69f205' does not exist. (ref: r-25082803093944faa0f2eb0e453b00cb)

In order to ensure a consistently running job manager we should be less dependent on the exact error codes.

Discussion:

https://teams.microsoft.com/l/message/19:2941a270bf8e48a2a8e8a23975051c11@thread.skype/1756359865243?tenantId=9e2777ed-8237-4ab9-9278-2c144d6f6da3&groupId=8c9c739d-2544-4def-8cd4-b65970551b70&parentMessageId=1756359865243&teamName=Unit%20TAP&channelName=openEO-users&createdTime=1756359865243

[HansVRP]

As discussed in: #632

There is an additional need to eventually force a job to the error state and continue.

[soxofaan]

more extensive stack trace of issue above:

line 497, in run_jobs
    self._job_update_loop(job_db=job_db, start_job=start_job, stats=stats)
  File ".../openeo/extra/job_management/__init__.py", line 537, in _job_update_loop
    self._launch_job(start_job, df=not_started, i=i, backend_name=backend_name, stats=stats)
  File ".../openeo/extra/job_management/__init__.py", line 602, in _launch_job
    status = job.status()
             ^^^^^^^^^^^^
  File ".../openeo/rest/job.py", line 88, in status
    return self.describe().get("status", "N/A")
           ^^^^^^^^^^^^^^^
  File ".../openeo/rest/job.py", line 78, in describe
    return self.connection.get(f"/jobs/{self.job_id}", expected_status=200).json()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
  File ".../openeo/rest/_connection.py", line 154, in _raise_api_error
    raise OpenEoApiError(
openeo.rest.OpenEoApiError: [404] JobNotFound: The batch job 'j-2508280309394033a7b090abae69f205' does not exist. (ref: r-25082803093944faa0f2eb0e453b00cb)

what might have happened here is that the job was actually created, but the status poll was too early and the job metadata was not properly propagated yet, causing the "not found" issue.

This is probably a very rare problem, but could improved by adding a simple retry

[soxofaan]

as discussed irl, adding retries does not guarantee that it will work eventually,
there should also be (maybe even more importantly than the retry) a try-except around _launch_job, to avoid crashing the whole run on a job launching problem/glitch.

[soxofaan]

While doing a load test for #736 I had a comparable crash
not during job starting, but when trying to download the results

stack trace

---------------------------------------------------------------------------
OpenEoApiError                            Traceback (most recent call last)
Cell In [12], line 1
----> 1 manager.run_jobs(
      2     job_db=job_db, 
      3     start_job=create_job,
      4 )

File [~/src/openeo/openeo-python-client/openeo/extra/job_management/__init__.py:536](http://localhost:8888/lab/workspaces/auto-S/tree/workspace/2025/src/openeo/openeo-python-client/openeo/extra/job_management/__init__.py#line=535), in MultiBackendJobManager.run_jobs(self, df, start_job, job_db, **kwargs)
    526 self._worker_pool = _JobManagerWorkerThreadPool()
    528 while (
    529     sum(
    530         job_db.count_by_status(
   (...)
    534     > 0
    535 ):
--> 536     self._job_update_loop(job_db=job_db, start_job=start_job, stats=stats)
    537     stats["run_jobs loop"] += 1
    539     # Show current stats and sleep

File [~/src/openeo/openeo-python-client/openeo/extra/job_management/__init__.py:591](http://localhost:8888/lab/workspaces/auto-S/tree/workspace/2025/src/openeo/openeo-python-client/openeo/extra/job_management/__init__.py#line=590), in MultiBackendJobManager._job_update_loop(self, job_db, start_job, stats)
    589 # TODO: move this back closer to the `_track_statuses` call above, once job done[/error](http://localhost:8888/error) handling is also handled in threads?
    590 for job, row in jobs_done:
--> 591     self.on_job_done(job, row)
    593 for job, row in jobs_error:
    594     self.on_job_error(job, row)

File [~/src/openeo/openeo-python-client/openeo/extra/job_management/__init__.py:746](http://localhost:8888/lab/workspaces/auto-S/tree/workspace/2025/src/openeo/openeo-python-client/openeo/extra/job_management/__init__.py#line=745), in MultiBackendJobManager.on_job_done(self, job, row)
    743 metadata_path = self.get_job_metadata_path(job.job_id)
    745 self.ensure_job_dir_exists(job.job_id)
--> 746 job.get_results().download_files(target=job_dir)
    748 with metadata_path.open("w", encoding="utf-8") as f:
    749     json.dump(job_metadata, f, ensure_ascii=False)

File [~/src/openeo/openeo-python-client/openeo/rest/job.py:602](http://localhost:8888/lab/workspaces/auto-S/tree/workspace/2025/src/openeo/openeo-python-client/openeo/rest/job.py#line=601), in JobResults.download_files(self, target, include_stac_metadata, chunk_size)
    599     raise OpenEoClientException(f"Target argument {target} exists but isn't a folder.")
    600 ensure_dir(target)
--> 602 downloaded = [a.download(target, chunk_size=chunk_size) for a in self.get_assets()]
    604 if include_stac_metadata:
    605     # TODO #184: convention for metadata file name?
    606     metadata_file = target [/](http://localhost:8888/) DEFAULT_JOB_RESULTS_FILENAME

File [~/src/openeo/openeo-python-client/openeo/rest/job.py:539](http://localhost:8888/lab/workspaces/auto-S/tree/workspace/2025/src/openeo/openeo-python-client/openeo/rest/job.py#line=538), in JobResults.get_assets(self)
    535 """
    536 Get all assets from the job results.
    537 """
    538 # TODO: add arguments to filter on metadata, e.g. to only get assets of type "image[/tiff](http://localhost:8888/tiff)"
--> 539 metadata = self.get_metadata()
    540 # API 1.0 style: dictionary mapping filenames to metadata dict (with at least a "href" field)
    541 assets = metadata.get("assets", {})

File [~/src/openeo/openeo-python-client/openeo/rest/job.py:527](http://localhost:8888/lab/workspaces/auto-S/tree/workspace/2025/src/openeo/openeo-python-client/openeo/rest/job.py#line=526), in JobResults.get_metadata(self, force)
    525 """Get batch job results metadata (parsed JSON)"""
    526 if self._results is None or force:
--> 527     self._results = self._job.connection.get(
    528         self._job.get_results_metadata_url(), expected_status=200
    529     ).json()
    530 return self._results

File [~/src/openeo/openeo-python-client/openeo/rest/_connection.py:204](http://localhost:8888/lab/workspaces/auto-S/tree/workspace/2025/src/openeo/openeo-python-client/openeo/rest/_connection.py#line=203), in RestApiConnection.get(self, path, params, stream, auth, **kwargs)
    186 def get(
    187     self,
    188     path: str,
   (...)
    193     **kwargs,
    194 ) -> Response:
    195     """
    196     Do GET request to REST API.
    197 
   (...)
    202     :return: response: Response
    203     """
--> 204     return self.request("get", path=path, params=params, stream=stream, auth=auth, **kwargs)

File [~/src/openeo/openeo-python-client/openeo/rest/connection.py:687](http://localhost:8888/lab/workspaces/auto-S/tree/workspace/2025/src/openeo/openeo-python-client/openeo/rest/connection.py#line=686), in Connection.request(self, method, path, headers, auth, check_error, expected_status, **kwargs)
    680     return super(Connection, self).request(
    681         method=method, path=path, headers=headers, auth=auth,
    682         check_error=check_error, expected_status=expected_status, **kwargs,
    683     )
    685 try:
    686     # Initial request attempt
--> 687     return _request()
    688 except OpenEoApiError as api_exc:
    689     if (
    690         api_exc.http_status_code in {HTTP_401_UNAUTHORIZED, HTTP_403_FORBIDDEN}
    691         and api_exc.code == "TokenInvalid"
    692     ):
    693         # Auth token expired: can we refresh?

File [~/src/openeo/openeo-python-client/openeo/rest/connection.py:680](http://localhost:8888/lab/workspaces/auto-S/tree/workspace/2025/src/openeo/openeo-python-client/openeo/rest/connection.py#line=679), in Connection.request.<locals>._request()
    679 def _request():
--> 680     return super(Connection, self).request(
    681         method=method, path=path, headers=headers, auth=auth,
    682         check_error=check_error, expected_status=expected_status, **kwargs,
    683     )

File [~/src/openeo/openeo-python-client/openeo/rest/_connection.py:141](http://localhost:8888/lab/workspaces/auto-S/tree/workspace/2025/src/openeo/openeo-python-client/openeo/rest/_connection.py#line=140), in RestApiConnection.request(self, method, path, params, headers, auth, check_error, expected_status, **kwargs)
    139 expected_status = ensure_list(expected_status) if expected_status else []
    140 if check_error and status >= 400 and status not in expected_status:
--> 141     self._raise_api_error(resp)
    142 if expected_status and status not in expected_status:
    143     raise OpenEoRestError(
    144         "Got status code {s!r} for `{m} {p}` (expected {e!r}) with body {body}".format(
    145             m=method.upper(), p=path, s=status, e=expected_status, body=resp.text
    146         )
    147     )

File [~/src/openeo/openeo-python-client/openeo/rest/_connection.py:163](http://localhost:8888/lab/workspaces/auto-S/tree/workspace/2025/src/openeo/openeo-python-client/openeo/rest/_connection.py#line=162), in RestApiConnection._raise_api_error(self, response)
    161     error_message = info.get("message")
    162     if error_code and isinstance(error_code, str) and error_message and isinstance(error_message, str):
--> 163         raise OpenEoApiError(
    164             http_status_code=status_code,
    165             code=error_code,
    166             message=error_message,
    167             id=info.get("id"),
    168             url=info.get("url"),
    169         )
    171 # Failed to parse it as a compliant openEO API error: show body as-is in the exception.
    172 text = response.text

OpenEoApiError: [400] JobNotFinished: Batch job has not finished computing the results yet. Please try again later or contact our support. (ref: r-2509091209134202a235556a6e99c5d8)