Custom build not working with new RTX 5000 series #1309
Description
I tried installing it through Docker Compose, but it does not work on an RTX 5000 GPU. It seems to have an issue with the TensorRT version or something similar. Do you have any suggestions to fix this, or will there be an update in the near future?
I tried using -SubCenter-ArcFace-r100-gpu with Docker Compose on Ubuntu 20.04
{"severity": "ERROR", "message": "Exception on /status [GET]", "request": {"method": "GET", "path": "/status", "filename": "", "api_key": "", "remote_addr": "172.19.0.5"}, "logger": "embedding-calculator", "module": "app", "traceback": "Traceback (most recent call last):\n File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app\n response = self.full_dispatch_request()\n File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1945, in full_dispatch_request\n self.try_trigger_before_first_request_functions()\n File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1993, in try_trigger_before_first_request_functions\n func()\n File "/app/ml/./src/_endpoints.py", line 52, in init_model\n detector(\n File "/app/ml/./src/services/facescan/plugins/mixins.py", line 46, in call\n faces = self._fetch_faces(img, det_prob_threshold)\n File "/app/ml/./src/services/facescan/plugins/mixins.py", line 53, in _fetch_faces\n boxes = self.find_faces(img, det_prob_threshold)\n File "/app/ml/./src/services/facescan/plugins/insightface/insightface.py", line 103, in find_faces\n model = self._detection_model\n File "/usr/local/lib/python3.8/dist-packages/cached_property.py", line 36, in get\n value = obj.dict[self.func.name] = self.func(obj)\n File "/app/ml/./src/services/facescan/plugins/insightface/insightface.py", line 80, in _detection_model\n model.prepare(ctx_id=self._CTX_ID, nms=self._NMS)\n File "/usr/local/lib/python3.8/dist-packages/insightface/app/face_analysis.py", line 32, in prepare\n self.det_model.prepare(ctx_id, nms)\n File "/usr/local/lib/python3.8/dist-packages/insightface/model_zoo/face_detection.py", line 223, in prepare\n out = model.get_outputs()[0].asnumpy()\n File "/usr/local/lib/python3.8/dist-packages/mxnet/ndarray/ndarray.py", line 2568, in asnumpy\n check_call(_LIB.MXNDArraySyncCopyToCPU(\n File "/usr/local/lib/python3.8/dist-packages/mxnet/base.py", line 246, in check_call\n raise get_last_ffi_error()\nmxnet.base.MXNetError: Traceback (most recent call last):\n File "/work/mxnet/include/mshadow/././././cuda/tensor_gpu-inl.cuh", line 128\nName: Check failed: err == cudaSuccess (209 vs. 0) : MapPlanKernel ErrStr:no kernel image is available for execution on the device\n", "build_version": "dev"}
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1945, in full_dispatch_request
self.try_trigger_before_first_request_functions()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1993, in try_trigger_before_first_request_functions
func()
File "/app/ml/./src/_endpoints.py", line 52, in init_model
detector(
File "/app/ml/./src/services/facescan/plugins/mixins.py", line 46, in call
faces = self._fetch_faces(img, det_prob_threshold)
File "/app/ml/./src/services/facescan/plugins/mixins.py", line 53, in _fetch_faces
boxes = self.find_faces(img, det_prob_threshold)
File "/app/ml/./src/services/facescan/plugins/insightface/insightface.py", line 103, in find_faces
model = self._detection_model
File "/usr/local/lib/python3.8/dist-packages/cached_property.py", line 36, in get
value = obj.dict[self.func.name] = self.func(obj)
File "/app/ml/./src/services/facescan/plugins/insightface/insightface.py", line 80, in _detection_model
model.prepare(ctx_id=self._CTX_ID, nms=self._NMS)
File "/usr/local/lib/python3.8/dist-packages/insightface/app/face_analysis.py", line 32, in prepare
self.det_model.prepare(ctx_id, nms)
File "/usr/local/lib/python3.8/dist-packages/insightface/model_zoo/face_detection.py", line 223, in prepare
out = model.get_outputs()[0].asnumpy()
File "/usr/local/lib/python3.8/dist-packages/mxnet/ndarray/ndarray.py", line 2568, in asnumpy
check_call(_LIB.MXNDArraySyncCopyToCPU(
File "/usr/local/lib/python3.8/dist-packages/mxnet/base.py", line 246, in check_call
raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
File "/work/mxnet/include/mshadow/././././cuda/tensor_gpu-inl.cuh", line 128
Name: Check failed: err == cudaSuccess (209 vs. 0) : MapPlanKernel ErrStr:no kernel image is available for execution on the device