annotation error

[Mortezanavidi]

Hi, regardless of if i host the app on a supervisely agent or one of my own agents, and regardless of if the owl-vit is on an agent with a gpu or cpu or both, whenever i try to predict an object in my dataset using a reference, i get this error:
Temporary connection error, please wait ...: Retrying (1/10). method=annotations.info, url=https://app.supervisely.com/public/api/v3/annotations.info, details=500 Server Error: Internal Server Error for url: https://app.supervisely.com/public/api/v3/annotations.info ({"error":"Internal Server Error"})
here is a screenshot:

the agents that owl-vit are served on each has 4x Nvidia A6000 and they are connected to a 10Gb internet ports with the supervisely api pingable without a problem.

how can i fix this problem? ( unfortunately the error doesn't say anything else regarding it is an internal error )

Edit 1: this error doesn't show on step 4 which is "Preview Result", it can predict objects and i can see my server gpus and cpu being used, but this error shows up on step 5 which is "Apply Model".

Here is the full error:
Warn2024.04.21 04:44:14 Temporary connection error, please wait ...: Retrying (1/10). method=annotations.info, url=https://app.supervisely.com/public/api/v3/annotations.info, details=500 Server Error: Internal Server Error for url: https://app.supervisely.com/public/api/v3/annotations.info ({"error":"Internal Server Error"}) Warn2024.04.21 04:44:15 Temporary connection error, please wait ...: Retrying (2/10). method=annotations.info, url=https://app.supervisely.com/public/api/v3/annotations.info, details=500 Server Error: Internal Server Error for url: https://app.supervisely.com/public/api/v3/annotations.info ({"error":"Internal Server Error"}) Warn2024.04.21 04:44:17 Temporary connection error, please wait ...: Retrying (3/10). method=annotations.info, url=https://app.supervisely.com/public/api/v3/annotations.info, details=500 Server Error: Internal Server Error for url: https://app.supervisely.com/public/api/v3/annotations.info ({"error":"Internal Server Error"}) Warn2024.04.21 04:44:21 Temporary connection error, please wait ...: Retrying (4/10). method=annotations.info, url=https://app.supervisely.com/public/api/v3/annotations.info, details=500 Server Error: Internal Server Error for url: https://app.supervisely.com/public/api/v3/annotations.info ({"error":"Internal Server Error"}) Warn2024.04.21 04:44:29 Temporary connection error, please wait ...: Retrying (5/10). method=annotations.info, url=https://app.supervisely.com/public/api/v3/annotations.info, details=500 Server Error: Internal Server Error for url: https://app.supervisely.com/public/api/v3/annotations.info ({"error":"Internal Server Error"}) Warn2024.04.21 04:44:45 Temporary connection error, please wait ...: Retrying (6/10). method=annotations.info, url=https://app.supervisely.com/public/api/v3/annotations.info, details=500 Server Error: Internal Server Error for url: https://app.supervisely.com/public/api/v3/annotations.info ({"error":"Internal Server Error"}) Warn2024.04.21 04:45:17 Temporary connection error, please wait ...: Retrying (7/10). method=annotations.info, url=https://app.supervisely.com/public/api/v3/annotations.info, details=500 Server Error: Internal Server Error for url: https://app.supervisely.com/public/api/v3/annotations.info ({"error":"Internal Server Error"}) Warn2024.04.21 04:46:17 Temporary connection error, please wait ...: Retrying (8/10). method=annotations.info, url=https://app.supervisely.com/public/api/v3/annotations.info, details=500 Server Error: Internal Server Error for url: https://app.supervisely.com/public/api/v3/annotations.info ({"error":"Internal Server Error"}) Warn2024.04.21 04:47:18 Temporary connection error, please wait ...: Retrying (9/10). method=annotations.info, url=https://app.supervisely.com/public/api/v3/annotations.info, details=500 Server Error: Internal Server Error for url: https://app.supervisely.com/public/api/v3/annotations.info ({"error":"Internal Server Error"}) Warn2024.04.21 04:48:18 Temporary connection error, please wait ...: Retrying (10/10). method=annotations.info, url=https://app.supervisely.com/public/api/v3/annotations.info, details=500 Server Error: Internal Server Error for url: https://app.supervisely.com/public/api/v3/annotations.info ({"error":"Internal Server Error"}) Info2024.04.21 04:48:46 INFO: connection closed Info2024.04.21 04:48:48 Start dumping app UI for offline mode Info2024.04.21 04:48:48 INFO: 192.168.32.2:51142 - "GET / HTTP/1.1" 200 OK Info2024.04.21 04:48:49 Start dumping app UI for offline mode Info2024.04.21 04:48:49 INFO: 192.168.32.2:51144 - "GET /?slyContext={%22serverAddress%22:%22https://app.supervisely.com%22,%22taskId%22:708946,%22teamId%22:88578,%22workspaceId%22:102354,%22isStaticVersion%22:false,%22env%22:{%22SLY_VERSION%22:%226.9.14%22,%22STAGING%22:false,%22COMMUNITY%22:true,%22PUBLIC_MODE%22:false,%22IMPORT_FILES_CHECK_URL%22:%22%22,%22IS_K8S%22:false}} HTTP/1.1" 200 OK Info2024.04.21 04:48:50 INFO: 192.168.32.2:51156 - "POST /session-info HTTP/1.1" 200 OK Info2024.04.21 04:48:50 INFO: 192.168.32.2:51164 - "POST /state HTTP/1.1" 200 OK Info2024.04.21 04:48:50 INFO: 192.168.32.2:51166 - "POST /data HTTP/1.1" 200 OK Info2024.04.21 04:48:50 INFO: ('192.168.32.2', 51178) - "WebSocket /ws" [accepted] Info2024.04.21 04:48:50 INFO: connection open Error2024.04.21 04:49:18 Something went wrong. Error: Retry limit exceeded ('https://app.supervisely.com/public/api/v3/annotations.info') stack=[ Traceback (most recent call last): File "/app/repo/./src/main.py", line 846, in run_model image_ann_json = g.api.annotation.download(image_info.id).annotation File "/usr/local/lib/python3.8/dist-packages/supervisely/api/annotation_api.py", line 292, in download response = self._api.post( File "/usr/local/lib/python3.8/dist-packages/supervisely/api/api.py", line 459, in post raise requests.exceptions.RetryError("Retry limit exceeded ({!r})".format(url)) requests.exceptions.RetryError: Retry limit exceeded ('https://app.supervisely.com/public/api/v3/annotations.info') ]