Error when importing 10000 images

[JRGit4UE]

My actions before raising this issue

• Read/searched the docs
• Searched past issues

I have 10000 images, 42MB each, mounted into a CVAT share.
Creating a task with 1 such image succeeds
Creating the task with 10000 of them fails - well, partly - as the imported images can be seen and annotated after re-login into CVAT,
but error logs show, some internal structure (i.e. a Task directory containing a client.log) has not been created.

Expected Behaviour

The Task should be created without error.

Current Behaviour

The Task is only partly created.
It can be seen after logout-login,
but whenever we annotate and save something,
an error panel pops up, telling the the log cannot be written

Possible Solution

Maybe, someone have a look at it, as it seems to be a timeout issue to me..

Steps to Reproduce (for bugs)

• logged into fresh CVAT as superuser: django

• created Project: Project1

• with Labels: L1

• created Task: Task1

• Connected file share /media/sda -> A.png (50 byte size)

• Advanced configuration

• Not copying data into CVAT

• Not using zip chunks

• Use cache

• Image quality: 100%

• Format: CVAT for images 1.1

• ok! Task created

---

• created Task2

• Connected file share /media/sda -> 0_16kx1k.png (42MB size)

• ok! Task created

---

• Try to create Task3

• Connected file share /media/sda -> 0_16kx1k.png .. 10000_16kx1k.png (10000 images with 42MB)

• Could not create the task

Context

We try to annotate large RGB images (16k x 1k), which usually works well..

Your Environment

2 CPU slots with 6 cores each (24 virtual cores)
128GB RAM

• Git hash commit (git log -1):
  commit 5e43171 (HEAD -> develop, origin/develop, origin/HEAD)
  Author: Dmitry Kruchinin [email protected]
  Server v1.7
  Core v3.17.0
  Canvas v2.8.0
  UI v1.25.0

• Docker version: Docker 20.10.12, Docker-compose 1.29.2

• Are you using Docker Swarm or Kubernetes? No

• Operating System and version: Ubuntu 20.04

• Code example or link to GitHub repo or gist to reproduce problem:

• Other diagnostic information / logs:

  Logs from `cvat` container

cvat_redis | 1:M 27 Jan 09:22:12.132 * 100 changes in 300 seconds. Saving...
cvat_redis | 1:M 27 Jan 09:22:12.133 * Background saving started by pid 14
cvat_redis | 14:C 27 Jan 09:22:12.170 * DB saved on disk
cvat_redis | 14:C 27 Jan 09:22:12.170 * RDB: 0 MB of memory used by copy-on-write
cvat_redis | 1:M 27 Jan 09:22:12.233 * Background saving terminated with success
cvat | 2022-01-27 09:23:21,547 DEBG 'rqworker_default_0' stderr output:
cvat | INFO - 2022-01-27 09:23:21,547 - worker - default: cvat.apps.engine.task._create_thread(2, {'chunk_size': None, 'size': 0, 'image_quality': 100, 'start_frame': 0, 'st...) (/api/v1/tasks/2)
cvat |
cvat | 2022-01-27 09:23:21,547 DEBG 'rqworker_default_0' stderr output:
cvat | DEBUG - 2022-01-27 09:23:21,547 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
cvat |
cvat | 2022-01-27 09:23:21,567 DEBG 'rqworker_default_0' stderr output:
cvat | DEBUG - 2022-01-27 09:23:21,566 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 90 seconds.
cvat |
cvat | 2022-01-27 09:23:21,578 DEBG 'rqworker_default_0' stderr output:
cvat | [2022-01-27 09:23:21,578] INFO cvat.server: create task #2
cvat |
cvat | 2022-01-27 09:23:21,578 DEBG 'rqworker_default_0' stderr output:
cvat | INFO - 2022-01-27 09:23:21,578 - task - create task #2
cvat |
cvat | 2022-01-27 09:23:22,058 DEBG 'rqworker_default_0' stderr output:
cvat | [2022-01-27 09:23:22,058] INFO cvat.server: Found frames 1 for Data #2
cvat |
cvat | 2022-01-27 09:23:22,058 DEBG 'rqworker_default_0' stderr output:
cvat | INFO - 2022-01-27 09:23:22,058 - task - Found frames 1 for Data #2
cvat |
cvat | 2022-01-27 09:23:22,059 DEBG 'rqworker_default_0' stderr output:
cvat | [2022-01-27 09:23:22,059] INFO cvat.server: New segment for task #2: start_frame = 0, stop_frame = 0
cvat |
cvat | 2022-01-27 09:23:22,059 DEBG 'rqworker_default_0' stderr output:
cvat | INFO - 2022-01-27 09:23:22,059 - task - New segment for task #2: start_frame = 0, stop_frame = 0
cvat |
cvat | 2022-01-27 09:23:22,083 DEBG 'rqworker_default_0' stderr output:
cvat | DEBUG - 2022-01-27 09:23:22,083 - worker - Handling successful execution of job /api/v1/tasks/2
cvat |
cvat | 2022-01-27 09:23:22,086 DEBG 'rqworker_default_0' stderr output:
cvat | INFO - 2022-01-27 09:23:22,086 - worker - default: Job OK (/api/v1/tasks/2)
cvat |
cvat | 2022-01-27 09:23:22,086 DEBG 'rqworker_default_0' stderr output:
cvat | INFO - 2022-01-27 09:23:22,086 - worker - Result is kept for 500 seconds
cvat |
cvat | 2022-01-27 09:23:22,103 DEBG 'rqworker_default_0' stderr output:
cvat | DEBUG - 2022-01-27 09:23:22,102 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
cvat |
cvat | 2022-01-27 09:23:22,103 DEBG 'rqworker_default_0' stderr output:
cvat | DEBUG - 2022-01-27 09:23:22,103 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
cvat |
cvat | 2022-01-27 09:23:22,104 DEBG 'rqworker_default_0' stderr output:
cvat | DEBUG - 2022-01-27 09:23:22,103 - worker - *** Listening on default...
cvat |
cvat | 2022-01-27 09:23:22,104 DEBG 'rqworker_default_0' stderr output:
cvat | DEBUG - 2022-01-27 09:23:22,104 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
cvat |
cvat_redis | 1:M 27 Jan 09:27:21.116 * 100 changes in 300 seconds. Saving...
cvat_redis | 1:M 27 Jan 09:27:21.117 * Background saving started by pid 15
cvat_redis | 15:C 27 Jan 09:27:21.145 * DB saved on disk
cvat_redis | 15:C 27 Jan 09:27:21.145 * RDB: 0 MB of memory used by copy-on-write
cvat_redis | 1:M 27 Jan 09:27:21.217 * Background saving terminated with success
cvat | 2022-01-27 09:27:44,886 DEBG 'rqworker_low' stderr output:
cvat | DEBUG - 2022-01-27 09:27:44,885 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
cvat |
cvat | 2022-01-27 09:28:57,181 DEBG 'rqworker_default_1' stderr output:
cvat | DEBUG - 2022-01-27 09:28:57,181 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
cvat |
cvat | 2022-01-27 09:29:05,943 DEBG 'runserver' stderr output:
cvat | [Thu Jan 27 09:29:05.943023 2022] [wsgi:error] [pid 743:tid 140638102210304] [client 172.20.0.5:55488] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: https://<my.ip.address>/tasks/create?projectId=1
cvat |
cvat | 2022-01-27 09:29:31,830 DEBG 'rqworker_default_0' stderr output:
cvat | INFO - 2022-01-27 09:29:31,830 - worker - default: cvat.apps.engine.task._create_thread(3, {'chunk_size': None, 'size': 0, 'image_quality': 100, 'start_frame': 0, 'st...) (/api/v1/tasks/3)
cvat |
cvat | 2022-01-27 09:29:31,831 DEBG 'rqworker_default_0' stderr output:
cvat | DEBUG - 2022-01-27 09:29:31,831 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
cvat |
cvat | 2022-01-27 09:29:31,850 DEBG 'rqworker_default_0' stderr output:
cvat | DEBUG - 2022-01-27 09:29:31,848 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 90 seconds.
cvat |
cvat | 2022-01-27 09:29:31,868 DEBG 'rqworker_default_0' stderr output:
cvat | [2022-01-27 09:29:31,867] INFO cvat.server: create task #3
cvat |
cvat | 2022-01-27 09:29:31,868 DEBG 'rqworker_default_0' stderr output:
cvat | INFO - 2022-01-27 09:29:31,867 - task - create task #3
cvat |
cvat | 2022-01-27 09:29:38,156 DEBG 'rqworker_default_0' stderr output:
cvat | [2022-01-27 09:29:38,155] INFO cvat.server: Found frames 10000 for Data #3
cvat |
cvat | 2022-01-27 09:29:38,156 DEBG 'rqworker_default_0' stderr output:
cvat | INFO - 2022-01-27 09:29:38,155 - task - Found frames 10000 for Data #3
cvat |
cvat | 2022-01-27 09:29:38,156 DEBG 'rqworker_default_0' stderr output:
cvat | [2022-01-27 09:29:38,156] INFO cvat.server: New segment for task #3: start_frame = 0, stop_frame = 9999
cvat |
cvat | 2022-01-27 09:29:38,157 DEBG 'rqworker_default_0' stderr output:
cvat | INFO - 2022-01-27 09:29:38,156 - task - New segment for task #3: start_frame = 0, stop_frame = 9999
cvat |
cvat | 2022-01-27 09:29:38,717 DEBG 'rqworker_default_0' stderr output:
cvat | DEBUG - 2022-01-27 09:29:38,716 - worker - Handling successful execution of job /api/v1/tasks/3
cvat |

As you may see in the logs, it tells that the creation has been executed successfully, but that's not the case.

Next steps

You may join our Gitter channel for community support.

[JRGit4UE]

Hi, just asking politely, if there is an update on the issue?

[veriff-javiervgd]

Another user here with very similar problem. Have 10K images. Generated annotations, downloaded them, parsed them, but when try to upload I get an error of file too big:

Failed to load resource: the server responded with a status of 413 (Request Entity Too Large)

cvat-ui.580642541947bec0314e.min.js:393 Error: Request failed with status code 413. "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n<!-- a padding to disable MSIE and Chrome friendly error page -->\r\n<!-- a padding to disable MSIE and Chrome friendly error page -->\r\n<!-- a padding to disable MSIE and Chrome friendly error page -->\r\n<!-- a padding to disable MSIE and Chrome friendly error page -->\r\n<!-- a padding to disable MSIE and Chrome friendly error page -->\r\n<!-- a padding to disable MSIE and Chrome friendly error page -->\r\n".

In my case the file size in my case is ~100MB. Do we have an update on this?

[nmanovic]

@veriff-javiervgd , CVAT got a couple of improvements in the uploading functionality. Please try CVAT v2.0.0. PR #3692

[veriff-javiervgd]

Thank you for your quick response and taking the time to do so. In my case I am just a simple developer who have access to CVATv3 tool, I do not have access to devops tooling and hence I can not deploy the old version 😞

[nmanovic]

@veriff-javiervgd , if you have CVAT v3, please share it with us. The latest official version of CVAT is v2.0.0

[veriff-javiervgd]

hahahaha ok ok, you are right, apparently v3 stands for the "3th try" of the deployment. Sorry for my incompetence, I am new to this. Ok, after researching a bit I found a default max upload file size in django so I am going to try to increase the FILE_UPLOAD_MAX_MEMORY_SIZE in django settings.py. If I am not successful I will try to divide the dataset Task into 100 jobs and will split the annotations file into 100 files so I can halve the size by 100.

[bsekachev]

Did you try the latest version?

[veriff-javiervgd]

Yes, I tried and it worked. When I upload a big CVAT file the UI keep this progress circle animation going on and seems stale but after 5 -10 mins the upload is actually performed so thank you for the effort 🙌🏼

[nmanovic]

@veriff-javiervgd , thanks for the feedback. I will close the issue.