Auto annotation GPU support

README.md

@@ -61,16 +61,16 @@ via its command line tool and Python library.
 
 ## Deep learning models for automatic labeling
 
-| Name                                                                                                    | Type       | Framework  |
-| ------------------------------------------------------------------------------------------------------- | ---------- | ---------- |
-| [Deep Extreme Cut](/serverless/openvino/dextr/nuclio)                                                   | interactor | OpenVINO   |
-| [Faster RCNN](/serverless/tensorflow/faster_rcnn_inception_v2_coco/nuclio)                              | detector   | TensorFlow |
-| [Mask RCNN](/serverless/openvino/omz/public/mask_rcnn_inception_resnet_v2_atrous_coco/nuclio)           | detector   | OpenVINO   |
-| [YOLO v3](/serverless/openvino/omz/public/yolo-v3-tf/nuclio)                                            | detector   | OpenVINO   |
-| [Text detection v4](/serverless/openvino/omz/intel/text-detection-0004/nuclio)                          | detector   | OpenVINO   |
-| [Semantic segmentation for ADAS](/serverless/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio) | detector   | OpenVINO   |
-| [Mask RCNN](/serverless/tensorflow/matterport/mask_rcnn/nuclio)                                         | detector   | TensorFlow |
-| [Object reidentification](/serverless/openvino/omz/intel/person-reidentification-retail-300/nuclio)     | reid       | OpenVINO   |
+| Name                                                                                                    | Type       | Framework  | CPU | GPU |
+| ------------------------------------------------------------------------------------------------------- | ---------- | ---------- | --- | --- |
+| [Deep Extreme Cut](/serverless/openvino/dextr/nuclio)                                                   | interactor | OpenVINO   | X   |
+| [Faster RCNN](/serverless/tensorflow/faster_rcnn_inception_v2_coco/nuclio)                              | detector   | TensorFlow | X   | X   |
+| [Mask RCNN](/serverless/openvino/omz/public/mask_rcnn_inception_resnet_v2_atrous_coco/nuclio)           | detector   | OpenVINO   | X   |
+| [YOLO v3](/serverless/openvino/omz/public/yolo-v3-tf/nuclio)                                            | detector   | OpenVINO   | X   |
+| [Text detection v4](/serverless/openvino/omz/intel/text-detection-0004/nuclio)                          | detector   | OpenVINO   | X   |
+| [Semantic segmentation for ADAS](/serverless/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio) | detector   | OpenVINO   | X   |
+| [Mask RCNN](/serverless/tensorflow/matterport/mask_rcnn/nuclio)                                         | detector   | TensorFlow | X   |
+| [Object reidentification](/serverless/openvino/omz/intel/person-reidentification-retail-300/nuclio)     | reid       | OpenVINO   | X   |
 
 ## Online demo: [cvat.org](https://cvat.org)
 

components/serverless/docker-compose.serverless.yml

@@ -2,7 +2,7 @@ version: '3.3'
 services:
   serverless:
     container_name: nuclio
-    image: quay.io/nuclio/dashboard:1.4.8-amd64
+    image: quay.io/nuclio/dashboard:1.5.7-amd64
     restart: always
     networks:
       default:

cvat/apps/documentation/installation.md

@@ -290,32 +290,7 @@ docker-compose -f docker-compose.yml -f components/analytics/docker-compose.anal
 
 ### Semi-automatic and automatic annotation
 
-- You have to install `nuctl` command line tool to build and deploy serverless
-  functions. Download [the latest release](https://github.com/nuclio/nuclio/releases).
-- Create `cvat` project inside nuclio dashboard where you will deploy new
-  serverless functions and deploy a couple of DL models. Commands below should
-  be run only after CVAT has been installed using docker-compose because it
-  runs nuclio dashboard which manages all serverless functions.
-
-```bash
-nuctl create project cvat
-```
-
-```bash
-nuctl deploy --project-name cvat \
-    --path serverless/openvino/dextr/nuclio \
-    --volume `pwd`/serverless/openvino/common:/opt/nuclio/common \
-    --platform local
-```
-
-```bash
-nuctl deploy --project-name cvat \
-    --path serverless/openvino/omz/public/yolo-v3-tf/nuclio \
-    --volume `pwd`/serverless/openvino/common:/opt/nuclio/common \
-    --platform local
-```
-
-Note: see [deploy.sh](/serverless/deploy.sh) script for more examples.
+Please follow [instructions](/cvat/apps/documentation/installation_automatic_annotation.md)
 
 ### Stop all containers
 

cvat/apps/documentation/installation_automatic_annotation.md

@@ -0,0 +1,84 @@
+
+### Semi-automatic and automatic annotation
+
+- To bring up cvat with auto annotation tool, **do not use** `docker-compose up`.If you did first make sure all containers are stopped `docker-compose down`
+
+
+  From cvat root directory, you need to run:
+  ```bash
+  docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d
+  ```
+  If you did any changes to the docker-compose files, make sure to add `--build` at the end.
+
+  To stop the containers, simply run:
+
+  ```bash
+  docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml down
+  ```
+
+
+- You have to install `nuctl` command line tool to build and deploy serverless
+  functions. Download [version 1.5.8](https://github.com/nuclio/nuclio/releases).
+  It is important that the version you download matches the version in
+  [docker-compose.serverless.yml](/components/serverless/docker-compose.serverless.yml)
+  when you downloaded the nuclio give them proper permission and do a softlin
+  ```
+  sudo chmod +x nuctl-<version>-linux-amd64
+  sudo ln -sf $(pwd)/nuctl-<version>-linux-amd64 /usr/local/bin/nuctl
+  ```
+
+- Create `cvat` project inside nuclio dashboard where you will deploy new
+  serverless functions and deploy a couple of DL models. Commands below should
+  be run only after CVAT has been installed using docker-compose because it
+  runs nuclio dashboard which manages all serverless functions.
+
+  ```bash
+  nuctl create project cvat
+  ```
+
+  ```bash
+  nuctl deploy --project-name cvat \
+    --path serverless/openvino/dextr/nuclio \
+    --volume `pwd`/serverless/openvino/common:/opt/nuclio/common \
+    --platform local
+  ```
+
+  ```bash
+  nuctl deploy --project-name cvat \
+    --path serverless/openvino/omz/public/yolo-v3-tf/nuclio \
+    --volume `pwd`/serverless/openvino/common:/opt/nuclio/common \
+    --platform local
+  ```
+
+
+  If your function is running on GPU, you should add `--resource-limit nvidia.com/gpu=1` to the above command or, alternatively, add gpu resources dircetly into the function.yaml see [tensorflow-fast-rcnn-gpu](../../../serverless/tensorflow/
+  faster_rcnn_inception_v2_coco_gpu/nuclio/function.yaml)
+
+    - Note: see [deploy.sh](/serverless/deploy.sh) script for more examples.
+
+####Debugging:
+
+- You can open nuclio dashboard at [localhost:8070](http://localhost:8070). Make sure status of your functions are up and running without any error.
+
+
+- To check for internal server errors, run `docker ps -a` to see the list of containers. Find the container that you are interested, e.g. `nuclio-nuclio-tf-faster-rcnn-inception-v2-coco-gpu`. Then check its logs by
+
+  ```bash
+  docker logs <name of your container>
+  ```
+  e.g.,
+
+  ```bash
+  docker logs nuclio-nuclio-tf-faster-rcnn-inception-v2-coco-gpu
+  ```
+
+
+- If you would like to debug a code inside a container, you can use vscode to directly attach to a container [instructions](https://code.visualstudio.com/docs/remote/attach-container). To apply changes, makse sure to restart the container.
+  ```bash
+  docker stop <name of the container>
+  ```
+  and then
+  ```bash
+  docker start <name of the container>
+  ```
+  Do not use nuclio dashboard to stop the container since with any change, it rebuilds the container and you'll lose your changes.

serverless/deploy.sh

@@ -46,6 +46,10 @@ nuctl deploy --project-name cvat \
     --path "$SCRIPT_DIR/tensorflow/faster_rcnn_inception_v2_coco/nuclio" \
     --platform local
 
+nuctl deploy --project-name cvat \
+    --path "$SCRIPT_DIR/tensorflow/faster_rcnn_inception_v2_coco_gpu/nuclio" \
+    --platform local
+
 nuctl deploy --project-name cvat \
     --path "$SCRIPT_DIR/pytorch/foolwood/siammask/nuclio" \
     --platform local

serverless/tensorflow/faster_rcnn_inception_v2_coco/nuclio/model_loader.py

@@ -35,7 +35,8 @@ def infer(self, image):
         width, height = image.size
         if width > 1920 or height > 1080:
             image = image.resize((width // 2, height // 2), Image.ANTIALIAS)
-        image_np = np.array(image.getdata()).reshape((image.height, image.width, 3)).astype(np.uint8)
+        image_np = np.array(image.getdata())[:, :3].reshape(
+            (image.height, image.width, -1)).astype(np.uint8)
         image_np = np.expand_dims(image_np, axis=0)
 
         return self.session.run(

serverless/tensorflow/faster_rcnn_inception_v2_coco_gpu/nuclio/function.yaml

@@ -0,0 +1,134 @@
+metadata:
+  name: tf-faster-rcnn-inception-v2-coco-gpu
+  namespace: cvat
+  annotations:
+    name: Faster RCNN via Tensorflow GPU
+    type: detector
+    framework: tensorflow
+    spec: |
+      [
+        { "id": 1, "name": "person" },
+        { "id": 2, "name": "bicycle" },
+        { "id": 3, "name": "car" },
+        { "id": 4, "name": "motorcycle" },
+        { "id": 5, "name": "airplane" },
+        { "id": 6, "name": "bus" },
+        { "id": 7, "name": "train" },
+        { "id": 8, "name": "truck" },
+        { "id": 9, "name": "boat" },
+        { "id":10, "name": "traffic_light" },
+        { "id":11, "name": "fire_hydrant" },
+        { "id":13, "name": "stop_sign" },
+        { "id":14, "name": "parking_meter" },
+        { "id":15, "name": "bench" },
+        { "id":16, "name": "bird" },
+        { "id":17, "name": "cat" },
+        { "id":18, "name": "dog" },
+        { "id":19, "name": "horse" },
+        { "id":20, "name": "sheep" },
+        { "id":21, "name": "cow" },
+        { "id":22, "name": "elephant" },
+        { "id":23, "name": "bear" },
+        { "id":24, "name": "zebra" },
+        { "id":25, "name": "giraffe" },
+        { "id":27, "name": "backpack" },
+        { "id":28, "name": "umbrella" },
+        { "id":31, "name": "handbag" },
+        { "id":32, "name": "tie" },
+        { "id":33, "name": "suitcase" },
+        { "id":34, "name": "frisbee" },
+        { "id":35, "name": "skis" },
+        { "id":36, "name": "snowboard" },
+        { "id":37, "name": "sports_ball" },
+        { "id":38, "name": "kite" },
+        { "id":39, "name": "baseball_bat" },
+        { "id":40, "name": "baseball_glove" },
+        { "id":41, "name": "skateboard" },
+        { "id":42, "name": "surfboard" },
+        { "id":43, "name": "tennis_racket" },
+        { "id":44, "name": "bottle" },
+        { "id":46, "name": "wine_glass" },
+        { "id":47, "name": "cup" },
+        { "id":48, "name": "fork" },
+        { "id":49, "name": "knife" },
+        { "id":50, "name": "spoon" },
+        { "id":51, "name": "bowl" },
+        { "id":52, "name": "banana" },
+        { "id":53, "name": "apple" },
+        { "id":54, "name": "sandwich" },
+        { "id":55, "name": "orange" },
+        { "id":56, "name": "broccoli" },
+        { "id":57, "name": "carrot" },
+        { "id":58, "name": "hot_dog" },
+        { "id":59, "name": "pizza" },
+        { "id":60, "name": "donut" },
+        { "id":61, "name": "cake" },
+        { "id":62, "name": "chair" },
+        { "id":63, "name": "couch" },
+        { "id":64, "name": "potted_plant" },
+        { "id":65, "name": "bed" },
+        { "id":67, "name": "dining_table" },
+        { "id":70, "name": "toilet" },
+        { "id":72, "name": "tv" },
+        { "id":73, "name": "laptop" },
+        { "id":74, "name": "mouse" },
+        { "id":75, "name": "remote" },
+        { "id":76, "name": "keyboard" },
+        { "id":77, "name": "cell_phone" },
+        { "id":78, "name": "microwave" },
+        { "id":79, "name": "oven" },
+        { "id":80, "name": "toaster" },
+        { "id":81, "name": "sink" },
+        { "id":83, "name": "refrigerator" },
+        { "id":84, "name": "book" },
+        { "id":85, "name": "clock" },
+        { "id":86, "name": "vase" },
+        { "id":87, "name": "scissors" },
+        { "id":88, "name": "teddy_bear" },
+        { "id":89, "name": "hair_drier" },
+        { "id":90, "name": "toothbrush" }
+      ]
+
+spec:
+  description: Faster RCNN from Tensorflow Object Detection GPU API
+  runtime: 'python:3.6'
+  handler: main:handler
+  eventTimeout: 30s
+
+  build:
+    image: cvat/tf.faster_rcnn_inception_v2_coco_gpu
+    baseImage: tensorflow/tensorflow:2.1.1-gpu
+
+    directives:
+      preCopy:
+        - kind: RUN
+          value: apt install curl
+        - kind: WORKDIR
+          value: /opt/nuclio
+
+      postCopy:
+        - kind: RUN
+          value: curl -O http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_v2_coco_2018_01_28.tar.gz
+        - kind: RUN
+          value: tar -xzf faster_rcnn_inception_v2_coco_2018_01_28.tar.gz && rm faster_rcnn_inception_v2_coco_2018_01_28.tar.gz
+        - kind: RUN
+          value: ln -s faster_rcnn_inception_v2_coco_2018_01_28 faster_rcnn
+        - kind: RUN
+          value: pip install pillow pyyaml
+  resources:
+    limits:
+      nvidia.com/gpu: "1"
+
+  triggers:
+    myHttpTrigger:
+      maxWorkers: 2
+      kind: 'http'
+      workerAvailabilityTimeoutMilliseconds: 10000
+      attributes:
+        maxRequestBodySize: 33554432 # 32MB
+
+  platform:
+    attributes:
+      restartPolicy:
+        name: always
+        maximumRetryCount: 3

serverless/tensorflow/faster_rcnn_inception_v2_coco_gpu/nuclio/main.py

@@ -0,0 +1,48 @@
+import json
+import base64
+import io
+from PIL import Image
+import yaml
+from model_loader import ModelLoader
+
+
+def init_context(context):
+    context.logger.info("Init context...  0%")
+    model_path = "/opt/nuclio/faster_rcnn/frozen_inference_graph.pb"
+    model_handler = ModelLoader(model_path)
+    setattr(context.user_data, 'model_handler', model_handler)
+    functionconfig = yaml.safe_load(open("/opt/nuclio/function.yaml"))
+    labels_spec = functionconfig['metadata']['annotations']['spec']
+    labels = {item['id']: item['name'] for item in json.loads(labels_spec)}
+    setattr(context.user_data, "labels", labels)
+    context.logger.info("Init context...100%")
+
+def handler(context, event):
+    context.logger.info("Run faster_rcnn_inception_v2_coco model")
+    data = event.body
+    buf = io.BytesIO(base64.b64decode(data["image"].encode('utf-8')))
+    threshold = float(data.get("threshold", 0.5))
+    image = Image.open(buf)
+
+    (boxes, scores, classes, num_detections) = context.user_data.model_handler.infer(image)
+
+    results = []
+    for i in range(int(num_detections[0])):
+        obj_class = int(classes[0][i])
+        obj_score = scores[0][i]
+        obj_label = context.user_data.labels.get(obj_class, "unknown")
+        if obj_score >= threshold:
+            xtl = boxes[0][i][1] * image.width
+            ytl = boxes[0][i][0] * image.height
+            xbr = boxes[0][i][3] * image.width
+            ybr = boxes[0][i][2] * image.height
+
+            results.append({
+                "confidence": str(obj_score),
+                "label": obj_label,
+                "points": [xtl, ytl, xbr, ybr],
+                "type": "rectangle",
+            })
+
+    return context.Response(body=json.dumps(results), headers={},
+        content_type='application/json', status_code=200)