[GSoC2024] Fix import of outside track shapes in Datumaro-based forma…

…ts (#7669)

<!-- Raise an issue to propose your change
(https://github.com/opencv/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution
guide](https://opencv.github.io/cvat/docs/contributing/). -->

<!-- Provide a general summary of your changes in the Title above -->

### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->

Main Issue: #7571 
Related Issue: #2339 

I've reproduced the issue mentioned in #7571 when exporting and
importing annotations using both the Datumaro and Coco 1.0 formats.
Specifically, the "Switch outside" attribute isn't being applied as
expected. After some investigation, I pinpointed the root cause to be
the absence of the "outside" attribute in the exported annotations.

To address this, I've made adjustments to the binding.py file to bypass
the track_id during annotation import. This modification appears to
solve the issue regarding the "Switch outside" attribute. However, it
introduces a new concern: the potential loss of information, including
keyframes and track_id.

While this workaround offers a temporary fix, I'm contemplating a more
holistic approach that entails properly handling the "outside" attribute
during both the export and import processes of annotations. This method
could preserve the integrity of the annotations while ensuring the
functionality of the "Switch outside" attribute.

I'm reaching out for feedback or suggestions on my proposed solution. Is
there a preference for one of these approaches, or might there be
another avenue I haven't considered?

Looking forward to your insights.

### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [x] I have added tests to cover my changes
- [x] I have linked related issues (see [GitHub docs](

https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary

([cvat-canvas](https://github.com/opencv/cvat/tree/develop/cvat-canvas#versioning),

[cvat-core](https://github.com/opencv/cvat/tree/develop/cvat-core#versioning),

[cvat-data](https://github.com/opencv/cvat/tree/develop/cvat-data#versioning)
and

[cvat-ui](https://github.com/opencv/cvat/tree/develop/cvat-ui#versioning))

### License

- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/opencv/cvat/blob/develop/LICENSE) that covers the
project.
  Feel free to contact the maintainers if that's a concern.

---------

Co-authored-by: Maxim Zhiltsov <[email protected]>

changelog.d/20240403_135916_yeek020407_switch_outside.md

@@ -0,0 +1,4 @@
+### Fixed
+
+- Formats with the custom `track_id` attribute should import `outside` track shapes properly (e.g. `COCO`, `COCO Keypoints`, `Datumaro`, `PASCAL VOC`)
+  (<https://github.com/opencv/cvat/pull/7669>)

cvat/apps/dataset_manager/bindings.py

@@ -15,6 +15,7 @@
 from typing import (Any, Callable, DefaultDict, Dict, Iterable, List, Literal, Mapping,
                     NamedTuple, Optional, OrderedDict, Sequence, Set, Tuple, Union)
 
+from attrs.converters import to_bool
 import datumaro as dm
 import defusedxml.ElementTree as ET
 import numpy as np
@@ -1953,7 +1954,8 @@ def import_dm_annotations(dm_dataset: dm.Dataset, instance_data: Union[ProjectDa
         'sly_pointcloud',
         'coco',
         'coco_instances',
-        'coco_person_keypoints'
+        'coco_person_keypoints',
+        'voc'
     ]
 
     label_cat = dm_dataset.categories()[dm.AnnotationType.label]
@@ -2031,9 +2033,9 @@ def reduce_fn(acc, v):
                     # because in some formats return type can be different
                     # from bool / None
                     # https://github.com/openvinotoolkit/datumaro/issues/719
-                    occluded = dm.util.cast(ann.attributes.pop('occluded', None), bool) is True
-                    keyframe = dm.util.cast(ann.attributes.get('keyframe', None), bool) is True
-                    outside = dm.util.cast(ann.attributes.pop('outside', None), bool) is True
+                    occluded = dm.util.cast(ann.attributes.pop('occluded', None), to_bool) is True
+                    keyframe = dm.util.cast(ann.attributes.get('keyframe', None), to_bool) is True
+                    outside = dm.util.cast(ann.attributes.pop('outside', None), to_bool) is True
 
                     track_id = ann.attributes.pop('track_id', None)
                     source = ann.attributes.pop('source').lower() \
@@ -2080,7 +2082,7 @@ def reduce_fn(acc, v):
                         ))
                         continue
 
-                    if keyframe or outside:
+                    if dm_dataset.format in track_formats:
                         if track_id not in tracks:
                             tracks[track_id] = {
                                 'label': label_cat.items[ann.label].name,
@@ -2107,11 +2109,8 @@ def reduce_fn(acc, v):
 
                         if ann.type == dm.AnnotationType.skeleton:
                             for element in ann.elements:
-                                element_keyframe = dm.util.cast(element.attributes.get('keyframe', None), bool, True)
                                 element_occluded = element.visibility[0] == dm.Points.Visibility.hidden
                                 element_outside = element.visibility[0] == dm.Points.Visibility.absent
-                                if not element_keyframe and not element_outside:
-                                    continue
 
                                 if element.label not in tracks[track_id]['elements']:
                                     tracks[track_id]['elements'][element.label] = instance_data.Track(
@@ -2120,6 +2119,7 @@ def reduce_fn(acc, v):
                                         source=source,
                                         shapes=[],
                                     )
+
                                 element_attributes = [
                                     instance_data.Attribute(name=n, value=str(v))
                                     for n, v in element.attributes.items()
@@ -2151,10 +2151,54 @@ def reduce_fn(acc, v):
                 raise CvatImportError("Image {}: can't import annotation "
                     "#{} ({}): {}".format(item.id, idx, ann.type.name, e)) from e
 
-    for track in tracks.values():
-        track['elements'] = list(track['elements'].values())
-        instance_data.add_track(instance_data.Track(**track))
-
+    def _validate_track_shapes(shapes):
+        shapes = sorted(shapes, key=lambda t: t.frame)
+        new_shapes = []
+        prev_shape = None
+        # infer the keyframe shapes and keep only them
+        for shape in shapes:
+            prev_is_visible = prev_shape and not prev_shape.outside
+            cur_is_visible = shape and not shape.outside
+
+            has_gap = False
+            if prev_is_visible:
+                has_gap = prev_shape.frame + instance_data.frame_step < shape.frame
+
+            if has_gap:
+                prev_shape = prev_shape._replace(outside=True, keyframe=True,
+                    frame=prev_shape.frame + instance_data.frame_step)
+                new_shapes.append(prev_shape)
+
+            if prev_is_visible != cur_is_visible or cur_is_visible and (has_gap or shape.keyframe):
+                shape = shape._replace(keyframe=True)
+                new_shapes.append(shape)
+
+            prev_shape = shape
+
+        if prev_shape and not prev_shape.outside and (
+            prev_shape.frame + instance_data.frame_step <= stop_frame
+            # has a gap before the current instance segment end
+        ):
+            prev_shape = prev_shape._replace(outside=True, keyframe=True,
+                frame=prev_shape.frame + instance_data.frame_step)
+            new_shapes.append(prev_shape)
+
+        return new_shapes
+
+    stop_frame = int(instance_data.meta[instance_data.META_FIELD]['stop_frame'])
+    for track_id, track in tracks.items():
+        track['shapes'] = _validate_track_shapes(track['shapes'])
+
+        if ann.type == dm.AnnotationType.skeleton:
+            new_elements = {}
+            for element_id, element in track['elements'].items():
+                new_element_shapes = _validate_track_shapes(element.shapes)
+                new_elements[element_id] = element._replace(shapes=new_element_shapes)
+            track['elements'] = new_elements
+
+        if track['shapes'] or track['elements']:
+            track['elements'] = list(track['elements'].values())
+            instance_data.add_track(instance_data.Track(**track))
 
 def import_labels_to_project(project_annotation, dataset: dm.Dataset):
     labels = []