Release v2.12.1 #7811
Merged
Release v2.12.1 #7811
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![cvat-bot[bot]](https://avatars.githubusercontent.com/in/406724?s=60&v=4){kind=small}
Update develop after v2.12.0
…ts (#7669) <!-- Raise an issue to propose your change (https://github.com/opencv/cvat/issues). It helps to avoid duplication of efforts from multiple independent contributors. Discuss your ideas with maintainers to be sure that changes will be approved and merged. Read the [Contribution guide](https://opencv.github.io/cvat/docs/contributing/). --> <!-- Provide a general summary of your changes in the Title above --> ### Motivation and context <!-- Why is this change required? What problem does it solve? If it fixes an open issue, please link to the issue here. Describe your changes in detail, add screenshots. --> Main Issue: #7571 Related Issue: #2339 I've reproduced the issue mentioned in #7571 when exporting and importing annotations using both the Datumaro and Coco 1.0 formats. Specifically, the "Switch outside" attribute isn't being applied as expected. After some investigation, I pinpointed the root cause to be the absence of the "outside" attribute in the exported annotations. To address this, I've made adjustments to the binding.py file to bypass the track_id during annotation import. This modification appears to solve the issue regarding the "Switch outside" attribute. However, it introduces a new concern: the potential loss of information, including keyframes and track_id. While this workaround offers a temporary fix, I'm contemplating a more holistic approach that entails properly handling the "outside" attribute during both the export and import processes of annotations. This method could preserve the integrity of the annotations while ensuring the functionality of the "Switch outside" attribute. I'm reaching out for feedback or suggestions on my proposed solution. Is there a preference for one of these approaches, or might there be another avenue I haven't considered? Looking forward to your insights. ### Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply. If an item isn't applicable for some reason, then ~~explicitly strikethrough~~ the whole line. If you don't do that, GitHub will show incorrect progress for the pull request. If you're unsure about any of these, don't hesitate to ask. We're here to help! --> - [x] I submit my changes into the `develop` branch - [x] I have created a changelog fragment <!-- see top comment in CHANGELOG.md --> - [ ] I have updated the documentation accordingly - [x] I have added tests to cover my changes - [x] I have linked related issues (see [GitHub docs]( https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword)) - [ ] I have increased versions of npm packages if it is necessary ([cvat-canvas](https://github.com/opencv/cvat/tree/develop/cvat-canvas#versioning), [cvat-core](https://github.com/opencv/cvat/tree/develop/cvat-core#versioning), [cvat-data](https://github.com/opencv/cvat/tree/develop/cvat-data#versioning) and [cvat-ui](https://github.com/opencv/cvat/tree/develop/cvat-ui#versioning)) ### License - [x] I submit _my code changes_ under the same [MIT License]( https://github.com/opencv/cvat/blob/develop/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern. --------- Co-authored-by: Maxim Zhiltsov <[email protected]>
<!-- Raise an issue to propose your change (https://github.com/cvat-ai/cvat/issues). It helps to avoid duplication of efforts from multiple independent contributors. Discuss your ideas with maintainers to be sure that changes will be approved and merged. Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/). --> <!-- Provide a general summary of your changes in the Title above --> ### Motivation and context <!-- Why is this change required? What problem does it solve? If it fixes an open issue, please link to the issue here. Describe your changes in detail, add screenshots. --> After analyzing the current implementation it turned out that we evaluate queryset and iterate over them only one time when merging table rows and initializing custom structure for storing objects. It means that generally, we can disable internal Django's default logic for caching querysets. This approach allows us to reduce the amount of memory used when there are a large number of objects. ### How has this been tested? <!-- Please describe in detail how you tested your changes. Include details of your testing environment, and the tests you ran to see how your change affects other areas of the code, etc. --> I've checked the amount of memory used, the number of queries to the database, and the required time. **Without iterator:** Memory usage: 18GB <details> <summary>Profile details</summary> ``` filename: /home/maya/Documents/cvat/cvat/apps/dataset_manager/task.py Line # Mem usage Increment Occurrences Line Contents ============================================================= 116 290.7 MiB 290.7 MiB 1 @Profile 117 def _merge_table_rows(rows, keys_for_merge, field_id): 118 # It is necessary to keep a stable order of original rows 119 # (e.g. for tracked boxes). Otherwise prev_box.frame can be bigger 120 # than next_box.frame. 121 290.7 MiB 0.0 MiB 1 merged_rows = OrderedDict() 122 123 # Group all rows by field_id. In grouped rows replace fields in 124 # accordance with keys_for_merge structure. 125 19053.0 MiB 17140.9 MiB 1721804 for row in rows: #.iterator(): 126 19053.0 MiB -567.5 MiB 1721803 row_id = row[field_id] 127 19053.0 MiB -567.6 MiB 1721803 if not row_id in merged_rows: 128 19053.0 MiB 127.0 MiB 373063 merged_rows[row_id] = dotdict(row) 129 19053.0 MiB -162.9 MiB 746126 for key in keys_for_merge: 130 19053.0 MiB -111.0 MiB 373063 merged_rows[row_id][key] = [] 131 132 19053.0 MiB -1121.3 MiB 3443606 for key in keys_for_merge: 133 19053.0 MiB -2753.5 MiB 10330818 item = dotdict({v.split('__', 1)[-1]:row[v] for v in keys_for_merge[key]}) 134 19053.0 MiB -485.9 MiB 1721803 if item.id is not None: 135 19053.0 MiB -525.5 MiB 1573530 merged_rows[row_id][key].append(item) 136 137 # Remove redundant keys from final objects 138 19053.0 MiB 0.0 MiB 7 redundant_keys = [item for values in keys_for_merge.values() for item in values] 139 19053.0 MiB -5728.4 MiB 373064 for i in merged_rows: 140 19053.0 MiB -22913.7 MiB 1492252 for j in redundant_keys: 141 19053.0 MiB -17185.2 MiB 1119189 del merged_rows[i][j] 142 143 19052.9 MiB -0.0 MiB 1 return list(merged_rows.values()) ``` ``` filename: /home/maya/Documents/cvat/cvat/apps/dataset_manager/task.py Line # Mem usage Increment Occurrences Line Contents ============================================================= 749 290.7 MiB 290.7 MiB 1 @Profile 750 def init_from_db(self): 751 290.7 MiB 0.0 MiB 1 self._init_tags_from_db() 752 19052.3 MiB 18761.7 MiB 1 self._init_shapes_from_db() 753 19052.3 MiB 0.0 MiB 1 self._init_tracks_from_db() 754 19052.3 MiB 0.0 MiB 1 self._init_version_from_db() ``` </details>  **With iterator:** Memory usage: 5.5GB <details> <summary>Profile details</summary> ``` filename: /home/maya/Documents/cvat/cvat/apps/dataset_manager/task.py Line # Mem usage Increment Occurrences Line Contents ============================================================= 116 290.9 MiB 290.9 MiB 1 @Profile 117 def _merge_table_rows(rows, keys_for_merge, field_id): 118 # It is necessary to keep a stable order of original rows 119 # (e.g. for tracked boxes). Otherwise prev_box.frame can be bigger 120 # than next_box.frame. 121 290.9 MiB 0.0 MiB 1 merged_rows = OrderedDict() 122 123 # Group all rows by field_id. In grouped rows replace fields in 124 # accordance with keys_for_merge structure. 125 4345.7 MiB 3783.7 MiB 1721804 for row in rows.iterator(): 126 4345.7 MiB 24.2 MiB 1721803 row_id = row[field_id] 127 4345.7 MiB 0.0 MiB 1721803 if not row_id in merged_rows: 128 4345.7 MiB 78.9 MiB 373063 merged_rows[row_id] = dotdict(row) 129 4345.7 MiB 5.9 MiB 746126 for key in keys_for_merge: 130 4345.7 MiB 0.0 MiB 373063 merged_rows[row_id][key] = [] 131 132 4345.7 MiB 0.3 MiB 3443606 for key in keys_for_merge: 133 4345.7 MiB 152.9 MiB 10330818 item = dotdict({v.split('__', 1)[-1]:row[v] for v in keys_for_merge[key]}) 134 4345.7 MiB 9.0 MiB 1721803 if item.id is not None: 135 4345.7 MiB 0.0 MiB 1573530 merged_rows[row_id][key].append(item) 136 137 # Remove redundant keys from final objects 138 4345.7 MiB 0.0 MiB 7 redundant_keys = [item for values in keys_for_merge.values() for item in values] 139 4345.7 MiB 0.0 MiB 373064 for i in merged_rows: 140 4345.7 MiB 0.0 MiB 1492252 for j in redundant_keys: 141 4345.7 MiB 0.0 MiB 1119189 del merged_rows[i][j] 142 143 4348.5 MiB 2.8 MiB 1 return list(merged_rows.values()) ``` ``` Filename: /home/maya/Documents/cvat/cvat/apps/dataset_manager/task.py Line # Mem usage Increment Occurrences Line Contents ============================================================= 590 290.9 MiB 290.9 MiB 1 @Profile 591 def _init_shapes_from_db(self): 592 290.9 MiB 0.0 MiB 3 db_shapes = self.db_job.labeledshape_set.prefetch_related( 593 290.9 MiB 0.0 MiB 1 "label", 594 290.9 MiB 0.0 MiB 1 "labeledshapeattributeval_set" 595 290.9 MiB 0.0 MiB 2 ).values( 596 290.9 MiB 0.0 MiB 1 'id', 597 290.9 MiB 0.0 MiB 1 'label_id', 598 290.9 MiB 0.0 MiB 1 'type', 599 290.9 MiB 0.0 MiB 1 'frame', 600 290.9 MiB 0.0 MiB 1 'group', 601 290.9 MiB 0.0 MiB 1 'source', 602 290.9 MiB 0.0 MiB 1 'occluded', 603 290.9 MiB 0.0 MiB 1 'outside', 604 290.9 MiB 0.0 MiB 1 'z_order', 605 290.9 MiB 0.0 MiB 1 'rotation', 606 290.9 MiB 0.0 MiB 1 'points', 607 290.9 MiB 0.0 MiB 1 'parent', 608 290.9 MiB 0.0 MiB 1 'labeledshapeattributeval__spec_id', 609 290.9 MiB 0.0 MiB 1 'labeledshapeattributeval__value', 610 290.9 MiB 0.0 MiB 1 'labeledshapeattributeval__id', 611 290.9 MiB 0.0 MiB 1 ).order_by('frame') 618 4328.6 MiB 4037.7 MiB 2 db_shapes = _merge_table_rows( 619 290.9 MiB 0.0 MiB 1 rows=db_shapes, 620 290.9 MiB 0.0 MiB 1 keys_for_merge={ 621 290.9 MiB 0.0 MiB 1 'labeledshapeattributeval_set': [ 622 'labeledshapeattributeval__spec_id', 623 'labeledshapeattributeval__value', 624 'labeledshapeattributeval__id', 625 ], 626 }, 627 290.9 MiB 0.0 MiB 1 field_id='id', 628 ) 649 4385.6 MiB 0.0 MiB 1 serializer = serializers.LabeledShapeSerializerFromDB(list(shapes.values()), many=True) 650 5990.2 MiB 1604.6 MiB 1 self.ir_data.shapes = serializer.data ``` ``` Filename: /home/maya/Documents/cvat/cvat/apps/dataset_manager/task.py Line # Mem usage Increment Occurrences Line Contents ============================================================= 749 290.6 MiB 290.6 MiB 1 @Profile 750 def init_from_db(self): 751 290.9 MiB 0.3 MiB 1 self._init_tags_from_db() 752 5990.2 MiB 5699.3 MiB 1 self._init_shapes_from_db() 753 5990.2 MiB 0.0 MiB 1 self._init_tracks_from_db() 754 5990.2 MiB 0.0 MiB 1 self._init_version_from_db() ``` </details>  ### Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply. If an item isn't applicable for some reason, then ~~explicitly strikethrough~~ the whole line. If you don't do that, GitHub will show incorrect progress for the pull request. If you're unsure about any of these, don't hesitate to ask. We're here to help! --> - [x] I submit my changes into the `develop` branch - [x] I have created a changelog fragment <!-- see top comment in CHANGELOG.md --> ~~- [ ] I have updated the documentation accordingly~~ ~~- [ ] I have added tests to cover my changes~~ ~~- [ ] I have linked related issues (see [GitHub docs]( https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))~~ ~~- [ ] I have increased versions of npm packages if it is necessary ([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning), [cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning), [cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning) and [cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))~~ ### License - [x] I submit _my code changes_ under the same [MIT License]( https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern.
There are three changes here: * Don't save the bundle to a file. There seems to be no reason to do that; the bundle is very small (<8000 bytes), and we already load it fully into memory to generate the ETag, so just generate it in memory and keep it there. This also fixes a possible race condition where a server process starts to read the bundle file right as another server process starts up and overwrites it. * Generate the bundle on-demand rather than at startup. This obviates the need for the `IAM_OPA_BUNDLE` environment variable. * Generate the ETag once, instead of on every request. There's no reason to hash the same bundle over and over.
…7734) This improves the dependency graph of apps with regards to IAM and other apps. IAM is a generic app, so it should not depend on specific apps, like `webhooks`. It's also easier to work with a bunch of smaller files rather than one giant one. Having permissions classes in separate modules introduces the risk that they will never be loaded directly. To mitigate that, add a function to ensure permissions are loaded and call it from every app. In a future PR, I plan to move the `.rego` files into their respective apps, as well.
In many cases what is actually meant is "authenticated"/"authentication". This patch fixes such mistakes across the codebase.
<!-- Raise an issue to propose your change (https://github.com/cvat-ai/cvat/issues). It helps to avoid duplication of efforts from multiple independent contributors. Discuss your ideas with maintainers to be sure that changes will be approved and merged. Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/). --> <!-- Provide a general summary of your changes in the Title above --> ### Motivation and context <!-- Why is this change required? What problem does it solve? If it fixes an open issue, please link to the issue here. Describe your changes in detail, add screenshots. --> This PR adds logs for further invistigation of `DatasetNotFound` error which frequently appears in case of importing datasets in various formats ### How has this been tested? <!-- Please describe in detail how you tested your changes. Include details of your testing environment, and the tests you ran to see how your change affects other areas of the code, etc. --> ### Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply. If an item isn't applicable for some reason, then ~~explicitly strikethrough~~ the whole line. If you don't do that, GitHub will show incorrect progress for the pull request. If you're unsure about any of these, don't hesitate to ask. We're here to help! --> - [x] I submit my changes into the `develop` branch - ~~[ ] I have created a changelog fragment <!-- see top comment in CHANGELOG.md -->~~ - ~~[ ] I have updated the documentation accordingly~~ - ~~[ ] I have added tests to cover my changes~~ - ~~[ ] I have linked related issues (see [GitHub docs]( https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))~~ - ~~[ ] I have increased versions of npm packages if it is necessary ([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning), [cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning), [cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning) and [cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))~~ ### License - [x] I submit _my code changes_ under the same [MIT License]( https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added detailed error logging for dataset import errors to enhance troubleshooting and user feedback. - **Enhancements** - Introduced a new logging configuration for handling dataset import errors, including file location and formatting details. - **Configuration Changes** - Added an environment variable `CVAT_LOG_IMPORT_ERRORS` to control the logging of dataset import errors, set to `'true'` by default in the Docker configuration. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
cvat-bot
bot
requested review from
azhavoro,
zhiltsov-max and
bsekachev
as code owners
cvat-bot
bot
requested review from
mdacoca,
SpecLad,
Marishka17 and
nmanovic
as code owners
|
Important Auto Review SkippedBot user detected. To trigger a single review, invoke the You can disable this status message by setting the Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixed
Formats with the custom
track_idattribute should importoutsidetrack shapes properly (e.g.COCO,COCO Keypoints,Datumaro,PASCAL VOC) ([GSoC2024] Fix import of outside track shapes in Datumaro-based formats #7669)Inefficient resources fetching in admin panel leading to 504 Gateway Timeout (Fixed 504 gateway timeout on admin page #7767)
Optimized memory usage when retrieving annotations by disabling internal Django QuerySet caching (Optimize the code for retrieving annotations #7748)
Annotations are not shown on the
0frame sometimes (Fixed showing annotations at 0 frame #7796)Extra requests in PolicyEnforcer when at least one policy is rejected, others are not checked (Disable further checking permissions if at least one is False #7803)
Project's
updated_datewas not updated after changing annotations in jobs (Update project's on save annotations #7808)