Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Spvcnn backbone #2320

Merged
merged 93 commits into from
Mar 28, 2023
Merged
Changes from 3 commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
b275217
add cylindrical voxelization & voxel feature encoder
sunjiahao1999 Jan 18, 2023
8c08f98
add cylindrical voxelization & voxel feature encoder
sunjiahao1999 Jan 18, 2023
ea4116b
add voxel-wise label & voxelization UT
sunjiahao1999 Feb 3, 2023
532d0f9
fix vfe
sunjiahao1999 Feb 3, 2023
ca3d097
fix vfe UT
sunjiahao1999 Feb 3, 2023
85416ce
rename voxel encoder & add more test case
sunjiahao1999 Feb 7, 2023
460a002
Merge branch 'dev-1.x' into cylinder_voxel
sunjiahao1999 Feb 7, 2023
a693b8b
fix type hint
sunjiahao1999 Feb 7, 2023
56b554d
temporarily refactoring mmcv's voxelize and dynamic in mmdet3d for da…
sunjiahao1999 Feb 7, 2023
953085d
_forward
sunjiahao1999 Feb 14, 2023
aee48b9
del checkpoints
sunjiahao1999 Feb 14, 2023
d5e835e
add if tp
sunjiahao1999 Feb 14, 2023
d4c96b2
add predict
sunjiahao1999 Feb 16, 2023
20a6eb4
fix vfe init bug & fix UT
sunjiahao1999 Feb 16, 2023
f0b7599
add grid_size & move voxelization code
sunjiahao1999 Feb 16, 2023
f8a310d
fix import bug
sunjiahao1999 Feb 16, 2023
42be1af
keep radian to follow origin
sunjiahao1999 Feb 16, 2023
9b5ac39
add doc string
sunjiahao1999 Feb 17, 2023
7715601
fix type hint
sunjiahao1999 Feb 17, 2023
546ddac
Merge branch 'cylinder_voxel' into minkunet
sunjiahao1999 Feb 17, 2023
def76bd
add minkunet voxelization and loss function
sunjiahao1999 Feb 17, 2023
c54ef5d
fix data
sunjiahao1999 Feb 17, 2023
6539d04
Merge branch 'dev-1.x' into minkunet
sunjiahao1999 Feb 17, 2023
8edd1c5
Merge branch 'dev-1.x' into minkunet 2
sunjiahao1999 Feb 17, 2023
cc0ea24
init train
sunjiahao1999 Feb 19, 2023
4865bc0
Merge branch 'minkunet_train' of https://github.com/sunjiahao1999/mmd…
sunjiahao1999 Feb 19, 2023
9ceb10c
fix sparsetensor typehint
sunjiahao1999 Feb 19, 2023
af36c48
Merge branch 'minkunet_train' of https://github.com/sunjiahao1999/mmd…
sunjiahao1999 Feb 19, 2023
84d4e99
rename dir
sunjiahao1999 Feb 19, 2023
84beaf2
Merge branch 'minkunet_train' of https://github.com/sunjiahao1999/mmd…
sunjiahao1999 Feb 19, 2023
3d1967f
fix data config
sunjiahao1999 Feb 19, 2023
abbb30b
Merge branch 'minkunet_train' of https://github.com/sunjiahao1999/mmd…
sunjiahao1999 Feb 19, 2023
7608b07
fix data config
sunjiahao1999 Feb 19, 2023
6f0116f
Merge branch 'minkunet_train' of https://github.com/sunjiahao1999/mmd…
sunjiahao1999 Feb 19, 2023
64546c9
fix batch_size & replace dynamic_scatter
sunjiahao1999 Feb 21, 2023
f5bdf1e
fix confilcts
sunjiahao1999 Feb 21, 2023
1896b30
fix conflicts 2
sunjiahao1999 Feb 21, 2023
d787a39
fix conflicts on s_70
sunjiahao1999 Feb 21, 2023
8db57c2
Alignment of the original implementation
sunjiahao1999 Feb 27, 2023
0e53760
rename config
sunjiahao1999 Feb 27, 2023
bd73a54
add worker_init_fn_hook
sunjiahao1999 Feb 27, 2023
2e002fd
remove test_config & worker hook
sunjiahao1999 Feb 27, 2023
df91ff7
Merge branch 'dev-1.x' into minkunet
sunjiahao1999 Feb 27, 2023
883586a
add UT
sunjiahao1999 Feb 27, 2023
8a629f5
fix polarmix UT
sunjiahao1999 Feb 27, 2023
56b9e2b
init spcvnn backbone
sunjiahao1999 Feb 28, 2023
10bdefe
Merge branch 'dev-1.x' of github.com:open-mmlab/mmdetection3d into de…
sunjiahao1999 Feb 28, 2023
55bffc5
add seed for cr0p5
sunjiahao1999 Mar 1, 2023
3317d67
Merge branch 'minkunet_train' into minkunet
sunjiahao1999 Mar 1, 2023
963d0e3
spvcnn_init
sunjiahao1999 Mar 1, 2023
e66e5a7
Merge branch 'dev-1.x' of github.com:open-mmlab/mmdetection3d into de…
sunjiahao1999 Mar 1, 2023
06ad676
format
sunjiahao1999 Mar 1, 2023
5223d96
rename SemanticKittiDataset
sunjiahao1999 Mar 2, 2023
372ecba
Merge branch 'dev-1.x' of github.com:open-mmlab/mmdetection3d into de…
sunjiahao1999 Mar 6, 2023
44d1e6f
merge from dev-1.x
sunjiahao1999 Mar 6, 2023
9f6fc1c
add platte & fix visual bug
sunjiahao1999 Mar 6, 2023
7ccd1f9
add platte & fix data info bug
sunjiahao1999 Mar 7, 2023
8c91004
Merge branch 'minkunet' into spvcnn
sunjiahao1999 Mar 7, 2023
8d8864e
fix ut
sunjiahao1999 Mar 7, 2023
93461ee
fix ut
sunjiahao1999 Mar 7, 2023
40fcdda
fix semantic_kitti ut
sunjiahao1999 Mar 7, 2023
9794292
merge format semantic_kitti
sunjiahao1999 Mar 7, 2023
cf9e518
train init
sunjiahao1999 Mar 7, 2023
221798f
fix docstring
sunjiahao1999 Mar 8, 2023
3652f2d
fix config name
sunjiahao1999 Mar 8, 2023
e78b860
Merge branch 'dev-1.x' of github.com:open-mmlab/mmdetection3d into de…
sunjiahao1999 Mar 8, 2023
b7cca29
merge from dev-1.x
sunjiahao1999 Mar 8, 2023
5506766
merge format_semantickitti & rename segmentor ut
sunjiahao1999 Mar 8, 2023
5d787e0
rename layer
sunjiahao1999 Mar 8, 2023
c45b268
fix doc string
sunjiahao1999 Mar 8, 2023
08d9df2
fix review
sunjiahao1999 Mar 15, 2023
ce325f5
Merge branch 'dev-1.x' of github.com:open-mmlab/mmdetection3d into de…
sunjiahao1999 Mar 15, 2023
4827aca
fix conflicts
sunjiahao1999 Mar 15, 2023
57ab8bd
remove filter data
sunjiahao1999 Mar 15, 2023
c7dc194
merge dev-1.x
sunjiahao1999 Mar 16, 2023
9a3689f
Merge branch 'minkunet' into spvcnn
sunjiahao1999 Mar 16, 2023
92d9a02
rename config
sunjiahao1999 Mar 16, 2023
6392474
rename backbone
sunjiahao1999 Mar 16, 2023
fe56a64
rename backbone 2
sunjiahao1999 Mar 16, 2023
c1dc4fe
refactor voxel2point
sunjiahao1999 Mar 16, 2023
6658913
fix coors typo
sunjiahao1999 Mar 16, 2023
9d9b13f
fix ut
sunjiahao1999 Mar 16, 2023
6092aae
fix ut
sunjiahao1999 Mar 16, 2023
dbb2e68
Merge branch 'dev-1.x' of github.com:open-mmlab/mmdetection3d into de…
sunjiahao1999 Mar 20, 2023
7d9b05f
merge dev-1.x
sunjiahao1999 Mar 20, 2023
f7a44a1
pred in segmentor
sunjiahao1999 Mar 20, 2023
8ac5ed0
merge from minkunet
sunjiahao1999 Mar 21, 2023
c2b57a7
fix get voxel seg
sunjiahao1999 Mar 22, 2023
afb9ecb
resolve comments
sunjiahao1999 Mar 22, 2023
98da87e
Merge branch 'minkunet' into spvcnn
sunjiahao1999 Mar 22, 2023
c983cb1
rename p2v and v2p
sunjiahao1999 Mar 22, 2023
3c9e898
rename points and voxels
sunjiahao1999 Mar 23, 2023
445f58a
merge minkunet
sunjiahao1999 Mar 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions configs/minkunet/minkunet_w16_8xb2-15e_semantickitti.py
Original file line number Diff line number Diff line change
@@ -7,4 +7,7 @@
decoder_channels=[128, 64, 48, 48]),
decode_head=dict(channels=48))

# NOTE: Due to TorchSparse backend, the model performance is relatively
# dependent on random seeds, and if random seeds are not specified the
# model performance will be different (± 1.5 mIoU).
randomness = dict(seed=1588147245)
26 changes: 19 additions & 7 deletions mmdet3d/models/data_preprocessors/data_preprocessor.py
Original file line number Diff line number Diff line change
@@ -456,14 +456,22 @@ def get_voxel_seg(self, res_coors: torch.Tensor, data_sample: SampleList):
Args:
res_coors (Tensor): The voxel coordinates of points, Nx3.
data_sample: (:obj:`Det3DDataSample`): The annotation data of
every samples. Add voxel-wise annotation for segmentation.
every samples. Add voxel-wise annotation forsegmentation.
"""
pts_semantic_mask = data_sample.gt_pts_seg.pts_semantic_mask
voxel_semantic_mask, _ = dynamic_scatter_3d(
F.one_hot(pts_semantic_mask.long()).float(),
res_coors.contiguous(), 'mean')
voxel_semantic_mask = torch.argmax(voxel_semantic_mask, dim=-1)
data_sample.gt_pts_seg.voxel_semantic_mask = voxel_semantic_mask

if self.training:
pts_semantic_mask = data_sample.gt_pts_seg.pts_semantic_mask
voxel_semantic_mask, _, point2voxel_map = dynamic_scatter_3d(
F.one_hot(pts_semantic_mask.long()).float(), res_coors, 'mean',
True)
voxel_semantic_mask = torch.argmax(voxel_semantic_mask, dim=-1)
data_sample.gt_pts_seg.voxel_semantic_mask = voxel_semantic_mask
data_sample.gt_pts_seg.point2voxel_map = point2voxel_map
else:
pseudo_tensor = res_coors.new_ones([res_coors.shape[0], 1]).float()
_, _, point2voxel_map = dynamic_scatter_3d(pseudo_tensor,
res_coors, 'mean', True)
data_sample.gt_pts_seg.point2voxel_map = point2voxel_map

def ravel_hash(self, x: np.ndarray) -> np.ndarray:
"""Get voxel coordinates hash for np.unique().
@@ -495,6 +503,10 @@ def sparse_quantize(self,

Args:
coords (np.ndarray): The voxel coordinates of points, Nx3.
return_index (bool): Whether to return the indices of the
unique coords, shape (M,).
return_inverse (bool): Whether to return the indices of the
original coords shape (N,).

Returns:
List[np.ndarray] or None: Return index and inverse map if
2 changes: 1 addition & 1 deletion mmdet3d/models/decode_heads/minkunet_head.py
Original file line number Diff line number Diff line change
@@ -73,7 +73,7 @@ def forward(self, x: SparseTensor) -> Tensor:
x (SparseTensor): Features from backbone.

Returns:
output (Tensor): Segmentation map of shape [N, C].
Tensor: Segmentation map of shape [N, C].
Note that output contains all points from each batch.
"""
output = self.cls_seg(x.F)
22 changes: 18 additions & 4 deletions mmdet3d/models/segmentors/minkunet.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,16 @@
# Copyright (c) OpenMMLab. All rights reserved.
from typing import Tuple

from torch import Tensor

from mmdet3d.models.layers.torchsparse import IS_TORCHSPARSE_AVAILABLE
from mmdet3d.registry import MODELS
from mmdet3d.structures.det3d_data_sample import OptSampleList, SampleList
from .encoder_decoder import EncoderDecoder3D

if IS_TORCHSPARSE_AVAILABLE:
from torchsparse import SparseTensor
else:
SparseTensor = None


@MODELS.register_module()
class MinkUNet(EncoderDecoder3D):
@@ -94,8 +97,19 @@ def _forward(self,
x = self.extract_feat(batch_inputs_dict)
return self.decode_head.forward(x)

def extract_feat(self, batch_inputs_dict: dict) -> Tuple[Tensor]:
"""Extract features from voxels."""
def extract_feat(self, batch_inputs_dict: dict) -> SparseTensor:
"""Extract features from voxels.

Args:
batch_inputs_dict (dict): Input sample dict which
includes 'points' and 'voxels' keys.

- points (List[Tensor]): Point cloud of each sample.
- voxels (dict): Voxel feature and coords after voxelization.

Returns:
SparseTensor: voxels with features.
"""
voxel_dict = batch_inputs_dict['voxels']
x = self.backbone(voxel_dict['voxels'], voxel_dict['coors'])
if self.with_neck:
1 change: 1 addition & 0 deletions tests/test_models/test_segmentor/test_minkunet.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# Copyright (c) OpenMMLab. All rights reserved.
import unittest

import pytest