V2X-R: Cooperative LiDAR-4D Radar Fusion for 3D Object Detection with Denoising Diffusion

🎈 Introduction

👋 This is the official repository for the V2X-R, including the V2X-R dataset and the implementation of the benchmark model, and MDD module.

This repo is also a unified and integrated multi-agent collaborative perception framework for LiDAR-based, 4D radar-based, LiDAR-4D radar fusion strategies!

Features

✨ Dataset Support
- V2X-R
- OPV2V
- DAIR-V2X
✨ Modality Support
- LiDAR
- 4D Radar
- LiDAR-4D Radar Fusion
✨ SOTA collaborative perception method support
- Late Fusion
- Early Fusion
- When2com (CVPR2020)
- V2VNet (ECCV2020)
- PFA-Net (ITSC2021)
- RTNH (NIPS2022)
- DiscoNet (NeurIPS2021)
- V2X-ViT (ECCV2022)
- CoBEVT (CoRL2022)
- Where2comm (NeurIPS2022)
- CoAlign (ICRA2023)
- BM2CP (CoRL2023)
- SCOPE (ICCV2023)
- How2comm (NeurIPS2023)
- InterFusion (IROS2023)
- L4DR (Arxiv2024)
- SICP (IROS2024)
✨ Visualization
- BEV visualization
- 3D visualization

🎈 V2X-R Dataset Manual

The first V2X dataset incorporates LiDAR, camera, and 4D radar. V2X-R contains 12,079 scenarios with 37,727 frames of LiDAR and 4D radar point clouds, 150,908 images, and 170,859 annotated 3D vehicle bounding boxes.

Dataset Collection

Thanks to the CARLA simulator and the OpenCDA framework, our V2X-R simulation dataset was implemented on top of them. In addition, our dataset route acquisition process partly references V2XViT, which researchers can reproduce according to the data_protocol in the dataset.

Download and Decompression

📒 Log in here using the username "Guest" and the password "guest_CMD" to download the dataset.

Since the data is large (including 3xLiDAR{normal, fog, snow}, 1xradar, 4ximages for each agent). We have compressed the sequence data of each agent, you can refer to this code for batch decompression after downloading.

import os
import subprocess
def decompress_v2x_r(root_dir, save_dir):
    for root, dirs, files in os.walk(root_dir):
        for file in files:
            if file.endswith('.7z'):
                file_path = os.path.join(root, file)
                path = root.split('/')
                extract_path = os.path.join(save_dir, path[-2], path[-1], file[:-3])
                os.makedirs(extract_path, exist_ok=True)
                subprocess.run(['7z', 'x', '-o' + extract_path + '/', file_path])

data_directory = #downloaded dataset path e.g: '/mnt/16THDD-2/hx/V2X-R_Dataset(compressed)'
output_directory =  #output dataset path  e.g: '/mnt/16THDD-2/hx/t'
decompress_v2x_r(data_directory, output_directory)

Structure

📂 After download and decompression are finished, the dataset is structured as following:

V2X-R # root path of v2x-r output_directory 
├── train
│   ├──Sequence name (time of data collection, e.g. 2024_06_24_20_24_02)
│   │   ├──Agent Number ("-1" denotes infrastructure, otherwise is CAVs)
│   │   │   ├──Data (including the following types of data)
│   │   │   │ Timestamp.Type, eg.
│   │   │   │ - 000060_camerai.png (i-th Camera),
│   │   │   │ - 000060.pcd (LiDAR),
│   │   │   │ - 000060_radar.pcd (4D radar),
│   │   │   │ - 000060_fog.pcd (LiDAR with fog simulation),
│   │   │   │ - 000060_snow.pcd (LiDAR with snow simulation),
│   │   │   │ - 000060.yaml (LiDAR with fog simulation)
├── validate
│   ├──...
├── test
│   ├──...

Calibration

We provide calibration information for each sensor (LiDAR, 4D radar, camera) of each agent for inter-sensor fusion. In particular, the exported 4D radar point cloud has been converted to the LiDAR coordinate system of the corresponding agent in advance of fusion, so the 4D radar point cloud is referenced to the LiDAR coordinate system.

🎈 Benchmark and Models Zoo

All benchmark model downloads require a login (using the username "Guest" and the password "guest_CMD")

4DRadar-based Cooperative 3D Detector (no-compression)

Method	Validation (IoU=0.3/0.5/0.7)	Testing (IoU=0.3/0.5/0.7)	Config	Model
ITSC2021:PFA-Net	76.90/68.00/39.30	85.10/79.90/52.50	√	model-25M
NIPS2022:RTNH	71.70/62.20/34.40	73.70/67.70/41.90	√	model-64M
ECCV2022:V2XViT	71.14/64.28/31.12	80.94/73.82/42.73	√	model-51M
ICRA2022:AttFuse	75.30/66.50/36.10	81.80/75.40/48.20	√	model-25M
NIPS2023:Where2comm	71.60/67.20/42.90	80.40/77.30/56.70	√	model-30M
ICCV2023:SCOPE	61.90/59.30/47.90	73.00/71.60/51.60	√	model-151M
CoRL2023:CoBEVT	80.20/73.40/41.10	85.80/80.60/52.90	√	model-40M
ICRA2023:CoAlign	65.80/59.20/34.70	76.90/70.20/46.20	√	model-43M
WACV2023:AdaFusion	77.84/72.48/42.85	82.20/78.08/55.51	√	model-27M
IROS2024:SICP	70.08/60.62/32.43	71.45/63.47/33.39	√	model-28M

LiDAR-based Cooperative 3D Detector (no-compression)

Method	Validation (IoU=0.3/0.5/0.7)	Testing (IoU=0.3/0.5/0.7)	Config	Model
ECCV2022:V2XViT	84.99/82.22/64.92	90.14/89.01/77.71	√	model-52M
ICRA2022:AttFuse	86.00/82.20/66.90	91.40/89.60/80.20	√	model-25M
NIPS2023:Where2comm	85.20/83.10/65.90	91.60/88.50/80.50	√	model-30M
ICCV2023:SCOPE	76.00/74.70/60.90	81.40/72.90/67.00	√	model-151M
CoRL2023:CoBEVT	87.64/84.79/71.01	92.29/91.44/82.45	√	model-40M
ICRA2023:CoAlign	89.08/87.57/80.05	89.59/88.89/83.29	√	model-43M
ICCV:AdaFusion	88.11/86.91/75.61	92.70/90.60/84.80	√	model-27M
IROS2024:SICP	81.14/77.62/58.14	84.64/82.17/66.71	√	model-28M
WACV2024:MACP	72.80/70.90/60.00	83.70/83.10/75.50	soon	model-61M

LiDAR-4D Radar Fusion Cooperative 3D Detector (no-compression)

Method	Validation (IoU=0.3/0.5/0.7)	Testing (IoU=0.3/0.5/0.7)	Config	Model
IROS2023:InterFusion	81.23/77.33/52.93	87.91/86.51/69.63	√	model-95M
Arxiv2024:L4DR	84.58/82.75/70.29	90.78/89.62/82.91	√	model-79M
ICRA2022:AttFuse	86.14/84.30/70.72	92.20/90.70/84.60	√	model-95M
ECCV2022:V2XViT	85.23/83.90/69.77	91.99/91.22/83.04	√	model-118M
ICCV2023:Scope	78.79/77.96/62.57	83.38/82.89/70.00	√	model-134M
NIPS2023:Where2comm	87.62/85.58/69.61	92.20/91.00/82.04	√	model-30M
CoRL2023:CoBEVT	87.72/87.02/78.56	94.00/93.21/86.68	√	model-40M
ICRA2023:CoAlign	87.16/85.54/73.79	92.14/91.21/83.77	√	model-49M
WACV2023:AdaFusion	89.24/87.31/74.86	93.10/92.15/85.48	√	model-27M
IROS2024:SICP	82.46/79.44/61.51	86.19/84.20/68.15	√	model-28M

🎈 Quickly Get Started

Installation

Refer to Installation of V2X-R

Train benchmark model

First of all, modify the dataset path in the setting file (root_dir, validate_dir), i.e. xxx.yaml.

The setting is same as OpenCOOD, which uses yaml file to configure all the parameters for training. To train your own model from scratch or a continued checkpoint, run the following commonds:

training command --hypes_yaml ${CONFIG_FILE} [--model_dir  ${CHECKPOINT_FOLDER}] [--tag  ${train_tag}] [--worker  ${number}]

Arguments Explanation:

hypes_yaml: the path of the training configuration file, e.g. opencood/hypes_yaml/second_early_fusion.yaml, meaning you want to train an early fusion model which utilizes SECOND as the backbone. See Tutorial 1: Config System to learn more about the rules of the yaml files.
model_dir (optional) : the path of the checkpoints. This is used to fine-tune the trained models. When the model_dir is given, the trainer will discard the hypes_yaml and load the config.yaml in the checkpoint folder.
tag (optional) : the path of the checkpoints. The training label is used to record additional information about the model being trained, with the default setting being 'default'.
worker (optional) : the number of workers in dataloader, default is 16.

Training with single-GPU

For example, to train V2XR_AttFuse (LiDAR-4D radar fusion version) from scratch:

CUDA_VISIBLE_DEVICES=0 python opencood/tools/train.py --hypes_yaml opencood/hypes_yaml/V2X-R/L_4DR_Fusion/V2XR_AttFuse.yaml --tag 'demo' --worker 16

To train V2XR_AttFuse from a checkpoint:

CUDA_VISIBLE_DEVICES=0 python opencood/tools/train.py --hypes_yaml opencood/hypes_yaml/V2X-R/L_4DR_Fusion/V2XR_AttFuse.yaml --model_dir opencood/logs/V2XR_AttFuse/test__2024_11_21_16_40_38 --tag 'demo' --worker 16

Training with distributed multi-GPUs

For example, to train V2XR_AttFuse (LiDAR-4D radar fusion version) from scratch with 4 GPUs:

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4  --use_env opencood/tools/train.py --hypes_yaml opencood/hypes_yaml/V2X-R/L_4DR_Fusion/V2XR_AttFuse.yaml --tag 'demo' --worker 16

Test the model

Before you run the following command, first make sure the validation_dir in config.yaml under your checkpoint folder

python opencood/tools/inference.py --model_dir ${CHECKPOINT_FOLDER} --fusion_method ${FUSION_STRATEGY} --eval_epoch ${epoch_number} --save_vis ${default False}

Arguments Explanation:

model_dir: the path to your saved model.
fusion_method: indicate the fusion strategy, currently support 'early', 'late', 'intermediate', 'no'(indicate no fusion, single agent), 'intermediate_with_comm'(adopt intermediate fusion and output the communication cost).
eval_epoch: int. Choose to inferece which epoch.
save_vis: bool. Wether to save the visualization result.

For example, to test V2XR_AttFuse (LiDAR-4D radar fusion version) from scratch:

CUDA_VISIBLE_DEVICES=0 python opencood/tools/inference.py --hypes_yaml opencood/hypes_yaml/V2X-R/L_4DR_Fusion/V2XR_AttFuse.yaml --model_dir opencood/logs/V2XR_AttFuse/test__2024_11_21_16_40_38

The evaluation results will be dumped in the model directory.

Train model with Multi-modal Diffusion Denoising (MDD) module

To be continued ...

Citation

If you are using our project for your research, please cite the following paper:

@article{V2X-R,
  title={V2X-R: Cooperative LiDAR-4D Radar Fusion for 3D Object Detection with Denoising Diffusion},
  author={Huang, Xun and Wang, Jinlong and Xia, Qiming and Chen, Siheng and Yang, Bisheng and Wang, Cheng and Wen, Chenglu},
  journal={arXiv preprint arXiv:2411.08402},
  year={2024}
}

Acknowledgements

Thank for the excellent cooperative perception codebases BM2CP, OpenCOOD, CoPerception and Where2comm.

Thank for the excellent cooperative perception dataset OPV2V.

Thank for the dataset and code support by DerrickXu and ZhaoAI.

Name		Name	Last commit message	Last commit date
Latest commit History 50 Commits
images		images
opencood		opencood
.gitignore		.gitignore
INSTALL.md		INSTALL.md
readme.md		readme.md
requirements.txt		requirements.txt
setup.py		setup.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

V2X-R: Cooperative LiDAR-4D Radar Fusion for 3D Object Detection with Denoising Diffusion

🎈 Introduction

Features

🎈 V2X-R Dataset Manual

Dataset Collection

Download and Decompression

Structure

Calibration

🎈 Benchmark and Models Zoo

4DRadar-based Cooperative 3D Detector (no-compression)

LiDAR-based Cooperative 3D Detector (no-compression)

LiDAR-4D Radar Fusion Cooperative 3D Detector (no-compression)

🎈 Quickly Get Started

Installation

Train benchmark model

Training with single-GPU

Training with distributed multi-GPUs

Test the model

Train model with Multi-modal Diffusion Denoising (MDD) module

Citation

Acknowledgements

About

Releases

Packages

Languages

ylwhxht/V2X-R

Folders and files

Latest commit

History

Repository files navigation

V2X-R: Cooperative LiDAR-4D Radar Fusion for 3D Object Detection with Denoising Diffusion

🎈 Introduction

Features

🎈 V2X-R Dataset Manual

Dataset Collection

Download and Decompression

Structure

Calibration

🎈 Benchmark and Models Zoo

4DRadar-based Cooperative 3D Detector (no-compression)

LiDAR-based Cooperative 3D Detector (no-compression)

LiDAR-4D Radar Fusion Cooperative 3D Detector (no-compression)

🎈 Quickly Get Started

Installation

Train benchmark model

Training with single-GPU

Training with distributed multi-GPUs

Test the model

Train model with Multi-modal Diffusion Denoising (MDD) module

Citation

Acknowledgements

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages