Skip to content

Latest commit

 

History

History
204 lines (185 loc) · 7.34 KB

README.md

File metadata and controls

204 lines (185 loc) · 7.34 KB

EINet

The official code and benchmark for our paper: Explicit Interaction for Fusion-Based Place Recognition.

This work has been accepted by IROS 2024 🎉

Jingyi Xu, Junyi Ma, Qi Wu, Zijie Zhou, Yue Wang, Xieyuanli Chen, Wenxian Yu, Ling Pei*.

image

Installation

We follow the installation instructions of our codebase LCPR, which are also posted here.

  • Create a conda virtual environment and activate it
git clone [email protected]:BIT-XJY/EINet.git
cd EINet
conda create -n EINet python=3.8
conda activate EINet
  • Install other dependencies
pip install -r requirements.txt

Data Download

Note that the download data structure should be like:

nuscenes
├─ raw_data
│    ├─ maps
│    │    ├─ ...
│    ├─ samples
│    │    ├─ CAM_BACK
│    │    ├─ CAM_BACK_LEFT
│    │    ├─ CAM_BACK_RIGHT
│    │    ├─ CAM_FRONT
│    │    ├─ CAM_FRONT_LEFT
│    │    ├─ CAM_FRONT_RIGHT
│    │    ├─ LIDAR_TOP
│    │    ├─ RADAR_BACK_LEFT
│    │    ├─ RADAR_BACK_RIGHT
│    │    ├─ RADAR_FRONT
│    │    ├─ RADAR_FRONT_LEFT
│    │    ├─ RADAR_FRONT_RIGHT
│    ├─ sweeps
│    │    ├─ CAM_BACK
│    │    ├─ CAM_BACK_LEFT
│    │    ├─ CAM_BACK_RIGHT
│    │    ├─ CAM_FRONT
│    │    ├─ CAM_FRONT_LEFT
│    │    ├─ CAM_FRONT_RIGHT
│    │    ├─ LIDAR_TOP
│    │    ├─ RADAR_BACK_LEFT
│    │    ├─ RADAR_BACK_RIGHT
│    │    ├─ RADAR_FRONT
│    │    ├─ RADAR_FRONT_LEFT
│    │    ├─ RADAR_FRONT_RIGHT
│    ├─ v1.0-test
│    │    ├─ attribute.json
│    │    ├─ calibrated_sensor.json
│    │    ├─ ...
│    ├─ v1.0-traninval
│    │    ├─ attribute.json
│    │    ├─ calibrated_sensor.json
│    │    ├─ ...

NUSC-PR

We propose the NUSC-PR benchmark to split nuScenes datasets with self-supervised and supervised learning schemes.

Self-supervised Data Preparation

  • Extract basic information from nuScenes datasets, and split query and database for NUSC-PR.
cd NUSC-PR
cd self_supervised
python generate_basic_infos.py
python split_dataset.py
cd ..
  • The data structure with a self-supervised learning scheme should be like:
self_supervised_data
├─ generate_basic_infos
│    ├─ nuscenes_infos-bs.pkl
│    ├─ nuscenes_infos-shv.pkl
│    ├─ nuscenes_infos-son.pkl
│    ├─ nuscenes_infos-sq.pkl
│    ├─ nuscenes_infos.pkl
├─ split_dataset
│    ├─ all_train_query_pos_neg_index_in_infos.pkl
│    ├─ bs_db_index_in_infos.npy
│    ├─ bs_test_query_gt_index_in_infos.pkl
│    ├─ bs_train_query_pos_neg_index_in_infos.pkl
│    ├─ shv_db_index_in_infos.npy
│    ├─ shv_test_query_gt_index_in_infos.pkl
│    ├─ shv_train_query_pos_neg_index_in_infos.pkl
│    ├─ son_db_index_in_infos.npy
│    ├─ son_test_query_gt_index_in_infos.pkl
│    ├─ son_train_query_pos_neg_index_in_infos.pkl
│    ├─ sq_db_index_in_infos.npy
│    ├─ sq_test_query_gt_index_in_infos.pkl
│    ├─ sq_train_query_pos_neg_index_in_infos.pkl

Supervised Data Preparation

  • Extract basic information from nuScenes datasets, and split query and database for NUSC-PR.
cd supervised
python generate_basic_infos.py
python split_dataset.py
python select_pos_neg_samples_by_dis.py
python generate_selected_indicies.py
cd ..
cd ..
  • The data structure with a supervised learning scheme should be like:
supervised_data
├─ generate_basic_infos
│    ├─ nuscenes_infos-bs.pkl
│    ├─ nuscenes_infos-shv.pkl
│    ├─ nuscenes_infos-son.pkl
│    ├─ nuscenes_infos-sq.pkl
│    ├─ nuscenes_infos.pkl
├─ generate_selected_indicies
│    ├─ bs_db_index_in_infos.npy
│    ├─ bs_test_query_gt_index_in_infos.pkl
│    ├─ bs_train_query_pos_neg_index_in_infos.pkl
│    ├─ shv_db_index_in_infos.npy
│    ├─ shv_test_query_gt_index_in_infos.pkl
│    ├─ shv_train_query_pos_neg_index_in_infos.pkl
│    ├─ son_db_index_in_infos.npy
│    ├─ son_test_query_gt_index_in_infos.pkl
│    ├─ son_train_query_pos_neg_index_in_infos.pkl
│    ├─ sq_db_index_in_infos.npy
│    ├─ sq_test_query_gt_index_in_infos.pkl
│    ├─ sq_train_query_pos_neg_index_in_infos.pkl
├─ select_pos_neg_samples_by_dis
│    ├─ bs_test_query_gt_tokens.pkl
│    ├─ bs_train_query_pos_neg_tokens.pkl
│    ├─ shv_test_query_gt_tokens.pkl
│    ├─ shv_train_query_pos_neg_tokens.pkl
│    ├─ son_test_query_gt_tokens.pkl
│    ├─ son_train_query_pos_neg_tokens.pkl
│    ├─ sq_test_query_gt_tokens.pkl
│    ├─ sq_train_query_pos_neg_tokens.pkl
├─ split_dataset
│    ├─ bs_db_sample_token.npy
│    ├─ bs_db.npy
│    ├─ bs_sample_token.npy
│    ├─ bs_test_query_sample_token.npy
│    ├─ bs_test_query.npy
│    ├─ bs_train_query_sample_token.npy
│    ├─ bs_train_query.npy
│    ├─ bs_val_query_sample_token.npy
│    ├─ bs_val_query.npy
│    ├─ shv_db_sample_token.npy
│    ├─ shv_db.npy
│    ├─ shv_sample_token.npy
│    ├─ shv_test_query_sample_token.npy
│    ├─ shv_test_query.npy
│    ├─ shv_train_query_sample_token.npy
│    ├─ shv_train_query.npy
│    ├─ shv_val_query_sample_token.npy
│    ├─ shv_val_query.npy
│    ├─ son_db_sample_token.npy
│    ├─ son_db.npy
│    ├─ son_sample_token.npy
│    ├─ son_test_query_sample_token.npy
│    ├─ son_test_query.npy
│    ├─ son_train_query_sample_token.npy
│    ├─ son_train_query.npy
│    ├─ son_val_query_sample_token.npy
│    ├─ son_val_query.npy
│    ├─ sq_db_sample_token.npy
│    ├─ sq_db.npy
│    ├─ sq_sample_token.npy
│    ├─ sq_test_query_sample_token.npy
│    ├─ sq_test_query.npy
│    ├─ sq_train_query_sample_token.npy
│    ├─ sq_train_query.npy
│    ├─ sq_val_query_sample_token.npy
│    ├─ sq_val_query.npy

TODO

  • Release the paper
  • Release the benchmark NUSC-PR code for EINet
  • Release the source code for EINet
  • Release our pretrained baseline model

Acknowledgement

We thank the fantastic works LCPR, ManyDepth, and AutoPlace for their pioneer code release, which provide codebase for this work.