Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors
Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors
Vladimir Guzov*, Aymen Mir*, Torsten Sattler , Gerard Pons-Moll
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition 2021
(* joint first authors with equal contribution)
HPS jointly estimates the full 3D human pose and location of a subject within large 3D scenes, using only wearable sensors. Left: subject wearing IMUs and a head mounted camera. Right: using the camera, HPS localizes the human in a pre-built map of the scene (bottom left). The top row shows the split images of the real and estimated virtual camera
Download the scenes used for optimization, predefined vertices, all the IMU txt files and MVNX files, the video files and the camera localization json files
Download the SMPL Model files from here
Change the corresponding global variables denoting locations of these files in global_vars.py
Change the location of all the global variables that specify the locations where the data generated by all the processes in this repo should be stored.
Create conda environment
conda env create -f hps_env.yml
Run Preprocessing code. This computes the frame where the clap - used to synchronize the IMUs with the head mounted camera - occurs. This also generates the foot contact arrays from the mvnx files and processes the txt files to produce basic IMU pkl files.
Due to copyright concerns, we have not released the audio files associated with the head mounted camera videos. Please use the initialization files provided with the data.
python preprocess.py --file_name seq_name
Run Initialization code. This finds the orientation matrix that aligns the IMU coordinate frame with the scene coordinate frame. This generates the IMU bake.pkl files which contain the IMU recovered pose and translation oriented correctly in the 3D scene.
python Initialization.py --file_name seq_name
Compute scene normals. These are used for refining sitting frames in the final optimizaiton step. This script processes all the scenes at once.
python scene_normals.py
Resample the IMU and contact files. The IMU frames are sampled at 29.97 Hz while the Camera Frames are sampled at 30.0 Hz. This step ensures the two sensors are synchronized properly.
python resample.py
Compute Sitting Frames. This uses a heuristic to detect which frames involve sitting motion. All the sequences are processed at once
python sit_frames.py
Run the Filter Code. This filters out the erroneous camera localization frames. Incorrectly localized frames are detected based on comparison between IMU velocities and Camera velocities. The erroneous camera localizations are replaced by correctly oriented IMU trajectories.
cd filter
python filter.py --file_name seq_name
Create a config file. A sample file is found in configs folder
Run the optimization code as follows
python main.py --config configs/sample.txt
If you find our code useful, please consider citing our paper
@inproceedings{HPS,
title = {Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors },
author = {Guzov, Vladimir and Mir, Aymen and Sattler, Torsten and Pons-Moll, Gerard},
booktitle = {{IEEE} Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {jun},
organization = {{IEEE}},
year = {2021},
}
This code is available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using this code you agree to the terms in the LICENSE.
The smplpytorch code comes from Gul Varol's repository
The ChamferDistancePytorch code comes from Thibault Groueix's repository