GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMs

Official implementation of GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMs.

🚩 Features

• [✅] Release physical property prediction code.
• [✅] Gradio online demo available. Try it at demo link.
• [TODO] Release physical-based simulation models and configurations.

⚙️ Dependencies and Installation

We recommend using Python>=3.10, PyTorch>=2.1.0, and CUDA>=12.1.

conda create -n gp python=3.10
conda install pytorch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 pytorch-cuda=12.1 -c pytorch -c nvidia

# install SAM, LangSplat used SAM for multi-level segmentation. Here we using for part-level segmentation.
git clone https://github.com/minghanqin/segment-anything-langsplat
cd segment-anything; pip install -e .
pip install opencv-python pycocotools matplotlib onnxruntime onnx 

pip install rembg
pip install numpy==1.26.4
pip install openai
pip install gradio

💫 Run

Remove Background(Optional) and Oganize the Data

We provide sample data in the gp_cases folder for testing. To test with your own data, simply organize it in the same format.

python folder_oganizer.py --folder_path gp_cases

Part-level Segmentation using SAM

First, download the checkpoints of SAM from here, then preprocess the data using SAM:

python sam_preprocess.py  

Physical Property Prediction using LMMS

you can choose between GPT-4V or Qwen-VL-MAX by adding the --vlm gpt or --vlm qwen flag. Make sure to update your api_key in utils/vim_utils.py before running:

python vlm_predict.py

Visualize Material Segmentation Result

To visualize the material segmentation result, run:

python visualize_material_segmentation.py

💻 Gradio Demo

To run the Gradio demo, execute the following command and access the demo in your local web browser:

python app.py

📚 Citation

If you find this project helpful in your research or applications, please cite it as follows:

@article{xu2024gaussianproperty,
  title={GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMs},
  author={Xinli Xu and Wenhang Ge and Dicong Qiu and ZhiFei Chen and Dongyu Yan and Zhuoyun Liu and Haoyu Zhao and Hanfeng Zhao and Shunsi Zhang and Junwei Liang and Ying-Cong Chen},
  journal={arXiv preprint arXiv:2412.11258},
  year={2024}
}

🤗 Acknowledgements

We thank the authors of the following projects for their excellent contributions to our project!

• NeRF2Physics
• PhysGaussian
• LangSplat