Video Instance Shadow Detection Under the Sun and Sky

Zhenghao Xing*, Tianyu Wang*, Xiaowei Hu, Haoran Wu, Chi-Wing Fu, Pheng-Ann Heng (* Joint first authors)

IEEE Transactions on Image Processing (TIP), 2024

[IEEE Xplore] [arXiv] [Video]

Installation

$ git clone https://github.com/HarryHsing/Video-Instance-Shadow-Detection.git
$ cd Video-Instance-Shadow-Detection
$ conda create -n VISD python=3.8
$ conda activate VISD
$ conda install pytorch==1.13.0 torchvision==0.14.0 torchaudio==0.13.0 pytorch-cuda=11.7 -c pytorch -c nvidia
$ pip install -r requirements.txt
$ cd visd
$ pip install -e .

Demo

Download Pre-trained Weights

$ cd ..
$ wget -P ./model_zoo/ https://github.com/HarryHsing/Video-Instance-Shadow-Detection/releases/download/weights/model_final.pth

Run the Demo

$ python vishadow_demo.py --input-name ./assets/videos/skateboard.mp4 --output-name demo_result

This command processes the specified input video (skateboard.mp4) and produces:

• Shadow-object association masks for each frame.
• Painted frames and a reconstructed output video.

All results will be saved with the prefix demo_result for easy access.

SOBA-VID Dataset

Please download the dataset using the links below:

• Video
• Annotation

Citation

The following is a BibTeX reference:

@article{xing2024video,
  title={Video Instance Shadow Detection Under the Sun and Sky},
  author={Xing, Zhenghao and Wang, Tianyu and Hu, Xiaowei and Wu, Haoran and Fu, Chi-Wing and Heng, Pheng-Ann},
  journal={IEEE Transactions on Image Processing},
  year={2024},
  publisher={IEEE}
}

TODO

• Training Code
• Evaluation Code
• Gradio Interface

Acknowledgement

This repo is largely based on SSIS and InternGPT. Thanks for their excellent works.