ComPhy

This repository holds the code for the paper.

ComPhy: Compositional Physical Reasoning ofObjects and Events from Videos, (ICLR 2022)

Project Website

Evaluation Server

Framework

Code Preparation

git clone https://github.com/comphyreasoning/compositional_physics_learner.git 

Installation

pip install -r requirements

Data Preparation

• Download videos, video annotation, questions from the project website.

Fast Evaluation

• Download the regional proposals with attribute and physical property prediction from the anonymous Google drive
• Download the dynamic predictions from the anonymous Google drive
• Run executor for factual questions.

sh scripts/test_oe_release.sh

• Run executor for multiple-choice questions.

sh scripts/test_mc_release.sh

• Submit results onto the evaluation server.

Supporting sub-modules

Physical Property Learner and Dynamic predictor

Please refer to this repo for property learning and dynamics prediction.

Perception

This module uses the public NS-VQA's perception module object detection and visual attribute extraction.

Program parser

This module uses the public NS-VQA's program parser module to tranform language into executable programs.