This project involves Implementation of Conference Paper,
Video stabilization using Sparse Optical Flow using a Mesh grid. This includes outlier rejection using median filtering, generating vertex profiles, and smoothening. The project was accomplished by a team of 3 members, which includes Naveen Mangla , Mahima Arora, and Charu Sharma.
git clone [email protected]:nvnmangla/Video-Stabilization.git
cd Video-Stabilization
python3 main.py --video="<your video path>"
There are many challenges faced by researchers when it comes to Video Stabilization but two challenges are prominent. First, camera motion estimation is often difficult and second, successful camera motion filtering often requires future frames. The more one reduces this buffer, the more will the future frame significantly deteriorate the results.
Moreover, doing optical flow is computationally expensive, therefore we aim to do feature matching with which we get cheaper results but we also use similar techniques as optical flow i.e. they both encode strong spatial smoothness. For another, optical flow is dense and the Mesh Flow is sparse since it uses less profiles and only takes into consideration the mesh vertices instead of all pixel profiles hence, Mesh Flow can be regarded as a down-sampled dense flow.
Taking Pixel Profiles Vs taking Mesh Vertices for computing flow
- We record a video and place a regular 2D mesh on the video frame
- We then track image corners or "features" between consecutive frames, which yields a motion vector at each feature location.
Motion Vectors
- Next, these motion vectors are transferred to their corresponding nearby mesh vertexes,such that each vertex accumulates several motions from its surrounding feature
- With regards to the camera motion smoothing, we design a filter to smooth the temporal changes of the motion vector at each mesh vertex. This filter is applied to each mesh vertex
- Use PAPS (Predicted Adaptive Path Smoothing is when old frame flow is used to predict future flow of frame. We reduce excessive cropping and wobble distortion using this) for strong stabilization by using previous frame
Original vs Optimized Path of Vertex
- Generate a stabilized video
Input Video | Stable Video