Pothole-Detection

Detecting road potholes using OAK-D camera

Table of Contents

• About the Project
  • Aim
  • Description
  • Tech Stack
  • File Structure
• Getting Started
  • Prerequisites
  • Installation
  • Execution
• Theory and Approach
• Algorithm Flowchart
• Results and Demo
• Future Work
• Contributors
• Acknowledgements and Resources

About The Project

Aim

Potholes are bowl-shaped openings on the road that are caused by the combined effect of wear-and-tear and weathering of the road. Potholes are not only main cause of car accidents, but also can be fatal to motorcycles. Drivers might see the pothole before they pass it, it is usually too late for them to react. Keeping this in mind , our project aims to locate the potholes present on a road through the input image and provide its distance in cm from the camera.

Description

We have used OAK-D camera to get the monochromatic stereo images of the surface which are then rectified using the inbuilt functioning of the camera. We have implemented stereo image processing algorithms to generate disparity maps and then labelled the appropriate areas in the image as pothole using the connected component labelling algorithm.

Tech Stack

• OpenCV
• Numpy
• Matplotlib
• Scikit Learn
• DepthAI
• Jupyter notebook
• Google Colab

File Structure

📦Pothole-Detection
 ┣ 📂Assets                           #contains images taken from the left-right stereo camera
 ┃ ┗ 📂Images
 ┣ 📂Results
 ┃ ┗ 📂Disparity		      #contains depth map in form of .npy file generated from the Oak-D camera
 ┃ ┃ ┣ 📜dispimage_7.npy
 ┃ ┃ ┣ 📜dispimage_8.npy
 ┃ ┃ ┣ 📜pothole_depth1.npy
 ┃ ┃ ┣ 📜pothole_depth2.npy
 ┃ ┃ ┣ 📜image7.png
 ┃ ┃ ┗ 📜image8.png
 ┃ ┗ 📂Pothole Detection	      #contains final result with detected potholes
 ┃ ┃ ┣ 📜Pothole_depthimage_1.png
 ┃ ┃ ┣ 📜Pothole_depthimage_2.png
 ┃ ┃ ┣ 📜Resultwith21Matrix7.png
 ┃ ┃ ┗ 📜Resultwith21Matrix8.png
 ┣ 📂notes                            
 ┃ ┣ 📂Dhruvanshu_notes		      #contains notes created by Dhruvanshu
 ┃ ┗ 📂Prachi_notes		      #contains notes created by Prachi
 ┣ 📂SRC                             #contains sorce codes
 ┃ ┣ 📂asset_npys		      #contains depth map in form of npys
 ┃ ┃ ┣ 📜depth_image_0.npy
 ┃ ┃ ┣ 📜depth_image_1.npy
 ┃ ┃ ┣ 📜depth_image_2.npy
 ┃ ┃ ┣ 📜depth_image_3.npy
 ┃ ┃ ┣ 📜depth_image_4.npy
 ┃ ┃ ┣ 📜image_0.npy
 ┃ ┃ ┣ 📜image_1.npy
 ┃ ┃ ┣ 📜image_2.npy
 ┃ ┃ ┣ 📜image_3.npy
 ┃ ┃ ┣ 📜testcase1.npy
 ┃ ┃ ┗ 📜testcase2.npy
 ┃ ┣ 📜Boxed.ipynb
 ┃ ┣ 📜CameraInfo
 ┃ ┣ 📜DepthGenerator.ipynb
 ┃ ┣ 📜Detection.ipynb
 ┃ ┣ 📜Get_Camera_Info.py
 ┃ ┣ 📜Get_Left_Right.py
 ┃ ┣ 📜Labelling.ipynb
 ┃ ┣ 📜MinimizeError.ipynb
 ┃ ┣ 📜Pothole_Detection.ipynb
 ┃ ┣ 📜Pothole_Detection.py
 ┃ ┣ 📜Result.ipynb
 ┃ ┣ 📜SurfaceFit.ipynb
 ┃ ┣ 📜data.ipynb
 ┃ ┣ 📜depth-saving.py
 ┃ ┗ 📜main.ipynb
 ┃ ┣ 📜Boxed.py
 ┃ ┣ 📜DepthGenerator.py
 ┃ ┣ 📜Detection.py
 ┃ ┣ 📜Get_Left_Right.py
 ┃ ┣ 📜Labelling.py
 ┃ ┣ 📜MinimizeError.py
 ┃ ┣ 📜Pothole_Detection.py
 ┃ ┣ 📜MonoImages.py
 ┃ ┣ 📜Result.py
 ┃ ┣ 📜SurfaceFit.py
 ┃ ┣ 📜data.py
 ┃ ┣ 📜main.py
 ┃ ┣ 📂depth-sav                    #Used to generate depth from Oak-D camera
 ┃ ┃ ┣ 📜depth-sav.py
 ┃ ┃ ┣ 📜image_0.npy
 ┃ ┃ ┣ 📜image_1.npy
 ┃ ┃ ┗ 📜image_2.npy
 ┃ ┗ 📂asset_npys
 ┗ 📜readme.md

Getting Started

Prerequisites

• Should have python environment. To establish python environment, refer here for the setup.
  OR
  You can create a virtual environment referring this
• Python libraries

  • DepthAI

  • OpenCV
    pip install opencv-python

  • Scikit
    pip install scikit-learn

  • numpy
    pip install numpy

  • Matplotlib
    pip install matplotlib

  • Modulling

    pip install ipynb

UPDATE: To run the code on Google colab, run the notebooks.

Installation

Clone the repo

git clone https://github.com/Dhruvanshu-Joshi/Pothole-Detection.git

Execution

• Connect the OAK-D camera
• Run Save_Image code to capture the stereo images.
• Run Rectification_Disparity code to get the disparity map
• For all further activities, run code

Theory and Approach

1. The stereo images are captured using the depth camera.
2. They are then rectified using the DepthAI functions that are based on the traditional geometric methods.
3. Then the disparity map is generated using StereoSGBM (semi global matching algorithm). Given the disparity map and stereo camera parameters, the corresponding coordinates in 3-dimensional coordinate system can be calculated.
4. Given all 3D points in an image, a road surface can be fitted using the bi-squares weighted robust least-squares algorithm.
5. Then all the outliers can be labelled as road potholes.

Algorithm Flowchart

Simplified code structure

Results and Demo

Future Work

• Detect Potholes in Images
• Detect Potholes in Videos
• Detect Potholes in Real-Time

Contributors

• Dhruvanshu Joshi
• Prachi Doshi

Acknowledgements and Resources

• SRA VJTI Eklavya 2022
• Our mentors Toshan Luktuke, Dhruv Kunjadiya and Rishabh Bali for their guidance throughout this project
• Yaqi-Li Research Paper for main refference
• John Lambert Research Paper for stereo related reference
• Video to know more on stereo
• Video on Camera Calibration
• For more resources refer References section in the Report.