The main project folder contains 6 subdirectories. These are as follows:
- Code contains all of the code which was written to actually make the product work. This folder is organized using Patrick Ball's method for principled data processing (video url: https://www.youtube.com/watch?v=ZSunU9GQdcI). This subdirectory's contents and structure are described in detail below.
- Deliverables contains the non-code related deliverables for this project. Specifically this includes the project proposal, milestone presentation slides, conference poster, report, and video.
- InteractiveTools contains a handful of jupyter notebooks which help with exploratory data analysis, such as plotting ground truth data on the image.
- Resources contains files that we used as resources throughout the project. Includes notes which were taken while we were deciding how to proceed with data preparation as well as evaluation. In addition, this directory contains the document published by WeRobotics to guide participants in the AI challenge as well as the data provided which was not used in the project.
The table below shows what software/tools are required for completing each task. Click on the software name to see its installation guide.
| Software | 1_import | 2_clean | 3_tileAnnotate | 4_trainModel | 5_testModel | 6_evaluate | webApp |
|---|---|---|---|---|---|---|---|
| Python3 | 🔹 | 🔹 | 🔹 | 🔹 | 🔹 | ||
| GeoPandas | 🔹 | ||||||
| Pillow | 🔹 | 🔹 | 🔹 | 🔹 | |||
| Darknet* | 🔹 | 🔹 | |||||
| Pandas | 🔹 | ||||||
| NumPy | 🔹 | ||||||
| Flask | 🔹 | ||||||
| * Follow the installation guide, except clone our fork (https://github.com/bgerspac/darknet.git) and place in the same directory as this repo. |
Download the aerial image provided by WeRobotics from this link. Rename this file to single_layer.tif and place it in Code/2_clean/input and Code/3_tileAnnotate/input.
Starting in the Code/ directory, execute the following commands to perform preprocessing as we did.
$ cd 1_import/src
$ ./shp2geojson.sh
$ cd ../..
$ cd 2_clean/src
$ ./clean.sh
$ cd ../..
$ cd 3_tileAnnotate/src
$ ./tileAnnotate.sh
$ cd ../..
$ cd 4_trainModel/src
$ ./trainModel.sh
$ cd ../..
$ cd 5_testModel/src
$ ./testModel.sh
$ cd ../..
$ cd 6_evaluate/src
$ ./mean_average_precision.sh
Once we finished data processing, we implemented one of the models in a user facing web app. Visit our finished web application
The code has been organized using a structure devised by Patrick Ball (the video url is above). Each task in the data science pipeline is given its own directory. Within these directories there are 3 folders:
- input contains all data required to perform the task.
- output contains all data generated by the task.
- src contains the code which performs the task (python and bash scripts in our case).
In addition, this folder contains 1 directory which does not adhere to the Ball structure):
webAppcontains the web application we built, allowing users to submit an image and return predictions and counts based on the CNN model we developed in the earlier tasks.
Converts original shapefile data provided by WeRobotics into the geoJSON open standard. To run this task, execute the following from Code/1_import/src.
$ python3 shp2geojson.py <INPUT_PATH> <OUTPUT_PATH>
INPUT_PATH: path to the directory containing the shapefile to convert.
OUTPUT_PATH: file path to the outputted geoJSON file.
$ python3 shp2geojson.py '../input/Trees/' '../output/points.json'
Maps longitude/latitude coordinates to x/y pixel values in image, assigning class and box width to each object, and removing objects outside the clipping boundary. To run this task, execute the following from Code/2_clean/src.
$ python3 clean.py <JSON_IN> <IMAGE_IN> <CLIP_IN> <OUTPUT_PATH> <CLASSES>
JSON_IN: file path to the json file containing tree data.
IMAGE_IN: file path to image.
CLIP_IN: file path to clipping boundary.
OUTPUT_PATH: file path to output file.
CLASSES: classes to include in the output, provided as a string of comma separated classes. Optional. Defaults to "all".
$ python3 clean.py ../input/points.json ../input/single_layer.tif ../input/clipping_boundary.geojson ../output/BCCleaned.json "banana tree,coconut tree"
Splits the large image up into many small images (tiles). Creates an annotation text file for each tile. And generates a json file mapping classes to integers. To run this task, execute the following from Code/3_tileAnnotate/src.
$ python 3tileAnnotate.py <IMAGE_IN> <JSON_IN> <PREFIX> <EXT> <HEIGHT> <WIDTH>
IMAGE_IN: File path the image file being split into tiles.
JSON_IN: File path to the JSON containing objects being assigned to tiles.
OUTPUT_PATH_PREFIX: Path and file prefix of the resulting tiles (tile numbers are added to complete the filename).
OUTPUT_FILE_EXTENSION: File suffix of the resulting tiles (e.g. png, jpeg).
TILE_HEIGHT: Height in pixels of the resulting tiles.
TILE_WIDTH: Width in pixels of the resulting tiles.
$ python3 tileAnnotate.py '../input/single_layer.tif' '../input/BCCleaned.json' '../output/tile' 'jpg' 596 593
Train a YOLOv2 model to detect objects using the Darknet framework. Execute the following command from Code/4_trainModel/src to run this task. Note that training a model often takes a very long time, even if you are using a GPU. For this reason, you may want to run this as a background process. Output also contains some sample weight files if you'd prefer to skip training all together.
$ <DARKNET_PATH> detector train <DATA_PATH> <CFG_PATH> <WEIGHT_PATH> ><OUT_PATH> 2> <ERROR_PATH>
DARKNET_PATH: File path to your Darknet installation.
DATA_PATH: File path to the .data file needed for training.
CFG_PATH: File path to the .cfg file needed for training.
PRETRAINED_WEIGHT_PATH: File path to the set of initialization weights (Don't supply if you want to train from scratch).
OUT_PATH: File path to where standard output is written.
ERROR_PATH: File path to where standard error is written.
$ ../../../../darknet/darknet detector train ../input/darknet-cfg/bc.data ../input/darknet-cfg/bc.cfg ../input/darknet-cfg/darknet19_448.conv.23 >../output/bc.out 2>../output/bc.error
Uses the trained YOLOv2 model and detects objects in a test set, outputting the predictions to later be evaluated. The following command can be run from Code/5_testModel to execute this task.
$ <DARKNET_PATH> detector valid <DATA_PATH> <CFG_PATH> <WEIGHTS_PATH>
DARKNET_PATH: File path to your Darknet installation.
DATA_PATH: File path to the config folder for the model which you want to use to make detections.
CFG_PATH: File path to the config folder for the model which you want to use.
WEIGHTS_PATH: File path to the weighs for the model which you want to use.
../../../../darknet/darknet detector valid ../input/darknet-cfg/bc.data ../input/darknet-cfg/bc.cfg ../input/bc_10.weights
From the predictions made on the test set, calculate the average precision for each class, and the mean Average Precision across all classes. To execute this task, run the following from Code/5_evaluate/src
$ python3 mean_average_precision.py <PREDICTION_DIR> <GROUND_TRUTH_DIR> <CLASS_PATH> <OUT_PATH> <IOU_THRESH>
PREDICTION_DIR: Path to directory containing the prediction files.
GROUND_TRUTH_DIR: Path to the directory containing all ground truth information (images & annotations).
CLASS_PATH: File path to the json containing class mapping between class names and integers.
OUT_PATH: Path to the file where output should be appended.
IOU_THRESH: The IoU threshold used to determine whether a detection is a true or false positive.
$ python3 mean_average_precision.py ../input/basic30000/ ../../3_tileAnnotate/output/ ../input/classes.json ../output/mAP.txt 0.5
A Python- and Flask-based application that allow an image to be submitted for prediction. Predictions for banana and coconut trees are overlaid on the image with their confidence level. The user can change the threshold to hide and show predictions with different confidence levels. The complete list of predictions of all confidence levels can be downloaded.
A sample of the application can be accessed at: http://ec2-35-161-243-71.us-west-2.compute.amazonaws.com:5000/
Requires that the PYTHONPATH environment variable contains the 3_tileAnnotate/src and webApp/src folders.
Only the --darknetpath argument is required but model files and output folders are assumed to exist in default locations if their locations are not provided as arguments.
$ python3 app.py --darknetpath DARKNETPATH [--cfg CFG] [--weights WEIGHTS] [--out OUT] [--err ERR] [--sample SAMPLE] [--result RESULT] [--temp TEMP] [--caveat CAVEAT]
DARKNETPATH: Path to the folder containing the darknet executable that does detections
CFG: The config file of the detection model to use (defaults to model.cfg)
WEIGHTS: The weights file of the detection model to use (defaults to model.weights)
OUT: The file containing output from the latest detection run
ERR: The file containing error output from the latest detection run
SAMPLE: The folder containing sample results that can be viewed without re-calculating predictions, relative to the app root folder (defaults to sample/)
RESULT: A folder that will contain results files for download, relative to the app root folder (defaults to result/)
TEMP: A folder that can be used to temporarily store working files during the prediction process, relative to the app root folder (defaults to temp/)
CAVEAT: Whether to show a caveat stating that the hardware is not capable of predictions (defaults to False)
$ export PYTHONPATH=$PYTHONPATH:/home/ubuntu/projects/cmpt733finalproject/Code/webApp/src:/home/ubuntu/projects/cmpt733finalproject/Code/3_tileAnnotate/src
$ python3 app.py --darknetpath ../../../../darknet --cfg model.cfg --weights model.weights --out app.out --err app.err --sample ../output/sample --result ../output/result --temp ../output/temp --caveat True