实时行人分析工具PP-Human中集成了视频分类的打架识别模块。本文档介绍如何基于PaddleVideo,完成打架识别模型的训练流程。
目前打架识别模型使用的是PP-TSM,并在PP-TSM视频分类模型训练流程的基础上修改适配,完成模型训练。
请先参考使用说明了解PaddleVideo模型库的使用。
打架识别静态图模型获取https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM_fight.zip。
打架识别demo。
首先需要将下载好的静态图模型解压并放到inference
目录下,然后执行下面的命令即可直接判断一个给定的视频中是否存在打架行为:
cd ${PaddleVideo_root}
python tools/predict.py --input_file fight.avi \
--config pptsm_fight_frames_dense.yaml \
--model_file inference/ppTSM/ppTSM.pdmodel \
--params_file inference/ppTSM/ppTSM.pdiparams \
--use_gpu=True \
--use_tensorrt=False
PP-TSM是一个基于视频片段进行预测的模型。在PaddleVideo中,训练数据为.mp4
、.avi
等格式视频或者是抽帧后的视频帧序列,标签则可以是.txt
格式存储的文件。
本项目基于6个公开的打架、暴力行为相关数据集合并后的数据进行模型训练。公开数据集具体信息如下:
数据集 | 下载连接 | 简介 | 标注 | 数量 | 时长 |
---|---|---|---|---|---|
Surveillance Camera Fight Dataset | https://github.com/sayibet/fight-detection-surv-dataset | 裁剪视频,监控视角 | 视频级别 | 打架:150;非打架:150 | 2s |
A Dataset for Automatic Violence Detection in Videos | https://github.com/airtlab/A-Dataset-for-Automatic-Violence-Detection-in-Videos | 裁剪视频,室内自行录制 | 视频级别 | 暴力行为:115个场景,2个机位,共230 ;非暴力行为:60个场景,2个机位,共120 | 几秒钟 |
Hockey Fight Detection Dataset | https://www.kaggle.com/datasets/yassershrief/hockey-fight-vidoes?resource=download | 裁剪视频,非真实场景 | 视频级别 | 打架:500;非打架:500 | 2s |
Video Fight Detection Dataset | https://www.kaggle.com/datasets/naveenk903/movies-fight-detection-dataset | 裁剪视频,非真实场景 | 视频级别 | 打架:100;非打架:101 | 2s |
Real Life Violence Situations Dataset | https://www.kaggle.com/datasets/mohamedmustafa/real-life-violence-situations-dataset | 裁剪视频,非真实场景 | 视频级别 | 暴力行为:1000;非暴力行为:1000 | 几秒钟 |
UBI Abnormal Event Detection Dataset | http://socia-lab.di.ubi.pt/EventDetection/ | 未裁剪视频,监控视角 | 帧级别 | 打架:216;非打架:784;裁剪后二次标注:打架1976,非打架1630 | 原视频几秒到几分钟不等,裁剪后2s |
打架(暴力行为)视频3956个,非打架(非暴力行为)视频3501个,共7457个视频,每个视频几秒钟。
为了加快训练速度,将视频进行抽帧。
cd ${PaddleVideo_root}
python data/ucf101/extract_rawframes.py dataset/ rawframes/ --level 2 --ext mp4
其中,视频存放在dataset
目录下,打架(暴力)视频存放在dataset/fight
中;非打架(非暴力)视频存放在dataset/nofight
中。rawframes
目录存放抽取的视频帧。
本项目验证集1500条,来自Surveillance Camera Fight Dataset、A Dataset for Automatic Violence Detection in Videos、UBI Abnormal Event Detection Dataset三个数据集。
也可根据下面的代码将数据按照0.8:0.2的比例划分成训练集和测试集:
import os
import glob
import random
import fnmatch
import re
class_id = {
"nofight":0,
"fight":1
}
def get_list(path,key_func=lambda x: x[-11:], rgb_prefix='img_', level=1):
if level == 1:
frame_folders = glob.glob(os.path.join(path, '*'))
elif level == 2:
frame_folders = glob.glob(os.path.join(path, '*', '*'))
else:
raise ValueError('level can be only 1 or 2')
def count_files(directory):
lst = os.listdir(directory)
cnt = len(fnmatch.filter(lst, rgb_prefix + '*'))
return cnt
# check RGB
video_dict = {}
for f in frame_folders:
cnt = count_files(f)
k = key_func(f)
if level==2:
k = k.split("/")[0]
video_dict[f]=str(cnt)+" "+str(class_id[k])
return video_dict
def fight_splits(video_dict, train_percent=0.8):
videos = list(video_dict.keys())
train_num = int(len(videos)*train_percent)
train_list = []
val_list = []
random.shuffle(videos)
for i in range(train_num):
train_list.append(videos[i]+" "+str(video_dict[videos[i]]))
for i in range(train_num,len(videos)):
val_list.append(videos[i]+" "+str(video_dict[videos[i]]))
print("train:",len(train_list),",val:",len(val_list))
with open("fight_train_list.txt","w") as f:
for item in train_list:
f.write(item+"\n")
with open("fight_val_list.txt","w") as f:
for item in val_list:
f.write(item+"\n")
frame_dir = "rawframes"
level = 2
train_percent = 0.8
if level == 2:
def key_func(x):
return '/'.join(x.split('/')[-2:])
else:
def key_func(x):
return x.split('/')[-1]
video_dict = get_list(frame_dir, key_func=key_func, level=level)
print("number:",len(video_dict))
fight_splits(video_dict, train_percent)
最终生成fight_train_list.txt和fight_val_list.txt两个文件。打架的标签为1,非打架的标签为0。
对于未裁剪的视频,需要先进行裁剪才能用于模型训练,这个给出视频裁剪的函数cut_video
,输入为视频路径,裁剪的起始帧和结束帧以及裁剪后的视频保存路径。
import cv2
def cut_video(video_path, frameToStart, frametoStop, saved_video_path):
cap = cv2.VideoCapture(video_path)
FPS = cap.get(cv2.CAP_PROP_FPS)
#print("FPS:",FPS)
TOTAL_FRAME = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) # 获取视频总帧数
#print("TOTAL_FRAME:",TOTAL_FRAME)
size = (cap.get(cv2.CAP_PROP_FRAME_WIDTH), cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
#print("size:",size)
videoWriter =cv2.VideoWriter(saved_video_path,apiPreference = 0,fourcc = cv2.VideoWriter_fourcc(*'mp4v'),fps=FPS,
frameSize=(int(size[0]),int(size[1])))
COUNT = 0
while True:
success, frame = cap.read()
if success:
COUNT += 1
if COUNT <= frametoStop and COUNT > frameToStart: # 选取起始帧
videoWriter.write(frame)
else:
print("cap.read failed!")
break
if COUNT > frametoStop:
break
cap.release()
videoWriter.release()
print(saved_video_path)
下载预训练模型:
wget https://videotag.bj.bcebos.com/PaddleVideo/PretrainModel/ResNet50_vd_ssld_v2_pretrained.pdparams
模型训练:
# 单卡训练
cd ${PaddleVideo_root}
python main.py --validate -c pptsm_fight_frames_dense.yaml
cd ${PaddleVideo_root}
# 多卡训练
export CUDA_VISIBLE_DEVICES=0,1,2,3
python -B -m paddle.distributed.launch --gpus=“0,1,2,3” \
--log_dir=log_pptsm_dense main.py --validate \
-c pptsm_fight_frames_dense.yaml
训练好的模型下载:https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM_fight.pdparams
模型评估:
cd ${PaddleVideo_root}
python main.py --test -c pptsm_fight_frames_dense.yaml \
-w ppTSM_fight_best.pdparams
其中ppTSM_fight_best.pdparams
为训练好的模型。
导出inference模型:
cd ${PaddleVideo_root}
python tools/export_model.py -c pptsm_fight_frames_dense.yaml \
-p ppTSM_fight_best.pdparams \
-o inference/ppTSM