Video Detection Module Development Tutorial¶
I. Overview¶
Video detection tasks are a critical component of computer vision systems, focusing on identifying and locating objects or events within video sequences. Video detection involves decomposing the video into individual frame sequences and then analyzing these frames to recognize detected objects or actions, such as detecting pedestrians in surveillance videos or identifying specific activities like "running," "jumping," or "playing guitar" in sports or entertainment videos.
The output of the video detection module includes bounding boxes and class labels for each detected object or event. This information can be used by other modules or systems for further analysis, such as tracking the movement of detected objects, generating alerts, or compiling statistics for decision-making processes. Therefore, video detection plays an important role in various applications ranging from security surveillance and autonomous driving to sports analytics and content moderation.
II. List of Supported Models¶
Model | Model Download Link | Frame-mAP (@ IoU 0.5) | Model Storage Size (M) | Description |
---|---|---|---|---|
YOWO | Inference Model/čŽįģæ¨Ąå | 80.94 | 462.891M | YOWO is a single-stage network with two branches. One branch extracts spatial features of key frames (i.e., the current frame) through a 2D-CNN, while the other branch captures spatiotemporal features of a clip composed of previous frames using a 3D-CNN. To accurately aggregate these features, YOWO employs a channel fusion and attention mechanism to maximize the utilization of inter-channel dependencies. Finally, the fused features are used for frame-level detection. |
Test Environment Description:
- Performance Test Environment
- Test Dataset: UCF101-24 test set.
-
Hardware Configuration:
- GPU: NVIDIA Tesla T4
- CPU: Intel Xeon Gold 6271C @ 2.60GHz
- Other Environments: Ubuntu 20.04 / cuDNN 8.6 / TensorRT 8.5.2.2
-
Inference Mode Description
Mode | GPU Configuration | CPU Configuration | Acceleration Technology Combination |
---|---|---|---|
Normal Mode | FP32 Precision / No TRT Acceleration | FP32 Precision / 8 Threads | PaddleInference |
High-Performance Mode | Optimal combination of pre-selected precision types and acceleration strategies | FP32 Precision / 8 Threads | Pre-selected optimal backend (Paddle/OpenVINO/TRT, etc.) |
III. Quick Integration¶
â Before quick integration, please install the PaddleX wheel package. For detailed instructions, refer to the PaddleX Local Installation Guide.
After installing the wheel package, you can complete video Detection module inference with just a few lines of code. You can switch between models in this module freely, and you can also integrate the model inference of the video Detection module into your project. Before running the following code, please download the demo video to your local machine.
from paddlex import create_model
model = create_model(model_name="YOWO")
output = model.predict(input="HorseRiding.avi", batch_size=1)
for res in output:
res.print()
res.save_to_video(save_path="./output/")
res.save_to_json(save_path="./output/res.json")
đ After running, the result is: (Click to expand)
{'res': {'input_path': 'HorseRiding.avi', 'result': [[[[110, 40, 170, 171], 0.8385784886274905, 'HorseRiding']], [[[112, 31, 168, 167], 0.8587647461352432, 'HorseRiding']], [[[106, 28, 164, 165], 0.8579590929730969, 'HorseRiding']], [[[106, 24, 165, 171], 0.8743957465404151, 'HorseRiding']], [[[107, 22, 165, 172], 0.8488322619908999, 'HorseRiding']], [[[112, 22, 173, 171], 0.8446755521458691, 'HorseRiding']], [[[115, 23, 177, 176], 0.8454028365262367, 'HorseRiding']], [[[117, 22, 178, 179], 0.8484261880748285, 'HorseRiding']], [[[117, 22, 181, 181], 0.8319480115446183, 'HorseRiding']], [[[117, 39, 182, 183], 0.820551099084625, 'HorseRiding']], [[[117, 41, 183, 185], 0.8202395831914338, 'HorseRiding']], [[[121, 47, 185, 190], 0.8261058921745246, 'HorseRiding']], [[[123, 46, 188, 196], 0.8307278306829033, 'HorseRiding']], [[[125, 44, 189, 197], 0.8259781361122833, 'HorseRiding']], [[[128, 47, 191, 195], 0.8227593229866699, 'HorseRiding']], [[[127, 44, 192, 193], 0.8205373129456817, 'HorseRiding']], [[[129, 39, 192, 185], 0.8223318812628619, 'HorseRiding']], [[[127, 31, 196, 179], 0.8501208612019866, 'HorseRiding']], [[[128, 22, 193, 171], 0.8315708410681566, 'HorseRiding']], [[[130, 22, 192, 169], 0.8318588228062005, 'HorseRiding']], [[[132, 18, 193, 170], 0.8310494469100611, 'HorseRiding']], [[[132, 18, 194, 172], 0.8302132445350239, 'HorseRiding']], [[[133, 18, 194, 176], 0.8339063714162727, 'HorseRiding']], [[[134, 26, 200, 183], 0.8365876380675275, 'HorseRiding']], [[[133, 16, 198, 182], 0.8395230321418268, 'HorseRiding']], [[[133, 17, 199, 184], 0.8198139782724922, 'HorseRiding']], [[[140, 28, 204, 189], 0.8344166596681291, 'HorseRiding']], [[[139, 27, 204, 187], 0.8412694521771158, 'HorseRiding']], [[[139, 28, 204, 185], 0.8500098862888805, 'HorseRiding']], [[[135, 19, 199, 179], 0.8506627974981384, 'HorseRiding']], [[[132, 15, 201, 178], 0.8495054272547193, 'HorseRiding']], [[[136, 14, 199, 173], 0.8451630721500223, 'HorseRiding']], [[[136, 12, 200, 167], 0.8366456814214907, 'HorseRiding']], [[[133, 8, 200, 168], 0.8457252233401213, 'HorseRiding']], [[[131, 7, 197, 162], 0.8400586356358062, 'HorseRiding']], [[[131, 8, 195, 163], 0.8320492682901985, 'HorseRiding']], [[[129, 4, 194, 159], 0.8298043752822792, 'HorseRiding']], [[[127, 5, 194, 162], 0.8348390851948722, 'HorseRiding']], [[[125, 7, 190, 164], 0.8299688814865505, 'HorseRiding']], [[[125, 6, 191, 164], 0.8303107088154711, 'HorseRiding']], [[[123, 8, 190, 168], 0.8348342187965798, 'HorseRiding']], [[[125, 14, 189, 170], 0.8356523950497134, 'HorseRiding']], [[[127, 18, 191, 171], 0.8392671764931521, 'HorseRiding']], [[[127, 30, 193, 178], 0.8441704160826191, 'HorseRiding']], [[[128, 18, 190, 181], 0.8438125326146775, 'HorseRiding']], [[[128, 12, 189, 186], 0.8390128962093542, 'HorseRiding']], [[[129, 15, 190, 185], 0.8471056476788448, 'HorseRiding']], [[[129, 16, 191, 184], 0.8536121834731034, 'HorseRiding']], [[[129, 16, 192, 185], 0.8488154629800881, 'HorseRiding']], [[[128, 15, 194, 184], 0.8417711698421471, 'HorseRiding']], [[[129, 13, 195, 187], 0.8412510238991473, 'HorseRiding']], [[[129, 14, 191, 187], 0.8404350980083457, 'HorseRiding']], [[[129, 13, 190, 189], 0.8382891279858882, 'HorseRiding']], [[[129, 11, 187, 191], 0.8318282305903217, 'HorseRiding']], [[[128, 8, 188, 195], 0.8043430817880264, 'HorseRiding']], [[[131, 25, 193, 199], 0.826184954516826, 'HorseRiding']], [[[124, 35, 191, 203], 0.8270462809459467, 'HorseRiding']], [[[121, 38, 191, 206], 0.8350931715324705, 'HorseRiding']], [[[124, 41, 195, 212], 0.8331239341053625, 'HorseRiding']], [[[128, 42, 194, 211], 0.8343046153103657, 'HorseRiding']], [[[131, 40, 192, 203], 0.8309784496027532, 'HorseRiding']], [[[130, 32, 195, 202], 0.8316640083647542, 'HorseRiding']], [[[135, 30, 196, 197], 0.8272172409555161, 'HorseRiding']], [[[131, 16, 197, 186], 0.8388410406147955, 'HorseRiding']], [[[134, 15, 202, 184], 0.8485738297037244, 'HorseRiding']], [[[136, 15, 209, 182], 0.8529430205135213, 'HorseRiding']], [[[134, 13, 218, 182], 0.8601191479922718, 'HorseRiding']], [[[144, 10, 213, 183], 0.8591963099263467, 'HorseRiding']], [[[151, 12, 219, 184], 0.8617965108346937, 'HorseRiding']], [[[151, 10, 220, 186], 0.8631923599955371, 'HorseRiding']], [[[145, 10, 216, 186], 0.8800860885204287, 'HorseRiding']], [[[144, 10, 216, 186], 0.8858840451538228, 'HorseRiding']], [[[146, 11, 214, 190], 0.8773644144886106, 'HorseRiding']], [[[145, 24, 214, 193], 0.8605544385867248, 'HorseRiding']], [[[146, 23, 214, 193], 0.8727294882672254, 'HorseRiding']], [[[148, 22, 212, 198], 0.8713131467067079, 'HorseRiding']], [[[146, 29, 213, 197], 0.8579099324651196, 'HorseRiding']], [[[154, 29, 217, 199], 0.8547794072847914, 'HorseRiding']], [[[151, 26, 217, 203], 0.8641733722316758, 'HorseRiding']], [[[146, 24, 212, 199], 0.8613466257602624, 'HorseRiding']], [[[142, 25, 210, 194], 0.8492670944810214, 'HorseRiding']], [[[134, 24, 204, 192], 0.8428117300203049, 'HorseRiding']], [[[136, 25, 204, 189], 0.8486779356971397, 'HorseRiding']], [[[127, 21, 199, 179], 0.8513896296400709, 'HorseRiding']], [[[125, 10, 192, 192], 0.8510201771386576, 'HorseRiding']], [[[124, 8, 191, 192], 0.8493999019508465, 'HorseRiding']], [[[121, 8, 192, 193], 0.8487097098892171, 'HorseRiding']], [[[119, 6, 187, 193], 0.847543279648022, 'HorseRiding']], [[[118, 12, 190, 190], 0.8503535936620565, 'HorseRiding']], [[[122, 22, 189, 194], 0.8427901493276977, 'HorseRiding']], [[[118, 24, 188, 195], 0.8418835400352087, 'HorseRiding']], [[[120, 25, 188, 205], 0.847192725785284, 'HorseRiding']], [[[122, 25, 189, 207], 0.8444105420674433, 'HorseRiding']], [[[120, 23, 189, 208], 0.8470784016639392, 'HorseRiding']], [[[121, 23, 188, 205], 0.843428111269418, 'HorseRiding']], [[[117, 23, 186, 206], 0.8420809714166708, 'HorseRiding']], [[[119, 5, 199, 197], 0.8288265053231356, 'HorseRiding']], [[[121, 8, 192, 195], 0.8197548738023599, 'HorseRiding']]]}}
[xmin, ymin, xmax, ymax]
: The coordinates of the top-left and bottom-right corners of the bounding box, both are floating-point numbers.
- float: The confidence score of the bounding box, a floating-point number.
- string: The category to which the bounding box belongs, a string.
The visualized video is as follows:
In the above Python script, the following steps are executed:
create_model
instantiates the video detection model (YOWO
), with specific explanations as follows:
Parameter | Parameter Description | Parameter Type | Options | Default Value |
---|---|---|---|---|
model_name |
Name of the model | str |
All model names supported by PaddleX | None |
model_dir |
Path to store the model | str |
None | None |
nms_thresh |
The IoU threshold parameter in the Non-Maximum Suppression (NMS) process; if not specified, the default configuration of the PaddleX official model will be used | float/None |
Floating-point number greater than 0/None | None |
score_thresh |
The prediction confidence threshold; if not specified, the default configuration of the PaddleX official model will be used | float/None |
Floating-point number greater than 0/None | None |
- The
predict
method of the video classification model is called to perform inference and prediction. The parameter of thepredict
method isinput
, which is used to input the data to be predicted and supports multiple input types, with specific explanations as follows:
Parameter | Parameter Description | Parameter Type | Options | Default Value |
---|---|---|---|---|
input |
Data to be predicted, supporting multiple input types | Python Var /str /list |
|
None |
batch_size |
Batch size | int |
None | 1 |
nms_thresh |
The IoU threshold parameter in the Non-Maximum Suppression (NMS) process | float|None |
|
None |
score_thresh |
Prediction confidence threshold | float|None |
|
None |
- Process the prediction results. Each sample's prediction result is a corresponding Result object, and it supports operations such as printing, saving as an image, and saving as a
json
file:
Method | Method Description | Parameter | Parameter Type | Parameter Description | Default Value |
---|---|---|---|---|---|
print() |
Print the result to the terminal | format_json |
bool |
Whether to format the output content using json indentation |
True |
indent |
int |
JSON formatting setting, only effective when format_json is True |
4 | ||
ensure_ascii |
bool |
JSON formatting setting, only effective when format_json is True |
False |
||
save_to_json() |
Save the result as a JSON file | save_path |
str |
The path to save the file. When it is a directory, the saved file name is consistent with the input file name | None |
indent |
int |
JSON formatting setting | 4 | ||
ensure_ascii |
bool |
JSON formatting setting | False |
||
save_to_video() |
Save the result as a video file | save_path |
str |
The path to save the file. When it is a directory, the saved file name is consistent with the input file name | None |
- Additionally, it supports obtaining the result visualization video and
json
results through attributes:
Attribute | Attribute Description |
---|---|
json |
Get the prediction result in json format |
video |
Get the visualization video and frame rate in dict format. Here, the visualization video is an np.array array with dimensions (number of video frames, video height, video width, number of video channels) |
For more information on using PaddleX's single-model inference APIs, please refer to the PaddleX Single-Model Python Script Usage Instructions.
IV. Custom Development¶
If you are seeking higher accuracy from existing models, you can use PaddleX's custom development capabilities to develop better video Detection models. Before using PaddleX to develop video Detection models, please ensure that you have installed the relevant model training plugins for video Detection in PaddleX. The installation process can be found in the custom development section of the PaddleX Local Installation Guide.
4.1 Data Preparation¶
Before model training, you need to prepare the dataset for the corresponding task module. PaddleX provides data validation functionality for each module, and only data that passes data validation can be used for model training. Additionally, PaddleX provides demo datasets for each module, which you can use to complete subsequent development. If you wish to use your own private dataset for subsequent model training, please refer to the PaddleX Video Detection Task Module Data Annotation Guide.
4.1.1 Demo Data Download¶
You can use the following command to download the demo dataset to a specified folder:
cd /path/to/paddlex
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/data/video_det_examples.tar -P ./dataset
tar -xf ./dataset/video_det_examples.tar -C ./dataset/
4.1.2 Data Validation¶
One command is all you need to complete data validation:
python main.py -c paddlex/configs/modules/video_detection/YOWO.yaml \
-o Global.mode=check_dataset \
-o Global.dataset_dir=./dataset/video_det_examples
Check dataset passed !
in the log. The validation results file is saved in ./output/check_dataset_result.json
, and related outputs are saved in the ./output/check_dataset
directory in the current directory, including visual examples of sample images and sample distribution histograms.
đ Validation Results Details (Click to Expand)
{
"done_flag": true,
"check_pass": true,
"attributes": {
"label_file": "..\/..\/dataset\/video_det_examples\/label_map.txt",
"num_classes": 24,
"train_samples": 6878,
"train_sample_paths": [
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/SoccerJuggling\/v_SoccerJuggling_g19_c06\/00296.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/SkateBoarding\/v_SkateBoarding_g17_c04\/00026.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/RopeClimbing\/v_RopeClimbing_g01_c03\/00055.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/HorseRiding\/v_HorseRiding_g11_c05\/00132.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/PoleVault\/v_PoleVault_g13_c03\/00089.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/Basketball\/v_Basketball_g13_c04\/00050.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/PoleVault\/v_PoleVault_g01_c05\/00024.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/RopeClimbing\/v_RopeClimbing_g03_c04\/00118.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/GolfSwing\/v_GolfSwing_g01_c06\/00231.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/TrampolineJumping\/v_TrampolineJumping_g02_c02\/00134.jpg"
],
"val_samples": 3916,
"val_sample_paths": [
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/IceDancing\/v_IceDancing_g22_c02\/00017.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/TennisSwing\/v_TennisSwing_g04_c02\/00046.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/SoccerJuggling\/v_SoccerJuggling_g08_c03\/00169.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/Fencing\/v_Fencing_g24_c02\/00009.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/Diving\/v_Diving_g16_c02\/00110.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/HorseRiding\/v_HorseRiding_g08_c02\/00079.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/PoleVault\/v_PoleVault_g17_c07\/00008.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/Skiing\/v_Skiing_g20_c06\/00221.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/PoleVault\/v_PoleVault_g17_c07\/00137.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/GolfSwing\/v_GolfSwing_g24_c01\/00093.jpg"
]
},
"analysis": {
"histogram": "check_dataset\/histogram.png"
},
"dataset_path": "video_det_examples",
"show_type": "video",
"dataset_type": "VideoDetDataset"
}
The above validation results, with check_pass being True, indicate that the dataset format meets the requirements. Explanations for other indicators are as follows:
attributes.num_classes
: The number of classes in this dataset is 24;attributes.train_samples
: The number of training set samples in this dataset is 6878;attributes.val_samples
: The number of validation set samples in this dataset is 3916;attributes.train_sample_paths
: A list of relative paths to the visual samples in the training set of this dataset;attributes.val_sample_paths
: A list of relative paths to the visual samples in the validation set of this dataset;
Additionally, the dataset validation analyzes the sample number distribution across all classes in the dataset and generates a distribution histogram (histogram.png):
4.1.3 Dataset Format Conversion/Dataset Splitting (Optional)¶
After completing data validation, you can convert the dataset format or re-split the training/validation ratio of the dataset by modifying the configuration file or appending hyperparameters.
đ Dataset Format Conversion/Dataset Splitting Details (Click to Expand)
(1) Dataset Format Conversion
Image Detection does not currently support data conversion.
(2) Dataset Splitting
Image Detection does not currently support data conversion.
### 4.2 Model Training A single command can complete the model training. Taking the training of the video Detection model YOWO as an example:python main.py -c paddlex/configs/modules/video_det_examples/YOWO.yaml \
-o Global.mode=train \
-o Global.dataset_dir=./dataset/video_det_examples
đ More Details (Click to Expand)
- During model training, PaddleX automatically saves the model weight files, with the default being
output
. If you need to specify a save path, you can set it through the-o Global.output
field in the configuration file. - PaddleX shields you from the concepts of dynamic graph weights and static graph weights. During model training, both dynamic and static graph weights are produced, and static graph weights are selected by default for model inference.
-
After completing the model training, all outputs are saved in the specified output directory (default is
./output/
), typically including: -
train_result.json
: Training result record file, recording whether the training task was completed normally, as well as the output weight metrics, related file paths, etc.; train.log
: Training log file, recording changes in model metrics and loss during training;config.yaml
: Training configuration file, recording the hyperparameter configuration for this training session;.pdparams
,.pdema
,.pdopt.pdstate
,.pdiparams
,.pdmodel
: Model weight-related files, including network parameters, optimizer, EMA, static graph network parameters, static graph network structure, etc.;
python main.py -c paddlex/configs/modules/video_detection/YOWO.yaml \
-o Global.mode=evaluate \
-o Global.dataset_dir=./dataset/video_det_examples
đ More Details (Click to Expand)
When evaluating the model, you need to specify the model weight file path. Each configuration file has a default weight save path built-in. If you need to change it, simply set it by appending a command line parameter, such as -o Evaluate.weight_path=./output/best_model/best_model.pdparams
.
After completing the model evaluation, an evaluate_result.json
file will be generated, which records the evaluation results. Specifically, it records whether the evaluation task was completed successfully and the model's evaluation metrics, including mAP;
python main.py -c paddlex/configs/modules/video_detection/YOWO.yaml \
-o Global.mode=predict \
-o Predict.model_dir="./output/best_model/inference" \
-o Predict.input="HorseRiding.avi"