Tutorial on Using the Human Keypoint Detection Module¶

I. Overview¶

Human keypoint detection is an important task in the field of computer vision, aiming to identify the specific keypoint locations of the human body in images or videos. By detecting these keypoints, various applications such as pose estimation, action recognition, human-computer interaction, and animation generation can be achieved. Human keypoint detection has a wide range of applications in augmented reality, virtual reality, motion capture, and other fields.

Keypoint detection algorithms mainly include two approaches: Top-Down and Bottom-Up. The Top-Down approach typically relies on an object detection algorithm to identify the bounding boxes of the objects of interest. The input to the keypoint detection model is a cropped single object, and the output is the keypoint prediction result for that object. The model's accuracy is higher, but its speed slows down with an increasing number of objects. In contrast, the Bottom-Up method does not rely on prior object detection but directly performs keypoint detection on the entire image, then groups or connects these points to form multiple pose instances. Its speed is fixed and does not slow down with an increasing number of objects, but its accuracy is lower.

II. Supported Model List¶

Model	Approach	Input Size	AP(0.5:0.95)	GPU Inference Time (ms) [Normal Mode / High-Performance Mode]	CPU Inference Time (ms)	Model Size (M)	Introduction
PP-TinyPose_128x96	Top-Down	128x96	58.4			4.9	PP-TinyPose is a real-time keypoint detection model optimized for mobile devices developed by the Baidu PaddlePaddle Vision Team. It can smoothly perform multi-person pose estimation tasks on mobile devices.
PP-TinyPose_256x192	Top-Down	256x192	68.3			4.9

Test Environment Description:

Performance Test Environment
Test Dataset: The above accuracy metrics are based on the COCO dataset AP(0.5:0.95) using ground truth annotations for bounding boxes.
Hardware Configuration:
- GPU: NVIDIA Tesla T4
- CPU: Intel Xeon Gold 6271C @ 2.60GHz
- Other Environments: Ubuntu 20.04 / cuDNN 8.6 / TensorRT 8.5.2.2
Inference Mode Description

Mode	GPU Configuration	CPU Configuration	Acceleration Technology Combination
Normal Mode	FP32 Precision / No TRT Acceleration	FP32 Precision / 8 Threads	PaddleInference
High-Performance Mode	Optimal combination of pre-selected precision types and acceleration strategies	FP32 Precision / 8 Threads	Pre-selected optimal backend (Paddle/OpenVINO/TRT, etc.)

III. Quick Integration¶

❗ Before quick integration, please install the PaddleX wheel package first. For details, please refer to the PaddleX Local Installation Guide

After completing the installation of the wheel package, you can perform inference for the human keypoint detection module with just a few lines of code. You can switch models under this module at will, and you can also integrate the model inference of the human keypoint detection module into your project. Before running the following code, please download the example image to your local machine.

from paddlex import create_model

model_name = "PP-TinyPose_128x96"

model = create_model(model_name)
output = model.predict("keypoint_detection_002.jpg", batch_size=1)

for res in output:
    res.print(json_format=False)
    res.save_to_img("./output/")
    res.save_to_json("./output/res.json")

👉 The result obtained after running is: (Click to expand)

{'res': {'input_path': 'keypoint_detection_002.jpg', 'kpts': [{'keypoints': [[175.2838134765625, 56.043609619140625, 0.6522828936576843], [181.32794189453125, 49.642051696777344, 0.7338210940361023], [169.46002197265625, 50.59111022949219, 0.6837076544761658], [193.3421173095703, 51.91969680786133, 0.8676544427871704], [164.50787353515625, 55.6519889831543, 0.8232858777046204], [219.7235870361328, 90.28710174560547, 0.8812915086746216], [152.90377807617188, 95.07806396484375, 0.9093065857887268], [233.1095733642578, 149.6704864501953, 0.7706904411315918], [139.5576629638672, 144.38327026367188, 0.7555014491081238], [245.22830200195312, 202.4243927001953, 0.706590473651886], [117.83794403076172, 188.56410217285156, 0.8892115950584412], [203.29542541503906, 200.2967071533203, 0.838330864906311], [172.00791931152344, 201.1993865966797, 0.7636935710906982], [181.18797302246094, 273.0669250488281, 0.8719099164009094], [185.1750030517578, 278.4797668457031, 0.6878190040588379], [171.55068969726562, 362.42730712890625, 0.7994316816329956], [201.6941375732422, 354.5953369140625, 0.6789217591285706]], 'kpt_score': 0.7831441760063171}]}}

Parameter meanings are as follows: - `input_path`: The path of the input image to be predicted. - `kpts`: Information of the predicted keypoints, a list of dictionaries. Each dictionary contains the following information: - `keypoints`: Keypoint coordinate information, a numpy array with shape [num_keypoints, 3], where each keypoint is composed of [x, y, score], and score represents the confidence of that keypoint. - `kpt_score`: The overall confidence of the keypoints, i.e., the average confidence of the keypoints.

The visualization image is as follows:

The explanations for the methods, parameters, etc., are as follows:

create_model instantiates a human keypoint detection model (here, PP-TinyPose_128x96 is used as an example), and the specific explanations are as follows:

Parameter	Parameter Description	Parameter Type	Options	Default Value
`model_name`	Name of the model	`str`	None	None
`model_dir`	Path to store the model	`str`	None	None
`flip`	Whether to perform flipped inference; if True, the model will infer the horizontally flipped input image and fuse the results of both inferences to increase the accuracy of keypoint predictions	`bool`	None	`False`

The model_name must be specified. After specifying model_name, the default model parameters built into PaddleX are used. If model_dir is specified, the user-defined model is used.
The predict() method of the human keypoint detection model is called for inference prediction. The predict() method has parameters input and batch_size, which are explained as follows:

Parameter	Parameter Description	Parameter Type	Options	Default Value
`input`	Data to be predicted, supporting multiple input types	`Python Var`/`str`/`list`	Python variable, such as image data represented by `numpy.ndarray` File path, such as the local path of an image file: `/root/data/img.jpg` URL link, such as the network URL of an image file: Example Local directory, the directory should contain data files to be predicted, such as the local path: `/root/data/` List, the elements of the list should be of the above-mentioned data types, such as `[numpy.ndarray, numpy.ndarray]`, `[\"/root/data/img1.jpg\", \"/root/data/img2.jpg\"]`, `[\"/root/data1\", \"/root/data2\"]`	None
`batch_size`	Batch size	`int`	Any integer	1

The prediction results are processed, and the prediction result for each sample is of type dict. It supports operations such as printing, saving as an image, and saving as a json file:

Method	Method Description	Parameter	Parameter Type	Parameter Description	Default Value
`print()`	Print the results to the terminal	`format_json`	`bool`	Whether to format the output content using `JSON` indentation	`True`
		`indent`	`int`	Specify the indentation level to beautify the output `JSON` data, making it more readable, only effective when `format_json` is `True`	4
		`ensure_ascii`	`bool`	Control whether to escape non-`ASCII` characters to `Unicode`. If set to `True`, all non-`ASCII` characters will be escaped; `False` retains the original characters, only effective when `format_json` is `True`	`False`
`save_to_json()`	Save the results as a JSON file	`save_path`	`str`	The path to save the file. If it is a directory, the saved file name will be consistent with the input file name	None
		`indent`	`int`	Specify the indentation level to beautify the output `JSON` data, making it more readable, only effective when `format_json` is `True`	4
		`ensure_ascii`	`bool`	Control whether to escape non-`ASCII` characters to `Unicode`. If set to `True`, all non-`ASCII` characters will be escaped; `False` retains the original characters, only effective when `format_json` is `True`	`False`
`save_to_img()`	Save the results as an image file	`save_path`	`str`	The path to save the file. If it is a directory, the saved file name will be consistent with the input file name	None

Additionally, it supports obtaining the visualization image with results and the prediction results through attributes, as follows:

Attribute	Attribute Description
`json`	Get the prediction result in `json` format
`img`	Get the visualization image in `dict` format

For more information on using the PaddleX single-model inference API, please refer to the PaddleX Single-Model Python Script Usage Instructions.

IV. Secondary Development¶

If you aim to improve the accuracy of existing models, you can leverage PaddleX's secondary development capabilities to create better keypoint detection models. Before developing keypoint detection models with PaddleX, make sure to install the PaddleDetection plugin for PaddleX. The installation process can be found in the PaddleX Local Installation Guide.

4.1 Data Preparation¶

Before training a model, you need to prepare the dataset for the specific task module. PaddleX provides a data validation feature for each module, and only datasets that pass the validation can be used for model training. Additionally, PaddleX offers demo datasets for each module, which you can use to complete subsequent development based on the official demo data. If you wish to use your private dataset for model training, please refer to the PaddleX Keypoint Detection Data Annotation Guide.

4.1.1 Downloading Demo Data¶

You can use the following commands to download the demo dataset to a specified folder:

cd /path/to/paddlex
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/data/keypoint_coco_examples.tar -P ./dataset
tar -xf ./dataset/keypoint_coco_examples.tar -C ./dataset/

4.1.2 Data Validation¶

A single command can complete the data validation:

python main.py -c paddlex/configs/keypoint_detection/PP-TinyPose_128x96.yaml \
    -o Global.mode=check_dataset \
    -o Global.dataset_dir=./dataset/keypoint_coco_examples

After executing the above command, PaddleX will validate the dataset and summarize its basic information. If the command runs successfully, it will print Check dataset passed ! in the log. The validation result file is saved at ./output/check_dataset_result.json, and related outputs are saved in the ./output/check_datasetdirectory under the current directory. This includes visualized sample images and sample distribution histograms.

👉 Validation Result Details (Click to expand)

The content of the validation result file is as follows:

{
  "done_flag": true,
  "check_pass": true,
  "attributes": {
    "num_classes": 1,
    "train_samples": 500,
    "train_sample_paths": [
      "check_dataset/demo_img/000000560108.jpg",
      "check_dataset/demo_img/000000434662.jpg",
      "check_dataset/demo_img/000000540556.jpg",
      ...
    ],
    "val_samples": 100,
    "val_sample_paths": [
      "check_dataset/demo_img/000000463730.jpg",
      "check_dataset/demo_img/000000085329.jpg",
      "check_dataset/demo_img/000000459153.jpg",
      ...
    ]
  },
  "analysis": {
    "histogram": "check_dataset/histogram.png"
  },
  "dataset_path": "keypoint_coco_examples",
  "show_type": "image",
  "dataset_type": "KeypointTopDownCocoDetDataset"
}

In the above validation results, `check_pass` being `True` indicates that the dataset format meets the requirements. Explanations for other metrics are as follows: * `attributes.num_classes`: The dataset contains 1 class. * `attributes.train_samples`: The training set contains 500 samples. * `attributes.val_samples`: The validation set contains 100 samples. * `attributes.train_sample_paths`: A list of relative paths to visualized training samples. * `attributes.val_sample_paths`: A list of relative paths to visualized validation samples. The data validation also analyzes the sample distribution across all classes in the dataset and generates a histogram (histogram.png): ![Histogram](https://raw.githubusercontent.com/cuicheng01/PaddleX_doc_images/main/images/modules/keypoint_det/01.png)

4.1.3 Dataset Format Conversion / Dataset Splitting (Optional)¶

After completing data validation, you can convert the dataset format or re-split the training/validation ratio by modifying the configuration file or adding hyperparameters.

👉 Details on Format Conversion / Dataset Splitting (Click to expand)

**（1）Dataset Format Conversion** Keypoint detection does not support dataset format conversion. **（2）Dataset Splitting** Parameters for dataset splitting can be set by modifying the fields under `CheckDataset` in the configuration file. Example explanations for some parameters in the configuration file are as follows: * `CheckDataset`: * `split`: * `enable`: Whether to re-split the dataset. Set to `True` to enable dataset splitting. Default is `False`. * `train_percent`: If re-splitting the dataset, set the percentage of the training set. This should be an integer between 0-100, ensuring the sum with `val_percent` equals 100. For example, if you want to re-split the dataset with 90% for training and 10% for validation, modify the configuration file as follows:

......
CheckDataset:
  ......
  split:
    enable: True
    train_percent: 90
    val_percent: 10
  ......
````

Then execute the following command:

```bash
python main.py -c paddlex/configs/keypoint_detection/PP-TinyPose_128x96.yaml \
    -o Global.mode=check_dataset \
    -o Global.dataset_dir=./dataset/keypoint_coco_examples

After the dataset splitting is executed, the original annotation file will be renamed to xxx.bak in the original path. The above parameters can also be set via command-line arguments:

python main.py -c paddlex/configs/keypoint_detection/PP-TinyPose_128x96.yaml  \
    -o Global.mode=check_dataset \
    -o Global.dataset_dir=./dataset/keypoint_coco_examples \
    -o CheckDataset.split.enable=True \
    -o CheckDataset.split.train_percent=90 \
    -o CheckDataset.split.val_percent=10

4.3 Model Evaluation¶

After completing model training, you can evaluate the specified model weight file on the validation set to verify the model's accuracy. Using PaddleX for model evaluation, you can complete the evaluation with a single command:

python main.py -c paddlex/configs/keypoint_detection/PP-TinyPose_128x96.yaml \
    -o Global.mode=evaluate \
    -o Global.dataset_dir=./dataset/keypoint_coco_examples

Similar to model training, the process involves the following steps:

Specify the path to the .yaml configuration file for the model（here it's MobileFaceNet.yaml）
Set the mode to model evaluation: -o Global.mode=evaluate
Specify the path to the validation dataset: -o Global.dataset_dir Other related parameters can be configured by modifying the fields under Global and Evaluate in the .yaml configuration file. For detailed information, please refer to PaddleX Common Configuration Parameters for Models。

👉 More Details (Click to Expand)

During model evaluation, the path to the model weights file needs to be specified. Each configuration file has a default weight save path built in. If you need to change it, you can set it by appending a command line parameter, such as `-o Evaluate.weight_path="./output/best_model/best_model/model.pdparams"`. After completing the model evaluation, an `evaluate_result.json` file will be produced, which records the evaluation results. Specifically, it records whether the evaluation task was completed normally and the model's evaluation metrics, including Accuracy.

4.4 Model Inference¶

After completing model training and evaluation, you can use the trained model weights for inference predictions. In PaddleX, model inference predictions can be implemented through two methods: command line and wheel package.

4.4.1 Model Inference¶

To perform inference predictions through the command line, you only need the following command. Before running the following code, please download the example image to your local machine.
```
python main.py -c paddlex/configs/keypoint_detection/PP-TinyPose_128x96.yaml \
    -o Global.mode=predict \
    -o Predict.model_dir="./output/best_model/inference" \
    -o Predict.input="keypoint_detection_002.jpg"
```
Similar to model training and evaluation, the following steps are required:
Specify the path to the model's .yaml configuration file (here it is MobileFaceNet.yaml)
Specify the mode as model inference prediction: -o Global.mode=predict
Specify the path to the model weights: -o Predict.model_dir="./output/best_model/inference"
Specify the path to the input data: -o Predict.input="..." Other related parameters can be set by modifying the fields under Global and Predict in the .yaml configuration file. For details, please refer to PaddleX Common Model Configuration File Parameter Description.

4.4.2 Model Integration¶

The model can be directly integrated into the PaddleX pipeline or into your own project.

Pipeline Integration

The human keypoint detection module can be integrated into the PaddleX pipeline for human keypoint detection. Simply replacing the model path will update the human keypoint detection module in the relevant pipeline. In pipeline integration, you can deploy your model using high-performance deployment or service-oriented deployment.

Module Integration

The weights you produced can be directly integrated into the face feature module. You can refer to the Python example code in Quick Integration and only need to replace the model with the path to the model you trained.

You can also use the PaddleX high-performance inference plugin to optimize the inference process of your model and further improve efficiency. For detailed procedures, please refer to the PaddleX High-Performance Inference Guide.