Why does AP and AR decrease if we increase the number of key points

[rubeea]

I am training a custom dataset with HRNet, HRNet+UDP and ResNet50. I am testing two annotation schemes: 6 key points versus 3 key points. If I use 6 key points to label my dataset, the performance of the model (AP,AR) is worse but the inference speed is better than that achieved with 3 key points. However, logically if the number of key points is more, the precision should increase and the speed should decrease right? Any reason why 3 key points give better AP and AR and less speed as compared to 6 key points?
Any help will be highly appreciated.

[jin-s13]

Increasing the number of keypoints will lead to performance drop. This is reasonable. Because the model needs more capacity to learn more things.
But the increase of the inference speed is surprising. How about testing the inference speed for multiple times?

[rubeea]

@jin-s13 I have one more question regarding your explanation. The performance metrics (mAP, AR etc.) are calculated using the OKS measure. How is OKS affected by the number of key points? mAP and AR values are averaged across all OKS values and OKS is defined according to https://cocodataset.org/#keypoints-eval so does increasing the number of key points affect the OKS values and in turn affect the performance metrics?

[rubeea]

@jin-s13 yes I tested on 3 different datasets with HRNet and HRNet+UDP. I am getting similar results in all cases, the inference speed is higher for 6 key points as compared to 3 key points. I used the following script to calculate the inference time:
https://github.com/open-mmlab/mmpose/blob/master/tools/analysis/benchmark_inference.py

How many times should I run this script approximately to check for accurate results?

[jin-s13]

What is the inference speed in your case？Could you provide some examples? 5 FPS, 10FPS?
Is the difference only marginal or large?

How many times should I run this script approximately to check for accurate results?

I am not sure. Maybe 3 or 5 times?

[rubeea]

The results for HRNet are as follows:
Dataset Key points AP AR Inference Speed (items/s) Overall Time (s)
PLDU 6 38.8 52.5 12.25 25.94 s
PLDU 3 57.9 70.1 8.85 35.93 s
PLDM 6 42.2 53.1 8.84 10.58 s
PLDM 3 49 58.8 6.63 14.14 s

[rubeea]

The results for HRNet are as follows: Dataset Key points AP AR Inference Speed (items/s) Overall Time (s) PLDU 6 38.8 52.5 12.25 25.94 s PLDU 3 57.9 70.1 8.85 35.93 s PLDM 6 42.2 53.1 8.84 10.58 s PLDM 3 49 58.8 6.63 14.14 s

I am running my experiments on Google colab. Can this be due to the fact that Colab assigns a different GPU each time?

[rubeea]

@jin-s13 I realized the difference in inference speeds was due to the fact that Colab was assigning a different GPU for every experiment. On the same GPU, the inference speed of 3 key points was much higher than 6 key points.