[Bug] Tensorrt will slow down other mmdetection and mmpose missions on cpu, can it be fixed?

[aixiaodewugege]

Checklist

• I have searched related issues but cannot get the expected help.
• 2. I have read the FAQ documentation but cannot get the expected help.
• 3. The bug has not been fixed in the latest version.

Describe the bug

Hi, I found when I using trt in action recognition, it would heavily slow down my detection and pose estimation model on other thread. I put detection and pose estimation on cpu thread, and action model on gpu thread to let them run more smoothly. But after using trt in gpu, the fps for cpu model became much worse, even if I didn't change them.

Reproduction

Originally posted by @aixiaodewugege in #1964 (comment)

Environment

03/28 21:53:33 - mmengine - INFO - **********Environmental information**********
03/28 21:53:33 - mmengine - INFO - sys.platform: linux
03/28 21:53:33 - mmengine - INFO - Python: 3.7.16 (default, Jan 17 2023, 22:20:44) [GCC 11.2.0]
03/28 21:53:33 - mmengine - INFO - CUDA available: True
03/28 21:53:33 - mmengine - INFO - numpy_random_seed: 2147483648
03/28 21:53:33 - mmengine - INFO - GPU 0: NVIDIA GeForce RTX 3090
03/28 21:53:33 - mmengine - INFO - CUDA_HOME: /usr/local/cuda
03/28 21:53:33 - mmengine - INFO - NVCC: Cuda compilation tools, release 11.7, V11.7.64
03/28 21:53:33 - mmengine - INFO - GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
03/28 21:53:33 - mmengine - INFO - PyTorch: 1.13.1+cu117
03/28 21:53:33 - mmengine - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.7
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.8.1  (built against CUDA 11.8)
    - Built with CuDNN 8.5
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

03/28 21:53:33 - mmengine - INFO - TorchVision: 0.14.1+cu117
03/28 21:53:33 - mmengine - INFO - OpenCV: 4.5.5
03/28 21:53:33 - mmengine - INFO - MMEngine: 0.6.0
03/28 21:53:33 - mmengine - INFO - MMCV: 2.0.0rc4
03/28 21:53:33 - mmengine - INFO - MMCV Compiler: GCC 9.3
03/28 21:53:33 - mmengine - INFO - MMCV CUDA Compiler: 11.7
03/28 21:53:33 - mmengine - INFO - MMDeploy: 1.0.0rc3+032ce75
03/28 21:53:33 - mmengine - INFO - 

03/28 21:53:33 - mmengine - INFO - **********Backend information**********
03/28 21:53:33 - mmengine - INFO - tensorrt:    8.4.2.4
03/28 21:53:33 - mmengine - INFO - tensorrt custom ops: Available
03/28 21:53:33 - mmengine - INFO - ONNXRuntime: None
03/28 21:53:33 - mmengine - INFO - ONNXRuntime-gpu:     1.8.1
03/28 21:53:33 - mmengine - INFO - ONNXRuntime custom ops:      Available
03/28 21:53:33 - mmengine - INFO - pplnn:       None
03/28 21:53:33 - mmengine - INFO - ncnn:        None
03/28 21:53:33 - mmengine - INFO - snpe:        None
03/28 21:53:33 - mmengine - INFO - openvino:    2022.3.0
03/28 21:53:33 - mmengine - INFO - torchscript: 1.13.1
03/28 21:53:33 - mmengine - INFO - torchscript custom ops:      NotAvailable
03/28 21:53:33 - mmengine - INFO - rknn-toolkit:        None
03/28 21:53:33 - mmengine - INFO - rknn-toolkit2:       None
03/28 21:53:33 - mmengine - INFO - ascend:      None
03/28 21:53:33 - mmengine - INFO - coreml:      None
03/28 21:53:33 - mmengine - INFO - tvm: None
03/28 21:53:33 - mmengine - INFO - vacc:        None
03/28 21:53:33 - mmengine - INFO - 

03/28 21:53:33 - mmengine - INFO - **********Codebase information**********
03/28 21:53:33 - mmengine - INFO - mmdet:       3.0.0rc6
03/28 21:53:33 - mmengine - INFO - mmseg:       None
03/28 21:53:33 - mmengine - INFO - mmcls:       None
03/28 21:53:33 - mmengine - INFO - mmocr:       None
03/28 21:53:33 - mmengine - INFO - mmedit:      None
03/28 21:53:33 - mmengine - INFO - mmdet3d:     None
03/28 21:53:33 - mmengine - INFO - mmpose:      1.0.0rc1
03/28 21:53:33 - mmengine - INFO - mmrotate:    None
03/28 21:53:33 - mmengine - INFO - mmaction:    1.0.0rc3
03/28 21:53:33 - mmengine - INFO - mmrazor:     None

Error traceback

No response

[irexyc]

Does the tensorrt slow down your fps or due to multi model infernence simultaneously? If you replace the tensorrt with pytorch, does it improve fps?

---

The detection, pose estimation and mmaction model are all competing for GPUresources. You could run each model independently and use nvidia-smi to see the GPU-Util of each model. If each has a high cpu/gpu usage, there is no doubt that the speed will slow down when you run three models simultaneously.

Therefore, you may implement a strategy like web-cam demo in mmaction2 for recognition model. Just control the fps of camera display and maintain a queue with fixed length, draw result when the model have a output, use previous result when the model is not fast enough.

[aixiaodewugege]

I put detection and pose model on CPU, action on GPU, for they are much lighter comparing to action model.

I have used a queue to let the result match with latest frame. When I use pytorch format in all three model, it looks good. Detection ,pose could reach 30 fps and action was 0.5 fps. But after I use trt format in action model. The action can reach 10fps, but the detection and pose drop to 4 fps. And the display thread was inferenced by the action model cline to 5 fps.

I am curious that why mission runing on gpu will inference all thread that on cpu? Is it because of data processing?

[irexyc]

Detection ,pose could reach 30 fps and action was 0.5 fps.

Are these three model run simultaneously? Both on cpu or both on cuda or detection, pose on cpu and action on cuda?

The VideoRecgnizer node should only consume one core of cpu and shouldn't affect much of other node if they don't have data dependencies.

Since mmaction use much gpu calculation resource and you want to avoid competition for gpu resources, you need to make sure that the data process and model inference of other models don't use cuda. You could run each model separately to confirm it.

And what I said mmaction web-camera strategy in last reply is that you can always control the fps in visual thread, but make other model(thread) to adapt to it. From your reply I think you may not get my point.

[aixiaodewugege]

Are these three model run simultaneously? Both on cpu or both on cuda or detection, pose on cpu and action on cuda?

detection, pose on cpu, action on cuda. I have setedt detect ,pose model ,init and inference on cpu. I tested it as you recommended.

For the mmaction web-camera strategy , I think I get your point. Display thread will do its job at setted fps, and the results from models will update the update the latest frame.

It works fine before I use trt for action model.

[irexyc]

Since every node use one thread, it takes up at most one core. I don't think the action node will consume cpu resource.

You can add some time point and profile yourself to see if it is due to data dependencies that caused the problem.

You could set the display thread to draw images the same speed with camera fps, but reduce action fps to 1 or some value (skip inference)

[aixiaodewugege]

Hi, I have confirmed that it is not due to data dependencies.

After I set the display thread to draw images the same speed with camera fps, but reduce action fps to 1, all the threads will be blocked by action trt model when it is runing, and go ahead when it is sleeping.

Here are the inference time record, of three model. You can see that detection model is heavily affected by action model at the moment.

detection 0.03336854791268706
pose 0.014350187964737415
detection 0.036842640955001116
pose 0.013412263011559844
detection 0.03686305298469961
pose 0.017130172112956643
detection 0.030933654867112637
pose 0.014054940780624747
action trt sdk 0.20969921117648482
detection 0.2437018440105021
pose 0.013913026079535484
detection 0.0341306971386075
pose 0.015826564049348235
detection 0.0316423149779439
pose 0.018784271087497473
detection 0.03421466099098325
pose 0.018799545941874385
detection 0.0344116201158613