Folder Structure Using DeepLabCut

[gapas-a]

Currently, we have 3D pose data that we gathered using DeepLabCut and are trying to go to an OpenSim model using Pose2Sim. I am having some trouble figuring out the folder structure to use for multiple trials and what needs to be changed in the config.toml files. Here is what I currently have:

--Main Project Folder
      --Config.toml
      --Trial N
            --Config.toml
            --Videos
                  -- Raw, unlabeled videos
            --Calibration
                  -- camera intrinsics/extrinsics/config
            --Pose
                  -- camN
                        --camN.h5
                        --camN.csv
What needs to be changed in each Config file to specify that I am using deeplabcut?

[davidpagnon]

Hi Giancarlo,

I would suggest you copy-paste one of the Demo folders. To do this, follow this section.

What you normally need in your root folder is your Config.toml file, a calibration folder, and a videos folder. Now, your case is slightly different:

• You want to batch process several trials. So you need to put your videos folder in sub-directories corresponding to each trial. You also need to add a Config.toml file. Make it an empty file, unless you want some trials to be processed differently: in this case, add the parameters that change from the root Config.toml file.
• Then, if you used DeepLabCut instead of the pose estimation models provided by Pose2Sim, you do not need the videos folders anymore, you can simply create pose-2d ones for each camera.
  • Be aware that the pose files need to be converted to .json.
  • Depending on the model you trained in DeepLabCut, you may need to create a skeleton tree for triangulation to work. See this section for more information.
    Aside from pose_model and [[pose.CUSTOM]], you should not have to change anything in Config.toml

If you are unsure, you can run the full demonstration so that you see how the folders are populated.

[gapas-a]

mode = 'balanced' # 'lightweight', 'balanced', 'performance', 
                  # or """{dictionary}""" (see below)
# A dictionary (WITHIN THREE DOUBLE QUOTES) allows you to manually select the person detection (if top_down approach) and/or pose estimation models (see https://github.com/Tau-J/rtmlib). 
# Models can be local paths or URLs.
# Make sure the input_sizes are within square brackets, and that they are in the opposite order from the one in the model path (for example, it would be [192,256] for rtmpose-m_simcc-body7_pt-body7-halpe26_700e-256x192-4d3e73dd_20230605.zip). 
# If your pose_model is not provided in skeletons.py, you may have to create your own one (see example at the end of the file).
# Example, equivalent to mode='balanced':
# mode = """{'det_class':'YOLOX',
#          'det_model':'https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/onnx_sdk/yolox_m_8xb8-300e_humanart-c2c7a14a.zip',
#          'det_input_size':[640, 640],
#          'pose_class':'RTMPose',
#          'pose_model':'https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/onnx_sdk/rtmpose-m_simcc-body7_pt-body7-halpe26_700e-256x192-4d3e73dd_20230605.zip',
#          'pose_input_size':[192,256]}"""
# Example with one-stage RTMO model (Requires pose_model = 'Body'):
# mode = """{'pose_class':'RTMO', 
#          'pose_model':'https://download.openmmlab.com/mmpose/v1/projects/rtmo/onnx_sdk/rtmo-m_16xb16-600e_body7-640x640-39e78cc4_20231211.zip', 
#          'pose_input_size':[640, 640]}"""
# Example with animal pose estimation:
# mode = """{'pose_class':'RTMPose',
#          'pose_model':'https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/onnx_sdk/rtmpose-m_simcc-ap10k_pt-aic-coco_210e-256x256-7a041aa1_20230206.zip',
#          'pose_input_size':[256,256]}"""

In regard to this section:

The DeepLabCut model I am using is a horse. What file do I have to put in the 'pose_model' section of the dictionary here? I keep getting this error when trying to run triangulation:

   1432     except:
   1433         print(pose_model)
-> 1434         raise NameError('{pose_model} not found in skeletons.py nor in Config.toml')
   1435 keypoints_ids = [node.id for _, _, node in RenderTree(model) if node.id!=None]
   1436 keypoints_names = [node.name for _, _, node in RenderTree(model) if node.id!=None]

[davidpagnon]

Hi, sorry for being so long to answer, I was just going over the issues and realized I had not answered this one!

Out of memory, you need to set pose_model = 'CUSTOM' and change the [[pose.CUSTOM]] lines (here) so that they correspond to the horse keypoints and their hierarchy.

But also, I just added a rather large contribution to rtmlib to support animal models: Tau-J/rtmlib#68
I would like to implement animal pose biomechanics in Pose2Sim and Sports2D. Now, the only missing part to get it done is calibrated animal videos. Would you mind sharing a set of horse videos with a calibration file with me? It would be greatly helpful :) contact@david-pagnon.com