[Question] Training UR Robots with Robotiq 2F-85 Gripper (Mimic Joint)

[HariKrishnan06082k]

Hello,

I am trying to train a UR5 robot equipped with a Robotiq 2F-85 gripper for a lifting task, similar to the example with the Franka Isaac-Lift-Cube-Franka-v0. However, I’m unsure how to set the init_state in the robot’s configuration file, given that the Robotiq 2F-85 gripper includes a mimic joint.

Could you please clarify if it’s possible to train robots with a mimic joint? If so, how should I set the configuration file?

Hi I'm trying to reopen this issue as I need help with the same and no solution was provided in the previous question.

I have two questions to be precise,

1. Rigging issue itself with UR robots and getting stable position control and optimize it using gain tuner. Please refer to this sub issue regarding this
   Link 1: https://forums.developer.nvidia.com/t/unstable-robot-assembler-in-isaac-sim-4-5/323628/12
   Link 2: https://github.com/isaac-sim/IsaacLab/discussions/1908 {No answer provided}

2. How would the assetCfg look like at all for mimic joint based config, say in this case if everything is tuned properly (currently it's not),

3. Ref https://github.com/isaac-sim/IsaacLab/discussions/1099 in this issue, you've informed that having two separate articulation root layers is not possible, so in this [https://forums.developer.nvidia.com/t/unstable-robot-assembler-in-isaac-sim-4-5/323628/12] I've tried to have only one articulation root layer which is to be the parent Xform of both Robotiq2F-85 and UR robot where the wrist_3 of UR is attached with base_link of Robotiq gripper with a fixed joint. I somehow tried all tuning stuff with Rigging Complicated Closed loop structure tutorial but with no luck on stable position control. Moreover the gain tuner extension fails to parse _metadata from physx view and (get/set joint_states) under that hierarchy.

Please consider this issue to be a priority as this is a very common configuration and industrial gripper which people try to use. In ROS2 scenario example with Robotiq 2F85 and UR10e the simulation configs (damping and stiffness) when running the gain tuner where not stable as well but it worked. If we could have a detailed tutorial on rigging Robotiq2f85 from scratch with UR and creating mimic joint cfg in IsaacLab that would be very helpful for researchers and Isaac Sim active users. (Side note , not a big deal but the UR5e asset in IsaacAsset(Beta) is missing a mesh at the base link of the robot).

[RandomOakForest]

Thank you for posting this, and clarifying the issues for us. Unfortunately, we haven't got to provide an answer earlier. We'll look into this. Please feel free to ping us back if we are taking too long.

To start, currently, Isaac Lab does support mimic joints. This has been mentioned earlier (#2209).

[HariKrishnan06082k]

Thanks I look forward to developing more with Isaacsim and Isaac lab but my workflow to create digital twin would need this to be sorted before I can proceed further. Eagerly waiting for your response.😀

[HariKrishnan06082k]

Hi just wanted to ping you back regarding this issue for IsaacLab, I was able to make the urdf -> usd work decently for position control in IsaacSim please refer to this link which has the respective files. However the way it is working right now is by having multiple mimic joints to close the articulation loop and single point of articulation being finger_joint with 0 - 45.0 limit. I'm uploading the drive link which will have parameters, which I tested for including rmpflow, lula description etc.,

robotiq_asset_drive_link

Just for additional info,

1. The usd which works is in relative path : robotiq_issue_assets/ur5/usd/ur5_robotiq_base_plate_default_q.usd
2. The urdf which is used is in relative path : robotiq_issue_assets/ur5/robot_description/ur5_baseplate.urdf

and to be able to control the drive for opening and closing I did write my own wrapper which worked specifically for robotiq, as it's single point of actuation and Parallelgripper class was implemented for 2 joint drives. The code for that should be in the link as well.

Now for this to work in IsaacLab what would be the ArticulationCfg, and how it should be defined is what I'm more interested about as it has multiple MimicJoint API applied to different joints of the gripper. Thanks.

[IOSMRobotics]

Hi @HariKrishnan06082k ,

We had a similar issue that was troubling us for a while and in the end managed to get a quick work around it by assigning binary open/close positions using BinaryJointPositionActionCfg. Our setup is a ur10e and a robotiq 2f85, the articulationcfg has all the arm joints and all the finger joints defined (even the mimic joints) as actuated joints. For the UR stiffness and dampening, I roughly used these values from the demo files in IsaacLab: https://github.com/isaac-sim/IsaacLab/blob/main/source/isaaclab_assets/isaaclab_assets/robots/universal_robots.py

Then to get the values of the joints for the binary position cfg, I used the values in IsaacSim for opening and closing the gripper for every joint (the joints are in radians in isaacLab):

gripper_actions = mdp.BinaryJointPositionActionCfg( asset_name="robot", joint_names=[ "finger_joint", "right_outer_knuckle_joint", "right_outer_finger_joint", "right_inner_finger_joint", "right_inner_finger_knuckle_joint", "left_outer_finger_joint", "left_inner_finger_knuckle_joint", "left_inner_finger_joint", ], open_command_expr={ "finger_joint": 0.0, "right_outer_knuckle_joint": 0.0, "right_outer_finger_joint": 0.0, "right_inner_finger_joint": 0.0, "right_inner_finger_knuckle_joint": 0.0, "left_outer_finger_joint": 0.0 , "left_inner_finger_knuckle_joint": 0.0, "left_inner_finger_joint": 0.0, }, close_command_expr={ "finger_joint": 0.785398163, "right_outer_knuckle_joint": 0.785398163, "right_outer_finger_joint": 0.0, "right_inner_finger_joint": 0.785398163, "right_inner_finger_knuckle_joint": -0.785398163, "left_outer_finger_joint": 0.0 , "left_inner_finger_knuckle_joint": -0.785398163, "left_inner_finger_joint": -0.785398163, } )

Now, when training, the gripper opens and closes in the way that the gripper does in real life and in isaacSim, with the 2 fingers staying parallel. If there's a better way to do it, or a way to assign partially closed commands (e.g sending the robotiq gripper a 0.4 command to half close it) than that would be great to know!

[IOSMRobotics]

Formats better for reading without the code block:

gripper = mdp.BinaryJointPositionActionCfg(
asset_name="robot",
joint_names=[
"finger_joint",
"right_outer_knuckle_joint",
"right_outer_finger_joint",
"right_inner_finger_joint",
"right_inner_finger_knuckle_joint",
"left_outer_finger_joint",
"left_inner_finger_knuckle_joint",
"left_inner_finger_joint",
],
open_command_expr={
"finger_joint": 0.0,
"right_outer_knuckle_joint": 0.0,
"right_outer_finger_joint": 0.0,
"right_inner_finger_joint": 0.0,
"right_inner_finger_knuckle_joint": 0.0,
"left_outer_finger_joint": 0.0 ,
"left_inner_finger_knuckle_joint": 0.0,
"left_inner_finger_joint": 0.0,
},
close_command_expr={
"finger_joint": 0.785398163,
"right_outer_knuckle_joint": 0.785398163,
"right_outer_finger_joint": 0.0,
"right_inner_finger_joint": 0.785398163,
"right_inner_finger_knuckle_joint": -0.785398163,
"left_outer_finger_joint": 0.0 ,
"left_inner_finger_knuckle_joint": -0.785398163,
"left_inner_finger_joint": -0.785398163,
}
)

[HariKrishnan06082k]

Hi just wanted to ping you back regarding this issue for IsaacLab, I was able to make the urdf -> usd work decently for position control in IsaacSim please refer to this link which has the respective files. However the way it is working right now is by having multiple mimic joints to close the articulation loop and single point of articulation being finger_joint with 0 - 45.0 limit. I'm uploading the drive link which will have parameters, which I tested for including rmpflow, lula description etc.,

robotiq_asset_drive_link

Just for additional info,

1. The usd which works is in relative path : robotiq_issue_assets/ur5/usd/ur5_robotiq_base_plate_default_q.usd

2. The urdf which is used is in relative path : robotiq_issue_assets/ur5/robot_description/ur5_baseplate.urdf

and to be able to control the drive for opening and closing I did write my own wrapper which worked specifically for robotiq, as it's single point of actuation and Parallelgripper class was implemented for 2 joint drives. The code for that should be in the link as well.

Now for this to work in IsaacLab what would be the ArticulationCfg, and how it should be defined is what I'm more interested about as it has multiple MimicJoint API applied to different joints of the gripper. Thanks.

Hi @IOSMRobotics idk if you've seen these files or not, but in isaac sim I was able to get precise position control too by giving in intermediate values for the gripper which made it move half way if wanted. But how I was doing that is by directly manipulating the angular drive by using USDPhysics Get and Set attributes. I was not able to get the parallel gripper implementation to work with one actuation point (left knuckle) So I wrote a MimicGripper method to actually do open and close by again using the USD schema. The full class implementation you could check the google drive link above.

In short my apply_action function did this,

target_position = control_actions.joint_positions[0]
robotiq_prim = get_prim_at_path("/World/ur5e_robotiq_2f_85/robotiq_joints/robotiq_85_left_knuckle_joint")
robotiq_drive = UsdPhysics.DriveAPI.Get(robotiq_prim, "angular")

if robotiq_drive:
    robotiq_drive.GetTargetPositionAttr().Set(float(target_position)) 

But idk how this would work in IsaacLab setting, on the other note could you please send me your complete articulation testing config in one of the sample formats, while doing something with simulation stepping via dt. Similar to the code present in Isaaclab/tutorials/01_assets/run_articulation.py

On a higher note I think there should be a better way of handling mimic configuration in IsaacLab and I saw someone working on the branch for it, but it has went stale i think or still a work in progress. Refer Isaaclab/unactuated-joints,

If you could provide me with the following usd file with just the robot and the gripper along with run_simulator.py for it I think I'll have an idea on how to fix it atleast temporarily. Cuz main fix would require changes to source API of Isaaclab itself.

gripper_isaacsim.mp4

[RandomOakForest]

Thank you for following up and for your interest in Isaac Lab. I will move this question to our Discussions section for follow up.