[Question] Franka Lift Cube Task with Visual Observation

[Ozzey]

Franka Lift Cube Task with Visual Observation

I was trying to modify the Franka-Lift task to use visual observation. The camera is mounted on the end-effector and from the camera's pov, the cube is visible at all the times (screenshot below).

The only change I made was to modify the observation space:

@configclass
class LiftCameraSceneCfg(ObjectTableSceneCfg):

    # add camera to the scene
    
        gripper_camera = TiledCameraCfg(
            prim_path="{ENV_REGEX_NS}/Robot/panda_hand/camera",
            height=480,
            width=640,
            data_types=["distance_to_image_plane"],
            spawn=sim_utils.PinholeCameraCfg(
                focal_length=24.0,
                focus_distance=400.0,
                horizontal_aperture=53.7,
                clipping_range=(0.01, 1.0e5),
            ),
            offset=TiledCameraCfg.OffsetCfg(
                pos=(0.05, 0, 0.05), rot=(0.70441603, -0.06162842, -0.06162842, 0.70441603), convention="ros"
            ),
        )
    ##


@configclass
class DepthObservationsCfg:
    """Observation specifications for the MDP."""

    @configclass
    class DepthCameraPolicyCfg(ObsGroup):
        """Observations for policy group with depth images."""

        image = ObsTerm(
            func=mdp.image, params={"sensor_cfg": SceneEntityCfg("gripper_camera"), "data_type": "distance_to_image_plane"}
        )

    policy: ObsGroup = DepthCameraPolicyCfg()




@configclass
class LiftCameraEnvCfg(LiftEnvCfg):
    """Configuration for the lift environment with depth camera."""

    scene: LiftCameraSceneCfg = LiftCameraSceneCfg(num_envs=256, env_spacing=20)
    observations: DepthObservationsCfg = DepthObservationsCfg()

    def __post_init__(self):
        super().__post_init__()
        # remove ground as it obstructs the camera
        # self.scene.ground = None
        # viewer settings
        self.viewer.eye = (-5.0, 0.0, 2.5)
        self.viewer.lookat = (0.0, 0.0, 1.5)

This env cfg is then used by

class FrankaLiftCameraEnvCfg(LiftCameraEnvCfg):

which contains details about the manipulator.

The agent was trained for roughly 45,109,248 steps but there is still no success at all, infact the manipulator doesnt even try to pickup the cube and stays fixed in a single position (as shown in the screenshot above). The average reward is 0.95

Algorithm: PPO
Library: RL_Games
GPU: Nvidia RTX 4060ti
VRAM: 16GB
RAM: 64GB
Envs: 16

horizon_length: 64
minibatch_size: 512

[RandomOakForest]

Thank you for posting this. I will move the post to our Discussions for the team to follow up.

[RandomOakForest]

Following up, have you tried using 100 environments? It's definitely looks as an under-training problem.

[Ozzey]

Thank you for your response.

Unfortunately with my current setup (mentioned above) I can only train visual tasks with a maximum of 16 parallel envs. Although I can try training more but overall I was expecting atleast a bit better performance after so many interactions.

Also I have made a similar implementation for Isaac-Open-Drawer-Franka task and after similar number of interactions I still had much higher average return (~30 reward, for non-visual variant the average return was around 90 after similar no. of interactions).

[WillMandil001]

I have the same problem but with a global camera (works fine with the object pose as input instead) - 200 envs, training for 100k steps, but no success :(

[Samyarrahimi]

@WillMandil001 @Ozzey have either of you succeeded? I'm also working on cube lifting but with SO100. My problem is that the robot does not grasp and lift the cube.
#2743

[WillMandil001]

@Samyarrahimi
Nothing so far! It half learns the lifting before forgetting. Here is what I got so far for the image vs object pose as input...

I tried a few things, like not pre-processing the images, different learning rates and policy layers, but this is the best I got!

Let me know if you find anything interesting!

[Ozzey]

Hi @WillMandil001 @Samyarrahimi ! Sorry currently working on another project so I haven't had the chance to debug the problem. Though I have faced similar issue with another variant of lifting task in the past where the manipulator would just get stuck in a certain initial position and wont even move towards the object. The task used ground-true observation. I was able to solve the issue by switching from RL-Games to SKRL for PPO. So maybe it's worth trying that out if you haven't already.