SO100 Arm Unable to Lift Cube with Camera + Privileged Information Training

[Samyarrahimi]

I'm training an SO100 robotic arm to lift a cube using a combination of camera observations and privileged information, but the arm is failing to successfully lift the cube.

The arm trains but cannot successfully grasp and lift the cube. It approaches the cube but fails to execute proper grasping and lifting motions.

Visual Overview

rl-video-step-60000.mp4

Setup Details

I'm using a manager-based environment with configs from lift_env_cfg.py. The only difference is that I've added a camera to the SO100 to incorporate visual observations.

Observations Config:

@configclass
class ObservationsCfg:
    """Observation specifications for the MDP."""
    @configclass
    class PolicyCfg(ObsGroup):
        """Observations for policy group."""
        joint_pos = ObsTerm(func=mdp.joint_pos_rel)
        joint_vel = ObsTerm(func=mdp.joint_vel_rel)
        object_position = ObsTerm(func=mdp.object_position_in_robot_root_frame)
        target_object_position = ObsTerm(func=mdp.generated_commands, params={"command_name": "object_pose"})
        actions = ObsTerm(func=mdp.last_action)

        # Camera features - will be flattened to match other observations
        camera_rgb_features = ObsTerm(func=mdp.image_features, params={"sensor_cfg": SceneEntityCfg("gripper_camera"), "data_type": "rgb", "model_name": "resnet18"})

        def __post_init__(self):
            self.enable_corruption = True
            self.concatenate_terms = True

    # observation groups
    policy: PolicyCfg = PolicyCfg()

Rewards Config:

@configclass
class RewardsCfg:
    """Reward terms for the MDP."""

    reaching_object = RewTerm(func=mdp.object_ee_distance, params={"std": 0.1}, weight=1.0)

    lifting_object = RewTerm(func=mdp.object_is_lifted, params={"minimal_height": 0.04}, weight=15.0)

    object_goal_tracking = RewTerm(
        func=mdp.object_goal_distance,
        params={"std": 0.3, "minimal_height": 0.04, "command_name": "object_pose"},
        weight=16.0,
    )

    object_goal_tracking_fine_grained = RewTerm(
        func=mdp.object_goal_distance,
        params={"std": 0.05, "minimal_height": 0.04, "command_name": "object_pose"},
        weight=5.0,
    )

    # action penalty
    action_rate = RewTerm(func=mdp.action_rate_l2, weight=-1e-4)

    joint_vel = RewTerm(
        func=mdp.joint_vel_l2,
        weight=-1e-4,
        params={"asset_cfg": SceneEntityCfg("robot")},
    )

SKRL PPO config:

seed: 42


# Models are instantiated using skrl's model instantiator utility
# https://skrl.readthedocs.io/en/latest/api/utils/model_instantiators.html
models:
  separate: False
  policy:  # see gaussian_model parameters
    class: GaussianMixin
    clip_actions: False
    clip_log_std: True
    min_log_std: -20.0
    max_log_std: 2.0
    initial_log_std: 0.0
    network:
      - name: net
        input: STATES
        layers: [256, 128, 64]
        activations: elu
    output: ACTIONS
  value:  # see deterministic_model parameters
    class: DeterministicMixin
    clip_actions: False
    network:
      - name: net
        input: STATES
        layers: [256, 128, 64]
        activations: elu
    output: ONE

# Rollout memory
# https://skrl.readthedocs.io/en/latest/api/memories/random.html
memory:
  class: RandomMemory
  memory_size: -1  # automatically determined (same as agent:rollouts)


# PPO agent configuration (field names are from PPO_DEFAULT_CONFIG)
# https://skrl.readthedocs.io/en/latest/api/agents/ppo.html
agent:
  class: PPO
  rollouts: 24
  learning_epochs: 8
  mini_batches: 4
  discount_factor: 0.99
  lambda: 0.95
  learning_rate: 1.0e-03
  learning_rate_scheduler: KLAdaptiveLR
  learning_rate_scheduler_kwargs:
    kl_threshold: 0.01
  state_preprocessor: RunningStandardScaler
  state_preprocessor_kwargs: null
  value_preprocessor: RunningStandardScaler
  value_preprocessor_kwargs: null
  random_timesteps: 0
  learning_starts: 0
  grad_norm_clip: 1.0
  ratio_clip: 0.2
  value_clip: 0.2
  clip_predicted_values: True
  entropy_loss_scale: 0.005
  value_loss_scale: 2.0
  kl_threshold: 0.0
  rewards_shaper_scale: 0.01
  time_limit_bootstrap: False
  # logging and checkpoint
  experiment:
    directory: "SO_100_Cube_Lift_With_Camera"
    experiment_name: "SO_100_Cube_lift_camera" 
    write_interval: auto
    checkpoint_interval: auto


# Sequential trainer
# https://skrl.readthedocs.io/en/latest/api/trainers/sequential.html
trainer:
  class: SequentialTrainer
  timesteps: 110000
  environment_info: log

Questions

Are there known issues with combining camera and privileged observations for manipulation tasks in IsaacLab?
Are there specific PPO hyperparameters or reward function considerations for this type of multi-modal observation setup?
Are there example configurations or scripts available for SO100 arm manipulation with visual observations?
Any guidance on debugging this setup or recommendations for alternative approaches would be greatly appreciated!

After 110,000 reinforcement learning steps, the robot has not learned to lift the cube. due to the hardware constraints (memory) I cannot train with more than 32 environments.

[Toni-SM]

Hi @Samyarrahimi

Please see the following comments for reference on your case

• [Question] Observation and SKRL setup for Camera and Robot in Manager-based Projects #2613 (comment)
• [Question] Observation and SKRL setup for Camera and Robot in Manager-based Projects #2613 (comment)