Tips for domain randomization for training legged locomotion

[TRo202]

Hi. I’m currently working on legged locomotion, and have sim-to-real issues. Training works fine and fast with my algorithm and settings for reward gains, hyperparameters, and etc. However, if I apply noise in observation and domain randomization, training doesn’t work well. I wonder how other people in robotics field deal with domain randomization.

Do you apply noise and domain randomization from the beginning right away or first train without them, then load the pre-trained model and start training again with noise and domain randomization?

Should I try to reduce the noise and randomization range or try to tune the hyperparameters? I honestly doubt if reducing the noise and randomization range is fine since then the trained model will be less robust to sim-to-real gaps.

[RandomOakForest]

Thank you for starting this discussion. Here is a summary that is still under review, but that you may find helpful.

In legged locomotion, domain randomization and observation noise are critical for bridging the sim-to-real gap, but their application requires careful strategy to avoid destabilizing training.

Best Practices for Applying Domain Randomization and Noise

1. Progressive Curriculum:

• Do not apply strong noise and domain randomization from the very start.
  • Begin training without or with minimal randomization and noise.
  • Once the policy demonstrates stable learning and good performance in the nominal (clean) environment, gradually introduce noise and randomization.
  • This staged approach is called curriculum domain randomization and is widely used to prevent early learning collapse and ensure policy robustness¹².

2. Gradual Increase of Randomization:

• Start with narrow/randomization ranges (e.g., friction, mass, sensor noise) and incrementally widen them as training progresses and the policy improves¹².
• Automatic Domain Randomization (ADR) techniques can automate this process by expanding parameter ranges only when the policy can maintain a certain performance threshold, providing a natural curriculum and improved robustness².

3. Hyperparameter Tuning:

• Introducing noise and randomization often requires retuning RL hyperparameters (e.g., learning rate, batch size, reward scaling) because the learning dynamics change.
• If training becomes unstable after adding randomization, first try adjusting these hyperparameters before reducing the randomization ranges¹.

4. Parameter Range Selection:

• Use system identification or real-world measurements to set realistic randomization bounds (e.g., measure friction on your lab floor to set friction randomization limits)¹³.
• Excessively wide ranges can destabilize learning and may lead to policies that are too conservative or fail to converge¹.

5. Implementation in Isaac Lab:

• In Isaac Lab, domain randomization is configured via a class specifying which parameters to randomize, their ranges, and the schedule (e.g., at reset or at intervals)⁴.
• Example configuration (from Isaac Lab docs):

@configclass
class EventCfg:
    robot_physics_material = EventTerm(
        func=mdp.randomize_rigid_body_material,
        mode="reset",
        params={
            "asset_cfg": SceneEntityCfg("robot", body_names=".*"),
            "static_friction_range": (0.7, 1.3),
            "dynamic_friction_range": (1.0, 1.0),
            "restitution_range": (1.0, 1.0),
            "num_buckets": 250,
        },
    )
    robot_joint_stiffness_and_damping = EventTerm(
        func=mdp.randomize_actuator_gains,
        mode="reset",
        params={
            "asset_cfg": SceneEntityCfg("robot", joint_names=".*"),
            "stiffness_distribution_params": (0.75, 1.5),
            "damping_distribution_params": (0.3, 3.0),
            "operation": "scale",
            "distribution": "log_uniform",
        },
    )

• This allows you to randomize robot friction, joint gains, and other properties at episode resets or at specified intervals⁴.

6. Types of Randomization:

• Randomize parameters that most affect sim-to-real transfer:
  • Physical properties: friction, mass, restitution, actuator gains³¹⁴
  • Sensor noise: add observation noise matching real sensor characteristics¹⁴
  • Environment dynamics: gravity, terrain roughness, initial robot state, external perturbations³⁴

Summary Table: Curriculum vs. Direct Randomization

Approach	When to Use	Pros	Cons
Curriculum (Progressive)	Most common, especially for complex tasks	Stable learning, better convergence	Requires schedule design
Direct (Full from start)	Simple tasks, robust policies	Simpler setup	High risk of early training collapse

Key Recommendations:

• Start with curriculum randomization: Train first in a clean environment, then progressively add noise and randomization.
• Tune hyperparameters after introducing randomization.
• Do not reduce randomization range just to make training easier; instead, use curriculum and tuning to reach robustness.
• Measure real-world parameters to set realistic randomization bounds.

Footnotes

1. https://app.studyraid.com/en/read/15451/536759/domain-randomization-to-handle-reality-gap ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

2. https://github.com/isaac-sim/IsaacGymEnvs/blob/main/docs/domain_randomization.md ↩ ↩² ↩³

3. https://arxiv.org/html/2502.10894v1 ↩ ↩² ↩³

4. https://isaac-sim.github.io/IsaacLab/main/source/migration/migrating_from_omniisaacgymenvs.html ↩ ↩² ↩³ ↩⁴ ↩⁵