Debugging simulation instability/NaN errors (with OmniPVD?)

[Creampelt]

I'm currently facing an issue with simulation instability, resulting in NaN errors when updating the actor distribution (usually around 8k steps into training). I've looked at these guides and have tried adjusting gains, decreasing sim.dt, and adjusting clip_param --- I still get the same issues, the only thing that changes is how soon or late it occurs.

I'm also tried enabling OmniPVD capture as described here, but this quickly uses up the ~250 GB of free space on my machine in <50 steps and takes 60-70 seconds per step. I am mainly concerned with recording the step that fails --- is there some way to make the OmniPVD recording faster and discard old recordings? I'm also open to any other suggestions on debugging this issue.

[RandomOakForest]

Thank you for starting this discussion. This is typically caused by numerical issues, as you describe. The manual for the OmniPVD is here. It seems you have to manually delete the recordings as needed.