3DGSim: Learning 3D-Gaussian Simulators from RGB Videos

Mikel Zhobro, Andreas René Geist, Georg Martius · ICML 2026

3DGSim is a learned 3D-Gaussian simulator trained directly from multi-view RGB videos — no ground-truth 3D supervision. It reconstructs a scene into 3D Gaussians and rolls out its dynamics autoregressively, enabling photorealistic simulation and latent-space scene editing of elastic objects and cloth.

Code Release

The source code for the synthetic experiments of the paper (elastic objects and cloth, simulated with Genesis) — including pretrained checkpoints, a scene-editing demo, and training/evaluation pipelines — is available on the synthetic_3dgsim branch:

git clone -b synthetic_3dgsim git@github.com:martius-lab/3DGSim.git

See the branch README for installation, demo, and evaluation instructions.

Resources

• Paper: arXiv:2503.24009
• Datasets (elastic & cloth, Genesis): Keeper
• Pretrained models & sample data: HuggingFace mzhobro/3dgsim

Roadmap

• Release datasets
• Release inference code + checkpoints (synthetic experiments, synthetic_3dgsim)
• Release refactored codebase (v2)

Citation

@inproceedings{zhobro2026_3dgsim,
  title     = {3DGSim: Learning 3D-Gaussian Simulators from RGB Videos},
  author    = {Zhobro, Mikel and Geist, Andreas Ren{\'e} and Martius, Georg},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2026}
}