Torchcchio

[lvjonok]

Hello Pinocchio Team,

Around six months ago, when there was no Pinocchio 3 with Casadi support in the Python bindings, I embarked on developing a fork of casadi_kin_dyn to streamline its usage. During this process, I wondered if it would be possible to instantiate templates in Pinocchio using torch variables.

To explore this, I created a small example. However, I encountered an issue where the computational graph was not preserved during computations.

I am reaching out to ask if integrating torch variables in Pinocchio is feasible. There might be a minor issue in my implementation, and if so, I would be happy to assist with integrating torch support into the core codebase.

Additionally, I found this discussion, which suggests the proposal might not be possible. Any insights or guidance would be greatly appreciated.

Thank you!

[simeon-ned]

Hello @jcarpent and @lvjonok,

I hope you don't mind me reviving this discussion. I recently stumbled upon an intriguing implementation from Facebook Research that appears to create a Torch wrapper around Pinocchio's functionality:

• C++ code with bindings
• Python docs

This implementation includes methods for forward kinematics, inverse dynamics, Jacobian computation, and inverse kinematics using Pinocchio, all wrapped to be compatible with PyTorch tensors. It's quite an impressive piece of work!

I wanted to bring this to the attention of the Pinocchio community and ask a few questions:

1. Are the Pinocchio developers aware of this implementation?
2. What are your thoughts on this approach to integrating Pinocchio with PyTorch?
3. Are there any plans within the Pinocchio project to create official PyTorch bindings or integration?
4. If not, would you be open to contributions in this direction?

I believe having PyTorch integration could be incredibly valuable for users who want to combine Pinocchio's efficient rigid body algorithms with PyTorch's machine learning capabilities, especially for applications in reinforcement learning for robotics.

Thank you for your time and consideration. I'm really looking forward to hearing your thoughts on this exciting development!

[jcarpent]

Dear @simeon-ned,

Thanks for coming with these news.

1. Are the Pinocchio developers aware of this implementation?

We were not aware of such initiative, which seems very interesting indeed.
We are very happy to know that Pinocchio has been integrated into PyTorch and could then be easily used inside this deep learning framework.

2. What are your thoughts on this approach to integrating Pinocchio with PyTorch?

I will have a deeper look to the code. But I think we could have a seeless integration, with full support of operations over Lie groups and algebra.

3. Are there any plans within the Pinocchio project to create official PyTorch bindings or integration?

We can think of integrating directly the Torch bindings into the main Pinocchio lib for extended support and integration with all new features, especially constrained dynamics part which seems an important block for future applications.

4. If not, would you be open to contributions in this direction?

We are of course open to external support (as your are currently doing). Thanks in advance for contributing to Pinocchio.
One important remark is that we are mostly missing time to provide proper code review, as we are now focus mostly focus on https://github.com/Simple-Robotics/Simple.

Best,
Justin
PS: and thanks to @lvjonok for starting the discussion

[simeon-ned]

Thank you for the update, @jcarpent! This is great news indeed. I'm excited to hear about the potential for integrating PyTorch support directly into Pinocchio.

@lvjonok and I are looking forward to helping integrate this feature into Pinocchio. We're sure it will be highly appreciated by the RL and robotics communities.

As an interim solution, we could create a small standalone module similar to casadi_kin_dyn, which served as a temporary solution before CasADi support was added to the Python bindings. Our module would instead focus specifically on Torch integration.

Then, when you have more time available after your work on the Simple simulator, we could collaborate on moving this functionality directly into Pinocchio. This approach would allow the community to start benefiting from PyTorch integration sooner while also paving the way for a more seamless integration in the future.

By the way, it's great to hear that you're putting so much attention into Simple. We're eagerly waiting to try it out - the paper looks very promising!

Looking forward to collaborating on this. Let us know how we can best support this effort moving forward.

Best regards,
Simeon

[LockedFlysher]

@simeon-ned I'm currently working based on your repository casadi_kin_dyn, but honestly, I haven't been able to get your project running yet—mainly because the readme is a bit... well... let's say not very "friendly" (ahem) 😅
The good news is that I've already implemented an SQP+ADMM algorithm for NMPC solving on CUDA and can perform simple control tasks. I really hope we can exchange ideas and experiences on this topic!
However, I urgently need to understand how to properly use your repository. If you have time, could you contact me? My email is [email protected]
Looking forward to your reply!

[lvjonok]

Hello @LockedFlysher, why do you need to use casadi_kin_dyn when pinocchio has already released their support for casadi?

[LockedFlysher]

aha, thanks, I have found the solution

[ManifoldFR]

Just wanted to chime in here: when #2616 is fully implemented and Pinocchio switches over to nanobind, we will be able to provide somewhat tighter integration with PyTorch, following this:
https://nanobind.readthedocs.io/en/latest/ndarray.html#returning-arrays-from-c-to-python

Then, users and developers could implement some C++ functions which return e.g. nb::ndarray<nb::pytorch, double>.

[lvjonok]

Sounds really cool! I would love to benchmark this kind of bindings with algorithms implemented in other GPU based dynamics libraries/simulators.

[ManifoldFR]

I think we'd love to see something like that! Although a more appropriate comparison would perhaps be with other CPU-based libraries (e.g. mainline MuJoCo, Bullet, Dart...), since Pinocchio is (so far) a CPU-focused library.