Support exact Hessians of Lagrangian `NonLinMPC` and `MovingHorizonEstimator`

[franckgaga]

@baggepinnen I think for Hessians, it would make sense to always default to sparse computations (when activated, I would keep the LBFGS approximation as the default)? Should it be pure ForwardDiff, or mixed mode?

[franckgaga]

Also maybe @gdalle has tips on that subject ? Thanks!

[gdalle]

If your problems are reaasonably small, pure-ForwardDiff sparse Hessians are not a bad idea.
Otherwise you'd have to combine backends, and that's where it gets tricky:

• ReverseDiff only remains alloc-free in the absence of constants
• Zygote is of course not alloc-free, nor does it support mutation
• Mooncake and Enzyme don't work as inner backends for ForwardDiff

Pure-Enzyme sparse Hessians would also be an option but at the moment they're not so optimized, and there are still bugs I don't understand with respect to DI.Cache in second-order Enzyme

[franckgaga]

Alright let's always sparse forwarddiff by default. The cost might be prohibitive on large systems, but the default setting will still be the LBFGS approximation anyway.

Any thoughts @baggepinnen 💭?

[baggepinnen]

sparse ForwardDiff is probably the only "safe" default in the sense that it's most likely to work, all others have quite spotty support in my experience.

[franckgaga]

Btw, how's your trip to Jordan ? 🇯🇴

Will visit the Iceland for the first time in July, really looking forward for this trip 🇮🇸!

[franckgaga]

Come to think of it, as mentionned by Robbybp here, it would not be efficient at all to compute individual Hessians for each element of the nonlinear constraint equality and inequality vectors. Maybe I should wait that Ippot.jl and JuMP officially support user-defined vector functions, at least for the nonlinear constraints.

A good compromise here would be to add the support for the objective function Hessian with a keyword argument hessian in the short term (since this part does not require user-defined vector functions). The default would be sparse AutoForwardDiff.

I would add a new keyword argument hessian_lagrangian later, when these functionalities will be ready and stable.