ForwardDiff overloads

[ChrisRackauckas]

A(p)x = b(p) does not need to differentiate w.r.t. p, since dA/dp dx/dp = db/dp is just another linear equation, so that can be solved. Thus the duals should just be split and it should solve using the same solver.

The implementation can follow the same setup. You find cases with Dual numbers:

https://github.com/SciML/NonlinearSolve.jl/blob/v4.8.0/lib/NonlinearSolveBase/ext/NonlinearSolveBaseForwardDiffExt.jl#L22-L32

If you catch that, then you remake using ForwardDiff.value to de-dual and solve the non-dual part:

https://github.com/SciML/NonlinearSolve.jl/blob/v4.8.0/lib/NonlinearSolveBase/ext/NonlinearSolveBaseForwardDiffExt.jl#L43-L59

and then you push forward the partials:

https://github.com/SciML/NonlinearSolve.jl/blob/v4.8.0/lib/NonlinearSolveBase/ext/NonlinearSolveBaseForwardDiffExt.jl#L64-L78

It's actually the same thing as this code, since f(x,p) = 0 when x(p) is the solution has the derivative f_x x_p + f_p = 0 which means x_p = - f_x \ f_p which is this line https://github.com/SciML/NonlinearSolve.jl/blob/v4.8.0/lib/NonlinearSolveBase/ext/NonlinearSolveBaseForwardDiffExt.jl#L66. So it's effectively the same thing as this code right here is just turned into x_p = A_p \ b_p instead, i.e. just grabbing the partials out of A and b.

This is the core of the downstream issue SciML/ModelingToolkit.jl#3589