Clearer JSON API

[paul-fornage]

Is your feature request related to a problem? Please describe.
I am writing a WLED OpenAPI spec so that interfaces can be implemented in any language, but after going back to my old rust library from the v14 era, almost everything has changed.

The spec would also allow for auto-generated documentation, as the docs are massively incomplete and out of date.

Describe the solution you'd like
This is more an RFC than a feature request, I could totally understand this not being the right call, but I am proposing a way to define the JSON structure in the WLED source code more clearly, instead of manually ser/deser the json object for every field.

I think having some single source of structure would make it much easier for consumer (me, I'm being selfish)

To be clear, I'm not suggesting OpenAPI generating the serialize/deserialize code, or the structure, but having some source of truth like a classic c struct that consumer can at least reference and easily diff for changes would be very useful.

It would obviously take implementing to know for sure, but I think doing it this way could also speed it up, as the keys are all known at compile time, and the JSON wouldn't have to be handled dynamically.

Describe alternatives you've considered
Coping and just doing it manually. Unfortunately I don't have the motivation to do this task (and maintain it, I have done it once before). It's not technically challenging and very tedious, and I only need the API when I'm doing specific projects.

Alternatively changes could trigger an AI workflow to update the struct based on the diff, which would definitely make it easier, or also without AI, but I assume you wouldn't want to dissuade contributors by making them write/manage docs.

Additional context
I might find some time to provide a demo PR, but I would want some hint that there is interest.

I am working on a project to use computer vision to map LEDs to a 3d pointcloud (like Matt parker's similar video) and needing to use the API for basic stuff (I have implemented DDP libraries for actual LED control in python and rust already), but trying to write robust and future proof code is hard with the current arrangement.

[softhack007]

@paul-fornage, thanks for your call for discussion.

Let's start wirh a few aspects that drive the WLED design:

• we are talking about a complex software that runs on CPU and memory constrained devices (ESP8266 has ~20KB of free RAM after starting WLED).
• The JSON APIs are just one interface for communication with external applications. In this context, no JSON data structure - or central C struct - should be the "single source of truth", as it's just one special purpose interface.
• keeping a complex JSON structure in memory and making it reflect the software state faithfully sounds very inefficient to me, as it would hinder performance optimizations and consume a lot of RAM.
• Afaik JSON (and Arduino-JSON) is basically a string-based nested object structure. It might be appropriate for the general case but doesn't look to me like an efficient way of storing real-time data.
• instead of enforcing centralized "global state" information, we are currently looking into parallelization of processes, and a tasks design that supports distributed data ownership and lock-free processing.

It's possible that I misunderstand the objectives of your proposal, but I don't see right now what the added value really is.

[netmindz]

OpenAPI spec would definitely be nice to have, the hard part is that with C you can't use the reflection or annotations used in JVM languages to automate the spec from the code.

Another user did use AI to try and create a spec based on a combination of the rust library and the WLED source, but when I asked how they made it and how it would be maintained going forward I got in a bizarre discussion where I was clearly chatting with an LLM which was failing miserably to pretend it wasn't

[paul-fornage]

Ok I do have an openAPI spec for 0.14.4 , if you want that, typed and everything. here

Most things work I think, although I think I have commits I didn't push on my home pc.

I fully used AI, but I made the rust lib artisanally, so I think I could just generate the structs through an OpenAPI generator and use unit testing to compare to the lib to verify it got that right

OpenAPI spec would definitely be nice to have, the hard part is that with C you can't use the reflection or annotations used in JVM languages to automate the spec from the code.

Another user did use AI to try and create a spec based on a combination of the rust library and the WLED source, but when I asked how they made it and how it would be maintained going forward I got in a bizarre discussion where I was clearly chatting with an LLM which was failing miserably to pretend it wasn't

[paul-fornage]

@paul-fornage, thanks for your call for discussion.

Let's start wirh a few aspects that drive the WLED design:

* we are talking about a complex software that runs on CPU and memory constrained devices (ESP8266 has ~20KB of free RAM after starting WLED).

* The JSON APIs are just one interface for communication with external applications. In this context, no JSON data structure - or central C `struct` - should be the "single source of truth", as it's just one special purpose interface.

* keeping a complex JSON structure in memory and making it reflect the software state faithfully sounds very inefficient to me, as it would hinder performance optimizations and consume a lot of RAM.

* Afaik JSON (and Arduino-JSON) is basically a string-based nested object structure. It might be appropriate for the general case but doesn't look to me like an efficient way of storing real-time data.

* instead of enforcing centralized "global state" information, we are currently looking into parallelization of processes, and a tasks design that supports distributed data ownership and lock-free processing.

It's possible that I misunderstand the objectives of your proposal, but I don't see right now what the added value really is.

I'm not suggesting we keep json in memory or anything, but to be fair I'm not really sure what I am suggesting. But what I meant by single source of truth is one agreed json schema, not single source of truth for the values in the variables at runtime. I'm gonna implement ser/deser on one of the structs as an example.

Basically I want to do something like compile time template generation, and only populate at runtime when requested. I know if feels expensive to keep the whole json template in flash, but all the string keys are already there, just spread out. You might save some bytes by not storing multiple curly braces, but if curly braces are gonna tip the scales, it will (I think) be offset by removing arduinoJson. Maybe I'm wrong, I'll do some experiments and provide some examples for various approaches. Including flash usage and hopefully rudimentary speed benchmarks.

Thank you for the response.

[paul-fornage]

I've done some experiments in compiler explorer and it looks like there should be NO penalty here.

With compiler Xtensa ESP32-S2 gcc 8.4.0 (2021r2) and options -O2 -std=gnu++17 -DESP32 -DARDUINO_ARCH_ESP32

compiler explorer with one approach: https://godbolt.org/z/W66sd3873
compiler explorer with control (baseline, current) approach: https://godbolt.org/z/xaYnhKzz1

I think this can be improved, but the assembly is literally identical. Seems to warrant further investigation if other people want this too.

[willmmiles]

Thanks for taking a look. I am somewhat skeptical that it will bear out at scale, but I'm interested to see what you come up with and where the edges are.

A big thing to keep in mind is that not all config elements are POD types; for example, an array of integers (such as a list of pins) must be represented as an std::vector<> which will almost certainly defeat the ability for the compiler to elide creation of the struct. (Also we are trying to move away from C-style strings wherever we can -- const char* is to be avoided.)

Consider https://godbolt.org/z/hqWz8s5G6 : the code and required stack size of serializeInfo_New is significantly larger than serializeInfo_Old, and the vector contents are being copied (see the call to memmove that only appears in serializeInfo_New).