-
Notifications
You must be signed in to change notification settings - Fork 34
Open
Labels
Description
@per1234 I saw the announcement about Github Actions for Arduino. Does this mean I should plan to drop such support from this project?
I get the sense that arduino_ci
is a bit more advanced at the moment, given that I can specify different dependencies for each compiled example. But I assume the Arduino action is going to be developed further as well.
Should I plan to drop support for some or all of arduino_ci
in favor of the official action?
Activity
per1234 commentedon Apr 10, 2021
Well, first I should state that I am very biased because the
arduino/compile-sketches
andarduino/report-size-deltas
are my pet projects. Of course I'm going to think they are the best option for sketch compilation tests. However, I also very much welcome competition in this space. I've never held back in my meager attempts to be of assistance with this project, as well as my other primary competitors:adafruit/ci-arduino
andArminJo/arduino-test-compile
.My opinion is that
arduino_ci
should focus on doing one thing very well. To me, that one thing is unit testing.I don't think anything about the sketch compilation action release should make you consider dropping the unit testing aspects of
arduino_ci
.I am not aware of any plans Arduino has to reproduce what
arduino_ci
offers in terms of unit tests. You can see the approach Arduino is taking to unit testing here:ArduinoCore-API
tests:
https://github.com/arduino/ArduinoCore-API/tree/master/test
workflow:
https://github.com/arduino/ArduinoCore-API/blob/master/.github/workflows/unit-tests.yml
ArduinoIoTCloud library:
tests:
https://github.com/arduino-libraries/ArduinoIoTCloud/tree/master/extras/test
workflow:
https://github.com/arduino-libraries/ArduinoIoTCloud/blob/master/.github/workflows/unit-tests.yml
ArduinoBLE library (WIP)
tests:
https://github.com/Polldo/ArduinoBLE/tree/ci-workflow/extras/test
workflow:
https://github.com/Polldo/ArduinoBLE/blob/ci-workflow/.github/workflows/unit-tests.yml
I did create the
arduino/cpp-test-action
action you see used in those workflows, but this is an extremely minimal action that is essentially just running CMake+Valgrind+LCOV commands from the shell. It's just a way to maintain all those commands in a single location, rather than duplicating them in every workflow, as we were doing initially. I think that's all thearduino/cpp-test-action
action will ever be. All the real magic is in the Catch2 and FakeIt frameworks, which Arduino has no involvement with, and the unit test code itself, which the firmware developers wrote.The unit testing efforts have focused on the low hanging fruit for host-side testing. The code it covers does not require extensive mocking of hardware.
The sketch compilation actions are actually nothing new. Arduino has been using the sketch compilation action for over 1.5 years now, and the report-size-deltas action for a year. They have been developed and used in public repositories from the start. And this is the continuation of work I started four years ago with my
per1234/arduino-ci-script
project. The only thing that changed yesterday is we finally got around to doing the 1.0.0 release of the actions and putting them in GitHub Marketplace.I definitely have a long "to do" list for the projects, but nothing very dramatic as far as the functionality and interface goes. I've been a bit hampered by the lack of a release in how much work I could do to the actions, so this will open things up for me a bit. However, I don't have any plans to add provisions for defining dependencies per sketch to the action. This can already be done via the GitHub Actions workflow system. You can see an example of something along those lines here:
https://github.com/arduino-libraries/ArduinoIoTCloud/blob/master/.github/workflows/compile-examples.yml#L64-L176
and here:
https://github.com/arduino/arduino-examples/blob/main/.github/workflows/compile-examples.yml
That approach seems to be working out just fine for the action's users.
There has been some talk on and off about adding some form of support to Arduino CLI for defining library dependencies via the sketch metadata, and in fact Arduino Web Editor already does allow associating specific libraries with a sketch. Since
arduino/compile-sketches
uses Arduino CLI under the hood, it will inherit any advancements made to Arduino CLI.ianfixes commentedon Apr 12, 2021
This is all good info. For clarity, I would only consider dropping support for the compilation of examples, and even then only if you set up a local version of the test runner so you could try out the CI on your local machine first.
I'm very glad to see CI/CD and testing start to pop up in an official capacity at Arduino -- my goal was to have such tooling, not write it myself. To that end, I'm wondering if/how the Arduino team would ever want to adopt the testing methodology I'm using (if not the ruby implementation itself) into a project owned by that organization. Which members of the Arduino team are part of that decision making process? I'd like to get a sense of what level of maturity I'd need to shoot for in order for that to be considered viable.