[ENH] - Allow plugins to re-use existing Nebari deployment/integration tests

[Adam-D-Lewis]

Feature description

I've created an MLflow plugin for Nebari (https://github.com/nebari-dev/nebari-mlflow-plugin). I want to set up CI to run the Nebari deployment tests and some deployment and e2e tests in order to validate that the plugin continues to work.

The issue is I need to modify the nebari-config.yaml used in the tests. e.g. I need to add

mlflow:
  enabled: true

This could be valuable for plugins in general.

We could accomplish something like this using the plugin hooks. I imagine we set up the nebari config and then allow the plugin to define a hook to modify the config before the test(s) proceed.

Alternatively, I think we can already do something similar using env vars e.g. NEBARI__MLFLOW__ENABLED=true. Update: This didn't work b/c mlflow wasn't found in the nebari-config.yaml and it failed.

Value and/or benefit

see above

Anything else?

No response

[viniciusdc]

You would like to bake in plugin tests within our integration tests, right? And consequently, have a way to target plugin-specific attributes in the nebari-config YAML file correctly.

Plugins in general should have their own testing handled in their respective repositories. Still, I agree that running the entire deploy stack on each repository would be excessive and bothersome for many different issues. I would like to have a separate repo just for testing all available plugins with Nebari deploys.

[Adam-D-Lewis]

You would like to bake in plugin tests within our integration tests, right? And consequently, have a way to target plugin-specific attributes in the nebari-config YAML file correctly.

I assume plugins should run their own tests in their own repo since anyone can develop a plugin and presumably they all will want to test their plugin regularly against the latest version of Nebari. We shouldn't test arbitrary plugins that aren't supported by us. So I'd like to make it easy for nebari plugin devs to run slightly modified (e.g update the nebari config before deploying) versions of our tests. e.g. run a cloud deployment. Add a new playwright test, but still use our setup for adding users to keycloak automatically for CI, etc.

In short, I think we should make some of our testing setup reusable by plugin developers in their repo.

[viniciusdc]

In short, I think we should make some of our testing setup reusable by plugin developers in their repo.

Would it make sense have those tests or custom ones in the plugin's repo template?

[Adam-D-Lewis]

Would it make sense have those tests or custom ones in the plugin's repo template?

IMO, the nebari tests may change over time, and as a plugin author, I would like to test my plugin against main regularly to ensure the plugin still works. If the tests are only on the rebari plugin template than you don't get those updates over time.

[Adam-D-Lewis]

@viniciusdc Here's an idea of what I'm thinking, but it only adds the hook for a single test, and we'd have to add a hook for each test. main...897f7c2

[viniciusdc]

A possible outcome of our recent discussion was to restructure the tests folder into a workable lib, while separating the pytest deploy/destroy somewhere outside of it. And then have a nebari[test] to install those