[ENH] Allow `"tests:vm": True` override in local pytests

[joshdunnlime]

Is your feature request related to a problem? Please describe.

As a developer, when running and maintaining regressors with "tests:vm": True, these tests do not run. It requires commenting out tests:vm. Perhaps there is already a way to do it?

Describe the solution you'd like

A way to pass a parameter to pytests or set an envvar such that it ignore "tests:vm": True and runs the relevant tests.

Describe alternatives you've considered

NA

Additional context

A clear example is that of XGBoostLSS where it has the vm parameter and without commenting it out, the tests fail to run.

[fkiraly]

Very good observation, thanks for pointing this out!

Perhaps there is already a way to do it?

Hm, not that I know of. There is a "hacky" workaround by using set_tags to override the tag, but that is unsafe and should not be used.

How about a pytest CLI argument? Would that be compatible with common user workflows, or not?

We could invert it and make the current remote CI behaviour the non-default, and "run all tests with dependencies present" the default, and then add the new CLI argument to the GHA test run.

Do you have a preference? What works for your current local testing workflow?

[joshdunnlime]

I think inverting is favourable. The main reason is that it lowers the barrier for entry for new contributors: running vm tagged estimators works out of the box. It's then easy for core devs to turn the vm flag back on in the GH CI.

Fyi, I typically run pytest via vscode test extension (as it makes debugging failed tests incredibly easy). It's trivial to add a pytest argument to the config... when you know what arg to add!

[fkiraly]

ok, I see - you have convinced me that the best solution is:

• adding an arg to the pytest.cfg that turns on and off "vm only" estimators
• making the default so that these estimators run on the local set when changed
• adding an argument to the GHA pytest

We should also change it in sktime, but I would say, let´s test in skpro first whether this works well.

Would you be able to make the changes, @joshdunnlime?

[joshdunnlime]

Yes. Happy to take a look at this after the PRs are closed out.