Add VSCode tutorial pipeline testing to CI #3737
Conversation
This commit adds a new CI workflow that tests all tutorial pipelines from the zenml-io/vscode-tutorial-extension repository to ensure ZenML core changes don't break user-facing tutorial examples. Key features: - Standalone workflow that can be called from ci-slow or run independently - Tests ZenML from current branch (not PyPI) for accurate regression testing - Runs all 10 tutorial pipelines with comprehensive error handling - Includes retry logic for network resilience and timeout protection - Updates documentation to reflect the new testing strategy The workflow is triggered as part of slow CI when the 'run-slow-ci' label is applied to PRs, providing early detection of breaking changes that would affect tutorial examples. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
github-actions
bot
added
internal
strickvl
added
CI
run-slow-ci
and removed
internal
There was a problem hiding this comment.
Pull Request Overview
Adds a new CI workflow and supporting scripts to automatically test all tutorial pipelines from the VSCode tutorial extension against the current ZenML branch, ensuring user-facing examples don’t break.
- Introduces a standalone GitHub Actions workflow (
vscode-tutorial-pipelines-test.yml) for pipeline regression testing - Adds
scripts/test-tutorial-pipelines.shto discover, run, and report on each tutorial pipeline with timeouts and retries - Integrates the new workflow into the slow CI (
ci-slow.yml) and updates documentation (tests/README.md,CLAUDE.md) to reflect the testing strategy
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/README.md | Updated test strategy to include tutorial pipeline regression tests |
| scripts/test-tutorial-pipelines.sh | New script to run and summarize all tutorial pipelines |
| CLAUDE.md | Expanded CI documentation to mention tutorial pipeline testing |
| .github/workflows/vscode-tutorial-pipelines-test.yml | New GH Actions workflow to clone the tutorial repo and run the test script |
| .github/workflows/ci-slow.yml | Added a job to invoke the tutorial pipelines test workflow in slow CI |
Comments suppressed due to low confidence (2)
.github/workflows/vscode-tutorial-pipelines-test.yml:64
- The clone step hardcodes the
developbranch of the tutorial repo; if the default branch changes (e.g., tomain), this may fail. Consider using the default branch or making the branch name configurable.
if git clone --branch develop https://github.com/zenml-io/vscode-tutorial-extension.git tutorial-repo; then
CLAUDE.md:154
- The indentation of this list item doesn't match the surrounding list (it lacks two leading spaces), which may break formatting. Align it with the other bullets under 'Continuous Integration'.
- Tutorial pipeline testing runs all VSCode tutorial examples against the current branch to catch breaking changes
Documentation Link Check Results❌ Absolute links check failed |
The tutorial pipelines import from a utils.py file located at the root of the tutorial repository. When running pipelines with `python pipelines/subfolder/pipeline.py`, Python couldn't find the utils module because the tutorial repo root wasn't in PYTHONPATH. This fix: - Sets PYTHONPATH to include the tutorial repository root when running pipelines - Adds validation to warn if utils.py is missing from the expected location - Should resolve the ModuleNotFoundError for utils imports 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Removes the -e flag from the bash script to prevent fail-fast behavior, ensuring all tutorial pipelines are tested even if some fail. This provides a complete picture of all issues in a single CI run. Changes: - Use `set -uo pipefail` instead of `set -euo pipefail` - Maintains undefined variable checking (-u) and pipe error handling (-o pipefail) - Allows script to continue testing all pipelines and report comprehensive results - More efficient for CI debugging - no need to rerun multiple times 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
There was a problem hiding this comment.
I have a general question about this. How do we version the changes in the tutorials?
If something breaks in ZenML, the tutorial will break. Let's say we fix it in the other repo, which will run it out-of-sync with the latest ZenML release for a while until we make the next release. Even when we release, the pipelines will be incompatible with earlier versions of ZenML. Do we have any precautions for it?
| exit 1 | ||
| fi | ||
| done | ||
| - name: Create virtual environment |
There was a problem hiding this comment.
Does this mean that we are using a singular venv for all the pipelines? In that case, we would also have to merge all the requirements which might become an issue down the road.
There was a problem hiding this comment.
Also, if any of the pipelines depend on previous runs, artifacts, etc., running everything on the same server might create unforeseen circumstances.
There was a problem hiding this comment.
They're all separate pipelines. I mean, all these tutorial pipelines get run with the same env, one after another in the Codespace, so it shouldn't be a problem here.
The main thing this is trying to ward against is someone making a breaking change in the ZenML core API and not realising that it's also broken the tutorial pipelines. The way we version changes to the tutorial pipelines is that it has it's own repo and people can make changes / PRs there etc and fix / update to work with the latest versions of ZenML. I don't think there's a problem with the pipelines being incompatible with earlier versions of ZenML. These tutorial pipelines will only ever be run with the latest release of ZenML. And any changes made to the tutorials repo won't be reflected in the Codespaces environment until someone rebuilds the extension + pushes a new Docker image. But maybe @safoinme has some thoughts on this as well. |
The PYTHONPATH variable might not be set in CI environments, causing the script to fail with "unbound variable" error due to set -u flag. Use parameter expansion to provide empty fallback. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Adds a new CI workflow that tests all tutorial pipelines from the
zenml-io/vscode-tutorial-extensionrepository to ensure ZenML core changes don't break user-facing tutorial examples.This provides early detection of breaking changes that would affect the tutorial pipelines that users follow when learning ZenML.
View a successful run of these tests here: https://github.com/zenml-io/zenml/actions/runs/15637115894/job/44055030989. (This took just over 2 mins to run end-to-end).
Key Features
Implementation Details
.github/workflows/vscode-tutorial-pipelines-test.ymlscripts/test-tutorial-pipelines.shwith validation and reportingci-slow.ymlasvscode-tutorial-pipelines-testjobFiles Changed
.github/workflows/vscode-tutorial-pipelines-test.yml(new standalone workflow)scripts/test-tutorial-pipelines.sh(new test execution script).github/workflows/ci-slow.yml(added job call)tests/README.md(updated testing strategy documentation)CLAUDE.md(updated CI documentation)Test Plan
scripts/format.shpassesThe workflow will be triggered when the
run-slow-cilabel is applied to PRs, ensuring tutorial pipeline compatibility is verified before merging changes that could break user examples.🤖 Generated with Claude Code