diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 18ab7650..a57a31c0 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -1,68 +1,161 @@ # Carpentries Workflows -This directory contains workflows to be used for Lessons using the {sandpaper} -lesson infrastructure. Two of these workflows require R (`sandpaper-main.yaml` -and `pr-receive.yaml`) and the rest are bots to handle pull request management. +This directory contains workflows to be used for Lessons using the Carpentries Workbench lesson infrastructure. -These workflows will likely change as {sandpaper} evolves, so it is important to -keep them up-to-date. To do this in your lesson you can do the following in your -R console: +The three `docker-` workflows build lessons and maintain packages. +The workflows run using the [workbench-docker](https://github.com/carpentries/workbench-docker) container. +This container comprises prebuilt and installed dependencies of the core Workbench packages, i.e. sandpaper, pegboard and varnish. + +Two `update-` workflows handle: + - checking for new renv packages and creating a Pull Request (PR) when a renv.lock file updates (`update-cache.yaml`) + - checking for updated versions of these workflow files (`update-workflows.yaml`) + +The rest of the `pr-` workflows handle pull request management via base GitHub Actions. + +For Carpentries Core Curriculum lessons across our lesson programmes, maintenance of these workflows should be minimal. +For your own lesson repositories, it is important to understand the different workflows and what they do. + +## Managing Updates + +By using prebuilt Docker containers that are managed by the Carpentries core Workbench maintainers, these workflows are designed to be rarely updated. + +However, is important to be able to keep them up-to-date when appropriate. +You can do this locally using your own R and Workbench installation, or via the "04 Maintain: Update Workflow Files" (`update-workflows.yaml`) GitHub Action. + +### Updating locally + +In a terminal/git bash, navigate to the lesson folder where you want to update the workflows. + +Then, start an R session and: ```r # Install/Update sandpaper -options(repos = c(carpentries = "https://carpentries.r-universe.dev/", - CRAN = "https://cloud.r-project.org")) +options(repos = c(carpentries = "https://carpentries.r-universe.dev/", CRAN = "https://cloud.r-project.org")) install.packages("sandpaper") # update the workflows in your lesson library("sandpaper") -update_github_workflows() +sandpaper::update_github_workflows() +quit() ``` -Inside this folder, you will find a file called `sandpaper-version.txt`, which -will contain a version number for sandpaper. This will be used in the future to -alert you if a workflow update is needed. +And then in a bash prompt/git bash terminal: -What follows are the descriptions of the workflow files: +```bash +$ git add .github/workflows +$ git commit -m "Manual update to docker workflows" +$ git push origin main +``` -## Deployment +This will automatically start the "01 Maintain: Build and Deploy Site" workflow. -### 01 Build and Deploy (sandpaper-main.yaml) +This will be the extent of requirements for non-renv lessons. -This is the main driver that will only act on the main branch of the repository. -This workflow does the following: +#### Lessons that use Rmd and {renv} + +For renv-enabled lessons: +- Cancel the "01 Maintain: Build and Deploy Site" run that automatically started following the push to main +- Run the "02 Maintain: Check for Updated Packages" +- Run the "03 Maintain: Apply Package Cache" +- Run the "01 Maintain: Build and Deploy Site" + +### Updating using GitHub + +This presumes you: + - already have a lesson repository available on GitHub + - have enabled workflows in the lesson repo + - have set up a SANDPAPER_WORKFLOW personal access token (PAT) in the lesson repo + +To go through these steps, please follow the [Forking a Workbench Lesson](https://docs.carpentries.org/resources/curriculum/lesson-forks.html#forking-a-workbench-lesson-repository) +documentation. + +Once set up, run the "04 Maintain: Update Workflow Files" (`update-workflows.yaml`) action. + +This will raise a PR with any changes to the workflows that are needed. +If you are happy with the changes made, you can merge the PR into your lesson repository. +## Lesson Builds and Deployment + +### 01 Maintain: Build and Deploy Site (docker_build_deploy.yaml) + +This is the main workflow that you will encounter most often. + +It will only act on the main branch of the lesson repository. + +This workflow does the following: 1. checks out the lesson 2. provisions the following resources - - R - - pandoc - - lesson infrastructure (stored in a cache) - - lesson dependencies if needed (stored in a cache) + - the Workbench Docker container + - lesson dependencies if needed (stored in a cache) 3. builds the lesson via `sandpaper:::ci_deploy()` +If your lesson contains rendered content using RMarkdown and/or any associated R package dependencies, you will need to generate and apply the renv cache. +Please read the [Caching](#caching) section below. + #### Caching -This workflow has two caches; one cache is for the lesson infrastructure and -the other is for the lesson dependencies if the lesson contains rendered -content. These caches are invalidated by new versions of the infrastructure and -the `renv.lock` file, respectively. If there is a problem with the cache, -manual invaliation is necessary. You will need maintain access to the repository -and you can either go to the actions tab and [click on the caches button to find -and invalidate the failing cache](https://github.blog/changelog/2022-10-20-manage-caches-in-your-actions-workflows-from-web-interface/) -or by setting the `CACHE_VERSION` secret to the current date (which will -invalidate all of the caches). +> [!NOTE] +> Caching is only relevant for lessons that use Rmd files and renv to manage R packages. +> If you are building basic markdown documents, caching will not apply to you, and the only +> workflow that needs to be run is "01 Maintain: Build and Deploy Site". + +In summary, generating a reusable package cache is achieved by running the "02 Maintain: Check for Updated Packages" workflow, and then the "03 Maintain: Apply Package Cache" workflow. + +These workflows are separated to ensure that once you have a successful build with a working renv cache, this cache is stored within GitHub's infrastructure, and will be reused by the Workbench Docker container. +This means that lesson builds will be faster once an renv cache is created and reused by the Docker container. + +Another major bonus of this setup is that you can keep using this cache indefinitely to build your lesson. +This is important if you need very specific versions of R packages ("pinning"). + +If and when you want to perform an update to the cache, you can re-run the "02 Maintain: Check for Updated Packages" and verify that your lesson still builds with the new packages. +If all looks good, re-run the "03 Maintain: Apply Package Cache" workflow, and this will write a new renv cache file to GitHub. + +In any case, the renv cache is invalidated by new versions of the `renv.lock` file. +This happens: + - if you update your lockfile locally by using the `sandpaper::update_cache()` function, and then push it to the lesson repository + - when you run the "02 Maintain: Check for Updated Packages" and there are new packages to install + +More information on managing local renv caches for lessons can be found in the [Sandpaper packages vignettes](https://carpentries.github.io/sandpaper/articles/building-with-renv.html). + +#### Using different package cache versions + +There are times when you may want to go back to a previous renv package cache file: + - if you run "02 Maintain: Check for Updated Packages" and "03 Maintain: Apply Package Cache" and the cache generation fails for some reason + - if there is a new R package that produces incorrect or broken lesson output + +To choose a previous cache file version for your builds, go to the Actions tab, and click Caches in the left hand pane. + +Cache files should have the following name format: + +``` + OS HASHSUM +[ | ] [ | ] +Linux--renv-2e499eb706112971b2cffceb49b55a6efe49f3ed75cd6579b10ff224489daca4 +``` + +Once you have 2 or more cache files, you can choose which one you want to use. + +Copy the hashsum part of the desired cache file you want to use, e.g. `2e499eb706112971b2cffceb49b55a6efe49f3ed75cd6579b10ff224489daca4`. + +Then either: + 1. Add a repository variable called CACHE_VERSION, and paste in the hash + - Go to ... + 2. Run the "01 Maintain: Build and Deploy Site" manually, supplying the CACHE_VERSION input + - Go to ... + +If you have no caches listed, make sure to run the "02 Maintain: Check for Updated Packages" and "03 Maintain: Apply Package Cache" to create a new renv cache file. ## Updates ### Setup Information -These workflows run on a schedule and at the maintainer's request. Because they -create pull requests that update workflows/require the downstream actions to run, +These workflows run on a mix of schedules, automatic triggers, and at the maintainer's request. +Because they create pull requests that update workflows/require the downstream actions to run, they need a special repository/organization secret token called `SANDPAPER_WORKFLOW` and it must have the `public_repo` and `workflow` scope. This can be an individual user token, OR it can be a trusted bot account. If you -have a repository in one of the official Carpentries accounts, then you do not +have a repository in one of the official Carpentries organisations, then you do not need to worry about this token being present because the Carpentries Core Team will take care of supplying this token. @@ -73,45 +166,69 @@ clipboard and then go to your repository's settings > secrets > actions and create or edit the `SANDPAPER_WORKFLOW` secret, pasting in the generated token. If you do not specify your token correctly, the runs will not fail and they will -give you instructions to provide the token for your repository. +give you instructions to provide the token for your repository. + +### "02 Maintain: Check for Updated Packages" (update-cache.yaml) + +For lessons that have generated content, we use {renv} to ensure that the output +is stable. This is controlled by a single lockfile which documents the packages +needed for the lesson and the version numbers. This workflow is skipped in +lessons that do not have generated content. + +Packages are frequently updated, fixing bugs or introducing new features. It's a +good idea to make sure these packages can be both: updated periodically, or; or left +static to ensure consistent lesson builds. + +The update cache workflow will do this by: +- checking repositories for updates +- updating the renv lockfile +- summarising the updated packages and their versions in a branch called `updates/packages` +- creating a pull request with _only the renv lockfile changed_ + +From here, the markdown documents will be rebuilt and you can inspect what has +changed based on how the packages have updated. -### 02 Maintain: Update Workflow Files (update-workflow.yaml) +If all steps pass in this workflow, you can safely merge the PR that is raised. +Once the PR is merged, the "03 Maintain: Apply Package Cache" workflow will run +automatically. + +### 03 Maintain: Apply Package Cache (docker_apply_cache.yaml) + +This workflow takes the updated lockfile produced in "02 Maintain: Check for Updated Packages" +and uses it to produce a cached file stored within GitHub's infrastructure. + +This cached file can then be reused repeatedly by the "01 Maintain: Build and Deploy Site" +workflow. + +This workflow is run automatically when the PR generated by "02 Maintain: Check for Updated Packages" +is closed and merged. + +You would only ever need to run this workflow manually: +- if your cache gets removed by GitHub due to age or non-use +- if your cache file contains packages that cannot be used by a Workbench Docker container's newer R version + +### "04 Maintain: Update Workflow Files" (update-workflows.yaml) The {sandpaper} repository was designed to do as much as possible to separate -the tools from the content. For local builds, this is absolutely true, but -there is a minor issue when it comes to workflow files: they must live inside -the repository. +the tools from the content. For local builds, this is absolutely true as you +can develop and build lessons without any GitHub workflows. When it comes to +workflow files on GitHub itself for managed builds online, the workflows must +live inside the lesson repository. -This workflow ensures that the workflow files are up-to-date. The way it work is -to download the update-workflows.sh script from GitHub and run it. The script -will do the following: +This workflow ensures that the workflow files are up-to-date. It downloads the +`update-workflows.sh` script from GitHub and runs it. The script will do the +following: -1. check the recorded version of sandpaper against the current version on github +1. check the recorded version of sandpaper against the current version on GitHub 2. update the files if there is a difference in versions -After the files are updated, if there are any changes, they are pushed to a +After the files are updated, and if there are any changes, they are pushed to a branch called `update/workflows` and a pull request is created. Maintainers are encouraged to review the changes and accept the pull request if the outputs are okay. This update is run weekly or on demand. -### 03 Maintain: Update Package Cache (update-cache.yaml) - -For lessons that have generated content, we use {renv} to ensure that the output -is stable. This is controlled by a single lockfile which documents the packages -needed for the lesson and the version numbers. This workflow is skipped in -lessons that do not have generated content. - -Because the lessons need to remain current with the package ecosystem, it's a -good idea to make sure these packages can be updated periodically. The -update cache workflow will do this by checking for updates, applying them in a -branch called `updates/packages` and creating a pull request with _only the -lockfile changed_. - -From here, the markdown documents will be rebuilt and you can inspect what has -changed based on how the packages have updated. - ## Pull Request and Review Management Because our lessons execute code, pull requests are a secruity risk for any @@ -140,11 +257,11 @@ Once the checks are finished, a comment is issued to the pull request, which will allow maintainers to determine if it is safe to run the "Receive Pull Request" workflow from new contributors. -### Receive Pull Request (pr-receive.yaml) +### Receive Pull Request (docker_pr_receive.yaml) **Note of caution:** This workflow runs arbitrary code by anyone who creates a pull request. GitHub has safeguarded the token used in this workflow to have no -priviledges in the repository, but we have taken precautions to protect against +privileges in the repository, but we have taken precautions to protect against spoofing. This workflow is triggered with every push to a pull request. If this workflow @@ -164,14 +281,11 @@ request. This builds the content and uploads three artifacts: 2. A summary of changes after the rendering process (diff) 3. The rendered files (build) -Because this workflow builds generated content, it follows the same general -process as the `sandpaper-main` workflow with the same caching mechanisms. - -The artifacts produced are used by the next workflow. +The artifacts produced are used by the "Comment on Pull Request" workflow. ### Comment on Pull Request (pr-comment.yaml) -This workflow is triggered if the `pr-receive.yaml` workflow is successful. +This workflow is triggered if the `docker_pr_receive.yaml` workflow is successful. The steps in this workflow are: 1. Test if the workflow is valid and comment the validity of the workflow to the diff --git a/.github/workflows/docker_apply_cache.yaml b/.github/workflows/docker_apply_cache.yaml new file mode 100644 index 00000000..d8e91a70 --- /dev/null +++ b/.github/workflows/docker_apply_cache.yaml @@ -0,0 +1,132 @@ +name: "03 Maintain: Apply Package Cache" + +on: + workflow_dispatch: + inputs: + name: + description: 'Who triggered this build?' + required: true + default: 'Maintainer (via GitHub)' + pull_request: + types: + - closed + +jobs: + preflight: + name: "Preflight: PR or Manual Trigger?" + runs-on: ubuntu-latest + outputs: + do-apply: ${{ steps.check.outputs.merged_or_manual }} + steps: + - name: "Should we run cache application?" + id: check + run: | + if [[ "${{ github.event_name }}" == "workflow_dispatch" || + ("${{ github.event.action }}" == "closed" && "${{ github.event.pull_request.merged }}" == "true") ]]; then + echo "merged_or_manual=true" >> $GITHUB_OUTPUT + else + echo "This was not a manual trigger and no PR was merged. No action taken." + echo "merged_or_manual=false" >> $GITHUB_OUTPUT + fi + + check-renv: + name: "Check If We Need {renv}" + runs-on: ubuntu-latest + needs: preflight + if: ${{ needs.preflight.outputs.do-apply == 'true' }} + outputs: + renv-needed: ${{ steps.check-for-renv.outputs.exists }} + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v4 + + - name: "Check for renv" + id: check-for-renv + run: | + if [[ -d renv ]]; then + echo "exists=true" >> $GITHUB_OUTPUT + fi + + prepare: + name: "Grab renv.lock hash" + runs-on: ubuntu-latest + needs: check-renv + if: ${{ needs.check-renv.outputs.renv-needed == 'true' }} + outputs: + renv-cache-hashsum: ${{ steps.set-hash.outputs.renv-cache-hashsum }} + steps: + - uses: actions/checkout@v4 + + - name: Calculate renv hash + id: set-hash + run: | + echo "renv-cache-hashsum=${{ hashFiles('renv/profiles/lesson-requirements/renv.lock') }}" >> $GITHUB_OUTPUT + + update-renv-cache: + name: "Update renv Cache" + if: | + github.event_name == 'workflow_dispatch' || + ( + github.event.pull_request.merged == true && + contains( + join(github.event.pull_request.labels.*.name, ','), + 'type: package cache' + ) + ) + runs-on: ubuntu-latest + needs: prepare + permissions: + checks: write + contents: write + pages: write + container: + image: carpentries/workbench-docker:${{ vars.WORKBENCH_TAG || 'latest' }} + env: + WORKBENCH_PROFILE: "ci" + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: /home/rstudio/lesson/renv + RENV_PROFILE: "lesson-requirements" + RENV_VERSION: ${{ needs.prepare.outputs.renv-cache-hashsum }} + RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library" + volumes: + - ${{ github.workspace }}:/home/rstudio/lesson + options: --cpus 2 + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v4 + + - name: Current env + run: env | sort + + - name: Debugging Info + run: | + echo "Current Directory: $(pwd)" + ls -lah /home/rstudio/.workbench + ls -lah $(pwd) + Rscript -e 'sessionInfo()' + + - name: Mark Repository as Safe + run: | + git config --global --add safe.directory $(pwd) + + - name: "Ensure sandpaper is loadable" + run: | + .libPaths() + library(sandpaper) + shell: Rscript {0} + + - name: Setup Lesson Dependencies + run: | + Rscript /home/rstudio/.workbench/setup_lesson_deps.R + + - name: Fortify renv Cache + run: | + Rscript /home/rstudio/.workbench/fortify_renv_cache.R + + - name: Cache renv Directory + uses: actions/cache@v4 + with: + path: /home/rstudio/lesson/renv + key: ${{ runner.os }}-${{ inputs.cache-version }}-renv-${{ needs.prepare.outputs.renv-cache-hashsum }} + restore-keys: + ${{ runner.os }}-${{ inputs.cache-version }}-renv- diff --git a/.github/workflows/docker_build_deploy.yaml b/.github/workflows/docker_build_deploy.yaml new file mode 100644 index 00000000..57c90f6c --- /dev/null +++ b/.github/workflows/docker_build_deploy.yaml @@ -0,0 +1,283 @@ +name: "01 Maintain: Build and Deploy Site" + +on: + push: + branches: + - main + schedule: + - cron: '0 0 * * 2' + workflow_dispatch: + inputs: + name: + description: 'Who triggered this build?' + required: true + default: 'Maintainer (via GitHub)' + CACHE_VERSION: + description: 'Optional renv cache version override' + required: false + default: '' + reset: + description: 'Reset cached markdown files' + required: false + default: false + type: boolean + sandpaper-version: + description: 'The version of sandpaper to use. You can use the remotes syntax to specify a version to use. Defaults to latest on the r-universe.' + default: 'latest' + pegboard-version: + description: 'The version of pegboard to use. You can use the remotes syntax to specify a version to use. Defaults to latest on the r-universe.' + default: 'latest' + varnish-version: + description: 'The version of varnish to use. You can use the remotes syntax to specify a version to use. Defaults to latest on the r-universe.' + default: 'latest' + workflow_run: + workflows: ["03 Maintain: Apply Package Cache"] + types: + - completed + +jobs: + preflight: + name: "Preflight: Schedule, Push, or PR?" + runs-on: ubuntu-latest + outputs: + do-build: ${{ steps.check.outputs.push_or_pr }} + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - name: "Should we run build and deploy?" + id: check + run: | + if [[ "${{ github.event_name }}" == "schedule" || + "${{ github.event_name }}" == "workflow_dispatch" || + "${{ github.event_name }}" == "workflow_run" ]]; then + echo "✅ Schedule, manual, or auto trigger. Build will run." + echo "push_or_pr=true" >> $GITHUB_OUTPUT + elif [[ "${{ github.event_name }}" == "push" && "${{ github.ref }}" == "refs/heads/main" ]]; then + PRE_PR=$(gh pr list --repo "${{ github.repository }}" --state merged --search "${{ github.sha }}" --json number -q ".[0].number") >> $GITHUB_ENV + + if [[ -n "$PRE_PR" ]]; then + LABELS=$(gh pr view "$PRE_PR" --repo "${{ github.repository }}" --json labels -q ".labels[].name") + if echo "$LABELS" | grep -q "type: package cache"; then + echo "❗Package Cache PR merged. Please run '03 Maintain: Apply Package Cache' if not already running!" + echo "push_or_pr=false" >> $GITHUB_OUTPUT + exit 1 + else + echo "✅ Valid PR merged. Build will run." + echo "push_or_pr=true" >> $GITHUB_OUTPUT + fi + else + echo "✅ Direct push to main. Build will run." + echo "push_or_pr=true" >> $GITHUB_OUTPUT + fi + else + echo "❗This was not a schedule or valid trigger. No build will run." + echo "push_or_pr=false" >> $GITHUB_OUTPUT + exit 1 + fi + + check-renv: + name: "Check if We Need {renv}" + runs-on: ubuntu-latest + needs: preflight + if: ${{ needs.preflight.outputs.do-build == 'true' }} + outputs: + renv-needed: ${{ steps.check-for-renv.outputs.exists }} + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v4 + + - name: "Check for renv" + id: check-for-renv + run: | + if [[ -d renv ]]; then + echo "exists=true" >> $GITHUB_OUTPUT + fi + + prepare-renv: + name: "Grab renv.lock hash" + runs-on: ubuntu-latest + needs: check-renv + if: ${{ needs.check-renv.outputs.renv-needed == 'true' }} + outputs: + renv-cache-hashsum: ${{ steps.set-hash.outputs.renv-cache-hashsum }} + steps: + - uses: actions/checkout@v4 + + - name: Calculate renv hash + id: set-hash + run: | + CACHE_VERSION_INPUT="${{ github.event.inputs.CACHE_VERSION || vars.CACHE_VERSION }}" + if [ -z "$CACHE_VERSION_INPUT" ]; then + echo "renv-cache-hashsum=${{ hashFiles('renv/profiles/lesson-requirements/renv.lock') }}" >> $GITHUB_OUTPUT + else + echo "renv-cache-hashsum=$CACHE_VERSION_INPUT" >> $GITHUB_OUTPUT + fi + + prepare-wb-package-override: + name: "Any Workbench package version inputs?" + runs-on: ubuntu-latest + outputs: + varnish-version: ${{ steps.input-wb-vers.outputs.varnish-version-override }} + sandpaper-version: ${{ steps.input-wb-vers.outputs.sandpaper-version-override }} + pegboard-version: ${{ steps.input-wb-vers.outputs.pegboard-version-override }} + steps: + - uses: actions/checkout@v4 + + - name: "Get var/input Workbench package versions" + id: input-wb-vers + run: | + VARNISH_VERSION_INPUT="${{ github.event.inputs.varnish-version || 'latest' }}" + if [ ! -z "$VARNISH_VERSION_INPUT" ]; then + echo "varnish-version-override=$VARNISH_VERSION_INPUT" >> $GITHUB_OUTPUT + fi + SANDPAPER_VERSION_INPUT="${{ github.event.inputs.sandpaper-version || 'latest' }}" + if [ ! -z "$SANDPAPER_VERSION_INPUT" ]; then + echo "sandpaper-version-override=$SANDPAPER_VERSION_INPUT" >> $GITHUB_OUTPUT + fi + PEGBOARD_VERSION_INPUT="${{ github.event.inputs.pegboard-version || 'latest' }}" + if [ ! -z "$PEGBOARD_VERSION_INPUT" ]; then + echo "pegboard-version-override=$PEGBOARD_VERSION_INPUT" >> $GITHUB_OUTPUT + fi + + full-build: + name: "Build Full Site" + runs-on: ubuntu-latest + needs: [preflight, check-renv, prepare-renv, prepare-wb-package-override] + if: ${{ needs.preflight.outputs.do-build == 'true' }} + env: + VARNISH_VER: ${{ needs.prepare-wb-package-override.outputs.varnish-version }} + SANDPAPER_VER: ${{ needs.prepare-wb-package-override.outputs.sandpaper-version }} + PEGBOARD_VER: ${{ needs.prepare-wb-package-override.outputs.pegboard-version }} + permissions: + checks: write + contents: write + pages: write + container: + image: carpentries/workbench-docker:${{ vars.WORKBENCH_TAG || 'latest' }} + env: + WORKBENCH_PROFILE: "ci" + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: /home/rstudio/lesson/renv + RENV_PROFILE: "lesson-requirements" + RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library" + RENV_VERSION: ${{ needs.prepare-renv.outputs.renv-cache-hashsum }} + VARNISH_VER: ${{ needs.prepare-wb-package-override.outputs.varnish-version }} + SANDPAPER_VER: ${{ needs.prepare-wb-package-override.outputs.sandpaper-version }} + PEGBOARD_VER: ${{ needs.prepare-wb-package-override.outputs.pegboard-version }} + volumes: + - ${{ github.workspace }}:/home/rstudio/lesson + options: --cpus 1 + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v4 + + - name: Check if renv Exists + id: check-renv + run: | + if [ -d "/home/rstudio/lesson/renv" ]; then + echo "RENV_EXISTS=true" >> $GITHUB_ENV + else + echo "RENV_EXISTS=false" >> $GITHUB_ENV + fi + + - name: "Grabbed renv.lock hash" + if: env.RENV_EXISTS == 'true' + run: echo "RENV_VERSION is $RENV_VERSION" + + - name: Current env + run: env | sort + + - name: Debugging Info + run: | + cd /home/rstudio/lesson + echo "Current Directory: $(pwd)" + ls -lah /home/rstudio/.workbench + ls -lah $(pwd) + Rscript -e 'sessionInfo()' + + - name: Mark Repository as Safe + run: | + git config --global --add safe.directory $(pwd) + + - name: Setup Lesson Dependencies + run: | + Rscript /home/rstudio/.workbench/setup_lesson_deps.R + + - name: Restore renv from cache + id: restore-renv-cache + uses: actions/cache@v4 + if: env.RENV_EXISTS == 'true' + with: + path: /home/rstudio/lesson/renv + key: ${{ runner.os }}-${{ inputs.cache-version }}-renv-${{ needs.prepare-renv.outputs.renv-cache-hashsum }} + restore-keys: + ${{ runner.os }}-${{ inputs.cache-version }}-renv- + + - name: Restore renv Dependencies + if: env.RENV_EXISTS == 'true' && steps.restore-renv-cache.outputs.cache-hit == 'true' + run: | + lsn_path <- fs::path("/home/rstudio/lesson") + renv::load(project = lsn_path) + renv_lib <- renv::paths$library(project = lsn_path) + renv_lock <- renv::paths$lockfile(project = lsn_path) + renv::restore(project = lsn_path, library = renv_lib, lockfile = renv_lock, prompt = FALSE) + shell: Rscript {0} + + - name: Fail on renv cache miss + if: env.RENV_EXISTS == 'true' && steps.restore-renv-cache.outputs.cache-hit != 'true' + run: | + echo "❌ renv cache required but none available. Please run the "02 Maintain: Check for Updated Packages" then the "03 Maintain: Apply Package Cache" workflows." + exit 1 + + - name: Override any Workbench packages + if: env.VARNISH_VER != 'latest' || env.SANDPAPER_VER != 'latest' || env.PEGBOARD_VER != 'latest' + run: | + varnish_version <- '${{ env.VARNISH_VER }}' + sandpaper_version <- '${{ env.SANDPAPER_VER }}' + pegboard_version <- '${{ env.PEGBOARD_VER }}' + + cfg_file <- '/home/rstudio/lesson/config.yaml' + + library("remotes") + cfg <- if (file.exists(cfg_file)) readLines(cfg_file) else character(0) + get_version <- function(x, key = "varnish") { + res <- x[grepl(paste0("^", key, "\\s?:"), x)] + if (length(res)) { + res <- trimws(strsplit(res, ":")[[1]][2]) + # trim quotes + res <- gsub("[\"']", "", res) + if (grepl("^[0-9]", res)) { + res <- paste0("carpentries/", key, "@", res) + } + } else { + res <- "latest" + } + res + } + + varnish_version <- get_version(cfg, key = "varnish") + sandpaper_version <- get_version(cfg, key = "sandpaper") + pegboard_version <- get_version(cfg, key = "pegboard") + if (varnish_version != "latest") { + cat("::group::Installing", varnish_version, "\n") + remotes::install_github(varnish_version) + cat("::endgroup::\n") + } + if (sandpaper_version != "latest") { + cat("::group::Installing", sandpaper_version, "\n") + remotes::install_github(sandpaper_version) + cat("::endgroup::\n") + } + if (pegboard_version != "latest") { + cat("::group::Installing", pegboard_version, "\n") + remotes::install_github(pegboard_version) + cat("::endgroup::\n") + } + shell: Rscript {0} + + - name: Run Container and Build Site + run: | + library(sandpaper) + sandpaper::package_cache_trigger(TRUE) + sandpaper:::ci_deploy(reset = TRUE) + shell: Rscript {0} diff --git a/.github/workflows/docker_pr_receive.yaml b/.github/workflows/docker_pr_receive.yaml new file mode 100644 index 00000000..56a4069b --- /dev/null +++ b/.github/workflows/docker_pr_receive.yaml @@ -0,0 +1,222 @@ +name: "Bot: Receive Pull Request" + +on: + pull_request: + types: + [opened, synchronize, reopened] + workflow_dispatch: + inputs: + pr_number: + type: number + required: true +concurrency: + group: ${{ github.ref }} + cancel-in-progress: true + +jobs: + + preflight: + name: "Preflight: md-outputs exists?" + runs-on: ubuntu-latest + outputs: + branch-exists: ${{ steps.check.outputs.exists }} + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v4 + + - name: "Check if md-outputs branch exists" + id: check + run: | + if [[ -n $(git ls-remote --exit-code --heads origin md-outputs) ]]; then + echo "exists=true" >> $GITHUB_OUTPUT + else + echo "exists=false" >> $GITHUB_OUTPUT + echo "❌ md-outputs branch required. Please merge any open package update PRs, and run the '03 Maintain: Apply Package Cache' and '01: Maintain: Build and Deploy Site' workflows." + exit 1 + fi + + test-pr: + name: "Record PR number" + if: ${{ github.event.action != 'closed' }} && ${{ needs.preflight.outputs.branch-exists == 'true' }} + runs-on: ubuntu-latest + needs: preflight + outputs: + is_valid: ${{ steps.check-pr.outputs.VALID }} + pr_branch: ${{ env.PR_BRANCH }} + steps: + - name: "Auto: Grab PR" + if: ${{ github.event_name == 'pull_request' }} + run: | + echo ${{ github.event.number }} > ${{ github.workspace }}/NR + echo "NR=${{ github.event.number }}" >> $GITHUB_ENV + echo "PR_BRANCH=$(gh -R ${{ github.repository }} pr view ${{ github.event.number }} --json headRefName --jq '.headRefName')" >> $GITHUB_ENV + + - name: "Manual: Grab PR" + if: ${{ github.event_name == 'workflow_dispatch' }} + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + echo ${{ inputs.pr_number }} > ${{ github.workspace }}/NR + echo "NR=${{ inputs.pr_number }}" >> $GITHUB_ENV + echo "PR_BRANCH=$(gh -R ${{ github.repository }} pr view ${{ inputs.pr_number }} --json headRefName --jq '.headRefName')" >> $GITHUB_ENV + + - name: "Upload PR number" + id: upload + if: ${{ always() }} + uses: actions/upload-artifact@v4 + with: + name: pr + path: ${{ github.workspace }}/NR + + - name: "Get Invalid Hashes File" + id: hash + run: | + echo "json<> $GITHUB_OUTPUT + + - name: "echo output" + run: | + echo "${{ steps.hash.outputs.json }}" + + - name: "Check PR" + id: check-pr + uses: carpentries/actions/check-valid-pr@main + with: + pr: ${{ env.NR }} + invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} + + check-renv: + name: "Check If We Need {renv}" + runs-on: ubuntu-latest + outputs: + renv-needed: ${{ steps.check-for-renv.outputs.exists }} + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v4 + + - name: "Check for renv" + id: check-for-renv + run: | + if [[ -d renv ]]; then + echo "exists=true" >> $GITHUB_OUTPUT + fi + + prepare: + name: "Grab renv.lock hash" + runs-on: ubuntu-latest + needs: check-renv + + outputs: + renv-cache-hashsum: ${{ steps.set-hash.outputs.renv-cache-hashsum }} + steps: + - uses: actions/checkout@v4 + + - name: Calculate renv hash + id: set-hash + run: | + echo "renv-cache-hashsum=${{ hashFiles('renv/profiles/lesson-requirements/renv.lock') }}" >> $GITHUB_OUTPUT + + build-md-source: + name: "Build markdown source files if valid" + needs: + - test-pr + - check-renv + runs-on: ubuntu-latest + if: ${{ needs.test-pr.outputs.is_valid == 'true' }} + env: + CHIVE: ${{ github.workspace }}/site/chive + PR: ${{ github.workspace }}/site/pr + GHWMD: ${{ github.workspace }}/site/built + PR_BRANCH: ${{ needs.test-pr.outputs.pr_branch }} + permissions: + checks: write + contents: write + pages: write + container: + image: carpentries/workbench-docker:${{ vars.WORKBENCH_TAG || 'latest' }} + env: + WORKBENCH_PROFILE: "ci" + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: /home/rstudio/lesson/renv + RENV_PROFILE: "lesson-requirements" + RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library" + volumes: + - ${{ github.workspace }}:/home/rstudio/lesson + options: --cpus 2 + steps: + - name: "Check Out PR Branch" + uses: actions/checkout@v4 + with: + ref: ${{ env.PR_BRANCH }} + + - name: "Check Out Staging Branch" + uses: actions/checkout@v4 + with: + ref: md-outputs + path: ${{ env.GHWMD }} + + - name: Mark Repository as Safe + run: | + git config --global --add safe.directory $(pwd) + + - name: "Ensure sandpaper is loadable" + run: | + .libPaths() + library(sandpaper) + shell: Rscript {0} + + - name: Setup Lesson Dependencies + run: | + Rscript /home/rstudio/.workbench/setup_lesson_deps.R + + - name: Fortify renv Cache + if: ${{ needs.check-renv.outputs.renv-needed == 'true' }} + run: | + Rscript /home/rstudio/.workbench/fortify_renv_cache.R + + - name: "Validate and Build Markdown" + id: build-site + run: | + sandpaper::package_cache_trigger(TRUE) + sandpaper::validate_lesson(path = '/home/rstudio/lesson') + sandpaper:::build_markdown(path = '/home/rstudio/lesson', quiet = FALSE) + shell: Rscript {0} + + - name: "Generate Artifacts" + id: generate-artifacts + run: | + sandpaper:::ci_bundle_pr_artifacts( + repo = '${{ github.repository }}', + pr_number = '${{ github.event.number }}', + path_md = '/home/rstudio/lesson/site/built', + path_pr = '/home/rstudio/lesson/site/pr', + path_archive = '/home/rstudio/lesson/site/chive', + branch = 'md-outputs' + ) + shell: Rscript {0} + + - name: "Upload PR" + uses: actions/upload-artifact@v4 + with: + name: pr + path: ${{ env.PR }} + overwrite: true + + - name: "Upload Diff" + uses: actions/upload-artifact@v4 + with: + name: diff + path: ${{ env.CHIVE }} + retention-days: 1 + + - name: "Upload Build" + uses: actions/upload-artifact@v4 + with: + name: built + path: ${{ env.GHWMD }} + retention-days: 1 + + - name: "Teardown" + run: sandpaper::reset_site() + shell: Rscript {0} diff --git a/.github/workflows/pr-receive.yaml b/.github/workflows/pr-receive.yaml deleted file mode 100644 index 7fbff6cd..00000000 --- a/.github/workflows/pr-receive.yaml +++ /dev/null @@ -1,132 +0,0 @@ -name: "Receive Pull Request" - -on: - pull_request: - types: - [opened, synchronize, reopened] - -concurrency: - group: ${{ github.ref }} - cancel-in-progress: true - -jobs: - test-pr: - name: "Record PR number" - if: ${{ github.event.action != 'closed' }} - runs-on: ubuntu-22.04 - outputs: - is_valid: ${{ steps.check-pr.outputs.VALID }} - steps: - - name: "Record PR number" - id: record - if: ${{ always() }} - run: | - echo ${{ github.event.number }} > ${{ github.workspace }}/NR # 2022-03-02: artifact name fixed to be NR - - name: "Upload PR number" - id: upload - if: ${{ always() }} - uses: actions/upload-artifact@v4 - with: - name: pr - path: ${{ github.workspace }}/NR - - name: "Get Invalid Hashes File" - id: hash - run: | - echo "json<> $GITHUB_OUTPUT - - name: "echo output" - run: | - echo "${{ steps.hash.outputs.json }}" - - name: "Check PR" - id: check-pr - uses: carpentries/actions/check-valid-pr@main - with: - pr: ${{ github.event.number }} - invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} - - build-md-source: - name: "Build markdown source files if valid" - needs: test-pr - runs-on: ubuntu-22.04 - if: ${{ needs.test-pr.outputs.is_valid == 'true' }} - env: - GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} - RENV_PATHS_ROOT: ~/.local/share/renv/ - CHIVE: ${{ github.workspace }}/site/chive - PR: ${{ github.workspace }}/site/pr - MD: ${{ github.workspace }}/site/built - steps: - - name: "Check Out Main Branch" - uses: actions/checkout@v4 - - - name: "Check Out Staging Branch" - uses: actions/checkout@v4 - with: - ref: md-outputs - path: ${{ env.MD }} - - - name: "Set up R" - uses: r-lib/actions/setup-r@v2 - with: - use-public-rspm: true - install-r: false - - - name: "Set up Pandoc" - uses: r-lib/actions/setup-pandoc@v2 - - - name: "Setup Lesson Engine" - uses: carpentries/actions/setup-sandpaper@main - with: - cache-version: ${{ secrets.CACHE_VERSION }} - - - name: "Setup Package Cache" - uses: carpentries/actions/setup-lesson-deps@main - with: - cache-version: ${{ secrets.CACHE_VERSION }} - - - name: "Validate and Build Markdown" - id: build-site - run: | - sandpaper::package_cache_trigger(TRUE) - sandpaper::validate_lesson(path = '${{ github.workspace }}') - sandpaper:::build_markdown(path = '${{ github.workspace }}', quiet = FALSE) - shell: Rscript {0} - - - name: "Generate Artifacts" - id: generate-artifacts - run: | - sandpaper:::ci_bundle_pr_artifacts( - repo = '${{ github.repository }}', - pr_number = '${{ github.event.number }}', - path_md = '${{ env.MD }}', - path_pr = '${{ env.PR }}', - path_archive = '${{ env.CHIVE }}', - branch = 'md-outputs' - ) - shell: Rscript {0} - - - name: "Upload PR" - uses: actions/upload-artifact@v4 - with: - name: pr - path: ${{ env.PR }} - overwrite: true - - - name: "Upload Diff" - uses: actions/upload-artifact@v4 - with: - name: diff - path: ${{ env.CHIVE }} - retention-days: 1 - - - name: "Upload Build" - uses: actions/upload-artifact@v4 - with: - name: built - path: ${{ env.MD }} - retention-days: 1 - - - name: "Teardown" - run: sandpaper::reset_site() - shell: Rscript {0} diff --git a/.github/workflows/sandpaper-main.yaml b/.github/workflows/sandpaper-main.yaml deleted file mode 100644 index b3d1de8c..00000000 --- a/.github/workflows/sandpaper-main.yaml +++ /dev/null @@ -1,64 +0,0 @@ -name: "01 Build and Deploy Site" - -on: - push: - branches: - - main - - master - schedule: - - cron: '0 0 * * 2' - workflow_dispatch: - inputs: - name: - description: 'Who triggered this build?' - required: true - default: 'Maintainer (via GitHub)' - reset: - description: 'Reset cached markdown files' - required: false - default: false - type: boolean -jobs: - full-build: - name: "Build Full Site" - - # 2024-10-01: ubuntu-latest is now 24.04 and R is not installed by default in the runner image - # pin to 22.04 for now - runs-on: ubuntu-22.04 - permissions: - checks: write - contents: write - pages: write - env: - GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} - RENV_PATHS_ROOT: ~/.local/share/renv/ - steps: - - - name: "Checkout Lesson" - uses: actions/checkout@v4 - - - name: "Set up R" - uses: r-lib/actions/setup-r@v2 - with: - use-public-rspm: true - install-r: false - - - name: "Set up Pandoc" - uses: r-lib/actions/setup-pandoc@v2 - - - name: "Setup Lesson Engine" - uses: carpentries/actions/setup-sandpaper@main - with: - cache-version: ${{ secrets.CACHE_VERSION }} - - - name: "Setup Package Cache" - uses: carpentries/actions/setup-lesson-deps@main - with: - cache-version: ${{ secrets.CACHE_VERSION }} - - - name: "Deploy Site" - run: | - reset <- "${{ github.event.inputs.reset }}" == "true" - sandpaper::package_cache_trigger(TRUE) - sandpaper:::ci_deploy(reset = reset) - shell: Rscript {0} diff --git a/.github/workflows/sandpaper-version.txt b/.github/workflows/sandpaper-version.txt index ea98690d..918eaca8 100644 --- a/.github/workflows/sandpaper-version.txt +++ b/.github/workflows/sandpaper-version.txt @@ -1 +1 @@ -0.16.12 +0.16.13.9000 diff --git a/.github/workflows/update-cache.yaml b/.github/workflows/update-cache.yaml index a011c0c0..2da7240c 100644 --- a/.github/workflows/update-cache.yaml +++ b/.github/workflows/update-cache.yaml @@ -1,4 +1,4 @@ -name: "03 Maintain: Update Package Cache" +name: "02 Maintain: Check for Updated Packages" on: workflow_dispatch: @@ -7,20 +7,40 @@ on: description: 'Who triggered this build (enter github username to tag yourself)?' required: true default: 'monthly run' + skip-restore: + description: 'Force lockfile update?' + required: false + default: false + type: boolean + update-packages: + description: 'Install any package updates?' + required: false + default: true + type: boolean + generate-cache: + description: 'Generate separate package cache?' + required: false + default: false + type: boolean schedule: # Run every tuesday - cron: '0 0 * * 2' +env: + LOCKFILE_CACHE_GEN: ${{ github.event.inputs.generate-cache || 'false' }} + SKIP_RENV_RESTORE: ${{ github.event.inputs.skip-restore || 'false' }} + UPDATE_PACKAGES: ${{ github.event.inputs.update-packages || 'true' }} + jobs: preflight: - name: "Preflight Check" - runs-on: ubuntu-22.04 + name: "Preflight: Manual or Scheduled Trigger?" + runs-on: ubuntu-latest outputs: ok: ${{ steps.check.outputs.ok }} steps: - id: check run: | - if [[ ${{ github.event_name }} == 'workflow_dispatch' ]]; then + if [[ "${{ github.event_name }}" == 'workflow_dispatch' ]]; then echo "ok=true" >> $GITHUB_OUTPUT echo "Running on request" # using single brackets here to avoid 08 being interpreted as octal @@ -34,47 +54,36 @@ jobs: echo "Not Running Today" fi - check_renv: - name: "Check if We Need {renv}" - runs-on: ubuntu-22.04 + check-renv: + name: "Check If We Need {renv}" + runs-on: ubuntu-latest needs: preflight if: ${{ needs.preflight.outputs.ok == 'true'}} outputs: - needed: ${{ steps.renv.outputs.exists }} + renv-needed: ${{ steps.check-for-renv.outputs.exists }} steps: - name: "Checkout Lesson" uses: actions/checkout@v4 - - id: renv + - id: check-for-renv run: | if [[ -d renv ]]; then echo "exists=true" >> $GITHUB_OUTPUT fi - check_token: - name: "Check SANDPAPER_WORKFLOW token" - runs-on: ubuntu-22.04 - needs: check_renv - if: ${{ needs.check_renv.outputs.needed == 'true' }} - outputs: - workflow: ${{ steps.validate.outputs.wf }} - repo: ${{ steps.validate.outputs.repo }} - steps: - - name: "validate token" - id: validate - uses: carpentries/actions/check-valid-credentials@main - with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} - update_cache: - name: "Update Package Cache" - needs: check_token - if: ${{ needs.check_token.outputs.repo== 'true' }} + name: "Create Package Update Pull Request" runs-on: ubuntu-22.04 + needs: check-renv + permissions: + contents: write + pull-requests: write + actions: write + issues: write + if: ${{ needs.check-renv.outputs.renv-needed == 'true' }} env: GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} RENV_PATHS_ROOT: ~/.local/share/renv/ steps: - - name: "Checkout Lesson" uses: actions/checkout@v4 @@ -86,8 +95,11 @@ jobs: - name: "Update {renv} deps and determine if a PR is needed" id: update - uses: carpentries/actions/update-lockfile@main + uses: froggleston/actions/update-lockfile@main with: + update: ${{ env.UPDATE_PACKAGES }} + skip-restore: ${{ env.SKIP_RENV_RESTORE }} + generate-cache: ${{ env.LOCKFILE_CACHE_GEN }} cache-version: ${{ secrets.CACHE_VERSION }} - name: Create Pull Request @@ -95,7 +107,7 @@ jobs: if: ${{ steps.update.outputs.n > 0 }} uses: carpentries/create-pull-request@main with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} + token: ${{ secrets.GITHUB_TOKEN }} delete-branch: true branch: "update/packages" commit-message: "[actions] update ${{ steps.update.outputs.n }} packages" @@ -123,3 +135,16 @@ jobs: [1]: https://github.com/carpentries/create-pull-request/tree/main labels: "type: package cache" draft: false + + - name: Skip PR creation + if: ${{ steps.update.outputs.n == 0 }} + run: | + echo "No updates needed, skipping PR creation" + + # thanks @Bisaloo! - https://github.com/carpentries/sandpaper/issues/646#issuecomment-2829578435 + - name: Trigger checks + if: ${{ steps.cpr.outputs.pull-request-number != '' }} + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + gh workflow run docker_pr_receive.yaml --field pr_number=${{ steps.cpr.outputs.pull-request-number }} diff --git a/.github/workflows/update-workflows.yaml b/.github/workflows/update-workflows.yaml index 6414cf28..7689c08a 100644 --- a/.github/workflows/update-workflows.yaml +++ b/.github/workflows/update-workflows.yaml @@ -1,4 +1,4 @@ -name: "02 Maintain: Update Workflow Files" +name: "04 Maintain: Update Workflow Files" on: workflow_dispatch: @@ -7,18 +7,19 @@ on: description: 'Who triggered this build (enter github username to tag yourself)?' required: true default: 'weekly run' + tarball: + description: 'Absolute URL to the desired sandpaper repo tarball' + required: false + default: '' clean: description: 'Workflow files/file extensions to clean (no wildcards, enter "" for none)' required: false default: '.yaml' - schedule: - # Run every Tuesday - - cron: '0 0 * * 2' jobs: check_token: name: "Check SANDPAPER_WORKFLOW token" - runs-on: ubuntu-22.04 + runs-on: ubuntu-latest outputs: workflow: ${{ steps.validate.outputs.wf }} repo: ${{ steps.validate.outputs.repo }} @@ -31,7 +32,7 @@ jobs: update_workflow: name: "Update Workflow" - runs-on: ubuntu-22.04 + runs-on: ubuntu-latest needs: check_token if: ${{ needs.check_token.outputs.workflow == 'true' }} steps: @@ -40,9 +41,10 @@ jobs: - name: Update Workflows id: update - uses: carpentries/actions/update-workflows@main + uses: froggleston/actions/update-workflows@main with: - clean: ${{ github.event.inputs.clean }} + repo: ${{ github.event.inputs.tarball }} || "https://carpentries.r-universe.dev" + clean: ${{ github.event.inputs.clean }} || ".yaml" - name: Create Pull Request id: cpr