Skip to content

Handle GitHub CI secrets in Nova jobs

Jump to bottom
Huy Do edited this page Apr 26, 2025 · 2 revisions

Nova Linux, MacOS, and Windows jobs are the basic building blocks for many CI jobs in PyTorch. They provide a common interface to hide the complexity of setting up the infrastructure from the repo owners so that they can focus more on the actual CI logic. Nevertheless, one of the remaining rough edges in Nova jobs is how it handles secrets. It's important to get this done right because of the security implication involved. Simply said, leaking secret is bad. And the remaining of this wiki will explain how to safely handle GitHub CI secrets in Nova jobs to avoid similar incidents in the future.

What is GitHub CI secret?

Short answer: a GitHub CI secret is a secret string that GitHub keeps in the repository settings. Only repo owners and org admins can add or modify its secrets. The secret can then be used in CI jobs to access remote services either in read-only mode, i.e. HF_TOKEN to download model weights, or with write access, i.e. PYPI_TOKEN.

  • GitHub secrets have scope and can be:
    • Repository secrets which are available for all workflows in the repo
    • [RECOMMEND] Environment secrets that are limited only to specific environments. The key difference is that the environment has protection rules saying the conditions where the access is allowed, i.e. only from the protected main branch.
    • There is also org-wide secrets, but I have never seen one before nor plan to add one. So, let's exclude this.
  • A fork pull request doesn't have access to the secrets of the target repo. In the context of PyTorch where ghstack is widely used, this is a common gotcha because ghstack PRs are non-fork so they have full access to PyTorch repo secrets while regular PRs could be both. Exporting an internal diff could go to either a fork or a non-fork PR too. This subtlety gives the impression that CI jobs that depending on the available of some secrets work for some while fail for other, i.e. #6865
  • Using secrets on a non-ephemeral self-hosted runner exposes the possibility of them being stolen. As such, GitHub recommends using ephemeral runners whenever a secret is needed in CI.
Clone this wiki locally