|
1 | 1 | # Fork PR Testing Guide |
2 | 2 |
|
3 | | -This document explains how to test external pull requests using the dispatch action workflow system. |
| 3 | +This document explains how testing works for external pull requests from forks. |
4 | 4 |
|
5 | 5 | ## Overview |
6 | 6 |
|
7 | | -The testing system consists of three main workflows: |
| 7 | +The testing system consists of two main workflows: |
8 | 8 |
|
9 | | -1. **E2E Tests** (`test-e2e.yml`) - Runs automatically for internal PRs, requires approval for external PRs |
10 | | -2. **Ok To Test** (`ok-to-test.yml`) - Processes slash commands to trigger fork testing |
11 | | -3. **E2E tests [fork]** (`test-e2e-fork.yml`) - Triggered manually via slash commands for external PRs |
| 9 | +1. **E2E Tests** (`test-e2e.yml`) - Runs automatically for internal PRs, need manual trigger on external PRs. |
| 10 | +2. **Ok To Test** (`ok-to-test.yml`) - Dispatches `repository_dispatch` event when maintainer put's `/ok-to-test sha=<commit hash>` comment in the forked PR thread. |
12 | 11 |
|
13 | 12 | ## How It Works |
14 | 13 |
|
15 | | -### 1. Initial PR State |
16 | | -When a PR is created or updated: |
17 | | -- The `test-e2e.yml` workflow automatically runs |
18 | | -- For **external PRs**: The `check-external-pr` job detects it's from a fork and **fails intentionally** |
19 | | -- For **internal PRs**: The workflow proceeds normally with e2e tests |
20 | | -- External PRs show ❌ for the check-external-pr job with a message asking for maintainer approval |
| 14 | +### 1. PR is created by maintainer: |
21 | 15 |
|
22 | | -### 2. Manual Approval Required |
23 | | -**Important**: External PRs require manual approval before workflows can run. |
| 16 | +For the PR created by maintainer `E2E Test` workflow starts automatically. The PR check will reflect the status of the job. |
24 | 17 |
|
25 | | -**Steps for maintainers:** |
26 | | -1. Go to the PR page |
27 | | -2. Click **"Approve workflow to run"** button. For contributors who have made more than one contribution, workflows will start automatically after approval. |
28 | | -3. The workflow will then execute |
| 18 | +### 2. PR is created by external contributor: |
29 | 19 |
|
30 | | -**Note**: The `test-e2e.yml` workflow will still fail for external PRs even after approval, as it's designed to prevent automatic execution. Need to use the slash command instead. |
| 20 | +For the PR created by external contributor `E2E Test` workflow **won't** start automatically. |
| 21 | +Maintainer should make a sanity check of the changes and run it manually by: |
| 22 | +1. Putting a comment `/ok-to-test sha=<latest commit hash>` in the PR thread. |
| 23 | +2. `E2E Test` workflow starts. |
| 24 | +3. After `E2E Test` workflow finishes, the commit with a link and workflow status will appear in the thread. |
| 25 | +4. Maintainer can merge PR or request the changes based on the `E2E Test` results. |
31 | 26 |
|
32 | | -### 3. Testing External PRs |
33 | | -Once the initial checks have run (and failed), maintainers can test the PR using slash commands: |
34 | 27 |
|
35 | | -#### Step-by-Step Process: |
| 28 | +## Notes |
36 | 29 |
|
37 | | -1. **Navigate to the PR** |
38 | | - - Go to the pull request you want to test |
39 | | - |
40 | | -2. **Add a comment with slash command** |
41 | | - ``` |
42 | | - /ok-to-test sha=<commit-sha> |
43 | | - ``` |
44 | | - Replace `<commit-sha>` with the latest commit SHA from the PR. |
45 | | - |
46 | | - **Note**: Use the short SHA. |
47 | | - |
48 | | -3. **Dispatch Action Triggers** |
49 | | - - The `ok-to-test.yml` workflow processes the slash command |
50 | | - - It triggers a `repository_dispatch` event with type `ok-to-test-command` |
51 | | - - The `test-e2e-fork.yml` workflow starts |
52 | | - |
53 | | -4. **Workflow Execution** |
54 | | - The fork workflow runs two jobs: |
55 | | - |
56 | | - **a) `run-e2e-tests` job** (conditional) |
57 | | - - Only runs if: |
58 | | - - Event is `repository_dispatch` |
59 | | - - SHA parameter is not empty |
60 | | - - PR head SHA contains the provided SHA |
61 | | - - Calls the reusable `run-e2e-tests.yml` workflow |
62 | | - - Runs the actual E2E tests if conditions are met |
63 | | - |
64 | | - **b) `update-check-status` job** (conditional) |
65 | | - - Runs after `e2e-tests` completes |
66 | | - - Updates the existing check for job named "run-e2e-tests" from 'test-e2e.yml' workflow |
67 | | - - Sets the conclusion based on `run-e2e-tests` result: |
68 | | - - ✅ **Success** if tests pass |
69 | | - - ❌ **Failure** if tests fail |
70 | | - |
71 | | -## Troubleshooting |
72 | | - |
73 | | -### Workflow Not Running |
74 | | -- **Check**: Ensure you've approved the workflow to run |
75 | | -- **Action**: Click "Approve workflow to run" in the PR checks tab |
76 | | - |
77 | | -### SHA Not Found |
78 | | -- **Check**: Verify the SHA exists in the PR commits |
79 | | -- **Action**: Use `git log --oneline` to find valid commit SHAs |
80 | | - |
81 | | -### Dispatch Not Triggering |
82 | | -- **Check**: Ensure the slash command format is correct |
83 | | -- **Action**: Use exact format: `/ok-to-test sha=<sha>` |
84 | | - |
85 | | -### Check Runs Not Updating |
86 | | -- **Note**: The fork workflow updates the existing "E2E Tests [reusable]" check run |
87 | | -- **Behavior**: The same check run is updated with new results rather than creating a new one |
88 | | - |
89 | | -## Security Notes |
90 | | - |
91 | | -- Only users with **write** permissions can trigger dispatch commands |
92 | | -- The fork workflow runs in the main repository context, allowing it to update check runs |
93 | | -- Manual approval is required for external PR workflows |
94 | | -- External PRs are automatically detected and prevented from running tests automatically |
95 | | - |
96 | | -## Workflow Files |
97 | | - |
98 | | -- **`.github/workflows/ok-to-test.yml`** - Ok To Test - Slash command processor |
99 | | -- **`.github/workflows/test-e2e.yml`** - E2E Tests - Initial PR checks |
100 | | -- **`.github/workflows/test-e2e-fork.yml`** - E2E tests [fork] - Dispatch action handler |
101 | | -- **`.github/workflows/e2e-tests.yml`** - E2E Tests [reusable] - Reusable workflow for actual testing |
102 | | - |
103 | | -## Testing Checklist |
104 | | - |
105 | | -- [ ] PR created with failing check-external-pr check (for external PRs) |
106 | | -- [ ] Workflow approved to run (if required) |
107 | | -- [ ] Slash command posted with valid SHA |
108 | | -- [ ] Dispatch action triggered successfully |
109 | | -- [ ] `check-external-pr` check run updated with correct conclusion |
| 30 | +- Only users with **write** permissions can trigger the `/ok-to-test` command. |
| 31 | +- External PRs are automatically detected and prevented from running e2e tests automatically. |
| 32 | +- Running e2e test on the external PR is optional. Maintainer can merge PR without running it. Maintainer decides whether it's needed to run an E2E test. |
0 commit comments