Skip to content

Commit 9d9d729

Browse files
bpaserobpasero
committed
Web: run integration tests as part of our product builds (fix microsoft#83923)
1 parent 16954b4 commit 9d9d729

File tree

18 files changed

+161
-113
lines changed

18 files changed

+161
-113
lines changed

‎.github/workflows/ci.yml

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -46,9 +46,12 @@ jobs:
4646
- run: yarn download-builtin-extensions
4747
name: Download Built-in Extensions
4848
- run: DISPLAY=:10 ./scripts/test.sh --tfs "Unit Tests"
49-
name: Run Unit Tests
49+
name: Run Unit Tests (Electron)
50+
# Fails with cryptic error (e.g. https://github.com/microsoft/vscode/pull/90292/checks?check_run_id=433681926#step:13:9)
51+
# - run: DISPLAY=:10 yarn test-browser --browser chromium
52+
# name: Run Unit Tests (Browser)
5053
- run: DISPLAY=:10 ./scripts/test-integration.sh --tfs "Integration Tests"
51-
name: Run Integration Tests
54+
name: Run Integration Tests (Electron)
5255

5356
windows:
5457
runs-on: windows-2016
@@ -78,9 +81,11 @@ jobs:
7881
- run: yarn download-builtin-extensions
7982
name: Download Built-in Extensions
8083
- run: .\scripts\test.bat --tfs "Unit Tests"
81-
name: Run Unit Tests
84+
name: Run Unit Tests (Electron)
85+
- run: yarn test-browser --browser chromium
86+
name: Run Unit Tests (Browser)
8287
- run: .\scripts\test-integration.bat --tfs "Integration Tests"
83-
name: Run Integration Tests
88+
name: Run Integration Tests (Electron)
8489

8590
darwin:
8691
runs-on: macos-latest
@@ -107,6 +112,8 @@ jobs:
107112
- run: yarn download-builtin-extensions
108113
name: Download Built-in Extensions
109114
- run: ./scripts/test.sh --tfs "Unit Tests"
110-
name: Run Unit Tests
115+
name: Run Unit Tests (Electron)
116+
- run: yarn test-browser --browser chromium --browser webkit
117+
name: Run Unit Tests (Browser)
111118
- run: ./scripts/test-integration.sh --tfs "Integration Tests"
112-
name: Run Integration Tests
119+
name: Run Integration Tests (Electron)

‎build/azure-pipelines/darwin/continuous-build-darwin.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,13 +40,13 @@ steps:
4040
displayName: Download Built-in Extensions
4141
- script: |
4242
./scripts/test.sh --tfs "Unit Tests"
43-
displayName: Run Unit Tests
43+
displayName: Run Unit Tests (Electron)
4444
- script: |
4545
yarn test-browser --browser chromium --browser webkit
46-
displayName: Run Unit Tests (Browsers)
46+
displayName: Run Unit Tests (Browser)
4747
- script: |
4848
./scripts/test-integration.sh --tfs "Integration Tests"
49-
displayName: Run Integration Tests
49+
displayName: Run Integration Tests (Electron)
5050
- task: PublishTestResults@2
5151
displayName: Publish Tests Results
5252
inputs:

‎build/azure-pipelines/darwin/product-build-darwin.yml

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -96,9 +96,7 @@ steps:
9696
- script: |
9797
set -e
9898
./scripts/test.sh --build --tfs "Unit Tests"
99-
# APP_NAME="`ls $(agent.builddirectory)/VSCode-darwin | head -n 1`"
100-
# yarn smoketest -- --build "$(agent.builddirectory)/VSCode-darwin/$APP_NAME"
101-
displayName: Run unit tests
99+
displayName: Run unit tests (Electron)
102100
condition: and(succeeded(), eq(variables['VSCODE_STEP_ON_IT'], 'false'))
103101

104102
- script: |
@@ -111,18 +109,22 @@ steps:
111109
INTEGRATION_TEST_ELECTRON_PATH="$APP_ROOT/$APP_NAME/Contents/MacOS/Electron" \
112110
VSCODE_REMOTE_SERVER_PATH="$(agent.builddirectory)/vscode-reh-darwin" \
113111
./scripts/test-integration.sh --build --tfs "Integration Tests"
114-
displayName: Run integration tests
112+
displayName: Run integration tests (Electron)
115113
condition: and(succeeded(), eq(variables['VSCODE_STEP_ON_IT'], 'false'))
116114

117115
- script: |
118116
set -e
119-
cd test/smoke
120-
yarn compile
121-
cd -
122117
VSCODE_REMOTE_SERVER_PATH="$(agent.builddirectory)/vscode-reh-web-darwin" \
123-
yarn smoketest --web --headless
118+
./resources/server/test/test-web-integration.sh --browser webkit
119+
displayName: Run integration tests (Browser)
120+
condition: and(succeeded(), eq(variables['VSCODE_STEP_ON_IT'], 'false'))
121+
122+
- script: |
123+
set -e
124+
VSCODE_REMOTE_SERVER_PATH="$(agent.builddirectory)/vscode-reh-web-darwin" \
125+
yarn smoketest --web --headless --browser webkit
124126
continueOnError: true
125-
displayName: Run web smoke tests
127+
displayName: Run smoke tests (Browser)
126128
condition: and(succeeded(), eq(variables['VSCODE_STEP_ON_IT'], 'false'))
127129

128130
- script: |

‎build/azure-pipelines/linux/continuous-build-linux.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,13 +48,13 @@ steps:
4848
displayName: Download Built-in Extensions
4949
- script: |
5050
DISPLAY=:10 ./scripts/test.sh --tfs "Unit Tests"
51-
displayName: Run Unit Tests
51+
displayName: Run Unit Tests (Electron)
5252
- script: |
5353
DISPLAY=:10 yarn test-browser --browser chromium
5454
displayName: Run Unit Tests (Browser)
5555
- script: |
5656
DISPLAY=:10 ./scripts/test-integration.sh --tfs "Integration Tests"
57-
displayName: Run Integration Tests
57+
displayName: Run Integration Tests (Electron)
5858
- task: PublishTestResults@2
5959
displayName: Publish Tests Results
6060
inputs:

‎build/azure-pipelines/linux/product-build-linux.yml

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ steps:
101101
- script: |
102102
set -e
103103
DISPLAY=:10 ./scripts/test.sh --build --tfs "Unit Tests"
104-
displayName: Run unit tests
104+
displayName: Run unit tests (Electron)
105105
condition: and(succeeded(), eq(variables['VSCODE_STEP_ON_IT'], 'false'))
106106

107107
- script: |
@@ -114,8 +114,23 @@ steps:
114114
INTEGRATION_TEST_ELECTRON_PATH="$APP_ROOT/$APP_NAME" \
115115
VSCODE_REMOTE_SERVER_PATH="$(agent.builddirectory)/vscode-reh-linux-x64" \
116116
DISPLAY=:10 ./scripts/test-integration.sh --build --tfs "Integration Tests"
117-
# yarn smoketest -- --build "$(agent.builddirectory)/VSCode-linux-x64"
118-
displayName: Run integration tests
117+
displayName: Run integration tests (Electron)
118+
condition: and(succeeded(), eq(variables['VSCODE_STEP_ON_IT'], 'false'))
119+
120+
# Fails due to weird error: Protocol error (Target.getBrowserContexts): Target closed.
121+
# - script: |
122+
# set -e
123+
# VSCODE_REMOTE_SERVER_PATH="$(agent.builddirectory)/vscode-reh-web-linux-x64" \
124+
# DISPLAY=:10 ./resources/server/test/test-web-integration.sh --browser chromium
125+
# displayName: Run integration tests (Browser)
126+
# condition: and(succeeded(), eq(variables['VSCODE_STEP_ON_IT'], 'false'))
127+
128+
- script: |
129+
set -e
130+
VSCODE_REMOTE_SERVER_PATH="$(agent.builddirectory)/vscode-reh-web-linux-x64" \
131+
yarn smoketest --web --headless --browser firefox
132+
continueOnError: true
133+
displayName: Run smoke tests (Firefox)
119134
condition: and(succeeded(), eq(variables['VSCODE_STEP_ON_IT'], 'false'))
120135

121136
- script: |

‎build/azure-pipelines/win32/continuous-build-win32.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,13 +45,13 @@ steps:
4545
displayName: Download Built-in Extensions
4646
- powershell: |
4747
.\scripts\test.bat --tfs "Unit Tests"
48-
displayName: Run Unit Tests
48+
displayName: Run Unit Tests (Electron)
4949
- powershell: |
50-
yarn test-browser --browser chromium --browser webkit
50+
yarn test-browser --browser chromium
5151
displayName: Run Unit Tests (Browser)
5252
- powershell: |
5353
.\scripts\test-integration.bat --tfs "Integration Tests"
54-
displayName: Run Integration Tests
54+
displayName: Run Integration Tests (Electron)
5555
- task: PublishTestResults@2
5656
displayName: Publish Tests Results
5757
inputs:

‎build/azure-pipelines/win32/product-build-win32.yml

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,7 @@ steps:
109109
$ErrorActionPreference = "Stop"
110110
exec { yarn electron $(VSCODE_ARCH) }
111111
exec { .\scripts\test.bat --build --tfs "Unit Tests" }
112-
displayName: Run unit tests
112+
displayName: Run unit tests (Electron)
113113
condition: and(succeeded(), eq(variables['VSCODE_STEP_ON_IT'], 'false'))
114114

115115
- powershell: |
@@ -122,7 +122,22 @@ steps:
122122
$AppProductJson = Get-Content -Raw -Path "$AppRoot\resources\app\product.json" | ConvertFrom-Json
123123
$AppNameShort = $AppProductJson.nameShort
124124
exec { $env:INTEGRATION_TEST_ELECTRON_PATH = "$AppRoot\$AppNameShort.exe"; $env:VSCODE_REMOTE_SERVER_PATH = "$(agent.builddirectory)\vscode-reh-win32-$(VSCODE_ARCH)"; .\scripts\test-integration.bat --build --tfs "Integration Tests" }
125-
displayName: Run integration tests
125+
displayName: Run integration tests (Electron)
126+
condition: and(succeeded(), eq(variables['VSCODE_STEP_ON_IT'], 'false'))
127+
128+
- powershell: |
129+
. build/azure-pipelines/win32/exec.ps1
130+
$ErrorActionPreference = "Stop"
131+
exec { $env:VSCODE_REMOTE_SERVER_PATH = "$(agent.builddirectory)\vscode-reh-web-win32-$(VSCODE_ARCH)"; .\resources\server\test\test-web-integration.bat --browser chromium }
132+
displayName: Run integration tests (Browser)
133+
condition: and(succeeded(), eq(variables['VSCODE_STEP_ON_IT'], 'false'))
134+
135+
- powershell: |
136+
. build/azure-pipelines/win32/exec.ps1
137+
$ErrorActionPreference = "Stop"
138+
exec { $env:VSCODE_REMOTE_SERVER_PATH = "$(agent.builddirectory)\vscode-reh-web-win32-$(VSCODE_ARCH)"; yarn smoketest --web --headless --browser chromium }
139+
continueOnError: true
140+
displayName: Run smoke tests (Browser)
126141
condition: and(succeeded(), eq(variables['VSCODE_STEP_ON_IT'], 'false'))
127142

128143
- task: SFP.build-tasks.custom-build-task-1.EsrpCodeSigning@1

‎extensions/vscode-api-tests/src/singlefolder-tests/debug.test.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ suite('Debug', function () {
3737
disposeAll(toDispose);
3838
});
3939

40+
// TODO@isidor seems to fail
4041
test.skip('start debugging', async function () {
4142
assert.equal(debug.activeDebugSession, undefined);
4243
let stoppedEvents = 0;

‎extensions/vscode-api-tests/src/singlefolder-tests/workspace.test.ts

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -214,7 +214,12 @@ suite('workspace-namespace', () => {
214214
});
215215
});
216216

217-
test('eol, change via onWillSave', () => {
217+
test('eol, change via onWillSave', function () {
218+
if (vscode.env.uiKind === vscode.UIKind.Web) {
219+
// TODO@Jo Test seems to fail when running in web due to
220+
// onWillSaveTextDocument not getting called
221+
return this.skip();
222+
}
218223

219224
let called = false;
220225
let sub = vscode.workspace.onWillSaveTextDocument(e => {

‎package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"name": "code-oss-dev",
33
"version": "1.43.0",
4-
"distro": "b975cdab7c0c0e7705fa7eac4c8411ad8a790f4b",
4+
"distro": "7568129186570b349ac1b07dea1c0d640a36304f",
55
"author": {
66
"name": "Microsoft Corporation"
77
},

‎src/vs/workbench/services/extensions/browser/extensionService.ts

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -140,8 +140,10 @@ export class ExtensionService extends AbstractExtensionService implements IExten
140140
}
141141

142142
public _onExtensionHostExit(code: number): void {
143-
console.log(`vscode:exit`, code);
144-
// ipc.send('vscode:exit', code);
143+
// We log the exit code to the console. Do NOT remove this
144+
// code as the automated integration tests in browser rely
145+
// on this message to exit properly.
146+
console.log(`vscode:exit ${code}`);
145147
}
146148
}
147149

‎test/README.md

Lines changed: 3 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -3,34 +3,7 @@
33
## Contents
44

55
This folder contains the various test runners for VSCode. Please refer to the documentation within for how to run them:
6-
* `unit`: our suite of unit tests
7-
* `integration`: our suite of API tests
8-
* `smoke`: our suite of automated UI tests
6+
* `unit`: our suite of unit tests ([README](unit/README.md))
7+
* `integration`: our suite of API tests ([README](integration/browser/README.md))
8+
* `smoke`: our suite of automated UI tests ([README](smoke/README.md))
99
* `ui`: our suite of manual UI tests
10-
11-
12-
13-
### Unit Tests (Electron-runner)
14-
15-
```
16-
./scripts/test.[sh|bat]
17-
```
18-
19-
All unit tests are run inside a electron-browser environment which access to DOM and Nodejs api. This is the closest to the enviroment in which VS Code itself ships. Notes:
20-
21-
- use the `--debug` to see an electron window with dev tools which allows for debugging
22-
- to run only a subset of tests use the `--run` or `--glob` options
23-
24-
### Unit Tests (Browser-runner)
25-
26-
```
27-
yarn test-browser --browser webkit --browser chromium
28-
```
29-
30-
Unit tests from layers `common` and `browser` are run inside `chromium`, `webkit`, and (soon’ish) `firefox` (using playwright). This complements our electron-based unit test runner and adds more coverage of supported platforms. Notes:
31-
32-
- these tests are part of the continuous build, that means you might have test failures that only happen with webkit on _windows_ or _chromium_ on linux
33-
- you can these tests locally via yarn `test-browser --browser chromium --browser webkit`
34-
- to debug, open `<vscode>/test/unit/browser/renderer.html` inside a browser and use the `?m=<amd_module>`-query to specify what AMD module to load, e.g `file:///Users/jrieken/Code/vscode/test/unit/browser/renderer.html?m=vs/base/test/common/strings.test` runs all tests from `strings.test.ts`
35-
- to run only a subset of tests use the `--run` or `--glob` options
36-

‎test/integration/browser/README.md

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,19 @@
1-
# VS Code Integration test
1+
# Integration test
22

3-
### Run
3+
## Compile
44

5-
```bash
5+
Make sure to run the following command to compile and install dependencies:
66

7-
# Dev (Electron)
8-
scripts/test-integration.sh
7+
yarn --cwd test/integration/browser
98

10-
# Dev (Web)
11-
node test/integration/browser/out/index.js
9+
## Run (inside Electron)
1210

13-
```
11+
scripts/test-integration.[sh|bat]
12+
13+
All integration tests run in an Electron instance. You can specify to run the tests against a real build by setting the environment variables `INTEGRATION_TEST_ELECTRON_PATH` and `VSCODE_REMOTE_SERVER_PATH` (if you want to include remote tests).
14+
15+
## Run (inside browser)
16+
17+
resources/server/test/test-web-integration.[sh|bat] --browser [chromium|webkit]
18+
19+
All integration tests run in a browser instance as specified by the command line arguments.

‎test/integration/browser/package.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@
1313
"@types/tmp": "^0.1.0",
1414
"rimraf": "^2.6.1",
1515
"tmp": "0.0.33",
16-
"typescript": "3.7.5"
16+
"tree-kill": "1.2.2",
17+
"typescript": "3.7.5",
18+
"vscode-uri": "2.1.1"
1719
}
1820
}

0 commit comments

Comments
 (0)