Skip to content

Commit 50ba1cd

Browse files
marcpruxfinagolfin
andauthored
Run the compiler validation suite for Android (#8)
* Run the compiler validation suite for Android * Add --build-compiler option * Add --cross-compile-build-swift-tools=0 from swiftlang/swift#38441 * Build with --build-llvm=0 * Check out Yams for Swift 6.1.1 build * Add --llvm-ninja-targets-for-cross-compile-hosts=help * Install pre-requisites * Install build prerequisites * Only setup local toolchain if build-compiler is not 0 * Fix Yams version checkout * Quote arguments to build scripts * Permit empty host-toolchain argument in build.sh * Fix check for BUILD_COMPILER * Re-order Docker PATH to system clang is used before Swift toolchain clang * Use --host-test to skip attempt to test on connected device/emulator * Install clang in Dockerfile * Add --skip-test-linux flag to build * Remove more folders to free up space * Update patches * Add docker-specific CI variants that run the compiler validation tests * meaningless edit * Update pull_request.yml to stop the 6.1 release builds * Update pull_request.yml to really only build the full docker compiler with tests * Update README.md with meaningless edit to bump build * Update build.sh to skip testing XCTest for linux, as a handful of the linux tests fail for some reason * Update build.sh to disable building libTesting and for 16K memory pages, as both don't work yet * Update build.sh to skip testing Foundation for linux, which requires building SwiftPM from source first * Update build.sh to not build each arch in a separate build directory * Update Dockerfile to use clang 19 instead * Add self-hosted runner CI variant * Fix CI workflow syntax * Update CI runner config * Update CI runner config * Fix name of self-hosted CI runner * Re-order arch run sequence * Update CI for self-hosted runner * Build for self-hosted runner * Add swift-6.2-branch to self-hosted run matrix * Update build.sh to disable aarch64 temporarily, as we know it built fine * Update pull_request.yml to disable non-compiler builds, as they all work, and comment out self-hosted till we know armv7 works * Run compiler validation on self-hosted with increased timeout * Update build-docker to only build for armv7 * Update pull_request.yml to disable self-hosted builds temporarily * Update pull_request.yml to build the full compiler in github runners with the armv7 stdlib and tests * Update build-docker to build for aarch64 first * Update toolchain-vars.sh to use older trunk snapahot toolchain that didn't crash * Update toolchain-vars.sh to only download the latest release compiler if building the Swift compiler from source * Update build-docker to only build for armv7 again * Update toolchain-vars.sh to fix setting branch variable * Update build-docker to build for all three supported arches * Update pull_request.yml to try self-hosted runs again * Update build.sh to only install SDK components and remove linux stdlib * Tolerate missing linux folder when attempting to clean up unnecessary build artifacts * Fix extraCLIOptions in swift-toolset.json * gcpdw * Fix patch application * Update apply-patches.sh to remove unused changes * Update swift-android-testing-except-release.patch to add Testing fix * Update swift-android.patch to remove unneeded patches * Update build.sh to try and fix Testing and clean up the bundle more * Fix README.md --------- Co-authored-by: finagolfin <[email protected]>
1 parent ad24ea5 commit 50ba1cd

12 files changed

+573
-76
lines changed

.github/workflows/pull_request.yml

Lines changed: 35 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -23,27 +23,38 @@ jobs:
2323
*.log
2424
2525
android-build:
26-
name: Build Android ${{ matrix.build-type }} ${{ matrix.swift-version }} ${{ matrix.arch }} SDK
26+
name: Android ${{ matrix.build-type }} ${{ matrix.swift-version }} ${{ matrix.arch }} ${{ matrix.runner }} (compiler=${{ matrix.build-compiler }})
2727
strategy:
2828
fail-fast: false
2929
matrix:
30-
#build-type: ['docker']
31-
build-type: ['docker', 'local']
32-
# blank arch builds all (aarch64,x86_64,armv7)
33-
#arch: ['']
34-
# builds only x86_64 to speed up the validation
35-
#arch: ['x86_64']
36-
# build both the quick (x86_64) and complete (aarch64,x86_64,armv7) SDKs
37-
arch: ['x86_64', '']
38-
swift-version: ['release', 'swift-6.2-branch', 'development']
39-
runs-on: ubuntu-24.04
30+
include:
31+
- swift-version: 'swift-6.2-branch'
32+
build-type: 'docker'
33+
build-compiler: '1'
34+
runner: 'self-hosted'
35+
- swift-version: 'development'
36+
build-type: 'docker'
37+
build-compiler: '1'
38+
runner: 'self-hosted'
39+
- swift-version: 'swift-6.2-branch'
40+
build-type: 'docker'
41+
build-compiler: '0'
42+
runner: 'ubuntu-24.04'
43+
- swift-version: 'development'
44+
build-type: 'docker'
45+
build-compiler: '0'
46+
runner: 'ubuntu-24.04'
47+
runs-on: ${{ matrix.runner }}
48+
# 15 hour timeout
49+
timeout-minutes: 900
4050
steps:
4151
- name: Free Disk Space
52+
if: ${{ matrix.runner != 'self-hosted' }}
4253
run: |
4354
df -h
4455
# brings available space from 25G to 32G
4556
# otherwise we sometimes run out of space during the build
46-
sudo rm -rf /usr/share/miniconda /usr/share/az* /usr/share/glade* /usr/local/share/chromium /usr/local/share/powershell /usr/share/dotnet /opt/ghc /opt/hostedtoolcache/CodeQL
57+
sudo rm -rf /usr/share/miniconda /usr/share/az* /usr/share/glade* /usr/local/share/chromium /usr/local/share/powershell /usr/share/dotnet /opt/ghc /opt/hostedtoolcache /usr/local/graalvm/ /usr/local/.ghcup/ /usr/local/lib/node_modules /usr/local/share/boost
4758
sudo docker image prune --all --force
4859
sudo docker builder prune -a
4960
df -h
@@ -53,6 +64,8 @@ jobs:
5364
# these variabes are used by build-docker and build-local
5465
# to determine which Swift version to build for
5566
echo "BUILD_SCHEME=${{ matrix.swift-version }}" >> $GITHUB_ENV
67+
# pass the build-compiler matrix through to the build script
68+
echo "BUILD_COMPILER=${{ matrix.build-compiler }}" >> $GITHUB_ENV
5669
echo "TARGET_ARCHS=${{ matrix.arch }}" >> $GITHUB_ENV
5770
echo "WORKDIR=${{ runner.temp }}/swift-android-sdk" >> $GITHUB_ENV
5871
- name: Checkout repository
@@ -61,7 +74,7 @@ jobs:
6174
if: ${{ matrix.build-type == 'local' }}
6275
working-directory: swift-ci/sdks/android
6376
run: |
64-
sudo apt install -q ninja-build patchelf
77+
sudo apt install -q patchelf build-essential cmake ninja-build python3 golang git gnupg2 libcurl4-openssl-dev libedit-dev libicu-dev libncurses5-dev libpython3-dev libsqlite3-dev libxml2-dev rsync uuid-dev uuid-runtime tzdata curl unzip
6578
./build-local ${BUILD_SCHEME} ${WORKDIR}
6679
- name: Build Android SDK (Docker)
6780
if: ${{ matrix.build-type == 'docker' }}
@@ -99,6 +112,9 @@ jobs:
99112
if [[ "${{ matrix.build-type }}" == 'local' ]]; then
100113
ARTIFACT_NAME="${ARTIFACT_NAME}-local"
101114
fi
115+
if [[ "${{ matrix.build-compiler }}" == '1' ]]; then
116+
ARTIFACT_NAME="${ARTIFACT_NAME}-hostbuild"
117+
fi
102118
# artifacts need a unique name so we suffix with the matrix arch(s)
103119
if [[ ! -z "${{ matrix.arch }}" ]]; then
104120
ARTIFACT_NAME="${ARTIFACT_NAME}-$(echo ${{ matrix.arch }} | tr ',' '-')"
@@ -117,6 +133,7 @@ jobs:
117133
name: ${{ steps.info.outputs.artifact-name }}
118134
path: ${{ steps.info.outputs.artifact-path }}
119135
- name: Cleanup
136+
if: ${{ matrix.runner != 'self-hosted' }}
120137
run: |
121138
# need to free up some space or else when installing we get: No space left on device
122139
df -h
@@ -125,6 +142,7 @@ jobs:
125142
sudo docker builder prune -a
126143
df -h
127144
- name: Install artifactbundle
145+
if: ${{ matrix.runner != 'self-hosted' }}
128146
shell: bash
129147
run: |
130148
set -ex
@@ -135,6 +153,7 @@ jobs:
135153
echo "ANDROID_NDK_ROOT=" >> $GITHUB_ENV
136154
137155
- name: Create Demo Project
156+
if: ${{ matrix.runner != 'self-hosted' }}
138157
run: |
139158
cd ${{ runner.temp }}
140159
mkdir DemoProject
@@ -149,6 +168,7 @@ jobs:
149168
echo 'import Android' >> Sources/DemoProject/DemoProject.swift
150169
- name: Test Demo Project on Android
151170
uses: skiptools/swift-android-action@main
171+
if: ${{ matrix.runner != 'self-hosted' }}
152172
with:
153173
# only test for the complete arch SDK build to speed up CI
154174
#run-tests: ${{ matrix.arch == '' }}
@@ -157,14 +177,15 @@ jobs:
157177
installed-swift: ${{ steps.info.outputs.swift-root }}
158178

159179
- name: Checkout swift-algorithms
180+
if: ${{ matrix.runner != 'self-hosted' }}
160181
uses: actions/checkout@v4
161182
with:
162183
repository: apple/swift-algorithms
163184
path: swift-algorithms
164185
- name: Test swift-algorithms
186+
if: ${{ matrix.runner != 'self-hosted' }}
165187
uses: skiptools/swift-android-action@main
166188
with:
167-
run-tests: ${{ matrix.arch == '' }}
168189
package-path: swift-algorithms
169190
installed-sdk: ${{ steps.info.outputs.sdk-id }}
170191
installed-swift: ${{ steps.info.outputs.swift-root }}

swift-ci/sdks/android/Dockerfile

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ ARG SWIFT_TOOLCHAIN_URL=
1818
RUN apt-get -q update \
1919
&& DEBIAN_FRONTEND=noninteractive apt-get -q install -y \
2020
build-essential \
21+
clang-19 \
2122
cmake \
2223
ninja-build \
2324
python3 \
@@ -54,7 +55,7 @@ ENV SWIFT_SIGNING_KEY=$SWIFT_SIGNING_KEY \
5455
COPY scripts/install-swift.sh /scripts/install-swift.sh
5556
RUN chmod ugo+x /scripts/install-swift.sh
5657
RUN /scripts/install-swift.sh /usr/local/swift
57-
ENV PATH="/usr/local/swift/bin:${PATH}"
58+
ENV PATH="/usr/lib/llvm-19/bin:/usr/local/swift/bin:${PATH}"
5859

5960
ARG ANDROID_NDK_VERSION=
6061

swift-ci/sdks/android/build-docker

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ source ./scripts/toolchain-vars.sh
2727

2828
# Check-out and patch the sources
2929
./scripts/fetch-source.sh --source-dir ${WORKDIR}/source --swift-tag ${SWIFT_TAG}
30-
${WORKDIR}/source/swift-android-patches/apply-patches.sh ${WORKDIR}/source/swift-project
30+
./patches/apply-patches.sh ${WORKDIR}/source/swift-project
3131

3232
mkdir -p ${WORKDIR}/products
3333
chmod ugo+rwx ${WORKDIR}/products
@@ -59,9 +59,10 @@ $DOCKER run -i --rm \
5959
-v ${WORKDIR}/products:/products:rw \
6060
${CONTAINER_NAME} \
6161
/scripts/build.sh \
62-
--source-dir /source \
63-
--products-dir /products \
64-
--host-toolchain /usr/local/swift \
65-
--android-api ${ANDROID_API} \
66-
--ndk-home /usr/local/ndk/${ANDROID_NDK_VERSION} \
67-
--archs ${TARGET_ARCHS}
62+
--source-dir "/source" \
63+
--products-dir "/products" \
64+
--host-toolchain "/usr/local/swift" \
65+
--build-compiler "${BUILD_COMPILER}" \
66+
--android-api "${ANDROID_API}" \
67+
--ndk-home "/usr/local/ndk/${ANDROID_NDK_VERSION}" \
68+
--archs "${TARGET_ARCHS}"

swift-ci/sdks/android/build-local

Lines changed: 19 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -25,16 +25,18 @@ WORKDIR=$(realpath ${WORKDIR})
2525
HOST_OS=ubuntu$(lsb_release -sr)
2626
source ./scripts/toolchain-vars.sh
2727

28-
SWIFT_ROOT=${WORKDIR}/host-toolchain
29-
HOST_TOOLCHAIN=$SWIFT_ROOT/$SWIFT_BASE/usr
30-
if [[ ! -d "$HOST_TOOLCHAIN" ]]; then
31-
./scripts/install-swift.sh ${HOST_TOOLCHAIN}
32-
fi
28+
if [[ "${BUILD_COMPILER}" != "1" ]]; then
29+
SWIFT_ROOT=${WORKDIR}/host-toolchain
30+
HOST_TOOLCHAIN=$SWIFT_ROOT/$SWIFT_BASE/usr
31+
if [[ ! -d "$HOST_TOOLCHAIN" ]]; then
32+
./scripts/install-swift.sh ${HOST_TOOLCHAIN}
33+
fi
3334

34-
$HOST_TOOLCHAIN/bin/swift --version
35+
$HOST_TOOLCHAIN/bin/swift --version
3536

36-
# ensure the correct Swift is first in the PATH
37-
export PATH=$HOST_TOOLCHAIN/bin:$PATH
37+
# ensure the correct Swift is first in the PATH
38+
export PATH=$HOST_TOOLCHAIN/bin:$PATH
39+
fi
3840

3941
export ANDROID_NDK_HOME=${WORKDIR}/ndk/${ANDROID_NDK_VERSION}
4042

@@ -50,15 +52,16 @@ fi
5052

5153
# Check-out and patch the sources
5254
./scripts/fetch-source.sh --source-dir ${WORKDIR}/source --swift-tag ${SWIFT_TAG}
53-
${WORKDIR}/source/swift-android-patches/apply-patches.sh ${WORKDIR}/source/swift-project
55+
./patches/apply-patches.sh ${WORKDIR}/source/swift-project
5456

5557
mkdir -p ${WORKDIR}/products
5658

5759
./scripts/build.sh \
58-
--source-dir ${WORKDIR}/source \
59-
--products-dir ${WORKDIR}/products \
60-
--build-dir ${WORKDIR}/build \
61-
--host-toolchain ${HOST_TOOLCHAIN} \
62-
--android-api ${ANDROID_API} \
63-
--ndk-home ${ANDROID_NDK_HOME} \
64-
--archs ${TARGET_ARCHS}
60+
--source-dir "${WORKDIR}/source" \
61+
--products-dir "${WORKDIR}/products" \
62+
--build-dir "${WORKDIR}/build" \
63+
--build-compiler "${BUILD_COMPILER}" \
64+
--host-toolchain "${HOST_TOOLCHAIN}" \
65+
--android-api "${ANDROID_API}" \
66+
--ndk-home "${ANDROID_NDK_HOME}" \
67+
--archs "${TARGET_ARCHS}"
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/bin/bash -ex
2+
3+
patches_dir=$(dirname $(realpath -- "${BASH_SOURCE[0]}"))
4+
cd ${1}
5+
6+
case "${BUILD_SCHEME}" in
7+
swift-*-branch)
8+
git apply -v -C1 ${patches_dir}/swift-android.patch
9+
git apply -v -C1 ${patches_dir}/swift-android-testing-except-release.patch
10+
;;
11+
development)
12+
git apply -v -C1 ${patches_dir}/swift-android.patch
13+
git apply -v -C1 ${patches_dir}/swift-android-trunk-libdispatch.patch
14+
git apply -v -C1 ${patches_dir}/swift-android-testing-except-release.patch
15+
;;
16+
*)
17+
echo "$0: invalid BUILD_SCHEME=${BUILD_SCHEME}"
18+
exit 1
19+
;;
20+
esac
21+
22+
# disable backtrace() for Android (needs either API33+ or libandroid-execinfo, or to manually add in backtrace backport)
23+
perl -pi -e 's;os\(Android\);os\(AndroidDISABLED\);g' swift-testing/Sources/Testing/SourceAttribution/Backtrace.swift

0 commit comments

Comments
 (0)