Skip to content

Commit efca54a

Browse files
rjtokenringrjtokenring
authored
[WIRE-575] CI/CD refactoring (#94)
Refactored PR to remove no appigentools and move to a pure openapi-generator implementation
1 parent 0e613c7 commit efca54a

File tree

26 files changed

+444
-287
lines changed

26 files changed

+444
-287
lines changed

‎.github/workflows/generate.yaml

Lines changed: 0 additions & 49 deletions
This file was deleted.

‎.github/workflows/release.yaml

Lines changed: 158 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,170 @@
11
name: release
22

3-
on: workflow_dispatch
4-
3+
on:
4+
workflow_dispatch:
5+
pull_request:
6+
push:
7+
tags:
8+
- 'v[0-9].*'
9+
510
env:
611
GENERATOR_VERSION: 'v7.9.0'
12+
INPUT_OPENAPI_FILE: 'openapi/openapi.yaml'
13+
GO_VERSION: '1.23'
14+
RELEASE_BRANCH: ''
15+
LATEST_COMMIT: ''
716

817
jobs:
918
push:
1019
runs-on: ubuntu-latest
1120

1221
steps:
22+
- name: Env setup
23+
run: |
24+
git config --global user.email "[email protected]"
25+
git config --global user.name "github-actions[bot]"
26+
echo "RELEASE_BRANCH="release-"$(date +"%Y%m%d%H%M%S")" >> ${GITHUB_ENV}
27+
28+
- name: Install Go
29+
uses: actions/setup-go@v5
30+
with:
31+
go-version: ${{ env.GO_VERSION }}
32+
1333
- name: Checkout
1434
uses: actions/checkout@v4
35+
36+
- name: Get latest commit
37+
run: |
38+
echo "LATEST_COMMIT=$(git log -1 | head -n1)" >> ${GITHUB_ENV}
39+
40+
- name: Generate openapi yaml definition
41+
run: |
42+
cd openapi/compose && go run compose.go && cd .. && cd ..
43+
44+
- name: Checkout generator templates
45+
uses: actions/checkout@v4
46+
with:
47+
repository: OpenAPITools/openapi-generator
48+
path: official-generator-templates
49+
ref: ${{ env.GENERATOR_VERSION }}
50+
token: ${{ secrets.GH_PUSH_TOKEN }}
51+
sparse-checkout: |
52+
.github
53+
modules/openapi-generator/src/main/resources
54+
55+
- name: Move required templates
56+
run: |
57+
mkdir -p generated
58+
mv official-generator-templates/modules/openapi-generator/src/main/resources/go official-generator-templates/
59+
mv official-generator-templates/modules/openapi-generator/src/main/resources/python official-generator-templates/
60+
mv official-generator-templates/modules/openapi-generator/src/main/resources/Javascript official-generator-templates/
61+
mv official-generator-templates/modules/openapi-generator/src/main/resources/htmlDocs2 official-generator-templates/
62+
63+
- name: Apply patches to templates
64+
run: |
65+
cd official-generator-templates
66+
for f in ../template-patches/*.patch; do git apply ../template-patches/${f}; done;
67+
cd ..
68+
69+
- name: Checkout GO client arduino repo
70+
uses: actions/checkout@v4
71+
with:
72+
repository: arduino/iot-client-go
73+
token: ${{ secrets.GH_PUSH_TOKEN }}
74+
path: generated/go
75+
- run: |
76+
cd generated/go && git checkout -b ${{ env.RELEASE_BRANCH }}
77+
78+
- name: Checkout PYTHON client arduino repo
79+
uses: actions/checkout@v4
80+
with:
81+
repository: arduino/iot-client-py
82+
token: ${{ secrets.GH_PUSH_TOKEN }}
83+
path: generated/python
84+
- run: |
85+
cd generated/python && git checkout -b ${{ env.RELEASE_BRANCH }}
86+
87+
- name: Checkout JAVASCRIPT client arduino repo
88+
uses: actions/checkout@v4
89+
with:
90+
repository: arduino/iot-client-js
91+
token: ${{ secrets.GH_PUSH_TOKEN }}
92+
path: generated/javascript
93+
- run: |
94+
cd generated/javascript && git checkout -b ${{ env.RELEASE_BRANCH }}
95+
96+
- name: Checkout DOCS client arduino repo
97+
uses: actions/checkout@v4
98+
with:
99+
repository: arduino/iot-client-docs
100+
token: ${{ secrets.GH_PUSH_TOKEN }}
101+
path: generated/html2
102+
- run: |
103+
cd generated/html2 && git checkout -b ${{ env.RELEASE_BRANCH }}
104+
105+
- name: Generate go client
106+
uses: openapi-generators/openapitools-generator-action@v1
107+
with:
108+
generator: go
109+
generator-tag: ${{ env.GENERATOR_VERSION }}
110+
config-file: config/languages/go_v2.json
111+
openapi-file: ${{ env.INPUT_OPENAPI_FILE }}
112+
template-dir: official-generator-templates/go/
113+
command-args: -o generated/go
114+
115+
- name: Generate python client
116+
uses: openapi-generators/openapitools-generator-action@v1
117+
with:
118+
generator: python
119+
generator-tag: ${{ env.GENERATOR_VERSION }}
120+
config-file: config/languages/python_v2.json
121+
openapi-file: ${{ env.INPUT_OPENAPI_FILE }}
122+
template-dir: official-generator-templates/python/
123+
command-args: -o generated/python
124+
125+
- name: Generate Javascript client
126+
uses: openapi-generators/openapitools-generator-action@v1
127+
with:
128+
generator: javascript
129+
generator-tag: ${{ env.GENERATOR_VERSION }}
130+
config-file: config/languages/javascript_v2.json
131+
openapi-file: ${{ env.INPUT_OPENAPI_FILE }}
132+
template-dir: official-generator-templates/Javascript/
133+
command-args: -o generated/javascript
134+
135+
- name: Generate Docs
136+
uses: openapi-generators/openapitools-generator-action@v1
137+
with:
138+
generator: html2
139+
generator-tag: ${{ env.GENERATOR_VERSION }}
140+
config-file: config/languages/html2_v2.json
141+
openapi-file: ${{ env.INPUT_OPENAPI_FILE }}
142+
template-dir: official-generator-templates/htmlDocs2/
143+
command-args: -o generated/html2
144+
145+
- name: Apply downstream templates
146+
run: |
147+
cp downstream-templates/go/README.md generated/go/README.md
148+
cp downstream-templates/javascript/README.md generated/javascript/README.md
149+
cp downstream-templates/python/README.md generated/python/README.md
150+
cp downstream-templates/python/setup.py generated/python/setup.py
151+
cp downstream-templates/python/utils.py generated/python/iot_api_client/utils.py
152+
153+
- name: Copy shared assets
154+
run: |
155+
cp -R assets/shared/* generated/go/
156+
cp -R assets/shared/* generated/python
157+
cp -R assets/shared/* generated/javascript
158+
cp -R assets/shared/* generated/html2
159+
160+
- name: Open branches on external client REPOs
161+
run: |
162+
cd generated
163+
for f in $(ls -d */); do cd ${f} && git add . && git commit -m "Release ${{ env.RELEASE_BRANCH }}" && git push -u origin ${{ env.RELEASE_BRANCH }} && cd ..; done;
164+
165+
- name: Open PRs on client REPOs
166+
run: |
167+
cd generated
168+
for f in $(ls -d */); do cd ${f} && gh pr create -B master -H ${{ env.RELEASE_BRANCH }} --title 'Regenerate clients - ${{ env.LATEST_COMMIT }}' --body 'Created by clients-iot-api clients release automation job' && cd ..; done;
169+
env:
170+
GH_TOKEN: ${{ secrets.GH_PUSH_TOKEN }}

‎.gitignore

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
!generated
22
generated/*
33
!generated/.gitkeep
4-
spec/*/full_spec.yaml
54
!templates
65
templates/*
7-
!templates/.gitkeep
6+
!templates/.gitkeep
7+
openapi/openapi.yaml
8+
official-generator-templates

‎README.md

Lines changed: 7 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,7 @@ This repo contains the informations and the tools needed to automatically
44
generate API clients for the `iot-api` service in [any language][0] supported by
55
OpenAPI generator.
66

7-
The OpenAPI generator is orchestrated by a Python tool called [apigentools][1]
8-
that let us keep the spec files and the configuration bits in one single repo
9-
(this one) without duplicating the boilerplate on each git repo hosting the
10-
actual clients.
11-
12-
There is a [blog post](https://blog.arduino.cc/2020/03/05/how-to-deal-with-api-clients-the-lazy-way-from-code-generation-to-release-management/)
13-
describing the system architecture.
7+
The OpenAPI generator is orchestrated by git workflow using openapi-generator official images.
148

159
## [IMPORTANT] Client generation process
1610

@@ -39,14 +33,6 @@ depending on the programming language).
3933

4034
The operations are detailed in the following paragraphs.
4135

42-
### Requirements
43-
44-
To be able to run the workflow locally in a developmnent environment, you'll
45-
need the following:
46-
47-
* Python 3.7+
48-
* Apigentools 1.0+ (`pip install apigentools`)
49-
5036
### Get an updated version of the API specification
5137

5238
In this case the specs are generated by Goa using Swagger and they can be found
@@ -64,60 +50,14 @@ easily integrated in a CI, so major updates will be performed by either:
6450
Minor updates might be done manually since v3 uses Yaml and the resulting spec
6551
is human friendly.
6652

67-
### Validate the Spec
68-
69-
The generator can validate the content of the spec, this should likely go in a
70-
CI step:
71-
72-
```sh
73-
apigentools validate
74-
```
75-
7653
### Generate the clients
7754

78-
To generate clients and documentation using a template, and apply pathces before
79-
template filling, this is the command to run:
80-
81-
```sh
82-
apigentools generate
83-
```
84-
85-
The previous command is fine for local development but in case the generated code
86-
has to be pushed upstream to each client's repository, run the previous command as
87-
follows:
88-
89-
```sh
90-
apigentools generate --clone-repo
91-
```
92-
93-
### Push generated code to each client's git repository
94-
95-
`apigentools` has a simple command that can be invoked for each client generated
96-
that will push the resulting code to different repos, using different branches
97-
so the code can be reviewed and merged through a regular PR:
98-
99-
```sh
100-
apigentools push
101-
```
102-
103-
A release process should take from here and shouldn't be part of this workflow.
104-
105-
### Render upstream templates
106-
107-
This step is not currently necessary as the `generate` step does it implicitly,
108-
but in case you want to only apply patches to the original (upstream) templates
109-
**before** the generation step, you can run it to check how the pathces will
110-
modify the generated code. The step consists of cloning the openapi-generator
111-
repo, applying one or more patches in the form of patch files to it and copy
112-
the relevant templates in the folder `templates`:
113-
114-
```sh
115-
apigentools templates
116-
```
117-
118-
There are other ways to provide upstream templates other than the currently configured
119-
`openapi-git` that might be useful to speed up development iterations, please refer to
120-
`apigentools` docs for more details.
55+
Update openapi definition in openapi/components folder.
56+
'Release' github workflow will take care of:
57+
* update and merge openapi specifications
58+
* update templates with provided patches
59+
* generate clients
60+
* create PRs with updated code in destination clients repositories
12161

12262
## Customization
12363

‎assets/shared/img/selection_1.png

-82.3 KB
Loading

‎assets/shared/img/selection_2.png

15.9 KB
Loading

‎assets/shared/img/selection_3.png

47.8 KB
Loading

0 commit comments

Comments
 (0)