Skip to content

Commit 4259f07

Browse files
ayildirim21ayildirim21
authored
chore: refactor example builds so that they no longer rely on tarball (#173)
Signed-off-by: a3hadi <[email protected]>
1 parent 02009c8 commit 4259f07

File tree

96 files changed

+382
-313
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

96 files changed

+382
-313
lines changed

‎.github/workflows/build-push.yaml

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,22 +8,21 @@ on:
88

99
jobs:
1010
docker_publish:
11-
# run it only on numaproj/numaflow-python repository
12-
# forked repositories normally don't have the proper permission setup.
11+
# run workflow only on numaproj/numaflow-python repository
1312
if: ${{ github.repository }} == "numaproj/numaflow-python"
1413
name: Build, Tag, and Push Image
1514
runs-on: ubuntu-latest
1615

1716
strategy:
1817
matrix:
19-
# TODO: rewrite asyncio-reduce example using latest SDK version, as it is currently using old methods
18+
# TODO: rewrite asyncio_reduce example using latest SDK version, as it is currently using old methods
2019
# then add to example_directories matrix
2120
example_directories: [
2221
"examples/map/even_odd", "examples/map/flatmap", "examples/map/forward_message",
2322
"examples/map/multiproc_map", "examples/mapstream/flatmap_stream", "examples/reduce/counter",
24-
"examples/reducestream/counter", "examples/reducestream/sum", "examples/sideinput/simple-sideinput",
25-
"examples/sideinput/simple-sideinput/udf", "examples/sink/async_log", "examples/sink/log",
26-
"examples/source/async-source", "examples/source/simple-source", "examples/sourcetransform/event_time_filter"
23+
"examples/reducestream/counter", "examples/reducestream/sum", "examples/sideinput/simple_sideinput",
24+
"examples/sideinput/simple_sideinput/udf", "examples/sink/async_log", "examples/sink/log",
25+
"examples/source/async_source", "examples/source/simple_source", "examples/sourcetransform/event_time_filter"
2726
]
2827

2928
steps:
@@ -48,5 +47,4 @@ jobs:
4847
password: ${{ secrets.NUMAIO_PASSWORD }}
4948
- name: Build, tag, and push images
5049
run: |
51-
./.hack/update_examples.sh --update ${{ matrix.example_directories }}
52-
./.hack/update_examples.sh --build-push-example ${{ matrix.example_directories }}
50+
./hack/update_examples.sh --build-push-example ${{ matrix.example_directories }}

‎.github/workflows/changelog.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ jobs:
1616
fetch-depth: 0
1717
- run: git fetch --prune --prune-tags
1818
- run: git tag -l 'v*'
19-
- run: ./.hack/changelog.sh > CHANGELOG.md
19+
- run: ./hack/changelog.sh > CHANGELOG.md
2020
- uses: peter-evans/create-pull-request@v3
2121
with:
2222
title: 'docs: updated CHANGELOG.md'

‎Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
clean:
2-
@rm -rf build dist .eggs *.egg-info
2+
@rm -rf build .eggs *.egg-info
33
@rm -rf .benchmarks .coverage coverage.xml htmlcov report.xml .tox
44
@find . -type d -name '.mypy_cache' -exec rm -rf {} +
55
@find . -type d -name '__pycache__' -exec rm -rf {} +

‎development.md

Lines changed: 6 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -6,28 +6,18 @@ This document explains the development process for the Numaflow Python SDK.
66

77
1. Install [Poetry](https://python-poetry.org/docs/) before starting your test. Make sure you have the correct Python version
88
2. Make your SDK changes
9-
3. Run `./.hack/update_examples.sh -u <example-directory-path>`. This command will create a tarball of the SDK, that includes your changes, in the directory of the example specified
10-
4. Run `poetry update -vv` inside the example directory you used in the step above. This will create a `poetry.lock` file
11-
5. Update the `example.py` inside the same example directory if needed
12-
6. Run `make image` inside the same example directory to build your image
13-
7. Now that you have the image with your customized example and code change, you can test it in a Numaflow pipeline. Example pipelines, `pipeline.yaml`, are also provided in most of the example directories.
9+
3. Update the `example.py` inside the same example directory if needed
10+
4. Run `make image` inside the same example directory to build your image
11+
5. Now that you have the image with your customized example and code change, you can test it in a Numaflow pipeline. Example pipelines, `pipeline.yaml`, are also provided in most of the example directories.
1412
Please check [Numaflow](https://numaflow.numaproj.io/) for more details
1513

1614
Each example directory has a Makefile which can be used to build, tag, and push images.
1715
If you want to build and tag the image and then immediately push it to quay.io, use the `image-push` target.
1816
If you want to build and tag a local image without pushing, use the `image` target.
1917

20-
Everytime you make a new SDK change, make sure to create the `dist/` folder in the example directory you would like
21-
to test, by running:
18+
If you would like to build and push a specific example, you can do so by running:
2219
```shell
23-
./.hack/update_examples.sh -u <example-directory-path>
24-
```
25-
When you are no longer using the `dist/` folders you can run `make clean` in the root directory, as well as in any example directory
26-
in order to remove them.
27-
28-
If you would like to build and push a specific example, you can do so by running (make sure that the `dist/` folder is present in the directory):
29-
```shell
30-
./.hack/update_examples.sh -bpe <example-directory-path> -t <tag>
20+
./hack/update_examples.sh -bpe <example-directory-path> -t <tag>
3121
```
3222
This is essentially equivalent to running `make image-push TAG=<tag>` in the example directory itself.
3323
The default tag is `stable`, but it is recommended you specify your own for testing purposes, as the Github Actions CI uses the `stable` tag.
@@ -38,7 +28,7 @@ Note: before running the script, ensure that through the CLI, you are logged int
3828
### Deploying
3929

4030
After confirming that your changes pass local testing:
41-
1. Clean up testing artifacts (remove any `dist/` folders created, remove any test images on quay.io, etc.)
31+
1. Clean up testing artifacts (remove any test images on quay.io, etc.)
4232
2. Create a PR for your changes. Once your PR has been merged, a Github Actions workflow (`Docker Publish`) will be triggered, to build, tag (with `stable`), and push
4333
all example images. This ensures that all example images are using the most up-to-date version of the SDK, i.e. the one including your
4434
changes

‎examples/map/even_odd/Dockerfile

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,12 +39,13 @@ RUN apt-get update \
3939
FROM builder AS udf
4040

4141
WORKDIR $PYSETUP_PATH
42-
COPY dist/ ./dist/
43-
COPY pyproject.toml .
42+
COPY ./ ./
43+
44+
WORKDIR $PYSETUP_PATH/examples/map/even_odd
4445
RUN poetry install --no-cache --no-root && \
4546
rm -rf ~/.cache/pypoetry/
4647

47-
ADD . /app
48+
COPY examples/map/even_odd /app
4849
WORKDIR /app
4950

5051
RUN chmod +x entry.sh

‎examples/map/even_odd/Makefile

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,22 @@
11
TAG ?= stable
22
PUSH ?= false
3+
IMAGE_REGISTRY = quay.io/numaio/numaflow-python/even-odd:${TAG}
4+
DOCKER_FILE_PATH = examples/map/even_odd/Dockerfile
5+
6+
.PHONY: update
7+
update:
8+
poetry update -vv
39

410
.PHONY: image-push
5-
image-push:
6-
docker buildx build -t "quay.io/numaio/numaflow-python/even-odd:${TAG}" --platform linux/amd64,linux/arm64 . --push
11+
image-push: update
12+
cd ../../../ && docker buildx build \
13+
-f ${DOCKER_FILE_PATH} \
14+
-t {IMAGE_REGISTRY} \
15+
--platform linux/amd64,linux/arm64 . --push
716

817
.PHONY: image
9-
image:
10-
docker build -t "quay.io/numaio/numaflow-python/even-odd:${TAG}" .
11-
@if [ "$(PUSH)" = "true" ]; then docker push "quay.io/numaio/numaflow-python/even-odd:${TAG}"; fi
12-
13-
.PHONY: clean
14-
clean:
15-
rm -rf ./dist
18+
image: update
19+
cd ../../../ && docker build \
20+
-f ${DOCKER_FILE_PATH} \
21+
-t ${IMAGE_REGISTRY} .
22+
@if [ "$(PUSH)" = "true" ]; then docker push ${IMAGE_REGISTRY}; fi

‎examples/map/even_odd/pipeline.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ spec:
1717
args:
1818
- python
1919
- example.py
20-
image: "quay.io/numaio/numaflow-python/even-odd:stable"
20+
image: quay.io/numaio/numaflow-python/even-odd:stable
2121
- name: even-sink
2222
scale:
2323
min: 1

‎examples/map/even_odd/pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ authors = ["Numaflow developers"]
66

77
[tool.poetry.dependencies]
88
python = "~3.10"
9-
pynumaflow = { path = "./dist/pynumaflow.tar.gz"}
9+
pynumaflow = { path = "../../../"}
1010

1111
[tool.poetry.dev-dependencies]
1212

‎examples/map/flatmap/Dockerfile

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,12 +39,13 @@ RUN apt-get update \
3939
FROM builder AS udf
4040

4141
WORKDIR $PYSETUP_PATH
42-
COPY dist/ ./dist/
43-
COPY pyproject.toml .
42+
COPY ./ ./
43+
44+
WORKDIR $PYSETUP_PATH/examples/map/flatmap
4445
RUN poetry install --no-cache --no-root && \
4546
rm -rf ~/.cache/pypoetry/
4647

47-
ADD . /app
48+
COPY examples/map/even_odd /app
4849
WORKDIR /app
4950

5051
RUN chmod +x entry.sh

‎examples/map/flatmap/Makefile

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,22 @@
11
TAG ?= stable
22
PUSH ?= false
3+
IMAGE_REGISTRY = quay.io/numaio/numaflow-python/map-flatmap:${TAG}
4+
DOCKER_FILE_PATH = examples/map/flatmap/Dockerfile
5+
6+
.PHONY: update
7+
update:
8+
poetry update -vv
39

410
.PHONY: image-push
5-
image-push:
6-
docker buildx build -t "quay.io/numaio/numaflow-python/map-flatmap:${TAG}" --platform linux/amd64,linux/arm64 . --push
11+
image-push: update
12+
cd ../../../ && docker buildx build \
13+
-f ${DOCKER_FILE_PATH} \
14+
-t ${IMAGE_REGISTRY} \
15+
--platform linux/amd64,linux/arm64 . --push
716

817
.PHONY: image
9-
image:
10-
docker build -t "quay.io/numaio/numaflow-python/map-flatmap:${TAG}" .
11-
@if [ "$(PUSH)" = "true" ]; then docker push "quay.io/numaio/numaflow-python/map-flatmap:${TAG}"; fi
12-
13-
.PHONY: clean
14-
clean:
15-
rm -rf ./dist
18+
image: update
19+
cd ../../../ && docker build \
20+
-f ${DOCKER_FILE_PATH} \
21+
-t ${IMAGE_REGISTRY} .
22+
@if [ "$(PUSH)" = "true" ]; then docker push ${IMAGE_REGISTRY}; fi

‎examples/map/flatmap/pipeline.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ spec:
1515
- name: flatmap
1616
udf:
1717
container:
18-
image: "quay.io/numaio/numaflow-python/map-flatmap:stable"
18+
image: quay.io/numaio/numaflow-python/map-flatmap:stable
1919
env:
2020
- name: PYTHONDEBUG
2121
value: "true"

‎examples/map/flatmap/pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ authors = ["Numaflow developers"]
66

77
[tool.poetry.dependencies]
88
python = "~3.10"
9-
pynumaflow = { path = "./dist/pynumaflow.tar.gz"}
9+
pynumaflow = { path = "../../../"}
1010

1111
[tool.poetry.dev-dependencies]
1212

‎examples/map/forward_message/Dockerfile

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,12 +39,13 @@ RUN apt-get update \
3939
FROM builder AS udf
4040

4141
WORKDIR $PYSETUP_PATH
42-
COPY dist/ ./dist/
43-
COPY pyproject.toml .
42+
COPY ./ ./
43+
44+
WORKDIR $PYSETUP_PATH/examples/map/forward_message
4445
RUN poetry install --no-cache --no-root && \
4546
rm -rf ~/.cache/pypoetry/
4647

47-
ADD . /app
48+
COPY examples/map/forward_message /app
4849
WORKDIR /app
4950

5051
RUN chmod +x entry.sh

‎examples/map/forward_message/Makefile

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,22 @@
11
TAG ?= stable
22
PUSH ?= false
3+
IMAGE_REGISTRY = quay.io/numaio/numaflow-python/map-forward-message:${TAG}
4+
DOCKER_FILE_PATH = examples/map/forward_message/Dockerfile
5+
6+
.PHONY: update
7+
update:
8+
poetry update -vv
39

410
.PHONY: image-push
5-
image-push:
6-
docker buildx build -t "quay.io/numaio/numaflow-python/map-forward-message:${TAG}" --platform linux/amd64,linux/arm64 . --push
11+
image-push: update
12+
cd ../../../ && docker buildx build \
13+
-f ${DOCKER_FILE_PATH} \
14+
-t ${IMAGE_REGISTRY} \
15+
--platform linux/amd64,linux/arm64 . --push
716

817
.PHONY: image
9-
image:
10-
docker build -t "quay.io/numaio/numaflow-python/map-forward-message:${TAG}" .
11-
@if [ "$(PUSH)" = "true" ]; then docker push "quay.io/numaio/numaflow-python/map-forward-message:${TAG}"; fi
12-
13-
.PHONY: clean
14-
clean:
15-
rm -rf ./dist
18+
image: update
19+
cd ../../../ && docker build \
20+
-f ${DOCKER_FILE_PATH} \
21+
-t ${IMAGE_REGISTRY} .
22+
@if [ "$(PUSH)" = "true" ]; then docker push ${IMAGE_REGISTRY}; fi

‎examples/map/forward_message/pipeline.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ spec:
1717
args:
1818
- python
1919
- example.py
20-
image: "quay.io/numaio/numaflow-python/map-forward-message:stable"
20+
image: quay.io/numaio/numaflow-python/map-forward-message:stable
2121
- name: log-output
2222
sink:
2323
log: {}

‎examples/map/forward_message/pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ authors = ["Numaflow developers"]
66

77
[tool.poetry.dependencies]
88
python = "~3.10"
9-
pynumaflow = { path = "./dist/pynumaflow.tar.gz"}
9+
pynumaflow = { path = "../../../"}
1010

1111
[tool.poetry.dev-dependencies]
1212

‎examples/map/multiproc_map/Dockerfile

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,12 +39,13 @@ RUN apt-get update \
3939
FROM builder AS udf
4040

4141
WORKDIR $PYSETUP_PATH
42-
COPY dist/ ./dist/
43-
COPY pyproject.toml .
42+
COPY ./ ./
43+
44+
WORKDIR $PYSETUP_PATH/examples/map/multiproc_map
4445
RUN poetry install --no-cache --no-root && \
4546
rm -rf ~/.cache/pypoetry/
4647

47-
ADD . /app
48+
COPY examples/map/multiproc_map /app
4849
WORKDIR /app
4950

5051
RUN chmod +x entry.sh

‎examples/map/multiproc_map/Makefile

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,22 @@
11
TAG ?= stable
22
PUSH ?= false
3+
IMAGE_REGISTRY = quay.io/numaio/numaflow-python/multiproc:${TAG}
4+
DOCKER_FILE_PATH = examples/map/multiproc_map/Dockerfile
5+
6+
.PHONY: update
7+
update:
8+
poetry update -vv
39

410
.PHONY: image-push
5-
image-push:
6-
docker buildx build -t "quay.io/numaio/numaflow-python/multiproc:${TAG}" --platform linux/amd64,linux/arm64 . --push
11+
image-push: update
12+
cd ../../../ && docker buildx build \
13+
-f ${DOCKER_FILE_PATH} \
14+
-t ${IMAGE_REGISTRY} \
15+
--platform linux/amd64,linux/arm64 . --push
716

817
.PHONY: image
9-
image:
10-
docker build -t "quay.io/numaio/numaflow-python/multiproc:${TAG}" .
11-
@if [ "$(PUSH)" = "true" ]; then docker push "quay.io/numaio/numaflow-python/multiproc:${TAG}"; fi
12-
13-
.PHONY: clean
14-
clean:
15-
rm -rf ./dist
18+
image: update
19+
cd ../../../ && docker build \
20+
-f ${DOCKER_FILE_PATH} \
21+
-t ${IMAGE_REGISTRY} .
22+
@if [ "$(PUSH)" = "true" ]; then docker push ${IMAGE_REGISTRY}; fi

‎examples/map/multiproc_map/pipeline.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ spec:
1515
- name: mult
1616
udf:
1717
container:
18-
image: "quay.io/numaio/numaflow-python/multiproc:stable"
18+
image: quay.io/numaio/numaflow-python/multiproc:stable
1919
env:
2020
- name: PYTHONDEBUG
2121
value: "true"

‎examples/map/multiproc_map/pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ authors = ["Numaflow developers"]
66

77
[tool.poetry.dependencies]
88
python = "~3.10"
9-
pynumaflow = { path = "./dist/pynumaflow.tar.gz"}
9+
pynumaflow = { path = "../../../"}
1010

1111
[tool.poetry.dev-dependencies]
1212

‎examples/mapstream/flatmap_stream/Dockerfile

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,12 +39,13 @@ RUN apt-get update \
3939
FROM builder AS udf
4040

4141
WORKDIR $PYSETUP_PATH
42-
COPY dist/ ./dist/
43-
COPY pyproject.toml .
42+
COPY ./ ./
43+
44+
WORKDIR $PYSETUP_PATH/examples/mapstream/flatmap_stream
4445
RUN poetry install --no-cache --no-root && \
4546
rm -rf ~/.cache/pypoetry/
4647

47-
ADD . /app
48+
COPY examples/mapstream/flatmap_stream /app
4849
WORKDIR /app
4950

5051
RUN chmod +x entry.sh

0 commit comments

Comments
 (0)