Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .mockery.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@ packages:
include-auto-generated: true
all: true
dir: 'gen/go/flyteidl2/workflow/mocks'
github.com/flyteorg/flyte/v2/gen/go/flyteidl2/workflow/workflowconnect:
config:
include-auto-generated: true
all: true
dir: 'gen/go/flyteidl2/workflow/workflowconnect/mocks'
github.com/flyteorg/flyte/v2/gen/go/flyteidl2/secret:
config:
include-auto-generated: true
Expand Down
154 changes: 154 additions & 0 deletions executor/DEVELOPMENT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
# Executor Development Guide

This guide provides steps on how to develop and iterates changes.

## Prerequisites
- go version v1.24.0+
- docker version 17.03+.
- kubectl version v1.11.3+.

### Setup on kind

We recommend using kind to create a Kubernetes cluster for local development.

### Use go v1.24

Currently, flyte-v2 use go v1.24 for development.

```sh
go install golang.org/dl/go1.24.0@latest
go1.24.0 download
export GOROOT=$(go1.24.0 env GOROOT)
export PATH="$GOROOT/bin:$PATH"
```

### Clean up local binaries

To keep consistent code generation and testing result, it is recommended to remove outdated binaries installed by the Makefile.

```sh
make clean
```

## Local development on kind cluster

Following steps requires you to switch your workign directory to the `executor/`.

```sh
cd executor
```

### Run executor inside the cluster

1. Create a kind cluster

```sh
kind create cluster --image=kindest/node:v1.26.0 --name flytev2
```

2. Build the image

```sh
IMG=flyteorg/executor:nightly make docker-build
```

3. Load image into kind cluster

```sh
kind load docker-image flyteorg/executor:nightly --name flytev2
```

4. Install CRD in to the cluster

```sh
make install
```

5. Deploy the Manager to the cluster with the image specified by `IMG`:

```sh
make deploy IMG=flyteorg/executor:nightly
```

6. Clean up

```sh
kind delete cluster --name flytev2
```

### Run executor outside the cluster

1. Create a kind cluster

```sh
kind create cluster --image=kindest/node:v1.26.0 --name flytev2
```

2. Install CRD in to the cluster

```sh
make install
```

3. Compile the source code and run

```sh
# Compile the source code
make build
# Run the executor
./bin/manager
```

## Tests

### Manual test

You can apply the samples from the config/sample using the following command:

```sh
kubectl apply -k config/samples/
```

You can see the `TaskAction` CRD created:

```sh
❯ k get taskactions
NAME RUN ACTION STATUS AGE
taskaction-sample sample-run sample-task Completed 59m
```

Or use `-o wide` to view more details:

```sh
❯ k get taskactions -o wide
NAME RUN ACTION STATUS AGE PROGRESSING SUCCEEDED FAILED
taskaction-sample sample-run sample-task Completed 59m False True
```

## Modify CRD

### Quick Steps

1. Modify the types file: Edit `api/v1/taskaction_types.go`
- Add/modify fields in `TaskActionSpec` or `TaskActionStatus`
- Add/modify printcolumns (the `// +kubebuilder:printcolumn` comments above the `TaskAction` struct)
- Add validation rules using `// +kubebuilder:validation:` markers

2. Generate CRD manifests and DeepCopy code

```sh
make manifests generate
```

This runs two commands:
- `make manifests`: Updates YAML manifests:
- `config/crd/bases/flyte.org_taskactions.yaml` (CRD with schema, validation, printcolumns)
- `config/rbac/role.yaml` (RBAC permissions)
- `make generate`: Updates Go code:
- `api/v1/zz_generated.deepcopy.go` (DeepCopy methods required by Kubernetes)

3. Update CRD in the cluster

```sh
make install
```
13 changes: 7 additions & 6 deletions executor/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,22 @@ ARG TARGETARCH

WORKDIR /workspace
# Copy the Go Modules manifests
COPY executor/go.mod go.mod
COPY executor/go.sum go.sum
COPY go.mod go.mod
COPY go.sum go.sum
# cache deps before building and copying source so that we don't need to re-download as much
# and so that source changes don't invalidate our downloaded layer
RUN go mod download

# Copy the Go source (relies on .dockerignore to filter)
COPY executor/. .
# Copy only the needed source files from monorepo
COPY executor/ executor/
COPY gen/ gen/

# Build
# Build the executor binary
# the GOARCH has no default value to allow the binary to be built according to the host where the command
# was called. For example, if we call make docker-build in a local env which has the Apple Silicon M1 SO
# the docker BUILDPLATFORM arg will be linux/arm64 when for Apple x86 it will be linux/amd64. Therefore,
# by leaving it empty we can ensure that the container and binary shipped on it will have the same platform.
RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager cmd/main.go
RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager executor/cmd/main.go

# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
Expand Down
6 changes: 5 additions & 1 deletion executor/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ run: manifests generate fmt vet ## Run a controller from your host.
# More info: https://docs.docker.com/develop/develop-images/build_enhancements/
.PHONY: docker-build
docker-build: ## Build docker image with the manager.
$(CONTAINER_TOOL) build -t ${IMG} .
$(CONTAINER_TOOL) build -t ${IMG} -f Dockerfile ..
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Set working dir to root so that we can copy all required things in Dockerfile


.PHONY: docker-push
docker-push: ## Push docker image with the manager.
Expand Down Expand Up @@ -178,6 +178,10 @@ LOCALBIN ?= $(shell pwd)/bin
$(LOCALBIN):
mkdir -p $(LOCALBIN)

.PHONY: clean
clean: ## Cleanup local binaries such as controller-gen and kustomize
rm -rf $(LOCALBIN)

## Tool Binaries
KUBECTL ?= kubectl
KIND ?= kind
Expand Down
23 changes: 19 additions & 4 deletions executor/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,20 @@
# executor
// TODO(user): Add simple overview of use/purpose
# Executor

The Executor is a Kubernetes operator that manages the execution of Flyte tasks in the data plane. Built using the Kubernetes Operator pattern with `controller-runtime`, it watches and reconciles `TaskAction` custom resources to orchestrate task execution.

## Description
// TODO(user): An in-depth paragraph about your project and overview of use

The Executor controller is responsible for:
- **Monitoring TaskAction CRs**: Watches for TaskAction resources created in the Kubernetes cluster
- **Task Execution**: Executes tasks based on the specifications defined in TaskAction resources
- **State Management**: Reports task execution state to the State Service
- **Lifecycle Management**: Manages the full lifecycle of task execution from queued to completed/failed states

The executor uses conditions to track task progress:
- `Progressing`: Indicates active execution (reasons: Queued, Initializing, Executing)
- `Succeeded`: Task completed successfully
- `Failed`: Task execution failed


## Getting Started

Expand Down Expand Up @@ -110,8 +122,11 @@ the '--force' flag and manually ensure that any custom configuration
previously added to 'dist/chart/values.yaml' or 'dist/chart/manager/manager.yaml'
is manually re-applied afterwards.


## Contributing
// TODO(user): Add detailed information on how you would like others to contribute to this project

Please read our [CONTRIBUTING](../CONTRIBUTING.md) guide before making a pull request. Refer to our
[DEVELOPMENT.md](DEVELOPMENT.md) to build and run tests for executor locally.

**NOTE:** Run `make help` for more information on all potential `make` targets

Expand Down
Loading