This repository contains Argo Workflows used by the Open Forest Observatory (OFO) to process drone imagery at scale on Jetstream2 Cloud.
For complete documentation, visit https://open-forest-observatory.github.io/ofo-argo/
Quick links:
The workflow runs the automate-metashape pipeline, followed by post-processing steps, simultaneously across multiple virtual machines. This type of scaling enables OFO to process many photogrammetry projects simultaneously with a single run command.
The system uses Kubernetes which orchestrates containers, scales the processing to multiple VMs, and balances the load between worker nodes. The current setup includes:
- Controller node: Manages the Kubernetes cluster and Argo workflows
- Worker nodes: Handle compute workloads, processing drone missions in parallel
- Manila shared storage: Provides working data storage to the nodes
- S3 storage: Stores the inputs/outputs of each step
The step-based workflow (photogrammetry-workflow-stepbased.yaml) provides optimized resource utilization by splitting Metashape processing into individual steps:
- 🎯 GPU steps (match_photos, build_depth_maps, build_mesh) run on expensive GPU nodes only when needed
- 💻 CPU steps (align_cameras, build_point_cloud, build_dem_orthomosaic, etc.) run on cheaper CPU nodes
- ⚡ Disabled steps are completely skipped (no pod creation, no resource allocation)
- 📊 Fine-grained monitoring - Track progress of each step in the Argo UI
- 🔧 Flexible GPU usage - Configure whether GPU-capable steps use GPU or CPU nodes
Documentation:
- 📖 Complete Guide - Full usage instructions, troubleshooting, and advanced topics
- ⚡ Quick Reference - Command cheat sheet and common patterns
| File/Directory | Purpose |
|---|---|
| photogrammetry-workflow.yaml | Original Argo Workflows configuration (monolithic, kept as reference) |
| photogrammetry-workflow-stepbased.yaml | Step-based workflow - Individual step execution with optimized CPU/GPU allocation |
| docs/ | Documentation site source (published to GitHub Pages) |
| mkdocs.yml | Configuration for the MkDocs documentation site |
| docker-workflow-utils/ | Utility docker container for workflow scripts (preprocessing, DB logging) |
| docker-photogrammetry-postprocessing/ | Files defining custom docker containers performing core workflow tasks (e.g. postprocessing) |
| setup/ | Kubernetes and Argo setup configurations (described in admin docs) |
| test-workflows/ | Test workflow definitions for development and validation |
| implementation-plans/ | Technical implementation plans for workflow development |
For complete setup and usage instructions, please visit the full documentation site.
Quick start guide:
For cluster administration, see the admin guides.