From 1ae7647d9c056a60c64645020378f57cd8b820ce Mon Sep 17 00:00:00 2001 From: Sarthak Agrawal Date: Thu, 26 Jun 2025 14:24:18 -0700 Subject: [PATCH 1/3] add simplified architecture --- docs/architecture/README.md | 83 ++++++++++ docs/architecture/configuration-flow.md | 163 ++++++++++++++++++ docs/architecture/gateway-lifecycle.md | 211 ++++++++++++++++++++++++ docs/architecture/traffic-flow.md | 139 ++++++++++++++++ 4 files changed, 596 insertions(+) create mode 100644 docs/architecture/README.md create mode 100644 docs/architecture/configuration-flow.md create mode 100644 docs/architecture/gateway-lifecycle.md create mode 100644 docs/architecture/traffic-flow.md diff --git a/docs/architecture/README.md b/docs/architecture/README.md new file mode 100644 index 0000000000..1e1b94daa2 --- /dev/null +++ b/docs/architecture/README.md @@ -0,0 +1,83 @@ +# NGINX Gateway Fabric Architecture + +This guide explains how NGINX Gateway Fabric works in simple terms. + +## What is NGINX Gateway Fabric? + +NGINX Gateway Fabric (NGF) turns Kubernetes Gateway API resources into working traffic routing. It has two main parts: + +- **Control Plane**: Watches Kubernetes and creates NGINX configs +- **Data Plane**: NGINX servers that handle your traffic + +## Control Plane vs Data Plane + +### Control Plane + +The **Control Plane** is the brain: + +- Watches for Gateway and Route changes in Kubernetes +- Converts Gateway API configs into NGINX configs +- Manages NGINX instances +- Handles certificates and security + +### Data Plane + +The **Data Plane** does the work: + +- Receives incoming traffic from users +- Routes traffic to your apps +- Handles SSL/TLS termination +- Applies load balancing and security rules + +## Architecture Diagrams + +### [Configuration Flow](./configuration-flow.md) + +How Gateway API resources become NGINX configurations + +### [Traffic Flow](./traffic-flow.md) + +How user requests travel through the system + +### [Gateway Lifecycle](./gateway-lifecycle.md) + +What happens when you create or update a Gateway + +## Key Concepts + +### Separation of Concerns + +- Control and data planes run in **separate pods** +- They communicate over **secure gRPC** +- Each can **scale independently** +- **Different security permissions** for each + +### Gateway API Integration + +NGF implements these Kubernetes Gateway API resources: + +- **Gateway**: Defines entry points for traffic +- **HTTPRoute**: Defines HTTP routing rules +- **GRPCRoute**: Defines gRPC routing rules +- **TLSRoute**: Defines TLS routing rules + +### NGINX Agent + +- **NGINX Agent v3** connects control and data planes +- Runs inside each NGINX pod +- Downloads configs from control plane +- Manages NGINX lifecycle (start, reload, monitor) + +## Security Model + +### Control Plane Security + +- **Full Kubernetes API access** (to watch resources) +- **gRPC server** for data plane connections +- **Certificate management** for secure communication + +### Data Plane Security + +- **No Kubernetes API access** (security isolation) +- **gRPC client** connects to control plane +- **Minimal permissions** (principle of least privilege) diff --git a/docs/architecture/configuration-flow.md b/docs/architecture/configuration-flow.md new file mode 100644 index 0000000000..92a9ad5640 --- /dev/null +++ b/docs/architecture/configuration-flow.md @@ -0,0 +1,163 @@ +# Configuration Flow + +This diagram shows how Gateway API resources become NGINX configurations. + +## Simple Overview + +```mermaid +%%{init: {'theme':'dark', 'themeVariables': {'fontSize': '16px', 'darkMode': true, 'primaryColor': '#4f46e5', 'primaryTextColor': '#e5e7eb', 'primaryBorderColor': '#6b7280', 'lineColor': '#9ca3af', 'secondaryColor': '#1f2937', 'tertiaryColor': '#374151', 'background': '#111827', 'mainBkg': '#1f2937', 'secondBkg': '#374151', 'tertiaryTextColor': '#d1d5db'}}}%% +graph TB + %% User Actions + USER[👤 User] --> K8S + + %% Kubernetes API Layer + subgraph "Kubernetes API" + K8S[🔵 API Server] + GW[Gateway] + ROUTE[HTTPRoute] + SVC[Service] + end + + %% Control Plane + subgraph "Control Plane Pod" + NGF[🎯 NGF Controller] + end + + %% Data Plane + subgraph "Data Plane Pod" + AGENT[🔧 NGINX Agent] + NGINX[🌐 NGINX] + CONF[nginx.conf] + end + + %% Flow + K8S --> NGF + GW --> NGF + ROUTE --> NGF + SVC --> NGF + + NGF --> AGENT + AGENT --> CONF + CONF --> NGINX + + %% Dark-friendly styling + style USER fill:#fbbf24,stroke:#f59e0b,stroke-width:2px,color:#1f2937 + style NGF fill:#3b82f6,stroke:#2563eb,stroke-width:2px,color:#ffffff + style NGINX fill:#8b5cf6,stroke:#7c3aed,stroke-width:2px,color:#ffffff + style K8S fill:#6b7280,stroke:#4b5563,stroke-width:2px,color:#ffffff +``` + +## Step-by-Step Process + +### 1. User Creates Resources + +```yaml +# User applies Gateway API resources +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: my-gateway +``` + +### 2. Kubernetes Stores Resources + +- Gateway, HTTPRoute, Service resources stored in etcd +- Kubernetes API Server notifies watchers of changes + +### 3. NGF Controller Processes Changes + +```text +NGF Controller: +├── Watches Gateway API resources +├── Validates configurations +├── Builds internal config graph +└── Generates NGINX configuration +``` + +### 4. Configuration Sent to Data Plane + +```text +Control Plane → Data Plane: +├── gRPC connection (secure) +├── nginx.conf file contents +├── SSL certificates +└── Other config files +``` + +### 5. NGINX Agent Updates Configuration + +```text +NGINX Agent: +├── Receives config from control plane +├── Validates NGINX syntax +├── Writes files to disk +├── Tests configuration +└── Reloads NGINX (if valid) +``` + +## Detailed Configuration Flow + +```mermaid +%%{init: {'theme':'dark', 'themeVariables': {'fontSize': '14px', 'darkMode': true, 'primaryColor': '#4f46e5', 'primaryTextColor': '#e5e7eb', 'primaryBorderColor': '#6b7280', 'lineColor': '#9ca3af', 'secondaryColor': '#1f2937', 'tertiaryColor': '#374151', 'background': '#111827', 'actorBkg': '#374151', 'actorBorder': '#6b7280', 'actorTextColor': '#e5e7eb', 'activationBkgColor': '#4f46e5', 'activationBorderColor': '#3730a3', 'signalColor': '#9ca3af', 'signalTextColor': '#e5e7eb', 'labelBoxBkgColor': '#1f2937', 'labelBoxBorderColor': '#6b7280', 'labelTextColor': '#e5e7eb', 'loopTextColor': '#e5e7eb', 'noteBkgColor': '#374151', 'noteBorderColor': '#6b7280', 'noteTextColor': '#e5e7eb'}}}%% +sequenceDiagram + participant User + participant API as K8s API + participant NGF as NGF Controller + participant Agent as NGINX Agent + participant NGINX + + User->>API: Apply Gateway/Route + API->>NGF: Watch notification + NGF->>NGF: Validate resources + NGF->>NGF: Build config graph + NGF->>NGF: Generate nginx.conf + NGF->>Agent: Send config (gRPC) + Agent->>Agent: Write config files + Agent->>NGINX: Test config + Agent->>NGINX: Reload (if valid) + Agent->>NGF: Report status + NGF->>API: Update resource status +``` + +## What Gets Generated? + +### NGINX Configuration Files + +- **nginx.conf**: Main configuration +- **servers.conf**: Virtual server definitions +- **upstreams.conf**: Backend service definitions +- **maps.conf**: Request routing maps + +### Example Generated Config + +```nginx +# Generated from Gateway API resources +server { + listen 80; + server_name api.example.com; + + location /users { + proxy_pass http://user-service; + } + + location /orders { + proxy_pass http://order-service; + } +} +``` + +## Error Handling + +### Invalid Configuration + +1. **NGF validates** Gateway API resources +2. **NGINX Agent tests** generated config +3. **Rollback** if configuration is invalid +4. **Status updates** report errors to Kubernetes + +### Recovery Process + +- Keep last known good configuration +- Report errors in resource status +- Retry configuration updates +- Graceful degradation when possible diff --git a/docs/architecture/gateway-lifecycle.md b/docs/architecture/gateway-lifecycle.md new file mode 100644 index 0000000000..71dcfba797 --- /dev/null +++ b/docs/architecture/gateway-lifecycle.md @@ -0,0 +1,211 @@ +# Gateway Lifecycle + +This shows what happens when you create, update, or delete a Gateway. + +## Gateway Creation Flow + +```mermaid +%%{init: {'theme':'dark', 'themeVariables': {'fontSize': '14px', 'darkMode': true, 'primaryColor': '#4f46e5', 'primaryTextColor': '#e5e7eb', 'primaryBorderColor': '#6b7280', 'lineColor': '#9ca3af', 'secondaryColor': '#1f2937', 'tertiaryColor': '#374151', 'background': '#111827', 'actorBkg': '#374151', 'actorBorder': '#6b7280', 'actorTextColor': '#e5e7eb', 'activationBkgColor': '#4f46e5', 'activationBorderColor': '#3730a3', 'signalColor': '#9ca3af', 'signalTextColor': '#e5e7eb', 'labelBoxBkgColor': '#1f2937', 'labelBoxBorderColor': '#6b7280', 'labelTextColor': '#e5e7eb', 'loopTextColor': '#e5e7eb', 'noteBkgColor': '#374151', 'noteBorderColor': '#6b7280', 'noteTextColor': '#e5e7eb'}}}%% +sequenceDiagram + participant User + participant API as K8s API Server + participant NGF as NGF Controller + participant K8s as Kubernetes + participant Agent as NGINX Agent + participant NGINX + + User->>API: Create Gateway resource + API->>NGF: Watch event: Gateway created + NGF->>NGF: Validate Gateway config + NGF->>K8s: Create NGINX Deployment + NGF->>K8s: Create LoadBalancer Service + K8s->>Agent: Start NGINX pod + Agent->>NGF: gRPC connect & register + NGF->>Agent: Send initial config + Agent->>NGINX: Write nginx.conf + Agent->>NGINX: Start NGINX process + Agent->>NGF: Report ready status + NGF->>API: Update Gateway status: Ready + API->>User: Gateway is ready +``` + +## What Happens During Creation + +### 1. User Creates Gateway + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: my-gateway + namespace: default +spec: + gatewayClassName: nginx + listeners: + - name: http + port: 80 + protocol: HTTP +``` + +### 2. NGF Controller Validates + +```text +Validation Checks: +├── GatewayClass exists and is valid +├── Listener configuration is correct +├── Required permissions are available +├── Resource limits are acceptable +└── No conflicting Gateways exist +``` + +### 3. Kubernetes Resources Created + +```text +Resources Created: +├── Deployment: nginx-gateway-{gateway-name} +├── Service: nginx-gateway-{gateway-name} +├── ConfigMap: nginx-config-{gateway-name} +├── ServiceAccount: nginx-gateway-{gateway-name} +└── Secrets: TLS certificates (if needed) +``` + +### 4. NGINX Pod Starts + +```text +Pod Startup: +├── Pull NGINX + Agent image +├── Mount configuration files +├── Start NGINX Agent process +├── Agent connects to control plane +└── Download initial configuration +``` + +### 5. Gateway Becomes Ready + +```text +Ready Conditions: +├── Deployment is available +├── Service has endpoints +├── NGINX is serving traffic +├── Health checks pass +└── Status updated in Kubernetes +``` + +## HTTPRoute Attachment Flow + +```mermaid +%%{init: {'theme':'dark', 'themeVariables': {'fontSize': '14px', 'darkMode': true, 'primaryColor': '#4f46e5', 'primaryTextColor': '#e5e7eb', 'primaryBorderColor': '#6b7280', 'lineColor': '#9ca3af', 'secondaryColor': '#1f2937', 'tertiaryColor': '#374151', 'background': '#111827', 'actorBkg': '#374151', 'actorBorder': '#6b7280', 'actorTextColor': '#e5e7eb', 'activationBkgColor': '#4f46e5', 'activationBorderColor': '#3730a3', 'signalColor': '#9ca3af', 'signalTextColor': '#e5e7eb', 'labelBoxBkgColor': '#1f2937', 'labelBoxBorderColor': '#6b7280', 'labelTextColor': '#e5e7eb', 'loopTextColor': '#e5e7eb', 'noteBkgColor': '#374151', 'noteBorderColor': '#6b7280', 'noteTextColor': '#e5e7eb'}}}%% +sequenceDiagram + participant User + participant API as K8s API Server + participant NGF as NGF Controller + participant Agent as NGINX Agent + participant NGINX + + User->>API: Create HTTPRoute + API->>NGF: Watch event: HTTPRoute created + NGF->>NGF: Validate route rules + NGF->>NGF: Check Gateway compatibility + NGF->>NGF: Generate updated config + NGF->>Agent: Send new nginx.conf + Agent->>NGINX: Test configuration + Agent->>NGINX: Reload if valid + Agent->>NGF: Report configuration status + NGF->>API: Update HTTPRoute status + NGF->>API: Update Gateway status +``` + +### Route Configuration Process + +```text +HTTPRoute Processing: +├── Parse route rules and matches +├── Validate backend references +├── Check Gateway listener compatibility +├── Generate NGINX location blocks +├── Update upstream definitions +├── Apply filters and policies +└── Send complete config to data plane +``` + +## Gateway Update Flow + +### Configuration Changes + +```mermaid +%%{init: {'theme':'dark', 'themeVariables': {'fontSize': '14px', 'darkMode': true, 'primaryColor': '#4f46e5', 'primaryTextColor': '#e5e7eb', 'primaryBorderColor': '#6b7280', 'lineColor': '#9ca3af', 'secondaryColor': '#1f2937', 'tertiaryColor': '#374151', 'background': '#111827', 'actorBkg': '#374151', 'actorBorder': '#6b7280', 'actorTextColor': '#e5e7eb', 'activationBkgColor': '#4f46e5', 'activationBorderColor': '#3730a3', 'signalColor': '#9ca3af', 'signalTextColor': '#e5e7eb', 'labelBoxBkgColor': '#1f2937', 'labelBoxBorderColor': '#6b7280', 'labelTextColor': '#e5e7eb', 'loopTextColor': '#e5e7eb', 'noteBkgColor': '#374151', 'noteBorderColor': '#6b7280', 'noteTextColor': '#e5e7eb'}}}%% +sequenceDiagram + participant User + participant API as K8s API Server + participant NGF as NGF Controller + participant Agent as NGINX Agent + participant NGINX + + User->>API: Update Gateway/Route + API->>NGF: Watch event: Resource updated + NGF->>NGF: Calculate configuration diff + NGF->>NGF: Generate new config + NGF->>Agent: Send updated config + Agent->>NGINX: Test new configuration + + alt Configuration Valid + Agent->>NGINX: Reload NGINX + Agent->>NGF: Report success + NGF->>API: Update status: Accepted + else Configuration Invalid + Agent->>NGF: Report validation error + NGF->>API: Update status: Invalid + Note over NGINX: Keep running old config + end +``` + +### Scaling Changes + +```text +Scaling Operations: +├── Update Deployment replica count +├── New pods start and register +├── Load balancer adds new endpoints +├── Traffic distributes to all pods +└── Old pods drain gracefully +``` + +## Gateway Deletion Flow + +```mermaid +%%{init: {'theme':'dark', 'themeVariables': {'fontSize': '14px', 'darkMode': true, 'primaryColor': '#4f46e5', 'primaryTextColor': '#e5e7eb', 'primaryBorderColor': '#6b7280', 'lineColor': '#9ca3af', 'secondaryColor': '#1f2937', 'tertiaryColor': '#374151', 'background': '#111827', 'actorBkg': '#374151', 'actorBorder': '#6b7280', 'actorTextColor': '#e5e7eb', 'activationBkgColor': '#4f46e5', 'activationBorderColor': '#3730a3', 'signalColor': '#9ca3af', 'signalTextColor': '#e5e7eb', 'labelBoxBkgColor': '#1f2937', 'labelBoxBorderColor': '#6b7280', 'labelTextColor': '#e5e7eb', 'loopTextColor': '#e5e7eb', 'noteBkgColor': '#374151', 'noteBorderColor': '#6b7280', 'noteTextColor': '#e5e7eb'}}}%% +sequenceDiagram + participant User + participant API as K8s API Server + participant NGF as NGF Controller + participant K8s as Kubernetes + participant Agent as NGINX Agent + + User->>API: Delete Gateway + API->>NGF: Watch event: Gateway deleted + NGF->>NGF: Check for attached routes + + alt Routes Still Attached + NGF->>API: Update route status: Gateway not found + end + + NGF->>Agent: Send shutdown signal + Agent->>Agent: Graceful shutdown + NGF->>K8s: Delete LoadBalancer Service + NGF->>K8s: Delete Deployment + NGF->>K8s: Delete ConfigMaps + NGF->>K8s: Delete ServiceAccount + K8s->>API: Resources deleted + API->>User: Gateway deletion complete +``` + +### Graceful Shutdown Process + +```text +Shutdown Steps: +├── Stop accepting new connections +├── Finish processing existing requests +├── Close upstream connections +├── Terminate NGINX process +└── Remove pod from endpoints +``` diff --git a/docs/architecture/traffic-flow.md b/docs/architecture/traffic-flow.md new file mode 100644 index 0000000000..fa57cd2264 --- /dev/null +++ b/docs/architecture/traffic-flow.md @@ -0,0 +1,139 @@ +# Traffic Flow + +This diagram shows how user requests travel through NGINX Gateway Fabric. + +## Simple Overview + +```mermaid +%%{init: {'theme':'dark', 'themeVariables': {'fontSize': '16px', 'darkMode': true, 'primaryColor': '#4f46e5', 'primaryTextColor': '#e5e7eb', 'primaryBorderColor': '#6b7280', 'lineColor': '#9ca3af', 'secondaryColor': '#1f2937', 'tertiaryColor': '#374151', 'background': '#111827', 'mainBkg': '#1f2937', 'secondBkg': '#374151', 'tertiaryTextColor': '#d1d5db'}}}%% +graph LR + %% Simple traffic flow + USER[👤 User] + + subgraph "Kubernetes Cluster" + NGINX[🌐 NGINX] + + subgraph "Your Apps" + SVC1[🔵 user-service] + SVC2[🔵 order-service] + POD1[Pod A] + POD2[Pod B] + POD3[Pod C] + end + end + + %% Simple flow + USER --> NGINX + NGINX --> SVC1 + NGINX --> SVC2 + SVC1 --> POD1 + SVC1 --> POD2 + SVC2 --> POD3 + + %% Dark-friendly styling + style USER fill:#fbbf24,stroke:#f59e0b,stroke-width:2px,color:#1f2937 + style NGINX fill:#8b5cf6,stroke:#7c3aed,stroke-width:2px,color:#ffffff + style SVC1 fill:#10b981,stroke:#059669,stroke-width:2px,color:#ffffff + style SVC2 fill:#10b981,stroke:#059669,stroke-width:2px,color:#ffffff +``` + +## Traffic Processing Steps + +### 1. User Sends Request + +```text +User Request: +├── GET /users +├── POST /orders +├── Headers: Authorization, Content-Type +└── Body: JSON data (if needed) +``` + +### 2. NGINX Receives Request + +```text +NGINX Gateway: +├── Receives request from user +├── Applies SSL termination +├── Matches routing rules +└── Selects backend service +``` + +### 3. Service Processes Request + +```text +Backend Service: +├── Receives request from NGINX +├── Processes business logic +├── Queries database (if needed) +├── Generates response +└── Returns response to NGINX +``` + +### 4. Response Returns to User + +```text +Response Flow: +├── Service → NGINX +├── NGINX → User +└── Request complete +``` + +## Detailed Request Flow + +```mermaid +%%{init: {'theme':'dark', 'themeVariables': {'fontSize': '14px', 'darkMode': true, 'primaryColor': '#4f46e5', 'primaryTextColor': '#e5e7eb', 'primaryBorderColor': '#6b7280', 'lineColor': '#9ca3af', 'secondaryColor': '#1f2937', 'tertiaryColor': '#374151', 'background': '#111827', 'actorBkg': '#374151', 'actorBorder': '#6b7280', 'actorTextColor': '#e5e7eb', 'activationBkgColor': '#4f46e5', 'activationBorderColor': '#3730a3', 'signalColor': '#9ca3af', 'signalTextColor': '#e5e7eb', 'labelBoxBkgColor': '#1f2937', 'labelBoxBorderColor': '#6b7280', 'labelTextColor': '#e5e7eb', 'loopTextColor': '#e5e7eb', 'noteBkgColor': '#374151', 'noteBorderColor': '#6b7280', 'noteTextColor': '#e5e7eb'}}}%% +sequenceDiagram + participant User + participant NGINX + participant Service as Backend Service + participant Pod + + User->>NGINX: HTTP Request + NGINX->>NGINX: Route matching + NGINX->>Service: Proxy to service + Service->>Pod: Forward to pod + Pod->>Pod: Process request + Pod->>Service: Return response + Service->>NGINX: Response + NGINX->>User: HTTP Response +``` + +## Request Routing Logic + +### Path-Based Routing + +```nginx +# Generated from HTTPRoute resources +location /users { + proxy_pass http://user-service; +} + +location /orders { + proxy_pass http://order-service; +} + +location /products { + proxy_pass http://product-service; +} +``` + + +### Host-Based Routing + +```nginx +# Different hosts route to different services +server { + server_name api.example.com; + location / { + proxy_pass http://api-service; + } +} + +server { + server_name admin.example.com; + location / { + proxy_pass http://admin-service; + } +} +``` From 33c5bc7e09bd22070e36a224c50ede80789b2727 Mon Sep 17 00:00:00 2001 From: Sarthak Agrawal <68310924+sarthyparty@users.noreply.github.com> Date: Thu, 3 Jul 2025 09:28:25 -0700 Subject: [PATCH 2/3] Change language from become to map to Co-authored-by: salonichf5 <146118978+salonichf5@users.noreply.github.com> --- docs/architecture/configuration-flow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/configuration-flow.md b/docs/architecture/configuration-flow.md index 92a9ad5640..5c365da8a3 100644 --- a/docs/architecture/configuration-flow.md +++ b/docs/architecture/configuration-flow.md @@ -1,6 +1,6 @@ # Configuration Flow -This diagram shows how Gateway API resources become NGINX configurations. +This diagram shows how Gateway API resources map to NGINX configurations. ## Simple Overview From 69a8e5e3620021c3dca06b51b4c5605b45eafa0c Mon Sep 17 00:00:00 2001 From: Sarthak Agrawal <68310924+sarthyparty@users.noreply.github.com> Date: Thu, 3 Jul 2025 09:28:57 -0700 Subject: [PATCH 3/3] Clarify controller instead of watchers in configuration flow Co-authored-by: salonichf5 <146118978+salonichf5@users.noreply.github.com> --- docs/architecture/configuration-flow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/configuration-flow.md b/docs/architecture/configuration-flow.md index 5c365da8a3..71ac88097c 100644 --- a/docs/architecture/configuration-flow.md +++ b/docs/architecture/configuration-flow.md @@ -62,7 +62,7 @@ metadata: ### 2. Kubernetes Stores Resources - Gateway, HTTPRoute, Service resources stored in etcd -- Kubernetes API Server notifies watchers of changes +- Kubernetes API Server notifies controllers(watchers) of changes ### 3. NGF Controller Processes Changes