Prefect Server (v2.20) silently crashing in Docker Compose on Ubuntu 24.04 LTS

[richelieuli]

Problem Description:

Hello Prefect Community,

I am trying to deploy Prefect Server using Docker Compose on my Ubuntu 24.04 LTS machine. The database service starts correctly, and the Prefect Server container itself starts, but the Prefect Server application process inside the container immediately exits after printing the Prefect ASCII logo in the logs. There are no error messages, warnings, or any other output following the logo. The container then enters a Restarting (0) loop according to docker compose ps.

The Prefect Agent container also starts but then restarts (Restarting (1)) as it cannot connect to a running server.

I've followed extensive troubleshooting steps over several hours, including fixes for common issues like missing dependencies and configuration, but the silent crash persists with the exact same log output pattern.

Environment:

• Docker Version:

  Client: Docker Engine - Community
  Version:           28.1.1
  API version:       1.49
  Go version:        go1.23.8
  Git commit:        4eba377
  Built:             Fri Apr 18 09:52:14 2025
  OS/Arch:           linux/amd64
  Context:           default
  
  Server: Docker Engine - Community
  Engine:
   Version:          28.1.1
   API version:      1.49 (minimum version 1.24)
   Go version:       go1.23.8
   Git commit:       01f442b
   Built:            Fri Apr 18 09:52:14 2025
   OS/Arch:          linux/amd64
   Experimental:     false
  containerd:
   Version:          1.7.27
   GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
  runc:
   Version:          1.2.5
   GitCommit:        v1.2.5-0-g59923ef
  docker-init:
   Version:          0.19.0
   GitCommit:        de40ad0
  

• Ubuntu Version:

  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:       Ubuntu 24.04.2 LTS
  Release:           24.04
  Codename:          noble
  

Docker Compose Configuration:

• docker-compose.yml:

  version: '3.8'
  
  # Define your services (containers)
  services:
  
    # Database service
    db: # **Service name used for networking**
      image: timescale/timescaledb:latest-pg16 # Use TimescaleDB image
      container_name: aoros_db # **Container name**
      restart: always # Set container to always restart unless manually stopped
      ports:
        - "5432:5432" # Map internal container port 5432 to host machine port 5432 for host access
      environment: # Set database initialization environment variables
        POSTGRES_USER: aoros # Database username
        POSTGRES_PASSWORD: Aoros # Database password
        POSTGRES_DB: aoros # Database name
      volumes:
        - db_data:/var/lib/postgresql/data # Mount a named volume to persist database data to prevent data loss on container removal
  
    # Application service
    aoros_app: # **Service name**
      image: framework # Use your locally built image
      container_name: my_app # **Container name**
      build: # Build from Dockerfile
        context: . # Directory containing the Dockerfile (current directory)
        dockerfile: Dockerfile # Dockerfile name
      working_dir: /app # Set working directory
      volumes:
        - .:/app # Mount project code directory for development (no need to rebuild image on code changes)
  
      environment: # Environment variables for the application
        DATABASE_HOST: db # **Important!** Use the database service name as the hostname
        DATABASE_PORT: 5432 # Default database port
        DATABASE_USER: aoros # Database username (must match db service)
        DATABASE_PASSWORD: Aorosquant # Database password (must match db service)
        DATABASE_NAME: aoros # Database name (must match db service)
  
      depends_on:
        - db # Indicates app service depends on db service, ensuring db starts before app
  
      # Keep original command or modify if needed for testing the app
      command: ["python", "datasources/yfinance_downloader.py"] # Example command
  
    # Corrected Prefect Server service definition
    prefect-server: # **Service name**
      # Build the image from the Dockerfile-prefect-server
      build:
        context: . # Build context is the current directory
        dockerfile: Dockerfile-prefect-server # Path to our custom Dockerfile
  
      container_name: aoros_server # Container name
      restart: always # Always restart
  
      ports:
        - "4200:4200" # Map internal port 4200 to host port 4200
  
      environment:
        # Prefect Server configuration environment variables
        - PREFECT_API_URL=http://prefect-server:4200/api # Server internal communication URL (using service name)
        # Database connection URL, using asyncpg driver, pointing to the correct database service name
        - PREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://aoros:Aorosquant@db:5432/aoros # <--- **Ensure credentials and database service name ('db') are correct**
        - PREFECT_LOGGING_LEVEL=DEBUG # Set logging level
        # Set the address Prefect Server binds to inside the container, allowing host access
        - PREFECT_API_HOST=0.0.0.0
  
      depends_on:
        # Prefect Server depends on the database service to start first
        - db # **Important!** Depend on the correct database service name ('db')
  
    # Prefect Agent service definition (corrected definition)
    prefect-agent: # **Service name**
      # Use the same Prefect base image version as the server
      image: prefecthq/prefect:2.20-python3.10
      container_name: aoros_agent
      restart: always
  
      environment:
        # Prefect Agent configuration
        - PREFECT_API_URL=http://prefect-server:4200/api # Agent connects to Server address
        - PREFECT_AGENT_LABELS=["default"] # Agent labels
  
      volumes:
        # Mount the Docker socket so the Agent can spin up new containers to run Task
        - /var/run/docker.sock:/var/run/docker.sock
        # Optional: Mount your application code volume if needed by Task execution
        # e.g., if Tasks need to run Python scripts from your project not in the Task image
        # - .:/opt/prefect/flows # Example: mount current directory to /opt/prefect/flows inside Agent
  
      depends_on:
        - prefect-server # Agent depends on Server to start
  
      # Corrected Agent startup command: Start Prefect Docker Agent
      command: prefect agent start -p docker
  
  # Define persistent volumes
  volumes:
    db_data: # Database data volume

• Dockerfile (for aoros_app):

  # Use official Python image as base image
  # Choose a suitable Python version consistent with your local development environment
  FROM python:3.10-slim-bullseye
  
  # Set working directory
  # This directory is where your project code resides inside the container
  WORKDIR /app
  
  # Copy project code into the container
  # . represents the host machine's current directory (project root)
  # /app represents the working directory inside the container
  COPY . /app
  
  # Install Python dependencies
  # Assuming you have a requirements.txt file in your project root listing all Python dependencies
  # If you don't have requirements.txt, you can generate one in your Conda environment using pip freeze > requirements.txt
  RUN pip install --no-cache-dir -r requirements.txt

• Dockerfile-prefect-server:

  # Dockerfile-prefect-server
  
  # Use a specific, known-stable Prefect image tag for Python 3.10
  # A version we successfully built before, e.g., 2.20-python3.10
  FROM prefecthq/prefect:2.20-python3.10
  
  # Install asyncpg and psycopg2-binary and their build dependencies
  # These are drivers needed for asynchronous PostgreSQL operations
  RUN apt-get update && \
      apt-get install -y libpq-dev gcc python3-dev && \
      pip install asyncpg psycopg2-binary && \
      # Clean up apt cache to reduce image size
      rm -rf /var/lib/apt/lists/*
  
  # Prefect images usually have ENTRYPOINT and CMD set; we rely on their defaults

Observed Behavior:

• Output of docker compose ps:

  NAME            IMAGE                              COMMAND                  SERVICE           CREATED          STATUS                      PORTS
  aoros_agent     prefecthq/prefect:2.20-python3.10  "/usr/bin/tini -g --…"   prefect-agent     21 minutes ago   Restarting (1) 38 seconds ago
  aoros_db        timescale/timescaledb:latest-pg16  "docker-entrypoint.s…"   db                2 minutes ago    Up 12 seconds               0.0.0.0:5432->5432/tcp, [::]:5432->5432/tcp
  aoros_server    aorosquant-prefect-server          "/usr/bin/tini -g --…"   prefect-server    21 minutes ago   Restarting (0) 44 seconds ago
  my_app          framework                          "python datasources/…"   aoros_app         2 minutes ago    Up 12 seconds
  

• Output of docker logs aoros_server:

  # (Paste the output you provided showing only repeated Prefect logos)
   ___ ___ ___ ___ ___ ___ _____
  | _ \ _ \ __| __| __/ __|_  _|
  |  _/   / _|| _|| _| (__  | |
  |_| |_|\___|_| |___\___| |_|
  
   ___ ___ ___ ___ ___ ___ _____
  ... (repeated logos, nothing else)
  

• When trying to access the UI (http://localhost:4200), the browser shows a "Connection reset" or "Unable to connect" error.

Troubleshooting Steps Taken:

I've followed extensive debugging steps over several hours, collaborating with an AI assistant. Here's a summary:

1. Initially faced ModuleNotFoundError for psycopg2 and Docker build network issues. Resolved by fixing host DNS and creating Dockerfile-prefect-server to install psycopg2 and asyncpg.
2. Encountered sqlalchemy.exc.InvalidRequestError related to async driver. Resolved by ensuring asyncpg is installed and using postgresql+asyncpg:// URL scheme. Verified asyncpg is importable in the container using docker compose run --entrypoint python prefect-server -c 'import asyncpg' successful).
3. Ensured all necessary environment variables (PREFECT_API_URL, PREFECT_API_DATABASE_CONNECTION_URL, PREFECT_LOGGING_LEVEL=DEBUG, PREFECT_API_HOST=0.0.0.0) are correctly set and passed. Verified with docker compose run printenv.
4. Tried manually overriding the command to run Uvicorn directly (--host 0.0.0.0), which resulted in even earlier crashes with minimal logs.
5. Tested multiple Prefect base image versions (including 2-latest and specific 2.20-python3.10), successfully built images each time, but the silent crash behavior remained identical.
6. Reverted application code to an older version from GitHub and combined it with the corrected Prefect Docker setup.
7. Used docker compose down --volumes --remove-orphans and docker stop/rm to ensure a completely clean Docker state between up attempts, resolving "port is already allocated" errors.
8. Validated docker-compose.yml syntax using docker compose config.

Despite all these steps, the Prefect Server consistently exhibits the same behavior: starts, prints the logo, and exits immediately with exit code 0, without logging any errors or further startup progress. The lack of logs at the point of failure is the main blocker for further diagnosis.

Request for Help:

I would be very grateful for any insights or suggestions from the community on what could be causing this very early, silent exit in the Prefect Server startup process.

• Are there any known issues with Prefect Server v2.x (specifically 2.20) on Ubuntu 24.04 or with Docker Engine 28.1.1 that could cause this?
• Could this be related to the tini entrypoint and its interaction with the Prefect process or the specific environment?
• Are there any lower-level debugging flags or environment variables I could set to get more verbose output before the standard logging is fully initialized?
• Any other ideas on how to diagnose a process that exits immediately with exit code 0 without logging?

Thank you for your time and expertise!

[desertaxle]

Hey @richelieuli!

I noticed this comment in your server Dockerfile:

# Prefect images usually have ENTRYPOINT and CMD set; we rely on their defaults

The Prefect Docker images set an ENTRYPOINT, but they don't set a default command. If no command is provided, the entrypoint will print the Prefect ASCII art and then immediately exit with a 0 exit code.

You can see this behavior by running something like this:

docker run prefecthq/prefect:2.20-python3.10

which will print the ASCII art and exit like you're seeing.

To fix this, I think you should add the prefect server start command to your prefect-server service in your docker-compose.yml.

Let me know if that solves the issue for you. We can dig in more to find the underlying issue if it doesn't.

[richelieuli]

Thank you so much for taking the time to look into my issue and for your suggestion!

You were absolutely right! Adding the command: prefect server start line to the prefect-server service in my docker-compose.yml did solve the issue!

The Prefect Server container is now starting and staying Up (docker compose ps confirms this), and I can successfully access the UI at http://localhost:4200.

Your explanation about the Prefect images not having a default command and the entrypoint exiting with code 0 was the key I was missing after many hours of debugging. It perfectly explains the silent exit behavior I was seeing.

Thank you again for your valuable insight and help! I really appreciate it.