[bug]: API container (and others) restarting due to missing table

[pccowboy]

Is there an existing issue for this?

• I have searched the existing issues

Current behavior

Running plane 2.6.3 self-hosted (commercial) on Alma Linux 9, via podman.

When I start plane via prime-cli start as root, I see this error in /var/log/messages:

Jun 25 10:49:14 wopr plane_plane-db_1[83676]: 2026-06-25 17:49:14.378 UTC [193] ERROR:  relation "embedding_models" does not exist at character 422
Jun 25 10:49:14 wopr plane_plane-db_1[83676]: 2026-06-25 17:49:14.378 UTC [193] STATEMENT:  SELECT embedding_models.deleted_at, embedding_models.created_by_id, embedding_models
.updated_by_id, embedding_models.created_at, embedding_models.updated_at, embedding_models.id, embedding_models.provider, embedding_models.model_name, embedding_models.base_api
_url, embedding_models.dimension, embedding_models.connector_id, embedding_models.model_id, embedding_models.deployment_status, embedding_models.is_active 
Jun 25 10:49:14 wopr plane_plane-db_1[83676]: #011FROM embedding_models 
Jun 25 10:49:14 wopr plane_plane-db_1[83676]: #011WHERE embedding_models.is_active IS true AND embedding_models.deleted_at IS NULL 
Jun 25 10:49:14 wopr plane_plane-db_1[83676]: #011 LIMIT 1
Jun 25 10:49:14 wopr plane_pi-api_1[92686]: #033[31m2026-06-25 17:49:14 pi.services.retrievers.pg_store.embedding_model ERROR   #033[0m Could not check ML model dynamic config:
 (psycopg2.errors.UndefinedTable) relation "embedding_models" does not exist
Jun 25 10:49:14 wopr plane_pi-api_1[92686]: LINE 2: FROM embedding_models 
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:             ^
Jun 25 10:49:14 wopr plane_pi-api_1[92686]: 
Jun 25 10:49:14 wopr plane_pi-api_1[92686]: [SQL: SELECT embedding_models.deleted_at, embedding_models.created_by_id, embedding_models.updated_by_id, embedding_models.created_a
t, embedding_models.updated_at, embedding_models.id, embedding_models.provider, embedding_models.model_name, embedding_models.base_api_url, embedding_models.dimension, embeddin
g_models.connector_id, embedding_models.model_id, embedding_models.deployment_status, embedding_models.is_active 
Jun 25 10:49:14 wopr plane_pi-api_1[92686]: FROM embedding_models 
Jun 25 10:49:14 wopr plane_pi-api_1[92686]: WHERE embedding_models.is_active IS true AND embedding_models.deleted_at IS NULL 
Jun 25 10:49:14 wopr plane_pi-api_1[92686]: LIMIT %(param_1)s]
Jun 25 10:49:14 wopr plane_pi-api_1[92686]: [parameters: {'param_1': 1}]
Jun 25 10:49:14 wopr plane_pi-api_1[92686]: (Background on this error at: https://sqlalche.me/e/20/f405)#033[0m
Jun 25 10:49:14 wopr plane_pi-api_1[92686]: Traceback (most recent call last):
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "<frozen runpy>", line 198, in _run_module_as_main
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "<frozen runpy>", line 88, in _run_code
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/app/pi/manage.py", line 39, in <module>
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    from pi.core.db.management.commands import embedding
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/app/pi/core/db/management/commands/embedding.py", line 29, in <module>
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    from pi.services.retrievers.pg_store import create_embedding_model as create_embedding_model_db
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/app/pi/services/retrievers/pg_store/__init__.py", line 14, in <module>
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    from .chat import favorite_chat
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/app/pi/services/retrievers/pg_store/chat.py", line 29, in <module>
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    from pi.agents.sql_agent.helpers import format_as_bullet_points
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/app/pi/agents/sql_agent/__init__.py", line 12, in <module>
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    from .base import text2sql
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/app/pi/agents/sql_agent/base.py", line 39, in <module>
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    from pi.services.chat.helpers.flow_tracking import FlowStepCollector
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/app/pi/services/chat/__init__.py", line 12, in <module>
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    from .chat import PlaneChatBot as PlaneChatBot
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/app/pi/services/chat/chat.py", line 40, in <module>
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    from . import action_planner
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/app/pi/services/chat/action_planner.py", line 76, in <module>
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    from .kit import TODO_STATUS_ICON
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/app/pi/services/chat/kit.py", line 44, in <module>
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    from pi.services.retrievers.docs_search import DocsRetriever
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/app/pi/services/retrievers/docs_search.py", line 21, in <module>
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    vector_db = VectorStore()
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:                ^^^^^^^^^^^^^
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/app/pi/core/vectordb/client.py", line 131, in __init__
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    self.os = OpenSearch(
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:              ^^^^^^^^^^^
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/usr/local/lib/python3.12/site-packages/opensearchpy/client/__init__.py", line 219, in __init__
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    super().__init__(hosts, transport_class, **kwargs)
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/usr/local/lib/python3.12/site-packages/opensearchpy/client/client.py", line 41, in __init__
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    self.transport = transport_class(_normalize_hosts(hosts), **kwargs)
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/usr/local/lib/python3.12/site-packages/opensearchpy/transport.py", line 201, in __init__
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    self.set_connections(hosts)
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/usr/local/lib/python3.12/site-packages/opensearchpy/transport.py", line 255, in set_connections
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    connections = list(zip(map(_create_connection, hosts), hosts))
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/usr/local/lib/python3.12/site-packages/opensearchpy/transport.py", line 253, in _create_connection
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    return self.connection_class(metrics=self.metrics, **kwargs)
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/usr/local/lib/python3.12/site-packages/opensearchpy/connection/http_urllib3.py", line 130, in __init__
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    super().__init__(
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:  File "/usr/local/lib/python3.12/site-packages/opensearchpy/connection/base.py", line 121, in __init__
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:    if ":" in host:  # IPv6
Jun 25 10:49:14 wopr plane_pi-api_1[92686]:       ^^^^^^^^^^^
Jun 25 10:49:14 wopr plane_pi-api_1[92686]: TypeError: argument of type 'NoneType' is not iterable
Jun 25 10:49:14 wopr systemd[1]: libpod-698a529a8d53e1da0cbf1ede090fe126c7350aa67d26f1c9a271e364a0febc29.scope: Deactivated successfully.
Jun 25 10:49:14 wopr systemd[1]: libpod-698a529a8d53e1da0cbf1ede090fe126c7350aa67d26f1c9a271e364a0febc29.scope: Consumed 1.281s CPU time.
Jun 25 10:49:14 wopr podman[91937]: @ - - [25/Jun/2026:10:49:14 -0700] "GET /v1.44/containers/json?all=1 HTTP/1.1" 200 40301 "" "Go-http-client/1.1"
Jun 25 10:49:14 wopr podman[91985]: 2026-06-25 10:49:14.492980071 -0700 PDT m=+10.287819741 container died 698a529a8d53e1da0cbf1ede090fe126c7350aa67d26f1c9a271e364a0febc29 (image=docker.io/makeplane/plane-pi-commercial:v2.6.3, name=plane_pi-beat_1, com.docker.compose.project.config_files=docker-compose.yml, io.podman.compose.service=pi-beat, PODMAN_SYSTEMD_UNIT=podman-compose@plane.service, com.docker.compose.project.working_dir=/var/plane, io.podman.compose.project=plane, com.docker.compose.container-number=1, com.docker.compose.project=plane, io.podman.compose.config-hash=24d315c745e62976757750955b76b23f9d989d1c03fbb76917e2fe64f2002f63, com.docker.compose.service=pi-beat, io.podman.compose.version=1.5.0)

This causes chaos on the host I run plane on, as the constant container restarts bounce my network connections for other apps. I have tried to "repair" the installation, but the repair process errors out with something else.

What can I provide to help troubleshoot this? Happy do dig in in whatever manner helps.

Steps to reproduce

prime-cli start
grep embedding_models /var/log/messages
prime-cli stop

Environment

Deploy preview

Browser

Google Chrome

Variant

Self-hosted

Version

v2.6.3

[makeplane]

Synced with Plane Workspace 🔄

[GIT-264] [bug]: API container (and others) restarting due to missing table

[pccowboy]

more data from digging in /var/log/messages with opus:

There are actually two separate problems here: the embedding_models table is missing, which gets logged as an error but is handled gracefully with a "Could not check" message. The real killer is that the OpenSearch host configuration is coming through as None, which causes a TypeError when the code tries to check if ":" is in the host string during the VectorStore initialization—this crashes the entire import chain and prevents the container from starting.

The compose topology is clear, and line 12 is the smoking gun: plane_pi-migrator_1 crashes on the same OpenSearch error. That means the migrator that should create embedding_models dies before it can. Let me confirm the migrator failure and find the prime-cli repair error.

[pccowboy]

FWIW, I removed the following from the docker compose files, and I am able to run without the container crashes, since the pi* containers don't get started

pi-db-init: (line 341)
pi-api: (line 450)
pi-beat: (line 462)
pi-worker: (line 475)
pi-migrator: (line 487)

[danciaclara]

@pccowboy Glad that stopped the crashes. One thing to be aware of though, prime-cli regenerates the docker-compose files when you upgrade or repair, so those manual edits will be overwritten and the crashing pi containers would come back.

The safe way to disable them is to set these in your env file
PI_API_REPLICAS=0
PI_BEAT_REPLICAS=0
PI_WORKER_REPLICAS=0
PI_MIGRATOR_REPLICAS=0

That file isn't touched on upgrade, so the pi services will stay disabled going forward. You can then restore your docker-compose files with prime-cli repair and it should be stable.
For context, all the pi services power Plane AI. If you ever want to enable that, you'd configure OpenSearch and set the replica counts back to 1. No need to do anything now if you're not using AI.

Also, you mentioned the repair process was erroring out. What error did you see? That might indicate a separate issue.
@akshat5302 @mguptahub @sunder-ch

[pccowboy]

PI_API_REPLICAS=0
PI_BEAT_REPLICAS=0
PI_WORKER_REPLICAS=0
PI_MIGRATOR_REPLICAS=0

Those are set to zero, they were in the original plane.env file.

What the content shows
In plane.env, the Pi services are already turned down:

PI_API_REPLICAS=0
PI_BEAT_REPLICAS=0
PI_WORKER_REPLICAS=0
PI_MIGRATOR_REPLICAS=0
And the Pi stack was never wired to function anyway:

OPENSEARCH_ENABLED=0
OPENSEARCH_URL=          # empty -> this is the None that crash-loops pi-api
OPENAI_API_KEY=          # all LLM provider keys empty
EMBEDDING_MODEL=         # empty
Yet the syslog shows pi-api/pi-migrator/pi-db-init running and crash-looping the whole Jun 22–25 window. So replicas: 0 is not being honored by this podman-compose setup — setting it to 0 (or lower) will not stop them. The only reliable disable is to remove the service definitions from the compose file.