'realtime' starts 404-ing immediately after starting. Continues to show up as "Unhealthy"

[subashc2023]

Bug report

• [x ] I confirm this is a bug with Supabase, not with my own application.
• [ x] I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

The supabase-realtime container starts 404-ing immediately after being started.

To Reproduce

Steps to reproduce the behavior, please provide code snippets or a repository:

1. Clone supabase
2. set env variables
3. docker compose up -d
4. docker ps
5. see error
6. Check logs
7. See these 404s

Expected behavior

Supabase-realtime should be active, and healthy.

Screenshots

 *  Executing task: docker logs --tail 1000 -f f6cce65be802fb4a263020a27a6eb147037dbfe6110e266a8f2937ab663306ef 

+ ulimit -n
+ '[' '!' -z 10000 ']'
+ echo 'Setting RLIMIT_NOFILE to 10000'
+ ulimit -Sn 10000
+ export ERL_CRASH_DUMP=/tmp/erl_crash.dump
+ ERL_CRASH_DUMP=/tmp/erl_crash.dump
1048576
Setting RLIMIT_NOFILE to 10000
+ '[' false = true ']'
+ echo 'Running migrations'
+ sudo -E -u nobody /app/bin/migrate
Running migrations
02:05:38.907 [info] == Running 20210706140551 Realtime.Repo.Migrations.CreateTenants.change/0 forward
02:05:38.909 [info] create table tenants
02:05:38.937 [info] create index tenants_external_id_index
02:05:38.947 [info] == Migrated 20210706140551 in 0.0s
02:05:38.995 [info] == Running 20220329161857 Realtime.Repo.Migrations.AddExtensionsTable.change/0 forward
02:05:38.995 [info] create table extensions
02:05:39.028 [info] create index extensions_tenant_external_id_type_index
02:05:39.038 [info] == Migrated 20220329161857 in 0.0s
02:05:39.043 [info] == Running 20220410212326 Realtime.Repo.Migrations.AddTenantMaxEps.up/0 forward
02:05:39.043 [info] alter table tenants
02:05:39.047 [info] == Migrated 20220410212326 in 0.0s
02:05:39.049 [info] == Running 20220506102948 Realtime.Repo.Migrations.RenamePollIntervalToPollIntervalMs.up/0 forward
02:05:39.051 [warning] Replica region not found, defaulting to Realtime.Repo
02:05:39.059 [info] == Migrated 20220506102948 in 0.0s
02:05:39.067 [info] == Running 20220527210857 Realtime.Repo.Migrations.AddExternalIdUniqIndex.change/0 forward
02:05:39.069 [info] execute "alter table tenants add constraint uniq_external_id unique (external_id)"
02:05:39.082 [info] == Migrated 20220527210857 in 0.0s
02:05:39.085 [info] == Running 20220815211129 Realtime.Repo.Migrations.NewMaxEventsPerSecondDefault.change/0 forward
02:05:39.085 [info] alter table tenants
02:05:39.091 [info] == Migrated 20220815211129 in 0.0s
02:05:39.094 [info] == Running 20220815215024 Realtime.Repo.Migrations.SetCurrentMaxEventsPerSecond.change/0 forward
02:05:39.094 [info] execute "update tenants set max_events_per_second = 1000"
02:05:39.103 [info] == Migrated 20220815215024 in 0.0s
02:05:39.105 [info] == Running 20220818141501 Realtime.Repo.Migrations.ChangeLimitsDefaults.change/0 forward
02:05:39.105 [info] alter table tenants
02:05:39.112 [info] == Migrated 20220818141501 in 0.0s
02:05:39.116 [info] == Running 20221018173709 Realtime.Repo.Migrations.AddCdcDefault.up/0 forward
02:05:39.116 [info] alter table tenants
02:05:39.119 [info] == Migrated 20221018173709 in 0.0s
02:05:39.122 [info] == Running 20221102172703 Realtime.Repo.Migrations.RenamePgType.up/0 forward
02:05:39.122 [info] execute "update extensions set type = 'postgres_cdc_rls'"
02:05:39.130 [info] == Migrated 20221102172703 in 0.0s
02:05:39.132 [info] == Running 20221223010058 Realtime.Repo.Migrations.DropTenantsUniqExternalIdIndex.change/0 forward
02:05:39.132 [info] execute "ALTER TABLE IF EXISTS tenants DROP CONSTRAINT IF EXISTS uniq_external_id"
02:05:39.138 [info] == Migrated 20221223010058 in 0.0s
02:05:39.142 [info] == Running 20230110180046 Realtime.Repo.Migrations.AddLimitsFieldsToTenants.change/0 forward
02:05:39.142 [info] alter table tenants
02:05:39.146 [info] == Migrated 20230110180046 in 0.0s
02:05:39.149 [info] == Running 20230810220907 Realtime.Repo.Migrations.AlterTenantsTableColumnsToText.change/0 forward
02:05:39.150 [info] alter table tenants
02:05:39.155 [info] == Migrated 20230810220907 in 0.0s
02:05:39.158 [info] == Running 20230810220924 Realtime.Repo.Migrations.AlterExtensionsTableColumnsToText.change/0 forward
02:05:39.158 [info] alter table extensions
02:05:39.163 [info] == Migrated 20230810220924 in 0.0s
02:05:39.166 [info] == Running 20231024094642 :"Elixir.Realtime.Repo.Migrations.Add-tenant-suspend-flag".change/0 forward
02:05:39.166 [info] alter table tenants
02:05:39.171 [info] == Migrated 20231024094642 in 0.0s
02:05:39.174 [info] == Running 20240306114423 Realtime.Repo.Migrations.AdddTenantJwtJwksColumn.change/0 forward
02:05:39.174 [info] alter table tenants
02:05:39.180 [info] == Migrated 20240306114423 in 0.0s
02:05:39.183 [info] == Running 20240418082835 Realtime.Repo.Migrations.AddAuthorizationFlag.change/0 forward
02:05:39.183 [info] alter table tenants
02:05:39.188 [info] == Migrated 20240418082835 in 0.0s
02:05:39.191 [info] == Running 20240625211759 Realtime.Repo.Migrations.RemoveEnableAuthorizationFlag.change/0 forward
02:05:39.191 [info] alter table tenants
02:05:39.199 [info] == Migrated 20240625211759 in 0.0s
02:05:39.202 [info] == Running 20240704172020 :"Elixir.Realtime.Repo.Migrations.Add-notify-private-alpha".change/0 forward
02:05:39.202 [info] alter table tenants
02:05:39.217 [info] == Migrated 20240704172020 in 0.0s
02:05:39.220 [info] == Running 20240902173232 Realtime.Repo.Migrations.AddExtensionExternalIdIndex.change/0 forward
02:05:39.220 [info] create index if not exists extensions_tenant_external_id_index
02:05:39.228 [info] == Migrated 20240902173232 in 0.0s
02:05:39.231 [info] == Running 20241106103258 Realtime.Repo.Migrations.AddPrivateOnlyFlagColumnToTenant.change/0 forward
02:05:39.231 [info] alter table tenants
02:05:39.238 [info] == Migrated 20241106103258 in 0.0s
+ '[' true = true ']'
+ echo 'Seeding selfhosted Realtime'
+ sudo -E -u nobody /app/bin/realtime eval 'Realtime.Release.seeds(Realtime.Repo)'
Seeding selfhosted Realtime
Starting Realtime
+ echo 'Starting Realtime'
+ ulimit -n
+ exec /app/bin/server
10000
02:05:40.849 [notice] SYN[realtime@127.0.0.1] Adding node to scope <Elixir.Realtime.Tenants.Connect>
02:05:40.850 [notice] SYN[realtime@127.0.0.1] Creating tables for scope <Elixir.Realtime.Tenants.Connect>
02:05:40.850 [notice] SYN[realtime@127.0.0.1|registry<Elixir.Realtime.Tenants.Connect>] Discovering the cluster
02:05:40.850 [notice] SYN[realtime@127.0.0.1|pg<Elixir.Realtime.Tenants.Connect>] Discovering the cluster
02:05:40.850 [notice] SYN[realtime@127.0.0.1] Adding node to scope <users>
02:05:40.850 [notice] SYN[realtime@127.0.0.1] Creating tables for scope <users>
02:05:40.850 [notice] SYN[realtime@127.0.0.1|registry<users>] Discovering the cluster        
02:05:40.850 [notice] SYN[realtime@127.0.0.1|pg<users>] Discovering the cluster
02:05:40.850 [notice] SYN[realtime@127.0.0.1] Adding node to scope <Elixir.RegionNodes>      
02:05:40.850 [notice] SYN[realtime@127.0.0.1] Creating tables for scope <Elixir.RegionNodes> 
02:05:40.850 [notice] SYN[realtime@127.0.0.1|registry<Elixir.RegionNodes>] Discovering the cluster
02:05:40.851 [notice] SYN[realtime@127.0.0.1|pg<Elixir.RegionNodes>] Discovering the cluster 
02:05:40.851 [warning] Replica region not found, defaulting to Realtime.Repo
02:05:40.863 [info] Running RealtimeWeb.Endpoint with cowboy 2.12.0 at :::4000 (http)        
02:05:40.864 [info] Access RealtimeWeb.Endpoint at http://realtime.fly.dev
02:05:40.865 [notice] SYN[realtime@127.0.0.1] Adding node to scope <Elixir.Extensions.PostgresCdcRls>
02:05:40.865 [notice] SYN[realtime@127.0.0.1] Creating tables for scope <Elixir.Extensions.PostgresCdcRls>
02:05:40.865 [notice] SYN[realtime@127.0.0.1|registry<Elixir.Extensions.PostgresCdcRls>] Discovering the cluster
02:05:40.865 [notice] SYN[realtime@127.0.0.1|pg<Elixir.Extensions.PostgresCdcRls>] Discovering the cluster
02:05:40.866 [info] Janitor started
02:05:40.866 [info] Starting MetricsCleaner
02:05:42.977 request_id=GCef2sYjqKYiF1IAAAAQ [info] HEAD /api/tenants/realtime-dev/health    
02:05:42.978 request_id=GCef2sYjqKYiF1IAAAAQ [info] Sent 403 in 290µs
02:05:47.996 request_id=GCef2_FDnAk9mX8AAABQ [info] HEAD /api/tenants/realtime-dev/health    
02:05:47.996 request_id=GCef2_FDnAk9mX8AAABQ [info] Sent 403 in 254µs
02:05:53.016 request_id=GCef3Rx8LnBzEBcAAACQ [info] HEAD /api/tenants/realtime-dev/health    
02:05:53.016 request_id=GCef3Rx8LnBzEBcAAACQ [info] Sent 403 in 232µs
02:05:58.032 request_id=GCef3keRhhUaeuIAAADQ [info] HEAD /api/tenants/realtime-dev/health    
02:05:58.033 request_id=GCef3keRhhUaeuIAAADQ [info] Sent 403 in 349µs
02:06:03.051 request_id=GCef33K43R2guJoAAAEQ [info] HEAD /api/tenants/realtime-dev/health    
02:06:03.052 request_id=GCef33K43R2guJoAAAEQ [info] Sent 403 in 264µs
02:06:08.070 request_id=GCef4J3aNEVIKnQAAAFQ [info] HEAD /api/tenants/realtime-dev/health    
02:06:08.070 request_id=GCef4J3aNEVIKnQAAAFQ [info] Sent 403 in 237µs
02:06:13.086 request_id=GCef4cjYLs40mnYAAAGQ [info] HEAD /api/tenants/realtime-dev/health
02:06:13.086 request_id=GCef4cjYLs40mnYAAAGQ [info] Sent 403 in 277µs
02:06:18.103 request_id=GCef4vPY74cpSpAAAAHQ [info] HEAD /api/tenants/realtime-dev/health
02:06:18.103 request_id=GCef4vPY74cpSpAAAAHQ [info] Sent 403 in 286µs
02:06:23.120 request_id=GCef5B7rTNRLeTMAAAIQ [info] HEAD /api/tenants/realtime-dev/health
02:06:23.120 request_id=GCef5B7rTNRLeTMAAAIQ [info] Sent 403 in 252µs

System information

• OS: Windows
• Version of supabase: latest
• Version of Node.js: latest

Additional context

I had some issues with analytics and stuff, but this was fixed by disabled autocrlf in git global config and re cloning the repo. This realtime issue was not there when I was experimenting with this supabase yesterday. I recloned and reran it today, and it stopped working. I made sure to purge all the volumes etc. using the docker purge command mentions in the other issue of Analytics and Realtime not working.

[filipecabaco]

From the logs provided it sees that it's a 403 which means that JWT secret for the API is not properly set which leads the healthcheck to fail

[mhaase90]

I had the same error until just now. After hours of trying, I came up with the idea that it could be due to the length of the JWT Secret. It was because the JWT Secret was too long (64 characters). I used 40 characters, as in the documentation, and now the unhealthy has disappeared. I hope it's the same for you.