🪲: Proxy IP blocked even though it is trusted

[VorpalBlade]

What happened?

Apparently stalwart i blocking the nginx reverse proxy on https even though:

$  grep trusted config.toml
server.listener.imaptls.proxy.trusted-networks = "172.16.0.0/12"
server.listener.sieve.proxy.trusted-networks = "172.16.0.0/12"
server.listener.smtp.proxy.trusted-networks = "172.16.0.0/12"
server.listener.submission.proxy.trusted-networks = "172.16.0.0/12"
server.listener.submissions.proxy.trusted-networks = "172.16.0.0/12"

Apparently https isn't there, adding it makes no difference. And I seem to remember I had configured that in the UI, so it is presumably in the db.

Yet in my logs I see:

2025-09-15T19:45:19Z INFO Blocked IP address (security.ip-blocked) listenerId = "https", localPort = 443, remoteIp = 172.19.0.1, remotePort = 51450
2025-09-15T19:45:36Z INFO Blocked IP address (security.ip-blocked) listenerId = "https", localPort = 443, remoteIp = 172.19.0.1, remotePort = 43172
2025-09-15T19:45:36Z INFO Blocked IP address (security.ip-blocked) listenerId = "https", localPort = 443, remoteIp = 172.19.0.1, remotePort = 43184
2025-09-15T19:45:37Z INFO Blocked IP address (security.ip-blocked) listenerId = "https", localPort = 443, remoteIp = 172.19.0.1, remotePort = 43196
2025-09-15T19:45:38Z INFO Blocked IP address (security.ip-blocked) listenerId = "https", localPort = 443, remoteIp = 172.19.0.1, remotePort = 43206

And in my nginx log I then see:

2025/09/15 19:45:00 [error] 214083#214083: *70975 peer closed connection in SSL handshake while SSL handshaking to upstream, client: xx.xx.xx.xx, server: mail.example.com, request: "GET / HTTP/2.0", upstream: "https://127.0.0.1:10443/", host: "mail.example.com"

I run with the following docker compose:

services:
  stalwart:
    image: stalwartlabs/stalwart
    restart: unless-stopped
    container_name: mail
    hostname: mail
    ports:
      # Mail
      - 127.0.0.1:10025:25 # SMTP
      - 127.0.0.1:10465:465 # SMTPS
      - 127.0.0.1:10587:587 # Submission
      - 127.0.0.1:10993:993 # IMAP TLS
      - 127.0.0.1:14190:4190 # ManageSieve
      # Admin
      - 127.0.0.1:10443:443
      - 127.0.0.1:18080:8080
      #- 110:110 # POP3
      #- 143:143 # IMAP
      #- 995:995 # POP3S
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/lib/stalwart-mail:/opt/stalwart
      - /var/lib/stalwart-mail:/opt/stalwart-mail
    networks:
      - mail

networks:
  mail:
    enable_ipv6: true
    ipam:
      config:
        - subnet: 2a01:4f9:c012:e134:0025::/80

And for nginx I do stream proxying for all protocols except https as I need to do virtual host handling on nginx. For that I instead do:

server {
    listen [::]:443 quic;
    listen 443 quic;
    listen [::]:443 ssl;
    listen 443 ssl;
    http2 on;
    http3 on;
    quic_retry on;
    ssl_early_data on;
    quic_gso on; 

    server_name mail.example.com autodiscover.example.com autoconfig.example.com mta-sts.example.com;

    # SSL configuration using Let's Encrypt certificates
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    # Optional security headers
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header Content-Security-Policy "frame-ancestors 'self'";
    add_header X-Frame-Options DENY;
    add_header Referrer-Policy same-origin;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
    #add_header Alt-Svc 'h3=":443"; ma=86400';

    client_max_body_size 50M;

    # Reverse Proxy for stalwart
    location / {
        proxy_pass https://127.0.0.1:10443;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        # Adjust the timeout if necessary
        proxy_read_timeout 90;
    }
}

Since stalwart now blocks my host I have no idea how to get in and fix the situation... Since everything is stored in a DB there isn't much I can do without getting into it. This only broke after the 0.13.x upgrade. This same config worked just fine on 0.12.x

How can we reproduce the problem?

I can reproduce the problem by doing the following steps:

Try to use stalwart behind a reverse proxy?

Version

v0.13.x

What database are you using?

RocksDB

What blob storage are you using?

RocksDB

Where is your directory located?

Internal

What operating system are you using?

None

Relevant log output

Yeah I can't get into the UI to change the log level...

Code of Conduct

• I agree to follow this project's Code of Conduct

[VorpalBlade]

I managed to run stalwart-cli from inside the docker container and have it talk to 127.0.0.1 in there. Which made it possible to unban the host. However it is still broken, giving me 502 Bad Gateway. I see:

2025-09-15T20:09:30Z WARN Proxy protocol error (network.proxy-error) listenerId = "https", localIp = ::, localPort = 443, tls = true, reason = "invalid proxy header"

Which is really odd: This used to work before the upgrade, using a normal HTTP proxy for this, rather than a stream proxy...

---

UPDATE: Removing the trusted network for https that I added as a test fixed that. Apparently that is not compatible with plain HTTP proxying. However that doesn't explain how to make sure that the host doesn't get banned when using a non-proxy-protocol proxy (as is needed if you have multiple virtual hosts and use nginx)

[VorpalBlade]

"Obtain remote IP from Forwarded header" in the UI had somehow gotten reset. I know for a fact this was set before. A few other settings have gotten lost by themselves before (e.g. %{file:/opt/stalwart/certs/fullchain.pem}% was replaced by the contents of the file), but I assumed I had done something wrong then. I don't believe it is my fault any longer, it is now the third time settings have been myseriously changed by themselves. And I'm the only admin and I use a very strong randomly generated password.

[mdecimus]

From which version have you upgraded from? Have you checked the CHANGELOG and UPGRADING notes to see if any relevant settings changed?

[VorpalBlade]

I upgraded from 0.12.5. I did read the upgrading notes. I don't see anything in the changelog that relates to proxy or forwarded headers. Are you saying that you don't mention all the possibly breaking changes in UPGRADING? If so it would be good to add a note to that effect.

[mdecimus]

Upgrading instructions (how to migrate, etc) are in UPGRADING while changes (both breaking and not) are in the CHANGELOG. I've just checked the changes since 0.12.5 and there were no changes in the configuration keys.

[VorpalBlade]

Then it does seem like a bug that the setting got trusting forwarding headers over Http turned itself off...

[freelon]

I've stumbled into the same problem and found that the docs and the source refer to http.use-x-forwarded while the UI sets http.use-x-forwarded-for. After setting it in the UI it started working again, i.e. in the logs I can get the proxies and my real IP shown as remoteIp (I'm running 0.13.3).

[VorpalBlade]

Intersting, so you also had it turn off by itself?

[freelon]

I would think so. I just had a look into an old backup of my config and there is was *server*.http.use-x-forwarded = true, which obviously didn't work in the newer version (I have no idea why it worked earlier, maybe there was a change in the config scheme awhile ago?).

[freelon]

Yeah, I just checked some old logs, it did work at some time in the past, the last entry where another remoteIp than my reverse proxy showed up for http was around June 20th, after which I updated from 0.11.8 to 0.12.

[Megver83]

I've exactly the same issue with traefik, cannot access the web UI, I see the Blocked IP address (security.ip-blocked) lines in my log indicating that Stalwart blocks my traefik internal IP, then if I add the network to server.proxy.trusted-networks I get the invalid proxy header error (no matter if http.use-x-forwarded is enabled). I tested traefik with whoami:

Hostname: bd9fae9b0f3c
IP: 127.0.0.1
IP: ::1
IP: 172.18.0.14
RemoteAddr: 172.18.0.9:33396
GET / HTTP/1.1
Host: [redacted]
User-Agent: curl/8.16.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 146.83.128.188
X-Forwarded-Host: [redacted]
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: c5e92de9af08
X-Real-Ip: 146.83.128.188

The X-Forwarded headers are correct, so it's not a traefik issue. This looks pretty much like a bug.

[Megver83]

I had to change my traefik ip to unblock it, now I see:

2025-10-17T19:17:55Z INFO Authentication successful (auth.success) listenerId = "http", localPort = 8080, remoteIp = 172.18.0.15, remotePort = 48306, accountName = "admin"

is this expected with http.use-x-forwarded? to have remoteIp = traefik ip? which is my case