Stalwart + Traefik 502 Bad Gateway error=EOF

[dalmatheo]

Hello everyone, I've been trying to setup stalwart on my NAS lately, and trying to connect it to my Traefik network. With the configuration files that I'm going to show, everything worked fine for a moment, until it doesn't and I become unable to connect to the Admin page. I don't really understand what is going on here, and I really need help. I tried to follow what was written here because it apparently come from someone that had a similar issue, without success.

Here are my different configuration files:

docker-compose.yml (Traefik)

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    networks:
      bridge:
      proxy:
        ipv4_address: 172.19.0.2
    environment:
      - CLOUDFLARE_DNS_API_TOKEN=TOKEN
      - CLOUDFLARE_EMAIL=EMAIL
    labels:
      - traefik.enable=true
      - traefik.http.routers.traefik-secure.rule=Host(`traefik.mydomain.com`)
      - traefik.http.routers.traefik-secure.entrypoints=https
      - traefik.http.routers.traefik-secure.service=api@internal
      - traefik.http.routers.traefik-secure.middlewares=traefik-auth
      - traefik.http.middlewares.traefik-auth.basicauth.users=dalmatheo:password
    ports:
      - 80:80
      - 443:443
      - 25:25
      - 465:465
      - 993:993
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/etc/traefik
      - acme:/etc/certs
      - /var/run/docker.sock:/var/run/docker.sock:ro

  traefik-certs-dumper:
    image: ghcr.io/kereis/traefik-certs-dumper:latest
    container_name: traefik-certs-dumper
    restart: unless-stopped
    depends_on:
      - traefik
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - acme:/traefik:ro
      - certs:/output


volumes:
  acme:
  certs:
  
networks:
  bridge:
    driver: bridge
  proxy:
    ipam:
      driver: default
      config:
        - subnet: 172.19.0.0/16
          gateway: 172.19.0.1

traefik.yml (Traefik)

global:
  checkNewVersion: true
  sendAnonymousUsage: false

api:
  dashboard: true

log:
  level: DEBUG

certificatesResolvers:
  letsencrypt:
    acme:
      keyType: EC256
      dnsChallenge:
        provider: cloudflare
      email: EMAIL
      storage: /etc/certs/acme.json

entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: "https"
          scheme: "https"
  https:
    address: ":443"
    http:
      tls:
        certResolver: letsencrypt

  smtp:
    address: :25

  smtps:
    address: :465
    proxyProtocol:
      trustedIPs:
        - 172.19.0.2
        - 172.19.0.5

  imaps:
    address: :993
    proxyProtocol:
      trustedIPs:
        - 172.19.0.2
        - 172.19.0.5

tls:
  options:
    default:
      minVersion: VersionTLS12

providers:
  docker:
    network: traefik_bridge
    exposedByDefault: false

docker-compose.yml (Stalwart)

services:
  mailserver:
    image: stalwartlabs/mail-server:latest
    container_name: mailserver
    restart: unless-stopped
    hostname: mail.mydomain.com
    networks:
      proxy:
        ipv4_address: 172.19.0.5
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./data:/opt/stalwart-mail
      - ./private:/opt/stalwart-smtp/etc/private
      - certs:/data/certs:ro
    labels:
      - traefik.enable=true
      - traefik.docker.network=traefik_proxy

      - traefik.http.routers.stalwart.rule=Host(`mail.mydomain.com`)
      - traefik.http.routers.stalwart.entrypoints=https
      - traefik.http.routers.stalwart.tls.certResolver=letsencrypt
      - traefik.http.routers.stalwart.service=stalwart
      - traefik.http.services.stalwart.loadbalancer.server.port=8080

      - traefik.tcp.routers.smtp.rule=HostSNI(`*`)
      - traefik.tcp.routers.smtp.entrypoints=smtp
      - traefik.tcp.routers.smtp.service=smtp
      - traefik.tcp.services.smtp.loadbalancer.server.port=25

      - traefik.tcp.routers.smtps.rule=HostSNI(`*`)
      - traefik.tcp.routers.smtps.tls.passthrough=true
      - traefik.tcp.routers.smtps.entrypoints=smtps
      - traefik.tcp.routers.smtps.service=smtps
      - traefik.tcp.services.smtps.loadbalancer.server.port=465
      - traefik.tcp.services.smtps.loadbalancer.proxyProtocol.version=2

      - traefik.tcp.routers.imaps.rule=HostSNI(`*`)
      - traefik.tcp.routers.imaps.tls.passthrough=true
      - traefik.tcp.routers.imaps.entrypoints=imaps
      - traefik.tcp.routers.imaps.service=imaps
      - traefik.tcp.services.imaps.loadbalancer.server.port=993
      - traefik.tcp.services.imaps.loadbalancer.proxyProtocol.version=2
      
volumes:
  certs:
    name: traefik_certs
    external: true

networks:
  proxy:
    name: traefik_proxy
    external: true

config.toml (Stalwart)

[authentication.fallback-admin]
secret = "PASSWORD"
user = "admin"

[directory.internal]
store = "rocksdb"
type = "internal"

[auth.dkim]
sign = [ { if = "is_local_domain('', sender_domain)", then = "['rsa-' + sender_domain, 'ed25519-' + sender_domain]" }, { else = false } ]
verify = "relaxed"

[report.dkim]
from-name = "'Report Subsystem'"
from-address = "'[email protected]'"
subject = "'DKIM Authentication Failure Report'"
sign = ["rsa", "ed25519"]
send = "1/1d"

[signature."ed25519"]
private-key = "%{file:/opt/stalwart-smtp/etc/private/dkim-ed.key}%"
domain = "mydomain.com"
selector = "ed-default"
headers = ["From", "To", "Date", "Subject", "Message-ID"]
algorithm = "ed25519-sha256"
canonicalization = "relaxed/relaxed"

[signature."rsa"]
private-key = "%{file:/opt/stalwart-smtp/etc/private/dkim-rsa.key}%"
domain = "mydomain.com"
selector = "rsa-default"
headers = ["From", "To", "Date", "Subject", "Message-ID"]
algorithm = "rsa-sha256"
canonicalization = "relaxed/relaxed"

[certificate."default"]
cert = "%{file:/data/certs/mail.mydomain.com/cert.pem}%"
default = true
private-key = "%{file:/data/certs/mail.mydomain.com/key.pem}%"

[lookup."default"]
hostname = "mail.mydomain.com"
[server."http"]
hsts = true
permissive-cors = false
url = "protocol + '://' + key_get('default', 'hostname') + ':' + local_port"
use-x-forwarded = true

[server.listener."http"]
bind = "[::]:8080"
protocol = "http"

[server.listener."https"]
bind = "[::]:443"
protocol = "http"
tls.implicit = true

[server.listener."imaptls"]
bind = "[::]:993"
protocol = "imap"
proxy.override = true
proxy.trusted-networks.0 = "172.19.0.2"
proxy.trusted-networks.1 = "172.19.0.0/16"
tls.implicit = true

[server.listener."smtp"]
bind = "[::]:25"
protocol = "smtp"
proxy.override = false

[server.listener."submissions"]
bind = "[::]:465"
protocol = "smtp"
proxy.override = true
proxy.trusted-networks.0 = "172.19.0.2"
proxy.trusted-networks.1 = "172.19.0.0/16"
tls.implicit = true

[storage]
blob = "rocksdb"
data = "rocksdb"
directory = "internal"
fts = "rocksdb"
lookup = "rocksdb"

[store."rocksdb"]
compression = "lz4"
path = "/opt/stalwart-mail/data"
type = "rocksdb"

[tracer.log]
ansi = false
enable = true
level = "info"
path = "/opt/stalwart-mail/logs"
prefix = "stalwart.log"
rotate = "daily"
type = "log"

Every time I try to connect to the admin panel, I get the error:

DBG github.com/traefik/traefik/v3/pkg/proxy/httputil/proxy.go:113 > 502 Bad Gateway error=EOF

[mdecimus]

I'm not familiar with Traefik but try setting Stalwart's log level to trace and then check the logs to see if the connections are reaching Stalwart.

[RiKaZaRai]

I don't have any errors in the logs, that's the worst part.
I use traefik with coolify

[mdecimus]

In that case your reverse proxy is not sending any requests to Stalwart.

[RiKaZaRai]

I don't understand precisely because I expose port 8080 to traefik, everything is configured correctly but I can't do it.

[bestagi]

same here but with nginx, 502 Bad Gateway

[dalmatheo]

Okay guys, I think I figured it out. I am not kidding, and everything I say here is true. I was getting a usual DDOS attack by a random bulgarian IP, and by removing the logs that became probably too long it fixed the issue. Crazy.

[antonsizikov]

@mdecimus I have same issue with Stalwart and Fast Reverse Proxy. I read logs and found out that Stalwart blocks one IP address and it is 172.18.0.1 - Gateway of docker network. So i guess when Stalwart works behind reverse proxy and after some time someone DDOSing it Stalwart see and blocks wrong IP. I think whitelisting private IP ranges will solve the problem of 502 but not the problem of blocking invalid addresses.

[antonsizikov]

Yeah, I saw a dedicated page in documentation about this - Proxy Protocol 😅

[mdecimus]

If Stalwart is blocking the Docker's internal address it means that the proxy protocol is not working for some reason.

[momardia54]

I had the same issue. For some reason, stalwart blocked my docker internal IP address. Removing it from the blocked list fixed the problem.