[Workaround] Improve stability under memory pressure; offloading BADBEHAVIOR enforcement to Fail2Ban

[rrrevin]

(Apologies in advance for the length of this. Also note: I use valkey, others use redis. Change as you like)

I’m posting this as an operational stability note / workaround related to the known memory behavior discussed in the linked issue below, rather than as a new bug report.

Issue reference: #3057

I want to start by saying: I love BunkerWeb. I love how it operates, the overall architecture is thoughtful, and full kudos to the developer(s) for building it. I’m posting this as an operational stability note rather than a traditional bug report.

Right now, I’m seeing memory pressure that can turn into reliability problems under load. Until the underlying memory behavior is resolved, I need stability over statistics.

Why I’m making this trade-off

BunkerWeb does a great job of recording rich per-event context (IP, UA, URL, route, country, etc.). At the time these data schema choices were introduced, representing these records in a straightforward way likely made sense. But once you introduce real-world scale, high request volume, hostile traffic, repeated scanners, memory pressure becomes an inevitability. Persistent per-event history and report generation can translate into large Valkey datasets and large transient memory spikes in the UI/backend. The result can be operational instability, timeouts, unresponsive hosts, and forced restarts, especially when reports are loaded and large structures get parsed, grouped, and sorted in memory.

From an operational standpoint: once something is blocked, it needs to be blocked, and I don’t need the system to continually spend memory and CPU recording the same offender repeatedly.

Yes, it’s cool to know an IP hit me 1,000 times, but I don’t need that to operate the system safely. What I need is the first event to trigger the enforcement decision, enforcement to be immediate and durable, and the platform to remain stable under repeat attempts.

The trade-offs I’m choosing

Once an IP is blocked, it becomes effectively invisible at the application layer until the ban expires. I lose per-request statistics such as URLs, UAs, per-service context, and the ever-increasing counters that come from repeat attempts during the ban window. In exchange, repeated attempts do not keep generating ever-growing datasets and report load.

There is also an important scope trade-off. This Fail2Ban enforcement is a global service ban, not a per-service ban. If one site triggers the ban decision, Fail2Ban blocks that IP for all services on the host. I am explicitly OK with that. Think of it like local crowdsourcing. If one site identifies abusive behavior, it blocks it for all sites. For my operational goals, that is an acceptable and desirable outcome.

This is an explicit choice: stability over statistics.

Approach: BunkerWeb decides, Fail2Ban enforces

The stepping-stone integration is:

1. BunkerWeb remains the decision-maker and emits the [BADBEHAVIOR] IP x.x.x.x is banned for ... line when thresholds are exceeded.
2. Fail2Ban becomes the enforcement layer and reads that log line and applies the ban at the firewall level. That means repeated requests do not keep hammering BunkerWeb, Valkey, and UI reporting structures.
3. To keep behavior sane and predictable, keep the ban times aligned between BunkerWeb and Fail2Ban (for example 14d, 1209600s). The intent is that you see it once, it gets enforced, and you do not keep accumulating log-driven state for the same offender.

Observed impact (before/after firewall enforcement)

Below is a time-series view of host memory usage that shows the practical effect of using Fail2Ban as a firewall-level enforcement layer in front of BunkerWeb.

Before ~Feb 5 (~17:00): memory usage would climb quickly over a few days, with repeated spikes that reached the full 16 GiB of available RAM. At that point the server could become unresponsive to the extent that remote access was no longer possible, and recovery required a hard power cycle to regain control.

After enabling Fail2Ban enforcement: memory usage stabilized dramatically. In this same workload profile, resident usage has rarely exceeded ~4 GiB.

The key operational difference is where repeat traffic is stopped. Once BunkerWeb emits a BADBEHAVIOR ban line, Fail2Ban applies a firewall ban and repeated attempts are dropped at the kernel firewall layer. That has two compounding effects:

1. State and reporting pressure collapses: BunkerWeb + Valkey + the UI no longer need to repeatedly ingest, store, and re-materialize increasingly large per-event and per-IP structures for traffic that is already determined to be abusive and destined to be blocked anyway. This reduces both persistent dataset growth and transient “report load” memory spikes.

2. Handshake/CPU pressure collapses: because banned sources are blocked at the firewall, the host avoids spending cycles on new TLS handshakes (and associated connection setup/teardown work) for IPs that have already been classified as abusive. In practice, this preserves CPU time and memory bandwidth for what BunkerWeb should be doing: routing legitimate traffic, applying security logic to new requests, and maintaining counters only until a threshold is crossed.

I recognize this doesn’t replace an underlying optimization pass (e.g., reducing the in-memory footprint of reporting pipelines and/or optimizing the Valkey/Redis data model and retention strategy). It is, however, a stability-first mitigation that has been highly effective for my environment. Your traffic profile and tolerance for global bans may differ, but for anyone seeing the same “memory growth → UI/report load → host instability” pattern, I wanted to share a configuration that made the system stable and functional.

Memory usage before/after enabling Fail2Ban firewall enforcement for BADBEHAVIOR bans]

The next level of this integration will honor the ban time emitted by BunkerWeb instead of relying on Fail2Ban’s static bantime. Also, if you really want post-ban statistics, the kernel firewall can provide counters (iptables or nftables). With some integration, those counters can be surfaced back into BunkerWeb. You will not get URL-level detail, but you can get number of attempts while banned. I'm also working on extracting the CIDR block the IP is registered in and adding a CIDR level block as an option.

---

Stepping-stone runbook: Fail2Ban enforcement for BunkerWeb BADBEHAVIOR (bwbb)

Phase 0 - Preconditions

• BunkerWeb is writing its logs to a file Fail2Ban can read (example: /var/log/bunkerweb/error.log).
• The ban emission line looks like:
  [BADBEHAVIOR] IP <ip> is banned for ...

Example:

2026/01/29 10:40:45 [warn] ... [BADBEHAVIOR] IP 47.128.121.88 is banned for 1209600s (7/5) on server example.com with scope global, context: ngx.timer

Goal: BunkerWeb decides; Fail2Ban enforces.

---

Phase 1 - Install Fail2Ban

Debian / Ubuntu

sudo apt update
sudo apt install -y fail2ban
sudo systemctl enable --now fail2ban
sudo fail2ban-client ping

RHEL / Rocky / Alma / CentOS

sudo dnf install -y fail2ban
sudo systemctl enable --now fail2ban
sudo fail2ban-client ping

---

Phase 1a - Disable the default sshd jail (optional)

On some distros, installing Fail2Ban enables an sshd jail by default (you will see Jail 'sshd' started). If you do not want Fail2Ban managing SSH bans, disable it via a local override:

sudo tee /etc/fail2ban/jail.d/sshd.local >/dev/null <<'EOF'
[sshd]
enabled = false
EOF

sudo systemctl restart fail2ban

Confirm it is gone:

sudo fail2ban-client status

---

Phase 2 - Create the filter: bwbb (bunkerweb-badbehavior)

Create:
/etc/fail2ban/filter.d/bwbb.conf

[Definition]
# Match the ban emission line and extract the offending IP.
# Stable spine:
#   [BADBEHAVIOR] IP <HOST> is banned for ...
failregex = ^.*\[BADBEHAVIOR\]\s+IP\s+<HOST>\s+is\s+banned\s+for\s.*$
ignoreregex =

Validate:

sudo fail2ban-regex /var/log/bunkerweb/error.log /etc/fail2ban/filter.d/bwbb.conf

If you get 0 matches:

• wrong logpath, or
• the log line differs (adjust tokens in failregex).

---

Phase 3 - Create the jail: bwbb (BunkerWeb bad behavior)

Create:
/etc/fail2ban/jail.d/bwbb.local

[bwbb]
enabled  = true
filter   = bwbb
logpath  = /var/log/bunkerweb/error.log
backend  = auto

# BunkerWeb already decided the request/IP crossed a threshold, so one hit is enough.
maxretry = 1
findtime = 1m

# Fail2Ban is the enforcement timer source of truth.
# Align to BunkerWeb ban_time if you want consistent behavior.
bantime  = 14d

# Make web ports explicit (prevents inheriting SSH-oriented defaults).
port     = 80,443
protocol = tcp

# Select ONE of the enforcement stacks below, or enable both if you explicitly want dual-stack enforcement.
# nftables-only:
# action   = nftables[type=multiport]
#
# iptables+ipset-only:
# action   = iptables-ipset-proto4[name=bwbb, port="80,443", protocol=tcp]
#
# dual enforcement (nftables + iptables/ipset):
# action   = nftables[type=multiport]
#            iptables-ipset-proto4[name=bwbb, port="80,443", protocol=tcp]

# Default: stick with the distro default action unless you are explicitly switching.
action   = nftables[type=multiport]

Notes on choosing actions

• If your host is already standardized on nftables, keep nftables as the primary enforcement.
• If you need broad compatibility or are on legacy systems, use iptables+ipset.
• If you want “belt and suspenders,” use both actions (it’s redundant but functional).

---

Phase 3a - Add counters to nftables (per-element)

You already have this conceptually correct. The one key requirement for “per-element counters” is defining the set with counter;.

Create:
/etc/fail2ban/action.d/nftables-common.local

[Definition]
# Recreate the set with per-element counters so each IP has its own packets/bytes.
_nft_add_set = <nftables> add set <table_family> <table> <addr_set> \{ type <addr_type>\; counter\; \}
              <_nft_for_proto-<type>-iter>
              <nftables> add rule <table_family> <table> <chain> %(rule_stat)s
              <_nft_for_proto-<type>-done>

This preserves your “kernel provides counters” strategy:

• The set tracks per-element packet/byte counters.
• The rule can also have a counter, but you’re primarily interested in element counters.

---

Phase 3b - Add counters to iptables (per-element) using ipset

iptables itself has rule counters, but per-element counters require ipset, and the set must be created with counters.

Option A: Use the distro’s ipset-based action (preferred)

Many distros ship an action like iptables-ipset-proto4 / iptables-ipset that uses ipset behind the scenes. When it creates the set, it may not enable element counters by default.

To force element counters reliably, define a dedicated ipset-based action that creates the set with counters.

Create:
/etc/fail2ban/action.d/iptables-ipset-bwbb.local

[Definition]
# Create a CIDR-capable ipset with per-element counters (works for single IPs too).
actionstart = ipset create bwbb hash:net family inet counters hashsize 4096 maxelem 200000 -exist
              iptables -C INPUT -m set --match-set bwbb src -j DROP 2>/dev/null || iptables -I INPUT 1 -m set --match-set bwbb src -j DROP
actionstop  =
actioncheck =

# Fail2Ban passes <ip> for bans; we store it as a /32 network element.
actionban   = ipset add bwbb <ip>/32 -exist
actionunban = ipset del bwbb <ip>/32 2>/dev/null || true

Then, in your bwbb jail, you can use:

action = iptables-ipset-bwbb

Or dual enforce:

action = nftables[type=multiport]
         iptables-ipset-bwbb

Why ipset is the correct mechanism here

• You get per-element counters (bytes/packets per banned element)
• You don’t accumulate one iptables rule per ban (which becomes unscalable)
• It remains compatible with both single-IP and later CIDR additions

---

Phase 4 - Restart Fail2Ban and confirm the jail is live

sudo systemctl restart fail2ban
sudo fail2ban-client status
sudo fail2ban-client status bwbb

---

Phase 5 - Functional test

1. Trigger a BunkerWeb BADBEHAVIOR ban event.
2. Confirm Fail2Ban recorded the ban:

sudo fail2ban-client get bwbb banip

Optional enforcement inspection:

nftables

sudo nft list ruleset | head -n 160

Check per-element counters (the set used by Fail2Ban’s nftables action):

sudo nft list set inet <table> <addr_set>

iptables + ipset

sudo iptables -L -n | head -n 80
sudo ipset list bwbb

With ipset element counters enabled, ipset list bwbb will show packet/byte counts per element.

---

Reset to steady state (stop, clear Valkey, start)

WARNING: This wipes Valkey data. Use only if you intend to clear BunkerWeb state.

Stop services:

sudo systemctl stop bunkerweb bunkerweb-scheduler bunkerweb-ui crowdsec

Clear Valkey (choose one):

valkey-cli FLUSHDB
# or
valkey-cli FLUSHALL

Verify empty (optional):

valkey-cli DBSIZE
valkey-cli INFO KEYSPACE

Start services (Valkey first):

sudo systemctl restart valkey
sudo systemctl start bunkerweb bunkerweb-scheduler bunkerweb-ui crowdsec

---

Minimal rollback (remove Fail2Ban integration files)

sudo rm -f /etc/fail2ban/jail.d/bwbb.local
sudo rm -f /etc/fail2ban/filter.d/bwbb.conf
sudo rm -f /etc/fail2ban/action.d/nftables-common.local
sudo rm -f /etc/fail2ban/action.d/iptables-ipset-bwbb.local
sudo systemctl restart fail2ban