Skip to content

Fix collector ingest instance binding#26572

Draft
thll wants to merge 19 commits into
masterfrom
fix/collector-ingest-instance-binding
Draft

Fix collector ingest instance binding#26572
thll wants to merge 19 commits into
masterfrom
fix/collector-ingest-instance-binding

Conversation

@thll

@thll thll commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Extends mTLS checks on the ingest path to verifying that a presented client certificate binds to an enrolled collector instance.

Previously, client certs were verified cryptographically, but it was not checked if they bind to an enrolled instance. This PR addresses that gap by checking if the fingerprint of the presented cert matches any of the certificate fingerprints in the collector_instances collection.
The check is performed during the TLS handshake, as well as for every message batch received on an already established connection. Since TLS never re-validates an established connection, the per-batch check now cuts a connection when a certificate has been revoked or has expired after the handshake.

Checking if a cert belongs to an enrolled collector requires a MongoDB lookup. Extra care has been taken to keep MongoDB I/O out of the hot path by caching certificate fingerprint bindings. The core mechanics are:

  • Prewarming the cache with active certificate fingerprints on server startup
  • Relying on background refreshes for cache entries, driven by cluster events and complemented by periodic revalidation (refreshAfterWrite) as a self-healing bound for missed or failed refreshes.
  • Relying on the TLS handshake to re-populate evicted fingerprints.
  • Moving cert lookups during the TLS handshake off the Netty event loop into a separate thread pool, and cache refreshes into another.
  • Setting the cache expiry > connection idle timeout. To guarantee this, the ingest input's idle_writer_timeout is no longer user-configurable and is forced to 60s, which was the default for that setting.

With the binding check in place, a regular certificate renewal would cut the established ingest connection: when the OpAMP control channel adopts the renewed certificate, the server promotes next to active and the superseded certificate stops binding. However, the collector's ingest exporter is still using it, because it only picks up the new certificate via an asynchronous restart. Every renewal would drop an in-flight log batch.

This issue is addressed by introducing a new previous slot which holds the previously active cert after a renewal.

After a renewal, the ingest path additionally accepts the superseded (previous) certificate for a configurable grace period (collector_cert_rotation_grace_period, default 5 minutes). Deletion and re-enrollment still cut access promptly.

thll and others added 18 commits June 30, 2026 15:02
When enabled, the logs trace a rotation end to end: the rotation write,
the event-driven cache refresh, the slot and validity stamp of each
loaded binding, per-connection binding verification at the TLS
handshake, and prewarm coverage at startup. The per-request rejection
warning now includes the remote address and fingerprint so a
post-grace cut can be correlated to its rotation. Steady-state ingest
logs nothing, so silence confirms per-request lookups stay cache hits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@thll thll added the collector label Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants