Fix collector ingest instance binding#26572
Draft
thll wants to merge 19 commits into
Draft
Conversation
…t-instance-binding
When enabled, the logs trace a rotation end to end: the rotation write, the event-driven cache refresh, the slot and validity stamp of each loaded binding, per-connection binding verification at the TLS handshake, and prewarm coverage at startup. The per-request rejection warning now includes the remote address and fingerprint so a post-grace cut can be correlated to its rotation. Steady-state ingest logs nothing, so silence confirms per-request lookups stay cache hits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends mTLS checks on the ingest path to verifying that a presented client certificate binds to an enrolled collector instance.
Previously, client certs were verified cryptographically, but it was not checked if they bind to an enrolled instance. This PR addresses that gap by checking if the fingerprint of the presented cert matches any of the certificate fingerprints in the
collector_instancescollection.The check is performed during the TLS handshake, as well as for every message batch received on an already established connection. Since TLS never re-validates an established connection, the per-batch check now cuts a connection when a certificate has been revoked or has expired after the handshake.
Checking if a cert belongs to an enrolled collector requires a MongoDB lookup. Extra care has been taken to keep MongoDB I/O out of the hot path by caching certificate fingerprint bindings. The core mechanics are:
refreshAfterWrite) as a self-healing bound for missed or failed refreshes.idle_writer_timeoutis no longer user-configurable and is forced to 60s, which was the default for that setting.With the binding check in place, a regular certificate renewal would cut the established ingest connection: when the OpAMP control channel adopts the renewed certificate, the server promotes
nexttoactiveand the superseded certificate stops binding. However, the collector's ingest exporter is still using it, because it only picks up the new certificate via an asynchronous restart. Every renewal would drop an in-flight log batch.This issue is addressed by introducing a new
previousslot which holds the previouslyactivecert after a renewal.After a renewal, the ingest path additionally accepts the superseded (
previous) certificate for a configurable grace period (collector_cert_rotation_grace_period, default 5 minutes). Deletion and re-enrollment still cut access promptly.