Are individual connection metrics exposed via Prometheus?

[daveofthedogs]

I was wondering if anybody knows of a way to replicate the old connections metric page vis prometheus? What I am looking for is the username, connected at, and the peer certificate subject. I have been looking at the RMQ Prom page https://www.rabbitmq.com/docs/prometheus#connectionchannel-metrics and attempting to pull the prometheus output and searhcing through it, but cannot ffind anything.

[michaelklishin]

This is not a metric that's exposed by the Prometheus scraping endpoint, including /metrics/per-object.

In part this is because Prometheus focusses on aggregated metrics as they are easy to visualize, in part because this is not a commonly asked for set of metrics.

I'm not sure what you mean by the "old" connections page. There haven't been almost any changes to
the management UI, even in the upcoming 4.0 release. For example, the same page looks like so
in current main:

[xDaevax]

I was wondering the same thing, as I add additional client details. Having the ability to return the client properties and connection information would enable me to do filtering in Grafana so I could see which clients / connections were having problems. This would especially be helpful for stream connections.

[mkuratczyk]

I'd rather avoid adding every possible detail to the Prometheus metrics. However, something we did discuss and I still believe could be a good idea is to get rid of the channel label and instead add a connection_name label (or just name or whatever). For example, we have:

rabbitmq_connection_incoming_bytes_total{channel="<0.1247.0>"} 588
rabbitmq_connection_outgoing_bytes_total{channel="<0.1247.0>"} 679
rabbitmq_connection_process_reductions_total{channel="<0.1247.0>"} 18195
rabbitmq_connection_incoming_packets_total{channel="<0.1247.0>"} 9
rabbitmq_connection_outgoing_packets_total{channel="<0.1247.0>"} 9

Channel PID is ephemeral and while it could be correlated with other connection details by parsing the logs and/or through the Management API (while the connection is open), this is a high cardinality value that is not useful on its own and changes whenever the client reconnects. I believe having the client-provided connection name as a label instead, would make it clear what connection this is (if your connection names are not descriptive, you can make them so) and would allow monitoring a given client even when it reconnects. The main drawback I see is that there could be multiple connections with the same name and that correlating with logs and/or Management UI based on an unstructured name could be harder than by an Erlang pid.

[ansd]

Exposing the username in Prometheus is an anti pattern described in every Prometheus 101.
If you need the username, you should use a logs based monitoring tool.

[tiagoefmoraes]

Channel PID is ephemeral and while it could be correlated with other connection details by parsing the logs and/or through the Management API (while the connection is open), this is a high cardinality value that is not useful on its own and changes whenever the client reconnects. I believe having the client-provided connection name as a label instead, would make it clear what connection this is (if your connection names are not descriptive, you can make them so) and would allow monitoring a given client even when it reconnects. The main drawback I see is that there could be multiple connections with the same name and that correlating with logs and/or Management UI based on an unstructured name could be harder than by an Erlang pid.

I'm troubleshooting some connection issues and I can't correlate Prometheus channel PIDs with anything in the logs or in the Management UI because they are not logged or visible in the UI.

Here's what I would find usefull in order of importance:

• Having the connection PID in Prometheus
  • are present in the logs and are searchable
  • they change with each connection making reconnections color changes standout in charts
  • we can identify the first accepting AMQP connection log
• Having the connection Client Name in Prometheus
  • are present in the logs and are searchable
  • they DON'T change with each connection allowing us to see all connections as a single consumer
  • we CAN'T identify the first accepting AMQP connection log
• Having the connection PID and channel PID in the Management UI the respective pages.
  • Helps correlate the logs and Prometheus PIDs with the UI
• Having the connection "UI name" (192.168.65.1:18524 -> 172.19.0.6:5672) in Prometheus
  • Similar to connection PID, but allows us to correlate with the UI too
  • Maybe it's an anti-pattern like exposing the username

accepting AMQP connection <0.8277.0> (192.168.65.1:18524 -> 172.19.0.6:5672)
connection <0.8277.0> (192.168.65.1:18524 -> 172.19.0.6:5672) has a client-provided name: ClientName
connection <0.8277.0> (192.168.65.1:18524 -> 172.19.0.6:5672 - ClientName): user 'user' authenticated and granted access to vhost 'vhost'
closing AMQP connection <0.8277.0> (192.168.65.1:18524 -> 172.19.0.6:5672 - ClientName, vhost: 'vhost', user: 'user')

accepting AMQP connection <0.8295.0> (192.168.65.1:37284 -> 172.19.0.6:5672)
connection <0.8295.0> (192.168.65.1:37284 -> 172.19.0.6:5672): user 'user' authenticated and granted access to vhost 'vhost'
closing AMQP connection <0.8295.0> (192.168.65.1:37284 -> 172.19.0.6:5672, vhost: 'vhost', user: 'user')

[mkuratczyk]

Thanks for sharing. I still feel the connection names are the best option.

are present in the logs and are searchable

Yup, valuable.

they DON'T change with each connection allowing us to see all connections as a single consumer

I'd say it's generally a good thing. Some of the metrics are counters, I'm pretty sure you can highlight counter resets if you want to or you can use logs to add annotations on your graph to highlight when a connection was re-established, etc.

we CAN'T identify the first accepting AMQP connection log

Why not? The log message with has a client-provided name has the same PID as the accepting AMQP connection line just above (and in general, they will almost always appear next very close to each other).