rabbitmq metric is not showing in prometheus

[MohdRashid01]

Hi All,

My question i want to add rabbitmq monitoring in prometheus. I already have rabbitmq running in Kubernetes but i dont know how to add rabbitmq metric in prometheus

I have install promethues and grafana through yaml file along with pv,pvc,storage,svc,config,deploy and cluster-role

Here is the screenshot of rabbitmq showing empty in promethues

have install Kubernetes in one vm with local storage i.e.., control-plane and node both install in one vm and everything is working fine

Here is my prometheus-config yaml file

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    rule_files:
      - /etc/prometheus/prometheus.rules
    alerting:
      alertmanagers:
      - scheme: http
        static_configs:
        - targets:
          - "alertmanager.monitoring.svc:9093"

    scrape_configs:
      - job_name: 'kubernetes-apiservers'

        kubernetes_sd_configs:
        - role: endpoints
        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
          action: keep
          regex: default;kubernetes;https

      - job_name: 'kubernetes-nodes'

        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics

      
      - job_name: 'kubernetes-pods'

        kubernetes_sd_configs:
        - role: pod

        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name
      
      - job_name: 'kube-state-metrics'
        static_configs:
          - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']

      - job_name: 'kubernetes-cadvisor'

        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
      
      - job_name: 'kubernetes-service-endpoints'

        kubernetes_sd_configs:
        - role: endpoints

        relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name
      
      - job_name: 'rabbitmq'
        metrics_path: /metrics
        scrape_interval: 5s
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
              - default

        relabel_configs:
          - source_labels: [__meta_kubernetes_service_label_app]
            separator: ;
            regex: rabbitmq
            replacement: $1
            action: keep
          - source_labels: [__meta_kubernetes_endpoint_port_name]
            separator: ;
            regex: prometheus

Even after enable rabbitmq_prometheus(plugin) metric it is not showing in prometheus url (target)

here is my rabbitmq-svc yaml file

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: xxx-rabbitmq
spec:
  selector:
    matchLabels:
      app: xxx-rabbitmq
  serviceName: xxx-rabbitmq
  replicas: 1
  template:
    metadata:
      labels:
        app: xxx-rabbitmq
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: xxx-rabbitmq
        image: rabbitmq:3.7.3-management
        ports:
        - containerPort: xxx
        - containerPort: xxx
        - containerPort: xxx
        - containerPort: xxx        
        volumeMounts:
        - name: xxx-rabbitmq-pvc
          mountPath: /var/lib/rabbitmq
          subPath: rabbitmq
        envFrom:
        - configMapRef:
            name: rabbitmq-config

  volumeClaimTemplates:
    - metadata:
        name: xxx-rabbitmq-pvc
      spec:
        accessModes: ["ReadWriteOnce"]
        volumeMode: Filesystem
        storageClassName: xxxx-storage
        resources:
          requests:
           storage: 10Gi  
---
apiVersion: v1
kind: Service
metadata:
  name: xxx-rabbitmq
  labels:
    app: xxx-rabbitmq
spec:
  type: ClusterIP
  ports: 
  - port: xxx
    name: main
  - port: xxx
    name: rabbitmqssl
  - port: xxx
    name: rabvitmqmgmt
  selector:
    app: xxx-rabbitmq

Please help me from 4 days im not able to get metric and what file do i need to change in prometheus or in rabbitmq-data.yaml or config.yaml file

Even i did curl -s localhost:15692/metrics | head -n 3 (got below respone)
root@rabbitmq-1:/# curl -s localhost:15692/metrics | head -n 3 # TYPE erlang_mnesia_held_locks gauge # HELP erlang_mnesia_held_locks Number of held locks. erlang_mnesia_held_locks 0

But still metric is not showing in prometheus

[michaelklishin]

According to the response from localhost:15692/metrics you do have metrics served locally, so either

• Other pods or external services cannot access that port
• Your Grafana instance is pointed at the wrong endpoint

[MohdRashid01]

@michaelklishin so what should i do now please explain me in clear im new to rabbitmq and prometheus

My VM is running in hetzner and also control-plane and node both are in 1 VM running and also i have LB which running my application

root@rabbitmq-0:/# rabbitmq-diagnostics -q cluster_status
Basics

Cluster name: [email protected]

Disk Nodes

rabbit@rabbitmq-0

Running Nodes

rabbit@rabbitmq-0

Versions

rabbit@rabbitmq-0: RabbitMQ 3.8.34 on Erlang 24.3.4.1

Maintenance status

Node: rabbit@rabbitmq-0, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@rabbitmq-0, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmq-0, interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Node: rabbit@rabbitmq-0, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-0, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: drop_unroutable_metric, state: disabled
Flag: empty_basic_get_metric, state: disabled
Flag: implicit_default_bindings, state: disabled
Flag: maintenance_mode_status, state: disabled
Flag: quorum_queue, state: disabled
Flag: user_limits, state: disabled
Flag: virtual_host_metadata, state: disabled
root@fx-rabbitmq-0:/# exit

root@fx-rabbitmq-0:/# curl http://PUBLIC-IP:15692/metrics
curl: (7) Failed to connect to PUBLIC-IP port 15692: Connection refused

[michaelklishin]

First of all, RabbitMQ 3.8 has reached end of life.

The Prometheus plugin is enabled and listens on all interfaces. However, you cannot
access the port via node's public IP address. "Connection refused" means curl failed to establish
a TCP connection. So you need to expose the port to external connections, here is one way of doing it.

[michaelklishin]

Here is an example Prometheus configuration used by the RabbitMQ Kubernetes Cluster Operator.

Perhaps you should simply use the Cluster Operator instead of rolling your own deployment.

[MohdRashid01]

ok got it i will try it
and i will update u