Dynamic Hostname Labelling

[tardoe]

When creating labels for Prometheus metrics (or Timescale, Influx etc.), add the machine hostname determined during process start-up. This also supports the dynamic inventory use-case and ensures metrics are uniquely labelled.

[nickbabcock]

Is this different than the existing hostname resolution features: https://github.com/nickbabcock/OhmGraphite?tab=readme-ov-file#hostname-resolution ?

Currently prometheus is the only one that doesn't transmit it. In the prometheus exporter ecosystem, how common is it to transmit the hostname?

[tardoe]

This is specifically to do with this bit in the doc linked:

Prometheus setups are unaffected by name_lookup, as Prometheus automatically creates the instance label when scraping. If renaming is desired, you'll want to change the instance label by either hardcoding the alias in the Prometheus config or using DNS.

Because I'm shipping Prometheus in agent-mode with OhmGraphite to enable remote-write-like functionality, prometheus is just scraping localhost and puts that in as the instance label, not the actual machine's hostname (which could be dynamic).

An opt-in, dedicated machine_name (e.g.) label created by OhmGraphite would be useful.

[Andor]

@tardoe In case of the static configuration scrapping localhost, you might want to just specify label in your prometheus config.
Example:

- job_name: my_localhost
  static_configs:
    - targets: [ localhost:9999 ]
      labels:
        hostname: mylocalhost11

This was all the metrics scraped from your localhost:9999 will have labels instance=localhost:9999 (automatically generated) and hostname=mylocalhost (custom specified by you).

You usually don't want to play around instance label, but rather create a separate label via your service discovery mechanism.

I would like to suggest creating a separate "info"-metric for prometheus which will contain resolved hostname.
Something like:

ohmgraphite_discovered_info{hostname="mylocalhost11"} 1