Add alerts for low available swap space #1075
Conversation
There was a problem hiding this comment.
Thanks @seunghun1ee, good job.
Please could you add a release note, and mention that we currently have a one-size fits all policy as an interim solution. We can advise that operators may need to tune the threshold for the alerts.
|
Renders fine: |
seunghun1ee
force-pushed
the
monitor-swap-usage
branch
from
50fc3d1 to
a7562bf
Compare
Thank you @dougszumski |
seunghun1ee
force-pushed
the
monitor-swap-usage
branch
from
a7562bf to
643aa78
Compare
| @@ -24,6 +24,24 @@ groups: | |||
| summary: "Prometheus exporter at {{ $labels.instance }} reports low memory" | |||
| description: "Available memory is {{ $value }} GiB." | |||
|
|
|||
| - alert: LowSwapSpace | |||
| expr: (node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes) < {% endraw %}{{ alertmanager_node_free_swap_warning_threshold_ratio }}{% raw %} | |||
There was a problem hiding this comment.
What happens when there is no swap, i.e. node_memory_SwapTotal_bytes is 0?
There was a problem hiding this comment.
That's good point. It seems like either adding conditional templating or updating expr for these are needed. @markgoddard How should I do it? Do you want me to revert this merge and re-do the PR?
There was a problem hiding this comment.
It depends on how long it will take for you to test it out and provide a fix. If it can be done quickly then no need to revert.
There was a problem hiding this comment.
In that case I think we should revert this change. I'm working on customer's system this week.
There was a problem hiding this comment.
The divide by zero evaluates to NaN, and the alert appears as 'OK'. Context: https://www.robustperception.io/get-thee-to-a-nannary/
There was a problem hiding this comment.
Example from a no swap test env:
There was a problem hiding this comment.
So I think this alert is fine as it is, but a good question to ask none-the-less.
As exhausting memory on hypervisor causes OOM kills, alerts for low available swap space are added.
Storage nodes usually utilise all of the swap spaces, so this alert can be spammed with them.
Therefore, silencing rules for storage nodes are going to be necessary.
We can implement per group alerting by overriding prometheus.yml.j2 from Kolla-ansible to avoid silencing approach but that needs separate discussion.