Skip to content

Add alerts for low available swap space #1075

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 31, 2024
Merged

Add alerts for low available swap space #1075

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8a64c5c
Add alerts for low available swap space
seunghun1ee May 16, 2024
643aa78
Add releasenote for swap space monitoring
seunghun1ee May 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions etc/kayobe/kolla/config/prometheus/system.rules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,24 @@ groups:
summary: "Prometheus exporter at {{ $labels.instance }} reports low memory"
description: "Available memory is {{ $value }} GiB."

- alert: LowSwapSpace
expr: (node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes) < {% endraw %}{{ alertmanager_node_free_swap_warning_threshold_ratio }}{% raw %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens when there is no swap, i.e. node_memory_SwapTotal_bytes is 0?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's good point. It seems like either adding conditional templating or updating expr for these are needed. @markgoddard How should I do it? Do you want me to revert this merge and re-do the PR?

Copy link
Contributor

@markgoddard markgoddard Jun 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It depends on how long it will take for you to test it out and provide a fix. If it can be done quickly then no need to revert.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case I think we should revert this change. I'm working on customer's system this week.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The divide by zero evaluates to NaN, and the alert appears as 'OK'. Context: https://www.robustperception.io/get-thee-to-a-nannary/

Copy link
Member

@dougszumski dougszumski Jun 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Example from a no swap test env:
image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I think this alert is fine as it is, but a good question to ask none-the-less.

for: 1m
labels:
severity: warning
annotations:
summary: "Swap space at {{ $labels.instance }} reports low memory"
description: "Available swap space is {{ $value | humanizePercentage }}. Running out of swap space causes OOM Kills."

- alert: LowSwapSpace
expr: (node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes) < {% endraw %}{{ alertmanager_node_free_swap_critical_threshold_ratio }}{% raw %}
for: 1m
labels:
severity: critical
annotations:
summary: "Swap space at {{ $labels.instance }} reports low memory"
description: "Available swap space is {{ $value | humanizePercentage }}. Running out of swap space causes OOM Kills."

- alert: HostOomKillDetected
expr: increase(node_vmstat_oom_kill[5m]) > 0
for: 5m
Expand Down
6 changes: 6 additions & 0 deletions etc/kayobe/stackhpc-monitoring.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@ alertmanager_low_memory_threshold_gib: 5
# link. Change to false to disable this alert.
alertmanager_warn_network_bond_single_link: true

# Threshold to trigger an LowSwapSpace alert on swap space depletion (ratio).
# When the ratio of free swap space is lower than each of these values, warning
# and critical alerts will be triggered respectively.
alertmanager_node_free_swap_warning_threshold_ratio: 0.25
alertmanager_node_free_swap_critical_threshold_ratio: 0.1

###############################################################################
# Exporter configuration

Expand Down
13 changes: 13 additions & 0 deletions releasenotes/notes/add-alerts-for-swap-availability-75e28ed7f913d1ec.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
features:
- |
Added two alerts (Warning and critical) that are triggered when the ratio
of (free_swap_sppace / total_swap_space) is below thresholds.
Each threshold can be modified by alterting value of
``alertmanager_node_free_swap_warning_threshold_ratio`` and
``alertmanager_node_free_swap_critical_threshold_ratio``.

Currently this solution has limitation of having one-size fits all policy.
This can cause unwanted alerts for the hosts which utilise swap heavily
Therefore it is recommended to tune the thresholds or apply silence rules
for the needs.
Loading