[fix] Don't evaluate threshold for historical data #666

[pandafy]

Checklist

• I have read the OpenWISP Contributing Guidelines.
• I have manually tested the changes proposed in this pull request.
• I have written new test cases for new code and/or updated existing tests for changes to existing code.
• I have updated the documentation.

Reference to Existing Issue

Fixes #666

[pandafy]

I have chosen 1 hour arbitrarily. We can change this if needed.

[nemesifier]

I argue that anything older than 5 minutes shouldn't trigger the check of the threshold, because it could come after more recent data has already been ingested and could conflict with recent data, hence generating inconsistent results.

[nemesifier]

this shouldn't happen right? If we flag the method as private I think we can save these 2 lines of code as this method is going to be used only internally.

[nemesifier]

I argue that anything older than 5 minutes shouldn't trigger the check of the threshold, because it could come after more recent data has already been ingested and could conflict with recent data, hence generating inconsistent results.

[nemesifier]

Changes discussed:

1. We could move the logic which check whether we're dealing with fresh or old data before the metric is being written, so that if it's historical data we could pass check=False, which is already supported
2. Some tests are writing old data and expecting notifications to be created, we'll have to update these tests, in some cases we could use freeze_time to write past data if needed.
3. Let's update this doc page to mention this behavior:
   /monitoring/user/device-checks-and-alert-settings.rst, we can add a new section here and also reference the
   send mode subsection in /openwrt-monitoring-agent/user/settings to explain that we support writing old data which couldn't be collected due to network outages, but when we write this data we don't check threshold nor change the health status (only fresh data will do this).