Skip to content

Commit 624987b

Browse files
nepythondevkapilbansal
authored andcommitted
[timeseries] Add initial support for elasticsearch #99
Closes #99
1 parent 7b2a410 commit 624987b

25 files changed

+1353
-152
lines changed

Dockerfile

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ WORKDIR /opt/openwisp/tests/
1818
ENV NAME=openwisp-monitoring \
1919
PYTHONBUFFERED=1 \
2020
INFLUXDB_HOST=influxdb \
21-
REDIS_HOST=redis
21+
REDIS_HOST=redis \
22+
ELASTICSEARCH_HOST=es01
2223
CMD ["sh", "docker-entrypoint.sh"]
2324
EXPOSE 8000

README.rst

Lines changed: 39 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ Available Features
7474
* Collects and displays `device status <#device-status>`_ information like uptime, RAM status, CPU load averages,
7575
Interface properties and addresses, WiFi interface status and associated clients,
7676
Neighbors information, DHCP Leases, Disk/Flash status
77+
* Collection of monitoring information in a timeseries database (`InfluxDB <https://www.influxdata.com/>`_ and `Elasticsearch <https://www.elastic.co/elasticsearch/>`_ are currently supported)
7778
* Monitoring charts for uptime, packet loss, round trip time (latency), associated wifi clients, interface traffic,
7879
RAM usage, CPU load, flash/disk usage
7980
* Charts can be viewed at resolutions of 1 day, 3 days, a week, a month and a year
@@ -108,6 +109,8 @@ beforehand.
108109
In case you prefer not to use Docker you can `install InfluxDB <https://docs.influxdata.com/influxdb/v1.8/introduction/install/>`_
109110
and Redis from your repositories, but keep in mind that the version packaged by your distribution may be different.
110111

112+
If you wish to use ``Elasticsearch`` for storing and retrieving timeseries data then `install Elasticsearch <https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html>`_.
113+
111114
Install spatialite and sqlite:
112115

113116
.. code-block:: shell
@@ -165,6 +168,20 @@ Follow the setup instructions of `openwisp-controller
165168
'PORT': '8086',
166169
}
167170
171+
In case, you wish to use ``Elasticsearch`` for timeseries data storage and retrieval,
172+
make use of the following settings
173+
174+
.. code-block:: python
175+
176+
TIMESERIES_DATABASE = {
177+
'BACKEND': 'openwisp_monitoring.db.backends.elasticsearch',
178+
'USER': 'openwisp',
179+
'PASSWORD': 'openwisp',
180+
'NAME': 'openwisp2',
181+
'HOST': 'localhost',
182+
'PORT': '9200',
183+
}
184+
168185
``urls.py``:
169186

170187
.. code-block:: python
@@ -461,6 +478,9 @@ This data is only used to assess the recent status of devices, keeping
461478
it for a long time would not add much benefit and would cost a lot more
462479
in terms of disk space.
463480

481+
**Note**: In case you use ``Elasticsearch`` then time shall be taken as integral multiple of a day.
482+
That means the time ``36h0m0s`` shall be interpreted as ``24h0m0s`` (integral multiple of a day).
483+
464484
``OPENWISP_MONITORING_AUTO_PING``
465485
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
466486

@@ -764,18 +784,30 @@ MB (megabytes) instead of GB (Gigabytes) you can use:
764784
"SUM(rx_bytes) / 1000000 AS download FROM {key} "
765785
"WHERE time >= '{time}' AND content_type = '{content_type}' "
766786
"AND object_id = '{object_id}' GROUP BY time(1d)"
767-
)
787+
),
788+
'elasticsearch': _make_query({
789+
'upload': {'sum': {'field': 'points.fields.tx_bytes'}},
790+
'download': {'avg': {'field': 'points.fields.rx_bytes'}},
791+
})
768792
},
769793
}
770794
}
771795
796+
# Please declare the operations separately in case you use elasticsearch as done below
797+
OPENWISP_MONITORING_ADDITIONAL_CHART_OPERATIONS = {
798+
'upload': {'operator': '/', 'value': 1000000},
799+
'download': {'operator': '/', 'value': 1000000},
800+
}
801+
772802
Or if you want to define a new chart configuration, which you can then
773803
call in your custom code (eg: a custom check class), you can do so as follows:
774804

775805
.. code-block:: python
776806
777807
from django.utils.translation import gettext_lazy as _
778808
809+
from openwisp_monitoring.db.backends.elasticsearch import _make_query
810+
779811
OPENWISP_MONITORING_CHARTS = {
780812
'ram': {
781813
'type': 'line',
@@ -789,7 +821,12 @@ call in your custom code (eg: a custom check class), you can do so as follows:
789821
"MEAN(buffered) AS buffered FROM {key} WHERE time >= '{time}' AND "
790822
"content_type = '{content_type}' AND object_id = '{object_id}' "
791823
"GROUP BY time(1d)"
792-
)
824+
),
825+
'elasticsearch': _make_query({
826+
'total': {'avg': {'field': 'points.fields.total'}},
827+
'free': {'avg': {'field': 'points.fields.free'}},
828+
'buffered': {'avg': {'field': 'points.fields.buffered'}},
829+
})
793830
},
794831
}
795832
}

docker-compose.yml

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ services:
1111
depends_on:
1212
- influxdb
1313
- redis
14+
- es01
15+
- es02
1416

1517
influxdb:
1618
image: influxdb:1.8-alpine
@@ -22,6 +24,45 @@ services:
2224
INFLUXDB_DB: openwisp2
2325
INFLUXDB_USER: openwisp
2426
INFLUXDB_USER_PASSWORD: openwisp
27+
# clustered version of elasticsearch is used as that might be used in production
28+
es01:
29+
image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
30+
container_name: es01
31+
environment:
32+
- "node.name=es01"
33+
- "discovery.seed_hosts=es02"
34+
- "cluster.initial_master_nodes=es01,es02"
35+
- "cluster.name=openwisp2"
36+
- "bootstrap.memory_lock=true"
37+
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
38+
ulimits:
39+
memlock:
40+
soft: -1
41+
hard: -1
42+
volumes:
43+
- esdata01:/usr/share/elasticsearch/data
44+
ports:
45+
- 9200:9200
46+
networks:
47+
- esnet
48+
es02:
49+
image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
50+
container_name: es02
51+
environment:
52+
- "node.name=es02"
53+
- "discovery.seed_hosts=es01"
54+
- "cluster.initial_master_nodes=es01,es02"
55+
- "cluster.name=openwisp2"
56+
- "bootstrap.memory_lock=true"
57+
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
58+
ulimits:
59+
memlock:
60+
soft: -1
61+
hard: -1
62+
volumes:
63+
- esdata02:/usr/share/elasticsearch/data
64+
networks:
65+
- esnet
2566

2667
redis:
2768
image: redis:5.0-alpine
@@ -31,3 +72,10 @@ services:
3172

3273
volumes:
3374
influxdb-data: {}
75+
esdata01:
76+
driver: local
77+
esdata02:
78+
driver: local
79+
80+
networks:
81+
esnet:

openwisp_monitoring/db/__init__.py

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
from .backends import timeseries_db
22

33
chart_query = timeseries_db.queries.chart_query
4-
default_chart_query = timeseries_db.queries.default_chart_query
5-
device_data_query = timeseries_db.queries.device_data_query
64

7-
__all__ = ['timeseries_db', 'chart_query', 'default_chart_query', 'device_data_query']
5+
__all__ = ['timeseries_db', 'chart_query']

openwisp_monitoring/db/backends/__init__.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,8 @@ def load_backend_module(backend_name=TIMESERIES_DB['BACKEND'], module=None):
4848
except ImportError as e:
4949
# The database backend wasn't found. Display a helpful error message
5050
# listing all built-in database backends.
51-
builtin_backends = ['influxdb']
51+
builtin_backends = ['influxdb', 'elasticsearch']
52+
raise e
5253
if backend_name not in [
5354
f'openwisp_monitoring.db.backends.{b}' for b in builtin_backends
5455
]:
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
from .queries import _make_query
2+
3+
__all__ = ['_make_query']

0 commit comments

Comments
 (0)