You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/index.asciidoc
+12-11Lines changed: 12 additions & 11 deletions
Original file line number
Diff line number
Diff line change
@@ -117,18 +117,21 @@ When <<plugins-{type}s-{plugin}-tracking_field>> is set, the plugin will record
117
117
a file (location defaults to <<plugins-{type}s-{plugin}-last_run_metadata_path>>).
118
118
119
119
The user can then inject this value in the query using the placeholder `:last_value`. The value will be injected into the query
120
-
before execution, and the updated after the query completes, assuming new data was found.
120
+
before execution, and the updated after the query completes if new data was found.
121
121
122
122
This feature works best when:
123
-
* the query sorts by the tracking field
124
-
* the field type has enough resolution so that two events are unlikely to have the same value for the field
125
123
126
-
The plugin also offers another placeholder called `:present` used to inject the nano-second based value of "now-30s".
124
+
. the query sorts by the tracking field;
125
+
. the timestamp field is added by {es};
126
+
. the field type has enough resolution so that two events are unlikely to have the same value.
127
127
128
-
A suggestion is to use a tracking field that has nanosecond second precision, like
129
-
https://www.elastic.co/guide/en/elasticsearch/reference/current/date_nanos.html[date nanoseconds] field type.
128
+
It is recommended to use a tracking field whose type is https://www.elastic.co/guide/en/elasticsearch/reference/current/date_nanos.html[date nanoseconds].
129
+
If the tracking field is of this data type, an extra placeholder called `:present` can be used to inject the nano-second based value of "now-30s".
130
+
This placeholder is useful as the right-hand side of a range filter, allowing the collection of
131
+
new data but leaving partially-searcheable bulk request data to the next scheduled job.
130
132
131
-
A good use case for this feature is to track new data in an index, which can be achieved by:
133
+
Below is a series of steps to help set up the "tailing" of data being written to a set of indices, using a date nanosecond field
134
+
added by an Elasticsearch ingest pipeline, and the `tracking_field` capability of this plugin.
132
135
133
136
. create ingest pipeline that adds Elasticsearch's `_ingest.timestamp` field to the documents as `event.ingested`:
134
137
@@ -205,9 +208,7 @@ A good use case for this feature is to track new data in an index, which can be
0 commit comments