Skip to content

Commit ee2da28

Browse files
updated docs for main (#1867)
* Fix include path to support docs 9.0 and later --------- Co-authored-by: Karen Metts <[email protected]>
1 parent 7284da3 commit ee2da28

33 files changed

+1254
-549
lines changed

docs/plugins/codecs/cloudfront.asciidoc

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@
66
///////////////////////////////////////////
77
START - GENERATED VARIABLES, DO NOT EDIT!
88
///////////////////////////////////////////
9-
:version: v7.1.8
10-
:release_date: 2024-07-26
11-
:changelog_url: https://github.com/logstash-plugins/logstash-integration-aws/blob/v7.1.8/CHANGELOG.md
9+
:version: v7.2.1
10+
:release_date: 2025-02-13
11+
:changelog_url: https://github.com/logstash-plugins/logstash-integration-aws/blob/v7.2.1/CHANGELOG.md
1212
:include_path: ../include
1313
///////////////////////////////////////////
1414
END - GENERATED VARIABLES, DO NOT EDIT!
@@ -36,7 +36,7 @@ This codec will read cloudfront encoded content
3636
&nbsp;
3737

3838
[id="plugins-{type}s-{plugin}-charset"]
39-
===== `charset`
39+
===== `charset`
4040

4141
* Value can be any of: `ASCII-8BIT`, `UTF-8`, `US-ASCII`, `Big5`, `Big5-HKSCS`, `Big5-UAO`, `CP949`, `Emacs-Mule`, `EUC-JP`, `EUC-KR`, `EUC-TW`, `GB2312`, `GB18030`, `GBK`, `ISO-8859-1`, `ISO-8859-2`, `ISO-8859-3`, `ISO-8859-4`, `ISO-8859-5`, `ISO-8859-6`, `ISO-8859-7`, `ISO-8859-8`, `ISO-8859-9`, `ISO-8859-10`, `ISO-8859-11`, `ISO-8859-13`, `ISO-8859-14`, `ISO-8859-15`, `ISO-8859-16`, `KOI8-R`, `KOI8-U`, `Shift_JIS`, `UTF-16BE`, `UTF-16LE`, `UTF-32BE`, `UTF-32LE`, `Windows-31J`, `Windows-1250`, `Windows-1251`, `Windows-1252`, `IBM437`, `IBM737`, `IBM775`, `CP850`, `IBM852`, `CP852`, `IBM855`, `CP855`, `IBM857`, `IBM860`, `IBM861`, `IBM862`, `IBM863`, `IBM864`, `IBM865`, `IBM866`, `IBM869`, `Windows-1258`, `GB1988`, `macCentEuro`, `macCroatian`, `macCyrillic`, `macGreek`, `macIceland`, `macRoman`, `macRomania`, `macThai`, `macTurkish`, `macUkraine`, `CP950`, `CP951`, `IBM037`, `stateless-ISO-2022-JP`, `eucJP-ms`, `CP51932`, `EUC-JIS-2004`, `GB12345`, `ISO-2022-JP`, `ISO-2022-JP-2`, `CP50220`, `CP50221`, `Windows-1256`, `Windows-1253`, `Windows-1255`, `Windows-1254`, `TIS-620`, `Windows-874`, `Windows-1257`, `MacJapanese`, `UTF-7`, `UTF8-MAC`, `UTF-16`, `UTF-32`, `UTF8-DoCoMo`, `SJIS-DoCoMo`, `UTF8-KDDI`, `SJIS-KDDI`, `ISO-2022-JP-KDDI`, `stateless-ISO-2022-JP-KDDI`, `UTF8-SoftBank`, `SJIS-SoftBank`, `BINARY`, `CP437`, `CP737`, `CP775`, `IBM850`, `CP857`, `CP860`, `CP861`, `CP862`, `CP863`, `CP864`, `CP865`, `CP866`, `CP869`, `CP1258`, `Big5-HKSCS:2008`, `ebcdic-cp-us`, `eucJP`, `euc-jp-ms`, `EUC-JISX0213`, `eucKR`, `eucTW`, `EUC-CN`, `eucCN`, `CP936`, `ISO2022-JP`, `ISO2022-JP2`, `ISO8859-1`, `ISO8859-2`, `ISO8859-3`, `ISO8859-4`, `ISO8859-5`, `ISO8859-6`, `CP1256`, `ISO8859-7`, `CP1253`, `ISO8859-8`, `CP1255`, `ISO8859-9`, `CP1254`, `ISO8859-10`, `ISO8859-11`, `CP874`, `ISO8859-13`, `CP1257`, `ISO8859-14`, `ISO8859-15`, `ISO8859-16`, `CP878`, `MacJapan`, `ASCII`, `ANSI_X3.4-1968`, `646`, `CP65000`, `CP65001`, `UTF-8-MAC`, `UTF-8-HFS`, `UCS-2BE`, `UCS-4BE`, `UCS-4LE`, `CP932`, `csWindows31J`, `SJIS`, `PCK`, `CP1250`, `CP1251`, `CP1252`, `external`, `locale`
4242
* Default value is `"UTF-8"`
@@ -50,3 +50,5 @@ weird cases like this, you can set the charset setting to the
5050
actual encoding of the text and logstash will convert it for you.
5151

5252
For nxlog users, you'll want to set this to "CP1252"
53+
54+

docs/plugins/codecs/cloudtrail.asciidoc

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@
66
///////////////////////////////////////////
77
START - GENERATED VARIABLES, DO NOT EDIT!
88
///////////////////////////////////////////
9-
:version: v7.1.8
10-
:release_date: 2024-07-26
11-
:changelog_url: https://github.com/logstash-plugins/logstash-integration-aws/blob/v7.1.8/CHANGELOG.md
9+
:version: v7.2.1
10+
:release_date: 2025-02-13
11+
:changelog_url: https://github.com/logstash-plugins/logstash-integration-aws/blob/v7.2.1/CHANGELOG.md
1212
:include_path: ../include
1313
///////////////////////////////////////////
1414
END - GENERATED VARIABLES, DO NOT EDIT!
@@ -36,7 +36,11 @@ This is the base class for logstash codecs.
3636
&nbsp;
3737

3838
[id="plugins-{type}s-{plugin}-charset"]
39-
===== `charset`
39+
===== `charset`
4040

4141
* Value can be any of: `ASCII-8BIT`, `UTF-8`, `US-ASCII`, `Big5`, `Big5-HKSCS`, `Big5-UAO`, `CP949`, `Emacs-Mule`, `EUC-JP`, `EUC-KR`, `EUC-TW`, `GB2312`, `GB18030`, `GBK`, `ISO-8859-1`, `ISO-8859-2`, `ISO-8859-3`, `ISO-8859-4`, `ISO-8859-5`, `ISO-8859-6`, `ISO-8859-7`, `ISO-8859-8`, `ISO-8859-9`, `ISO-8859-10`, `ISO-8859-11`, `ISO-8859-13`, `ISO-8859-14`, `ISO-8859-15`, `ISO-8859-16`, `KOI8-R`, `KOI8-U`, `Shift_JIS`, `UTF-16BE`, `UTF-16LE`, `UTF-32BE`, `UTF-32LE`, `Windows-31J`, `Windows-1250`, `Windows-1251`, `Windows-1252`, `IBM437`, `IBM737`, `IBM775`, `CP850`, `IBM852`, `CP852`, `IBM855`, `CP855`, `IBM857`, `IBM860`, `IBM861`, `IBM862`, `IBM863`, `IBM864`, `IBM865`, `IBM866`, `IBM869`, `Windows-1258`, `GB1988`, `macCentEuro`, `macCroatian`, `macCyrillic`, `macGreek`, `macIceland`, `macRoman`, `macRomania`, `macThai`, `macTurkish`, `macUkraine`, `CP950`, `CP951`, `IBM037`, `stateless-ISO-2022-JP`, `eucJP-ms`, `CP51932`, `EUC-JIS-2004`, `GB12345`, `ISO-2022-JP`, `ISO-2022-JP-2`, `CP50220`, `CP50221`, `Windows-1256`, `Windows-1253`, `Windows-1255`, `Windows-1254`, `TIS-620`, `Windows-874`, `Windows-1257`, `MacJapanese`, `UTF-7`, `UTF8-MAC`, `UTF-16`, `UTF-32`, `UTF8-DoCoMo`, `SJIS-DoCoMo`, `UTF8-KDDI`, `SJIS-KDDI`, `ISO-2022-JP-KDDI`, `stateless-ISO-2022-JP-KDDI`, `UTF8-SoftBank`, `SJIS-SoftBank`, `BINARY`, `CP437`, `CP737`, `CP775`, `IBM850`, `CP857`, `CP860`, `CP861`, `CP862`, `CP863`, `CP864`, `CP865`, `CP866`, `CP869`, `CP1258`, `Big5-HKSCS:2008`, `ebcdic-cp-us`, `eucJP`, `euc-jp-ms`, `EUC-JISX0213`, `eucKR`, `eucTW`, `EUC-CN`, `eucCN`, `CP936`, `ISO2022-JP`, `ISO2022-JP2`, `ISO8859-1`, `ISO8859-2`, `ISO8859-3`, `ISO8859-4`, `ISO8859-5`, `ISO8859-6`, `CP1256`, `ISO8859-7`, `CP1253`, `ISO8859-8`, `CP1255`, `ISO8859-9`, `CP1254`, `ISO8859-10`, `ISO8859-11`, `CP874`, `ISO8859-13`, `CP1257`, `ISO8859-14`, `ISO8859-15`, `ISO8859-16`, `CP878`, `MacJapan`, `ASCII`, `ANSI_X3.4-1968`, `646`, `CP65000`, `CP65001`, `UTF-8-MAC`, `UTF-8-HFS`, `UCS-2BE`, `UCS-4BE`, `UCS-4LE`, `CP932`, `csWindows31J`, `SJIS`, `PCK`, `CP1250`, `CP1251`, `CP1252`, `external`, `locale`
4242
* Default value is `"UTF-8"`
43+
44+
45+
46+

docs/plugins/filters/elastic_integration.asciidoc

Lines changed: 87 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,10 @@
55
///////////////////////////////////////////
66
START - GENERATED VARIABLES, DO NOT EDIT!
77
///////////////////////////////////////////
8-
:version: v8.17.0
9-
:release_date: 2025-01-08
10-
:changelog_url: https://github.com/elastic/logstash-filter-elastic_integration/blob/v8.17.0/CHANGELOG.md
11-
:include_path: ../include
8+
:version: v9.0.0
9+
:release_date: 2025-04-28
10+
:changelog_url: https://github.com/elastic/logstash-filter-elastic_integration/blob/v9.0.0/CHANGELOG.md
11+
include_path: ../include
1212
///////////////////////////////////////////
1313
END - GENERATED VARIABLES, DO NOT EDIT!
1414
///////////////////////////////////////////
@@ -19,7 +19,7 @@ END - GENERATED VARIABLES, DO NOT EDIT!
1919

2020
=== {elastic-integration-name} filter plugin
2121

22-
include::{include_path}/plugin_header-nonstandard.asciidoc[]
22+
include::{include_path}/plugin_header.asciidoc[]
2323

2424
.Elastic Enterprise License
2525
****
@@ -53,21 +53,21 @@ Events that _fail_ ingest pipeline processing will be tagged with `_ingest_pipel
5353

5454
- This plugin requires Java 17 minimum with {ls} `8.x` versions and Java 21 minimum with {ls} `9.x` versions.
5555
- When you upgrade the {stack}, upgrade {ls} (or this plugin specifically) _before_ you upgrade {kib}.
56-
(Note that this requirement is a departure from the typical {stack} https://www.elastic.co/guide/en/elastic-stack/current/installing-elastic-stack.html#install-order-elastic-stack[installation order].)
56+
(Note that this requirement is a departure from the typical {stack} https://www.elastic.co/guide/en/elastic-stack/current/installing-elastic-stack.html#install-order-elastic-stack[installation order].)
5757
+
58-
The {es}-{ls}-{kib} installation order ensures the best experience with {agent}-managed pipelines, and embeds functionality from a version of {es} Ingest Node that is compatible with the plugin version (`major`.`minor`).
58+
The {es}-{ls}-{kib} installation order recommended here ensures the best experience with {agent}-managed pipelines, and embeds functionality from a version of {es} Ingest Node that is compatible with the plugin version (`major`.`minor`).
5959

6060
[id="plugins-{type}s-{plugin}-es-tips"]
6161
===== Using `filter-elastic_integration` with `output-elasticsearch`
6262

6363
Elastic {integrations} are designed to work with {logstash-ref}/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data-streams[data streams] and {logstash-ref}/plugins-outputs-elasticsearch.html#_compatibility_with_the_elastic_common_schema_ecs[ECS-compatible] output.
64-
Be sure that these features are enabled in the {logstash-ref}/plugins-outputs-elasticsearch.html[`output-elasticsearch`] plugin.
64+
Be sure that these features are enabled in the {logstash-ref}/plugins-outputs-elasticsearch.html[`output-elasticsearch`] plugin.
6565

66-
* Set {logstash-ref}/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data_stream[`data-stream`] to `true`. +
67-
(Check out {logstash-ref}/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data-streams[Data streams] for additional data streams settings.)
66+
* Set {logstash-ref}/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data_stream[`data-stream`] to `true`. +
67+
(Check out {logstash-ref}/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data-streams[Data streams] for additional data streams settings.)
6868
* Set {logstash-ref}/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ecs_compatibility[`ecs-compatibility`] to `v1` or `v8`.
6969

70-
Check out the {logstash-ref}/plugins-outputs-elasticsearch.html[`output-elasticsearch` plugin] docs for additional settings.
70+
Check out the {logstash-ref}/plugins-outputs-elasticsearch.html[`output-elasticsearch` plugin] docs for additional settings.
7171

7272
[id="plugins-{type}s-{plugin}-minimum_configuration"]
7373
==== Minimum configuration
@@ -145,12 +145,12 @@ At the startup phase, this plugin confirms that current user has sufficient priv
145145

146146
|=======================================================================
147147

148-
[NOTE]
148+
[NOTE]
149149
--
150150
This plugin cannot determine if an anonymous user has the required privileges when it connects to an {es} cluster that has security features disabled or when the user does not provide credentials.
151151
The plugin starts in an unsafe mode with a runtime error indicating that API permissions are insufficient, and prevents events from being processed by the ingest pipeline.
152152

153-
To avoid these issues, set up user authentication and ensure that security in {es} is enabled (default).
153+
To avoid these issues, set up user authentication and ensure that security in {es} is enabled (default).
154154
--
155155

156156
[id="plugins-{type}s-{plugin}-supported_ingest_processors"]
@@ -166,27 +166,27 @@ It has access to the Painless and Mustache scripting engines where applicable:
166166

167167
| `append` | _none_
168168
| `bytes` | _none_
169-
| `communityid` | _none_
169+
| `community_id` | _none_
170170
| `convert` | _none_
171171
| `csv` | _none_
172172
| `date` | _none_
173-
| `dateindexname` | _none_
173+
| `date_index_name` | _none_
174174
| `dissect` | _none_
175-
| `dotexpander` | _none_
175+
| `dot_expander` | _none_
176176
| `drop` | _none_
177177
| `fail` | _none_
178178
| `fingerprint` | _none_
179179
| `foreach` | _none_
180180
| `grok` | _none_
181181
| `gsub` | _none_
182-
| `htmlstrip` | _none_
182+
| `html_strip` | _none_
183183
| `join` | _none_
184184
| `json` | _none_
185-
| `keyvalue` | _none_
185+
| `kv` | _none_
186186
| `lowercase` | _none_
187-
| `networkdirection` | _none_
187+
| `network_direction` | _none_
188188
| `pipeline` | resolved pipeline _must_ be wholly-composed of supported processors
189-
| `registereddomain` | _none_
189+
| `registered_domain` | _none_
190190
| `remove` | _none_
191191
| `rename` | _none_
192192
| `reroute` | _none_
@@ -207,7 +207,6 @@ h| GeoIp
207207

208208
|=======================================================================
209209

210-
211210
[id="plugins-{type}s-{plugin}-field_mappings"]
212211
===== Field Mappings
213212

@@ -280,6 +279,73 @@ To achieve this, mappings are cached for a maximum of {cached-entry-ttl}, and ca
280279
* when a reloaded mapping is newly _empty_, the previous non-empty mapping is _replaced_ with a new empty entry so that subsequent events will use the empty value
281280
* when the reload of a mapping _fails_, this plugin emits a log warning but the existing cache entry is unchanged and gets closer to its expiry.
282281

282+
[id="plugins-{type}s-{plugin}-troubleshooting"]
283+
==== Troubleshooting
284+
285+
Troubleshooting ingest pipelines associated with data streams requires a pragmatic approach, involving thorough analysis and debugging techniques.
286+
To identify the root cause of issues with pipeline execution, you need to enable debug-level logging.
287+
The debug logs allow monitoring the plugin's behavior and help to detect issues.
288+
The plugin operates through following phases: pipeline _resolution_, ingest pipeline _creation_, and pipeline _execution_.
289+
290+
[ingest-pipeline-resolution-errors]
291+
===== Ingest Pipeline Resolution Errors
292+
293+
*Plugin does not resolve ingest pipeline associated with data stream*
294+
295+
If you encounter `No pipeline resolved for event ...` messages in the debug logs, the error indicates that the plugin is unable to resolve the ingest pipeline from the data stream.
296+
To further diagnose and resolve the issue, verify whether the data stream's index settings include a `default_pipeline` or `final_pipeline` configuration.
297+
You can inspect the index settings by running a `POST _index_template/_simulate_index/{type}-{dataset}-{namespace}` query in the {kib} Dev Tools.
298+
Make sure to replace `{type}-{dataset}-{namespace}` with values corresponding to your data stream.
299+
For further guidance, we recommend exploring {fleet-guide}/integrations.html[Manage Elastic Agent Integrations], {es} {ref}/ingest.html#pipelines-for-fleet-elastic-agent[Ingest pipelines for fleet] and {integrations-docs}[Elastic {integrations}] resources.
300+
301+
*Ingest pipeline does not exist*
302+
303+
If you notice `pipeline not found: ...` messages in the debug logs or `Pipeline {pipeline-name} could not be loaded` warning messages, it indicates that the plugin has successfully resolved the ingest pipeline from `default_pipeline` or `final_pipeline`, but the specified pipeline does not exist.
304+
To confirm whether pipeline exists, run a `GET _ingest/pipeline/{ingest-pipeline-name}` query in the {kib} Dev Tools console.
305+
For further guidance, we recommend exploring {fleet-guide}/integrations.html[Manage Elastic Agent Integrations], {es} {ref}/ingest.html#pipelines-for-fleet-elastic-agent[Ingest pipelines for fleet] and {integrations-docs}[Elastic {integrations}] resources.
306+
307+
[ingest-pipeline-creation-errors]
308+
===== Ingest Pipeline Creation Errors
309+
310+
If you encounter `failed to create ingest pipeline {pipeline-name} from pipeline configuration` error messages, it indicates that the plugin is unable to create an ingest pipeline from the resolved pipeline configuration.
311+
This issue typically arises when the pipeline configuration contains unsupported or invalid processor(s) that the plugin cannot execute.
312+
In such situations, the log output includes information about the issue.
313+
For example, the following error message indicating `inference` processor in the pipeline configuration which is not supported processor type.
314+
315+
[source]
316+
----
317+
2025-01-21 12:29:13 [2025-01-21T20:29:13,986][ERROR][co.elastic.logstash.filters.elasticintegration.IngestPipelineFactory][main] failed to create ingest pipeline logs-my.custom-1.0.0 from pipeline configuration
318+
2025-01-21 12:29:13 org.elasticsearch.ElasticsearchParseException: No processor type exists with name [inference]
319+
2025-01-21 12:29:13 at org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:470) ~[logstash-filter-elastic_integration-0.1.16.jar:?]
320+
2025-01-21 12:29:13 at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:635)
321+
----
322+
323+
For further guidance, we recommend exploring {fleet-guide}/integrations.html[Manage Elastic Agent Integrations], {es} {ref}/ingest.html#handling-pipeline-failures[Handling pipeline failures] resources.
324+
325+
[ingest-pipeline-execution-errors]
326+
===== Ingest Pipeline Execution Errors
327+
328+
These errors typically fall into two main categories, each requiring specific investigation and resolution steps:
329+
330+
*Logstash catches issues while running ingest pipelines*
331+
332+
When errors occur during the execution of ingest pipelines, {ls} attaches the `_ingest_pipeline_failure` tag to the event, making it easier to identify and investigate problematic events.
333+
The detailed logs are available in the {ls} logs for your investigation.
334+
The root cause may depend on configuration, environment or integration you are running.
335+
For further guidance, we recommend exploring {fleet-guide}/integrations.html[Manage Elastic Agent Integrations], {es} {ref}/ingest.html#handling-pipeline-failures[Handling pipeline failures] resources.
336+
337+
*Errors internally occurred in the ingest pipeline*
338+
339+
If an ingest pipeline is configured with `on_failure` conditions, failures during pipeline execution are internally handled by the ingest pipeline itself and not be visible to {ls}.
340+
This means that errors are captured and processed within the pipeline, rather than being passed to {ls} for logging or tagging.
341+
To identify and analyze such cases, go to the {kib} -> Stack Management -> Ingest pipelines and find the ingest pipeline you are using.
342+
Click on it and navigate to the _Failure processors_ section. If processors are configured, they may specify which field contains the failure details.
343+
For example, the pipeline might store error information in a `error.message` field or a custom field defined in the _Failure processors_ configuration.
344+
Go to the {kib} Dev Tools and search for the data (`GET {index-ingest-pipeline-is-writing}/_search`) and look for the fields mentioned in the failure processors .
345+
The fields have error details which help you to analyze the root cause.
346+
347+
For further guidance, we recommend exploring {fleet-guide}/integrations.html[Manage Elastic Agent Integrations], {es} {ref}/ingest.html#handling-pipeline-failures[Handling pipeline failures] resources.
348+
283349
[id="plugins-{type}s-{plugin}-options"]
284350
==== {elastic-integration-name} Filter Configuration Options
285351

0 commit comments

Comments
 (0)