Document LS output changes necessary and how it works with curator + rollover API setup

[ppf2]

On the curator side, ES rollover API support has been added to master (and will be available in a 5.x curator version in the future).

We will have a lot of questions on what it means for Logstash since by default Logstash rolls indices every day and these are created as actual indices via the ES output, while the rollover API based on rolling aliases to a new index.

This generally means that if users use the upcoming/new curator+rollover feature, they are effectively "outsourcing" the index rolling feature of Logstash to the curator, i.e. they will have to create an alias ahead of time to point to an index (single index only), and set it to be the index name for the LS ES output (instead of having LS roll new indices), and then schedule the curator to run regularly to check for when it needs to roll a new timebased index based on max_age or max_docs.

It will be nice to document this Logstash use case with curator rollover api support. I am not sure where this information should go, but having something in the LS ES output documentation can be useful since they can't really make use of the curator+rollover feature unless they reconfigure the LS ES output to stop it from rolling indices automatically. Having similar use case description in the curator docs will also be helpful. But I will let you (@suyograo @untergeek) determine the appropriate place(s) to add this information :)

Or maybe the right way to approach this is to add rollover support directly to LS ES output? But until that is available, having some documentation available to go with the release timeframe for Curator 5.0 (when we add rollover support) will be helpful.

[acchen97]

Thanks @ppf2

@untergeek when is Curator 5.0 planned for?

@dedemorton my feel on this one is that it should be at least in the LS ES output docs. If they're using both LS and Curator with the index rollover feature, they should use the respective alias directly for indexing with the ES output. WDYT?

[untergeek]

@acchen97 A few of us discussed this in Tahoe. I think there may be room for an option in the Elasticsearch output plugin to create an alias + rollover-friendly index. We'd need a flag that says rollover_name or something so it knows to do that. For example:

elasticsearch {
   # ... other options
   index => "aliasname"
   rollover_name => "index-00001"
}

If users don't append the counter, we could auto-append that, of course. And at initialization, we could detect whether an index is already associated with aliasname, or an index by that name already exists and act accordingly.

[untergeek]

I hope to release Curator 5 in or before the middle of April (crosses fingers)...

[ppf2]

+1 . If we can do this directly in LS, it's a cleaner solution.

[acchen97]

@untergeek I do like the native solution. Mind creating an issue for it in the ES output repo?

@suyograo this stack integration is important; perhaps we could make this change around the Curator 5 release and bundle the new plugin version in 5.5?

[ppf2]

@acchen97 @suyograo Curator 5 with rollover support has been released. Until native rollover solution is available for the LS ES output, I think it will be helpful to document what users will have to do in the LS ES output in order to "outsource" the index creation management to the curator for rollover.

[untergeek]

@ppf2 @acchen97 I finally got around to creating that issue: #636