Remove 'document_type' setting

[jordansissel]

We've been advising users (for years!) to use document_type for different schemas of logs on the same index. However, recently, I have learned that this is not a correct way to use Elasticsearch!

TODO: Add link/references to this from the Elasticsearch repo

Proposal: We should remove the document_type setting, and update our docs to encourage using different index names for different (and possibly conflicting) document schemas within Elasticsearch.

[AlexRRR]

@jordansissel I am just about to implement document_type, would you care to share why this is a bad idea? could not find anything on this?

PS: Found this for anyone else interested. https://www.elastic.co/blog/index-vs-type

[untergeek]

@AlexRRR, Elasticsearch still requires a _type for each document, and Logstash provides this via the type field automatically, or via the document_type setting.

What @jordansissel is referring to are the changes since Elasticsearch 2.0 which disallow conflicting field mappings within an index, even if they exist within different _types. Earlier versions of Elasticsearch allowed for this, and this was sub-optimal.

Please see mapping changes. The blog post you linked has a lighter explanation of the same.

[tuananhtd]

Sorry guys, in the path of finding how to use the document_type I landed here. I've just started playing with ELK stack a few days. And my newbie's thought is that if I have different types of event to log I should use different type.
I could didn't using the document_type setting yet and now I just find out we have type and _type what is the difference then?
Hope to hear your thought @untergeek

[untergeek]

@tuananhtd

As previously mentioned, Elasticsearch requires a _type. Logstash uses the value of the type field to provide Elasticsearch with a value for _type. If there is no type in a Logstash event, Logstash will automatically provide Elasticsearch with a value of logs as the value for _type.

The document_type setting simply overrides whatever Logstash would have set and provides the value to Elasticsearch.

As far as the difference is concerned, Elasticsearch uses _type as part of document mapping. Every document can be directly discovered if you know the index name, the _type, and the document id. Using a type field in Elasticsearch, you can more easily distinguish between different data sets. However, the most current recommendation for Elasticsearch is to use a single data type (as in a single mapping, or _type) per index for the reasons mentioned in #259 (comment)

This can make it seem like having a type field redundant. Regardless, Logstash frequently needs this type in order to route events through specific filters, and/or to the correct destinations. At present, you could tag/type your fields using the @metadata field in Logstash, as nothing in that object will propagate to Elasticsearch.

[untergeek]

@tuananhtd This is why @jordansissel is recommending we remove document_type altogether, and perhaps even use @metadata fields for event routing. It will clean things up in Logstash and in Elasticsearch as well.

[tuananhtd]

@untergeek thank you so much for the response!

As a result of my debugging I find out that I just cannot use the field type in my JSON object. However change the field's name will make it work. After read your comments I think may be the mapping changes that you mentioned is the reason. Besides that the type field (without the underscore) is always set to log. document_type setting will only affect the _type field.

Regarding to single data type per index I'm just curious that will it generate to much overhead since each index will have multiple shards so based on my naive though each query will have a overhead as (number of shards) * (number of types)?

[andrewvc]

I still think users should get this level of control. We should discourage users from using document_type, but there's no reason to not allow them to use it IMHO.

[andrewvc]

I'll mention its also necessary for parent/child docs.

[arn-k]

Omitting document_type in a config causes documents to be created in Elasticsearch with type "doc". Yet the Elasticsearch documentation for the planned removal of type recommends using "_doc" for type to aid the transition to the removal of types.

Should this plugin default the type to "_doc" instead of "doc" when document_type is omitted?

[Dris101]

I think that is correct. I am doing a re-indexing from 5.6 to 6.4 and if I want logstash to use the now recommended "_doc" rather than "doc" (see https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html) I am having to use document_type => "_doc", which is itself deprecated

[arn-k]

Yes, same for us, which means we get deprecation warnings in our logs about using document_type. Would gladly remove it IF the default was _doc instead of doc.

[SajtiDH]

The issue is still there. I have only one mapping/type for my index as the 6.x documentation says, but still if I drop the deprecated document_type it breaks the output because _type=>"doc" will be on the output.