Unexpected error when @timestamp field missing

[gmoskovicz]

• Version: Logstash 5.x and Logstash 6.x

• Operating System: Any

• Config File (if you have sensitive info, please remove it):

input {
  generator {}
}
filter {
  mutate {
    rename => {
      "@timestamp" => "timestamp"
    }
  }
}

output {
  elasticsearch {
    user => elastic
    password => xxxxx
  }
}

• Sample Data:

Any data

• Steps to Reproduce:

Run logstash with that configuration.

Output:

5.x

[2018-02-22T13:49:42,425][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: timestamp field is missing>, :backtrace=>["org/logstash/ext/JrubyEventExtLibrary.java:205:in sprintf'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:169:in event_action_params'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:44:in event_action_tuple'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:38:in multi_receive'", "org/jruby/RubyArray.java:2414:in map'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:38:in multi_receive'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in multi_receive'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/logstash-core/lib/logstash/output_delegator.rb:49:in multi_receive'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/logstash-core/lib/logstash/pipeline.rb:436:in output_batch'", "org/jruby/RubyHash.java:1342:in each'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/logstash-core/lib/logstash/pipeline.rb:435:in output_batch'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/logstash-core/lib/logstash/pipeline.rb:381:in worker_loop'", "/Users/Gabriel/Documents/ElasticSearch/logstash-5.6.2/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}

6.x

[2018-02-22T13:50:52,960][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: timestamp field is missing>, :backtrace=>["org/logstash/ext/JrubyEventExtLibrary.java:168:in sprintf'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:169:in event_action_params'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:39:in event_action_tuple'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:34:in block in multi_receive'", "org/jruby/RubyArray.java:2486:in map'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:34:in multi_receive'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in multi_receive'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/output_delegator.rb:50:in multi_receive'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:487:in block in output_batch'", "org/jruby/RubyHash.java:1343:in each'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:486:in output_batch'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:438:in worker_loop'", "/Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:393:in `block in start_workers'"]}

LogStash::Error: timestamp field is missing
                 sprintf at org/logstash/ext/JrubyEventExtLibrary.java:168
     event_action_params at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:169
      event_action_tuple at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:39
  block in multi_receive at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:34
                     map at org/jruby/RubyArray.java:2486
           multi_receive at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:34
           multi_receive at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13
           multi_receive at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/output_delegator.rb:50
   block in output_batch at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:487
                    each at org/jruby/RubyHash.java:1343
            output_batch at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:486
             worker_loop at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:438
  block in start_workers at /Users/Gabriel/Documents/ElasticSearch/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:393

---

Should this be a better error rather than FATAL? Given that this could happen just with a specific document (if you have conditionals), best is to fail that event rather than a FATAL error?

[guyboertje]

The @timestamp is missing so the default index string can't be built

  mod.config :index, :validate => :string, :default => "logstash-%{+YYYY.MM.dd}"

[KevSex]

Has there been any fix for this? Just ran into this FATAL error in 6.2.3

Had to create the following workaround in hope that this stops it from crashing logstash in future.

filter {
  if ![@timestamp] {
    drop { }
  }
}

[webmat]

Fix for this incoming: #777.

[webmat]

@gmoskovicz In case you're interested, discussions on my fix for this issue took a turn. Discussion will be happening here #779 :-)

Perhaps you can chime in with what you think our customers would prefer?

[Jmainguy]

@KevSex did that actually stop it from crashing for you? I tried adding that in /etc/logstash/conf.d/ files but didnt seem to change anything, logstash keeps flapping for me.

EDIT:

Ignore me, we had a bad mutate that was removing the timestamp field later, womp.

[vacri]

Still a problem in Logstash 6.4.

I believe the problem for us came from using the documented Nginx snippet at https://www.elastic.co/guide/en/logstash/current/logstash-config-for-filebeat-modules.html#parsing-nginx

For the nginx access log, the 'read_timestamp' is an added field
For the nginx error log, the 'read_timestamp' is renamed from @timestamp, removing @timestamp