An item with the same key has already been added (Bis)

[jeremielec]

We have an random issue with Streamiz, under a high rate processing message.

Sould like #181, but as the stack differs and persist on latest version, i have created a new issue

The error "An item with the same key has already been added Key: xxxx" occurs randomly , then the whole stream task goes into error state.

The topic mentionned in the key change basicly every time. This can be a well known topic with business data or an internal generated topic like :

=> ?-?Join?To?-repartition [[0]]

Stack trace :

• at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
• at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
• at System.Collections.Generic.Dictionary2.AddRange(IEnumerable1 enumerable)
• at System.Collections.Generic.Dictionary2..ctor(IEnumerable1 collection, IEqualityComparer`1 comparer)
• at System.Linq.Enumerable.ToDictionary[TKey,TValue](IEnumerable1 source, IEqualityComparer1 comparer)
• at System.Linq.Enumerable.ToDictionary[TKey,TValue](IEnumerable`1 source)
• at Streamiz.Kafka.Net.Processors.StreamTask.MayWriteCheckpoint(Boolean force)
• at Streamiz.Kafka.Net.Processors.StreamTask.PostCommit(Boolean enforceCheckpoint)
• at Streamiz.Kafka.Net.Processors.Internal.TaskManager.CommitAll()
• at Streamiz.Kafka.Net.Processors.StreamThread.Commit()
• at Streamiz.Kafka.Net.Processors.StreamThread.<>c__DisplayClass66_1.b__3()
• at Streamiz.Kafka.Net.Crosscutting.ActionHelper.MeasureLatency(Action action)
• at Streamiz.Kafka.Net.Processors.StreamThread.Run()

[LGouellec]

Hey @jeremielec ,

thanks for your issue, let me take some time to reproduce and fix the issue. How your topology looks like ?