Partitioning logic change in 1.7 causes partition changing and breaks joins

[acordmike]

Issue: Partitioning logic change in 1.7 causes partition changing and breaks joins

After upgrading to streamiz.net v1.7, we are encountering serious issues due to a change in how partitions are selected in DefaultStreamPartitioner (see relevant code). This was also mentioned in #404.

Impact

• When we change the key of a record and then join to another stream, the new partitioning logic causes:
  • Missing records
  • Old records showing up on incorrect partitions
• This affects multiple production streams after upgrading, causing data inconsistencies and outages.
• The only workaround so far is to override the default partitioner for every stream and delete + repopulate all compacted topics, which is not practical.

Expected vs. Actual Behavior

• Expected:
  When sending to a topic, the partition should always be recalculated based on the destination topic's partition count and the (possibly new) key—consistent with previous versions.
• Actual (since 1.7):
  If the destination has the same or more partitions as the source, the source partition is used, even if the key has changed.

Suggestions

• Preferred:
  Restore the previous logic, where the partition is always recalculated for the destination topic, regardless of partition counts.
• Alternative:
  Introduce a global configuration option to select the partitioning algorithm, allowing projects to revert to the pre-1.7 behavior without having to override the partitioner in every TO<>() call.

Additional Notes

• We are happy to provide a minimal reproducible example if helpful.
• Please clarify if there is a recommended migration path or workaround for large projects.

[LGouellec]

Pretty fair ! Let me revert this code change !