Skip to content

Allow streams to opt into less consistent writes for high throughput #6784

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ripienaar opened this issue Apr 10, 2025 · 0 comments
Open
Labels
2.12+ Features for 2.12 and beyond accepted The defect or proposal as been accepted proposal Enhancement idea or proposal
Milestone

Comments

@ripienaar
Copy link
Contributor

Proposed change

Streams do a lot of consistency checks on each publish - ensuring that the message hit a disk on the quorum before marking it as accepted. This comes at a huge cost in performance.

We should allow a stream to opt-in to a less strict mode favouring throughput which could relax some consistency checks.

JetStream used to have this as the primary model for writes early on but this was moved to a more strict model, the old code is still there and we can revive that under a opt-in flag

Use case

Fast publishers might be ok with lesser consistency models for data where some being lost is not a problem. Combined with Batch Publishes (#6549) this could allow for very high throughput use cases

Contribution

No response

@ripienaar ripienaar added 2.12+ Features for 2.12 and beyond accepted The defect or proposal as been accepted proposal Enhancement idea or proposal labels Apr 10, 2025
@ripienaar ripienaar added this to the 2.12.0 milestone May 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.12+ Features for 2.12 and beyond accepted The defect or proposal as been accepted proposal Enhancement idea or proposal
Projects
None yet
Development

No branches or pull requests

1 participant