Poor performance with streams (high CPU and memory usage) #2213
Description
Describe the bug
Maybe I am using it wrong.
To support replication between nodes, I use hashicorp/memberlist. Every second, a "Push/Pull" is triggered, during which I queue a Backup/Load (incremental) on the pair of nodes.
It's a very dumb way of doing replication, there are probably better ways.
When using the default settings for the stream, I can observe a huge increase in CPU usage and memory usage. In fact, as we can see in the pprof output in the linked PR, most of the time is spent in Go's garbage collector, and traces back to BadgerDB's Stream.
If I reduce the concurrency by manually setting the number of goroutine used, performance increase (but I'm afraid that replication could become too slow during peak usage of the service (more testing needed on my side).
What are your recommendations and/or tips?
Environment
- OS: Ubuntu 25.04
- Language: Go
- Version: v4.7.0