In memory databases leak memory on Close/DropAll due to orphaned skiplists

[weberr13]

SkipLists leak on various destructive operations primarily used in testing

When a user like myself creates an in-memory badgerDB for use in test the underlying Open call creates a skl.NewSkipList with a rather large skl.newArena that is impossible for the garbage collector to find and remove. The underlying code that "cleans up" the skiplist when the reference count becomes 0 simply drops the head node, but this is a skiplist with each node retaining reference to other nodes (sometimes multiple references). I reproduced and profiled multiple cases in our code where running tests with go test -count=1000 ./ these lost skipLists quickly accumulated and caused the test to crash. Adding a DropAll only made things worse, causing an additional lost skiplist. I also confirmed that adding a runtime.GC() call to the end of our tests did nothing to find and remove this leaked memory.

To Reproduce

Steps to reproduce the behavior:

1. Create a loop that creates an in-memory db, writes to it once, closes it
2. profile with pprof
3. observe the leak

Expected behavior

The only way to assure that the skiplist can be dereferenced completely is to walk the list destroying and unreferencing all adjacent nodes in the list (eg walk the list and delete everything). Simply setting head = nil will not suffice.

Screenshots

pprof output from a test that does the above 1000 times (no example code, this is unfortunately part of our proprietary system). If you need me to create an example code snippet I can circle back.

Environment

• OS: Tested on Fedora 6.14.8-300.fc42.x86_64

Additional context

Add any other context about the problem here.

[linear]

BGR-12 In memory databases leak memory on Close/DropAll due to orphaned skiplists