refactor: replace deepsize crate with custom DeepSizeOf trait for Arrow-aware memory accounting

[wjones127]

The external deepsize crate does not correctly account for Arrow buffers
that are shared across Arc references, causing double-counting in cache
size calculations.

This PR introduces a custom DeepSizeOf trait in lance-core::deepsize
with a Context that tracks both Arc and raw buffer pointers in a
unified HashSet. It also adds a lance-derive proc-macro crate for
#[derive(DeepSizeOf)] and removes the dependency on the external
deepsize crate.

Changes:

• Add lance-core::deepsize module with custom DeepSizeOf trait and pointer-tracking Context
• Add lance-derive proc-macro crate with #[derive(DeepSizeOf)]
• Remove external deepsize crate from all crates; update all impls to use lance_core::deepsize
• Fix all deep_size_of_children implementations to thread context through (previously some ignored the parameter)
• Add DeepSizeOf for arrow_buffer::ScalarBuffer<T> with proper pointer tracking
• Fix PlainPostingList and Positions to use deep_size_of_children with context

[github-actions]

Review

Good motivation — the external deepsize crate doesn't track Arc/buffer pointer identity, so shared Arrow buffers get double-counted in cache size calculations. The custom trait with a pointer-tracking Context is the right approach.

Issues to address

P0: dyn Array::deep_size_of_children calls self.to_data() which clones all buffers

Array::to_data() can be expensive — it materializes an ArrayData by cloning Arc<Buffer>s and for some array types does non-trivial work. This is called every time you measure a column's size in a RecordBatch, which happens on every cache insertion/eviction. Consider using Array::get_array_memory_size() for the total and only using the Context for dedup at the Arc<dyn Array> / Arc<RecordBatch> level. Alternatively, use array.to_data() only once and cache the result, or see if ArrayData can be obtained without cloning (e.g. array.into_data() if you own it, or accessing internal data references).

P1: CompressedPostingList still uses get_buffer_memory_size() without context tracking

CompressedPostingList::deep_size_of_children (rust/lance-index/src/scalar/inverted/index.rs) was only half-migrated — it still calls self.blocks.get_buffer_memory_size() and self.positions.as_ref().map(Array::get_buffer_memory_size) directly, bypassing the Context dedup. This means shared buffers in compressed posting lists can still be double-counted.

P1: ScalarBuffer reports len * size_of::<T>() instead of capacity

In deepsize.rs, ScalarBuffer::deep_size_of_children returns self.len() * size_of::<T>(), but the underlying Buffer may have been allocated with more capacity (e.g. after slicing). Other impls in the same file (like Vec) correctly use capacity. For consistency and accuracy, consider using the underlying buffer's capacity.

Minor

• The HashMap size estimate uses capacity * (size_of::<K>() + size_of::<V>() + 1), which is a reasonable approximation for hashbrown/Swiss tables but the actual overhead per bucket is closer to 1 byte of control metadata per slot (not per occupied entry). This is fine as an estimate since you're using capacity() which already accounts for the total slot count — just noting it's approximate.

• No tests for the derive macro on enums or generics. Consider adding at least one test struct/enum in deepsize.rs tests that uses #[derive(DeepSizeOf)] to verify the proc macro generates correct code.

[codecov]

Codecov Report

❌ Patch coverage is 85.86667% with 53 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-core/src/deepsize.rs	87.08%	26 Missing and 1 partial ⚠️
rust/lance-derive/src/lib.rs	92.53%	4 Missing and 1 partial ⚠️
rust/lance-index/src/scalar/inverted/index.rs	58.33%	5 Missing ⚠️
rust/lance/src/index/vector/fixture_test.rs	0.00%	2 Missing ⚠️
rust/lance/src/index/vector/pq.rs	81.81%	2 Missing ⚠️
rust/lance-core/src/container/list.rs	0.00%	1 Missing ⚠️
rust/lance-index/src/mem_wal.rs	50.00%	1 Missing ⚠️
rust/lance-index/src/scalar/inverted/builder.rs	0.00%	1 Missing ⚠️
rust/lance-index/src/scalar/json.rs	0.00%	1 Missing ⚠️
rust/lance-index/src/scalar/ngram.rs	66.66%	1 Missing ⚠️
... and 7 more

📢 Thoughts on this report? Let us know!