feat: VectorIndex serialization and zero-IO reconstruction

[wjones127]

Depends on #6222.

Add serialization support for IVF vector index state, enabling
reconstruction from cache without re-reading index files.

Changes:

• partition_serde.rs: Spillable trait with serialize(&self, writer)
  and deserialize(data) for all quantizer types (PQ, SQ, Flat, RabitQ)
• IvfIndexState / VectorIndexData: serializable snapshot of vector
  index state cached via cacheable_state() on VectorIndex
• reconstruct_vector_index: rebuilds a live IVFIndex from cached
  IvfIndexState, dispatching on sub-index and quantization types
• File metadata and sizes are cached during try_new so reconstruction
  avoids both data reads and HEAD requests
• from_cached constructor on IvfQuantizationStorage to skip global
  buffer reads when reconstructing from cached metadata
• Zero-IO test verifying reconstruction performs no IO after initial open

[github-actions]

PR Review: feat: add serialize/deserialize for IVF PQ partition cache entries

Clean implementation overall — the zero-copy IPC approach is well-suited for cache serde. A few issues to flag:

P1: Integer overflow in deserialization offset arithmetic

In deserialize(), the section boundary calculations use unchecked addition:

let sub_index_end = sub_index_start + header.sub_index_len as usize;
let codebook_end = codebook_start + header.codebook_len as usize;
let storage_end = storage_start + header.storage_len as usize;

If the header contains corrupted or adversarial values (e.g., lengths close to u64::MAX), these additions can wrap around, causing storage_end to be a small value that passes the data.len() < storage_end check. The subsequent Buffer::slice_with_length calls would then read incorrect regions or panic.

Use checked_add and return an error on overflow, same as the defensive pattern already used for trailer_start / footer_start in read_ipc_zero_copy.

P1: Missing Hamming in distance type roundtrip test

test_roundtrip_preserves_distance_type covers L2, Cosine, and Dot but omits Hamming, which is handled in the distance_type_to_u8/u8_to_distance_type mapping. Either add it to the test or document why it's excluded (e.g., PQ storage doesn't support Hamming).

Minor

• The module is pub mod partition_serde — if this is only intended for internal caching, consider pub(crate) to avoid leaking it as public API.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 84.53333% with 116 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/vector/ivf/partition_serde.rs	84.53%	58 Missing and 58 partials ⚠️

📢 Thoughts on this report? Let us know!

[wjones127]

My big worry with this is that the DeepSizeOf will no longer be accurate after a roundtrip.

[wjones127]

This concern will be addressed by #6229. As long as we aren't sharing buffers across cache entries, we should be fine.

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.