required memory

[agoriwmt]

I’m currently using version 1.2.5.290. I’ve created an index, and the folder size is around 800-900 MB. The dataset consists of roughly 10,000 video transcripts, each averaging about 10 minutes in length, so substantial, but not massive.

When I start FastPlaid using search.FastPlaid(index_path), the memory usage exceeds 5 GB of RAM, eventually causing my current EC2 instance to crash. I can try running it on a larger instance, but I’m starting to worry about FastPlaid’s scalability.

Does memory usage scale linearly with data size? Ultimately, I’ll need to index more than 30 times this amount of data, and running an instance with 100 GB of RAM isn’t feasible due to cost constraints.

The embedding dimension is 128.

Do you have any recommendations or advice?

[agoriwmt]

A related question is whether it would make sense to split the index into multiple shards. Each document could be stored in a single shard, while queries would run concurrently across all shards (actually in my case for some of these queries I know in advance what shard I should go to). Obviously this approach would be only viable if the resulting scores are comparable across shards.

From what I understand, the scores should indeed be comparable, since they are simply the sum of the maxsim values for each query token, right?

[raphaelsty]

Hi @agoriwmt, thank you for this feedback.

While version 1.2.5.290 did reduce memory usage, it unfortunately remains high. We do store the set of centroids in memory.

In the next version of FastPlaid, I plan to add a low_memory parameter. This will use memory mapping to avoid loading the entire set of centroids into memory at once. It might take a bit of time until this is ready. Also it will be slower, we can expect 50% slower time to answer I think so compared to the in-memory version.

At some point you could maybe also consider to create your index with a reduced set of centroids. It might help there, might reduce a bit the accuracy. You can rely on the update method to update an existing index without updating the set of centroids. You would create your index with a bit of each sequence to get a representative index and then in the end add back the big part of the sequence with the update method. Not sure how it will perform :)

Sharding might help as you will only store 1/n_shards of the centroids within memory.

But you should set preload_index to False when initialising your FastPlaid object before doing search when doing sharding :)

You could consider also pooling, I don't know if it's compatible with ColPali yet but you can divide by twice the number of embeddings with this method without hurting actual accuracy of the index.

[agoriwmt]

Hi, many thanks for the prompt reply.

I hadn't realized the number of centroids was that relevant. This makes me think I could try increasing the max_points_per_centroid parameter (which defaults to 256). Doing so should reduce the total number of centroids but increase the size of the posting lists, (although it should increase the scanning time too).

That said, I feel I might be missing something, especially regarding the actual memory usage. My corpus should contain roughly 12 million tokens. Assuming max_points_per_centroid=256, 128-dimensional embeddings, and fp32 precision (4 bytes), I’d expect the memory footprint to be around 20 MB...so I’m not sure where the reported 5 GB comes from :/

[raphaelsty]

There are the residuals (quantized distances to the centroids it belong to) as well which we do store in memory. I'm sure we can drastically reduce memory but I have quite some stuff to do in the following days. It will need to wait a bit but I'll make something to reduce memory usage :).

We would like to write incrementally the index within create / update method and also add memory mapping in search.

[agoriwmt]

Not blaming anyone, of course, I still think this is a great project with a lot of potential ahead :)
Just wanted to share that I'm moving to a different strategy. The reasons are:

• High memory consumption.

• Scalability issues: updates become unfeasible even with a GPU once you reach around 20,000 chunks (at ~150 tokens each). Beyond that, the index becomes hard to manage, and even a single update can take 30 seconds or more.

• Limitations for reranking: it seems filters are applied after the search, so if a subset ID isn’t part of the selected centroids, no results are returned (at least that’s my understanding).

• Potential concurrency issues: when updating and searching at the same time, the index sometimes breaks, I've seen odd errors during both processes. I assume it's a concurrency problem, though I haven’t had time to investigate further.

It’s a shame because the accuracy is excellent. My hope is to get close to PLAID performance with a first step using single vectors and a subsequent reranking with ColBERT.

I'll definitely keep an eye on this great project and hope I can use it again in the future!

[raphaelsty]

Thank you for your feedback @agoriwmt, this is very helpful as it brings a lot of ideas to improve Fast Plaid.

Concurrency writing / searching would be possible to add soon by putting a lock on some files when writing, this is a good point.

For the size of the index in memory, can't do much better yet buy I'm sure we might improve by trading at least 50% of speed. I need to refactor few things and I don't have the time to do so yet but I will look into it.

For exact filtering this would be possible I think so.

Don't hesitate to try Muvera, it will not work with every existing colbert but it will consumme much less memory and be compatible with every vector databases. Given your constraint I think this it what you need.

Weaviate has also good tutorials with their multi-vector databases.

Thank's again for the honest feedback, and for hilighting key points that need to be improved @agoriwmt

Fast-Plaid right now really shine on static indexes re-created from scratch once or twice daily and reasonnable amount of embeddings :)

[agoriwmt]

Thank you as well for all the hard work!

Regarding Muvera, I gave it a try, but honestly, it just doesn't seem to work as expected. I'm not sure why. They claim near ColBERT-level performance, but in my tests, the results were actually worse than strong single vector models like bge-m3...despite using 40k dimensions compared to 1024! That’s even higher than the multivector representations I use for my chunks, so I’m not sure what’s going wrong there. I tested Muvera with jina-colbert-v2, which from what I've read, should be a compatible setup.

As for Weaviate, I'm still unclear on how they handle ColBERT. From my understanding, it’s served through their HNSW index, which I worry won't scale to the level I need.

That’s why I decided to go with a single vector retrieval (I would have preferred starting with Muvera) combined with ColBERT for reranking (all using Qdrant). The hope is that the first step provides sufficiently high recall for my use case.