Joint embeddings in uform v3 vs v2

[fbbdev]

Dear Unum team,

first of all, at the risk of sounding trite, thank you for the awesome software you are creating and for open sourcing it.

I am building a tool for navigating personal photographic archives, and I'd like to support offline semantic search (in the sense of off-the-web). Uform looks very promising for this use case.

My initial hypothesis, after experimenting with CLIP, was to perform a kNN query followed by reranking. This fits well with the general architecture as users will be offered a number of interactive refinement tools which would result in various reranking operations anyways.

I see however that joint embedding and matching scores have been removed from v3. I'd like to inquire about this difference:

• is it temporary?
• if not, is there a reason for giving up joint embeddings?
• v2 docs mention joint embeddings being finer grained and more accurate than unimodal similarity scores: is this still true w.r.t. v3 unimodal similarities?
• all in all, do you have any recommendations re choosing v2 over v3 for this use case?

Thank you!

UPDATE: after a deep dive through commit history it seems to me that model structure has not changed from v2 to v3, so it should be possible to reintroduce multimodal embedding functionality in a personal fork, at least for the torch implementation. What about the ONNX model though? Would it be possible and/or practical to reintroduce it there? I see it was removed from ONNX first, I'm curious about the reasons for this.

[ashvardanian]

Hey! We thought it's a very niche feature that would require a lot of support across all of the backends, and decided to deprecate it in v3. No thoughts on v4 for now.