Tokenizer Warning when Loading DPR

[BukuBukuChagma]

So, I'm building a LFQA pipeline, and when the load the models inside DPR it gives some tokenizer warning(mismatch tokenizer). I wanted to ask if it's something that might cause problem in the retrieving process?
If yes, then what's the solution to it? Also curious what's actually causing this problem?, as I'm totally following the tutorial in your docs.

Code:

retriever = DensePassageRetriever(
    document_store=document_store,
    query_embedding_model="vblagoje/dpr-question_encoder-single-lfqa-wiki",
    passage_embedding_model="vblagoje/dpr-ctx_encoder-single-lfqa-wiki",
    use_gpu=True,
    
)

document_store.update_embeddings(retriever)

Warnings:

haystack.modeling.model.language_model - Could not find vblagoje/dpr-question_encoder-single-lfqa-wiki locally.
INFO - haystack.modeling.model.language_model - Looking on Transformers Model Hub (in local cache and online)...
INFO - haystack.modeling.model.language_model - Loaded vblagoje/dpr-question_encoder-single-lfqa-wiki
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is 'DPRQuestionEncoderTokenizer'.
The class this function is called from is 'DPRContextEncoderTokenizerFast'.

[sjrl]

Hi, @Zeni69 I'm not entirely sure what is causing this error to be thrown. It is an error that comes from the HuggingFace library and it's unclear why there is a mismatch. You can check that the Tokenizer classes for both the query_embeddingmodel and the passage_embedding_model are correct by running the following:

type(retriever.query_tokenizer)
# transformers.models.dpr.tokenization_dpr_fast.DPRQuestionEncoderTokenizerFast

gives the expected DPRQuestionEncoderTokenizerFast tokenizer for the query_embedding_model and

type(retriever.passage_tokenizer)
# transformers.models.dpr.tokenization_dpr_fast.DPRContextEncoderTokenizerFast

gives the expected DPRContextEncoderTokenizerFast tokenizer for the passage_embedding_model.

So to answer your question you are doing everything correctly and everything is being loaded properly despite the warning being thrown by HuggingFace. We will have to investigate further why this might be occurring.

[ZanSara]

You've probably hit this known issue in HF Transformers: huggingface/transformers#12926. It's just a misleading log message, nothing wrong is actually happening behind the scenes.