Cannot load Qodo Embed 1 1.5b (upgrade to tokenizers 0.21.0)

[lightsofapollo]

System Info

Cannot load https://huggingface.co/Qodo/Qodo-Embed-1-1.5B when I investigated it there was an error reguarding the tokenizer... I was curious so I tried the latest version (0.21.0) of tokenizers and it worked fine.

The error message is:

tokenizer.json not found. text-embeddings-inference only supports fast tokenizers: Error("data did not match any variant of untagged enum ModelWrapper", line: 757321, column: 1)

When running tokenizers locally with the same version in the Cargo.toml here I also get this error locally.

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

text-embeddings-router --port 7997 --model-id Qodo/Qodo-Embed-1-1.5B

Expected behavior

Should load the tokenizer