Short inputs cause /embed to randomly return empty vectors.

[superkelvint]

System Info

version: text-embedding-inference 1.6.1
OS: ubuntu 24.04
python: 3.12.3

Embedding short (single-word) inputs randomly cause null vectors.

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

curl 127.0.0.1:8082/embed \
    -X POST \
    -d '{"inputs":"trust"}' \
    -H 'Content-Type: application/json'
[[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null]]

But the next invocation to the exact same URL yields vectors:

curl 127.0.0.1:8082/embed     -X POST     -d '{"inputs":"trust"}'     -H 'Content-Type: application/json'
[[-0.001264368,0.021291267,-0.06020143,0.04895081,-0.04007768,-0.05485208,-0.023668757,-0.021429246,0.009828372,0.010762385,0.0037068669,-0.012025427,-0.07450882,-0.0008550736,-0.0038236186,0.07157941,0.037042134,0.0347071,-0.004489634,-0.039780494,0.04166975,-0.027914274,-0.022628605,-0.045299664,-0.06414976,0.08338195,-0.015283861,-0.02320175,-0.063512936,0.0035184722,0.013532585,-0.016854702,-0.018616593,0.011866219,0.05969197,-0.020452777,-0.009260533,-0.011186937,-0.07578248,-0.005890655,-0.0636403,-0.014180026,-0.042115528,-0.024135763,0.0034813238,-0.0088996645,-0.02969739,0.09688269,-0.054257706,-0.0020537688,-0.0353227,0.030652633,-0.041117832,0.018213268,-0.023753667,0.025006095,-0.02723499,-0.0156978,0.10027911,0.038846478,-0.046530865,-0.020866716,-0.022480013,-0.038995072,-0.0098496,-0.003529086,0.005503251,0.022182826,0.06431958,-0.025940109,-0.08244794,-0.0033088496,-0.027086398,0.0030833066,0.08308476,0.083636686,0.0006063129,0.014318006,-0.025727833,-0.044705294,0.00323986,-0.053493515,-0.022777198,0.011579648,-0.019794723,-0.008252223,-0.017066978,-0.07616457,0.03504674,0.040756963,0.012067881,0.031522963,-0.00562531,-0.056125734,-0.0196992,-0.023668757,-0.07892416,0.047464877,0.071027495,0.046106312,-0.024135763,-0.011813151,0.06873492,0.031586647,0.016504446,-0.011420441,0.003948331,0.007360665,0.03436746,-0.01070401,0.0025035283,0.016515061,-0.0011330224,-0.030886136,-0.054894533,-0.037997376,0.05790885,-0.044620384,0.007721534,-0.009440968,0.04154238,-0.0439411,-0.04542703,0.017342936,-0.16846211,-0.01883948,-0.05930987,-0.07493337,-0.07336253,0.0050415513,0.06308838,0.03935594,-0.028126549,-0.0632582,0.017300481,0.03472833,-0.014816854,0.1035057,-0.041945707,0.025473101,-0.12906371,-0.009504651,-0.06372521,0.04273113,-0.019179123,-0.035004288,0.0067715994,-0.08278758,0.04534212,0.008289372,-0.021906868,0.052177403,0.0004942047,0.039907858,0.0109428195,0.020272342,0.011367371,0.047592245,0.0350892,0.028657239,0.08206584,0.08427351,0.045299664,0.052517045,0.035131652,0.007159003,-0.053153872,0.04007768,0.06296101,-0.032839075,-0.09323155,0.0046514943,0.017162502,0.06512623,-0.015241406,-0.00088426156,-0.03837947,0.003998746,-0.059649512,-0.022246508,0.031119639,-0.045214754,-0.02874215,0.0098496,0.027723225,-0.006601779,-0.07637685,-0.03330608,-0.06355539,0.05977688,0.017077591,-0.054979444,-0.07068785,0.040013995,-0.032987665,-0.06835282,-0.048229072,0.0028391895,0.022373874,0.059734423,0.040650826,0.008061175,-0.025812743,0.016621199,-0.036320396,-0.012715323,0.028105322,-0.023944715,0.011738854,-0.069966115,0.027404811,0.06020143,-0.024305584,-0.015464296,-0.008766992,-0.03402782,-0.028784605,0.009971658,0.020930398,0.0004225616,0.007907275,-0.032817844,0.0010573991,0.007705613,0.019624902,-0.007159003,-0.006644234,0.031565417,0.0936561,-0.08745765,-0.035726026,-0.028614784,-0.001178131,-0.0006603769,0.006214375,0.010794227,-0.01125062,-0.010215775,-0.035386384,0.04912063,0.049078174,-0.03479201,-0.010051262,0.033157486,0.06045616,0.01916851,-0.0057314476,0.018425543,0.008830675,-0.03731809,0.02407208,0.018085903,-0.042221665,0.01017332,0.03190506,-0.061559994,0.04207307,0.0036803326,0.013861612,0.0007688366,0.014551509,0.0049486808,0.043474093,0.015803937,-0.046445955,-0.010815455,0.056083277,0.05553136,0.025112232,0.006867124,-0.034091502,0.012354454,-0.0012411503,0.018085903,0.032456975,-0.03988663,-0.004250824,0.0402475,-0.022522466,-0.019667357,-0.024623998,0.029994577,-0.03109841,0.022607377,-0.041945707,0.079688355,-0.0049725617,-0.003133722,-0.0006225653,0.004850503,0.023689983,0.003369879,0.021259425,-0.006532789,0.05069147,0.019805336,0.03619303,-0.009287069,0.009563027,0.069371745,-0.0048345826,-0.032372065,0.036086895,-0.02821146,-0.028699694,0.048908353,0.011898061,-0.025430646,-0.003062079,-0.0058322786,-0.056974836,-0.003322117,0.020665053,-0.0155916605,-0.031438053,-0.025006095,0.051328298,-0.0053732325,-0.055998366,-0.011590261,0.010316606,-0.012471206,-0.020081295,0.037997376,-0.051795308,0.007116548,-0.024645226,-0.0097116195,-0.037339322,-0.029845985,0.027001487,0.034388687,0.031862605,-0.016769791,0.06822546,0.0002645156,0.006792827,0.040799417,-0.023711212,0.0122058615,0.03392168,-0.016674267,-0.033412218,0.0114735095,0.027701998,-0.021227585,0.05536154,-0.047252603,-0.012216475,-0.03893139,-0.013575041,0.030716315,-0.014551509,-0.0316291,-0.004349001,-0.028508646,-0.0021532732,0.029676164,-0.0075994753,-0.0042534773,0.0027569325,0.011314303,-0.022947019,0.035195336,-0.0071112406,0.014700103,-0.065423414,-0.015581047,-0.0046674153,0.0073341304,-0.016069282,0.008406123,0.033412218,0.0018202653,-0.026852895,0.0069095786,-0.011569033,0.0020511153,-0.0013134568,0.031607874,0.022097915,-0.0036458375,0.055573817,0.009547106,0.0034123342,-0.041330107,-0.001278962,-0.0050680856,0.025282053,0.017194344,0.022097915,0.031756468,-0.017693192,0.0007177577,0.046106312,-0.0010912305,-0.040990464,0.033369765,0.0011011809,0.007960344,0.019179123,0.05446998,-0.042858493,0.021843184,-0.053281236,0.030546494,-0.0057526752,0.034070272,0.017088205,-0.003096574,0.011155096,-0.032053653,0.036935996,0.0035901153,0.06720653,0.043601457,0.032287154,0.0066760755,0.089070946,-0.06826791,-0.0038740342,0.060413707,0.014222481,-0.061475083,-0.0039350633,0.03392168,0.00052704115,0.033688176,-0.019688584,0.03927103,-0.036086895,0.011218779,-0.0350892,-0.044535473,-0.01203604,-0.010263537,-0.013904068,-0.011707013,-0.018903164,-0.03145928,-0.02540942,-0.005773903,0.05905514,-0.010417437,0.035216562,-0.015039744,-0.018860709,0.008215075,0.0056306166,0.0030859602,-0.01743846,0.033688176,0.033709403,0.03428255,-0.03876157,0.010964047,0.0024398456,0.040438548,-0.004364922,-0.012747165,0.0012312,-0.013596267,-0.016334627,0.00060133764,0.05922496,0.053493515,0.031353142,-0.056720104,0.02882706,-0.008220382,-0.030334217,-0.0052379067,-0.0028763376,-0.009870827,-0.033857998,0.04050223,-0.015878234,0.03428255,0.037487913,-0.01010433,0.10138294,0.035386384,0.029909667,-0.022480013,-0.030864907,0.008315906,-0.011982972,-0.033624493,0.03131069,0.022161597,0.03780633,-0.0031655636,0.026916577,-0.0019290567,0.013383992,0.013086806,0.019242804,0.018648433,-0.00018607304,0.031183321,-0.01278962,0.008093016,-0.025685377,-0.001749949,-0.011059572,-0.058842864,0.011038344,0.03708459,-0.02345648,-0.05281423,-0.05396052,-0.020484619,-0.040841874,-0.035365157,-0.038082287,-0.01713066,-0.037339322,0.0010222408,0.0042189825,-0.0069838753,-0.056805015,-0.0035450065,0.010560723,0.0077852164,0.023286661,0.012832074,0.00028524565,0.038846478,0.03621426,-0.05854568,0.028954426,0.047422424,0.04712524,-0.008655547,0.03058895,0.032053653,-0.03823088,0.03313626,-0.008809447,-0.027956728,-0.06635743,0.009064179,-0.015888847,-0.04220044,-0.016663654,0.022034233,0.074806005,0.0114416685,0.017342936,-0.051625486,0.020675667,0.027595859,-0.028084094,0.00524056,-0.063343115,-0.028126549,0.00025854533,-0.009833679,0.020282958,0.033284854,0.013118647,0.017332323,0.015305089,-0.02689535,0.006999796,-0.02678921,-0.007917889,-0.023244206,0.026937805,-0.01648322,0.04704033,-0.0178524,0.039865404,-0.0042800116,0.025239598,-0.017693192,-0.036787406,0.0027038637,-0.045639306,-0.0064584925,-0.08019781,0.0010832702,0.063300654,0.0094144335,0.006453186,0.022692287,-0.000021414155,0.0017764835,0.05090375,0.057866395,-0.01718373,0.019826563,0.029103018,-0.00047098703,0.031480506,-0.019104825,0.03305135,-0.048823446,-0.022968246,0.028466191,-0.0019940662,-0.023222977,0.028614784,-0.030440357,-0.007944424,-0.020049453,-0.029103018,-0.040438548,0.033475902,0.058757953,-0.053153872,-0.01903053,0.004595772,0.0402475,0.022161597,-0.05264441,0.013925295,-0.0001945309,0.037148274,-0.04198816,-0.01371302,-0.050394285,0.026088702,-0.035471294,0.020898556,-0.037233185,-0.0011270521,-0.06924438,-0.004715177,-0.013118647,0.038273335,-0.0015708413,-0.019094212,-0.041903254,0.021429246,0.01155842,-0.012227088,-0.002552617,0.0038289255,0.0004895612,0.037403002,0.03691477,-0.0041367253,-0.038995072,-0.046191223,-0.0024305584,0.02197055,0.008193848,-0.020378482,0.009101327,-0.021864412,-0.004139379,-0.020824261,0.030567722,-0.01849984,-0.053451058,-0.0032345532,-0.0021373525,-0.015188336,0.022840882,0.006230296,0.010348448,0.032393295,0.024241902,-0.02012375,0.046573322,-0.03523779,-0.0024637266,0.050309375,0.0059437235,-0.028551102,-0.019847792,0.0041075377,0.017703805,-0.01709882,0.03417641,-0.030100714,-0.044153377,-0.020887943,-0.0002665057,-0.015803937,0.0045931186,-0.013415833,0.021333722,-0.07399936,-0.03927103,0.07293798,-0.005938417,0.037466686,0.07726841,0.032923985,0.046191223,0.020017613,-0.035959527,0.051710397,-0.01774626,-0.012439365,-0.0017101472,0.0040040533,-0.020389095,-0.023435254,-0.003937717,0.043176908,0.009021724,0.0009983599,-0.043155678,0.032287154,0.03954699,-0.006739758,0.06809809,-0.0060020997,-0.009769996,-0.007705613,0.03540761,-0.0051715705,-0.036235485,-0.0065858583,-0.019274646,0.020060068,0.26644865,0.0012265564,0.035789706,0.026025018,0.014328619,-0.004073043,0.018022219,0.013999592,0.080707274,-0.023838578,-0.033879224,-0.030334217,0.028805831,0.037148274,0.01025823,-0.017332323,0.01827695,-0.028444963,0.01911544,0.01225893,-0.034091502,0.0006600452,0.027362356,-0.029824756,0.027935501,-0.022798426,0.0038819944,-0.015071585,0.013755475,0.032456975,0.00043251202,-0.059904244,-0.02364753,0.018627206,0.026661847,0.044238288,-0.012396909,-0.03784878,-0.05281423,0.033263624,-0.04704033,0.016982067,-0.025473101,0.022628605,-0.016865317,0.009722234,-0.016504446,-0.04623368,-0.04094801]]

This seems to be happening for all models.

When inputs have > ~5 words, they consistently return vectors.

Expected behavior

Expected to return vectors regardless of input length.

[alvarobartt]

Hey @superkelvint, thanks for reporting! Do you see this happening with a specific model / architecture type? Could you share the model IDs or architecture types of the models you tried that ran into that issue on empty / null embeddings? In order to have a reproducer could you share which device are you using GPU, CPU or MPS, which architecture and model ID from the Hugging Face Hub? Thanks in advance 🤗

e.g. I tried ibm-granite/granite-embedding-125m-english and didn't see the issue happening!

[superkelvint]

@alvarobartt Thanks for your reply.

I'm using https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v2.0 on turing-1.6. Unfortunately this is the only GPU I have, otherwise I would try it on another gpu architecture. I can confirm this doesn't happen at all on cpu-1.6.

[superkelvint]

I just tried using ibm-granite/granite-embedding-125m-english on turing-1.6 and I'm not seeing the issue either.

[alvarobartt]

Hey @superkelvint thanks for the information, I'll try to reproduce on my own before closing the issue then, but please let us know if this happens ever again or if you happen to have a consistent reproducer 🤗 Thanks again!

[vrdn-23]

This has been a long-time know issue first documented in #53. The solution would be to turn off Flash Attention for that model!
cc @superkelvint @alvarobartt