Open
Description
System Info
version: text-embedding-inference 1.6.1
OS: ubuntu 24.04
python: 3.12.3
Embedding short (single-word) inputs randomly cause null vectors.
Information
- DockerThe CLI directly
Tasks
- An officially supported commandMy own modifications
Reproduction
curl 127.0.0.1:8082/embed \
-X POST \
-d '{"inputs":"trust"}' \
-H 'Content-Type: application/json'
[[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null]]
But the next invocation to the exact same URL yields vectors:
curl 127.0.0.1:8082/embed -X POST -d '{"inputs":"trust"}' -H 'Content-Type: application/json'
[[-0.001264368,0.021291267,-0.06020143,0.04895081,-0.04007768,-0.05485208,-0.023668757,-0.021429246,0.009828372,0.010762385,0.0037068669,-0.012025427,-0.07450882,-0.0008550736,-0.0038236186,0.07157941,0.037042134,0.0347071,-0.004489634,-0.039780494,0.04166975,-0.027914274,-0.022628605,-0.045299664,-0.06414976,0.08338195,-0.015283861,-0.02320175,-0.063512936,0.0035184722,0.013532585,-0.016854702,-0.018616593,0.011866219,0.05969197,-0.020452777,-0.009260533,-0.011186937,-0.07578248,-0.005890655,-0.0636403,-0.014180026,-0.042115528,-0.024135763,0.0034813238,-0.0088996645,-0.02969739,0.09688269,-0.054257706,-0.0020537688,-0.0353227,0.030652633,-0.041117832,0.018213268,-0.023753667,0.025006095,-0.02723499,-0.0156978,0.10027911,0.038846478,-0.046530865,-0.020866716,-0.022480013,-0.038995072,-0.0098496,-0.003529086,0.005503251,0.022182826,0.06431958,-0.025940109,-0.08244794,-0.0033088496,-0.027086398,0.0030833066,0.08308476,0.083636686,0.0006063129,0.014318006,-0.025727833,-0.044705294,0.00323986,-0.053493515,-0.022777198,0.011579648,-0.019794723,-0.008252223,-0.017066978,-0.07616457,0.03504674,0.040756963,0.012067881,0.031522963,-0.00562531,-0.056125734,-0.0196992,-0.023668757,-0.07892416,0.047464877,0.071027495,0.046106312,-0.024135763,-0.011813151,0.06873492,0.031586647,0.016504446,-0.011420441,0.003948331,0.007360665,0.03436746,-0.01070401,0.0025035283,0.016515061,-0.0011330224,-0.030886136,-0.054894533,-0.037997376,0.05790885,-0.044620384,0.007721534,-0.009440968,0.04154238,-0.0439411,-0.04542703,0.017342936,-0.16846211,-0.01883948,-0.05930987,-0.07493337,-0.07336253,0.0050415513,0.06308838,0.03935594,-0.028126549,-0.0632582,0.017300481,0.03472833,-0.014816854,0.1035057,-0.041945707,0.025473101,-0.12906371,-0.009504651,-0.06372521,0.04273113,-0.019179123,-0.035004288,0.0067715994,-0.08278758,0.04534212,0.008289372,-0.021906868,0.052177403,0.0004942047,0.039907858,0.0109428195,0.020272342,0.011367371,0.047592245,0.0350892,0.028657239,0.08206584,0.08427351,0.045299664,0.052517045,0.035131652,0.007159003,-0.053153872,0.04007768,0.06296101,-0.032839075,-0.09323155,0.0046514943,0.017162502,0.06512623,-0.015241406,-0.00088426156,-0.03837947,0.003998746,-0.059649512,-0.022246508,0.031119639,-0.045214754,-0.02874215,0.0098496,0.027723225,-0.006601779,-0.07637685,-0.03330608,-0.06355539,0.05977688,0.017077591,-0.054979444,-0.07068785,0.040013995,-0.032987665,-0.06835282,-0.048229072,0.0028391895,0.022373874,0.059734423,0.040650826,0.008061175,-0.025812743,0.016621199,-0.036320396,-0.012715323,0.028105322,-0.023944715,0.011738854,-0.069966115,0.027404811,0.06020143,-0.024305584,-0.015464296,-0.008766992,-0.03402782,-0.028784605,0.009971658,0.020930398,0.0004225616,0.007907275,-0.032817844,0.0010573991,0.007705613,0.019624902,-0.007159003,-0.006644234,0.031565417,0.0936561,-0.08745765,-0.035726026,-0.028614784,-0.001178131,-0.0006603769,0.006214375,0.010794227,-0.01125062,-0.010215775,-0.035386384,0.04912063,0.049078174,-0.03479201,-0.010051262,0.033157486,0.06045616,0.01916851,-0.0057314476,0.018425543,0.008830675,-0.03731809,0.02407208,0.018085903,-0.042221665,0.01017332,0.03190506,-0.061559994,0.04207307,0.0036803326,0.013861612,0.0007688366,0.014551509,0.0049486808,0.043474093,0.015803937,-0.046445955,-0.010815455,0.056083277,0.05553136,0.025112232,0.006867124,-0.034091502,0.012354454,-0.0012411503,0.018085903,0.032456975,-0.03988663,-0.004250824,0.0402475,-0.022522466,-0.019667357,-0.024623998,0.029994577,-0.03109841,0.022607377,-0.041945707,0.079688355,-0.0049725617,-0.003133722,-0.0006225653,0.004850503,0.023689983,0.003369879,0.021259425,-0.006532789,0.05069147,0.019805336,0.03619303,-0.009287069,0.009563027,0.069371745,-0.0048345826,-0.032372065,0.036086895,-0.02821146,-0.028699694,0.048908353,0.011898061,-0.025430646,-0.003062079,-0.0058322786,-0.056974836,-0.003322117,0.020665053,-0.0155916605,-0.031438053,-0.025006095,0.051328298,-0.0053732325,-0.055998366,-0.011590261,0.010316606,-0.012471206,-0.020081295,0.037997376,-0.051795308,0.007116548,-0.024645226,-0.0097116195,-0.037339322,-0.029845985,0.027001487,0.034388687,0.031862605,-0.016769791,0.06822546,0.0002645156,0.006792827,0.040799417,-0.023711212,0.0122058615,0.03392168,-0.016674267,-0.033412218,0.0114735095,0.027701998,-0.021227585,0.05536154,-0.047252603,-0.012216475,-0.03893139,-0.013575041,0.030716315,-0.014551509,-0.0316291,-0.004349001,-0.028508646,-0.0021532732,0.029676164,-0.0075994753,-0.0042534773,0.0027569325,0.011314303,-0.022947019,0.035195336,-0.0071112406,0.014700103,-0.065423414,-0.015581047,-0.0046674153,0.0073341304,-0.016069282,0.008406123,0.033412218,0.0018202653,-0.026852895,0.0069095786,-0.011569033,0.0020511153,-0.0013134568,0.031607874,0.022097915,-0.0036458375,0.055573817,0.009547106,0.0034123342,-0.041330107,-0.001278962,-0.0050680856,0.025282053,0.017194344,0.022097915,0.031756468,-0.017693192,0.0007177577,0.046106312,-0.0010912305,-0.040990464,0.033369765,0.0011011809,0.007960344,0.019179123,0.05446998,-0.042858493,0.021843184,-0.053281236,0.030546494,-0.0057526752,0.034070272,0.017088205,-0.003096574,0.011155096,-0.032053653,0.036935996,0.0035901153,0.06720653,0.043601457,0.032287154,0.0066760755,0.089070946,-0.06826791,-0.0038740342,0.060413707,0.014222481,-0.061475083,-0.0039350633,0.03392168,0.00052704115,0.033688176,-0.019688584,0.03927103,-0.036086895,0.011218779,-0.0350892,-0.044535473,-0.01203604,-0.010263537,-0.013904068,-0.011707013,-0.018903164,-0.03145928,-0.02540942,-0.005773903,0.05905514,-0.010417437,0.035216562,-0.015039744,-0.018860709,0.008215075,0.0056306166,0.0030859602,-0.01743846,0.033688176,0.033709403,0.03428255,-0.03876157,0.010964047,0.0024398456,0.040438548,-0.004364922,-0.012747165,0.0012312,-0.013596267,-0.016334627,0.00060133764,0.05922496,0.053493515,0.031353142,-0.056720104,0.02882706,-0.008220382,-0.030334217,-0.0052379067,-0.0028763376,-0.009870827,-0.033857998,0.04050223,-0.015878234,0.03428255,0.037487913,-0.01010433,0.10138294,0.035386384,0.029909667,-0.022480013,-0.030864907,0.008315906,-0.011982972,-0.033624493,0.03131069,0.022161597,0.03780633,-0.0031655636,0.026916577,-0.0019290567,0.013383992,0.013086806,0.019242804,0.018648433,-0.00018607304,0.031183321,-0.01278962,0.008093016,-0.025685377,-0.001749949,-0.011059572,-0.058842864,0.011038344,0.03708459,-0.02345648,-0.05281423,-0.05396052,-0.020484619,-0.040841874,-0.035365157,-0.038082287,-0.01713066,-0.037339322,0.0010222408,0.0042189825,-0.0069838753,-0.056805015,-0.0035450065,0.010560723,0.0077852164,0.023286661,0.012832074,0.00028524565,0.038846478,0.03621426,-0.05854568,0.028954426,0.047422424,0.04712524,-0.008655547,0.03058895,0.032053653,-0.03823088,0.03313626,-0.008809447,-0.027956728,-0.06635743,0.009064179,-0.015888847,-0.04220044,-0.016663654,0.022034233,0.074806005,0.0114416685,0.017342936,-0.051625486,0.020675667,0.027595859,-0.028084094,0.00524056,-0.063343115,-0.028126549,0.00025854533,-0.009833679,0.020282958,0.033284854,0.013118647,0.017332323,0.015305089,-0.02689535,0.006999796,-0.02678921,-0.007917889,-0.023244206,0.026937805,-0.01648322,0.04704033,-0.0178524,0.039865404,-0.0042800116,0.025239598,-0.017693192,-0.036787406,0.0027038637,-0.045639306,-0.0064584925,-0.08019781,0.0010832702,0.063300654,0.0094144335,0.006453186,0.022692287,-0.000021414155,0.0017764835,0.05090375,0.057866395,-0.01718373,0.019826563,0.029103018,-0.00047098703,0.031480506,-0.019104825,0.03305135,-0.048823446,-0.022968246,0.028466191,-0.0019940662,-0.023222977,0.028614784,-0.030440357,-0.007944424,-0.020049453,-0.029103018,-0.040438548,0.033475902,0.058757953,-0.053153872,-0.01903053,0.004595772,0.0402475,0.022161597,-0.05264441,0.013925295,-0.0001945309,0.037148274,-0.04198816,-0.01371302,-0.050394285,0.026088702,-0.035471294,0.020898556,-0.037233185,-0.0011270521,-0.06924438,-0.004715177,-0.013118647,0.038273335,-0.0015708413,-0.019094212,-0.041903254,0.021429246,0.01155842,-0.012227088,-0.002552617,0.0038289255,0.0004895612,0.037403002,0.03691477,-0.0041367253,-0.038995072,-0.046191223,-0.0024305584,0.02197055,0.008193848,-0.020378482,0.009101327,-0.021864412,-0.004139379,-0.020824261,0.030567722,-0.01849984,-0.053451058,-0.0032345532,-0.0021373525,-0.015188336,0.022840882,0.006230296,0.010348448,0.032393295,0.024241902,-0.02012375,0.046573322,-0.03523779,-0.0024637266,0.050309375,0.0059437235,-0.028551102,-0.019847792,0.0041075377,0.017703805,-0.01709882,0.03417641,-0.030100714,-0.044153377,-0.020887943,-0.0002665057,-0.015803937,0.0045931186,-0.013415833,0.021333722,-0.07399936,-0.03927103,0.07293798,-0.005938417,0.037466686,0.07726841,0.032923985,0.046191223,0.020017613,-0.035959527,0.051710397,-0.01774626,-0.012439365,-0.0017101472,0.0040040533,-0.020389095,-0.023435254,-0.003937717,0.043176908,0.009021724,0.0009983599,-0.043155678,0.032287154,0.03954699,-0.006739758,0.06809809,-0.0060020997,-0.009769996,-0.007705613,0.03540761,-0.0051715705,-0.036235485,-0.0065858583,-0.019274646,0.020060068,0.26644865,0.0012265564,0.035789706,0.026025018,0.014328619,-0.004073043,0.018022219,0.013999592,0.080707274,-0.023838578,-0.033879224,-0.030334217,0.028805831,0.037148274,0.01025823,-0.017332323,0.01827695,-0.028444963,0.01911544,0.01225893,-0.034091502,0.0006600452,0.027362356,-0.029824756,0.027935501,-0.022798426,0.0038819944,-0.015071585,0.013755475,0.032456975,0.00043251202,-0.059904244,-0.02364753,0.018627206,0.026661847,0.044238288,-0.012396909,-0.03784878,-0.05281423,0.033263624,-0.04704033,0.016982067,-0.025473101,0.022628605,-0.016865317,0.009722234,-0.016504446,-0.04623368,-0.04094801]]
This seems to be happening for all models.
When inputs have > ~5 words, they consistently return vectors.
Expected behavior
Expected to return vectors regardless of input length.
Metadata
Metadata
Assignees
Labels
No labels
Activity
alvarobartt commentedon Apr 4, 2025
Hey @superkelvint, thanks for reporting! Do you see this happening with a specific model / architecture type? Could you share the model IDs or architecture types of the models you tried that ran into that issue on empty / null embeddings? In order to have a reproducer could you share which device are you using GPU, CPU or MPS, which architecture and model ID from the Hugging Face Hub? Thanks in advance 🤗
e.g. I tried
ibm-granite/granite-embedding-125m-english
and didn't see the issue happening!superkelvint commentedon Apr 4, 2025
@alvarobartt Thanks for your reply.
I'm using https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v2.0 on turing-1.6. Unfortunately this is the only GPU I have, otherwise I would try it on another gpu architecture. I can confirm this doesn't happen at all on cpu-1.6.
superkelvint commentedon Apr 4, 2025
I just tried using ibm-granite/granite-embedding-125m-english on turing-1.6 and I'm not seeing the issue either.
alvarobartt commentedon Apr 7, 2025
Hey @superkelvint thanks for the information, I'll try to reproduce on my own before closing the issue then, but please let us know if this happens ever again or if you happen to have a consistent reproducer 🤗 Thanks again!
vrdn-23 commentedon Apr 22, 2025
This has been a long-time know issue first documented in #53. The solution would be to turn off Flash Attention for that model!
cc @superkelvint @alvarobartt