Skip to content

Short inputs cause /embed to randomly return empty vectors. #557

Open
@superkelvint

Description

@superkelvint

System Info

version: text-embedding-inference 1.6.1
OS: ubuntu 24.04
python: 3.12.3

Embedding short (single-word) inputs randomly cause null vectors.

Information

  • Docker
    The CLI directly

Tasks

  • An officially supported command
    My own modifications

Reproduction

curl 127.0.0.1:8082/embed \
    -X POST \
    -d '{"inputs":"trust"}' \
    -H 'Content-Type: application/json'
[[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null]]

But the next invocation to the exact same URL yields vectors:

curl 127.0.0.1:8082/embed     -X POST     -d '{"inputs":"trust"}'     -H 'Content-Type: application/json'
[[-0.001264368,0.021291267,-0.06020143,0.04895081,-0.04007768,-0.05485208,-0.023668757,-0.021429246,0.009828372,0.010762385,0.0037068669,-0.012025427,-0.07450882,-0.0008550736,-0.0038236186,0.07157941,0.037042134,0.0347071,-0.004489634,-0.039780494,0.04166975,-0.027914274,-0.022628605,-0.045299664,-0.06414976,0.08338195,-0.015283861,-0.02320175,-0.063512936,0.0035184722,0.013532585,-0.016854702,-0.018616593,0.011866219,0.05969197,-0.020452777,-0.009260533,-0.011186937,-0.07578248,-0.005890655,-0.0636403,-0.014180026,-0.042115528,-0.024135763,0.0034813238,-0.0088996645,-0.02969739,0.09688269,-0.054257706,-0.0020537688,-0.0353227,0.030652633,-0.041117832,0.018213268,-0.023753667,0.025006095,-0.02723499,-0.0156978,0.10027911,0.038846478,-0.046530865,-0.020866716,-0.022480013,-0.038995072,-0.0098496,-0.003529086,0.005503251,0.022182826,0.06431958,-0.025940109,-0.08244794,-0.0033088496,-0.027086398,0.0030833066,0.08308476,0.083636686,0.0006063129,0.014318006,-0.025727833,-0.044705294,0.00323986,-0.053493515,-0.022777198,0.011579648,-0.019794723,-0.008252223,-0.017066978,-0.07616457,0.03504674,0.040756963,0.012067881,0.031522963,-0.00562531,-0.056125734,-0.0196992,-0.023668757,-0.07892416,0.047464877,0.071027495,0.046106312,-0.024135763,-0.011813151,0.06873492,0.031586647,0.016504446,-0.011420441,0.003948331,0.007360665,0.03436746,-0.01070401,0.0025035283,0.016515061,-0.0011330224,-0.030886136,-0.054894533,-0.037997376,0.05790885,-0.044620384,0.007721534,-0.009440968,0.04154238,-0.0439411,-0.04542703,0.017342936,-0.16846211,-0.01883948,-0.05930987,-0.07493337,-0.07336253,0.0050415513,0.06308838,0.03935594,-0.028126549,-0.0632582,0.017300481,0.03472833,-0.014816854,0.1035057,-0.041945707,0.025473101,-0.12906371,-0.009504651,-0.06372521,0.04273113,-0.019179123,-0.035004288,0.0067715994,-0.08278758,0.04534212,0.008289372,-0.021906868,0.052177403,0.0004942047,0.039907858,0.0109428195,0.020272342,0.011367371,0.047592245,0.0350892,0.028657239,0.08206584,0.08427351,0.045299664,0.052517045,0.035131652,0.007159003,-0.053153872,0.04007768,0.06296101,-0.032839075,-0.09323155,0.0046514943,0.017162502,0.06512623,-0.015241406,-0.00088426156,-0.03837947,0.003998746,-0.059649512,-0.022246508,0.031119639,-0.045214754,-0.02874215,0.0098496,0.027723225,-0.006601779,-0.07637685,-0.03330608,-0.06355539,0.05977688,0.017077591,-0.054979444,-0.07068785,0.040013995,-0.032987665,-0.06835282,-0.048229072,0.0028391895,0.022373874,0.059734423,0.040650826,0.008061175,-0.025812743,0.016621199,-0.036320396,-0.012715323,0.028105322,-0.023944715,0.011738854,-0.069966115,0.027404811,0.06020143,-0.024305584,-0.015464296,-0.008766992,-0.03402782,-0.028784605,0.009971658,0.020930398,0.0004225616,0.007907275,-0.032817844,0.0010573991,0.007705613,0.019624902,-0.007159003,-0.006644234,0.031565417,0.0936561,-0.08745765,-0.035726026,-0.028614784,-0.001178131,-0.0006603769,0.006214375,0.010794227,-0.01125062,-0.010215775,-0.035386384,0.04912063,0.049078174,-0.03479201,-0.010051262,0.033157486,0.06045616,0.01916851,-0.0057314476,0.018425543,0.008830675,-0.03731809,0.02407208,0.018085903,-0.042221665,0.01017332,0.03190506,-0.061559994,0.04207307,0.0036803326,0.013861612,0.0007688366,0.014551509,0.0049486808,0.043474093,0.015803937,-0.046445955,-0.010815455,0.056083277,0.05553136,0.025112232,0.006867124,-0.034091502,0.012354454,-0.0012411503,0.018085903,0.032456975,-0.03988663,-0.004250824,0.0402475,-0.022522466,-0.019667357,-0.024623998,0.029994577,-0.03109841,0.022607377,-0.041945707,0.079688355,-0.0049725617,-0.003133722,-0.0006225653,0.004850503,0.023689983,0.003369879,0.021259425,-0.006532789,0.05069147,0.019805336,0.03619303,-0.009287069,0.009563027,0.069371745,-0.0048345826,-0.032372065,0.036086895,-0.02821146,-0.028699694,0.048908353,0.011898061,-0.025430646,-0.003062079,-0.0058322786,-0.056974836,-0.003322117,0.020665053,-0.0155916605,-0.031438053,-0.025006095,0.051328298,-0.0053732325,-0.055998366,-0.011590261,0.010316606,-0.012471206,-0.020081295,0.037997376,-0.051795308,0.007116548,-0.024645226,-0.0097116195,-0.037339322,-0.029845985,0.027001487,0.034388687,0.031862605,-0.016769791,0.06822546,0.0002645156,0.006792827,0.040799417,-0.023711212,0.0122058615,0.03392168,-0.016674267,-0.033412218,0.0114735095,0.027701998,-0.021227585,0.05536154,-0.047252603,-0.012216475,-0.03893139,-0.013575041,0.030716315,-0.014551509,-0.0316291,-0.004349001,-0.028508646,-0.0021532732,0.029676164,-0.0075994753,-0.0042534773,0.0027569325,0.011314303,-0.022947019,0.035195336,-0.0071112406,0.014700103,-0.065423414,-0.015581047,-0.0046674153,0.0073341304,-0.016069282,0.008406123,0.033412218,0.0018202653,-0.026852895,0.0069095786,-0.011569033,0.0020511153,-0.0013134568,0.031607874,0.022097915,-0.0036458375,0.055573817,0.009547106,0.0034123342,-0.041330107,-0.001278962,-0.0050680856,0.025282053,0.017194344,0.022097915,0.031756468,-0.017693192,0.0007177577,0.046106312,-0.0010912305,-0.040990464,0.033369765,0.0011011809,0.007960344,0.019179123,0.05446998,-0.042858493,0.021843184,-0.053281236,0.030546494,-0.0057526752,0.034070272,0.017088205,-0.003096574,0.011155096,-0.032053653,0.036935996,0.0035901153,0.06720653,0.043601457,0.032287154,0.0066760755,0.089070946,-0.06826791,-0.0038740342,0.060413707,0.014222481,-0.061475083,-0.0039350633,0.03392168,0.00052704115,0.033688176,-0.019688584,0.03927103,-0.036086895,0.011218779,-0.0350892,-0.044535473,-0.01203604,-0.010263537,-0.013904068,-0.011707013,-0.018903164,-0.03145928,-0.02540942,-0.005773903,0.05905514,-0.010417437,0.035216562,-0.015039744,-0.018860709,0.008215075,0.0056306166,0.0030859602,-0.01743846,0.033688176,0.033709403,0.03428255,-0.03876157,0.010964047,0.0024398456,0.040438548,-0.004364922,-0.012747165,0.0012312,-0.013596267,-0.016334627,0.00060133764,0.05922496,0.053493515,0.031353142,-0.056720104,0.02882706,-0.008220382,-0.030334217,-0.0052379067,-0.0028763376,-0.009870827,-0.033857998,0.04050223,-0.015878234,0.03428255,0.037487913,-0.01010433,0.10138294,0.035386384,0.029909667,-0.022480013,-0.030864907,0.008315906,-0.011982972,-0.033624493,0.03131069,0.022161597,0.03780633,-0.0031655636,0.026916577,-0.0019290567,0.013383992,0.013086806,0.019242804,0.018648433,-0.00018607304,0.031183321,-0.01278962,0.008093016,-0.025685377,-0.001749949,-0.011059572,-0.058842864,0.011038344,0.03708459,-0.02345648,-0.05281423,-0.05396052,-0.020484619,-0.040841874,-0.035365157,-0.038082287,-0.01713066,-0.037339322,0.0010222408,0.0042189825,-0.0069838753,-0.056805015,-0.0035450065,0.010560723,0.0077852164,0.023286661,0.012832074,0.00028524565,0.038846478,0.03621426,-0.05854568,0.028954426,0.047422424,0.04712524,-0.008655547,0.03058895,0.032053653,-0.03823088,0.03313626,-0.008809447,-0.027956728,-0.06635743,0.009064179,-0.015888847,-0.04220044,-0.016663654,0.022034233,0.074806005,0.0114416685,0.017342936,-0.051625486,0.020675667,0.027595859,-0.028084094,0.00524056,-0.063343115,-0.028126549,0.00025854533,-0.009833679,0.020282958,0.033284854,0.013118647,0.017332323,0.015305089,-0.02689535,0.006999796,-0.02678921,-0.007917889,-0.023244206,0.026937805,-0.01648322,0.04704033,-0.0178524,0.039865404,-0.0042800116,0.025239598,-0.017693192,-0.036787406,0.0027038637,-0.045639306,-0.0064584925,-0.08019781,0.0010832702,0.063300654,0.0094144335,0.006453186,0.022692287,-0.000021414155,0.0017764835,0.05090375,0.057866395,-0.01718373,0.019826563,0.029103018,-0.00047098703,0.031480506,-0.019104825,0.03305135,-0.048823446,-0.022968246,0.028466191,-0.0019940662,-0.023222977,0.028614784,-0.030440357,-0.007944424,-0.020049453,-0.029103018,-0.040438548,0.033475902,0.058757953,-0.053153872,-0.01903053,0.004595772,0.0402475,0.022161597,-0.05264441,0.013925295,-0.0001945309,0.037148274,-0.04198816,-0.01371302,-0.050394285,0.026088702,-0.035471294,0.020898556,-0.037233185,-0.0011270521,-0.06924438,-0.004715177,-0.013118647,0.038273335,-0.0015708413,-0.019094212,-0.041903254,0.021429246,0.01155842,-0.012227088,-0.002552617,0.0038289255,0.0004895612,0.037403002,0.03691477,-0.0041367253,-0.038995072,-0.046191223,-0.0024305584,0.02197055,0.008193848,-0.020378482,0.009101327,-0.021864412,-0.004139379,-0.020824261,0.030567722,-0.01849984,-0.053451058,-0.0032345532,-0.0021373525,-0.015188336,0.022840882,0.006230296,0.010348448,0.032393295,0.024241902,-0.02012375,0.046573322,-0.03523779,-0.0024637266,0.050309375,0.0059437235,-0.028551102,-0.019847792,0.0041075377,0.017703805,-0.01709882,0.03417641,-0.030100714,-0.044153377,-0.020887943,-0.0002665057,-0.015803937,0.0045931186,-0.013415833,0.021333722,-0.07399936,-0.03927103,0.07293798,-0.005938417,0.037466686,0.07726841,0.032923985,0.046191223,0.020017613,-0.035959527,0.051710397,-0.01774626,-0.012439365,-0.0017101472,0.0040040533,-0.020389095,-0.023435254,-0.003937717,0.043176908,0.009021724,0.0009983599,-0.043155678,0.032287154,0.03954699,-0.006739758,0.06809809,-0.0060020997,-0.009769996,-0.007705613,0.03540761,-0.0051715705,-0.036235485,-0.0065858583,-0.019274646,0.020060068,0.26644865,0.0012265564,0.035789706,0.026025018,0.014328619,-0.004073043,0.018022219,0.013999592,0.080707274,-0.023838578,-0.033879224,-0.030334217,0.028805831,0.037148274,0.01025823,-0.017332323,0.01827695,-0.028444963,0.01911544,0.01225893,-0.034091502,0.0006600452,0.027362356,-0.029824756,0.027935501,-0.022798426,0.0038819944,-0.015071585,0.013755475,0.032456975,0.00043251202,-0.059904244,-0.02364753,0.018627206,0.026661847,0.044238288,-0.012396909,-0.03784878,-0.05281423,0.033263624,-0.04704033,0.016982067,-0.025473101,0.022628605,-0.016865317,0.009722234,-0.016504446,-0.04623368,-0.04094801]]

This seems to be happening for all models.

When inputs have > ~5 words, they consistently return vectors.

Expected behavior

Expected to return vectors regardless of input length.

Activity

alvarobartt

alvarobartt commented on Apr 4, 2025

@alvarobartt
Member

Hey @superkelvint, thanks for reporting! Do you see this happening with a specific model / architecture type? Could you share the model IDs or architecture types of the models you tried that ran into that issue on empty / null embeddings? In order to have a reproducer could you share which device are you using GPU, CPU or MPS, which architecture and model ID from the Hugging Face Hub? Thanks in advance 🤗

e.g. I tried ibm-granite/granite-embedding-125m-english and didn't see the issue happening!

self-assigned this
on Apr 4, 2025
superkelvint

superkelvint commented on Apr 4, 2025

@superkelvint
Author

@alvarobartt Thanks for your reply.

I'm using https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v2.0 on turing-1.6. Unfortunately this is the only GPU I have, otherwise I would try it on another gpu architecture. I can confirm this doesn't happen at all on cpu-1.6.

superkelvint

superkelvint commented on Apr 4, 2025

@superkelvint
Author

I just tried using ibm-granite/granite-embedding-125m-english on turing-1.6 and I'm not seeing the issue either.

alvarobartt

alvarobartt commented on Apr 7, 2025

@alvarobartt
Member

Hey @superkelvint thanks for the information, I'll try to reproduce on my own before closing the issue then, but please let us know if this happens ever again or if you happen to have a consistent reproducer 🤗 Thanks again!

vrdn-23

vrdn-23 commented on Apr 22, 2025

@vrdn-23

This has been a long-time know issue first documented in #53. The solution would be to turn off Flash Attention for that model!
cc @superkelvint @alvarobartt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    Short inputs cause /embed to randomly return empty vectors. · Issue #557 · huggingface/text-embeddings-inference