NoKeysError: nats: no keys found error, but kv has keys

[kevinhikaruevans]

I'm receiving this fluke 'no keys found' error. It's odd though because this rarely seems to happen and the other async tasks loaded the keys without issue.

2025-04-11 10:31:44 4fd96e3b93dc taskiq.receiver.receiver[102] ERROR Exception found while executing function: nats: no keys found                                                         
Traceback (most recent call last):                                                                                                                                                         
  File "/usr/local/lib/python3.12/site-packages/taskiq/receiver/receiver.py", line 248, in run_task                                                                                        
    kwargs = await dep_ctx.resolve_kwargs()                                                                                                                                                
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                
  File "/usr/local/lib/python3.12/site-packages/taskiq_dependencies/ctx.py", line 364, in resolve_kwargs                                                                                   
    kwargs = await self.resolver(dependency, self.initial_cache)                                                                                                                           
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                           
  File "/usr/local/lib/python3.12/site-packages/taskiq_dependencies/ctx.py", line 335, in resolver                                                                                         
    sub_result = await executed_func                                                                                                                                                       
                 ^^^^^^^^^^^^^^^^^^^                                                                                                                                                       
  File "/usr/src/app/orchestrator/models/api/state.py", line 117, in get_state                                                                                                             
    await state.load()                                                                                                                                                                     
  File "/usr/src/app/orchestrator/models/api/state.py", line 83, in load                                                                                                                   
    remote_keys = await kv.keys()                                                                                                                                                          
                  ^^^^^^^^^^^^^^^                                                                                                                                                          
  File "/usr/local/lib/python3.12/site-packages/nats/js/kv.py", line 358, in keys                                                                                                          
    raise nats.js.errors.NoKeysError                                                                                                                                                       
nats.js.errors.NoKeysError: nats: no keys found          

But what's also strange is the kv store should definitely have keys in it. I can see them when using nats kv ls. Here's the code that I'm using:

  async def load(self):
    nc = await nats.connect(Config.Nats.URL, max_reconnect_attempts=-1)
    js = nc.jetstream()
    try:
      kv = await js.key_value(bucket='my_bucket_name')
    except:
      kv = await js.create_key_value(bucket='my_bucket_name')
    
    await kv.put('last_retrieved', str(datetime.utcnow()).encode()) # <- won't this always put an entry in kv?

    prefix = self.prefix
    remote_keys = await kv.keys() # <-- error occurs here

I would assume at the minimum, the last_retrieved key should definitely be in there.

I'm using nats-py version 2.8.0 with the nats:alpine (v2.11.1) Docker image. What would cause this error to occur? Should I be initializing this differently?

[SOORAJTS2001]

Hi @kevinhikaruevans, What version of nats-server do you have? I had the same issue of nats.js.errors.NoKeysError: nats: no keys found even with the keys in the bucket. There is a chance of race condition because I had a lot of reads and writes in the bucket. So i used semaphore lock to control max_concurrency

[kevinhikaruevans]

I believe it's v2.11.1. I'm not really reading/writing all too much to nats, like maybe a couple operations per second

[SOORAJTS2001]

Hi @kevinhikaruevans can you try semaphore ?

[kevinhikaruevans]

For the kv read/writes? Not easily, I have several different processes on different systems performing operations on a single nats server. I'll see what I can do though.