Conversations leaking into each other during parallelization on RPC cluster

[becky-soda]

I'm sorry to bother you all. My colleague and I been having trouble with the latest versions of llama.cpp, running over RPC. We do have a rather exotic set up, which I'll describe:

Primary Llama.cpp Machine: 1x Geforce 4090, 1x Geforce 4060 (CUDA)
Secondary Llama.cpp Machine: 1x Geforce 4060 (CUDA)
Tertiary Llama.cpp Machine: 1x RX 7900 XTX (Vulkan)

The primary machine starts with this command line:

set "LLAMA_SET_ROWS=1" && llama-server -m "X:\models\ML2-123B-Magnum-Diamond-IQ3_M-00001-of-00002.gguf" --host 192.168.1.69 --port 5002 -c 50000 --rpc 192.168.1.94:50052,192.168.1.70:50052 --n_gpu_layers 99 --no-warmup -fa -t 16 --threads-batch 16 -ctk f16 -ctv f16 -sm row --parallel 2

We're operating with the LLAMA_SET_ROWS=1 environment variable, and no CPUs are involved in inference.

We have parallelization running, and two of us access the RPC cluster simultaneously. Strangely, using llama.cpp compiled with various commits from the past week, we've been seeing conversations 'leak' or 'cross-talk' over into each other, for want of a better word, and sometimes even produce semi-lucid 'stream of thought' output based upon both conversations.

The last version we had working okay was from the 15th of July (commit cbc68be). There were some commits after that which outright broke parallelization for us, so we didn't use those (around and including commit 21c0217). When we switched to a newer version (commit 9008328), the parallelization began working again.

At this point, when we starting sending messages back and forth to the LLM, it became evident that something was wrong. I'd often get messages that seems to include some of his context, and he'd often get messages that seem to include some of my context. As an example, if I'm talking to the LLM about bash scripting, and he's talking to the LLM about writing professional letters, he would start getting elements of bash scripting seamlessly woven into his conversation, and I'd start getting information about formal writing in bash scripts. And sometimes it would just be messy stream-of-thought, with made-up-words, illegible grammar, and nonsensical punctuation.

I wanted to do a git bisect, but the trouble is that we invariably end up in a commit that has broken parallelization, so we can't test for the conversation leakage/cross-talk.

I'm at a loss as to how to handle this situation. Does anyone have any ideas, please?

[ggerganov]

Most likely fixed with #14853

[becky-soda]

I'm really sorry to say this, but we just tested it with a git pull from about an hour ago (commit 89d1029). It's still producing strange output after a few posts. We both tested with SillyTavern this time, so I could see the output in the consoles. My friend ended up getting my characters being mentioned in his dialogue, and I got characters from his.

Here's an example from my friend's conversation, while I (on a seperate PC, with the profile 'Becky') was conversing with a character named Gina: "Conversely, a stale odor or sudden violence, lips curling onto the balls, lips cursinglingerseverbalances convey meaning convey meaningfully work of overexplaining stating Gina, Always,** Creative Progressionarrative Progression:** Engaging Character Interactions: Respond thoughtfully to Becky's dialogue, actions, and environmental cues. Let Gina's reactions stem from subtle shifts: distant voices softening as they move deeper, the weight settling spine beneath their fingertips, or a gentle breeze carrying a hint of fallen rain."

I checked my friend's SillyTavern console, and there were no references in there to a character named Becky or Gina, meaning that llama.cpp is most likely getting those two characters from my parallel conversation.

Shortly after he got that comment, I got this one:

We both got partway through our conversations before this strangeness started occurring. I'm really sorry, but is there anything we can try at this point, please?

[becky-soda]

A little update. We tested it again with flash attention off, and if anything the situation seems more prevalent. We're also able to reproduce the leaking and gibberish consistently if one of us tries running inference while the other is already running inference. In this example, I was generating a response, and as soon as my friend started generating a response on his PC, the stream on my PC got messed up.

He started running his inference at the time the words 'between herself and plugrelage' were appearing on my screen. At that point, it goes a bit wonky.

[ggerganov]

See if any of the following changes fixes the issue and report the results:

• Remove -sm row
• Remove LLAMA_SET_ROWS=1
• Keep LLAMA_SET_ROWS=1, add -kvu
• Add GGML_CUDA_DISABLE_FUSION=1 GGML_VK_DISABLE_FUSION=1

[becky-soda]

See if any of the following changes fixes the issue and report the results:

* Remove `-sm row`

* Remove `LLAMA_SET_ROWS=1`

* Keep `LLAMA_SET_ROWS=1`, add `-kvu`

* Add `GGML_CUDA_DISABLE_FUSION=1 GGML_VK_DISABLE_FUSION=1`

Thank you so much for your time. I really do appreciate the help. This is what we found so far:

Remove -sm row
Produces cross-leaking and nonsensical output, if a message is sent to llama.cpp while it's processing another parallel message.

Remove LLAMA_SET_ROWS=1
If flash attention is disabled, it works and output is sane and not leaking.
If flash attention is enabled, it produces no (or empty?) output.

Keep LLAMA_SET_ROWS=1, add -kvu
If flash attention is disabled, it works and output is sane and not leaking.
If flash attention is enabled, it produces no (or empty?) output.

Add GGML_CUDA_DISABLE_FUSION=1 GGML_VK_DISABLE_FUSION=1
Produces cross-leaking and nonsensical output, if a message is sent to llama.cpp while it's processing another parallel message.

(Unfortunately for us, flash attention being off is less than ideal. This required us to include additional 40-series Geforce GPU cards to provide more VRAM, and these cards are much slower, significantly reducing the token per second for output.)

[slaren]

If the issue is with FA, this was probably fixed in #14916

[becky-soda]

If the issue is with FA, this was probably fixed in #14916

We just did a git pull and tried it (commit 8ad7b3e) and it's still leaking between chats, and producing nonsense in between, but only when one of us submits while it's already processing for the other.

Thank you for the suggestion, though.

[becky-soda]

A quick update. We've been trying various configurations, but nothing's worked so far.

We tried removing the -ctk f16 parameter.
We tried recompiling the RX 7900 XTX machine to use ROCm instead of Vulkan.
We tried using a base Mistral model (Mistral-Large-Instruct-2411-IQ3_M.gguf).
We removed the RX 7900 XTX from the cluster, and replaced it with another Geforce 4060, so that all the cards were using a CUDA backend, but with a smaller context to fit the smaller VRAM.

Sadly, none of these helped, and they all generated situations where the contexts leaked into each other.

[becky-soda]

What is the smallest model that you can reproduce the Vulkan issue with? I tried with Llama-8B-v3-Instruct Q4_K and cannot reproduce on my RTX 2060.

All of these models produce the scrambled output on a single RX 7900 XTX using the Vulkan backend on an Ubuntu 24.04 host:

gemma-3-12b-it-Q2_K_L.gguf
Meta-Llama-3.1-8B-Instruct-Q6_K.gguf
Mistral-Nemo-Instruct-2407.i1-IQ4_XS.gguf
Mistral-Nemo-Instruct-2407-Q6_K.gguf

I am running requests in 2 browsers in parallel - any specific conditions under which the problem occurs?

This is the command line I'm using:

LLAMA_SET_ROWS=1 ./llama-server -m "/mnt/knudgles/AI Models/LLMs/Meta-Llama-3.1-8B-Instruct-Q6_K.gguf" --host 192.168.1.70 --port 5002 -c 20000 --n_gpu_layers 99 -sm none --parallel 2

I've tried the test using two separate PCs connecting to the same host+port, and two browsers (Firefox and Chromium) on the one PC connecting to the same host+port. Both of these arrangements result in one set of anomalous output. I screenshotted an example of how I tested on a single PC, triggering the submit buttons one after the other, and you can see one of the outputs went nonsensical:

p.s. thanks for the tests - these are very useful.

Thank you too. I'm just sorry I can't do more to help.

[ggerganov]

I am not able to reproduce on my end using the same Q6_K model on RTX 2060 (just changed to -c 10000 so I can fit it in VRAM). My best guess is that this is something similar to #14974 (comment). But I don't know much about how to investigate it further.

[ggerganov]

Ah wait, I just noticed that you don't have -fa in your command. When I remove it - I am able to reproduce it.

[ggerganov]

Should be fixed on latest master. Let us know if you spot any additional problems.

[becky-soda]

Thank you!