Performance of llama.cpp on Nvidia CUDA

[olegshulyakov]

This is similar to the Performance of llama.cpp on Apple Silicon M-series, Performance of llama.cpp on AMD ROCm(HIP) and Performance of llama.cpp with Vulkan, but for CUDA! I think it's good to consolidate and discuss our results here.

We'll be testing the Llama 2 7B model like the other thread to keep things consistent, and use Q4_0 as it's simple to compute and small enough to fit on a 4GB GPU. You can download it here.

Instructions

Either run the commands below or download one of our CUDA releases. If you have multiple GPUs please run the test on a single GPU using -sm none -mg YOUR_GPU_NUMBER unless the model is too big to fit in VRAM.

wget https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q4_0.gguf
llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1

Share your llama-bench results along with the git hash and CUDA info string in the comments. Feel free to try other models and compare backends, but only valid runs will be placed on the scoreboard.

If multiple entries are posted for the same device I'll prioritize newer commits with substantial CUDA updates, otherwise I'll pick the one with the highest overall score at my discretion. Performance may vary depending on driver, operating system, board manufacturer, etc. even if the chip is the same. For integrated graphics note that your memory speed and number of channels will greatly affect your inference speed!

CUDA Scoreboard for Llama 2 7B, Q4_0 (no FA)

Chip	Memory	pp512 t/s	tg128 t/s	Commit	Thanks to
RTX 5090	32 GB / GDDR7 / 512 bit	14751.98 ± 136.24	239.62 ± 0.37	9c35706	@RodriMora
RTX 4090	24 GB / GDDR6X / 384 bit	11992.70 ± 107.99	186.21 ± 0.13	2241453	@lhl
RTX 3090 Ti	24 GB / GDDR6X / 384 bit	6567.49 ± 20.30	171.19 ± 3.98	9c35706	@slaren
RTX 3090	24 GB / GDDR6X / 384 bit	5174.69 ± 21.83	158.16 ± 0.21	c76b420	@m18coppola
RTX 4080	16 GB / GDDR6X / 256 bit	8031.64 ± 26.49	142.49 ± 0.16	20638e4	@Ristovski
RTX 3080	10 GB / GDDR6X / 320 bit	5013.86 ± 24.80	139.65 ± 0.99	9c35706	@slaren
RTX 4070 Ti SUPER	16 GB / GDDR6X / 256 bit	6924.53 ± 13.87	132.26 ± 0.16	9c35706	@Ristovski
RTX 2080 Ti	11 GB / GDDR6 / 352 bit	2890.66 ± 2.42	107.51 ± 0.21	9c35706	@ariya
RTX 5060 Ti	16 GB / GDDR7 / 128 bit	3737.25 ± 6.79	90.94 ± 0.02	89d1029	@mike-llamacpp
RTX 3060	12 GB / GDDR6 / 192 bit	2020.40 ± 3.75	68.38 ± 0.09	860a9e4	@pebaryan
RTX 4060 Ti	8 GB / GDDR6 / 128 bit	3394.63 ± 7.44	63.86 ± 0.01	89d1029	@mike-llamacpp
GTX 1080 Ti	11 GB / GDDR5X / 352 bit	1084.41 ± 3.01	62.49 ± 0.06	9c35706	@ariya
RTX 2060 SUPER	8 GB / GDDR6 / 256 bit	1420.24 ± 1.95	60.04 ± 0.01	5c0eb5e	@ggerganov
Tesla P40	24 GB / GDDR5 / 384 bit	1007.42 ± 1.23	54.74 ± 0.07	c76b420	@m18coppola
RTX 2000 Ada	16 GB / GDDR6 / 128 bit	1956.22 ± 7.74	50.62 ± 0.04	756cfea	@DigitalRudeness
GTX 1660 Ti Mobile	6 GB / GDDR6 / 192 bit	152.78 ± 3.35	44.11 ± 0.41	e54d41b	@pt13762104
RTX 4050 Laptop	6 GB / GDDR6 / 96 bit	1725.85 + 17.85	43.72 + 0.41	d79d8f3	@TimCabbage
GTX 1070 Ti	8 GB / GDDR5 / 256 bit	714.44 ± 2.04	37.82 ± 0.02	79c1160	@pebaryan
Tesla P4	8 GB / GDDR5 / 256 bit	514.53 ± 3.06	33.29 ± 0.00	c76b420	@m18coppola
P106-100	6 GB / GDDR5 / 192 bit	406.94 ± 0.25	30.40 ± 0.02	5fd160b	@pebaryan
GTX 1060	6 GB / GDDR5 / 192 bit	416.85 ± 1.75	27.79 ± 0.02	5fd160b	@pebaryan

CUDA Scoreboard for Llama 2 7B, Q4_0 (with FA)

Chip	Memory	pp512 t/s	tg128 t/s	Commit	Thanks to
RTX 5090	32 GB / GDDR7 / 512 bit	16041.54 ± 85.27	248.57 ± 0.05	9c35706	@RodriMora
RTX 4090	24 GB / GDDR6X / 384 bit	14770.63 ± 102.93	188.96 ± 0.05	2241453	@lhl
RTX 3090 Ti	24 GB / GDDR6X / 384 bit	6924.01 ± 10.76	172.26 ± 1.31	9c35706	@slaren
RTX 3090	24 GB / GDDR6X / 384 bit	5560.06 ± 16.28	161.89 ± 0.18	c76b420	@m18coppola
RTX 4080	16 GB / GDDR6X / 256 bit	9205.93 ± 22.31	143.47 ± 0.02	20638e4	@Ristovski
RTX 3080	10 GB / GDDR6X / 320 bit	5569.56 ± 14.04	139.95 ± 0.95	9c35706	@slaren
RTX 4070 Ti SUPER	16 GB / GDDR6X / 256 bit	7612.32 ± 37.35	132.85 ± 0.31	9c35706	@Ristovski
RTX 2080 Ti	11 GB / GDDR6 / 352 bit	3107.61 ± 4.34	109.17 ± 0.07	9c35706	@ariya
RTX 5060 Ti	16 GB / GDDR7 / 128 bit	4195.53 ± 1.98	93.46 ± 0.01	89d1029	@mike-llamacpp
RTX 3060	12 GB / GDDR6 / 192 bit	2229.42 ± 2.98	70.78 ± 0.07	860a9e4	@pebaryan
RTX 4060 Ti	8 GB / GDDR6 / 128 bit	3803.45 ± 70.80	64.03 ± 0.53	89d1029	@mike-llamacpp
GTX 1080 Ti	11 GB / GDDR5X / 352 bit	1138.14 ± 2.02	61.38 ± 0.03	9c35706	@ariya
RTX 2060 SUPER	8 GB / GDDR6 / 256 bit	1563.77 ± 0.51	61.13 ± 0.05	5c0eb5e	@ggerganov
Tesla P40	24 GB / GDDR5 / 384 bit	1079.66 ± 0.18	53.73 ± 0.05	c76b420	@m18coppola
RTX 2000 Ada	16 GB / GDDR6 / 128 bit	2250.14 ± 5.91	50.71 ± 0.01	756cfea	@DigitalRudeness
GTX 1660 Ti Mobile	6 GB / GDDR6 / 192 bit	158.09 ± 2.06	44.92 ± 0.68	e54d41b	@pt13762104
GTX 1070 Ti	8 GB / GDDR5 / 256 bit	790.52 ± 2.39	37.87 ± 0.00	79c1160	@pebaryan
Tesla P4	8 GB / GDDR5 / 256 bit	529.53 ± 2.12	33.12 ± 0.03	c76b420	@m18coppola
P106-100	6 GB / GDDR5 / 192 bit	438.49 ± 0.38	30.64 ± 0.06	5fd160b	@pebaryan
GTX 1060	6 GB / GDDR5 / 192 bit	446.19 ± 0.81	28.18 ± 0.01	5fd160b	@pebaryan

More detailed test

The main idea of this test is to show a decrease in performance with increasing size.

llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1 -p 512,1024,2048,4096,8192,16384,32768 -n 128,256,512,1024,2048

[m18coppola]

Here's the results for my devices. Not sure how to get a "cuda info string" though.

CUDA Scoreboard for Llama 2 7B, Q4_0 (no FA)

Chip	pp512 t/s	tg128 t/s	Commit
Tesla P4	514.53 ± 3.06	33.29 ± 0.00	c76b420
Tesla P40	1007.42 ± 1.23	54.74 ± 0.07	c76b420
RTX 3090	5174.69 ± 21.83	158.16 ± 0.21	c76b420

CUDA Scoreboard for Llama 2 7B, Q4_0 (with FA)

Chip	pp512 t/s	tg128 t/s	Commit
Tesla P4	529.53 ± 2.12	33.12 ± 0.03	c76b420
Tesla P40	1079.66 ± 0.18	53.73 ± 0.05	c76b420
RTX 3090	5560.06 ± 16.28	161.89 ± 0.18	c76b420

[bennmann]

While technically not directly related, there may also be value in comparing AMD ROCM build here too, as ROCM acts a replacement (sometimes a directly compatible layer) for most CUDA calls.

I admit risk of confusion for Nvidia users in the thread if this path is taken.

[olegshulyakov]

As I know you cannot run ROCm on Nvidia GPU. If you would like to see compared results check Vulkan thread. You can find there results for Vulkan/CUDA and Vulkan/ROCm.

UPD: Created ROCm discussion.

[slaren]

Device 0: NVIDIA GeForce RTX 3090 Ti, compute capability 8.6, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	pp512	6567.49 ± 20.30
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	tg128	171.19 ± 3.98
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	6924.01 ± 10.76
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	172.26 ± 1.31

build: 9c35706 (6060)

Device 0: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	pp512	5013.86 ± 24.80
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	tg128	139.65 ± 0.99
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	5569.56 ± 14.04
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	139.95 ± 0.95

build: 9c35706 (6060)

[Ristovski]

Device 0: NVIDIA GeForce RTX 4070 Ti SUPER, compute capability 8.9, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	pp512	6924.53 ± 13.87
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	tg128	132.26 ± 0.16
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	7612.32 ± 37.35
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	132.85 ± 0.31

build: 9c35706 (647)

[Ristovski]

@olegshulyakov One more benchmark for RTX 4080:

Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	pp512	8031.64 ± 26.49
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	tg128	142.49 ± 0.16
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	9205.93 ± 22.31
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	143.47 ± 0.02

build: 20638e4 (2)

[olegshulyakov]

@Ristovski why so slow? Have you undervolted it? It pretty as RTX 3080, I expected somewhere between RTX 3090 and 3080 Ti =(

[Ristovski]

@Ristovski why so slow? Have you undervolted it? It pretty as RTX 3080, I expected somewhere between RTX 3090 and 3080 Ti =(

Hmm indeed, I didn't give much thought to the score at first. It should be stock but not completely sure as that is one of our work machines. I didn't have much time to investigate today, will check again tomorrow!

[RodriMora]

Device 0: 3090. Power limit to 250w

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	pp512	4175.47 ± 27.79
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	tg128	137.72 ± 0.46
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	4377.03 ± 89.10
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	138.34 ± 0.96

build: 9c35706 (6060)

Device 2: 5090. Power limit to 400w

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	pp512	12706.26 ± 13.30
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	tg128	236.73 ± 1.29
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	13823.36 ± 20.99
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	245.02 ± 1.08

build: 9c35706 (6060)

[olegshulyakov]

Can you please launch them without a limit on full power?

[RodriMora]

Sure, results with defaults power limits:

3090 at 390W
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	pp512	5405.83 ± 5.80
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	tg128	151.04 ± 0.24
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	5932.44 ± 10.87
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	155.36 ± 0.09

build: 9c35706 (6060)

5090 at 600W
Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	pp512	14751.98 ± 136.24
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	tg128	239.62 ± 0.37
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	16041.54 ± 85.27
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	248.57 ± 0.05

build: 9c35706 (6060)

[ariya]

Device 0: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	100	pp512	1084.41 ± 3.01
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	100	tg128	62.49 ± 0.06

Device 0: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	100	1	pp512	1138.14 ± 2.02
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	100	1	tg128	61.38 ± 0.03

build: 9c35706 (6060)

[ariya]

@olegshulyakov To help users quickly understand the approximate largest models that can run on each GPU, I suggest adding a VRAM column next to the GPU name on the main scoreboard.

Example:

Chip	VRAM	pp512 t/s	tg128 t/s	Commit
RTX 3090 Ti	24 GB	6567.49 $\pm$ 20.30	171.19 $\pm$ 3.98	9c35706
RTX 3090	24 GB	5174.69 $\pm$ 21.83	158.16 $\pm$ 0.21	c76b420
RTX 3080	10 GB	5013.86 $\pm$ 24.80	139.65 $\pm$ 0.99	9c35706

[olegshulyakov]

Made it a little bit better 🙂

[ggerganov]

Device 0: NVIDIA GeForce RTX 2060 SUPER, compute capability 7.5, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	pp512	1420.24 ± 1.95
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	tg128	60.04 ± 0.01
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	1563.77 ± 0.51
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	61.13 ± 0.05

build: 5c0eb5e (6075)

[olegshulyakov]

@ggerganov Can you please add "performance" label?

[mike-llamacpp]

@olegshulyakov I see you grabbed some of my numbers from the Vulkan thread. However, I flooded that post with a bunch of data that probably came across as noise. While you quoted my correct numbers for Non-FA, the FA results you grabbed were actually when run on two GPUs instead of one. To make things easier, here are the numbers from a single card:

RTX 5060 Ti 16 GB

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes
  Device 1: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes
  Device 2: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes
  Device 3: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes

model	size	params	backend	ngl	sm	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	none	0	pp512	3737.25 ± 6.79
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	none	0	tg128	90.94 ± 0.02
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	none	1	pp512	4195.53 ± 1.98
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	none	1	tg128	93.46 ± 0.01

build: 89d10295 (6002)

And here's another GPU for the collection:

RTX 4060 Ti 8 GB

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	pp512	3394.63 ± 7.44
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	tg128	63.86 ± 0.01
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	3803.45 ± 70.80
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	64.03 ± 0.53

build: 89d10295 (6002)

[rohan-sircar]

Nice 64GB VRAM setup you got there!

And here's another GPU for the collection:

We all be here showing off our GPU collections 😅

[mike-llamacpp]

Thanks. It isn't the fastest setup around, especially when working with 70B+ models, but it is completely usable for inference. There are also some benefits I like about these particular cards (Gigabyte Windforce):

• Two slots thick and only ~200 mm in length makes them easy to fit in a wide variety of cases
• Physical x8 PCI-e connector lets them fit in either x8 or x16 slots without modification (5060 TIs only use 8 lanes anyhow)
• Quiet (Silent when idle)
• Low idle power consumption (~5 watts per card)
• Relatively low power draw under full load (<180W each), so easy to power all four with an inexpensive PSU

[ariya]

Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	100	0	pp512	2890.66 ± 2.42
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	100	0	tg128	107.51 ± 0.21
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	100	1	pp512	3107.61 ± 4.34
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	100	1	tg128	109.17 ± 0.07

build: 9c35706 (6060)

[lhl]

Yeah also saw numbers for my 4090 taken from the Vulkan thread. Re-ran CUDA results so you can get the latest FA and non-FA results from same build:

FA:

❯ CUDA_VISIBLE_DEVICES=0 build/bin/llama-bench -m /models/llm/gguf/llama-2-7b.Q4_0.gguf -fa 1
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	14770.63 ± 102.93
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	188.96 ± 0.05

Non-FA:

❯ CUDA_VISIBLE_DEVICES=0 build/bin/llama-bench -m /models/llm/gguf/llama-2-7b.Q4_0.gguf
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	pp512	11992.70 ± 107.99
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	tg128	186.21 ± 0.13

build: 224145325 (6098)

nvidia-dkms 575.64.03-1

❯ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Tue_May_27_02:21:03_PDT_2025
Cuda compilation tools, release 12.9, V12.9.86
Build cuda_12.9.r12.9/compiler.36037853_0

[pebaryan]

NVIDIA P106-100
6GB VRAM
Win 11
Driver Version: 566.36 CUDA Version: 12.7

I ran two times, took the best on 2 different build

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA P106-100, compute capability 6.1, VMM: no

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	0	pp512	406.94 ± 0.25
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	0	tg128	30.40 ± 0.02
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	1	pp512	438.49 ± 0.38
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	1	tg128	30.64 ± 0.06

build: 5fd160b (6106)

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA P106-100, compute capability 6.1, VMM: no

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	0	pp512	425.73 ± 0.82
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	0	tg128	29.42 ± 0.03
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	1	pp512	436.90 ± 0.88
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	1	tg128	29.94 ± 0.03

build: 860a9e4 (5688)

Sadly, nvidia was not supporting this device for the vulkan driver

[pebaryan]

I just bricked my gtx 1070 Ti :( so i would not be able to reproduce the result with newer build

[olegshulyakov]

@pebaryan I've taken the last build one.

[DigitalRudeness]

Would like to participate with a slightly exotic one from my cute server cube.. :-) (RTX 2000 Ada, 16GB, 75W)

I did two runs:

1. pull/compilation of llama.cpp from yesterday:

gml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX 2000 Ada Generation, compute capability 8.9, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	pp512	1956.22 ± 7.74
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	tg128	50.62 ± 0.04
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	2250.14 ± 5.91
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	50.71 ± 0.01

build: 756cfea (6105)

2. fresh pull/compilation of llama.cpp ~5min ago:

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX 2000 Ada Generation, compute capability 8.9, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	pp512	1952.82 ± 7.35
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	tg128	50.59 ± 0.04
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	2237.16 ± 6.18
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	50.67 ± 0.01

build: 1d72c84 (6109)

Seems to make no big difference... ^^

[pebaryan]

I finally got my hands on similar card as before (NP106) but with display output

NVIDIA GTX 1060
6GB GDDR5 192-bit
Driver 566.36

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1060 6GB, compute capability 6.1, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	0	pp512	416.85 ± 1.75
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	0	tg128	27.79 ± 0.02
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	1	pp512	446.19 ± 0.81
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	1	tg128	28.18 ± 0.01

build: 5fd160b (6106)

[pebaryan]

just realized i didn't use the latest build, not that difference though

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1060 6GB, compute capability 6.1, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	0	pp512	413.59 ± 2.04
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	0	tg128	27.74 ± 0.06
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	1	pp512	443.66 ± 0.25
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	1	tg128	28.08 ± 0.04

build: 79c1160 (6123)

[pebaryan]

Apparently, my gtx 1070 ti still works. its just the m.2 oculink adapter that is not detecting it properly.

here's the updated result with latest build

GTX 1070 Ti Mini (Zotac)
8GB GDDR5 256bit
on PCIE 3.0 x 2 (5.0 GT/s)

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1070 Ti, compute capability 6.1, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	0	pp512	714.44 ± 2.04
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	0	tg128	37.82 ± 0.02
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	1	pp512	790.52 ± 2.39
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA,RPC	99	1	tg128	37.87 ± 0.00

build: 79c1160 (6123)

[Tom94]

RTX PRO 6000 Blackwell

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	pp512	14401.14 ± 17.25
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	0	tg128	268.52 ± 0.53
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	pp512	16652.18 ± 13.96
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	1	tg128	275.76 ± 0.21

build: 79c1160 (6123)