guide : running gpt-oss with llama.cpp

[ggerganov]

Note

This guide is a live document. Feedback and benchmark numbers are welcome - the guide will be updated accordingly.

---

Overview

This is a detailed guide for running the new gpt-oss models locally with the best performance using llama.cpp. The guide covers a very wide range of hardware configurations. The gpt-oss models are very lightweight so you can run them efficiently in surprisingly low-end configurations.

Obtaining `llama.cpp` binaries for your system

Make sure you are running the latest release of llama.cpp: https://github.com/ggml-org/llama.cpp/releases

Obtaining the `gpt-oss` model data (optional)

The commands used below in the guide will automatically download the model data and store it locally on your device. So this step is completely optional and provided for completeness.

The original models provided by OpenAI are here:

• https://huggingface.co/openai/gpt-oss-20b
• https://huggingface.co/openai/gpt-oss-120b

First, you need to manually convert them to GGUF format. For convenience, we host pre-converted models here in ggml-org.

Pre-converted GGUF models:

• https://huggingface.co/ggml-org/gpt-oss-20b-GGUF
• https://huggingface.co/ggml-org/gpt-oss-120b-GGUF

Tip

Running the commands below will automatically download the latest version of the model and store it locally on your device for later usage. A WebUI chat and an OAI-compatible API will become available on localhost.

Note

This guide currently covers general chat use cases with llama-server. Tool-calling and agentic use cases are mostly supported via the OAI-compatible API of llama-server, but some minor issues are still being resolved, both on llama.cpp side and in the reference upstream model implementations (chat template, harmony). As things get more polished in the next few days, you should be able to use the commands below in combination with your favorite 3rd party applications that support the OAI interface (such as chat interfaces, coding agents, etc.).

Sample output of using gpt-oss-120b with the built-in llama-server WebUI

Minimum requirements

Here are some hard memory requirements for the 2 models. These numbers could vary a little bit by adjusting the CLI arguments, but should give a good reference point.

Model	Model data (GB)	Compute buffers (GB)	KV cache per 8 192 tokens (GB)	Total @ 8 192 tokens (GB)	Total @ 32 768 tokens (GB)	Total @ 131 072 tokens (GB)
gpt‑oss 20B	12.0	2.7	0.2	14.9	15.5	17.9
gpt‑oss 120B	61.0	2.7	0.3	64.0	64.9	68.5

Note

It is not necessary to fit the entire model in VRAM to get good performance. Offloading just the attention tensors and the KV cache in VRAM and keeping the rest of the model in the CPU RAM can provide decent performance as well. This is taken into account in the rest of the guide.

Relevant CLI arguments

Using the correct CLI arguments in your commands is crucial for getting the best performance for your hardware. Here is a summary of the important flags and their meaning:

Argument	Purpose
-hf	Specify the Hugging Face model ID to use. The model will be downloaded using curl from the respective model repository
-c	Specify the context size to use. More context requires more memory. Both gpt-oss models have a maximum context of 128k tokens. Use -c 0 to set to the model's default
-ub N -b N	Specify the max batch size N during processing. Larger size increases the size of compute buffers, but can improve the performance in some cases
-fa	Enable Flash Attention kernels. This improves the performance on backends that support the operator
--n-gpu-layers N	Number of layers N to offload to the GPU. For this guide, keep this number at -ngl 99
--n-cpu-moe N	Number of MoE layers N to keep on the CPU. This is used in hardware configs that cannot fit the models fully on the GPU. The specific value depends on your memory resources and finding the optimal value requires some experimentation
--jinja	Tell llama.cpp to use the Jinja chat-template embedded in the GGUF model file

 
 
 

Apple Silicon

Apple Silicon devices have unified memory that is seamlessly shared between the CPU and GPU. For optimal performance it is recommended to not exceed 70% of the total memory that your device has.

Tip

Install the latest llama.cpp package from Homebrew with:

brew install llama.cpp

Tip

To increase the amount of RAM available to the llama-server process, use the following command:

# on a 192GB machine, raise the limit from 154GB (default) to 180GB
sudo sysctl iogpu.wired_limit_mb=180000

---

• ✅ Devices with more than 96GB RAM

  The M2 Max, M3 Max, M4 Max, M1 Ultra, M2 Ultra, M3 Ultra, etc. chips can run both models at full context:

  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟢 Benchmarks on M2 Ultra (192GB) for `gpt-oss-20b`

  llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

  model	size	params	backend	n_ubatch	fa	test	t/s
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp2048	2191.13 ± 2.65
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp8192	1889.83 ± 3.91
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp16384	1594.51 ± 2.42
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp32768	1218.99 ± 0.44
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	tg128	116.08 ± 0.18

  build: 79c1160 (6123)

  llama-batched-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 132096 -b 2048 -ub 2048 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1,2,4

  PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
  0	128	1	128	0.000	0.00	1.112	115.06	1.113	115.05
  0	128	2	256	0.000	0.00	1.601	159.92	1.601	159.91
  0	128	4	512	0.000	0.00	2.463	207.85	2.463	207.84
  2048	128	1	2176	0.990	2068.28	1.163	110.03	2.154	1010.44
  2048	128	2	4352	1.916	2137.49	1.710	149.72	3.626	1200.17
  2048	128	4	8704	3.775	2169.82	2.656	192.78	6.431	1353.37
  8192	128	1	8320	4.344	1885.93	1.279	100.11	5.622	1479.81
  8192	128	2	16640	8.689	1885.52	1.929	132.69	10.619	1567.04
  8192	128	4	33280	17.359	1887.62	3.053	167.69	20.413	1630.35
  16384	128	1	16512	10.202	1606.01	1.397	91.63	11.599	1423.63
  16384	128	2	33024	20.715	1581.82	2.186	117.08	22.902	1441.98
  16384	128	4	66048	41.721	1570.80	3.653	140.14	45.375	1455.61
  32768	128	1	32896	26.611	1231.39	1.665	76.88	28.276	1163.40
  32768	128	2	65792	54.977	1192.06	2.794	91.64	57.771	1138.85
  32768	128	4	131584	111.278	1177.88	4.883	104.85	116.161	1132.77

  llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟢 Benchmarks on M2 Ultra (192 GB) for `gpt-oss-120b`

  llama-bench -m gpt-oss-120b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

  model	size	params	backend	n_ubatch	fa	test	t/s
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	pp2048	1244.57 ± 5.10
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	pp8192	1101.31 ± 0.99
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	pp16384	955.41 ± 0.64
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	pp32768	752.31 ± 1.02
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	tg128	79.68 ± 0.12

  build: 79c1160 (6123)

  llama-batched-bench -m gpt-oss-120b-mxfp4.gguf -ngl 99 -fa -c 132096 -b 2048 -ub 2048 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1,2,4

  PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
  0	128	1	128	0.000	0.00	1.610	79.48	1.611	79.48
  0	128	2	256	0.000	0.00	2.284	112.08	2.284	112.08
  0	128	4	512	0.000	0.00	3.477	147.27	3.477	147.27
  2048	128	1	2176	1.776	1152.89	1.711	74.82	3.487	623.99
  2048	128	2	4352	3.382	1211.16	2.458	104.14	5.840	745.18
  2048	128	4	8704	6.505	1259.34	3.747	136.65	10.252	849.02
  8192	128	1	8320	7.294	1123.16	1.857	68.94	9.150	909.25
  8192	128	2	16640	14.467	1132.48	2.767	92.53	17.234	965.53
  8192	128	4	33280	28.801	1137.74	4.358	117.50	33.159	1003.66
  16384	128	1	16512	16.580	988.15	2.058	62.18	18.639	885.89
  16384	128	2	33024	33.426	980.31	3.174	80.66	36.600	902.29
  16384	128	4	66048	67.190	975.39	5.245	97.61	72.435	911.83
  32768	128	1	32896	42.075	778.81	2.452	52.20	44.527	738.79
  32768	128	2	65792	86.615	756.64	4.029	63.54	90.644	725.83
  32768	128	4	131584	173.762	754.32	7.020	72.94	180.782	727.86

---

• ✅ Devices with less than 96GB RAM

  The small gpt-oss-20b model can run efficiently on Macs with at least 16GB RAM:

  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟢 Benchmarks on M4 Max (36GB) for `gpt-oss-20b`

  llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

  model	size	params	backend	n_ubatch	fa	test	t/s
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp2048	1277.42 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp8192	1030.28 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp16384	779.44 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp32768	568.13 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	tg128	92.36 ± 0.00

  build: 79c1160 (6123)

  llama-batched-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 132096 -b 2048 -ub 2048 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1

  PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
  0	128	1	128	0.000	0.00	1.359	94.17	1.359	94.15
  2048	128	1	2176	1.676	1222.17	1.450	88.30	3.125	696.26
  8192	128	1	8320	7.624	1074.47	1.552	82.47	9.176	906.67
  16384	128	1	16512	19.210	852.91	1.669	76.67	20.879	790.84
  32768	128	1	32896	55.684	588.46	1.976	64.76	57.661	570.51

  🟢 Benchmarks on M1 Max (64GB) for `gpt-oss-20b`

  llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

  model	size	params	backend	n_ubatch	fa	test	t/s
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp2048	994.75 ± 4.11
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp8192	843.01 ± 2.20
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp16384	698.82 ± 0.20
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp32768	497.65 ± 8.92
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	tg128	75.15 ± 0.98

  build: 2e2b22b (6180)

  🟢 Benchmarks on M1 Pro (32GB) for `gpt-oss-20b`

  llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384

  model	size	params	backend	n_ubatch	fa	test	t/s
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp2048	515.76 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp8192	437.22 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp16384	361.29 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	tg128	45.68 ± 0.00

  build: 79c1160 (6123)

  llama-batched-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 132096 -b 2048 -ub 2048 -npp 0,2048,8192,16384 -ntg 128 -npl 1

  PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
  0	128	1	128	0.000	0.00	2.806	45.62	2.806	45.62
  2048	128	1	2176	4.054	505.14	3.076	41.61	7.130	305.18
  8192	128	1	8320	18.444	444.15	3.329	38.45	21.773	382.12
  16384	128	1	16512	44.683	366.67	3.780	33.86	48.464	340.71

---

• 🟥 Devices with 8GB RAM

  Unfortunately, you are out of luck. The gpt-oss models are not possible to run on Macs with that small amount of memory.

---

 
 
 

NVIDIA

• ✅ Devices with more than 64GB VRAM

  With more than 64B VRAM, you can run both models by offloading everything (both the model and the KV cache) to the GPU(s).

  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟢 Benchmarks on RTX Pro 6000 Max-Q (96GB) for `gpt-oss-20b`

  llama-bench -m ./gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

  ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
  ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
  ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, compute capability 12.0, VMM: yes

  model	size	params	backend	n_ubatch	fa	test	t/s
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	pp2048	9480.55 ± 44.01
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	pp8192	8921.62 ± 4.21
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	pp16384	8196.12 ± 19.16
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	pp32768	7050.35 ± 12.36
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	tg128	249.96 ± 0.99

  build: f08c4c0 (6199)

  llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟢 Benchmarks on RTX Pro 6000 Max-Q (96GB) for `gpt-oss-120b`

  llama-bench -m ./gpt-oss-120b-mxfp4/gpt-oss-120b-mxfp4-00001-of-00003.gguf -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

  ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
  ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
  ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, compute capability 12.0, VMM: yes

  model	size	params	backend	n_ubatch	fa	test	t/s
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	pp2048	4494.20 ± 20.87
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	pp8192	4327.73 ± 16.04
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	pp16384	4114.04 ± 12.84
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	pp32768	3718.01 ± 19.67
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	tg128	170.62 ± 0.47

  build: f08c4c0 (6199)

---

• ✅ Devices with less than 64GB VRAM

  In this case, you can fit the small gpt-oss-20b model fully in VRAM for optimal performance.

  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟡 TODO: benchmarks for `gpt-oss-20b`

  TODO

  The large model has to be partially kept on the CPU.

  🟡 TODO: add commands for gpt-oss-120b

---

• ✅ Devices with 16GB VRAM

  For example: NVIDIA V100

  This config is just at the edge to fit the full context of gpt-oss-20b in VRAM, so we have to restrict the maximum context down to 32k tokens.

  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 32768 --jinja -ub 4096 -b 4096 -ngl 99 -fa

  🟢 Benchmarks on NVIDIA V100 (16GB) for `gpt-oss-20b`

  llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa 1 -b 4096 -ub 4096 -p 2048,8192,16384,32768

  ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
  ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
  ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla V100-PCIE-16GB, compute capability 7.0, VMM: yes

  model	size	params	backend	ngl	n_batch	n_ubatch	fa	test	t/s
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	pp2048	3526.65 ± 346.86
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	pp8192	3320.62 ± 44.98
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	pp16384	2768.99 ± 19.73
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	pp32768	2096.44 ± 8.58
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	tg128	117.71 ± 0.30

  build: 228f724 (6129)

  llama-batched-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 33792 -b 4096 -ub 4096 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1

  PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
  0	128	1	128	0.000	0.00	1.106	115.74	1.106	115.72
  2048	128	1	2176	0.481	4257.50	1.201	106.60	1.682	1293.82
  8192	128	1	8320	2.247	3646.05	1.417	90.31	3.664	2270.69
  16384	128	1	16512	5.421	3022.12	1.659	77.14	7.081	2331.96
  32768	128	1	32896	15.031	2180.10	2.121	60.35	17.151	1917.98

  Running the large gpt-oss-120b model with 16GB of VRAM requires to keep some of the layers on the CPU since it does not fit completely in VRAM:

  llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 32768 --jinja -ub 4096 -b 4096 -ngl 99 -fa --n-cpu-moe 32

---

• ✅ Devices with less than 16GB VRAM

  For this config, it is recommended to tell llama.cpp to run the entire model on the GPU while keeping enough layers on the CPU. Here is a specific example with an RTX 2060 8GB machine:

  # gpt-oss-20b, full context, 22 layers on the CPU
  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0     --jinja -ub 2048 -b 2048 -ngl 99 -fa --n-cpu-moe 22
  
  # gpt-oss-20b, 32k context, 16 layers on the CPU (faster, but has less total context)
  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 32768 --jinja -ub 2048 -b 2048 -ngl 99 -fa --n-cpu-moe 16

  Note that even with just 8GB of VRAM, we can adjust the CPU layers so that we can run the large 120B model too:

  # gpt-oss-120b, 32k context, 35 layers on the CPU
  llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 32768 --jinja -ub 2048 -b 2048 -ngl 99 -fa --n-cpu-moe 35

Tip

For more information about how to adjust the CPU layers, see the "Tips" section at the end of this guide.

---

 
 
 

AMD

Note

If you have AMD hardware, please provide feedback about running the gpt-oss models on it and the performance that you observe. See the sections above for what kind of commands to try and try to adjust respectively.

With AMD devices, you can use either the ROCm or the Vulkan backends. Depending on your specific hardware, the results can vary.

---

• ✅ RX 7900 XT (20GB VRAM) using ROCm backend

  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟢 Benchmarks for `gpt-oss-20b`

  llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

  ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
  ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
  ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 7900 XT, gfx1100 (0x1100), VMM: no, Wave Size: 32

  model	size	params	backend	ngl	threads	n_batch	n_ubatch	fa	test	t/s
  gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	pp2048	4251.56 ± 21.68
  gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	pp8192	3567.45 ± 11.84
  gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	pp16384	2948.39 ± 10.34
  gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	pp32768	2099.25 ± 13.17
  gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	tg128	101.92 ± 0.27

  build: 3007baf (6194)

More information: #15396 (comment)

 
 
 

Tips

Determining the optimal number of layers to keep on the CPU

Good general advice for most MoE models would be to offload the entire model, and use -n-cpu-moe to keep as many MoE layers as necessary on the CPU. The minimum amount of VRAM to do this with the 120B model is about 8GB, below that you will need to start reducing context size and the number of layers offloaded. You can get for example about 30 t/s at zero context on a 5090 with --n-cpu-moe 21.

Caveat: on Windows it is possible to allocate more VRAM than available, and the result will be slow swapping to RAM and very bad performance. Just because the model loads without errors, it doesn't mean you have enough VRAM for the settings that you are using. A good way to avoid this is to look at the "GPU Memory" in Task Manager and check that it does not exceed the GPU VRAM.

Example on 5090 (32GB):
good, --n-cpu-moe 21, GPU Memory < 32:

bad, --n-cpu-moe 20, GPU Memory > 32:

Using `gpt-oss` + `llama.cpp` with coding agents (such as Claude Code, crush, etc.)

• Setup the coding agent of your choice to look for a localhost OAI endpoint (see Tutorial: Offline Agentic coding with llama-server #14758)

• Start llama-server like this:

  # adjust this command for your hardware
  llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa
  
  # some agents such as Claude Code can benefit from multiple parallel server slots
  # note: currently this requires extra memory!
  llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 524288 -np 4 --jinja -ub 2048 -b 2048 -ngl 99 -fa

• Sample usage with crush: guide : running gpt-oss with llama.cpp #15396 (reply in thread)

Configure the default sampling and reasoning settings

When starting a llama-server command, you can change the default sampling and reasoning settings like so:

# use recommended gpt-oss sampling params
llama-server ... --temp 1.0 --top-p 1.0

# set default reasoning effort
llama-server ... --chat-template-kwargs '{"reasoning_effort": "high"}'

Note that these are just the default settings and they could be overridden by the client connecting to the llama-server.

Frequently asked questions

Q: Which quants to use?

Always use the original MXFP4 model files. The gpt-oss models are natively "quantized". I.e. they are trained in the MXFP4 format which is roughly equivalent to ggml's Q4_0. The main difference with Q4_0 is that the MXFP4 models get to keep their full quality. This means that no quantization in the usual sense is necessary.

Q: What sampling parameters to use?

OpenAI recommends: temperature=1.0 and top_p=1.0.

Do not use repetition penalties! Some clients tend to enable repetition penalties by default - make sure to disable those.

Be careful when you disable the Top K sampler. Although recommended by OpenAI, this can lead to significant CPU overhead and small but non-zero probability of sampling low-probability tokens.

Known issues

Some rough edges in the implementation are still being polished. Here is a list of issue to keep track of:

• Eval bug: Jinja fails on gpt-oss-120b when using Vulkan #15274

[SteelPh0enix]

I can provide some numbers for AMD part of the guide.

My hardware is RX 7900 XT (20GB VRAM) + Ryzen 9 5900X + 32GB of RAM, running on latest Arch Linux with locally built llama.cpp version 6194 (3007baf), built with ROCm 6.4.1-1 (from official Arch repo)

Pulled the gpt-oss-20b repository and converted to GGUF using convert_hf_to_gguf.py, should probably result in the same GGUF file as on huggingface.

7900XT can load the full 20B model with full context without offloading MoE layers to CPU (although barely, because it will fill up the whole VRAM), by running

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa

With that, i get generation speeds (as reported by llama.cpp webui) at ~94 tokens/second, slowly going down as the context fills up.

I've also tested whether setting K/V cache quantization would help with model size or performance, but the result was... bad, performance was halved and CPU got involved... is this because of mxfp4 format of gpt-oss?

I'd also like to note that my PC likes to hang when i fill up my VRAM to the brim, so i've also checked out how gpt-oss-20b behaves when i off-load MoE layers to CPU.

When running with all MoE layers on CPU, as below:

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa -cmoe

my GPU VRAM usage (as reported by btop) is around 10GB, RAM usage went up only ~2GB. However, the performance took a major 80% hit, as now my generation speed is in ~20tok/s - CPU takes most of the load. If you have better CPU and faster RAM (i'm still running dual-channel DDR4s @ 3200MHz CL16, mind you), you probably will get better results. I wonder how X3Ds behave in that case...

I assume that gpt-oss-20b has 24 MoE layers, so let's see how it behaves when i load only, let's say, 4 onto CPU:

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa -ncmoe 4

VRAM is at 18GB (previously it was at 19, as reported by btop, so there's a decrease), RAM usage went up by around 1.5GB, generation speed is ~60tok/s. Neat, this is usable.

How about 8 layers?

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa -ncmoe 8

In that case, i get 16GB VRAM usage, ~1.5GB RAM bump as previously, and generation speed went down to 38 tokens/s. Still pretty usable. How about 16 layers?

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa -ncmoe 16

VRAM: 13GB, RAM: as previously, not more than 2GB, generation speed: 27-25tok/s, this is getting pretty bad.

As mentioned before - your results may vary, i'm not running current-gen top-tier hardware and IIRC the largest performance bottleneck will be the RAM/PCIe link speed anyway - i'm pretty curious to see what the performance with this GPU is on more recent platform, especially with an X3D CPU.

[aldehir]

I had issues with a higher batch/ubatch size than the default but I'm not seeing that problem anymore so that was probably user error on my end.

I believe you are likely hitting the case where the model needs the CoT from the past tool call but the client isn't sending it in or there is a mismatch in the reasoning field. That is an open issue across all client and inference servers/providers with GPT-OSS.

If you can collect any dumps of this happening, I'm happy to dig in further.

[SteelPh0enix]

@SteelPh0enix I've been able to get crush to work with moderate success.

Can you share the llama-server command line arguments you pass in?

Sure @aldehir, here's my config for this model in crush.json:

 "providers": {
        "llamacpp": {
            "type": "openai",
            "base_url": "http://steelph0enix.pc:51536/v1",
            "name": "Llama.cpp",
            "id": "llamacpp",
            "models": [
                {
                    "id": "gpt-oss-20b.auto",
                    "name": "GPT-OSS 20B",
                    "context_window": 131072,
                    "default_max_tokens": 51200,
                    "has_reasoning_efforts": true,
                    "can_reason": true,
                    "supports_attachments": false,
                    "default_reasoning_effort": "high",
                    "cost_per_1m_in": 0,
                    "cost_per_1m_in_cached": 0,
                    "cost_per_1m_out": 0,
                    "cost_per_1m_out_cached": 0
                }
            ]
        }
    }
// ...

my llama-server invocation:

llama-server --ctx-size 0 --model gpt-oss-20b.auto.gguf --alias "gpt-oss-20b.auto" --jinja

i keep most of my llama-server settings in env vars, as following:

export LLAMA_ARG_HOST="0.0.0.0"
export LLAMA_ARG_PORT="51536"
export LLAMA_ARG_BATCH=2048
export LLAMA_ARG_UBATCH=2048
export LLAMA_ARG_SWA_FULL=false
export LLAMA_ARG_KV_SPLIT=false
export LLAMA_SET_ROWS=1 # for ARG_KV_SPLIT=false to work
export LLAMA_ARG_FLASH_ATTN=true
export LLAMA_ARG_MLOCK=true
export LLAMA_ARG_NO_MMAP=false
export LLAMA_ARG_N_GPU_LAYERS=999
export LLAMA_OFFLINE=true
export LLAMA_ARG_ENDPOINT_SLOTS=true
export LLAMA_ARG_ENDPOINT_PROPS=true

I've opened my test project (a Rust app) with gpt-oss-20b as chosen model in Crush and told it to initialize the project... and it seems to work just fine now!

I've tested Crush back on 0.6.0 (or 0.6.1) with gpt-oss, if not on 0.5.x, and i definitely had issues (for example, the chat description below CRUSH logo contained gpt-oss chat template tags...) so you must've fixed it already - i just haven't noticed :)

Thanks!

If you want, you may add my piece of crush.json to the post as a config example (change the IP to localhost though ;) ) @ggerganov, the invocation from the original post should work just fine

[SteelPh0enix]

I had issues with a higher batch/ubatch size than the default but I'm not seeing that problem anymore so that was probably user error on my end.

I believe you are likely hitting the case where the model needs the CoT from the past tool call but the client isn't sending it in or there is a mismatch in the reasoning field. I believe that's an open issue across all client and inference servers/providers with GPT-OSS.

If you can collect any dumps of this happening, I'm happy to dig in further.

Yes, i think i did notice that on some other models, i've been mostly working with Qwen... If this happens again, how can i get some more logs/info?

Oh, and one potential "issue" i've just noticed - i've set my reasoning in model config to "high", but crush seem to force "minimal", is this "by design", or is it some issue?

[aldehir]

I can't say, but it likely has no effect since llama-server only respects the chat_template_kwargs.reasoning_effort field in the request. I doubt crush is setting it, so it defaults to "medium" unless you change it via command line.

I usually run mitmproxy in the background, but enabling verbose and searching for parse errors in the server log will likely show the root of the problem--if there is one.

[SteelPh0enix]

btw @ggerganov i think you made a mistake labeling my test results, i don't have a mac, and they sure don't use AMD GPUs anymore 😄

[gcp]

Configure the default sampling and reasoning settings

When starting a llama-server command, you can change the default sampling and reasoning settings like so:

# use recommended gpt-oss sampling params
llama-server ... --temp 1.0 --top-p 1.0

Q: What sampling parameters to use?

OpenAI recommends: temperature=1.0 and top_p=1.0.

The problem I see is that the llama-server defaults are

--samplers SAMPLERS                     samplers that will be used for generation in the order, separated by
                                        ';'                                                                  
                                        (default:                                                            
                                        penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature)                                                                                            
--temp N                                temperature (default: 0.8)                                           
--top-k N                               top-k sampling (**default: 40**, 0 = disabled)                         
--top-p N                               top-p sampling (default: 0.9, 1.0 = disabled)
--min-p N                               min-p sampling (**default: 0.1**, 0.0 = disabled)   

So the above command is actually equivalent to:

llama-server ... --temp 1.0 --top-p 1.0 --top-k 40 --min-p 0.1

Which seems quite a bit different from the actual recommendation from OpenAI. Notably "min-p 0.1" will prune a lot of low-probability tokens, whereas the OpenAI recommendation is basically to follow the model output probabilities.

If you look at a lot of guides and settings for other SOTA LLM, they all recommend min-p 0.01 or 0.00.

Should the command line be changed to:

llama-server ... --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0

[Mushoz]

Note that with the recommended settings you are effectively sampling each token out of a ~200k vocabulary, which can be slow especially if you are offloading to the CPU. Setting a low --min-p value (0.01) or high top-k (1000) doesn't really change the result very much, but improves sampling speeds considerably. as the distribution that is being samples from will shrink considerably.

[SmallAndSoft]

It is not enough to define sampling parameters, one has to define the sampler chain itself, such as --sampling-seq tp for the chain recommended by OAI.

That said, I concur that their recommendation is nonsensical:
Temperature of 1 says "multiply all probabilities by 1" which leaves them unchanged.
top_p of 1 says "select tokens that add up to 100% probability" which is all of them.

As for me, I prefer --sampling-seq ks with top_k=2048 for speedup and then top_n_sigma with the default parameter of 1.

[gcp]

one has to define the sampler chain itself...for the chain recommended by OAI

With the settings given combined with the built-in defaults, the extra samplers are already disabled, so I do not think that is actually a requirement?

then top_n_sigma with the default parameter of 1.

This isn't supported in llama-server. (A number of claims in top-n-sigma paper fall flat when temperature is applied last, as is the case for llama.cpp, so I'm not sure this is going to change any time soon)

That said, I concur that their recommendation is nonsensical

The one from OpenAI at least is internally consistent: it means to sample the model output exactly as given. This is unusual however because it means there's no filtering on low-probability tokens whatsoever, so there's a low (but non-zero) chance you get a low probability token.

The llama.cpp default of "min-p 0.1" filters quite a bit more, so that's where we get the inconsistency, i.e. if you run llama.cpp with those parameters, the defaults will change the behavior quite a bit from the intention, it ends up actually quite aggressive in pruning low probability tokens.

[ggerganov]

Regardless of the OpenAI recommendation, I still think it's a good idea to filter low-probability tokens (for example with Top-K or Min-P).

For now, I've updated the guide with the following paragraph:

Be careful when you disable the `Top K` sampler. Although recommended by OpenAI, this can lead to significant CPU overhead and small but non-zero probability of sampling low-probability tokens.

We can revisit if we determine that sampling from the full vocab is actually important.

[SmallAndSoft]

then top_n_sigma with the default parameter of 1.

This isn't supported in llama-server. (A number of claims in top-n-sigma paper fall flat when temperature is applied last, as is the case for llama.cpp, so I'm not sure this is going to change any time soon)

Not sure what you mean. The support was merged in #13264
Temperature can be applied at any step you want if you define your own sampling chain.

[QuantiusBenignus]

To fill better the low-end CUDA edge cases, here are some benchmarks for gpt-oss-20B (both MXFP4 and Unsloth UD quant) on 12GB VRAM:
Ryzen 7 7700 with 32GB RAM (PCIe 4), NVIDIA RTX3060, 12GB VRAM, with CUDA 13.0:
llama.cpp build: 6139

Optimal settings at 16K context window:
Comparing -ncmoe N vs. offloading just some of the later up-projection layers, e.g. -ot "\.([2-9][0-9])\.ffn_up_exps.=CPU". My reasoning was that front of NN is more 'expensive' to offload due to early layers seeing the full sequence - more work per token. In practice, not huge difference in my scenario:

❯llama-server -t 8 -m redacted/gpt-oss-20b-UD-Q4_K_XL.gguf -ngl 99 -fa -c 16384 --min-p 0.0 --temp 1.0 --top-p 1.0 --top-k 0.0 --jinja --reasoning-format none --no-mmap -ot "\.([2-9][0-9])\.ffn_up_exps.=CPU"

(Leaves about 600 MB VRAM budget, with 67 tok/sec initial generation rate)
or

❯llama-server -t 8 -m redacted/gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 16384 --min-p 0.0 --temp 1.0 --top-p 1.0 --top-k 0.0 --jinja --reasoning-format none --no-mmap -ncmoe 2

(Leaves about 600 MB VRAM budget, with 64 tok/sec initial generation rate)

Optimal settings at 32K context window:

❯llama-server -t 8 -m redacted/gpt-oss-20b-UD-Q4_K_XL.gguf -ngl 99 -fa -c 32768 --min-p 0.0 --temp 1.0 --top-p 1.0 --top-k 0.0 --jinja --reasoning-format none --no-mmap -ot "\.([1-9][0-9])\.ffn_up_exps.=CPU"

(Leaves about 1.2 GB VRAM budget, with 53 tok/sec initial generation rate)
or

❯llama-server -t 8 -m redacted/gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 32768 --min-p 0.0 --temp 1.0 --top-p 1.0 --top-k 0.0 --jinja --reasoning-format none --no-mmap -ncmoe 3

(Leaves about 600 MB VRAM budget, with 56 tok/sec initial generation rate - this is too aggresive, will likely OOM before reaching context limit.)

#llama-bench for gpt-oss-20b-mxfp4.gguf:
bin❯master❯./llama-bench -m redacted/gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384 -ot "\.([1-9][0-9])\.ffn_up_exps.=CPU"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
| model                          |       size |     params | backend    | ngl | threads | n_ubatch | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -------: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss ?B MXFP4 MoE           |  11.27 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |          pp2048 |       2229.95 ± 1.93 |
| gpt-oss ?B MXFP4 MoE           |  11.27 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |          pp8192 |       2108.57 ± 6.36 |
| gpt-oss ?B MXFP4 MoE           |  11.27 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |         pp16384 |       1960.34 ± 2.08 |
| gpt-oss ?B MXFP4 MoE           |  11.27 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |           tg128 |         30.64 ± 0.08 |



#gpt-oss-20b-UD-Q4_K_XL.gguf (Unsloth)
bin❯master❯./llama-bench -m redacted/gpt-oss-20b-UD-Q4_K_XL.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384 -ot "\.([1-9][0-9])\.ffn_up_exps.=CPU"  
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
| model                          |       size |     params | backend    | ngl | threads | n_ubatch | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -------: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss ?B Q4_K - Medium       |  11.04 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |          pp2048 |       2212.30 ± 4.04 |
| gpt-oss ?B Q4_K - Medium       |  11.04 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |          pp8192 |       2092.57 ± 6.44 |
| gpt-oss ?B Q4_K - Medium       |  11.04 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |         pp16384 |       1948.17 ± 0.93 |
| gpt-oss ?B Q4_K - Medium       |  11.04 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |           tg128 |         31.30 ± 0.07 |

build: 6139

Similar results for latest build: 6195.

For completeness, when running with a small context window of 2048 tokens ( when everything fits in the GPU, i.e. no offloading), the inference speed reaches 75 tok/sec. This is not irrelevant even for a reasoning model because there are scenarios where this window is sufficient for one-off tasks.
Excellent inference speed with a decent LLM, indeed.

[aldehir]

Several people are having issues with tool calling in Cline/Roo Code when using gpt-oss-20b. This is because those clients do not use native tool calling and the model insists on native tool calls. There is a workaround by using a custom grammar that inhibits native tool calling:

root ::= analysis? start final .+
analysis ::= "<|channel|>analysis<|message|>" ( [^<] | "<" [^|] | "<|" [^e] )* "<|end|>"
start ::= "<|start|>assistant"
final ::= "<|channel|>final<|message|>"

Passing this in a file with --grammar-file yields good results when coupled with this system prompt:

Valid channels: analysis, final. Channel must be included for every message.

Is this something useful to include in the docs?

[ggerganov]

Could you ELI5 the difference between native and non-native tool calls? Or point me to a reference document.

[Mushoz]

Are we sure tool calling is currently implemented correctly? Openai has released a test script ( https://cookbook.openai.com/articles/gpt-oss/verifying-implementations ) to test backend implementations, but it's currently failing me with llama.cpp. Steps to run the test script:

git clone https://github.com/openai/openai-cookbook.git

cd gpt-oss/compatibility-test/

npm install
npm i -D tsx typescript @types/node

Then edit the providers.ts file (edit the correct details in):

export const PROVIDERS = {
  openai: {
    apiBaseUrl: "http://localhost:3001/v1",
    apiKey: "key",
    apiType: ["chat"], // choose from responses, chat, or both
    modelName: "GPT-OSS-120B",
    providerDetails: {
      // add any provider-specific details here. These will be passed as part of every request
      // for example to fix the provider for openrouter, you can do:
      // provider: {
      //   only: ["example"],
      // },
    },
  },
};

And then lastly start the test: npm start -- --provider openai

These are the results I obtained:

Summary:
  Provider: openai
  Total input cases: 30
  Tries: 1
  Total tasks: 29
  Total runs: 29
  Invalid Chat Completions API responses: 29 (out of 29)
  pass@k (k=1..1): 1=0.000
  pass^k (k=1..1): 1=0.000
  pass@k (k=1): 0.000
  pass^k (k=1): 0.000
  Wrong-input tool calls: 5
  Invalid cases.jsonl lines: 0

Expected outcome according to the guide: If your tests are successful, the output should show 0 invalid requests and over 90% on both pass@k and pass^k. This means the implementation should likely be correct.

Could anyone try replicating my findings? If they find the same, what should be done to fix this?

[Mushoz]

I see the following in runCase.ts:

      // this is the actual HTTP response from the provider
      // Since it's not guaranteed that every response has a reasoning field, we check if it's present
      // at least once across all responses
      const data = response.providerData;
      const message = data.choices[0].message;
      if (message.role === "assistant" && !message.refusal) {
        details.hasReasoningField =
          details.hasReasoningField ||
          ("reasoning" in message && typeof message.reasoning === "string");
        details.hasReasoningContentField =
          details.hasReasoningContentField ||
          ("reasoning_content" in message &&
            typeof message.reasoning_content === "string");

        validResponse =
          validResponse ||
          (details.hasReasoningField && message.reasoning.length > 0);
      }

It looks like it's already checking both reasoning as well as reasoning_content?

[aldehir]

Yes, but it still fails the test. Look at the validity check at the bottom, it does not test for reasoning content.

[Mushoz]

Do you happen to have the patch to enable reasoning_content compatibility? Typescript is definitely not my strong suit. I tried changing the if (item.type === "reasoning") { check to if (item.type === "reasoning_content") {, but that didn't work.

[aldehir]

Do you happen to have the patch to enable reasoning_content compatibility? Typescript is definitely not my strong suit. I tried changing the if (item.type === "reasoning") { check to if (item.type === "reasoning_content") {, but that didn't work.

I can't produce one right this moment, but if you duplicate the validResponse line and change hasReasoningField to hasReasoningContentField, and message.reasoning to message.reasoning_content, it should work.

[aldehir]

It does look like they may have intended to pass the test if reasoning_content is set but forgot to add another line. That said, I do think the divergence between projects is problematic for clients. I created the discussion to see if there is community support behind adding a way to change the field.

[ericcurtin]

Maybe not relevant as the models are kinda large... But perf tested CPU inferencing on an Ampere system before --threads cores/2 was the sweet spot... Also --cache-reuse what are the considerations for use?

[mounta11n]

@ggerganov I've tested one more Apple Silicon. Here are the results of my MBP M1 Max 64GB

🟢 Benchmarks on M1 Max (64GB) for gpt-oss-20b

time llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

model	size	params	backend	threads	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal,BLAS	1	2048	1	pp2048	994.75 ± 4.11
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal,BLAS	1	2048	1	pp8192	843.01 ± 2.20
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal,BLAS	1	2048	1	pp16384	698.82 ± 0.20
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal,BLAS	1	2048	1	pp32768	497.65 ± 8.92
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal,BLAS	1	2048	1	tg128	75.15 ± 0.98

build: 2e2b22b (6180)

llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 10,31s user 2,38s system 2% cpu 10:15,13 total

[Art9681]

Getting the following error when attempting to use @playwright/mcp@latest MCP server. It's working well with other tools. This MCP server works fine with other models such as Devstral so its an issue with the gpt-oss-model implementation:

got exception: {"code":500,"message":"JSON schema conversion failed:\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}","type":"server_error"}

[aldehir]

I'm not seeing this problem with Chatbox + @playright/mcp@latest. Which client are you using?

[Art9681]

I'm using the official OpenAI Go SDK. I can enable other MCP servers and internal tool implementations according to spec and they work fine.

[aldehir]

I'm afraid I'm stumped, as I cannot reproduce this with Chatbox or Crush. Neither produce a JSON schema with "not": {} for the playwright MCP server.

From what I can tell, not is unsupported, but I'm not equipped to give you a good answer. I recommend creating an issue and maybe someone more knowledgeable can help.

[SoftwareRenderer]

Adding benchmarks for NVIDIA > 64 GB VRAM.

🟢 Benchmarks on RTX Pro 6000 Max-Q (96GB) for `gpt-oss-20b`

llama-bench -m ./gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, compute capability 12.0, VMM: yes

model	size	params	backend	ngl	threads	n_ubatch	fa	test	t/s
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	pp2048	9480.55 ± 44.01
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	pp8192	8921.62 ± 4.21
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	pp16384	8196.12 ± 19.16
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	pp32768	7050.35 ± 12.36
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	tg128	249.96 ± 0.99

build: f08c4c0 (6199)

🟢 Benchmarks on RTX Pro 6000 Max-Q (96GB) for `gpt-oss-120b`

llama-bench -m ./gpt-oss-120b-mxfp4/gpt-oss-120b-mxfp4-00001-of-00003.gguf -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, compute capability 12.0, VMM: yes

model	size	params	backend	ngl	threads	n_ubatch	fa	test	t/s
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	pp2048	4494.20 ± 20.87
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	pp8192	4327.73 ± 16.04
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	pp16384	4114.04 ± 12.84
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	pp32768	3718.01 ± 19.67
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	tg128	170.62 ± 0.47

build: f08c4c0 (6199)

[ggerganov]

Thanks for the data!

p.s. Accidentally, edited your comment instead of the guide - sorry about that :)

[kj-c0d3s]

Since I spent more time than I probably should have, here is some info for the list

🧱 Hardware Specifications

🖥️ CPU

• Model: AMD Ryzen Threadripper 1950X 16-Core Processor
• Cores/Threads: 16 cores / 32 threads
• Base Clock: 2.2 GHz
• Boost Clock: 3.75 GHz
• Sockets: 1
• NUMA Nodes: 1 (CPUs 0–31)

🧠 Memory (RAM)

• Total Capacity: 64 GB (4 × 16 GB)
• Speed: 3200 MT/s
• Channels: Quad-channel
• Type: DDR4

🎮 GPUs

5 × AMD Radeon Pro VII (Vega 20 (gfx906: xnack-), 16 GB HBM2 each)

GPU ID	Memory Vendor	VBIOS Version
GPU 0	Hynix	113-D1640600-104
GPU 1	Hynix	113-D1640600-104
GPU 2	Hynix	113-D1640600-104
GPU 3	Samsung	113-D1640600-104
GPU 4	Samsung	113-D1640600-104

• Total VRAM: 80 GB
• ECC Support: Enabled
• IOMMU + HMM/SVM: Enabled (shared virtual memory for ROCm)
• Firmware & ROCm: Custom-built ROCm HIP stack with Flash attention support to enable functionality outside official compatibility list

🧩 PCIe Configuration

• Risers: 5 total
• Bifurcation Cards: 2 × 16x-to-8x dual-split
• Layout:
  • 3 GPUs on straight x16 risers
  • 2 GPUs connected via bifurcation (x8/x8) splitters

🐧 OS

• Distribution: Ubuntu 24.04 LTS

🧠 Inference Benchmark Summary

🏃 Run Command

./llama-server \
  --model gpt-oss-120b-F16.gguf \
  --threads 16 \
  --no-mmap \    #Prevents hang at 75% model loading (for me anyway)
  --flash-attn \
  --prio 2 \
  --n-gpu-layers 99 \
  --temp 1.0 \
  --top-p 1.0 \
  --top-k 0 \
  --min-p 0 \
  --no-warmup \
  --ubatch-size 2048 \
  --jinja \
  --chat-template-kwargs '{"reasoning_effort": "medium"}' \
  --ctx-size 32768

🔍 Inference Performance: 9K Prompt + Generation (~11.4K tokens total)

📈 Performance Metrics

📤 Prompt Tokenization

• Tokens: 9044
• Time: 77,223.972 ms
• Speed: ⚡ 117.1 tokens/sec

🧠 Token Generation

• Tokens: 2381
• Time: 134,268.908 ms
• Speed: 🐢 17.7 tokens/sec

⚙️ Total Workload: ~11,425 tokens

Let me know if there is anything else you'd like to see or know that may be helpful to others

[SmallAndSoft]

Try --split-mode row so your GPUs all work simultaneously instead of taking turns computing layers. You should see a massive tg speedup.

[ggerganov]

The --top-k 0 option is likely slowing text generation a lot.