@@ -5,7 +5,7 @@ Intel® Extension for PyTorch\*
55
66</div >
77
8- ** CPU** [ 💻main branch] ( https://github.com/intel/intel-extension-for-pytorch/tree/main )   ;  ;  ; |  ;  ;  ; [ 🌱Quick Start] ( https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/getting_started.html )   ;  ;  ; |  ;  ;  ; [ 📖Documentations] ( https://intel.github.io/intel-extension-for-pytorch/cpu/latest/ )   ;  ;  ; |  ;  ;  ; [ 🏃Installation] ( https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=cpu&version=v2.6 .0%2Bcpu )   ;  ;  ; |  ;  ;  ; [ 💻LLM Example] ( https://github.com/intel/intel-extension-for-pytorch/tree/main/examples/cpu/llm ) <br >
8+ ** CPU** [ 💻main branch] ( https://github.com/intel/intel-extension-for-pytorch/tree/main )   ;  ;  ; |  ;  ;  ; [ 🌱Quick Start] ( https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/getting_started.html )   ;  ;  ; |  ;  ;  ; [ 📖Documentations] ( https://intel.github.io/intel-extension-for-pytorch/cpu/latest/ )   ;  ;  ; |  ;  ;  ; [ 🏃Installation] ( https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=cpu&version=v2.7 .0%2Bcpu )   ;  ;  ; |  ;  ;  ; [ 💻LLM Example] ( https://github.com/intel/intel-extension-for-pytorch/tree/main/examples/cpu/llm ) <br >
99** GPU** [ 💻main branch] ( https://github.com/intel/intel-extension-for-pytorch/tree/xpu-main )   ;  ;  ; |  ;  ;  ; [ 🌱Quick Start] ( https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/getting_started.html )   ;  ;  ; |  ;  ;  ; [ 📖Documentations] ( https://intel.github.io/intel-extension-for-pytorch/xpu/latest/ )   ;  ;  ; |  ;  ;  ; [ 🏃Installation] ( https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=gpu )   ;  ;  ; |  ;  ;  ; [ 💻LLM Example] ( https://github.com/intel/intel-extension-for-pytorch/tree/xpu-main/examples/gpu/llm ) <br >
1010
1111Intel® Extension for PyTorch\* extends PyTorch\* with up-to-date features optimizations for an extra performance boost on Intel hardware. Optimizations take advantage of Intel® Advanced Vector Extensions 512 (Intel® AVX-512) Vector Neural Network Instructions (VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel X<sup >e</sup > Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs through the PyTorch* xpu device.
@@ -16,51 +16,60 @@ In the current technological landscape, Generative AI (GenAI) workloads and mode
1616
1717### Optimized Model List
1818
19- | MODEL FAMILY | MODEL NAME (Huggingface hub) | FP32 | BF16 | Static quantization INT8 | Weight only quantization INT8 | Weight only quantization INT4 |
20- | :---:| :---:| :---:| :---:| :---:| :---:| :---:|
21- | LLAMA| meta-llama/Llama-2-7b-hf | ✅ | ✅ | ✅ | ✅ | ✅ |
22- | LLAMA| meta-llama/Llama-2-13b-hf | ✅ | ✅ | ✅ | ✅ | ✅ |
23- | LLAMA| meta-llama/Llama-2-70b-hf | ✅ | ✅ | ✅ | ✅ | ✅ |
24- | LLAMA| meta-llama/Meta-Llama-3-8B | ✅ | ✅ | ✅ | ✅ | ✅ |
25- | LLAMA| meta-llama/Meta-Llama-3-70B | ✅ | ✅ | ✅ | ✅ | ✅ |
26- | LLAMA| meta-llama/Meta-Llama-3.1-8B-Instruct | ✅ | ✅ | ✅ | ✅ | ✅ |
27- | LLAMA| meta-llama/Llama-3.2-3B-Instruct | ✅ | ✅ | ✅ | ✅ | ✅ |
28- | LLAMA| meta-llama/Llama-3.2-11B-Vision-Instruct | ✅ | ✅ | | ✅ | ✅ |
29- | GPT-J| EleutherAI/gpt-j-6b | ✅ | ✅ | ✅ | ✅ | ✅ |
30- | GPT-NEOX| EleutherAI/gpt-neox-20b | ✅ | ✅ | ✅ | ✅ | ✅ |
31- | DOLLY| databricks/dolly-v2-12b | ✅ | ✅ | ✅ | ✅ | ✅ |
32- | FALCON| tiiuae/falcon-7b | ✅ | ✅ | ✅ | ✅ | ✅ |
33- | FALCON| tiiuae/falcon-11b | ✅ | ✅ | ✅ | ✅ | ✅ |
34- | FALCON| tiiuae/falcon-40b | ✅ | ✅ | ✅ | ✅ | ✅ |
35- | OPT| facebook/opt-30b | ✅ | ✅ | ✅ | ✅ | ✅ |
36- | OPT| facebook/opt-1.3b | ✅ | ✅ | ✅ | ✅ | ✅ |
37- | Bloom| bigscience/bloom-1b7 | ✅ | ✅ | ✅ | ✅ | ✅ |
38- | CodeGen| Salesforce/codegen-2B-multi | ✅ | ✅ | ✅ | ✅ | ✅ |
39- | Baichuan| baichuan-inc/Baichuan2-7B-Chat | ✅ | ✅ | ✅ | ✅ | ✅ |
40- | Baichuan| baichuan-inc/Baichuan2-13B-Chat | ✅ | ✅ | ✅ | ✅ | ✅ |
41- | Baichuan| baichuan-inc/Baichuan-13B-Chat | ✅ | ✅ | ✅ | ✅ | ✅ |
42- | ChatGLM| THUDM/chatglm3-6b | ✅ | ✅ | ✅ | ✅ | ✅ |
43- | ChatGLM| THUDM/chatglm2-6b | ✅ | ✅ | ✅ | ✅ | ✅ |
44- | GPTBigCode| bigcode/starcoder | ✅ | ✅ | ✅ | ✅ | ✅ |
45- | T5| google/flan-t5-xl | ✅ | ✅ | ✅ | ✅ | ✅ |
46- | MPT| mosaicml/mpt-7b | ✅ | ✅ | ✅ | ✅ | ✅ |
47- | Mistral| mistralai/Mistral-7B-v0.1 | ✅ | ✅ | ✅ | ✅ | ✅ |
48- | Mixtral| mistralai/Mixtral-8x7B-v0.1 | ✅ | ✅ | | ✅ | ✅ |
49- | Stablelm| stabilityai/stablelm-2-1_6b | ✅ | ✅ | ✅ | ✅ | ✅ |
50- | Qwen| Qwen/Qwen-7B-Chat | ✅ | ✅ | ✅ | ✅ | ✅ |
51- | Qwen| Qwen/Qwen2-7B | ✅ | ✅ | ✅ | ✅ | ✅ |
52- | LLaVA| liuhaotian/llava-v1.5-7b | ✅ | ✅ | | ✅ | ✅ |
53- | GIT| microsoft/git-base | ✅ | ✅ | | ✅ | ✅ |
54- | Yuan| IEITYuan/Yuan2-102B-hf | ✅ | ✅ | | ✅ | |
55- | Phi| microsoft/phi-2 | ✅ | ✅ | ✅ | ✅ | ✅ |
56- | Phi| microsoft/Phi-3-mini-4k-instruct | ✅ | ✅ | ✅ | ✅ | ✅ |
57- | Phi| microsoft/Phi-3-mini-128k-instruct | ✅ | ✅ | ✅ | ✅ | ✅ |
58- | Phi| microsoft/Phi-3-medium-4k-instruct | ✅ | ✅ | ✅ | ✅ | ✅ |
59- | Phi| microsoft/Phi-3-medium-128k-instruct | ✅ | ✅ | ✅ | ✅ | ✅ |
60- | Whisper| openai/whisper-large-v2 | ✅ | ✅ | ✅ | ✅ | ✅ |
61- | Maira| microsoft/maira-2 | ✅ | ✅ | | ✅ | ✅ |
62- | Jamba| ai21labs/Jamba-v0.1 | ✅ | ✅ | | ✅ | ✅ |
63- | DeepSeek| deepseek-ai/DeepSeek-V2.5-1210 | ✅ | ✅ | | ✅ | ✅ |
19+ We have supported a long list of LLMs, including the most notable open-source models
20+ like Llama series, Qwen series, Phi-3/Phi-4 series,
21+ and the phenomenal high-quality reasoning model DeepSeek-R1.
22+
23+ | MODEL FAMILY | MODEL NAME (Huggingface hub) | FP32 | BF16 | Weight only quantization INT8 | Weight only quantization INT4 |
24+ | :---:| :---:| :---:| :---:| :---:| :---:|
25+ | LLAMA| meta-llama/Llama-2-7b-hf | ✅ | ✅ | ✅ | ✅ |
26+ | LLAMA| meta-llama/Llama-2-13b-hf | ✅ | ✅ | ✅ | ✅ |
27+ | LLAMA| meta-llama/Llama-2-70b-hf | ✅ | ✅ | ✅ | ✅ |
28+ | LLAMA| meta-llama/Meta-Llama-3-8B | ✅ | ✅ | ✅ | ✅ |
29+ | LLAMA| meta-llama/Meta-Llama-3-70B | ✅ | ✅ | ✅ | ✅ |
30+ | LLAMA| meta-llama/Meta-Llama-3.1-8B-Instruct | ✅ | ✅ | ✅ | ✅ |
31+ | LLAMA| meta-llama/Llama-3.2-3B-Instruct | ✅ | ✅ | ✅ | ✅ |
32+ | LLAMA| meta-llama/Llama-3.2-11B-Vision-Instruct | ✅ | ✅ | ✅ | ✅ |
33+ | GPT-J| EleutherAI/gpt-j-6b | ✅ | ✅ | ✅ | ✅ |
34+ | GPT-NEOX| EleutherAI/gpt-neox-20b | ✅ | ✅ | ✅ | ✅ |
35+ | DOLLY| databricks/dolly-v2-12b | ✅ | ✅ | ✅ | ✅ |
36+ | FALCON| tiiuae/falcon-7b | ✅ | ✅ | ✅ | ✅ |
37+ | FALCON| tiiuae/falcon-11b | ✅ | ✅ | ✅ | ✅ |
38+ | FALCON| tiiuae/falcon-40b | ✅ | ✅ | ✅ | ✅ |
39+ | FALCON| tiiuae/Falcon3-7B-Instruct | ✅ | ✅ | ✅ | ✅ |
40+ | OPT| facebook/opt-30b | ✅ | ✅ | ✅ | ✅ |
41+ | OPT| facebook/opt-1.3b | ✅ | ✅ | ✅ | ✅ |
42+ | Bloom| bigscience/bloom-1b7 | ✅ | ✅ | ✅ | ✅ |
43+ | CodeGen| Salesforce/codegen-2B-multi | ✅ | ✅ | ✅ | ✅ |
44+ | Baichuan| baichuan-inc/Baichuan2-7B-Chat | ✅ | ✅ | ✅ | ✅ |
45+ | Baichuan| baichuan-inc/Baichuan2-13B-Chat | ✅ | ✅ | ✅ | ✅ |
46+ | Baichuan| baichuan-inc/Baichuan-13B-Chat | ✅ | ✅ | ✅ | ✅ |
47+ | ChatGLM| THUDM/chatglm3-6b | ✅ | ✅ | ✅ | ✅ |
48+ | ChatGLM| THUDM/chatglm2-6b | ✅ | ✅ | ✅ | ✅ |
49+ | GPTBigCode| bigcode/starcoder | ✅ | ✅ | ✅ | ✅ |
50+ | T5| google/flan-t5-xl | ✅ | ✅ | ✅ | ✅ |
51+ | MPT| mosaicml/mpt-7b | ✅ | ✅ | ✅ | ✅ |
52+ | Mistral| mistralai/Mistral-7B-v0.1 | ✅ | ✅ | ✅ | ✅ |
53+ | Mixtral| mistralai/Mixtral-8x7B-v0.1 | ✅ | ✅ | ✅ | ✅ |
54+ | Stablelm| stabilityai/stablelm-2-1_6b | ✅ | ✅ | ✅ | ✅ |
55+ | Qwen| Qwen/Qwen-7B-Chat | ✅ | ✅ | ✅ | ✅ |
56+ | Qwen| Qwen/Qwen2-7B | ✅ | ✅ | ✅ | ✅ |
57+ | Qwen| Qwen/Qwen2.5-7B-Instruct | ✅ | ✅ | ✅ | ✅ |
58+ | LLaVA| liuhaotian/llava-v1.5-7b | ✅ | ✅ | ✅ | ✅ |
59+ | GIT| microsoft/git-base | ✅ | ✅ | ✅ | ✅ |
60+ | Yuan| IEITYuan/Yuan2-102B-hf | ✅ | ✅ | ✅ | |
61+ | Phi| microsoft/phi-2 | ✅ | ✅ | ✅ | ✅ |
62+ | Phi| microsoft/Phi-3-mini-4k-instruct | ✅ | ✅ | ✅ | ✅ |
63+ | Phi| microsoft/Phi-3-mini-128k-instruct | ✅ | ✅ | ✅ | ✅ |
64+ | Phi| microsoft/Phi-3-medium-4k-instruct | ✅ | ✅ | ✅ | ✅ |
65+ | Phi| microsoft/Phi-3-medium-128k-instruct | ✅ | ✅ | ✅ | ✅ |
66+ | Phi| microsoft/Phi-4-mini-instruct | ✅ | ✅ | ✅ | |
67+ | Phi| microsoft/Phi-4-multimodal-instruct | ✅ | ✅ | ✅ | |
68+ | Whisper| openai/whisper-large-v2 | ✅ | ✅ | ✅ | ✅ |
69+ | Maira| microsoft/maira-2 | ✅ | ✅ | ✅ | ✅ |
70+ | Jamba| ai21labs/Jamba-v0.1 | ✅ | ✅ | ✅ | ✅ |
71+ | DeepSeek| deepseek-ai/DeepSeek-V2.5-1210 | ✅ | ✅ | ✅ | ✅ |
72+ | DeepSeek| meituan/DeepSeek-R1-Channel-INT8 | | | ✅ | |
6473
6574* Note* : The above verified models (including other models in the same model family, like "codellama/CodeLlama-7b-hf" from LLAMA family) are well supported with all optimizations like indirect access KV cache, fused ROPE, and customized linear kernels.
6675We are working in progress to better support the models in the tables with various data types. In addition, more models will be optimized in the future.
0 commit comments