Models
Sources
There are a few sources of preconverted models which can be used with OpenArc
If you need help converting a particular model join Discord and we can help you!
| Models |
|---|
| gemma-3-4b-it-int8_asym-ov |
| Gemma-3-12b-it-qat-int4_asym-ov |
| Qwen2.5-VL-7B-Instruct-int4_sym-ov |
| Nanonets-OCR2-3B-LM-INT4_ASYM-VE-FP16-ov |
| Qwen3-VL-4B-Instruct-int4_asym-ov |
Whisper
| Models |
|---|
| distil-whisper-large-v3-int8-ov |
| distil-whisper-large-v3-fp16-ov |
| whisper-large-v3-int8-ov |
| openai-whisper-large-v3-fp16-ov |
Qwen3-ASR
| Models |
|---|
| Qwen3-ASR-0.6B-INT8_ASYM-OpenVINO |
Kokoro
| Models |
|---|
| Kokoro-82M-FP16-OpenVINO |
Qwen3-TTS
| Models |
|---|
| Qwen3-TTS-12Hz-CustomVoice-1.7B-INT8-OpenVINO |
| Qwen3-TTS-12Hz-VoiceDesign-1.7B-INT8-OpenVINO |
| Qwen3-TTS-12Hz-Base-1.7B-INT8-OpenVINO |
| Models |
|---|
| Qwen3-Embedding-0.6B-int8_asym-ov |
| Models |
|---|
| Qwen3-Reranker-0.6B-fp16-ov |
Model-Specific Instructions
Qwen3.5/3.6
Is Qwen3.5/3.6 supported?
Qwen3.5 models has unofficial support. However, they do require you to build openvino and openvino.genai from source. You will also need to install the latest version of optimum-intel.
To add a model, run the command openarc add --model-name MODEL_NAME --model-path /path/to/model --model-type vlm --device GPU|CPU --runtime-config '{"ATTENTION_BACKEND": "SDPA"}'. OpenArc resolves the VLM vision token from the model's config.json. Intel is currently working on adding support for Qwen3.5 to utilize the PA attention backend but it has not been merged yet. This currently appears to be much more performant. If you have built openvino.genai with the support included, you may change the runtime config parameter to use PA instead.
How do I control thinking?
Qwen3.5 utilizes chat instructions for thinking control. You can enable thinking by using the parameter chat_template_kwargs with a value of {"enable_thinking": true} and disable it by setting the value to {"enable_thinking": false}.
Previous reasoning is also retained within a conversation. You can enable or disable it similarily by adding preserve_previous_think to chat_template_kwargs.
For example, to enable thinking and disable previous reasoning, you would pass chat_template_kwargs with a value of {"enable_thinking": true, "preserve_previous_think": false}.
Qwen3 MoE
GPU loading fails with "FuseMOE3GemmCompressed transformation did not match the routing subgraph"?
Qwen3 MoE models (e.g., Qwen3-Coder-30B-A3B-Instruct) require a recent OpenVINO build on GPU. Older builds fail during pipeline creation with:
[GPU] MOECompressed (GEMM3_SWIGLU) reached the GPU backend without being fused: FuseMOE3GemmCompressed transformation did not match the routing subgraph
This was a bug in the OpenVINO GPU plugin's MoE fusion pass and is fixed upstream in nightly 2026.4.0.dev20260730 and newer. Upgrade with:
uv pip install --pre -U openvino openvino-genai openvino-tokenizers --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
After upgrading, clear the model cache directory (if one is configured) so the model recompiles, and restart the server if it was running before the upgrade.