Models
Sources
There are a few sources of preconverted models which can be used with OpenArc
If you need help converting a particular model join Discord and we can help you!
| Models |
|---|
| gemma-3-4b-it-int8_asym-ov |
| Gemma-3-12b-it-qat-int4_asym-ov |
| Qwen2.5-VL-7B-Instruct-int4_sym-ov |
| Nanonets-OCR2-3B-LM-INT4_ASYM-VE-FP16-ov |
| Qwen3-VL-4B-Instruct-int4_asym-ov |
Whisper
| Models |
|---|
| distil-whisper-large-v3-int8-ov |
| distil-whisper-large-v3-fp16-ov |
| whisper-large-v3-int8-ov |
| openai-whisper-large-v3-fp16-ov |
Qwen3-ASR
| Models |
|---|
| Qwen3-ASR-0.6B-INT8_ASYM-OpenVINO |
Kokoro
| Models |
|---|
| Kokoro-82M-FP16-OpenVINO |
Qwen3-TTS
| Models |
|---|
| Qwen3-TTS-12Hz-CustomVoice-1.7B-INT8-OpenVINO |
| Qwen3-TTS-12Hz-VoiceDesign-1.7B-INT8-OpenVINO |
| Qwen3-TTS-12Hz-Base-1.7B-INT8-OpenVINO |
| Models |
|---|
| Qwen3-Embedding-0.6B-int8_asym-ov |
| Models |
|---|
| Qwen3-Reranker-0.6B-fp16-ov |
Model-Specific Instructions
Qwen3.5/3.6
Is Qwen3.5/3.6 supported?
Qwen3.5 models has unofficial support. However, they do require you to build openvino and openvino.genai from source. You will also need to install the latest version of optimum-intel.
To add a model, run the command openarc add --model-name MODEL_NAME --model-path /path/to/model --model-type vlm --device GPU|CPU --runtime-config '{"ATTENTION_BACKEND": "SDPA"}'. OpenArc resolves the VLM vision token from the model's config.json. Intel is currently working on adding support for Qwen3.5 to utilize the PA attention backend but it has not been merged yet. This currently appears to be much more performant. If you have built openvino.genai with the support included, you may change the runtime config parameter to use PA instead.
How do I control thinking?
Qwen3.5 utilizes chat instructions for thinking control. You can enable thinking by using the parameter chat_template_kwargs with a value of {"enable_thinking": true} and disable it by setting the value to {"enable_thinking": false}.
Previous reasoning is also retained within a conversation. You can enable or disable it similarily by adding preserve_previous_think to chat_template_kwargs.
For example, to enable thinking and disable previous reasoning, you would pass chat_template_kwargs with a value of {"enable_thinking": true, "preserve_previous_think": false}.