Skip to content

Run Gemma with MLX

Models Gemma 4 Uses MLX

MLX is Apple's array framework for machine learning on Apple silicon. It runs Gemma natively on the unified memory architecture, without a GGUF conversion step.

Install

pip install mlx mlx-lm mlx-vlm

Run

mlx_lm.generate --model mlx-community/gemma-4-E2B-it-qat-4bit --prompt "Who are you?"

mlx_vlm.generate --model mlx-community/gemma-4-E2B-it-qat-4bit \
  --prompt "Describe this image." --image <path_to_image>

Model repositories

Community-converted weights live under mlx-community:

Size Repository
E2B mlx-community/gemma-4-E2B-it-qat-4bit
E4B mlx-community/gemma-4-E4B-it-qat-4bit
12B mlx-community/gemma-4-12B-it-qat-4bit
26B A4B mlx-community/gemma-4-26B-A4B-it-qat-4bit
31B mlx-community/gemma-4-31B-it-qat-4bit

Note

Coverage and naming vary — these are community conversions, not first-party releases. Search mlx-community for Gemma 4 for the current list. Ollama also ships MLX builds for every size under its -mlx tags, if you would rather not pick a repository.

Server

mlx_vlm.server --model mlx-community/gemma-4-E2B-it-qat-4bit --port 8080

OpenAI-compatible API at http://localhost:8080/v1.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="mlx-community/gemma-4-E2B-it-qat-4bit",
    messages=[{"role": "user", "content": "Say this is a test"}],
)
print(response.choices[0].message.content)