Run Gemma with llama.cpp¶

llama.cpp is the engine underneath most local LLM tooling. Using it directly gives full control over quantization and GPU offload, and it is the most capable local runtime for Gemma 4 — image, audio and video input all work.

Install¶

brew install llama.cpp

winget install llama.cpp

conda install -c conda-forge llama.cpp

nix profile install nixpkgs#llama-cpp

Run¶

llama-cli -hf ggml-org/gemma-4-E2B-it-GGUF --prompt "Write a poem about the Kraken."

-hf downloads the weights on first use, including the multimodal projector. With a system prompt:

llama-cli -hf ggml-org/gemma-4-E2B-it-GGUF -sys "You are Hong Gildong." -p "Who are you?"

Model repositories¶

For 4-bit, prefer Google's QAT weights — trained with quantization simulated rather than compressed afterwards:

llama-cli -hf google/gemma-4-12B-it-qat-q4_0-gguf -p "Explain PLE in one paragraph"

Available as google/gemma-4-{E2B,E4B,12B,26B-A4B,31B}-it-qat-q4_0-gguf.

Multimodal¶

llama-mtmd-cli -hf ggml-org/gemma-4-E2B-it-GGUF

With local files, pass the projector explicitly:

llama-server -m gemma-4-12B-it-Q4_K_M.gguf --mmproj mmproj-gemma-4-12B-it.gguf

Server¶

llama-server -hf ggml-org/gemma-4-E2B-it-GGUF

Browser UI at http://localhost:8080, OpenAI-compatible API at http://localhost:8080/v1.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="gemma-4",  # llama-server serves whichever model it was started with
    messages=[{"role": "user", "content": "Say this is a test"}],
)
print(response.choices[0].message.content)

Multimodal input consumes context quickly; set it with -c, for example -c 32768.