Run Gemma with Unsloth¶

Unsloth publishes dynamic-quantized GGUF builds of Gemma 4 and ships a desktop application, Unsloth Studio, that downloads and runs them locally. It is also the fastest route to fine-tuning Gemma on a single consumer GPU.

There are two ways to run Gemma 4 here: the Studio app, or Unsloth's GGUFs directly in llama.cpp.

Hardware requirements¶

Unsloth's own recommended totals — RAM + VRAM combined, or unified memory:

Unsloth recommends dynamic 4-bit (UD-Q4_K_XL) for 12B, 26B A4B and 31B, and 8-bit (Q8_0) for E2B and E4B.

Recommended settings¶

Use Google's default Gemma 4 sampling parameters — the same ones described in Best practices:

temperature = 1.0
top_p = 0.95
top_k = 64

Unsloth Studio¶

1. Install Download the Unsloth app for macOS, Windows or Linux, or install it manually. macOS, Linux, WSL:

   curl -fsSL https://unsloth.ai/install.sh | sh
   

   Windows PowerShell:

   irm https://unsloth.ai/install.ps1 | iex
   

2. Launch

   unsloth studio -H 0.0.0.0 -p 8888
   

   Then open http://127.0.0.1:8888. To serve it over HTTPS through a free Cloudflare tunnel instead, use unsloth studio --secure.
3. Download and run Gemma 4 Search for Gemma 4 in the Chat tab and pick a quant. Inference parameters are set automatically, and both GGUF and MLX files are supported.

llama.cpp¶

Build llama.cpp first — set -DGGML_CUDA=OFF for CPU-only machines and for Apple silicon, where Metal is on by default:

git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first \
    --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp

Then pull and run a model straight from Hugging Face. There is no need to set a context length — llama.cpp uses exactly what is required.

export LLAMA_CACHE="unsloth/gemma-4-E2B-it-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-E2B-it-GGUF:Q8_0 \
    --temp 1.0 --top-p 0.95 --top-k 64

export LLAMA_CACHE="unsloth/gemma-4-E4B-it-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-E4B-it-GGUF:Q8_0 \
    --temp 1.0 --top-p 0.95 --top-k 64

export LLAMA_CACHE="unsloth/gemma-4-12B-it-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL \
    --temp 1.0 --top-p 0.95 --top-k 64

export LLAMA_CACHE="unsloth/gemma-4-26B-A4B-it-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL \
    --temp 1.0 --top-p 0.95 --top-k 64

export LLAMA_CACHE="unsloth/gemma-4-31B-it-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL \
    --temp 1.0 --top-p 0.95 --top-k 64

Vision¶

Download the weights together with the multimodal projector, then pass both:

hf download unsloth/gemma-4-26B-A4B-it-GGUF \
    --local-dir unsloth/gemma-4-26B-A4B-it-GGUF \
    --include "*mmproj-BF16*" \
    --include "*UD-Q4_K_XL*"

./llama.cpp/llama-cli \
    --model unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \
    --mmproj unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf \
    --temp 1.0 --top-p 0.95 --top-k 64

Server¶

./llama.cpp/llama-server \
    --model unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \
    --mmproj unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf \
    --temp 1.0 --top-p 0.95 --top-k 64 \
    --alias "unsloth/gemma-4-26B-A4B-it-GGUF" \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'

Thinking mode¶

Thinking is enabled by putting <|think|> at the start of the system prompt. To turn it off through llama.cpp instead, pass:

--chat-template-kwargs '{"enable_thinking":false}'

--chat-template-kwargs "{\"enable_thinking\":false}"

See Best practices for how thinking output is structured and what to keep in conversation history.

Fine-tuning¶

Unsloth trains Gemma with substantially less memory than a stock Transformers loop, which is what makes single-GPU and free-Colab fine-tuning practical. See Unsloth's Gemma 4 fine-tuning guide, and our own fine-tuning overview for the QLoRA and full fine-tune paths.

Resources¶

• Gemma 4 GGUF collection
• Unsloth documentation