Skip to content

MoE fine-tuning

Gemma 4 26B A4B is a Mixture-of-Experts (MoE) model with 26B total parameters, but it only activates 4B parameters per token.

Compared to dense models, training a Mixture-of-Experts model requires two important things to take into account. First, you must keep the router frozen so it does not send every token to the same few experts. Second, you should target the fused 3D expert tensors so all 128 routed experts receive LoRA updates.


1. Mixture of Experts

In a dense model like Gemma 4 12B or 31B, every token runs through the same feed-forward network. In Gemma 4 26B A4B, each layer splits that step into two parallel paths:

  1. Shared expert: One always-on MLP that processes every token.
  2. Routed experts: A bank of 128 smaller MLPs controlled by a router, which picks 8 experts for each token.

One Gemma 4 26B A4B MoE layer and its LoRA adapters

When you fine-tune 26B A4B with LoRA, you attach LoRA adapters to both the shared expert and the 128 routed experts. The router itself stays frozen.


2. Freezing the router

During pre-training across trillions of tokens, the router learns to spread work evenly across all 128 experts. Fine-tuning on a small dataset can disturb that routing:

  1. Training the router causes collapse: If during training only a few experts are chosen often, then only those experts will continue to receive updates while the rest are not trained.
  2. 4-bit quantization vs. 16-bit precision: Because capacity is split across 128 smaller expert matrices, rounding all of them to 4-bit NF4 introduces minor activation shifts that change the router's choices in later layers.

Trainable router collapse versus a frozen router


3. Speeding up training with grouped GEMM

Gemma 4 26B A4B stores its 128 routed experts as fused 3D parameter tensors (experts.gate_up_proj and experts.down_proj) rather than 128 separate linear layers.

If a trainer loops over all 128 experts one by one, the GPU spends most of its time waiting. Grouped matrix multiplication (grouped GEMM) sorts tokens by expert ID and runs all active experts in a single GPU launch:

Expert loop versus grouped GEMM

Batching all active experts into a single launch keeps the GPU busy instead of needing to wait between each of them.


4. Set up the environment

We fine-tune google/gemma-4-26B-A4B-it on the google/smol benchmark across five low-resource Indigenous and African languages (Quechua, Guarani, Ewe, Akan, and Bambara). Out of the box, 26B A4B translates high-resource languages fluently, but on low-resource languages it often leaves English phrases untranslated or falls into repetition loops. Adapting all 128 routed experts lets the model specialize across all five languages simultaneously on a single GPU.

First, we install the post-training and evaluation libraries:

# Install Hugging Face post-training, quantization, multimodal, and evaluation libraries
%pip install -q -U "transformers>=5.10.1" "trl>=0.16.0" "peft>=0.19.0" accelerate datasets bitsandbytes torchao
%pip install -q -U evaluate sacrebleu jiwer pillow soundfile librosa matplotlib tensorboard

If you plan to push your trained adapter to the Hugging Face Hub, log in with a write token:

from huggingface_hub import login

# Optional: uncomment to push your trained adapter to the Hugging Face Hub
# login()

5. Prepare the dataset

Next up is loading the data, namely google/smol which contains pairs of translations from five low-resource languages. We choose a subset, namely smolsent which contains sentences across various languages and allows us to choose the five low-resource languages. We will keep 4,000 training examples and 315 held-out test examples:

from datasets import concatenate_datasets, load_dataset

LANGUAGES = {
    "smolsent__en_qu": "Quechua",
    "smolsent__en_gn": "Guarani",
    "smolsent__en_ee": "Ewe",
    "smolsent__en_ak": "Akan",
    "smolsent__en_bm": "Bambara",
}


# Format each translation pair into prompt and completion turns
def to_chat(sample, lang):
    return {
        "prompt": [
            {
                "role": "system",
                "content": (
                    f"You are a fluent {lang} translator. Translate the user's "
                    f"English sentence into natural {lang}. Output only the translation."
                ),
            },
            {"role": "user", "content": sample["src"]},
        ],
        "completion": [{"role": "assistant", "content": sample["trg"]}],
    }


# Build non-overlapping training (rows 0..799) and held-out test (rows 800+) splits
train_parts, eval_parts, test_by_lang = [], [], {}
for subset, lang in LANGUAGES.items():
    ds = load_dataset("google/smol", subset, split="train")
    formatted = ds.map(lambda row: to_chat(row, lang), remove_columns=ds.column_names)
    train_parts.append(formatted.select(range(800)))
    eval_parts.append(formatted.select(range(800, len(formatted))))
    test_by_lang[lang] = formatted.select(range(800, 820))

train_dataset = concatenate_datasets(train_parts).shuffle(seed=3407)
eval_dataset = concatenate_datasets(eval_parts)
print(f"Train size: {len(train_dataset)} | Eval size: {len(eval_dataset)}")
Train size: 4000 | Eval size: 315

6. Load the model

To attach the low-rank adapters to all 128 experts, we load the model and set target_parameters to target "experts.gate_up_proj" and "experts.down_proj". These represent the experts themselves and will make sure that the router remains frozen.

Running on a 24–40 GB GPU?

Standard bitsandbytes 4-bit quantization skips fused 3D expert tensors, so this bfloat16 Hugging Face setup needs ~56 GB of VRAM (an 80 GB GPU, or multiple GPUs via FSDP2). To train 26B A4B in ~21 GB on a single 24–40 GB GPU, swap this loading cell with FastVisionModel(..., load_in_4bit=True) from Fine-tuning with Unsloth and keep the rest of the notebook unchanged.

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor
from peft import LoraConfig, get_peft_model

model_id = "google/gemma-4-26B-A4B-it"

# Load the 26B A4B model in bfloat16 and its processor
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

# Attach LoRA to attention, the shared expert, and all 128 routed 3D expert tensors
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.0,
    bias="none",
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],
    target_parameters=[
        "experts.gate_up_proj",
        "experts.down_proj",
    ],
    exclude_modules=r"(model\.(vision_tower|audio_tower)\..*|.*router.*)",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, peft_config)

We can verify that the router (router.proj) is frozen while the attention and expert adapters are trainable:

# Confirm that 0 router parameters are marked trainable
trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
all_params = sum(p.numel() for p in model.parameters())
router_trainable = [n for n, p in model.named_parameters() if p.requires_grad and "router" in n]

print(f"Trainable params: {trainable_params:,} / {all_params:,} ({100 * trainable_params / all_params:.2f}%)")
print(f"Trainable router parameters (must be 0): {len(router_trainable)}")
Trainable params: 494,376,960 / 26,300,310,832 (1.88%)
Trainable router parameters (must be 0): 0

As you can see, there is a small portion of the model's parameters that are trainable whereas none of the router parameters can be trained.


7. Evaluate the base model

Before training, we create a baseline by evaluating across 100 held-out test sentences (20 per language) using evaluate.load("chrf") (character n-gram F-score, the standard metric for morphologically rich low-resource languages). Because LoRA initializes its output matrix B to zero, running inference before trainer.train() evaluates the unmodified base model directly:

import evaluate

# Load the standard chrF translation metric
chrf = evaluate.load("chrf")
stop_ids = [processor.tokenizer.eos_token_id, processor.tokenizer.convert_tokens_to_ids("<turn|>")]


# Evaluate corpus chrF across 20 unseen test sentences per language
def evaluate_translations(tag):
    model.eval()
    orig_padding = processor.tokenizer.padding_side
    processor.tokenizer.padding_side = "left"
    all_preds, all_refs, scores = [], [], {}

    for lang, subset in test_by_lang.items():
        prompts = [
            processor.apply_chat_template(p, tokenize=False, add_generation_prompt=True)
            for p in subset["prompt"]
        ]
        refs = [c[0]["content"] for c in subset["completion"]]
        inputs = processor(text=prompts, return_tensors="pt", padding=True).to(model.device)
        with torch.no_grad():
            outputs = model.generate(
                **inputs, max_new_tokens=128, do_sample=False, eos_token_id=stop_ids, use_cache=True
            )

        prompt_len = inputs["input_ids"].shape[1]
        preds = [
            processor.decode(seq[prompt_len:], skip_special_tokens=True).split("\n")[0].strip()
            for seq in outputs
        ]
        scores[lang] = chrf.compute(predictions=preds, references=refs)["score"]
        all_preds.extend(preds)
        all_refs.extend(refs)
        print(f"  [{tag}] {lang:10s} chrF = {scores[lang]:5.2f}")

    processor.tokenizer.padding_side = orig_padding
    overall = chrf.compute(predictions=all_preds, references=all_refs)["score"]
    print(f"  [{tag}] OVERALL    chrF = {overall:5.2f}")
    return overall, scores


base_overall, base_lang = evaluate_translations("BEFORE")
  [BEFORE] Quechua    chrF = 28.56
  [BEFORE] Guarani    chrF = 24.23
  [BEFORE] Ewe        chrF = 19.09
  [BEFORE] Akan       chrF = 26.01
  [BEFORE] Bambara    chrF = 12.00
  [BEFORE] OVERALL    chrF = 22.02

This is our baseline. Let's see if we can improve it!


8. Train the model

We now switch the model back to training mode and train for 2 epochs (500 optimizer steps at an effective batch size of 16):

from trl import SFTConfig, SFTTrainer

model.train()

# Set the configuration for the trainer
args = SFTConfig(
    output_dir="gemma-4-26b-a4b-smol-lora",
    max_length=512,
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
    optim="adamw_torch_8bit",
    learning_rate=2e-4,
    warmup_steps=0.05,
    lr_scheduler_type="cosine",
    weight_decay=0.01,
    max_grad_norm=1.0,
    bf16=True,
    logging_steps=25,
    eval_strategy="epoch",
    eval_on_start=True,
    save_strategy="epoch",
    report_to="tensorboard",
    seed=3407,
)

# Define and run the trainer
trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    processing_class=processor.tokenizer,
)

trainer.train()
trainer.save_model()
[500/500 29:55, Epoch 2/2]

| Epoch | Training Loss | Validation Loss |
|---|---|---|
| 0 | — | 6.5089 |
| 1 | 2.5725 | 2.2794 |
| 2 | 1.9231 | 2.2304 |
Inspect the training curve

Training and validation loss across epochs


9. Evaluate the fine-tuned model

After training, we can re-run evaluate_translations on the same 100 unseen test sentences to measure how much adapting the shared and 128 routed experts improved translation quality across all five languages:

# Re-evaluate chrF across the 100 unseen test sentences after fine-tuning
after_overall, after_lang = evaluate_translations("AFTER")

print("\nSummary (chrF):")
for lang in LANGUAGES.values():
    b, a = base_lang[lang], after_lang[lang]
    print(f"  {lang:10s}: {b:5.2f} -> {a:5.2f} ({a - b:+5.2f})")
print(f"  {'OVERALL':10s}: {base_overall:5.2f} -> {after_overall:5.2f} ({after_overall - base_overall:+5.2f})")
  [AFTER] Quechua    chrF = 36.95
  [AFTER] Guarani    chrF = 35.55
  [AFTER] Ewe        chrF = 27.98
  [AFTER] Akan       chrF = 31.16
  [AFTER] Bambara    chrF = 15.72
  [AFTER] OVERALL    chrF = 29.60

Summary (chrF):
  Quechua   : 28.56 -> 36.95 (+8.39)
  Guarani   : 24.23 -> 35.55 (+11.32)
  Ewe       : 19.09 -> 27.98 (+8.89)
  Akan      : 26.01 -> 31.16 (+5.15)
  Bambara   : 12.00 -> 15.72 (+3.73)
  OVERALL   : 22.02 -> 29.60 (+7.57)

We get a decent quality boost all around, with a +7.57 chrF overall!

After evaluation, you can merge the adapter into the model through .merge_and_unload:

# Merge the trained MoE LoRA adapter directly into the bfloat16 model and save
merged_model = model.merge_and_unload()
merged_model.save_pretrained("gemma-4-26b-a4b-smol-merged", safe_serialization=True, max_shard_size="4GB")
AutoProcessor.from_pretrained(model_id).save_pretrained("gemma-4-26b-a4b-smol-merged")

What's next