# Unused tokens Every Gemma 4 checkpoint ships with **6,227 reserved token slots**, namely `` (token ID `6`) through `` (token ID `262143`). Those unused tokens are already built into its 262,144-token vocabulary. This allows us to keep the vocabulary size and reuse the `` slots in place. Keeping the vocabulary size fixed at `262,144` works on **every** Gemma 4 model without breaking per-layer embeddings or downstream GGUF and LiteRT-LM exports. In this guide, we turn eight unused slots into single-token routing and source tags on `google/gemma-4-E2B-it`. --- ## 1. Reusing slots versus resizing the vocabulary On **Gemma 4 E2B** and **E4B**, each decoder layer looks up a separate per-layer embedding table alongside the main embedding table. If we were to expand the actual vocabulary, we would have a problem since the per-layer embeddings stay at `262,144` tokens, which causes an error: ![Comparing vocabulary resizing against reusing an unused slot across main and per-layer embeddings](https://gemma.ai/assets/tuning/vocab_01_unused_slots_vs_resize.svg) --- ## 2. Initializing and training selected tokens Because `` through `` were, as their name imply, unused, their embeddings have no task meaning yet. Training them from scratch takes longer, while unfreezing the full 262,144-row embedding table adds over 400 million trainable weights. We can avoid both problems in two steps, namely initializing an unused token from subword embeddings and training only that row: 1. **Initialize from existing words:** Before attaching LoRA, split a short description of each new token (such as `"route billing"` for ``{.token-tool}) into its existing subwords and copy their average into that slot's row. 2. **Train only the rows you use:** Pass `trainable_token_indices` in `LoraConfig` so PEFT updates only your chosen rows (for 8 tokens on E2B, fewer than 100K weights instead of 402M) and folds them cleanly back into the model when you merge. ![Initializing an unused token from subword embeddings and training only that row](https://gemma.ai/assets/tuning/vocab_02_trainable_token_indices.svg) It's a very neat trick that prevents us from training entirely from scratch. !!! important "Initialize rows before wrapping the model with PEFT" PEFT copies the starting embedding rows at the moment you call `get_peft_model()`. Always run your subword initialization **before** calling `get_peft_model()` so training starts from your averaged vectors. --- ## 3. Creating single tokens Plain-text tags like `[ROUTE: billing]` or `[DOC2]` split into several tokens every time the model generates them. Mapping those tags to `` through `` turns each one into a single generation step: ![Comparing an 11-token plain-text response against a 4-token unused-token response](https://gemma.ai/assets/tuning/vocab_03_fertility_and_action_compression.svg) Using single tokens for structured tags helps in three ways: - **Faster generation:** Writing ``{.token-tool} and ``{.token-multi} takes one step each instead of nine bracket, colon, and label tokens. - **One-step routing scores:** Looking at the first generated token's probabilities across IDs `6` to `11` gives you a direct confidence score for every route in a single forward pass. - **No broken brackets:** A single token ID cannot be cut off mid-tag like `[ROUTE: billing`. --- ## 4. Set up the environment As always, we install the Hugging Face post-training and evaluation libraries: ```python { .notebook-cell } # Install Hugging Face post-training, quantization, multimodal, and evaluation libraries %pip install -q -U "transformers>=5.10.1" "trl>=0.16.0" "peft>=0.19.0" accelerate datasets bitsandbytes torchao %pip install -q -U evaluate sacrebleu jiwer pillow soundfile librosa matplotlib tensorboard ``` If you plan to push your adapter to the Hugging Face Hub, log in with a write token: ```python { .notebook-cell } from huggingface_hub import login # Optional: uncomment to push your trained adapter to the Hugging Face Hub # login() ``` --- ## 5. Register unused tokens To register the unused tokens, we start by loading the `google/gemma-4-E2B-it` tokenizer. We then check how many `` slots it has and register `` through `` (IDs `6` through `13`): ```python { .notebook-cell .model-swap } from tokenizers import AddedToken from transformers import AutoTokenizer model_id = "google/gemma-4-E2B-it" tokenizer = AutoTokenizer.from_pretrained(model_id) # Count all reserved slots in Gemma 4's 262,144-token vocabulary vocab = tokenizer.get_vocab() unused_slots = sorted( [(tok, idx) for tok, idx in vocab.items() if tok.startswith("")], key=lambda item: item[1], ) print(f"Reserved slots: {len(unused_slots)} ({unused_slots[0]} .. {unused_slots[-1]})") # Show why unregistered '' after a space splits into 5 subwords print("Before registration:", tokenizer.tokenize("Answer ")) # Map 6 task-routing intents + 2 source-citation badges to IDs 6..13 SLOT_PHRASES = { "": "route closed question answering", "": "route information extraction", "": "route text summarization", "": "route classification", "": "route open question answering", "": "route creative brainstorming", "": "cite provided reference context", "": "cite general model knowledge", } # Register with special=False, normalized=False so vocab stays 262,144 and decode() keeps tags tokenizer.add_tokens( [AddedToken(slot, special=False, normalized=False) for slot in SLOT_PHRASES] ) target_token_ids = [tokenizer.convert_tokens_to_ids(slot) for slot in SLOT_PHRASES] print("After registration: ", tokenizer.tokenize("Answer ")) print(f"Target IDs: {target_token_ids} | Vocab size: {len(tokenizer)}") # Compare token count on a sample routed + cited answer plain_example = "[ROUTE: billing] Refund sent [DOC2]" compact_example = "Refund sent" print( f"Token compression: {len(tokenizer.encode(plain_example, add_special_tokens=False))} plain tokens " f"-> {len(tokenizer.encode(compact_example, add_special_tokens=False))} compact tokens" ) ``` ```{ .no-copy .output } Reserved slots: 6227 (('', 6) .. ('', 262143)) Before registration: ['Answer', '▁<', 'unused', '0', '>'] After registration: ['Answer', '▁', ''] Target IDs: [6, 7, 8, 9, 10, 11, 12, 13] | Vocab size: 262144 Token compression: 11 plain tokens -> 4 compact tokens ``` --- ## 6. Prepare the dataset We take six task categories from [`databricks/databricks-dolly-15k`](https://huggingface.co/datasets/databricks/databricks-dolly-15k) and map them to `` through ``. These are specific categories that we want to add to the data so that we can fill the unused tokens with information about these categories. Note that this is merely an example and that you can use any type of data, like different languages or domains. At the end of each answer, we add `` when the answer uses a reference passage or `` when it relies on general knowledge: ```python { .notebook-cell } from datasets import load_dataset CATEGORY_TO_SLOT = { "closed_qa": "", "information_extraction": "", "summarization": "", "classification": "", "open_qa": "", "brainstorming": "", } SYSTEM_INSTRUCTION = ( "Prefix every response with its route token (..) and end with " " when grounded in a Reference block or when using general knowledge." ) # Filter Dolly-15k to our six target categories and keep concise examples raw_ds = load_dataset("databricks/databricks-dolly-15k", split="train").shuffle(seed=42) filtered_ds = raw_ds.filter( lambda r: r["category"] in CATEGORY_TO_SLOT and len(r["context"]) < 1200 and 20 < len(r["response"]) < 600 ) def format_routed_example(row): route_tok = CATEGORY_TO_SLOT[row["category"]] has_ctx = bool(row["context"].strip()) cite_tok = "" if has_ctx else "" user_body = ( f"Reference:\n{row['context'].strip()}\n\nQuestion: {row['instruction'].strip()}" if has_ctx else f"Question: {row['instruction'].strip()}" ) return { "prompt": [ {"role": "system", "content": SYSTEM_INSTRUCTION}, {"role": "user", "content": user_body}, ], "completion": [ {"role": "assistant", "content": f"{route_tok}{row['response'].strip()}{cite_tok}"}, ], "expected_route": route_tok, "expected_cite": cite_tok, } formatted_ds = filtered_ds.select(range(1300)).map( format_routed_example, remove_columns=filtered_ds.column_names ) train_dataset = formatted_ds.select(range(1200)) eval_dataset = formatted_ds.select(range(1200, 1300)) print(f"Prepared {len(train_dataset)} train and {len(eval_dataset)} eval examples") print("Sample completion:", train_dataset[0]["completion"][0]["content"][:140]) ``` ```{ .no-copy .output } Prepared 1200 train and 100 eval examples Sample completion: Garth the Gardener, John the Oak, Gilbert of the Vines, Brandon of the Bloody Blade, Foss the Archer, Owen Oakenshield, Harlon the ``` --- ## 7. Load the model We now load the model, `google/gemma-4-E2B-it`, in 4-bit NF4 precision. We initialize rows `6` through `13` from the average of their subwords. Finally, we configure LoRA to train only those eight token rows alongside the linear adapters: ```python { .notebook-cell } import torch from transformers import AutoModelForMultimodalLM, BitsAndBytesConfig from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training # Configure 4-bit NF4 quantization (embedding tables remain in bfloat16) bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_storage=torch.bfloat16, ) model = AutoModelForMultimodalLM.from_pretrained( model_id, dtype=torch.bfloat16, device_map="auto", quantization_config=bnb_config, ) model = prepare_model_for_kbit_training(model) # Copy the mean of subwords into embed_tokens and (on E2B/E4B) embed_tokens_per_layer @torch.no_grad() def initialize_unused_tokens(model, tokenizer, slot_to_phrase): text_model = model.model.language_model embedding_tables = [text_model.embed_tokens.weight] if hasattr(text_model, "embed_tokens_per_layer"): embedding_tables.append(text_model.embed_tokens_per_layer.weight) for slot_token, phrase in slot_to_phrase.items(): slot_id = tokenizer.convert_tokens_to_ids(slot_token) subword_ids = tokenizer.encode(phrase, add_special_tokens=False) for weight in embedding_tables: weight[slot_id] = weight[subword_ids].mean(dim=0).to(weight.dtype) # Initialize rows 6..13 BEFORE get_peft_model() snapshots original weights initialize_unused_tokens(model, tokenizer, SLOT_PHRASES) text_model = model.model.language_model trainable_tokens = {"embed_tokens": target_token_ids} if hasattr(text_model, "embed_tokens_per_layer"): trainable_tokens["embed_tokens_per_layer"] = target_token_ids peft_config = LoraConfig( r=16, lora_alpha=32, lora_dropout=0.0, bias="none", target_modules=[ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ], exclude_modules=r"model\.(vision_tower|audio_tower)\..*", trainable_token_indices=trainable_tokens, task_type="CAUSAL_LM", ) model = get_peft_model(model, peft_config) # Expose bias=None on PEFT's TrainableTokensWrapper for TRL's fused chunked LM head if not hasattr(model.get_output_embeddings(), "bias"): model.get_output_embeddings().bias = None model.print_trainable_parameters() ``` ```{ .no-copy .output } trainable params: 24,242,176 || all params: 5,128,539,680 || trainable%: 0.4727 ``` ??? tip "Shortening long domain terms with ``" You can use the same `initialize_unused_tokens` helper to shorten recurring domain terms (such as medical codes, product IDs, or financial metrics) that normally split into many subwords: ```python # Map recurring multi-token terms to .. DOMAIN_TERMS = { "EGFR_T790M": "", "HbA1c_mmol": "", "Q3_EBITDA": "", } tokenizer.add_tokens( [AddedToken(slot, special=False, normalized=False) for slot in DOMAIN_TERMS.values()] ) initialize_unused_tokens( model, tokenizer, {slot: term for term, slot in DOMAIN_TERMS.items()} ) ``` Replace those strings in your dataset before tokenizing (and map `` back to `"EGFR_T790M"` after decoding) to shorten sequences while keeping the vocabulary at `262,144`. --- ## 8. Evaluate the base model First, we check how often the base model writes valid `` tags on 40 held-out examples: ```python { .notebook-cell } from torch.optim import AdamW from trl import SFTConfig, SFTTrainer ROUTE_SLOTS = set(CATEGORY_TO_SLOT.values()) # Check route accuracy and citation accuracy on 40 unseen examples def evaluate_primitives(dataset_slice): model.eval() route_hits, cite_hits, valid_structure = 0, 0, 0 for row in dataset_slice: prompt_text = tokenizer.apply_chat_template( row["prompt"], tokenize=False, add_generation_prompt=True, enable_thinking=False ) inputs = tokenizer(prompt_text, return_tensors="pt").to(model.device) with torch.no_grad(): out_ids = model.generate( **inputs, max_new_tokens=96, do_sample=False, use_cache=True ) gen_ids = out_ids[0][inputs["input_ids"].shape[1] :] decoded = tokenizer.decode(gen_ids, skip_special_tokens=True).strip() starts_valid = any(decoded.startswith(s) for s in ROUTE_SLOTS) ends_valid = decoded.endswith("") or decoded.endswith("") valid_structure += int(starts_valid and ends_valid) route_hits += int(decoded.startswith(row["expected_route"])) cite_hits += int(decoded.endswith(row["expected_cite"])) n = len(dataset_slice) return { "valid_tag_pct": 100.0 * valid_structure / n, "route_acc_pct": 100.0 * route_hits / n, "cite_acc_pct": 100.0 * cite_hits / n, } test_slice = eval_dataset.select(range(40)) before_metrics = evaluate_primitives(test_slice) print(f"Before SFT: {before_metrics}") ``` ```{ .no-copy .output } Before SFT: {'valid_tag_pct': 0.0, 'route_acc_pct': 0.0, 'cite_acc_pct': 0.0} ``` Before training, the model outputs `0.0%` valid `` tag pairs, even when asked in the system prompt, because rows `6` through `13` have never been trained in conversation. --- ## 9. Train the model Now, we can train the model for two epochs using a `2e-4` learning rate on LoRA adapters and `1e-4` on the selected token rows: ```python { .notebook-cell } lora_lr = 2e-4 token_lr = 1e-4 token_params, lora_params = [], [] for name, param in model.named_parameters(): if not param.requires_grad: continue if "trainable_tokens" in name: token_params.append(param) else: lora_params.append(param) optimizer = AdamW( [ {"params": lora_params, "lr": lora_lr, "weight_decay": 0.01}, {"params": token_params, "lr": token_lr, "weight_decay": 0.0}, ] ) # Train only on the assistant completion (route tag + answer + source tag) sft_args = SFTConfig( output_dir="gemma-4-unused-tokens", max_length=1024, completion_only_loss=True, num_train_epochs=2, per_device_train_batch_size=4, gradient_accumulation_steps=4, learning_rate=lora_lr, warmup_steps=0.05, lr_scheduler_type="cosine", max_grad_norm=1.0, bf16=True, logging_steps=15, eval_strategy="epoch", eval_on_start=True, save_strategy="epoch", report_to="tensorboard", seed=42, ) trainer = SFTTrainer( model=model, args=sft_args, train_dataset=train_dataset.remove_columns(["expected_route", "expected_cite"]), eval_dataset=eval_dataset.remove_columns(["expected_route", "expected_cite"]), processing_class=tokenizer, optimizers=(optimizer, None), ) model.train() trainer.train() ``` ```{ .no-copy .output } [150/150 08:49, Epoch 2/2] | Epoch | Training Loss | Validation Loss | Mean Token Accuracy | |---|---|---|---| | 0 | — | 4.9008 | 0.5439 | | 1 | 1.4174 | 1.4244 | 0.6762 | | 2 | 1.3033 | 1.4140 | 0.6764 | ``` ??? info "Inspect the training curve" ![Training and validation loss across epochs](https://gemma.ai/assets/tuning/unused-tokens_loss.svg) --- ## 10. Evaluate the fine-tuned model After training, we re-run the evaluation on the held-out examples and save the adapter and updated tokenizer afterwards: ```python { .notebook-cell } after_metrics = evaluate_primitives(test_slice) print(f"Before SFT: {before_metrics}") print(f"After SFT: {after_metrics}\n") # Check route probabilities from a single forward pass sample = test_slice[0] print(f"Expected route: {sample['expected_route']} ({SLOT_PHRASES[sample['expected_route']]})") prompt_text = tokenizer.apply_chat_template( sample["prompt"], tokenize=False, add_generation_prompt=True, enable_thinking=False ) inputs = tokenizer(prompt_text, return_tensors="pt").to(model.device) with torch.no_grad(): next_token_logits = model(**inputs).logits[0, -1, :] route_ids = target_token_ids[:6] route_probs = torch.softmax(next_token_logits[route_ids].float(), dim=-1) for slot, prob in zip(list(SLOT_PHRASES)[:6], route_probs.tolist()): print(f" {slot} ({SLOT_PHRASES[slot]}): {prob:.1%}") # Save both the adapter and the updated tokenizer.json trainer.save_model("gemma-4-unused-tokens") tokenizer.save_pretrained("gemma-4-unused-tokens") ``` ```{ .no-copy .output } Before SFT: {'valid_tag_pct': 0.0, 'route_acc_pct': 0.0, 'cite_acc_pct': 0.0} After SFT: {'valid_tag_pct': 95.0, 'route_acc_pct': 82.5, 'cite_acc_pct': 90.0} Expected route: (route open question answering) (route closed question answering): 0.6% (route information extraction): 0.6% (route text summarization): 0.4% (route classification): 42.9% (route open question answering): 44.2% (route creative brainstorming): 11.2% ``` After two epochs of supervised fine-tuning, the performance is quite a bit higher! By leveraging unused tokens, there is less of a risk of a regression in performance since those will be mostly targeted. ---