Get started with Gemma and LangChain¶

Models Gemma 4 Gemma Uses LangChain Transformers Google Cloud Keras JAX

This tutorial shows you how to get started with Gemma and LangChain, running in Google Cloud or in your Colab environment. Gemma is a family of light-weight, state-of-the-art open models built from the same research and technology used to create the Gemini models. LangChain is a framework for building and deploying context-aware applications backed by language models.

Note

This tutorial runs on A100 GPU in Google Colab. Free Colab hardware acceleration is insufficient to run all the code.

Run Gemma 4 with LangChain¶

The current path is langchain-huggingface, which works with any Gemma repository on Hugging Face. The examples below follow this page's model selector.

pip install -qU langchain-huggingface transformers accelerate

Local pipelineHosted endpoint

Runs the weights on your own hardware.

from langchain_huggingface import ChatHuggingFace, HuggingFacePipeline

llm = HuggingFacePipeline.from_model_id(
    model_id="google/gemma-4-12B-it",
    task="text-generation",
    pipeline_kwargs=dict(
        max_new_tokens=512,
        do_sample=False,
        repetition_penalty=1.03,
    ),
)

chat_model = ChatHuggingFace(llm=llm)

Calls a hosted provider instead, so nothing runs locally.

from langchain_huggingface import ChatHuggingFace, HuggingFaceEndpoint

llm = HuggingFaceEndpoint(
    repo_id="google/gemma-4-12B-it",
    task="text-generation",
    max_new_tokens=512,
    do_sample=False,
    repetition_penalty=1.03,
    provider="auto",
)

chat_model = ChatHuggingFace(llm=llm)

Then invoke it like any other LangChain chat model:

messages = [
    ("system", "You are a helpful assistant."),
    ("human", "What is the capital of France?"),
]
print(chat_model.invoke(messages).content)

The sections below cover earlier Gemma releases

GemmaLocalKaggle, GemmaChatLocalHF and the Vertex AI one-click deployment were written for Gemma 1 and still reference its gemma_2b_en and google/gemma-2b checkpoints. They are kept because that deployment path still exists, not because it is the recommended way to run Gemma 4.

Run Gemma in Google Cloud¶

The langchain-google-vertexai package provides LangChain integration with Google Cloud models.

Install dependencies¶

pip install --upgrade -q langchain langchain-google-vertexai

Authenticate¶

Unless you're using Colab Enterprise, you need to authenticate.

from google.colab import auth
auth.authenticate_user()

Deploy the model¶

Vertex AI is a platform for training and deploying AI models and applications. Model Garden is a curated collection of models that you can explore in the Google Cloud console.

To deploy Gemma, open the model in Model Garden for Vertex AI and complete the following steps:

Select Deploy.
Make any desired changes to the deployment form fields, or leave them as is, if you're okay with the defaults. Make note of the following fields, which you'll need later:
- Endpoint name (for example, google_gemma-7b-it-mg-one-click-deploy)
- Region (for example, us-west1)
Select Deploy to deploy the model to Vertex AI. The deployment will take a few minutes to complete.

When the endpoint is ready, copy its project ID, endpoint ID, and location, and enter them as parameters.

# @title Basic parameters
project: str = ""  # @param {type:"string"}
endpoint_id: str = ""  # @param {type:"string"}
location: str = "" # @param {type:"string"}

Run the model¶

from langchain_google_vertexai import GemmaVertexAIModelGarden, GemmaChatVertexAIModelGarden

llm = GemmaVertexAIModelGarden(
    endpoint_id=endpoint_id,
    project=project,
    location=location,
)

output = llm.invoke("What is the meaning of life?")
print(output)

Prompt:
What is the meaning of life?
Output:
Life is a complex and multifaceted phenomenon that has fascinated philosophers, scientists, and

You can also use Gemma for multi-turn chat:

from langchain_core.messages import (
    HumanMessage
)

llm = GemmaChatVertexAIModelGarden(
    endpoint_id=endpoint_id,
    project=project,
    location=location,
)

message1 = HumanMessage(content="How much is 2+2?")
answer1 = llm.invoke([message1])
print(answer1)

message2 = HumanMessage(content="How much is 3+3?")
answer2 = llm.invoke([message1, answer1, message2])

print(answer2)

content='Prompt:\n<start_of_turn>user\nHow much is 2+2?<end_of_turn>\n<start_of_turn>model\nOutput:\nSure, the answer is 4.\n\n2 + 2 = 4'
content='Prompt:\n<start_of_turn>user\nHow much is 2+2?<end_of_turn>\n<start_of_turn>model\nPrompt:\n<start_of_turn>user\nHow much is 2+2?<end_of_turn>\n<start_of_turn>model\nOutput:\nSure, the answer is 4.\n\n2 + 2 = 4<end_of_turn>\n<start_of_turn>user\nHow much is 3+3?<end_of_turn>\n<start_of_turn>model\nOutput:\nSure, the answer is 6.\n\n3 + 3 = 6'

You can post-process responses to avoid repetitions:

answer1 = llm.invoke([message1], parse_response=True)
print(answer1)

answer2 = llm.invoke([message1, answer1, message2], parse_response=True)

print(answer2)

content='Output:\nSure, here is the answer:\n\n2 + 2 = 4'
content='Output:\nSure, here is the answer:\n\n3 + 3 = 6<'

Run Gemma from a Kaggle download¶

This section shows you how to download Gemma from Kaggle and then run the model.

To complete this section, you'll first need to complete the setup instructions at Gemma setup.

Then move on to the next section, where you'll set environment variables for your Colab environment.

Note

This section of the tutorial runs on A100 GPU in Google Colab.

Set environment variables¶

Set environment variables for KAGGLE_USERNAME and KAGGLE_KEY.

import os
from google.colab import userdata

# Note: `userdata.get` is a Colab API. If you're not using Colab, set the env
# vars as appropriate for your system.
os.environ["KAGGLE_USERNAME"] = userdata.get('KAGGLE_USERNAME')
os.environ["KAGGLE_KEY"] = userdata.get('KAGGLE_KEY')

Install dependencies¶

# Install Keras 3 last. See https://keras.io/getting_started/ for more details.
pip install -q -U keras-nlp
pip install -q -U keras>=3

Run the model¶

from langchain_google_vertexai import GemmaLocalKaggle

You can specify the Keras backend (by default it's tensorflow, but you can change it to jax or torch).

# @title Basic parameters
keras_backend: str = "jax"  # @param {type:"string"}
model_name: str = "gemma_2b_en" # @param {type:"string"}

llm = GemmaLocalKaggle(model_name=model_name, keras_backend=keras_backend)

output = llm.invoke("What is the meaning of life?", max_tokens=30)
print(output)

What is the meaning of life?

The question is one of the most important questions in the world.

It’s the question that has

Run the chat model¶

As in the Google Cloud example above, you can use a local deployment of Gemma for multi-turn chat. You might need to re-start the notebook and clean your GPU memory in order to avoid OOM errors:

from langchain_google_vertexai import GemmaChatLocalKaggle

# @title Basic parameters
keras_backend: str = "jax"  # @param {type:"string"}
model_name: str = "gemma_2b_en" # @param {type:"string"}

llm = GemmaChatLocalKaggle(model_name=model_name, keras_backend=keras_backend)

from langchain_core.messages import (
    HumanMessage
)

message1 = HumanMessage(content="Hi! Who are you?")
answer1 = llm.invoke([message1], max_tokens=30)
print(answer1)

message2 = HumanMessage(content="What can you help me with?")
answer2 = llm.invoke([message1, answer1, message2], max_tokens=60)

print(answer2)

You can post-process the response if you want to avoid multi-turn statements:

answer1 = llm.invoke([message1], max_tokens=30, parse_response=True)
print(answer1)

answer2 = llm.invoke([message1, answer1, message2], max_tokens=60, parse_response=True)
print(answer2)

Run Gemma from a Hugging Face download¶

Setup¶

Like Kaggle, Hugging Face requires that you accept the Gemma terms and conditions before accessing the model. To get access to Gemma through Hugging Face, go to the Gemma model card.

You'll also need to get a user access token with read permissions, which you can enter below.

Note

This section of the tutorial runs on A100 GPU in Google Colab.

# @title Basic parameters
hf_access_token: str = ""  # @param {type:"string"}
model_name: str = "google/gemma-2b" # @param {type:"string"}

Run the model¶

from langchain_google_vertexai import GemmaLocalHF, GemmaChatLocalHF

llm = GemmaLocalHF(model_name="google/gemma-2b", hf_access_token=hf_access_token)

output = llm.invoke("What is the meaning of life?", max_tokens=50)
print(output)

What is the meaning of life?

The question is one of the most important questions in the world.

It’s the question that has been asked by philosophers, theologians, and scientists for centuries.

And it’s the question that

As in the examples above, you can use a local deployment of Gemma for multi-turn chat. You might need to re-start the notebook and clean your GPU memory in order to avoid OOM errors:

Run the chat model¶

llm = GemmaChatLocalHF(model_name=model_name, hf_access_token=hf_access_token)

from langchain_core.messages import (
    HumanMessage
)

message1 = HumanMessage(content="Hi! Who are you?")
answer1 = llm.invoke([message1], max_tokens=60)
print(answer1)

message2 = HumanMessage(content="What can you help me with?")
answer2 = llm.invoke([message1, answer1, message2], max_tokens=140)

print(answer2)

As in the previous examples, you can post-process the response:

answer1 = llm.invoke([message1], max_tokens=60, parse_response=True)
print(answer1)

answer2 = llm.invoke([message1, answer1, message2], max_tokens=120, parse_response=True)
print(answer2)

What's next¶

Learn how to finetune a Gemma model.
Learn how to use Gemma models with Vertex AI.