Skip to content

gemma3

1 Overview

1.1 Background

Gemma3 is a large language model open-sourced by the Google DeepMind team. The open source project address is:

https://github.com/google-deepmind/gemma

The model download address is:

https://huggingface.co/google/gemma-3-1b-it

2 Model Conversion

2.1 Offline Model Conversion

For model conversion, please refer to the detailed large model conversion process documented in: doc/SDK_Doc_Release/M60/module/LLMConverter/llm_index.html under the SGS_IPU_Toolchain path.

2.2 Offline Model

The converted model is split into:

  • decoder_model_fixed.sim_sgsimg.img

  • norm_head_fixed.sim_sgsimg.img

  • token_embedding_weight.npy

  • local_cos_weight.npy

  • local_sin_weight.npy

  • global_cos_weight.npy

  • global_sin_weight.npy

Generally, the head and embedding weights of large models are shared, so the converted model will be in tie_embedding mode, with parameters converted to token_embedding_weight.npy.

Model Name Model Description
decoder_model_fixed.sim_sgsimg.img Model decoder part
norm_head_fixed.sim_sgsimg.img Model head part
token_embedding_weight.npy Model embedding part
local_cos_weight.npy Weights of the cos part of model local positional encoding
local_sin_weight.npy Weights of the sin part of model local positional encoding
global_cos_weight.npy Weights of the cos part of model positional encoding
global_sin_weight.npy Weights of the sin part of model positional encoding

2.3 Model Implementation

  • sdk/verify/opendla/source/llm/modular/gemma3_decoder.cpp

    Implements the loading of decoder_model_fixed.sim_sgsimg.img, token_embedding.npy, local_cos_weight.npy, local_sin_weight.npy, global_cos_weight.npy, global_sin_weight.npy, management of subgraph and kv_cache memory, and the complete decoder model inference process.

    The decoder_model_fixed.sim_sgsimg.img contains multiple subgraphs for prefill and decode processes. The prefill subgraph generally sets the number of tokens inferred each time to 128, and kv_cache corresponds to the maximum token_length gradually increasing, such as 512/1024/2048, etc. The decode subgraph infers 1 token at a time, with maximum token_length corresponding to prefill. When calling the decoder_model_fixed.sim_sgsimg.img model, different subgraphs are called according to different stages and inputs. For details, please refer to sdk/verify/opendla/source/llm/gemma3/gemma3.cpp

    • Note: Since Gemma3's decoder uses sliding_window, special handling is required when processing local kv cache. The IPU memory implementation uses circular memory management. The prefill process proceeds with inference according to the normal flow. When the prefill ends and the decode process begins, it needs to determine whether the current running position exceeds the sliding window. When it exceeds the sliding window, select the local_kv_cache of sliding_window size, then cyclically reuse this buffer. Pay special attention to the assignment of u32SlidingOffset, which marks the starting position of the circular memory.

  • sdk/verify/opendla/source/llm/modular/norm_head.cpp

    Implements the loading and inference process of the norm_head_fixed.sim_sgsimg.img model.

3 Board Deployment

3.1 Program Compilation

Before compiling the sample program, you need to select the defconfig for SDK full package compilation based on the board specifications (nand/nor/emmc, DDR model, etc.). For details, please refer to the "Development Environment Setup" document in the Alkaid SDK sigdoc.

  • Compile the on-board Gemma3 example.

    $cd sdk/verify/opendla
$make clean && make source/llm/gemma3 -j8

  • The final generated executable file address:

    sdk/verify/opendla/out/${AARCH}/app/prog_llm_gemma3

3.2 Runtime Files

When running the program, the following files need to be prepared:

  • prog_llm_gemma3
  • gemma3 offline model

3.3 Runtime Instructions

  • Usage: ./prog_llm_gemma3 -d gemma3_model_path -p "who are you" (command to execute the file)

    • -d points to the converted Gemma3 model, which contains the converted model files and corresponding JSON files
    • -p refers to the prompt, the demo currently passes "who are you"

  • Typical Output:

    ./prog_llm_gemma3 -d gemma3_model_path -p "who are you"

Assistant:  there! I'm Gemma, a large language model created by the Gemma team at Google DeepMind. I'm an open-weights model, which means I'm openly available for use and research.

I can take text and images as inputs and respond with text.

How can I help you today?