Skip to content

qwen3

1 Overview

1.1 Background

Qwen3 is a large language model open-sourced by the Qwen team. The open source project address is:

https://github.com/QwenLM/Qwen3

The model download address is:

https://huggingface.co/Qwen/Qwen3-1.7B

2 Model Conversion

2.1 Offline Model Conversion

For model conversion, please refer to the detailed large model conversion process documented in: doc/SDK_Doc_Release/M60/module/LLMConverter/llm_index.html under the SGS_IPU_Toolchain path.

2.2 Offline Model

The converted model is split into:

  • decoder_model_fixed.sim_sgsimg.img

  • norm_head_fixed.sim_sgsimg.img

  • token_embedding_weight.npy

  • cos_weight.npy

  • sin_weight.npy

Generally, the head and embedding weights of large models are shared, so the converted model will be in tie_embedding mode, with parameters converted to token_embedding_weight.npy.

Model Name Model Description
decoder_model_fixed.sim_sgsimg.img Weights and graph of the model decoder part
norm_head_fixed.sim_sgsimg.img Graph of the model head part
token_embedding_weight.npy Weights of the model embedding part
cos_weight.npy Weights of the cos part of model positional encoding
sin_weight.npy Weights of the sin part of model positional encoding

2.3 Model Implementation

  • sdk/verify/opendla/source/llm/modular/decoder.cpp

    Implements the loading of decoder_model_fixed.sim_sgsimg.img, token_embedding.npy, cos_weight.npy, sin_weight.npy, management of subgraph and kv_cache memory, and the complete decoder model inference process.

    The decoder_model_fixed.sim_sgsimg.img contains multiple subgraphs for prefill and decode processes. The prefill subgraph generally sets the number of tokens inferred each time to 128, and kv_cache corresponds to the maximum token_length gradually increasing, such as 512/1024/2048, etc. The decode subgraph infers 1 token at a time, with maximum token_length corresponding to prefill. When calling the decoder_model_fixed.sim_sgsimg.img model, different subgraphs are called according to different stages and inputs. For details, please refer to sdk/verify/opendla/source/llm/qwen3/qwen3.cpp

  • sdk/verify/opendla/source/llm/modular/norm_head.cpp

    Implements the loading and inference process of the norm_head_fixed.sim_sgsimg.img model.

3 Board Deployment

3.1 Program Compilation

Before compiling the sample program, you need to select the defconfig for SDK full package compilation based on the board specifications (nand/nor/emmc, DDR model, etc.). For details, please refer to the "Development Environment Setup" document in the Alkaid SDK sigdoc.

  • Compile the on-board Qwen3 example.

    $cd sdk/verify/opendla
$make clean && make source/llm/qwen3 -j8

  • The final generated executable file address:

    sdk/verify/opendla/out/${AARCH}/app/prog_llm_qwen3

  • Note: This demo can also be used for other large model inference, as long as the model is decoder-only and the converted decoder input is hidden_in/attention/cos/sin/k_cache/v_cache, and the output is hidden_out/k_cache/v_cache. For example, it can infer qwen2/qwen2.5

3.2 Runtime Files

When running the program, the following files need to be prepared:

  • prog_llm_qwen3
  • qwen3 offline model

3.3 Runtime Instructions

  • Usage: ./prog_llm_qwen3 -d qwen3_model_path -p "who are you" (command to execute the file)

    • -d points to the converted Qwen3 model, which contains the converted model files and corresponding JSON files
    • -p refers to the prompt, the demo currently passes "who are you"

  • Typical Output:

    ./prog_llm_qwen3 -d qwen3_model_path -p "who are you"

Okay, the user asked, "who are you," and I need to respond. Let me start by addressing their question directly. I should mention that I'm an AI assistant developed by Alibaba Group, but I should also highlight that I'm here to help with various tasks.

Wait, the user might not be familiar with the company, so I should explain that I'm a large-scale AI model created by Alibaba Group. But I need to make sure the response is clear and not too technical. Also, I should mention that I can assist with different tasks like answering questions, providing information, or helping with creative projects.

Hmm, should I include the name of the company? Maybe just mention "Alibaba Group" without the exact name of the AI model. Also, I should keep the tone friendly and approachable. Let me structure the response to first answer the question, then provide my capabilities, and end with a polite offer to help. Let me check for any possible errors or unclear parts. I think that's it.

Hello! I'm an AI assistant developed by Alibaba Group. I'm here to help with various tasks, such as answering questions, providing information, or assisting with creative projects. Feel free to ask me anything! 😊