qwen3

1 概述¶

1.1 背景介绍¶

qwen3是qwen团队开源的大语言模型，起开源项目地址为：

https://github.com/QwenLM/Qwen3

模型下载地址为：

https://huggingface.co/Qwen/Qwen3-1.7B

2 模型转换¶

2.1 offline模型转换¶

模型转换参考SGS_IPU_Toolchain路径下的doc/SDK_Doc_Release/M60/module/LLMConverter/llm_index.html，里面记录详细的大模型转换过程

2.2 offline模型¶

转换后模型拆分为

decoder_model_fixed.sim_sgsimg.img
norm_head_fixed.sim_sgsimg.img
token_embedding_weight.npy
cos_weight.npy
sin_weight.npy

一般大模型的head和embedding的权重是共享的，所以转换模型会是tie_embedding模式，参数转换到了token_embedding_weight.npy。

模型名称	模型介绍
decoder_model_fixed.sim_sgsimg.img	模型decoder部分的权重和graph
norm_head_fixed.sim_sgsimg.img	模型 head 部分的graph
token_embedding_weight.npy	模型embedding 部分的权重
cos_weight.npy	模型位置编码的cos部分的权重
sin_weight.npy	模型位置编码的sin部分的权重

2.3 模型实现¶

sdk/verify/opendla/source/llm/modular/decoder.cpp

实现了decoder_model_fixed.sim_sgsimg.img， token_embedding.npy，cos_weight.npy，sin_weight.npy的加载，子图和kv_cache内存的管理，实现了decoder模型推理的全流程。

decoder_model_fixed.sim_sgsimg.img中含有多张prefill 和decode 过程的子图，prefill的子图每次推理的token数一般设为了128，kv_cache对应最大的token_legnth逐渐增大，比如512/1024/2048等。decode的子图每次推理的数量为1，最大的token_legnth和prefill对应。在调用decoder_model_fixed.sim_sgsimg.img模型时，会根据不同的阶段和输入调用不同的子图，具体参考sdk/verify/opendla/source/llm/qwen3/qwen3.cpp
sdk/verify/opendla/source/llm/modular/norm_head.cpp

实现了norm_head_fixed.sim_sgsimg.img 模型的加载和推理流程。

3 板端部署¶

3.1 程序编译¶

示例程序编译之前需要先根据板子（nand/nor/emmc, ddr型号等）选择deconfig进行sdk整包编译, 具体可以参考alkaid sdk sigdoc《开发环境搭建》文档。

编译板端qwen3示例。

$cd sdk/verify/opendla
$make clean && make source/llm/qwen3 -j8

最终生成的可执行文件地址

sdk/verify/opendla/out/${AARCH}/app/prog_llm_qwen3

备注：此demo同样可用于其他的大模型推理，只要其模型是decoder-only 且转换后的decoder的输入是hidden_in/attention/cos/sin/k_cahce/v_cache,输出是hidden_out/k_cache/v_cache。比如可以推理qwen2/qwen2.5

3.2 运行文件¶

运行程序时, 需要准备以下几个文件

prog_llm_qwen3
qwen3 离线模型

3.3 运行说明¶

Usage: ./prog_llm_qwen3 -d qwen3_model_path -p "who are you"（执行文件使用命令）
- -d 指向转换后的qwen3模型，里面包含转换的模型文件以及对应的json文件
- -p 指的是提示词，demo现在传的是 "who are you"

Typical Output:

./prog_llm_qwen3 -d qwen3_model_path -p "who are you"

Okay, the user asked, "who are you," and I need to respond. Let me start by addressing their question directly. I should mention that I'm an AI assistant developed by Alibaba Group, but I should also highlight that I'm here to help with various tasks.

Wait, the user might not be familiar with the company, so I should explain that I'm a large-scale AI model created by Alibaba Group. But I need to make sure the response is clear and not too technical. Also, I should mention that I can assist with different tasks like answering questions, providing information, or helping with creative projects.

Hmm, should I include the name of the company? Maybe just mention "Alibaba Group" without the exact name of the AI model. Also, I should keep the tone friendly and approachable. Let me structure the response to first answer the question, then provide my capabilities, and end with a polite offer to help. Let me check for any possible errors or unclear parts. I think that's it.
</think>

Hello! I'm an AI assistant developed by Alibaba Group. I'm here to help with various tasks, such as answering questions, providing information, or assisting with creative projects. Feel free to ask me anything! 😊