1. Quick Start

1 LLM Model Quick Conversion¶

It is recommended that the conversion instructions for IPU_Toolchain be executed in a Docker environment. For more details, please refer to LLM Environment Setup.

Initialize SGS_IPU_Toolchain

cd SGS_IPU_Toolchain
source cfg_env.sh

To view the specific supported CHIP and version information for the IPU Toolchain, please execute

python3 SGS_IPU_Toolchain/DumpDebug/show_sdk_info.py

Download HuggingFace pre-trained model

https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-AWQ

1.1 Model Conversion & Running¶

1.1.1 Native AWQ Model Conversion¶

Run the following command to convert

python3 SGS_IPU_Toolchain/Scripts/LLM_Converter/convert_hf_to_sim.py \
    -d Qwen2.5-1.5B-Instruct-AWQ \
    --soc_version CHIP

Output Result: Automatically create an output_models subdirectory in the input directory to save the generated IPU-optimized model files and configuration files required for model conversion.

After conversion, the output directory contains:

output_models/
├── model_sim.img           # IPU format model file deployable on board
├── *.ini,*.json,*.npy      # Input configuration
└── tokenize_template.py    # Tokenizer template

For more detailed explanations of output file contents, please refer to the <Model Conversion> section.

1.1.2 SGS Quantized Model Conversion¶

In addition to native AWQ models, IPU Toolchain also supports converting models quantized using the SGS quantization library. The following uses Qwen3-0.6B as an example.

Constructing Quantization Calibration Data

The quantization process requires preparing a calibration dataset in JSON array format, where each element contains a prompt field. It is recommended to provide 10-20 high-quality texts covering diverse topics — the more diverse the data, the higher the quantization precision.

⚠️ Due to quantization algorithm requirements, each prompt should be as long as possible, recommended 100+ characters. Text that is too short will result in insufficient quantization statistics, affecting model accuracy.

Example inputs_text.json:

[
    {"prompt": "In the context of the rapid development of artificial intelligence, large language models like Qwen2.5 have made significant progress in reasoning capabilities, multilingual support, and context understanding. However, they still face limitations in handling ambiguous references, common-sense reasoning, and real-time information updates. How can we enhance model interpretability and factual consistency by combining symbolic logic with neural network architectures?"},
    {"prompt": "Facing the severe reality of intensifying global warming and frequent extreme weather events, although nations have signed the Paris Agreement and committed to carbon neutrality goals, implementation still faces multiple obstacles including economic growth dependence on fossil fuels, high costs of green technologies, and lack of trust in international cooperation. How can we build a global climate governance mechanism that balances fairness, feasibility, and enforceability?"},
    {"prompt": "As gene editing technologies like CRISPR-Cas9 mature, humanity now has the capability to modify embryonic genes to prevent genetic diseases. However, this also opens the Pandora's box of designer babies and genetic enhancement. What ethical review frameworks and legal boundaries should the international community establish to balance scientific exploration freedom with prevention of eugenics abuse?"}
]

Executing Model Conversion

python3 ../Tool/Scripts/LLM_Converter/convert_hf_to_sim.py \
    -d /SSARTIFACTS/SYNC_IPU/SGS_LLM_Models/Qwen/Qwen3-0.6B/ \
    -o /path/to/output_model/ \
    --inputs inputs_text.json \
    --soc_version CHIP

Parameter	Description
`-d`	Input model directory, pointing to the SGS-quantized model path
`-o`	Output directory for saving converted IPU model files
`--inputs`	Quantization calibration data JSON file, format: `[{"prompt": "text"}, ...]`
`--soc_version`	Target chip model

After conversion, the output directory structure is consistent with section 1.1.1, containing model_sim.img and supporting configuration files.

1.1.3 PC Simulation Running Offline Model¶

python3 run_llm.py \
    -d Qwen2.5-1.5B-Instruct-AWQ/output_models \
    --prompt "Give me a short introduction to large language model." \
    --soc_version CHIP

1.1.4 Development Board Running Offline Model¶

The Linux SDK-alkaid has provided the app located at sdk/verify/release_feature/source/dla/ipu_server.

Compile the board-side ipu_server example.

cd sdk/verify/release_feature
make clean && make source/dla/ipu_server -j8

Final generated executable file location

sdk/verify/release_feature/out/${AARCH}/app/prog_dla_ipu_server

Run ipu_server on the board to start the RPC service (PORT is the designated port number)

./prog_dla_ipu_server -p PORT

The prog_dla_ipu_server app has been run on the board to enable the RPC service, and the PC uses run_llm.py to run the AI offline model.

python3 run_llm.py \
    -d Qwen2.5-1.5B-Instruct-AWQ/output_models \
    --prompt "Give me a short introduction to large language model." \
    --soc_version CHIP \
    --host <board_ip_address> \
    --port PORT \
    --model_onboard_dir /path/to/model/on/board/

1.2 Cascaded Model Conversion & Running¶

1.2.1 Model Conversion¶

Cascaded model conversion commands are essentially the same as non-cascaded models, with only the addition of the --num_soc 2 parameter, indicating dual-SoC cascaded inference. Currently only dual-SoC cascading is supported.

Using Qwen2.5-1.5B-Instruct-AWQ as an example:

python3 SGS_IPU_Toolchain/Scripts/LLM_Converter/convert_hf_to_sim.py \
    -d Qwen2.5-1.5B-Instruct-AWQ \
    --num_soc 2 \
    --soc_version CHIP

Parameter	Description
`-d`	Input model directory
`--num_soc`	Number of SoC chips, must be set to 2 for cascaded mode
`--soc_version`	Target chip model

The output result is consistent with non-cascaded models, generating output_models/ in the input directory containing model_sim.img and supporting configuration files.

1.2.2 PC Simulation Running Offline Model¶

The PC simulation running command for cascaded models is the same as non-cascaded models, requiring no additional parameters:

python3 run_llm.py \
    -d Qwen2.5-1.5B-Instruct-AWQ/output_models \
    --prompt "Give me a short introduction to large language model." \
    --soc_version CHIP

1.2.3 Development Board Running Offline Model¶

Cascaded mode uses two SoC chips for collaborative inference, with one acting as the rc end and the other as the ep end. Both boards need to run their respective service programs.

1. Compile Board-Side Programs

Linux SDK-alkaid provides the required source code. Compile ipu_server (rc end) and cascade_service (ep end) separately:

cd sdk/verify/release_feature

# Compile rc end — ipu_server
make clean && make source/dla/ipu_server -j8

# Compile ep end — cascade_service
make clean && make source/ipu/cascade_service -j8

Compilation output paths:

sdk/verify/release_feature/out/${AARCH}/app/prog_dla_ipu_server (rc end)
sdk/verify/release_feature/out/${AARCH}/app/prog_ipu_cascade_service (ep end)

2. Start Board-Side Services

Start cascade_service on the ep end first, then ipu_server on the rc end:

# ep end: Start cascaded service (run directly, no additional parameters needed)
./prog_ipu_cascade_service

# rc end: Start ipu_server and specify RPC port number
./prog_dla_ipu_server -p PORT

Once both board-side services are running, the PC communicates with the cascaded system through the rc end's ipu_server.

3. PC-Side Inference

python3 run_llm.py \
    -d Qwen2.5-1.5B-Instruct-AWQ/output_models \
    --prompt "Give me a short introduction to large language model." \
    --soc_version CHIP \
    --host <rc_end_IP_address> \
    --port PORT \
    --model_onboard_dir /path/to/model/on/board/