Introduction

1. Overview

LLM Converter is a large language model conversion tool specifically designed for the IPU hardware platform. Its core function is to efficiently and accurately adapt open-source pre-trained models to the specialized computing architecture. This tool provides a complete solution from model conversion, structure optimization to deployment, enabling developers to achieve high-performance operation and stable deployment of models on the IPU platform without modifying the core model logic.

LLM Converter is a complete LLM model conversion and inference framework:

1. 📋 File Structure Description:

   (1) Core Scripts

   - converter_hf_to_sim.py:
   
          Conversion tool for converting Hugging Face models to IPU format
   
   - run_llm.py:
   
           Single inference run script for LLM models
   
   - run_serve.py:
   
            Multi-turn dialogue inference run script for LLM models
   

2. 🔧 Workflow Description:

   (1) Model Conversion:

   Use converter_hf_to_sim.py to convert Hugging Face models to IPU format
   

   (2) Model Inference:

   Use run_llm.py or run_serve.py to run the model.
   

3. 💡 Supported Features Description:

   (1) Model Types:

   Text, Multimodal, Voice, etc.
   

   (2) Support Limitations:

   The current version of the LLM converter tool supports the following two model conversion approaches:
   
   - Native AWQ/GPTQ Models: One-step conversion of official pre-trained weight-quantized models (such as AWQ, GPTQ formats) to offline networks.
   
   - SGS Quantized Models: Use the SGS quantization library with calibration data to quantize floating-point models before conversion. See the SGS Quantized Model Conversion section in [Quick Start](./1_Quick_Start/Quick_Start.md) for details.
   

   ---

2. Purpose

Through the tools and Python interfaces provided by LLM Converter, you can conveniently accomplish the following functions:

(1) Model Conversion

• Supports converting Hugging Face format large language models into inference models executable on the IPU platform, providing two conversion approaches:

  • Native AWQ/GPTQ Models: One-click conversion of officially pre-trained weight-quantized models (such as AWQ, GPTQ formats);

  • SGS Quantized Models: Quantize floating-point models using the SGS quantization library with calibration data for high-precision deployment;

(2) Model Inference

• Support for running converted models on the IPU platform for inference;

• Provide multimodal input support, including text, image, video, audio, and other data types.

---

3. Supported Model Types

This tool extensively supports current mainstream large language model architectures, covering the following major series:

- Llama Series: LLaMA3

- Qwen Series: Qwen2.5/3, Qwen-VL, Qwen-Omni

- Gemma Series: Gemma 1/2/3

- Multimodal Models: InternVL, Florence2, Whisper

- Other Common Open-source Models: Phi3, SmolVLM, Minimind

and more, with continuous expansion of adaptation scope

Compatibility

• Conversion tool development basis: transformers==4.57.3
• Model version requirement: Must be a HuggingFace model based on transformers 4.57.3
• Incompatibility consequences: Conversion process may result in errors or generate unavailable results

For a more detailed list of supported models, please refer to Supported Models.

This toolchain aims to bridge the adaptation gap between open-source large language models and specialized hardware platforms, helping developers conveniently and efficiently deploy and run various large language models on IPU.

---