Skip to content

Stransformers Algorithm Library

REVISION HISTORY

Revision No.
Description
 Date
1.0 First version 01/06/2026
1.1 New features include abort and callback, supporting multi-role input 04/23/2026
1.2 Add function tools setting feature 10/06/2026
1.3 Add cache_user_prefix feature, multimodal prefix KV reuse 07/24/2026

1. Overview

1.1. Stransformers Function Introduction

The stransformers runtime library is an inference runtime library adapted for SGS Offline models. Its core functions are as follows: load SGS Offline models converted by the IPU Toolchain, and implement inference acceleration on the edge side by calling IPU hardware. For the inference process of SGS Offline models, users can independently set inference parameters, configure differentiated text generation methods, and obtain model inference results in real time.

1.2. Configuration File Description

The runtime library relies on the following configuration files to implement model initialization and inference control. The functions of each file are as follows:

Configuration File Description
tokenizer.json Stores core parameters such as vocabulary mapping relations, tokenization rules, and character encoding formats
tokenizer_config.json Stores chat templates and tokenizer class types
generation_config.json Stores text generation inference parameter configurations
ipu.json Stores IPU hardware and model path configurations
vocab.json Stores basic vocabulary mapping
merges.txt Stores BPE tokenization merging rules

Among these files, tokenizer.json, tokenizer_config.json, generation_config.json, vocab.json,config.json and mergers.txt are model-related files downloaded from the Hugging Face official website; for details, please refer to the official Hugging Face documentation. The ipu.json file is associated with the SGS IPU hardware platform, and the specific parameters are as follows (the parameters used in practice need to be configured according to the model conversion results, and unused parameters must be deleted):

Parameter Description
ipu_model_class model name,for example "qwen3"
cos Path of the pre-generated cos_weight.npy file
sin Path of the pre-generated sin_weight.npy file
local_cos Path of the pre-generated local_cos_weight.npy file
local_sin Path of the pre-generated local_sin_weight.npy file
head_model Path of the head model
vision_model Path of the vision model
encoder_model Path of the language encoder model
decoder_model Path of the language decoder model
embedding_model Path of the embedding_weight file
embedding_position Path of the embedding_position file

2. API Reference

The runtime library provides the following APIs:

API Name Function
ALGO_STS_Init Model initialization
ALGO_STS_GetInputAttr Model input attribute acquisition
ALGO_STS_CallBack Set the callback function for debug
ALGO_STS_SetParams Multimodal model parameter configuration
ALGO_STS_Generate Model inference
ALGO_STS_LoadPromptCache Load prompt cache file
ALGO_STS_SetFunctionTools Set function tools
ALGO_STS_Abort Exit the model inference halfway
ALGO_STS_ClearKVCache Historical context clearing
ALGO_STS_Deinit Model resource release

2.1. ALGO_STS_Init

  • Function

    Create a handle and initialize the model.

  • Syntax

    MI_S32 ALGO_STS_Init(STSHandle- handle, STSInit_t init);

  • Parameters

    Parameter Name Description Input/Output
    handle Handle pointer Input
    init Large model configuration file path and hardware resource parameters Input

  • Return Values

    Return Value Description
    0 Success
    Others Failure (see Error Code for details)

  • Related Structures

    STSInit_t、IpuConfig_t

  • Dependencies

    • Header Files：sgs_sts_api.h、sgs_algo_datatype.h

    • Library Files：libsgsalgo_stransformers.a / libsgsalgo_stransformers.so

2.2. ALGO_STS_GetInputAttr

  • Function

    Obtain model input attributes (resolution, data type: 1->NV12, 7->ARGB888)）

  • Syntax

    MI_S32 ALGO_STS_GetInputAttr(STSHandle handle, ModelConfig_t- model_attr);

  • Parameters

    Parameter Name Description Input/Output
    handle Handle Input
    model_attr Pointer to store attribute information Input

  • Return Values

    Return Values Description
    0 Success
    Others Failure (see Error Code for details)

  • Related Structures

    ModelConfig_t

  • Dependencies

    • Header Files：sgs_sts_api.h、sgs_algo_datatype.h

    • Library Files：libsgsalgo_stransformers.a / libsgsalgo_stransformers.so

2.3. ALGO_STS_CallBack

  • Function

    set callback for debug

  • Syntax

    MI_S32 ALGO_STS_CallBack(STSHandle handle, STSCallBack callback);

  • Parameters

    Parameter Name Description Input/Output
    handle Handle Input
    STSCallBack Callback function structure Input

  • Return Values

    Return Values Description
    0 Success
    Others Failure (see Error Code for details)

  • Related Structures

    STSCallBack

  • Dependencies

    • Header Files：sgs_sts_api.h、sgs_algo_datatype.h

    • Library Files：libsgsalgo_stransformers.a / libsgsalgo_stransformers.so

2.4. ALGO_STS_SetParams

  • Function

    Set the start and end positions and placeholders of visual tokens for multimodal models.

  • Syntax

    MI_S32 ALGO_STS_SetParams(STSHandle handle, STSParams_t params);

  • Parameters

    Parameter Name Description Input/Output
    handle Handle Input
    params Multimodal model-related parameters Input

  • Return Values

    Return Values Description
    0 Success
    Others Failure (see Error Code for details)

  • Related Structures

    STSParams_t

  • Dependencies

    • Header Files：sgs_sts_api.h

    • Library Files：libsgsalgo_stransformers.a / libsgsalgo_stransformers.so

2.5. ALGO_STS_Generate

  • Function

    Execute model inference and output token results.

  • Syntax

    MI_S32 ALGO_STS_Generate(STSHandle handle, STSInput_t- input, STSOutput_t- output);

  • Parameters

    Parameter Name Description Input/Output
    handle Handle Input
    input Prompt input and callback function Input
    output Model output Output

  • Return Values

    Return Values Description
    0 Success
    Others Failure (see Error Code for details)

  • Related Structures

    STSInput_t、STSOutput_t

  • Dependencies

    • Header Files：sgs_sts_api.h、sgs_algo_datatype.h

    • Library Files：libsgsalgo_stransformers.a / libsgsalgo_stransformers.so

2.6. ALGO_STS_LoadPromptCache

  • Function

    Load the pre-generated prompt cache file.

  • Syntax

    MI_S32 ALGO_STS_LoadPromptCache(STSHandle handle,MI_U8* cache_path);

  • Parameters

    Parameter Name Description Input/Output
    handle Handle Input
    cache_path Prompt cache file Input

  • Return Values

    Return Values Description
    0 Success
    Others Failure (see Error Code for details)

  • Related Structures

  • Dependencies

    • Header Files：sgs_sts_api.h、sgs_algo_datatype.h

    • Library Files：libsgsalgo_stransformers.a / libsgsalgo_stransformers.so

2.7. ALGO_STS_SetFunctionTools

  • Function

    Set function tools to support tool calling capability.

  • Syntax

    MI_S32 ALGO_STS_SetFunctionTools(STSHandle handle, MI_U8* tools);

  • Parameters

    Parameter Name Description Input/Output
    handle Handle Input
    tools Tool string; calling again will overwrite the previous tools; setting tools="" will clear the tools

  • Return Values

    Return Values Description
    0 Success
    Others Failure (see Error Code for details)

  • Related Structures

  • Dependencies

    • Header Files：sgs_sts_api.h、sgs_algo_datatype.h

    • Library Files：libsgsalgo_stransformers.a / libsgsalgo_stransformers.so

2.8. ALGO_STS_Abort

  • Function

    Exit the model inference halfway.

  • Syntax

    MI_S32 ALGO_STS_Abort(STSHandle handle);

  • Parameters

    Parameter Name Description Input/Output
    handle Handle Input

  • Return Values

    Return Values Description
    0 Success
    Others Failure (see Error Code for details)

  • Related Structures

  • Dependencies

    • Header Files：sgs_sts_api.h

    • Library Files：libsgsalgo_stransformers.a / libsgsalgo_stransformers.so

2.9. ALGO_STS_ClearKVCache

  • Function

    Clear historical context and support multi-turn conversation reset.

  • Syntax

    MI_S32 ALGO_STS_ClearKVCache(STSHandle handle, MI_BOOL keep_system_prompt, MI_U32 cache_pos_start, MI_U32 cache_pos_end);

  • Parameters

    Parameter Name Description Input/Output
    handle Handle Input
    keep_system_prompt Whether to retain the system prompt Input
    cache_pos_start Start position of the cache to clear Input
    cache_pos_end End position of the cache to clear Input

  • Return Values

    Return Values Description
    0 Success
    Others Failure (see Error Code for details)

  • Related Structures

  • Dependencies

    • Header Files：sgs_sts_api.h

    • Library Files：libsgsalgo_stransformers.a / libsgsalgo_stransformers.so

2.10. ALGO_STS_Deinit

  • Function

    Destroy the handle and release all model resources.

  • Syntax

    MI_S32 ALGO_STS_Deinit(STSHandle handle);

  • Parameters

    Parameter Name Description Input/Output
    handle Handle Input

  • Return Values

    Return Values Description
    0 Success
    Others Failure (see Error Code for details)

  • Related Structures

  • Dependencies

    • Header Files：sgs_sts_api.h

    • Library Files：libsgsalgo_stransformers.a / libsgsalgo_stransformers.so

3. Structure Description

The relevant data types are defined as follows:

Data Type Definition
IpuConfig_t IPU hardware-related structure
STSInit_t Model initialization structure
STSParams_t Multimodal model input parameter structure
STSInput_t Model inference input parameter structure
STSOutput_t Model inference output structure
STSPromptCacheParams_t prompt cache parameters
ImageTensor_t Image input-related structure
AudioTensor_t Audio input-related structure
VideoTensor_t Video input-related structure
ModelConfig_t Model input attribute structure
GenerationConfig_t Text inference control parameter structure
STSCallBack Callback function structure

3.1 IpuConfig_t

  • Description

    IPU hardware configuration.

  • Definition

    typedef struct IpuConfig
{
    MI_BOOL create_device;
    MI_BOOL destroy_device;
    MI_U32 max_variable_size;
    char ipu_firmware_path[128];
}IpuConfig_t;

  • Members

    Member Name Description
    create_device Whether to create an IPU Device inside the algorithm library; default is true (created inside the library). For multi-library calls, set to false and manually create the IPU Device externally
    destroy_device Whether to destroy the IPU Device inside the algorithm library; default is true (destroyed inside the library). For multi-library calls, set to false and manually destroy the IPU Device externally
    max_variable_size Maximum buffer size required to create the device; fill in 0
    ipu_firmware_path[128] Path of the IPU firmware file

  • Related Data Types and Interfaces

    ALGO_STS_Init

3.2 STSInit_t

  • Description

    Model initialization configuration.

  • Definition

    typedef struct
{
    MI_U8 model_path[MAX_STS_STRLEN];
    IpuConfig_t ipu_config;
}STSInit_t;

  • Members

    Member Name Description
    model_path[MAX_STS_STRLEN] Path of the model configuration file
    ipu_config IPU hardware resource configuration

  • Related Data Types and Interfaces

    ALGO_STS_Init

3.3 STSParams_t

  • Description

    Language model and multimodal model parameter configuration.

  • Definition

    typedef struct
{
    MI_U32 img_start_token;
    MI_U32 img_end_token;
    MI_U8 img_pad_token[MAX_INPUT_NUM];
    MI_U8 video_pad_token[MAX_INPUT_NUM];
    MI_BOOL set_config;
    GenerationConfig_t generation_config;
}STSParams_t;

  • Members

    Member Name Description
    img_start_token Start position of image tokens
    img_end_token End position of image tokens
    img_pad_token[MAX_INPUT_NUM] Placeholder for image tokens
    video_pad_token[MAX_INPUT_NUM] Placeholder for video tokens
    set_config Whether to set the generation_config parameter
    generation_config generation config

  • Related Data Types and Interfaces

    ALGO_STS_SetParams

3.4 STSInput_t

  • Description

    Inference input parameter configuration.

  • Definition

    typedef struct
{
    ImageTensor_t image;
    AudioTensor_t audio;
    VideoTensor_t video;
     MI_U8 role[MAX_ROLE_NUM][MAX_INPUT_NUM];
    MI_U8 prompt[MAX_ROLE_NUM][MAX_INPUT_NUM];
    MI_U8 role_prompt_num;
    MI_BOOL stream;
    MI_BOOL enable_thinking;
    MI_BOOL cache_user_prefix;
    STSPromptCacheParams_t system_prompt_cache;
    void (*streamer)(const char* token,bool is_end);
}STSInput_t;

  • Members

    Member Name Description
    image Image input buffer
    audio Audio input buffer
    video Video input buffer
    role[MAX_ROLE_NUM][MAX_INPUT_NUM] Role identifier for custom chat templates
    prompt[MAX_ROLE_NUM][MAX_INPUT_NUM] User prompt text or system prompt
    role_prompt_num Number of roles
    stream Whether to enable streaming inference; set to true to output each token immediately after decoding, set to false to output the entire sentence after decoding
    enable_thinking Whether to enable the thinking mode takes effect only when the model that supports thinking is enabled
    cache_user_prefix Whether to cache the fixed prefix before the first vision token. Only valid for multimodal input; when enabled, multi-round multimodal conversations can reuse the prefix KV
    system_prompt_cache Prompt cache parameter
    (streamer)(const char token, bool is_end) Streaming output callback function

    Note: When a system prompt is set, the first call to generate caches the system prompt. On subsequent calls to generate, the algorithm library automatically checks whether the system prompt has changed. If it has not changed, the cached information is used and the system prompt is not inferred repeatedly; if it has changed, inference starts from scratch and the new system prompt is cached. If no new system prompt is set, the previous system prompt is used by default.

  • Related Data Types and Interfaces

    ALGO_STS_Generate

3.5 STSOutput_t

  • Description

    Model inference result acquisition.

  • Definition

    typedef struct
{
    MI_U8 output_string[MAX_OUTPUT_NUM];
    MI_U32 output_tokens[MAX_OUTPUT_NUM];
    ImageTensor_t embedding;
}STSOutput_t;

  • Members

    Member Name Description
    output_string[MAX_OUTPUT_NUM] For non-streaming inference, stores all responses generated by the model
    output_tokens[MAX_OUTPUT_NUM] For non-streaming inference, stores the token IDs corresponding to the model-generated text
    embedding output embedding buffer

  • Related Data Types and Interfaces

    ALGO_STS_Generate

3.6 STSPromptCacheParams_t

  • Description

    prompt cache parameters.

  • Definition

    typedef struct
{
    MI_BOOL save_cache;
    MI_U8 save_cache_path[MAX_STS_STRLEN];
}STSPromptCacheParams_t;

  • Members

    Member Name Description
    save_cache Whether to save cache
    save_cache_path Path to save the cache file; when saving system prompt, if tools are set, they will also be saved in the cache

  • Related Data Types and Interfaces

    ALGO_STS_Generate

3.7 ImageTensor_t

  • Description

    Image input data configuration.

  • Definition

    typedef struct ImageTensor
{
    void* p_vir_addr;
    MI_U64 phy_addr;
    MI_U32 buf_size;
    MI_U64 pts;
    uint16_t width;
    uint16_t height;
}ImageTensor_t;

  • Members

    Member Name Description
    p_vir_addr Virtual address of the input buffer
    phy_addr Physical address of the input buffer
    buf_size Length of the input buffer
    pts Timestamp of the input buffer
    width Image width
    height Image height

  • Related Data Types and Interfaces

    ALGO_STS_Generate

3.8 AudioTensor_t

  • Description

    audio input data configuration.

  • Definition

    typedef struct AudioTensor
{
    void* p_vir_addr;
    MI_U64 phy_addr;
    MI_U32 buf_size;
} AudioTensor_t;

  • Members

    Member Name Description
    p_vir_addr Virtual address of the input buffer
    phy_addr Physical address of the input buffer
    buf_size Length of the input buffer

  • Related Data Types and Interfaces

    ALGO_STS_Generate

3.9 VideoTensor_t

  • Description

    video input data configuration.

  • Definition

    typedef struct VideoTensor { void* p_vir_addr; MI_U64 phy_addr; MI_U32 buf_size; uint16_t width; uint16_t height; uint16_t frame_num; uint16_t internal_frames; uint16_t fps; }VideoTensor_t;

  • Members

    Member Name Description
    p_vir_addr Virtual address of the input buffer
    phy_addr Physical address of the input buffer
    buf_size Length of the input buffer
    width Width
    height Height
    frame_num Number of frames
    internal_frames Frame interval
    fps Frame rate

  • Related Data Types and Interfaces

    ALGO_STS_Generate

3.10 ModelConfig_t

  • Description

    Model input attribute acquisition.

  • Definition

    typedef struct ModelConfig
{
    MI_IPU_ELEMENT_FORMAT format;
    MI_U32 width;
    MI_U32 height;
}ModelConfig_t;

  • Members

    Member Name Description
    width Width of the model input data
    height Height of the model input data
    format Data type of the model input data

  • Related Data Types and Interfaces

    ALGO_STS_GetInputAttr

3.11 GenerationConfig_t

  • Description

    Text inference control parameter configuration.

  • Definition

    typedef struct GenerationConfig
{
    bool do_sample;
    int top_k;
    float top_p;
    float temperature;
    float repetition_penalty;
    bool stream;
    int eos_token_id[MAX_EOS_NUM];
    int eos_num;
    bool use_cache;
    int max_length;
    int max_new_tokens;
}GenerationConfig_t;

  • Members

    Member Name Description
    do_sample Sampling strategy: true = random sampling, false = deterministic strategy
    top_k Sample from the top k tokens with the highest probabilities, valid only when do_sample=true
    top_p Sample from the smallest set of tokens whose cumulative probability reaches p, valid only when do_sample=true
    temperature Temperature coefficient; values > 1 increase randomness, values < 1 decrease randomness, valid only when do_sample=true
    repetition_penalty Repetition penalty coefficient, valid only when the value > 1 and do_sample=true
    stream Whether to enable streaming inference
    eos_token_id[MAX_EOS_NUM] End-of-sequence token identifier
    eos_num Number of eos
    use_cache Whether to enable KV cache for multi-turn conversations
    max_length Maximum length of the KV cache
    max_new_tokens Maximum number of tokens generated per inference

  • Related Data Types and Interfaces

    ALGO_STS_SetParams

3.12 STSCallBack

  • Description

    Callback function structure.

  • Definition

    typedef void (*TokenizerCallBack)(char* text, int32_t* tokens, uint64_t num_tokens);
typedef void (*ModelCallBack)(void* tensor, int tensor_size,uint32_t* tensor_shape,int tensor_dims, MI_IPU_ELEMENT_FORMAT format,char* model_name, char* layer_name);

typedef struct
{
    TokenizerCallBack tokenizer_callback;
    ModelCallBack     model_callback;
}STSCallBack;

  • Members

    Member Name Description
    tokenizer_callback Tokenizer callback function
    model_callback Model input/output callback function

  • Related Data Types and Interfaces

    ALGO_STS_CallBack

4. Advanced Function and Configuration Description

4.1. KV Cache Management

  • Function Description

    During multi-turn conversations, it supports manually clearing the specified range of KV cache [cache_pos_start, cache_pos_end] to reset the KV cache.

  • Key Features

    When clearing the cache, you can select whether to retain the system prompt cache through the keep_system_prompt parameter to avoid repeated inference:

    • keep_system_prompt=1，Retain the KV cache of the system prompt；
    • keep_system_prompt=0，Clear all caches.

  • Related Interfaces/Structures

    ALGO_STS_ClearKVCache

4.2. Historical Context Management

  • Function Description

    During multi-turn conversations, when the total length of historically stored tokens (system tokens + prompt tokens + generate tokens) exceeds (max_length - max_new_tokens), the runtime library will automatically delete the earliest historical content.

  • Notes

    Automatic historical content cleaning is time-consuming. It is recommended to manually manage the cache with ALGO_STS_ClearKVCache.

  • Related Interfaces/Structures

    ALGO_STS_ClearKVCache、ALGO_STS_Generate

    GenerationConfig_t

4.3. Chat Template Management

  • Function Description

    In text input scenarios, the large model runtime library will automatically parse the chat_template field in tokenizer_config.json to obtain the prompt template. If users need to set their own prompt templates, they can reset them through the role configuration in the STSInput_t structure.

    In multimodal input scenarios, the prompt needs to be configured according to specifications to automatically obtain the prompt template from tokenizer_config.json. An example is as follows:

    STSInput_t.prompt = R"(
    [
        {"type":"text", "text":"Please analyze this image："},
        {"type":"image", "url":"https://example.com/img.png"}
    ]
)";

  • Related Interfaces/Structures

    ALGO_STS_Generate

    STSInput_t

4.4. Streaming Inference Callback Setting

  • Function Description

    Streaming inference supports real-time output of inference results token by token. To enable it, set STSInput_t.stream or GenerationConfig_t.stream to true, and implement the STSInput_t.streamer callback function. This function will be triggered after each token is generated, and you can obtain the currently generated token and the identifier indicating whether it is the last token through callback parameters, thereby realizing real-time reception and processing of inference results.

  • Related Interfaces/Structures

    ALGO_STS_Generate

    STSInput_t

4.5. PCIe Multi-Board Cascading Setting

To be supplemented

4.6. LoRA Model Management

To be supplemented

5. Error Code

Error Code Value Description
E_ALGO_SUCCESS 0 Operation successful
E_ALGO_HANDLE_NULL 1 Algorithm handle is null
E_ALGO_INVALID_PARAM 2 Invalid input parameter
E_ALGO_DEVICE_FAULT 3 Hardware error
E_ALGO_LOADMODEL_FAIL 4 Model loading failed
E_ALGO_INIT_FAIL 5 Algorithm initialization failed
E_ALGO_NOT_INIT 6 Algorithm has not been initialized
E_ALGO_INPUT_DATA_NULL 7 Algorithm input data is null
E_ALGO_INVALID_INPUT_SIZE 8 Invalid dimensions of the algorithm input data
E_ALGO_INVALID_LICENSE 9 Invalid license permission
E_ALGO_MEMORY_OUT 10 Insufficient memory
E_ALGO_FILEIO_ERROR 11 File read/write operation error
E_ALGO_INVALID_OUTPUT_SIZE 12 Invalid dimensions of the algorithm output data