Retrieval Text-to-Image Search Algorithm Description

REVISION HISTORY

Revision No. Description Date
1.0 First version 07/03/2024
1.1 Add new APIs 06/05/2025
1.2 Add new APIs 06/19/2025
1.3 Modify model descriptions 09/08/2025

1. Overview

1.1. Algorithm Description

Text-to-image search (Retrieval, Ret) can achieve the function of retrieving image content using text and can be used in conjunction with detection + tracking algorithms to implement the ability to search for attributes of detected targets. The models used are described as follows:

Model Function Input size(w*h) Input Format
ret_en_img_256al.img Extract English image feature(Large) 256*256 BGRA8888
ret_en_img_256as.img Extract English image feature(Small) 256*256 BGRA8888
ret_en_text77l.img Extract English text feature(Large) 1*77 U16
ret_en_text77s.img Extract English text feature(Small) 1*77 U16
ret_zh_img_256as.img Extract Chinese image feature(Small) 256*256 BGRA8888
ret_zh_text77s.img Extract Chinese text feature(Small) 1*77 U16

2. API Reference

This functional module provides the following APIs:

API Names Functions
ALGO_RET_CreateHandle Create algorithm handle
ALGO_RET_InitHandle Initialize algorithm handle
ALGO_RET_GetInputAttr Get model attribute information
ALGO_RET_SetParams Set configurable parameters of the algorithm
ALGO_RET_LoadDb Load a previously saved retrieval database file
ALGO_RET_GetCurrentDbSize Get the number of targets in the current database
ALGO_RET_AddBoxToDb Add a retrieval target to the database
ALGO_RET_SearchDb Search the data using the input text
ALGO_RET_SaveDb Export the database as a file
ALGO_RET_ClearDb Clear the database
ALGO_RET_DeInitHandle Deinitialize algorithm handle
ALGO_RET_ReleaseHandle Delete algorithm handle

2.1. ALGO_RET_CreateHandle

  • Function

    Create algorithm handle

  • Syntax

    MI_S32 ALGO_RET_CreateHandle(void **handle);

  • Parameters

    Names Descriptions Input/Output
    handle algo handle Input

  • Return Value

    Values Descriptions
    0 Success
    Others Error Code

  • Dependencies

    • Header File: sgs_ret_api.h

    • Lib File: libsgsalgo_ret.a/libsgsalgo_ret.so

2.2. ALGO_RET_InitHandle

  • Function

    Initialize algorithm handle

  • Syntax

    MI_S32 ALGO_RET_InitHandle(void *handle, const RetInit_t *init);

  • Parameters

    Names Descriptions Input/Output
    handle algo handle Input
    init Configurate parameters such as ipu_firmware_path, model paths, etc. Input

  • Return Value

    Values Descriptions
    0 Success
    Others Error Code

  • Dependencies

    • Header File: sgs_ret_api.h

    • Lib File: libsgsalgo_ret.a/libsgsalgo_ret.so

2.3. ALGO_RET_GetInputAttr

  • Function

    Get model attribute information, including input resolution and input data type

  • Syntax

    MI_S32 ALGO_RET_GetInputAttr(void *handle, RetInputAttr_t *ret_input_attr);

  • Parameters

    Names Descriptions Input/Output
    handle algo handle Input
    ret_input_attr Algo Input Attributes Output

  • Return Value

    Values Descriptions
    0 Success
    Others Error Code

  • Dependencies

    • Header File: sgs_ret_api.h

    • Lib File: libsgsalgo_ret.a/libsgsalgo_ret.so

2.4. ALGO_RET_SetParams

  • Function

    Set configurable parameters of the algorithm

  • Syntax

    MI_S32 ALGO_RET_SetParams(void *handle, const RetParams_t *params);

  • Parameters

    Names Descriptions Input/Output
    handle algo handle Input
    params Parameter structure Input

  • Return Value

    Values Descriptions
    0 Success
    Others Error Code

  • Dependencies

    • Header File: sgs_ret_api.h

    • Lib File: libsgsalgo_ret.a/libsgsalgo_ret.so

2.5. ALGO_RET_LoadDb

  • Function

    Load a previously saved retrieval database file

  • Syntax

    MI_S32 ALGO_RET_LoadDb(void *handle, const char *img_db_path, const char *box_db_path);

  • Parameters

    Names Descriptions Input/Output
    handle algo handle Input
    img_db_path image data database files Input
    box_db_path box data database files Input

  • Return Value

    Values Descriptions
    0 Success
    Others Error Code

  • Dependencies

    • Header File: sgs_ret_api.h

    • Lib File: libsgsalgo_ret.a/libsgsalgo_ret.so

2.6. ALGO_RET_GetCurrentDbSize

  • Function

    Get the number of targets in the current database

  • Syntax

    MI_S32 ALGO_RET_GetCurrentDbSize(void *handle, MI_U32 *db_size);

  • Parameters

    Names Descriptions Input/Output
    handle algo handle Input
    db_size number of targets in the current database Output

  • Return Value

    Values Descriptions
    0 Success
    Others Error Code

  • Dependencies

    • Header File: sgs_ret_api.h

    • Lib File: libsgsalgo_ret.a/libsgsalgo_ret.so

2.7. ALGO_RET_AddBoxToDb

  • Function

    Add a retrieval target to the database

  • Syntax

    MI_S32 ALGO_RET_AddBoxToDb(void *handle, const RetImage_t *image, const RetBox_t *box);

  • Parameters

    Names Descriptions Input/Output
    handle algo handle Input
    image Image buffer of the retrieval target Input
    box Information about the retrieval target box Input

  • Return Value

    Values Descriptions
    0 Success
    Others Error Code

  • Dependencies

    • Header File: sgs_ret_api.h

    • Lib File: libsgsalgo_ret.a/libsgsalgo_ret.so

2.8. ALGO_RET_SearchDb

  • Function

    Search the data using the input text

  • Syntax

    MI_S32 ALGO_RET_SearchDb(void *handle, const char *search_text, RetBox_t results    [MAX_RET_SEARCH_RESULTS], MI_S32 *num_results);

  • Parameters

    Names Descriptions Input/Output
    handle algo handle Input
    search_text Search text Input
    results Information about the target boxes of the search results Output
    num_results Number of search results Output

  • Return Value

    Values Descriptions
    0 Success
    Others Error Code

  • Dependencies

    • Header File: sgs_ret_api.h

    • Lib File: libsgsalgo_ret.a/libsgsalgo_ret.so

2.9. ALGO_RET_SaveDb

  • Function

    Export the database as a file

  • Syntax

    MI_S32 ALGO_RET_SaveDb(void *handle, const char *img_db_path, const char *box_db_path);

  • Parameters

    Names Descriptions Input/Output
    handle algo handle Input
    img_db_path Path to the image database file Input
    box_db_path Path to the target box database file Input

  • Return Value

    Values Descriptions
    0 Success
    Others Error Code

  • Dependencies

    • Header File: sgs_ret_api.h

    • Lib File: libsgsalgo_ret.a/libsgsalgo_ret.so

2.10. ALGO_RET_ClearDb

  • Function

    Clear the database

  • Syntax

    MI_S32 ALGO_RET_ClearDb(void *handle);

  • Parameters

    Names Descriptions Input/Output
    handle algo handle Input

  • Return Value

    Values Descriptions
    0 Success
    Others Error Code

  • Dependencies

    • Header File: sgs_ret_api.h

    • Lib File: libsgsalgo_ret.a/libsgsalgo_ret.so

2.11. ALGO_RET_DeInitHandle

  • Function

    Deinitialize algorithm handle

  • Syntax

    MI_S32 ALGO_RET_DeinitHandle(void *handle);

  • Parameters

    Names Descriptions Input/Output
    handle algo handle Input

  • Return Value

    Values Descriptions
    0 Success
    Others Error Code

  • Dependencies

    • Header File: sgs_ret_api.h

    • Lib File: libsgsalgo_ret.a/libsgsalgo_ret.so

2.12. ALGO_RET_ReleaseHandle

  • Function

    Delete algorithm handle

  • Syntax

    MI_S32 ALGO_RET_ReleaseHandle(void *handle);

  • Parameters

    Names Descriptions Input/Output
    handle algo handle Input

  • Return Value

    Values Descriptions
    0 Success
    Others Error Code

  • Dependencies

    • Header File: sgs_ret_api.h

    • Lib File: libsgsalgo_ret.a/libsgsalgo_ret.so

3. Structure Descriptions

Algorithm-related data types are defined as follows:

Data Type Definition
RetInit_t Initialization information structure of the algorithm
RetParams_t Configurable parameter structure of the algorithm
RetInputAttr_t Input attribute-related structure of the algorithm
RetImage_t Image buffer input structure
RetBox_t Target box information structure

3.1 RetInit_t

  • Description

    Initialization information structure of the algorithm

  • Definition

    typedef struct
{
    char ipu_firmware_path[MAX_RET_STRLEN]; // ipu_firmware.bin path
    char img_model_path[MAX_RET_STRLEN];    // image model path
    char txt_model_path[MAX_RET_STRLEN];    // text model path
    MI_BOOL create_device;                  // set false to create ipu device outside algo lib
    MI_BOOL destroy_device;                 // set false to destroy ipu device outside algo lib
    void *img_model_buffer;                 // set it when load image model from memory
    MI_U32 img_model_buffer_len;            // set it when load image model from memory
    void *txt_model_buffer;                 // set it when load text  model from memory
    MI_U32 txt_model_buffer_len;            // set it when load text  model from memory
    MI_U32 max_db_size;                     // set max size of image/text database default = 30,000
    char vocab_path[MAX_RET_STRLEN];        // vocab path
    RetLanguage_e language;                 // text language type; default = E_RET_EN
    MI_BOOL low_memory_mode;                // keep only one model in memory at the same time to save memory, default = false
} RetInit_t;

  • Members

    Name Descriptions
    ipu_firmware_bin Path to ipu_firmware.bin
    img_model_path Image model path
    txt_model_path Text model path
    create_device Whether to create IPU_Device inside the algorithm library, default is false
    destroy_device Whether to destroy IPU_Device inside the algorithm library, default is false
    img_model_buffer Image model memory buffer (only used when loading the image model from a memory buffer)
    img_model_buffer_len Length of the image model memory buffer (only used when loading the image model from a memory buffer)
    txt_model_buffer Text model memory buffer (only used when loading the text model from a memory buffer)
    txt_model_buffer_len Length of the text model memory buffer (only used when loading the text model from a memory buffer)
    max_db_size Maximum number of targets the database can contain (the earliest targets will be automatically removed when exceeded)
    vocab_path Vocabulary .txt file for the text model
    language Language of the retrieval text
    low_memory_mode low memory mode,set true to enable,default = false

  • Related data types and interfaces

    ALGO_RET_InitHandle

3.2 RetParams_t

  • Description

    Configurable parameter structure of the algorithm

  • Definition

    typedef struct
{
    MI_U32 search_top_k;              // return topk of search, result; range from 0 to MAX_RET_SEARCH_RESULTS, default 10
    MI_BOOL update_existed_track_id;  // set true to update box with track_id already existed in db; default false
    MI_BOOL allow_duplicate_track_id; // set true to allow multi boxes with same track_id co-exist in db; default false
} RetParams_t;

  • Members

    Name Descriptions
    search_top_k Take the most matching k results after the search, default is 10
    update_existed_track_id Whether to update the database with later input of the same track_id target, default is false, i.e., ignore later input of the same track_id target
    allow_duplicate_track_id Allow keeping multiple targets with the same track_id, default is false, i.e., keep one target per track_id in the database

  • Related data types and interfaces

    ALGO_RET_SetParams

3.3 RetInputAttr_t

  • Description

    Algorithm input information structure definition

  • Definition

    typedef struct
{
    MI_U32 image_width;
    MI_U32 image_height;
    MI_U32 image_format;
    MI_U32 max_text_len;
} RetInputAttr_t;

  • Members

    Name Descriptions
    image_width Input width of the image model
    image_height Input height of the image model
    image_format Input format of the image model
    max_text_len Maximum input length of the text model

  • Related data types and interfaces

    ALGO_RET_GetInputAttr

3.4 RetImage_t

  • Description

    Image buffer input structure definition

  • Definition

    typedef struct
{
    void *p_vir_addr;
    MI_PHY phy_addr;
    MI_U32 buf_size;
    MI_U64 pts;
    MI_U16 width;
    MI_U16 height;
} RetImage_t;

  • Members

    Name Descriptions
    p_vir_addr Virtual address of the input buffer
    phy_addr Physical address of the input buffer
    buf_size Length of the input buffer
    pts Timestamp of the input buffer
    width Frame width of the input buffer
    height Frame height of the input buffer

  • Related data types and interfaces

    ALGO_RET_AddBoxToDb

3.5 RetBox_t

  • Description

    Target box information structure definition

  • Definition

    typedef struct
{
    MI_U32 x;
    MI_U32 y;
    MI_U32 width;
    MI_U32 height;
    MI_U32 class_id;
    MI_FLOAT score;
    MI_U64 pts;
    MI_U64 track_id;
    MI_FLOAT search_score;
    char path[MAX_RET_STRLEN];
} RetBox_t;

  • Members

    Name Descriptions
    x X coordinate of the upper left corner of the target box
    y Y coordinate of the upper right corner of the target box
    width Width of the target box
    height Height of the target box
    class_id Detection category of the target box
    score Detection score of the target box
    pts Timestamp of the frame where the target box is located
    track_id Tracking ID of the target box
    search_score Retrieval score of the target box
    path path of input,set it when doing images search,ignore when doing detect results search

  • Related data types and interfaces

    ALGO_RET_AddBoxToDb

    ALGO_RET_SearchDb

4. Enumerated Types Description

RET enumerated types are defined as follows:

Data Type Definitions
RetLanguage_e Retrieval text language enumerated type

4.1 RetLanguage_e

  • Description

    Retrieval text language enumerated type definition

  • Definition

    typedef enum
{
    E_RET_EN = 0,
    E_RET_CN = 1
} RetLanguage_e;

  • Members

    Name Descriptions
    E_RET_EN English
    E_RET_CN Chinese

  • Related data types and interfaces

    RetInit_t

    ALGO_RET_InitHandle

5. Error Code

Error Code Value Description
E_ALGO_SUCCESS 0 Operation successful
E_ALGO_HANDLE_NULL 1 Algorithm handle is null
E_ALGO_INVALID_PARAM 2 Invalid input parameter
E_ALGO_DEVICE_FAULT 3 Hardware error
E_ALGO_LOADMODEL_FAIL 4 Model loading failed
E_ALGO_INIT_FAIL 5 Algorithm initialization failed
E_ALGO_NOT_INIT 6 Algorithm has not been initialized
E_ALGO_INPUT_DATA_NULL 7 Algorithm input data is null
E_ALGO_INVALID_INPUT_SIZE 8 Invalid dimensions of the algorithm input data
E_ALGO_INVALID_LICENSE 9 Invalid license permission
E_ALGO_MEMORY_OUT 10 Insufficient memory
E_ALGO_FILEIO_ERROR 11 File read/write operation error
E_ALGO_INVALID_OUTPUT_SIZE 12 Invalid dimensions of the algorithm output data