Vlm Pipeline Multimodal Algorithm Description

---

REVISION HISTORY

Revision No.	Description	Date
1.0	First version	09/16/2025

---

1. Overview

1.1. Algorithm Description

The VLM pipeline multimodal algorithm can achieve various functionalities including object detection in images, multi-object tracking, person-in-picture tracking, text-based image search, image generation from text, and video search by text.

As this pipeline integrates multiple functions that can complement each other for better results, this document provides explanations of different functionalities for users to better utilize and integrate the algorithm.

• The first usage method: Detection -> Text-based Image Search in the visual part, detailed code can be referenced in test.cpp under runRetVisionDemo.

  The algorithm activates detection and the visual part of the text-based image search.

  The interface calling sequence is: ALGO_VLM_Create -> ALGO_VLM_Init -> ALGO_VLM_Run -> ALGO_VLM_Deinit -> ALGO_VLM_Release.

  Initialization parameters and runtime parameters can be found in runRetVisionDemo in the cpp file.

• The second method: Text-based Image Search, specific code can be referenced in test.cpp under runRetTextDemo.

  The algorithm activates only the text part of the text-based image search (search), as the text model is larger; during the search process, the registration process of the first method can be released.

  The interface calling sequence is: ALGO_VLM_Create -> ALGO_VLM_Init -> ALGO_VLM_Run -> ALGO_VLM_GetCosSimi -> ALGO_VLM_Deinit -> ALGO_VLM_Release.

  Initialization and runtime parameters can be found in runRetTextDemo in test.cpp. The main difference is the initialization parameters of the handle and the settings during the runtime.

• The third operation method: Detection, multi-object tracking, specific code can be referenced in test.cpp, runDetMotDemo.

  The algorithm will perform detection and multi-object tracking, inputting detection boxes and tracking IDs.

  The interface calling sequence is: ALGO_VLM_Create -> ALGO_VLM_Init -> ALGO_VLM_Run -> ALGO_VLM_Deinit -> ALGO_VLM_Release.

  Specific parameters can be found in the code in test.cpp.

  Configuring the environment variable export SGS_DET_SAVE_DIR=dump can save the input image data for the detection algorithm (the data type matches the model's input data type), where dump is the folder for saving inputs, which needs to be created by the user.

• The fourth operation method: Long-distance detection/tracking, specific code can be referenced in test.cpp, runLongDisDetDemo.

  The algorithm enables three algorithms: motion detection (md), object detection (det), and single-object tracking (sot).

  The specific process is to first locate the motion area in the image, then perform object detection on the motion area, and select one target from the detection results to start tracking.

  The interface calling sequence is: ALGO_VLM_Create -> ALGO_VLM_Init -> ALGO_VLM_Run -> ALGO_VLM_SetSot -> ALGO_VLM_Deinit -> ALGO_VLM_Release.

• The fifth operation method: Video search by text, specific code can be referenced in test.cpp, runVRetDemo.

  If it is a video search by text, the process is similar to the text-based image search, first testing the embedding of the offline video, and then using the embedding of the text for searching.

  runVRetDemo demonstrates a classification usage of the video search by text. Specific usage can be referenced in the code.

  The interface calling sequence is: ALGO_VLM_Create -> ALGO_VLM_Init -> ALGO_VLM_Run -> ALGO_VLM_GetCosSimi -> ALGO_VLM_Deinit -> ALGO_VLM_Release.

• The sixth operation method: Person/vehicle re-identification (reid) algorithm, specific code can be referenced in test.cpp, runReIdDemo.

  First, detect person/vehicle targets in the scene, then perform quality evaluation of targets (optional). If the target quality meets the requirements, extract the features of the corresponding target area and match them against the feature library to determine whether it is the same target.

  The interface calling sequence is: ALGO_VLM_Create -> ALGO_VLM_Init -> ALGO_VLM_Run -> ALGO_VLM_Deinit -> ALGO_VLM_Release.

• The seventh method: Image generation from text, specific code can be referenced in test.cpp, runSgptvDemo.

  The interface calling sequence is: ALGO_VLM_Create -> ALGO_VLM_Init -> ALGO_VLM_Run -> ALGO_VLM_Deinit -> ALGO_VLM_Release.

  Note: The prompt for image generation from text in the Chinese model must be "简要描述图片内容"; in the English model, it should be "Briefly describe the content of the picture." The detection area is "What does the region describe?" where "" should be replaced with actual area coordinates. The file encoding format must be UTF-8.

1.2. Model and Algorithm Performance

Algorithm List	Model	Execution Time (ms)	ROM (MB)	RAM (MB)	Bandwidth (MB)
Image Generation	Vision	17.578	31.56	32.18	37
	LLM	71.78+5.1/token	52.1	54.32	44/token+27
Text-based Search	Vision	17.578	31.56	32.18	37
	Text	6.1	98.9	98.98	49
Universal Detection	Ovdet	23.2	8.8	15.4	171
Video Search	VRet_Text	6.5	102.9	102.9	53.6
	VRet_Vision	275	34.7	43	539

Pipeline	Model List	Execution Time (ms)	ROM (MB)	RAM (MB)	Bandwidth (MB)
(Visual Understanding Process) Universal Detection + Image Features + Image Generation	Ovdet+Vision+LLM	112.558+5.1/token	92.46	101.9	235+44/token
Text-based Search Process	Text	6.1	98.9	98.98	49

1.3. Detection Model Category Description

Model	Function	Input Dimensions (w*h)	Input Format
sovd_sd_sfsd_sbdy48.img	Outputs a total of 13 categories: Pedestrian (class_id=0), Bicycle (class_id=1), Car (class_id=2), Motorcycle (class_id=3), Bus (class_id=4), Truck (class_id=5), Cat (class_id=6), Dog (class_id=7), Human Head (class_id=8), Human Face (class_id=9), Fire (class_id=10), Smoke (class_id=11), Package (class_id=12)	800*480	yuvsp420_nv12
sovd_sad_sd_sfsd_sbdy48.img	Outputs a total of 15 categories: Bear (class_id=0), Crocodile (class_id=1), Pedestrian (class_id=2), Bicycle (class_id=3), Car (class_id=4), Motorcycle (class_id=5), Bus (class_id=6), Truck (class_id=7), Cat (class_id=8), Dog (class_id=9), Human Head (class_id=10), Human Face (class_id=11), Fire (class_id=12), Smoke (class_id=13), Package (class_id=14)	800*480	yuvsp420_nv12

2. API Reference

This module provides the following APIs:

API Name	Function
ALGO_VLM_Create	Create handle
ALGO_VLM_Init	Initialize handle
ALGO_VLM_Run	Run algorithm
ALGO_VLM_SetSot	Run image description and understanding
ALGO_VLM_GetCosSimi	Calculate cosine similarity
ALGO_VLM_Deinit	De-initialize handle
ALGO_VLM_Release	Delete handle

2.1. ALGO_VLM_Create

• Function

  Create handle.

• Syntax

  MI_S32 ALGO_VLM_Create(VLMPipeline_t* pipeline);
  

• Parameters

  Parameter Name	Description	Input/Output
  pipeline	Handle	Input

• Return Value

  Return Value	Description
  0	Success
  Other	Failure (see Error Codes)

• Dependencies

  • Header file: sgs_vlm_api.h
  • Library file: libsgsalgo_vlm.a/libsgsalgo_vlm.so

2.2. ALGO_VLM_Init

• Function

  Initialize handle.

• Syntax

  MI_S32 ALGO_VLM_Init(VLMPipeline_t pipeline, const VLMInit_t *init);
  

• Parameters

  Parameter Name	Description	Input/Output
  pipeline	Handle	Input
  init	Configure tokenizer folder path, model file path, etc.	Input

• Return Value

  Return Value	Description
  0	Success
  Other	Failure (see Error Codes)

• Dependencies

  • Struct: VLMInit_t
  • Header file: sgs_vlm_api.h
  • Library file: libsgsalgo_vlm.a/libsgsalgo_vlm.so

2.3. ALGO_VLM_Run

• Function

  Run the algorithm; depending on different parameters, different functionalities are executed.

• Syntax

  MI_S32 ALGO_VLM_Run(VLMPipeline_t pipeline, const VLMParams_t *params, const VLMInput_t *input, VLMOutput_t *output);
  

• Parameters

  Parameter Name	Description	Input/Output
  pipeline	Handle	Input
  params	Configuration parameters; different parameters enable different features	Input
  input	Image data or prompt	Input
  output	Results of various functionalities	Output

• Return Value

  Return Value	Description
  0	Success
  Other	Failure (see Error Codes)

• Dependencies

  • Structs: VLMParams_t, VLMInput_t
  • Header file: sgs_vlm_api.h
  • Library file: libsgsalgo_vlm.a/libsgsalgo_vlm.so

2.4. ALGO_VLM_SetSot

• Function

  Set target for single object tracking.

• Syntax

  MI_S32 ALGO_VLM_SetSot(VLMPipeline_t pipeline, const VLMInput_t *input, VLMOutput_t *output);
  

• Parameters

  Parameter Name	Description	Input/Output
  pipeline	Handle	Input
  input	Image and target coordinate information	Input
  output	Result of target tracking	Output

• Return Value

  Return Value	Description
  0	Success
  Other	Failure (see Error Codes)

• Dependencies

  • Header file: sgs_vlm_api.h
  • Library file: libsgsalgo_vlm.a/libsgsalgo_vlm.so

2.5. ALGO_VLM_GetCosSimi

• Function

  Calculate cosine similarity in text-based image search.

• Syntax

  MI_S32 ALGO_VLM_GetCosSimi(VLMPipeline_t pipeline, const MI_FLOAT* query, MI_FLOAT* source, MI_U32 batch, MI_FLOAT* similarity);
  

• Parameters

  Parameter Name	Description	Input/Output
  pipeline	Handle	Input
  query	Feature of text	Input
  source	Registered image features	Input
  batch	Number of registered photo features	Input
  similarity	Cosine similarity	Output

• Return Value

  Return Value	Description
  0	Success
  Other	Failure (see Error Codes)

• Dependencies

  • Header file: sgs_vlm_api.h
  • Library file: libsgsalgo_vlm.a/libsgsalgo_vlm.so

2.6. ALGO_VLM_Deinit

• Function

  De-initialize the handle.

• Syntax

  MI_S32 ALGO_VLM_Deinit(VLMPipeline_t pipeline);
  

• Parameters

  Parameter Name	Description	Input/Output
  pipeline	Handle	Input

• Return Value

  Return Value	Description
  0	Success
  Other	Failure (see Error Codes)

• Dependencies

  • Header file: sgs_vlm_api.h
  • Library file: libsgsalgo_vlm.a/libsgsalgo_vlm.so

2.7. ALGO_VLM_Release

• Function

  Delete the handle.

• Syntax

  MI_S32 ALGO_VLM_Release(VLMPipeline_t pipeline);
  

• Parameters

  Parameter Name	Description	Input/Output
  pipeline	Handle	Input

• Return Value

  Return Value	Description
  0	Success

• Dependencies

  • Header file: sgs_vlm_api.h
  • Library file: libsgsalgo_vlm.a/libsgsalgo_vlm.so

3. Structure Definitions

Related data types for VLM are defined as follows:

Data Type	Definition
VLMInit_t	Algorithm Initialization Structure
VLMImage_t	Image Data Structure
VLMInput_t	Algorithm Input Parameter Structure
VLMBox_t	Detection Tracking Box Structure
VLMParams_t	Algorithm Input Structure
VLMRetSgptvResult	Visual Output Structure for Text-based Image Search and Image Generation
VLMOutput_t	Algorithm Output Structure
VLMCropFunc	Crop Function
VLMVRetResult	Video Search Output Structure
VLMFrResult_t	Face Recognition Algorithm Output Structure
VLMReIdResult_t	Person/Vehicle Re-identification Algorithm Output Structure

3.1 VLMInit_t

• Description

  Algorithm initialization structure.

• Definition

  typedef struct
  {
      // common
      char ipu_firmware_path[MAX_VLM_STRLEN]; // Path for ipu_firmware.bin
      MI_BOOL create_device;                  // Set false to create ipu device outside algo lib
      MI_BOOL destroy_device;                 // Set false to destroy ipu device outside algo lib
      VLMCropFunc crop_func;
      // md
      MI_BOOL init_md;
      MI_S32 ive_handle;
      MI_U32 md_in_width;
      MI_U32 md_in_height;
      MI_U32 md_in_stride;
      //det and track
      MI_BOOL init_det;
      MI_BOOL init_mot;
      char det_model[MAX_VLM_STRLEN];
      char det_cls[MAX_VLM_STRLEN];
  
      // sot
      MI_BOOL init_sot;
      char det4sot_model[MAX_VLM_STRLEN];
      char sot_model[MAX_VLM_STRLEN];
      //fr
      MI_BOOL init_fr;
      char fr_det_model[MAX_VLM_STRLEN];
      char fr_quality_model[MAX_VLM_STRLEN];
      char fr_feat_model[MAX_VLM_STRLEN];
      char fr_emo_model[MAX_VLM_STRLEN];
      //ret_sgptv
      MI_BOOL init_ret;
      char vision_model[MAX_VLM_STRLEN];
      char json_path4ret[MAX_VLM_STRLEN];
      char ret_text_model[MAX_VLM_STRLEN];
      MI_BOOL init_sgptv;
      char sgptv_vision_model[MAX_VLM_STRLEN];
      char llm_head_model[MAX_VLM_STRLEN];
      char llm_decoder_model[MAX_VLM_STRLEN];
      char llm_embedding_model[MAX_VLM_STRLEN];
      char json_path4spgtv[MAX_VLM_STRLEN];
      //vret
      MI_BOOL init_vret;
      char vret_vision_model[MAX_VLM_STRLEN];
      char json_path4vret[MAX_VLM_STRLEN];
      char vret_text_model[MAX_VLM_STRLEN];
      //reid
      MI_BOOL init_reid;
      char reid_feat_model[MAX_VLM_STRLEN];
      char reid_quality_model[MAX_VLM_STRLEN];
  } VLMInit_t;
  

• Members

  Member Name	Description
  ipu_firmware_path	Path for ipu_firmware; not needed if not used
  create_device	Whether to create IPU device within the algorithm library; default is true (create within library), can be set to false to create IPU device externally when calling concurrently
  destroy_device	Whether to destroy IPU device within the algorithm library; default is true (destroy within library), can be set to false to destroy IPU device externally when calling concurrently
  crop_func	Crop function; refer to cropByScalar function in test.cpp
  init_md	Whether to initialize motion target detection algorithm
  ive_handle	Handle of ive needed when enabling md
  md_in_width	Input width for ive sad (generally scale the long edge of input frame to about 480, e.g., for input 1920x1080, set to 480)
  md_in_height	Input height for ive sad (generally scale the long edge of input frame to about 480, e.g., for input 1920x1080, set to 272)
  md_in_stride	Input stride for ive sad
  init_det	Whether to initialize detection function
  init_mot	Initialize multi-object tracking algorithm
  det_model	Detection model
  det_cls	Class configuration for universal detection, necessary only for ovdet
  init_sot	Whether to enable single object tracking
  det4sot_model	Detection model required for enabling single object tracking
  sot_model	Model for single object tracking
  init_ret	Whether to initialize text-based image search function
  init_sgptv	Whether to initialize image generation from text function
  vision_model	Vision model for text-based image search
  sgptv_vision_model	Vision model for image generation from text
  json_path4ret	JSON folder for tokenizer in text-based image search, configured only when performing search
  ret_text_model	Text model for text-based image search
  llm_head_model	Path for head model of language model for image generation from text
  llm_decoder_model	Path for decoder model of language model for image generation from text
  llm_embedding_model	Path for embedding model for image generation from text
  json_path4spgtv	JSON folder for tokenizer in image generation from text
  init_vret	Whether to initialize video search by text function
  vret_vision_model	Vision model for video search by text
  json_path4vret	JSON folder for tokenizer in video search
  vret_text_model	Text model for video search by text
  init_reid	Whether to enable person/vehicle re-identification
  reid_feat_model	Feature extraction model for person/vehicle re-identification
  reid_quality_model	Quality assessment model for person/vehicle re-identification

• Related Data Types and Interfaces

  ALGO_VLM_Init

3.2 VLMImage_t

• Description

  Image data structure.

• Definition

  typedef struct
  {
      void *p_vir_addr;
      MI_PHY phy_addr;
      MI_U32 buf_size;
      MI_U64 pts;
      MI_U16 width;
      MI_U16 height;
      MI_IPU_ELEMENT_FORMAT format;
      MI_U16 width_stride;
      MI_U16 height_stride;
  } VLMImage_t;
  

• Members

  Member Name	Description
  p_vir_addr	Virtual address of image data
  phy_addr	Physical address of image data
  buf_size	Size of image data buffer
  pts	PTS of image data
  width	Width of image
  height	Height of image
  format	Data format of the image
  width_stride	Stride of image width
  height_stride	Stride of image height

• Related Data Types and Interfaces

  VLMInput_t

3.3 VLMInput_t

• Description

  Algorithm input data structure.

• Definition

  typedef struct
  {
      // common
      VLMImage_t current_frame;
      // sot
      VLMBox_t track_obj;
      //ret
      char prompt[MAX_VLM_STRLEN];
      VLMBox_t ret_roi;
      //sgptv
      char sgptv_prompt[MAX_VLM_STRLEN];
      VLMBox_t vision_roi;
      //vret
      char vret_prompt[MAX_VLM_STRLEN];
  } VLMInput_t;
  

• Members

  Member Name	Description
  current_frame	Data of the current frame
  track_obj	Tracking target set for single object tracking
  prompt	Prompt in the text search process
  ret_roi	Specified search area (set ret_topk_boxes to 1)
  sgptv_prompt	Prompt for image generation from text
  vision_roi	Specified image generation area, highest priority; visual features are also within this area
  vret_prompt	Prompt for video search by text

• Related Data Types and Interfaces

  ALGO_VLM_Run

  ALGO_VLM_SetSot

3.4 VLMBox_t

• Description

  Detection tracking box structure.

• Definition

  typedef struct
  {
      MI_U32 x;
      MI_U32 y;
      MI_U32 width;
      MI_U32 height;
      MI_U32 class_id;
      MI_FLOAT score;
      MI_FLOAT distance;
      MI_U64 pts;
      MI_U64 track_id;
  } VLMBox_t;
  

• Members

  Member Name	Description
  x	The x-coordinate of the top left corner
  y	The y-coordinate of the top left corner
  width	Width of the box
  height	Height of the box
  class_id	The detection category ID of the box
  score	Confidence score of the class
  distance	Distance to the single-object tracking box
  pts	Timestamp of the box
  track_id	Tracking ID

• Related Data Types and Interfaces

  ALGO_VLM_Run

  VLMInput_t

3.5 VLMParams_t

• Description

  Real-time parameter configuration for algorithm execution.

• Definition

  typedef struct
  {
      // common
      MI_S32 disp_width;               // default=1920
      MI_S32 disp_height;              // default=1080
      // md
      MI_BOOL run_md;
      MI_U8    md_threshold;         // 0 ~ 255, default 8
      MI_FLOAT md_min_width;         // 0.0 ~ 1.0 default 0.06
      MI_FLOAT md_min_height;        // 0.0 ~ 1.0 default 0.1
      MI_FLOAT md_max_width;         // 0.0 ~ 1.0 default 0.3
      MI_FLOAT md_max_height;        // 0.0 ~ 1.0 default 1.0
      MI_FLOAT md_max_frame_gap;     // 0.0 ms ~ INF ms, default 100 ms
      MI_U32   md_max_num_boxes;     // max num detected boxes, default 1
      // det and track
      MI_BOOL run_det;
      MI_BOOL run_mot;
      MI_FLOAT detect_threshold;         // default=0.5
      MI_BOOL ignore_static_objects;   // default=false
      MI_FLOAT static_sensitive;       // default=0.85
      MI_BOOL stable_bbox;             // default=false
      MI_FLOAT stable_sensitive;       // default=0.63
      MI_S32 ignore_frame_number;      // [0-5] ignore the id of first detected box, default=0
  
      // sot
      MI_BOOL run_sot;
      MI_U32   template_min_width;
      MI_U32   missing_detection_times_end_track;
      MI_U32   lost_times_end_track;
      MI_FLOAT track_threshold;
      MI_S32 head_cls_id;
      //fr
      MI_BOOL run_fr;
      VLMFrDetMode_e fr_det_mode;
      MI_FLOAT fr_det_threshold;
      MI_FLOAT fr_quality_threshold;
      MI_FLOAT fr_min_eye_distance;
      MI_FLOAT fr_min_face_size;
      MI_FLOAT fr_emo_threshold;
      //ret
      MI_BOOL run_ret;
      MI_U32  ret_topk_boxes; //top k by box size
      MI_BOOL run_ret_cls;     //
      MI_S32 person_class; // class in detection model; if not exists, set to -1
      MI_S32 dog_class;    // class in detection model; if not exists, set to -1
      MI_S32 cat_class;    // class in detection model; if not exists, set to -1
      //sgptv
      MI_BOOL run_sgptv;
      MI_U32 sgptv_top_k;                           // top-K sampling parameter for token generation; currently, only top-K=1 is supported
      MI_FLOAT sgptv_top_p;
      MI_FLOAT sgptv_temperature;                     // sampling temperature affecting the randomness of token selection
      MI_FLOAT sgptv_repeat_penalty;
      MI_BOOL sgptv_do_sample;
      //vret
      MI_BOOL run_vret;
      MI_BOOL first_frame;
      MI_U32 frame_step;
      MI_U32 frame_shift;
      // reid
      MI_BOOL run_reid;
      MI_S32 reid_class;
      MI_FLOAT reid_quality_threshold;
  } VLMParams_t;
  

• Members

  Member Name	Description
  disp_width	Display width; should match the input image width
  disp_height	Display height; should match the input image height
  run_md	Whether to run motion target detection
  md_threshold	Threshold for motion detection; recommended value is 8
  md_min_width	Minimum normalized width for motion target; recommended value is 0.06
  md_min_height	Minimum normalized height for motion target; recommended value is 0.1
  md_max_width	Maximum normalized width for motion target; recommended value is 0.3
  md_max_height	Maximum normalized height for motion target; recommended value is 1.0
  run_det	Whether to enable detection
  run_mot	Whether to enable multi-object tracking
  detect_threshold	Detection threshold
  ignore_static_objects	Whether to ignore static objects
  static_sensitive	Sensitivity for ignoring static objects; recommended value is 0.85
  stable_bbox	Whether to enable stable bounding box
  stable_sensitive	Sensitivity for stable bounding box; recommended value is 0.63
  ignore_frame_number	Ignore the first few frames' detection results
  run_sot	Whether to run single object tracking
  template_min_width	Minimum width for tracking template
  missing_detection_times_end_track	Number of missed detections to end tracking
  lost_times_end_track	Number of losses to end tracking
  track_threshold	Tracking threshold
  head_cls_id	ID for head category in detection
  run_ret	Whether to enable text-based image search
  ret_topk_boxes	Perform feature extraction on the top k largest boxes; recommended value is 1; if detection is not enabled, feature extraction is performed on the original image's min(width, height)
  run_ret_cls	Whether to perform fine classification of persons/pets; requires detection configuration
  person_class	Category for persons in the detector
  dog_class	Category for dogs in the detector
  cat_class	Category for cats in the detector
  run_sgptv	Whether to run image generation from text
  sgptv_top_k	Choose top k for post-processing after sampling; currently only supports top k = 1
  sgptv_top_p	Choose cumulative probability greater than top_p during post-processing; recommended value is 0.85
  sgptv_temperature	Temperature parameter for post-processing; takes effect when do_sample is enabled; recommended value is 1.0
  sgptv_repeat_penalty	Penalty coefficient for repetition; recommended value is 1.0
  sgptv_do_sample	Whether performing sampling; recommended value is false
  run_vret	Whether to run video search by text
  first_frame	Whether this is the first frame of the video; each first frame of offline video needs to be set to true
  frame_step	Interval in frames; how many frames to take into the model inference cache; once there are 8 frames in the cache, model inference is done once
  frame_shift	How many frames to shift the model cache; total model input is 8 frames, shift value is [1-8]; recommended value is 4
  run_reid	Whether to run person/vehicle re-identification
  reid_class	Detection category class_id of the re-identified target
  reid_quality_threshold	Quality score threshold for the re-identified target

  Note:

  • The detailed categories for dogs are: [Border Collie, Shiba Inu, German Shepherd, French Bulldog, Poodle, Chihuahua, Golden Retriever, Labrador Retriever, Dachshund, Yorkshire Terrier]

  • The detailed categories for cats are: [Ragdoll, Mackerel Tabby, American Shorthair, Maine Coon, Norwegian Forest Cat, Sphynx Cat, Scottish Fold, Siamese Cat, Exotic Shorthair, British Shorthair]

  • The detailed categories for people are: [Person with knife, person with gun, person without knife and gun]

• Related Data Types and Interfaces

  ALGO_VLM_Run

3.6 VLMRetSgptvResult

• Explanation

  Structure for visual results of text-based image search and image generation from text.

• Definition

  typedef struct
  {
      MI_U32 x;
      MI_U32 y;
      MI_U32 width;
      MI_U32 height;
      MI_U32 dog_cls;
      MI_U32 person_cls;
      MI_U32 cat_cls;
      MI_FLOAT feature[VLM_FEATURE_LEN];
  }VLMRetSgptvResult;
  

• Members

  Member Name	Description
  x	Top-left x-coordinate of crop region
  y	Top-left y-coordinate of crop region
  width	Width of crop region
  height	Height of crop region
  dog_cls	Subcategory of the dog
  person_cls	Subcategory of the person
  cat_cls	Subcategory of the cat
  feature	Features of the crop region in the image

• Related Data Types and Interfaces

  ALGO_VLM_Run

3.7 VLMOutput_t

• Explanation

  Structure for algorithm results.

• Definition

  typedef struct
  {
      // md
      VLMBox_t md_rois[MAX_VLM_ROIS];
      MI_S32 num_md_rois;
      MI_FLOAT md_diff_ratio;
      VLMImage_t md_mask;
  
      //det
      VLMBox_t det_boxes[MAX_VLM_OBJS];
      MI_S32 num_det_boxes;
      // sot
      VLMBox_t track_result;
      VLMBox_t track_roi;
      MI_BOOL tracking;
      //ret
      VLMRetSgptvResult vision_features[MAX_RET_VISIOM_NUM];
      MI_U32 ret_feature_num;
      MI_FLOAT text_feature[VLM_FEATURE_LEN];
      //sgptv
      MI_U8 description[MAX_VLM_STRLEN];
      //fr
      VLMFrResult_t fr_result[VLM_MAX_FACE];
      int num_fr_results;
      //vret
      VLMVRetResult vret_result;
      //reid
      VLMReIdResult_t reid_result[MAX_VLM_REID_OBJS];
      int num_reid_results;
  } VLMOutput_t;
  

• Members

  Member Name	Description
  md_rois	Motion areas
  num_md_rois	Number of motion areas
  md_diff_ratio	motion area ratio（0.0-1.0）
  md_mask	（optional）The mask output of the motion area needs to allocate and provide a memory block of size md_in_width x md_in_height x sizeof(MI_U8), where the motion area corresponds to 255 and the stationary area corresponds to 0.
  det_boxes	Detection boxes
  num_det_boxes	Number of detection boxes
  track_result	Result of single object tracking
  track_roi	Search area for single object tracking
  tracking	Tracking status
  vision_features	Visual results from text-based image search and image generation
  ret_feature_num	Number of features in text-based image search
  description	Description results from image generation
  text_feature	Image features from text-based image search
  fr_result	Output results from face recognition algorithm
  num_fr_results	Number of output results from face recognition algorithm
  vret_result	Results from video search by text
  reid_result	Output results from person/vehicle re-identification algorithm
  num_reid_results	Number of output results from person/vehicle re-identification algorithm

• Related Data Types and Interfaces

  ALGO_VLM_Run

  VLMVRetResult

3.8 VLMCropFunc

• Explanation

  Crop function.

• Definition

  typedef int (*VLMCropFunc)(const VLMImage_t *in, const VLMBox_t *box, VLMImage_t *out);
  

• Members

  Member Name	Description
  in	Original image
  box	Corresponding crop area
  out	Result after cropping

3.9 VLMVRetResult

• Explanation

  Output structure for video search by text.

• Definition

  typedef struct
  {
      MI_BOOL has_vision_feature;
      MI_FLOAT vision_feature[VLM_FEATURE_LEN];
      MI_FLOAT text_feature[VLM_FEATURE_LEN];
  }VLMVRetResult;
  

  • Members

  Member Name	Description
  has_vision_feature	Whether visual features are returned; since it is a video model, not every frame has visual features returned, which depends on the parameters set
  vision_feature	Visual features
  text_feature	Text features

3.10 VLMFrResult_t

• Explanation

  Output structure for the face recognition algorithm.

• Definition

  typedef struct
  {
      MI_U32 x;
      MI_U32 y;
      MI_U32 width;
      MI_U32 height;
      MI_U64 track_id;
      MI_FLOAT keypts[VLM_FACE_KEYPTS][3];
      MI_FLOAT pitch;
      MI_FLOAT yaw;
      MI_FLOAT roll;
      MI_FLOAT quality_score;
      MI_BOOL befiltered;
      MI_FLOAT feature[VLM_FEATURE_LEN];
      MI_U64 recog_id;
      MI_FLOAT recog_simi;
      MI_S32 emo_type;
  }VLMFrResult_t;
  

• Members

  Member Name	Description
  x	Top-left x-coordinate of crop area
  y	Top-left y-coordinate of crop area
  width	Width of crop area
  height	Height of crop area
  track_id	Tracking ID of the face
  keypts	Coordinates of face keypoints
  pitch	Pitch angle of the face
  yaw	Yaw angle of the face
  roll	Roll angle of the face
  quality_score	Comprehensive score based on face angle
  befiltered	Whether the face has been filtered (due to low quality score)
  feature	Face feature vector
  recog_id	Face recognition ID
  recog_simi	Face recognition similarity
  emo_type	Face emotion category

3.11 VLMReIdResult_t

• Explanation

  Output structure for person/vehicle re-identification algorithm.

• Definition

  typedef struct
  {
      MI_U32 x;
      MI_U32 y;
      MI_U32 width;
      MI_U32 height;
      MI_U32 class_id;
      MI_FLOAT score;
      MI_U64 pts;
      MI_FLOAT quality_score;
      MI_FLOAT feature[VLM_FEATURE_LEN];
  }VLMReIdResult_t;
  

• Members

  Member Name	Description
  x	Top-left x-coordinate of crop area
  y	Top-left y-coordinate of crop area
  width	Width of crop area
  height	Height of crop area
  class_id	Category ID of the target
  score	Detection score of the target
  pts	Timestamp of the target frame
  quality_score	Quality score of the target
  feature	Target feature vector

4. errorcode

Error Code	Value	Description
E_ALGO_SUCCESS	0	Operation successful
E_ALGO_HANDLE_NULL	1	Algorithm handle is null
E_ALGO_INVALID_PARAM	2	Invalid input parameter
E_ALGO_DEVICE_FAULT	3	Hardware error
E_ALGO_LOADMODEL_FAIL	4	Failed to load the model
E_ALGO_INIT_FAIL	5	Algorithm initialization failed
E_ALGO_NOT_INIT	6	Algorithm not initialized
E_ALGO_INPUT_DATA_NULL	7	Algorithm input data is null
E_ALGO_INVALID_INPUT_SIZE	8	Invalid algorithm input data dimension
E_ALGO_INVALID_LICENSE	9	Invalid license
E_ALGO_MEMORY_OUT	10	Memory insufficient
E_ALGO_FILEIO_ERROR	11	File read/write error
E_ALGO_INVALID_OUTPUT_SIZE	12	Invalid algorithm output data dimension
E_ALGO_INVALID_DECODE_MODE	13	Invalid decode mode
E_ALGO_MODEL_INVOKE_ERROR	14	Invoke failed