Detection Algorithm

---

REVISION HISTORY

Revision No.	Description	Date
1.0	First version	04/25/2023
1.1	Second version	10/30/2023
3.0	Third version	04/11/2024
3.1	Remove deprecated models introduction	10/31/2024
3.2	Add ALGO_DET_SortResult API	12/03/2024
3.3	Adjust models description	12/06/2024

1. General Description

1.1. Algorithm Description

The mainly specific detection labels are described as follows:

Model	Function	Input size(w*h)	Input Format
sbdy36.img	Bag detection(Small), outputs one label, i.e. bag(class_id=0)	640*352	yuvsp420_nv12
sbdy36l.img	Bag detection(Large), outputs one label, i.e. bag(class_id=0)	640*352	yuvsp420_nv12
sbdy48.img	Bag detection(Small), outputs one label, i.e. bag(class_id=0)	800*480	yuvsp420_nv12
sbdy48l.img	Bag detection(Large), outputs one label, i.e. bag(class_id=0)	800*480	yuvsp420_nv12
sdy36.img	Person + vehicle + pet + head + face detection(Small), outputs ten labels, i.e. person(class_id=0), bicycle(class_id=1), cat(class_id=2), motorcycle(class_id=3), bus(class_id=4), truck(class_id=5), cat(class_id=6), dog(class_id=7), head(class_id=8), face(class_id=9)	640*352	yuvsp420_nv12
sdy36l.img	Person + vehicle + pet + head + face detection(Large) , outputs ten labels, i.e. person(class_id=0), bicycle(class_id=1), cat(class_id=2), motorcycle(class_id=3), bus(class_id=4), truck(class_id=5), cat(class_id=6), dog(class_id=7), head(class_id=8), face(class_id=9)	640*352	yuvsp420_nv12
sdy48.img	Person + vehicle + pet + head + face detection(Small) , outputs ten labels, i.e. person(class_id=0), bicycle(class_id=1), cat(class_id=2), motorcycle(class_id=3), bus(class_id=4), truck(class_id=5), cat(class_id=6), dog(class_id=7), head(class_id=8), face(class_id=9)	800*480	yuvsp420_nv12
sdy48l.img	Person + vehicle + pet + head + face detection(Large), outputs ten labels, i.e. person(class_id=0), bicycle(class_id=1), cat(class_id=2), motorcycle(class_id=3), bus(class_id=4), truck(class_id=5), cat(class_id=6), dog(class_id=7), head(class_id=8), face(class_id=9)	800*480	yuvsp420_nv12
sfsdy36.img	Fire and smoke detection, outputs two labels, i.e. fire(class_id=0), smoke(class_id=1)	640*352	yuvsp420_nv12
sfsdy48.img	Fire and smoke detection, outputs two labels, i.e. fire(class_id=0), smoke(class_id=1)	800*480	yuvsp420_nv12
spdy36.img	Person + head + face detection(Small), outputs three labels, i.e. person(class_id=0), head(class_id=1), face(class_id=2)	640*352	yuvsp420_nv12
spdy36l.img	Person + head + face detection(Large), outputs three labels, i.e. person(class_id=0), head(class_id=1), face(class_id=2)	640*352	yuvsp420_nv12
spdy48.img	Person + head + face detection(Small), outputs three labels, i.e. person(class_id=0), head(class_id=1), face(class_id=2)	800*480	yuvsp420_nv12
spdy48l.img	Person + head + face detection(Large), outputs three labels, i.e. person(class_id=0), head(class_id=1), face(class_id=2)	800*480	yuvsp420_nv12

2. API Reference

This function module provides the following APIs:

API Name	Function
ALGO_DET_CreateHandle	Create handle
ALGO_DET_InitHandle	Initialize handle
ALGO_DET_SetParams	Set configurable parameters
ALGO_DET_GetInputAttr	Get model attribute information
ALGO_DET_Run	Run detection algorithm
ALGO_DET_SortResult	Sort detection results
ALGO_DET_DeinitHandle	Deinitialize detection algorithm
ALGO_DET_ReleaseHandle	Release handle

2.1 ALGO_DET_CreateHandle

• Function

  Create handle

• Syntax

  MI_S32 ALGO_DET_CreateHandle(void **handle);
  

• Parameter

  Parameter Name	Description	Input/Output
  handle	Handle	Input

• Return Value

  Return Value	Description
  0	Successful.
  Other	Failed, see Error Code for details

• Dependency

  • Header file: sgs_det_api.h
  • Library file: libsgsalgo_det.a / libsgsalgo_det.so

2.2 ALGO_DET_InitHandle

• Function

  Initialize handle

• Syntax

  MI_S32 ALGO_DET_InitHandle(void *handle, DetInit_t *init_info);
  

• Parameter

  Parameter Name	Description	Input/Output
  handle	Handle	Input
  init_info	Detection algorithm configuration option, see DetInit_t for details	Input

• Return Value

  Return Value	Description
  0	Successful.
  Other	Failed, see Error Code for details

• Dependency

  • Header file: sgs_det_api.h
  • Library file: libsgsalgo_det.a / libsgsalgo_det.so

2.3 ALGO_DET_SetParams

• Function

  Set configurable parameters

• Syntax

  MI_S32 ALGO_DET_SetParams(void *handle, const DetParams_t *params);
  

• Parameter

  Parameter Name	Description	Input/Output
  handle	Handle	Input
  params	Configurable parameters	Input

• Return Value

  Return Value	Description
  0	Successful.
  Other	Failed, see Error Code for details

• Dependency

  • Header file: sgs_det_api.h
  • Library file: libsgsalgo_det.a / libsgsalgo_det.so

2.4 ALGO_DET_GetInputAttr

• Function

  Get model attribute information, including model input resolution and input data type

• Syntax

  MI_S32 ALGO_DET_GetInputAttr(void *handle, DetInputAttr_t *input_attr);
  

• Parameter

  Parameter Name	Description	Input/Output
  handle	Handle	Input
  input_attr	Pointer to the saved attribute information, see DetInputAttr_t for details	Input

• Return Value

  Return Value	Description
  0	Successful.
  Other	Failed, see Error Code for details

• Dependency

  • Header file: sgs_det_api.h
  • Library file: libsgsalgo_det.a / libsgsalgo_det.so

2.5 ALGO_DET_Run

• Function

  Run detection algorithm

• Syntax

  MI_S32 ALGO_DET_Run(void *handle, const DetInput_t *algo_input, DetBox_t bboxes[MAX_DET_OBJECT], MI_S32 *num_bboxes);
  

• Parameter

  Parameter Name	Description	Input/Output
  handle	Handle	Input
  algo_input	Input image buffer information	Input
  bboxes	Used to save the array of detection result boxes	Input
  num_bboxes	Pointer to the saved number of detection result boxes	Input

• Return Value

  Return Value	Description
  0	Successful.
  Other	Failed, see Error Code for details

• Dependency

  • Header file: sgs_det_api.h
  • Library file: libsgsalgo_det.a / libsgsalgo_det.so

2.6. ALGO_DET_SortResult

• Function

  Sort detection results

• Syntax

  MI_S32 ALGO_DET_SortResult(void *handle, const DetSortInput_t *sort_input, DetBox_t bboxes[MAX_DET_OBJECT], MI_S32 *num_bboxes);
  

• Parameter

  Parameter Name	Description	Input/Output
  handle	Handle	Input
  sort_input	sort input structure	Input
  bboxes	Used to save the array of sorted result boxes	Input/Output
  num_bboxes	Pointer to the saved number of sorted result boxes	Input/Output

• Return Value

  Return Value	Description
  0	Successful. NOTE：this api will change bboxes and num_bboxes
  Other	Failed, see Error Code for details

• Dependency

  • Header file: sgs_det_api.h
  • Library file: libsgsalgo_det.a / libsgsalgo_det.so

2.7 ALGO_DET_DeinitHandle

• Function

  Deinitialize detection algorithm

• Syntax

  MI_S32 ALGO_DET_DeinitHandle(void *handle);
  

• Parameter

  Parameter Name	Description	Input/Output
  handle	Handle	Input

• Return Value

  Return Value	Description
  0	Successful.
  Other	Failed, see Error Code for details

• Dependency

  • Header file: sgs_det_api.h
  • Library file: libsgsalgo_det.a / libsgsalgo_det.so

2.8 ALGO_DET_ReleaseHandle

• Function

  Release the resources occupied by the handle

• Syntax

  MI_S32 ALGO_DET_ReleaseHandle(void *handle);
  

• Parameter

  Parameter Name	Description	Input/Output
  handle	Handle	Input

• Return Value

  Return Value	Description
  0	Successful.
  Other	Failed, see Error Code for details

• Dependency

  • Header file: sgs_det_api.h
  • Library file: libsgsalgo_det.a / libsgsalgo_det.so

3. Structure Type

The detection related data types are defined as follows:

Data Type	Definition
DetInputAttr_t	Algorithm input structure
DetBox_t	Algorithm output structure
DetInit_t	Algorithm initial parameter structure
DetInput_t	Algorithm input image data information
DetParams_t	Algorithm configurable parameter structure
DetSortInput_t	Sort input related structure

3.1 DetInputAttr_t

• Description

  Define the resolution size and pixel format

• Definition

  typedef struct
  {
      MI_U32 width;
      MI_U32 height;
      MI_IPU_ELEMENT_FORMAT format;
  } DetInputAttr_t;
  

• Member

  Member Name	Description
  width	Model input data width
  height	Model input data height
  format	Model input pixel format

• Related data type and interface

  ALGO_DET_GetInputAttr

3.2 DetBox_t

• Description

  Algorithm output structure

• Definition

  typedef struct
  {
      MI_U32 x;
      MI_U32 y;
      MI_U32 width;
      MI_U32 height;
      MI_U32 class_id;
      MI_FLOAT score;
      MI_U64 pts;
  }DetBox_t;
  

• Member

  Member Name	Description
  x, y, width, height	Detection result box position
  class_id	Detection result box class ID
  score	Detection result box score
  pts	Timestamp of the incoming detection frame

• Related data type and interface

  ALGO_DET_Run

3.3 DetInit_t

• Description

  Detection algorithm initial parameter structure

• Definition

  typedef struct
  {
      char ipu_firmware_path[MAX_DET_STRLEN]; // ipu_firmware.bin path
      char model_path[MAX_DET_STRLEN];        // detect model path
      MI_BOOL create_device;                  // set false to create ipu device outside algo lib
      MI_BOOL destroy_device;                 // set false to destroy ipu device outside algo lib
      void *model_buffer;                     // set it when load model from memory
      MI_U32 model_buffer_len;                // set it when load model from memory
  }DetInit_t;
  

• Member

  Member Name	Description
  ipu_firmware_path	Ipu firmware path
  model_path	Model path
  create_device	Whether to create IPU device in detection algorithm,default is true; You can set false when create IPU device outside
  destroy_device	Whether to destory IPU device in detection algorithm,default is true; You can set false when destory IPU device outside
  model_buffer	Set model_path to NULL when using model_buffer; set model_buffer and model_buffer_len to load model from RAM to initialize detection algorithm
  model_buffer_len	Set model_path to NULL when using model_buffer_len; set model buffer length in RAM

• Related data type and interface

  ALGO_DET_InitHandle

3.4 DetInput_t

• Description

  Algorithm input image data information

• Definition

  typedef struct
  {
      void *p_vir_addr;
      MI_PHY phy_addr;
      MI_U32 buf_size;
      MI_U64 pts;
      MI_U16 width;
      MI_U16 height;
  }DetInput_t;
  

• Member

  Member Name	Description
  p_vir_addr	Input buffer virtual address
  phy_addr	Input buffer physical address
  buf_size	Input buffer size
  pts	Input buffer timestamp
  width	Input buffer width
  height	Input buffer height

• Related data type and interface

  ALGO_DET_Run

3.5 DetParams_t

• Description

  Algorithm configurable parameter structure

• Definition

  typedef struct
  {
      MI_FLOAT conf_threshold;         // default=0.5
      MI_S32 disp_width;               // default=1920
      MI_S32 disp_height;              // default=1080
      MI_BOOL ignore_static_objects;   // default=false
      MI_FLOAT static_sensitive;       // default=0.85
      MI_BOOL stable_bbox;             // default=false
      MI_FLOAT stable_sensitive;       // default=0.63
      MI_S32 ignore_frame_number;      // [0-5] ignore the id first number box of detect,default=0
      MI_BOOL strict_mode;             // default=false
  }DetParams_t;
  

• Member

  Member Name	Description
  conf_threshold	Detection threshold (0.0~1.0), default is 0.5. Larger threshold will reduce false alarm and recall; Smaller threshold will increase false alarm and recall
  disp_width	Display width (used for mapping the location of detection boxes), default is 1920
  disp_height	Display height (used for mapping the location of detection boxes), default is 1080
  ignore_static_objects	Whether to filter static objects, default is false, i.e. do not filter static objects
  static_sensitive	Static detecton threshold(0.0~1.0), default is 0.85. Larger threshold will keep boxes exhibiting minor movement
  stable_bbox	Whether to enable stable box, default is false, i.e. disable stable box
  stable_sensitive	Stable box threshold (0.0~1.0), default is 0.63. Smaller threshold will get better stable boxes
  ignore_frame_number	Filter the first frames, using to filter occasional false alarm, (0~5), default is 0, i.e. do not filter the first frames
  strict_mode	Whether to run algorithm in strict mode, which can reduce false alarm when enable, default is false

• Related data type and interface

  ALGO_DET_SetParams

3.6 DetSortInput_t

• Description

  Sort input related structure

• Definition

  typedef struct
  {
      MI_S32 class_indexs[MAX_DET_CLASSES];
      MI_S32 class_num;
      DetSortType_e sort_type;
  }DetSortInput_t;
  

• Member

  Member Name	Description
  class_indexs	Index of the classes to be sorted
  class_num	The number of classes to sort
  sort_type	Sorting method type

• Related data type and interface

  ALGO_DET_SortResult

4. Enumeration Type

The detection related data types are defined as follows:

Data Type	Definition
Label_FD_Face_e	Correspondence between model type and class_id of face detection
Label_FSD_e	Correspondence between model type and class_id of fire and smoke detection
Label_SD_e	Correspondence between model type and class_id of person + vehicle + pet + head + face detection
Label_SPD_e	Correspondence between model type and class_id of person + head + face detection
Label_BD_e	Correspondence between model type and class_id of bag detection

4.1 Label_FD_Face_e

• Description

  Correspondence between model type and class_id of face detection

• Definition

  typedef enum
  {
      E_FD_FACE = 0,
  }Label_FD_Face_e;
  

• Member

  Member Name	Description
  E_FD_FACE	Face(class_id=0)

• Related data type and interface

  DetBox_t

4.2 Label_FSD_e

• Description

  Correspondence between model type and class_id of fire and smoke detection

• Definition

  typedef enum
  {
      E_FSD_FIRE,
      E_FSD_SMOKE
  }Label_FSD_e;
  

• Member

  Member Name	Description
  E_FSD_FIRE	Fire(class_id=0)
  E_FSD_SMOKE	Smoke(class_id=1)

• Related data type and interface

  DetBox_t

4.3 Label_SD_e

• Description

  Correspondence between model type and class_id of person + vehicle + pet + head + face detection

• Definition

  typedef enum
  {
      E_SD_PERSON = 0,
      E_SD_BICYCLE,
      E_SD_CAR,
      E_SD_MOTOCYCLE,
      E_SD_BUS,
      E_SD_TRUCK,
      E_SD_CAT,
      E_SD_DOG,
      E_SD_HEAD,
      E_SD_FACE,
  }LABEL_SD_e;
  

• Member

  Member Name	Description
  E_SD_PERSON	Person(class_id=0)
  E_SD_BICYCLE	Bicycle(class_id=1)
  E_SD_CAR	Car (class_id=2)
  E_SD_MOTOCYCLE	Motocyle(class_id=3)
  E_SD_BUS	Bus(class_id=4)
  E_SD_TRUCK	Truck(class_id=5)
  E_SD_CAT	Cat(class_id=6)
  E_SD_DOG	Dog(class_id=7)
  E_SD_HEAD	Head(class_id=8)
  E_SD_FACE	Face(class_id=9)

• Related data type and interface

  DetBox_t

4.4 Label_SPD_e

• Description

  Correspondence between model type and class_id of person + head + face detection

• Definition

  typedef enum
  {
      E_SPD_PERSON = 0,
      E_SPD_HEAD,
      E_SPD_FACE
  }LABEL_SPD_e;
  

• Member

  Member Name	Description
  E_SPD_PERSON	Person(class_id=0)
  E_SPD_HEAD	Head(class_id=1)
  E_SPD_FACE	Face(class_id=2)

• Related data type and interface

  DetBox_t

4.5 Label_BD_e

• Description

  Correspondence between model type and class_id of bag detection

• Definition

  typedef enum
  {
      E_BD_BAG = 0,
  }LABEL_BD_e;
  

• Member

  Member Name	Description
  E_BD_BAG	Bag(class_id=0)

• Related data type and interface

  DetBox_t

5. Error code

Error Code	Value	Description
E_ALGO_SUCCESS	0	Operation successful
E_ALGO_HANDLE_NULL	1	Algorithm handle is null
E_ALGO_INVALID_PARAM	2	Invalid input parameter
E_ALGO_DEVICE_FAULT	3	Hardware error
E_ALGO_LOADMODEL_FAIL	4	Failed to load model
E_ALGO_INIT_FAIL	5	Algorithm initialization failed
E_ALGO_NOT_INIT	6	Algorithm not initialized
E_ALGO_INPUT_DATA_NULL	7	Algorithm input data is null
E_ALGO_INVALID_INPUT_SIZE	8	Invalid algorithm input data dimension
E_ALGO_INVALID_LICENSE	9	Invalid license
E_ALGO_MEMORY_OUT	10	Out of memory
E_ALGO_FILEIO_ERROR	11	File read and write operation error
E_ALGO_INVALID_OUTPUT_SIZE	12	Invalid algorithm output data dimension
E_ALGO_INVALID_DECODE_MODE	13	Invalid decode mode
E_ALGO_MODEL_INVOKE_ERROR	14	Invoke fail

6. Badcase Optimization Solutions

6.1. SGS Training

User Process for Badcase Optimization via SGS:

• Users provide problem information and badcase material files to the contact person.

  1. Users provide problem information, including a description of the issue, the model used, thresholds, input resolution, etc.
     • For example, human shape miss detection/mis-detection, model sdy48s.img, threshold 0.5, input resolution 800x480.
  2. Users provide badcase material files (dump save detection algorithm input image data to more accurately reproduce and confirm whether the issue is with the algorithm itself).
     • Set environment variable export SGS_DET_SAVE_DIR=dump to save the detection algorithm input image data (data type is consistent with model input data type), where 'dump' is the folder where the input is saved; users need to create this folder themselves.

• SGS aligns and reproduces results.

• User Badcase Material Collection.

  1. Users collect more badcase materials for the contact person, including images or videos (requires compliance with actual use cases, with each video scene lasting about 10 seconds).

• SGS Badcase Optimization Training.

6.2. Dlatrain Fine-Tuning

User Self-Utilization of Dlatrain Fine-Tuning Process:

• Discover and Confirm Badcases.

  1. Python side reproduction (for detailed process, refer to dlatrain/detection/det/ReadMe.md Section I. File Description, Item 3. predict.py).
  2. Board-level reproduction (model conversion, refer to dlatrain/detection/det/ReadMe.md Section I. File Description, Item 4. convert.py).

• Badcase Material Collection and Processing.

  1. Collect more badcase materials, including images or videos.
  2. Data processing and annotation to generate dataset.yaml (for detailed format, refer to dlatrain/detection/configs/datasets/coco.yaml).

• Dlatrain Fine-Tuning Training.

  1. Use Dlatrain for fine-tuning training (for detailed process, refer to dlatrain/detection/det/ReadMe.md Section I. File Description, Item 1. train.py, training based on an existing pre-trained model).
  2. Fine-tuned model badcase reproduction; if it still can reproduce, expand the dataset for further optimization training.

• Regularly return badcase materials to SGS to continuously improve base model performance.

6.3. Comparison of Solutions

COMPARISON DIMENSION	SGS TRAINING	DLATRAIN FINE-TUNING
Optimization Cycle	Medium (overall process takes about 1-2 weeks)	Short (self-controlled, reduced iteration cycle)
Technical Barrier	Low	Medium (requires certain development skills)
Flexibility	Medium (generalized optimization)	High (precise adaptation to business scenarios)