Human segmentation algorithm

REVISION HISTORY

Revision No.
Description
 Date
100
  • Initial release
    		•  05/04/2023
    101
  • Update
    		•  11/07/2024

    1. Algorithm Description

    The function of the human segmentation algorithm is to output a mask of the foreground portrait, which can be applied to background blur and background replacement.

    2. Algorithm Performance

    Algorithm Resolution Inference Time Rom Ram BW
    hseg_y36.img 640*360 39.08ms 4409KB 9828KB 101619KB

    Note: The following model is only released on the iford platform

    Algorithm Resolution Inference Time Rom Ram BW
    hseg_y36_single.img 640*360 13.93ms 1987KB 4428KB 45810KB
    hseg_y46_single.img 640*480 19.30ms 2021KB 6746KB 63350KB

    3. Functional Module API

    API Name Function
    ALGO_HSEG_CreateHandle Create handle
    ALGO_HSEG_InitHandle Initialization handle
    ALGO_HSEG_GetInputAttr Get the attribute information of the model input
    ALGO_HSEG_GetOutputAttr Get the attribute information of the model output
    ALGO_HSEG_Run Perform human segmentation
    ALGO_HSEG_DeinitHandle Deinitialization handle
    ALGO_HSEG_ReleaseHandle Release handle

    3.1. ALGO_HSEG_CreateHandle

    • Function

      Create handle.

    • Syntax

      MI_S32 ALGO_HSEG_CreateHandle(void **handle);

    • Parameters

      Parameter Name Description Input/Output
      handle handle input

    • Return value

      Return value Description
      0 Success
      Others Error code

    • Dependency

      Header file：sgs_hseg_api.h

      Library file：libsgsalgo_hseg.so/libsgsalgo_hseg.a

    3.2. ALGO_HSEG_InitHandle

    • Function

      Initialize the handle.

    • Syntax

      MI_S32 ALGO_HSEG_InitHandle(void *handle, const HsegInit_t *init_info);

    • Parameters

      Parameter Name Description Input/Output
      handle handle input
      init_info initialization parameters input

    • Return value

      Return value Description
      0 Success
      Others Error Code

    • Dependency

      Header file：sgs_hseg_api.h

      Library file：libsgsalgo_hseg.so/libsgsalgo_hseg.a

    3.3. ALGO_HSEG_GetInputAttr

    • Function

      Gets the model input property information, including the model input resolution and the type of input data.

    • Syntax

      MI_S32 ALGO_HSEG_GetInputAttr(void *handle, HsegAttr_t *input_attr);

    • Parameters

      Parameter Name Description Input/Output
      handle handle input
      input_attr output structure output

    • Return value

      Return value Description
      0 Success
      Others Error Code

    • Dependency

      Header file：sgs_hseg_api.h

      Library file：libsgsalgo_hseg.so/libsgsalgo_hseg.a

    3.4. ALGO_HSEG_GetOutputAttr

    • Function

      Gets the model output property information, including the model output resolution and the type of output data.

    • Syntax

      MI_S32 ALGO_HSEG_GetOutputAttr(void *handle, HsegAttr_t *y_mask_attr, HsegAttr_t *uv_mask_attr);

    • Parameters

      Parameter Name Description Input/Output
      handle handle input
      y_mask_attr Output structure, get the attribute information of model output Output
      uv_mask_attr Output structure, get the attribute information of model output Output

    • Return value

      Return value Description
      0 Success
      Others Error Code

    • Dependency

      Header file：sgs_hseg_api.h

      Library file：libsgsalgo_hseg.so/libsgsalgo_hseg.a

    3.5. ALGO_HSEG_Run

    • Function

      Perform human segmentation.

    • Syntax

      MI_S32 ALGO_HSEG_Run(void *handle, const HsegInput_t *algo_input, const HsegInput_t *y_mask, const HsegInput_t *uv_mask);

    • Parameters

      Parameter Name Description Input/Output
      handle handle input
      algo_input input buffer input
      y_mask output buffer output
      uv_mask output buffer output

    • Return value

      Return value Description
      0 Success
      Others Error Code

    • Dependency

      Header file：sgs_hseg_api.h

      Library file：libsgsalgo_hseg.so/libsgsalgo_hseg.a

    3.6. ALGO_HSEG_DeinitHandle

    • Function

      Deinitialize the handle.

    • Syntax

      MI_S32 ALGO_HSEG_DeinitHandle(void *handle);

    • Parameters

      Parameter Name Description Input/Output
      handle handle input

    • Return value

      Return value Description
      0 Success
      Others Error Code

    • Dependency

      Header file：sgs_hseg_api.h

      Library file：libsgsalgo_hseg.so/libsgsalgo_hseg.a

    3.7. ALGO_HSEG_ReleaseHandle

    • Function

      Release the handle.

    • Syntax

      MI_S32 ALGO_HSEG_ReleaseHandle(void *handle);

    • Parameters

      Parameter Name Description Input/Output
      handle handle input

    • Return value

      Return value Description
      0 Success
      Others Error Code

    • Dependency

      Header file：sgs_hseg_api.h

      Library file：libsgsalgo_hseg.so/libsgsalgo_hseg.a

    4. Data Types

    Data Types Definition
    HsegAttr_t Model input structure
    HsegInit_t Algorithm initialization parameter related structure
    HsegInput_t Algorithm input related structure

    4.1. HsegAttr_t

    • Description

      Model input structure.

    • Definition

      typedef struct
{
    MI_U32 width;
    MI_U32 height;
    MI_IPU_ELEMENT_FORMAT format;
} HsegAttr_t;

    • Member

      Member Name Description
      width width of model input data
      height Height of model input data
      format Type of model input data

    • Related interfaces

      ALGO_HSEG_GetInputAttr，ALGO_HSEG_GetOutputAttr

    4.2. HsegInit_t

    • Description

      Algorithm initialization parameter related structure.

    • Definition

      typedef struct
{
    char ipu_firmware_path[MAX_HSEG_STRLEN];  // ipu_firmware.bin path
    char model_path[MAX_HSEG_STRLEN];         // segmentation model path
    MI_BOOL create_device;                    // set false to create ipu device outside algo lib
    MI_BOOL destroy_device;                   // set false to destroy ipu device outside algo lib
    void *model_buffer;                       // set it when load model from memory
    MI_U32 model_buffer_len;                  // set it when load model from memory
} HsegInit_t;

    • Member

      Member Name Description
      ipu_firmware_path[IPU_MAX_LENGTH] ipu_firmware_path path
      model_path[MODEL_MAX_LENGTH] model path
      create_device Whether to create a device
      destroy_device Whether to destroy the device
      model_buffer Please set it to NULL when using model_path. Setting model_path to NULL and setting model_buffer and model_buffer_len can be used to initialize the algorithm from the model memory block.
      model_buffer_len Please set it to 0 when using model_path, set the model_buffer memory block length

    • Related interfaces

      ALGO_HSEG_InitHandle

    4.3. HsegInput_t

    • Description

      Algorithm input related structure.

    • Definition

      typedef struct
{
    void *p_vir_addr;
    MI_PHY phy_addr;
    MI_U32 buf_size;
    MI_U64 pts;
} HsegInput_t;

    • Member

      Member Name Description
      p_vir_addr Virtual address of input buffer
      phy_addr Physical address of input buffer
      buf_size Length of input buffer
      pts Timestamp of input buffer

    • Related interfaces

      ALGO_HSEG_Run

    5. Error Code

    Error Code Value Description
    E_ALGO_SUCCESS 0 Operation success
    E_ALGO_HANDLE_NULL 1 Algorithm handle is null
    E_ALGO_INVALID_PARAM 2 Invalid input parameter
    E_ALGO_DEVICE_FAULT 3 Hardware error
    E_ALGO_LOADMODEL_FAIL 4 Failed to load model
    E_ALGO_INIT_FAIL 5 Algorithm initialization failed
    E_ALGO_NOT_INIT 6 Algorithm initialization failed
    E_ALGO_INPUT_DATA_NULL 7 Algorithm input data is null
    E_ALGO_INVALID_INPUT_SIZE 8 Invalid algorithm input data dimension
    E_ALGO_INVALID_LICENSE 9 Invalid license
    E_ALGO_MEMORY_OUT 10 Out of memory
    E_ALGO_FILEIO_ERROR 11 File read and write operation error
    E_ALGO_INVALID_OUTPUT_SIZE 12 Invalid algorithm output data dimension