Face Recognition Algorithm¶
REVISION HISTORY¶
| Revision No. | Description |
Date |
|---|---|---|
| 100 | 04/27/2023 | |
| 101 | 07/04/2023 | |
| 102 | 05/20/2024 |
1. Algorithm Description¶
Face recognition mainly distinguishes faces and determines whether the captured face is from a whitelisted person. The entire algorithm includes face detection, face attribute recognition, face expression recognition, facial key points, face filtering, face tracking, face alignment, feature extraction, and face comparison.
Among them, face attributes can determine five attributes of the face: glasses, gender, mask, mustache, and age; face expressions can identify seven expressions, which are normal, happy, sad, surprised, scared, disgusted, and angry.
-
Algorithm Accuracy
ALGO Threshold 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 FR TAR% 99.4 98.9 98.5 98.1 97.6 96.9 96.4 95.2 93.9 93.4 92.1 90.9 89.1 86.7 85.4 84.2 FR FAR% 2.3 1.8 1 0.7 0.4 0.1 0 0 0 0 0 0 0 0 0 0 -
Model Introduction
model Function Resolution(w*h) Input Format fr_det_y24s.img Face Detection 480*288 yuvsp420_nv12 fr_det_y36s.img Face Detection 640*352 yuvsp420_nv12 fr_det_y48s.img Face Detection 800*480 yuvsp420_nv12 fr_feature_as.img Face Feature Extraction (Small) 112*112 BGRA8888 fr_feature_am.img Face Feature Extraction (Medium) 112*112 BGRA8888 fr_feature_all.img Face Feature Extraction (Large) 112*112 BGRA8888 fr_angle_y66.img Face Angle Estimation 64*64 yuvsp420_nv12 fr_cos256.img Face Feature Comparison Model 256*512 INT16 far101_224y_emo.img Face Emotion Recognition Model 224*224 yuvsp420_nv12 far101_224y_emo_s45.img Face Emotion Recognition Model (Small) 224*224 yuvsp420_nv12 far103_224y.img Face Attribute Recognition Model 224*224 yuvsp420_nv12
2. Interface Calling Process¶
Face recognition mainly involves two processes: one is the face registration process, which creates the face whitelist (database). The other is the face recognition process, which identifies the captured face to determine if it is from a whitelisted person.
2.1. Registration Process¶
The interface call for the registration process: ALGO_FR_Init → ALGO_FR_CreateHandle → ALGO_FR_Detect → ALGO_FR_Align → ALGO_FR_FeatureExtract. Finally, the extracted features are saved to become the database.
2.2. Recognition Process¶
The interface call for the recognition process is: ALGO_FR_Init → ALGO_FR_CreateHandle → ALGO_FR_Detect → ALGO_FR_Align → ALGO_FR_FeatureExtract → ALGO_FR_FeatureCompare, comparing the features from the captured image with the features in the database to determine whether they belong to the same person.
2.3. Face Attribute Recognition Process¶
The interface call for face attribute recognition is: ALGO_FR_Init → ALGO_FR_CreateHandle → ALGO_FR_Detect → ALGO_FR_Attr, using the captured face image for face attribute recognition.
Note: The face image sent to this interface after cropping needs to be expanded by a certain range, the expansion width and height are: 1.25*bbox_w, 1.25*bbox_h
2.4. Multi-Detector Recognition Process¶
First Detector:
ALGO_FR_Init (Only needs to be done once) --> ALGO_FR_CreateHandle (Create handle using the detection model path specified in InitFrParam_t) --> ALGO_FR_Detect --> ALGO_FR_Align --> ALGO_FR_FeatureExtract --> ALGO_FR_FeatureCompare
Subsequent Detectors:
ALGO_FR_AddHandle (Create a new handle by specifying the path to the new detection model) --> ALGO_FR_Detect --> ALGO_FR_Align --> ALGO_FR_FeatureExtract --> ALGO_FR_FeatureCompare
Note: The largest detector should be used as the first detector!
3. Function Module API¶
| API Name | Function |
|---|---|
| ALGO_FR_Init | SDK Initialization |
| ALGO_FR_CreateHandle | Create Handle |
| ALGO_FR_Detect | Face Detection |
| ALGO_FR_Attr | Face Attribute Recognition |
| ALGO_FR_Align | Face Alignment |
| ALGO_FR_FeatureExtract | Feature Extraction |
| ALGO_FR_FeatureCompare | Feature Comparison |
| ALGO_FR_ReleaseHandle | Release Handle |
| ALGO_FR_Cleanup | Release Entire SDK Environment |
| ALGO_FR_HandleResetParam | Reset Algorithm Configurable Parameters |
| ALGO_FR_FaceQuanlity | Perform Face Angle Estimation and Quality Filtering |
| ALGO_FR_BatchFeatureCompare | Perform Batch Face Feature Comparison (requires a copy of the comparison features) |
| ALGO_FR_BatchFeatureCompareV2 | Perform Batch Face Feature Comparison (no need to copy comparison features) |
| ALGO_FR_AddHandle | Create Handle for New Detection Model |
3.1. ALGO_FR_Init¶
-
Function
Initialize the SDK, only needs to be initialized once.
-
Syntax
MI_S32 ALGO_FR_Init(InitFrParam_t initParam); -
Parameters
Parameter Name Description Input/Output initParam Initialization Parameters Input -
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
-
Example
{ int ret=0; InitFrParam_t initParam; memcpy(initParam.ipu_firware_bin,"/config/dla/ipu_firmware.bin", IPU_MAX_LENGTH); memcpy(initParam.det_model_path, "./models/fr_det_y24s.img", MODEL_MAX_LENGTH); memcpy(initParam.attr_model_path, "./models/far101_224y_s45.img", MODEL_MAX_LENGTH); memcpy(initParam.emo_model_path, "./models/far101_224y_emo_s45.img", MODEL_MAX_LENGTH); memcpy(initParam.feature_model_path, "./models/fr_feature_as.img", MODEL_MAX_LENGTH); memcpy(initParam.cos_model_path, "./models/i6c/fr_cos256.img", MODEL_MAX_LENGTH); initParam.box_min_size = 5; initParam.det_thredhold = 0.5; initParam.attr_thredhold = 0.3; initParam.eye_distance = 20; initParam.filter_angle_ratio = 0.2; initParam.fr_mode = FR_MODE_FACE_DET | FR_MODE_FACE_DET_RECOG | FR_MODE_FACE_ATTR | FR_MODE_FACE_EMOTION; ret= ALGO_FR_Init(initParam); }
3.2. ALGO_FR_CreateHandle¶
-
Function
Create a handle; one thread creates one handle.
-
Syntax
MI_S32 ALGO_FR_CreateHandle(MI_S64* detectorId) -
Parameters
Parameter Name Description Input/Output detectorId Handle Pointer Input -
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
3.3. ALGO_FR_Detect¶
-
Function
Face Detection.
-
Syntax
MI_S32 ALGO_FR_Detect(MI_S64 detectId, AlgoFrInputInfo_t *stBufInfo, MI_S32 width, MI_S32 height, ParamDet_t* params, DetectBox_t** detectout, MI_S32* facecount) -
Parameters
Parameter Name Description Input/Output detectorId Handle Input stBufInfo Data Pointer (yuv420_nv12) (Size 480x288) Input width Width of the input raw image Input height Height of the input raw image Input params Configuration parameters for detection Input detectOut Output for detected face box Output faceCount Number of detected face boxes Output -
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
3.4. ALGO_FR_Attr¶
-
Function
Face Attribute Recognition.
-
Syntax
MI_S32 ALGO_FR_Attr(MI_S64 detectId, AlgoFrInputInfo_t *stBufInfo, FaceAttr_t *results); -
Parameters
Parameter Name Description Input/Output detectorId Handle Input stBufInfo Data Pointer (argb8888) (Size 224x224) Input results Face Attribute Results Output -
Note
The face buffer sent to this interface needs to be expanded by a certain range, the expansion width and height are: +0.25*bbox_w, +0.25*bbox_h
-
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
3.5. ALGO_FR_Align¶
-
Function
Face Alignment.
-
Syntax
MI_S32 ALGO_FR_Align(MI_U8* imageData, MI_S32 width, MI_S32 height, MI_S32 type, DetectBox_t detectOut, MI_U8* outData); -
Parameters
Parameter Name Description Input/Output imageData Original Image Data Pointer Input width Width of the original image Input height Height of the original image Input type Image type, 1 for yuv420_nv12, 0 for argb8888 Input detectOut Face Box Input outData Output aligned 112x112 ARGB888 Data, cannot be empty Output -
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
3.6. ALGO_FR_FeatureExtract¶
-
Function
Face Feature Extraction.
-
Syntax
MI_S32 ALGO_FR_FeatureExtract(MI_S64 detectId, MI_U8* imageData, MI_S16* featureOut); -
Parameters
Parameter Name Description Input/Output detectId Handle imageData Input image data output from Align interface (112x112) Input featureOut Face's feature data, 512-length MI_S16 feature Output -
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
3.7. ALGO_FR_FeatureCompare¶
-
Function
Feature Comparison.
-
Syntax
MI_S32 ALGO_FR_FeatureCompare(MI_S16* feature1, MI_S16* feature2, MI_S32 length, MI_FLOAT* similarity); -
Parameters
Parameter Name Description Input/Output feature1 Features of the first face Input feature2 Features of the second face Input length Feature length, currently 512 Input similarity Similarity between the two faces Output -
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
3.8. ALGO_FR_ReleaseHandle¶
-
Function
Release Handle.
-
Syntax
MI_S32 ALGO_FR_ReleaseHandle(MI_S64 detectorId); -
Parameters
Parameter Name Description Input/Output detectorId Handle Input -
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
3.9 ALGO_FR_Cleanup¶
-
Function
Release the entire SDK environment.
-
Syntax
MI_S32 ALGO_FR_Cleanup(); -
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
3.10 ALGO_FR_HandleResetParam¶
-
Function
Reset the algorithm's configurable parameters
-
Syntax
MI_S32 ALGO_FR_HandleResetParam(MI_S64 detectorID, AlgoFrFilterParam_t param); -
Parameters
Parameter Name Description Input/Output detectorID Handle Input param Algorithm parameters, see AlgoFrFilterParam_t Input -
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
3.11 ALGO_FR_FaceQuanlity¶
-
Function
Perform face angle estimation and quality filtering.
-
Syntax
MI_S32 ALGO_FR_FaceQuanlity(MI_S64 detectId, AlgoFrInputInfo_t* stBufInfo, DetectBox_t* box, AlgoFaceQuanlityParam_t param, FaceQuanlityAngle_t* angle); -
Parameters
Parameter Name Description Input/Output detectorID Handle Input stBufInfo Frame data pointer (yuv420_nv12) (Size 64x64) Input box Pointer for output face detection box Input/Output param Face quality parameters, see AlgoFaceQuanlityParam_t Input angle Output result of face angle, see FaceQuanlityAngle_t Output -
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
3.12 ALGO_FR_BatchFeatureCompare¶
-
Function
Perform batch face feature comparison (requires a copy of the comparison features)
-
Syntax
MI_S32 ALGO_FR_BatchFeatureCompare(MI_S64 detectId, MI_S16* feature1, MI_S16* feature2, MI_S32 batch, MI_FLOAT* similarity); -
Parameters
Parameter Name Description Input/Output detectorID Handle Input feature1 Target feature for comparison (1) Input feature2 Source features for comparison (batch) Input batch Number of source features for comparison Input similarity Similarity output of feature comparison Output -
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
3.13 ALGO_FR_BatchFeatureCompareV2¶
-
Function
Perform batch face feature comparison (no need to copy comparison features)
-
Syntax
MI_S32 ALGO_FR_BatchFeatureCompareV2(MI_S64 detectId, AlgoFrInputInfo_t* feature1, AlgoFrInputInfo_t* batchFeature, MI_S32 batch, MI_FLOAT* similarity); -
Parameters
Parameter Name Description Input/Output detectorID Handle Input feature1 Target feature for comparison (1), passed in MMA buffer format Input feature2 Source features for comparison (batch), passed in MMA buffer format Input batch Number of source features for comparison Input similarity Similarity output of feature comparison Output -
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
3.14. ALGO_FR_AddHandle¶
-
Function
Add a handle for the new detector, allowing specification of the detection model path so that each handle can use a different face detector.
-
Syntax
MI_S32 ALGO_FR_AddHandle(MI_S64* detectorID, const MI_U8* det_model_path); -
Parameters
Parameter Name Description Input/Output detectorId Handle Pointer Input det_model_path Path to detection model Input -
Return Value
Return Value Description 0 Success Others Failure (See Error Code) -
Dependencies
Header File: algo_fr_api.h
Library File: libsstaralgo_fr.so, libsstaralgo_fr.a
4. Data Types¶
| Data Type | Definition |
|---|---|
| ALGO_FrMode_e | Algorithm Mode. |
| InitFrParam_t | Initialization Parameters for Face Algorithm. |
| ParamDet_t | Algorithm Detection Input Parameters for Selecting Face Detection Mode. |
| AlgoFrInputInfo_t | Algorithm Input Data Structure. |
| FrPoint_t | Defines Key Point Coordinates. |
| DetectBox_t | Detection Box Information. |
| FaceAttr_t | Face Attribute Category Information. |
| AlgoFrFilterParam_t | Face Filtering Parameter Structure |
| FaceQuanlityAngle_t | Face Angle Output Structure |
| AlgoFaceQuanlityParam_t | Face Quality Parameter Structure |
4.1. ALGO_FrMode_e¶
-
Description
Algorithm Mode.
-
Definition
typedef enum { FR_MODE_FACE_DET, FR_MODE_FACE_DET_RECOG, FR_MODE_FACE_DET_ATTR, FR_MODE_FACE_EMOTION, } ALGO_FrMode_e; -
Members
Member Name Description FR_MODE_FACE_DET Face Detection FR_MODE_FACE_RECOG Face Recognition FR_MODE_FACE_ATTR Face Attribute Recognition FR_MODE_FACE_EMOTION Face Emotion Recognition -
Related Data Types and Interfaces
4.2. InitFrParam_t¶
-
Description
Initialization Parameters for Face Algorithm.
-
Definition
typedef struct InitFrParam_ { MI_U8 ipu_firware_bin[IPU_MAX_LENGTH]; MI_U8 det_model_path[MODEL_MAX_LENGTH]; MI_U8 feature_model_path[MODEL_MAX_LENGTH]; MI_U8 cos_model_path[MODEL_MAX_LENGTH]; MI_U8 attr_model_path[MODEL_MAX_LENGTH]; MI_U8 emo_model_path[MODEL_MAX_LENGTH]; MI_FLOAT det_thredhold; MI_FLOAT attr_thredhold; MI_S32 box_min_size; MI_FLOAT filter_angle_ratio; MI_FLOAT eye_distance; MI_U8 fr_mode; MI_BOOL had_create_device; } InitFrParam_t; -
Members
Member Name Description ipu_firware_bin Path of the Firmware_bin det_model_path Path of the Detection Model feature_model_path Path of the Feature Extraction Model cos_model_path Path of the Similarity Model attr_model_path Path of the Face Attribute Recognition Model emo_model_path Path of the Face Emotion Recognition Model det_thredhold Threshold for detection, recommended to set to 0.5 attr_thredhold Threshold for face attribute recognition, recommended to set to 0.5 box_min_size Minimum detection box size, recommended to set to 10 filter_angle_ratio Ratio for filtering face angle, recommended to set to 0.23 eye_distance Distance between eyes, recommended to set to 30 fr_mode Algorithm Mode, defaults to face recognition mode had_create_device Whether a device has been created for multiple handles -
Related Interfaces
4.3. ParamDet_t¶
-
Description
Algorithm Detection Input Parameters for Selecting Face Detection Mode.
-
Definition
typedef struct ParamDet_ { MI_S32 datatype; }ParamDet_t; -
Members
Member Name Description datatype 0 indicates detect the maximum face; 1 indicates detect and take all faces; 2 indicates video input, take the maximum face and add tracking id; 3 indicates video input, take all faces and add tracking id -
Related Data Types and Interfaces
4.4. AlgoFrInputInfo_t¶
-
Description
Algorithm Input Data Structure.
-
Definition
typedef struct AlgoFrInputInfo_ { void* pt_tensor_data; MI_PHY phy_tensor_addr; MI_U32 bufsize; MI_S64 pts; } AlgoFrInputInfo_t; -
Members
Member Name Description pt_tensor_data Virtual address of input data phy_tensor_addr Physical address of input data bufsize Size of input data pts Timestamp of input data -
Related Data Types and Interfaces
4.5. FrPoint_t¶
-
Description
Defines Key Point Coordinates.
-
Definition
typedef struct FrPoint_ { MI_FLOAT x; MI_FLOAT y; }FrPoint_t; -
Members
Member Name Description x x-coordinate of the key point y y-coordinate of the key point -
Related Data Types and Interfaces
4.6. DetectBox_t¶
-
Description
Detection box information.
-
Definition
typedef struct DetectBox_ { MI_FLOAT x1; MI_FLOAT y1; MI_FLOAT x2; MI_FLOAT y2; MI_FLOAT score; FrPoint_t landmark[FR_POINT_LEN]; MI_S32 face_id; MI_S32 befiltered; }DetectBox_t; -
Members
Member Name Description x1 x-coordinate of the top-left point of the face box y1 y-coordinate of the top-left point of the face box x2 x-coordinate of the bottom-right point of the face box y2 y-coordinate of the bottom-right point of the face box score Confidence of the face landmark Five key points of the face face_id Tracking id of the face befiltered Whether the face is filtered, True if filtered -
Related Data Types and Interfaces
4.7. FaceAttr_t¶
-
Description
Attribute category information.
-
Definition
typedef struct FaceAttr_t_ { MI_BOOL eye_glass; MI_BOOL male; MI_BOOL mask; MI_S32 age; MI_BOOL mustache; MI_S32 emotion; }FaceAttr_t; -
Members
Member Name Description eye_glass Glasses male Male mask Mask age Age mustache Mustache emotion Emotion -
Related Data Types and Interfaces
4.8. AlgoFrFilterParam_t¶
-
Description
Face filtering parameter structure.
-
Definition
typedef struct AlgoFrFilterParam_ { MI_S32 box_min_size; MI_FLOAT filter_angle_ratio; MI_FLOAT eye_distance; MI_FLOAT det_thredhold; MI_FLOAT attr_thredhold; MI_FLOAT motion_sensitive; //[0.0-1.0] }AlgoFrFilterParam_t; -
Members
Member Name Description box_min_size Minimum face detection box size, default 20 (pixels) filter_angle_ratio Head tilt/pitch angle requirement (0.0~1.0), larger values require more straight angles, default is 0.25 eye_distance Minimum interpupillary distance, default is 20 (pixels) det_thredhold Minimum face detection score, default is 0.3 attr_thredhold Score threshold for face attribute recognition motion_sensitive Required motion magnitude of the face (0.0~1.0), larger values require less motion of the face -
Related Data Types and Interfaces
4.9. FaceQuanlityAngle_t¶
-
Description
Face angle output structure.
-
Definition
typedef struct FaceQuanlityAngle_ { MI_FLOAT pitch; MI_FLOAT yaw; MI_FLOAT roll; MI_FLOAT quanlity; }FaceQuanlityAngle_t; -
Members
Member Name Description pitch Face pitch angle yaw Face yaw angle roll Face roll angle quanlity Comprehensive score of face angle (value range 0.0~1.0) -
Related Data Types and Interfaces
4.10. AlgoFaceQuanlityParam_t¶
-
Description
Face quality parameter structure.
-
Definition
typedef struct AlgoFaceQuanlityParam_ { MI_FLOAT thredhold; }AlgoFaceQuanlityParam_t; -
Members
Member Name Description thredhold Threshold for face quality score (value range 0.0~1.0) -
Related Data Types and Interfaces
5. Error Codes ¶
| Error Code | Value | Description |
|---|---|---|
| E_ALGO_SUCCESS | 0 | Operation Successful |
| E_ALGO_HANDLE_NULL | 1 | Algorithm Handle is Null |
| E_ALGO_INVALID_PARAM | 2 | Invalid Input Parameter |
| E_ALGO_DEVICE_FAULT | 3 | Hardware Error |
| E_ALGO_LOADMODEL_FAIL | 4 | Model Loading Failed |
| E_ALGO_INIT_FAIL | 5 | Algorithm Initialization Failed |
| E_ALGO_NOT_INIT | 6 | Algorithm Not Initialized |
| E_ALGO_INPUT_DATA_NULL | 7 | Algorithm Input Data is Null |
| E_ALGO_INVALID_INPUT_SIZE | 8 | Invalid Algorithm Input Data Dimension |
| E_ALGO_INVALID_LICENSE | 9 | Invalid License |
| E_ALGO_MEMORY_OUT | 10 | Memory Insufficient |
| E_ALGO_FILEIO_ERROR | 11 | File Read/Write Operation Error |
| E_ALGO_INVALID_OUTPUT_SIZE | 12 | Invalid Algorithm Output Data Dimension |