Feature-Distance

1 Overview

1.1 Background

Feature_distance works by calculating the similarity between images to complete the task of image feature comparison, accelerating the computation of feature comparison using IPU. The key idea is that the feature distances between similar image pairs are closer, while the feature distances between dissimilar images are further apart.

1.2 Usage Instructions

The Linux SDK-alkaid comes with default board-side examples, and the relevant file paths are as follows:

• Board-side example program path:

  Linux_SDK/sdk/verify/opendla/source/feature_distance
  

  • Board-side test image paths:

    Linux_SDK/sdk/verify/opendla/source/resource/feature_library/feat_gallery.bin Linux_SDK/sdk/verify/opendla/source/resource/feature_library/name_gallery.list Linux_SDK/sdk/verify/opendla/source/resource/feature_library/feat_query.bin Linux_SDK/sdk/verify/opendla/source/resource/feature_library/name_query.list

If the user does not need to convert the model, they can directly jump to section 3.

2 Model Conversion

2.1 Generating ONNX Model

• The model used for accelerating image similarity calculation with IPU does not require loading weights, so no model weight files are necessary. The following code can be executed to directly convert the required ONNX modelcos.onnx:

  Note: Modify gallery_size according to the size of the image library (e.g., if there are 256 images in the library, then gallery_size = 256).

  import torch
  import numpy as np
  import torch.nn as nn
  import torch.nn.functional as F
  gallery_size = 256
  class Net(nn.Module):
      def __init__(self):
          super(Net, self).__init__()
      def forward(self, x, y):
          dot = torch.matmul(x, y.permute(1, 0))
          norm = torch.sqrt(torch.matmul(x, x.permute(1, 0))).repeat(1, gallery_size) * torch.norm(y, dim=1).view(1, -1)
          return dot, norm
  torch_model = Net()
  x = torch.randn(1, 512, requires_grad=False)
  y = torch.randn(gallery_size, 512, requires_grad=False)
  torch.onnx.export(
      torch_model,
      (x, y),
      "cos.onnx",
      opset_version=12,
      input_names=['input0', 'input1'],
      output_names=['output0', 'output1']
  )
  

2.2 Offline Model Conversion

Note: 1) OpenDLAModel corresponds to the smodel file in the compressed package image-dev_model_convert.tar after extraction. 2) The conversion command needs to be run in the Docker environment; please first set up the SGS Docker environment according to the Docker development environment tutorial.

• Copy the ONNX model to the conversion code directory:

  $cp ./cos.onnx OpenDLAModel/feature_distance/onnx
  

• Generate quantization data cos.txt based on the number of images in the library (gallery_size): this model is a feature similarity calculation model, and the quantization data is derived from the output features of the feature extraction model; shape=(1, gallery_size, 512).

• Conversion command:

  $cd IPU_SDK_Release/docker
  $bash run_docker.sh
  # Enter the OpenDLAModel directory in the Docker environment
  $cd /work/SGS_XXX/OpenDLAModel
  $bash convert.sh -a feature_distance -c config/feature_distance.cfg -p SGS_IPU_Toolchain (absolute path) -s false
  

• Final generated model address:

  output/${chip}_${time}/cos.img
  

2.2.3 Key Script Parameter Explanation

-   config.ini

        [INPUT_CONFIG]
        inputs = input0,input1;                                 # ONNX input node names, separate multiple names with commas
        input_formats = RAWDATA_S16_NHWC;RAWDATA_S16_NHWC       # Board-side input format, can choose BGRA or YUV_NV12 as appropriate
        quantizations = TRUE,TRUE;                              # Enable input quantization, do not modify
        [OUTPUT_CONFIG]
        outputs = output0,output1;                              # ONNX output node names, separate multiple names with commas
        dequantizations = TRUE,TRUE;                            # Whether to enable dequantization, fill in based on actual needs; recommended to set to TRUE. Set to False for int16 output; set to True for float32 output

-   feature_distance.cfg

        [FEATURE]
        CHIP_LIST=pcupid                                            # Platform name, must match the board platform, otherwise the model will not run
        Model_LIST=cos                                              # Input ONNX model name
        INPUT_SIZE_LIST=256                                         # Model input length (size of the image library)
        INPUT_INI_LIST=config.ini                                   # Configuration file
        CLASS_NUM_LIST=Unknown                                      # Simply fill in Unknown
        SAVE_NAME_LIST=cos.img                                      # Output model name
        QUANT_DATA_PATH=./cos.txt                                   # Quantization data path: data is the features outputted by the feature extraction model

3 Board Deployment

3.1 Program Compilation

Before compiling the example program, you need to select the deconfig based on the board (e.g., nand/nor/emmc, DDR model, etc.) for a full SDK compilation. For specific instructions, refer to the Alkaid SDK sigdoc "Development Environment Setup" documentation.

• Compile the board-side clip example.

  $cd sdk/verify/opendla
  $make clean && make source/feature_distance -j8
  

• Final address of the generated executable file:

  sdk/verify/opendla/out/${AARCH}/app/prog_feature_distance

3.2 Run Files

1. Before running the program, you need to copy the following files to the board:

   • Run script:
     • prog_feature_distance
   • Offline model:
     • cos.img

2. Prepare the image files:

   • Image library:
     • feat_gallery.bin
     • name_gallery.list
   • Query images:
     • feat_query.bin
     • name_query.list

3.3 Running Instructions

• Usage: ./prog_feature_distance modelPath galleryBin galleryNames queryBin queryNames topk (Command to execute the file)

• Required Input:

  • modelPath: Path to the offline model
  • galleryBin: Image library (bin file)
  • galleryNames: Names of all images in the image query library (list file)
  • queryBin: Query image (bin file)
  • queryNames: Names of query images (list file)
  • topk: Output the top k prediction values

• Typical output:

  /prog_feature_distance models/cos.img resource/feature_library/feat_gallery.bin resource/feature_library/name_gallery.list resource/feature_library/feat_query.bin resource/feature_library/name_query.list 3
  
  client [812] connected, module:ipu
  query_0001, the image path is 044, the score is 0.570715
  query_0001, the image path is 194, the score is 0.299766
  query_0001, the image path is 052, the score is 0.297114
  
  query_0002, the image path is 224, the score is 0.510769
  query_0002, the image path is 294, the score is 0.301047
  query_0002, the image path is 266, the score is 0.295059
  
  query_0003, the image path is 164, the score is 0.549499
  query_0003, the image path is 010, the score is 0.317435
  query_0003, the image path is 052, the score is 0.303460
  
  query_0004, the image path is 298, the score is 0.634573
  query_0004, the image path is 044, the score is 0.230089
  query_0004, the image path is 258, the score is 0.225121
  
  collect channel0's resource
  ------shutdown IPU0------
  client [812] disconnected, module:ipu
  

---