Zipformer

1 Overview

1.1 Background

Zipformer is the next-generation speech recognition (ASR) encoder promoted by the k2-fsa/icefall ecosystem. It balances efficient modeling with low-latency streaming inference, featuring fast inference, low memory footprint, and high accuracy. icefall provides a complete implementation of training and inference pipelines for both streaming/non-streaming scenarios under the RNNT-T architecture.

For details, please refer to the official documentation:

https://github.com/k2-fsa/icefall/blob/master/README.md

This example uses streaming inference + RNNT-T architecture for deployment. The model download URL is as follows: - kws_librispeech

Note: This model is trained on the LibriSpeech dataset. For other requirements, you may download other models from the icefall official website for conversion.

1.2 Usage Instructions

The Linux SDK-alkaid comes with pre-converted offline models and board-side examples by default. The related file paths are as follows:

• Board-side example program path

  Linux_SDK/sdk/verify/opendla/source/asr/zipformer
  

• Board-side offline model path

  Linux_SDK/project/board/${chip}/dla_file/ipu_open_models/asr/zipformer_encoder.img
  Linux_SDK/project/board/${chip}/dla_file/ipu_open_models/asr/zipformer_decoder.img
  Linux_SDK/project/board/${chip}/dla_file/ipu_open_models/asr/zipformer_joiner.img
  

• Board-side test audio path

  Linux_SDK/sdk/verify/opendla/source/resource/zipformer/1188-133604-0000.wav
  

• Board-side test dictionary path

  Linux_SDK/sdk/verify/opendla/source/resource/zipformer/tokens.txt
  Linux_SDK/sdk/verify/opendla/source/resource/zipformer/tokenizer_config.json
  Linux_SDK/sdk/verify/opendla/source/resource/zipformer/preprocessor_config.json
  

If you do not need to convert the model, you can skip directly to Chapter 3.

2 Model Conversion

2.1 ONNX Model Conversion

• Python Environment Setup

  $conda create -n zipformer python==3.11
  $conda activate zipformer
  $git clone https://github.com/k2-fsa/icefall.git
  

Note: The Python environment setup provided here is only a reference example. For the actual setup process, please refer to the official source code tutorial:

https://k2-fsa.github.io/icefall/installation/index.html

• Model Export

  • Install dependencies

    $pip install onnx -i https://pypi.tuna.tsinghua.edu.cn/simple
    

  • The downloaded model contains

    • pretrianed.pt
    • data/lang_bpe_500/tokens.txt

  • Run the model conversion script included with icefall. Ensure the icefall environment is configured correctly.

    $cd icefall/egs/librispeech/ASR/
    $export PYTHONPATH=$PYTHONPATH:$(pwd)/../../../
    $python ./zipformer/export-onnx-streaming-new.py \
        --exp-dir icefall-asr-librispeech-streaming-zipformer-2023-05-17/exp \
        --tokens icefall-asr-librispeech-streaming-zipformer-2023-05-17/data/lang_bpe_500/tokens.txt \
        --use-averaged-model 0 \
        --epoch 30 \
        --avg 1 \
        --chunk-size 16 \
        --left-context-frames 128 \
        --decoder-dim 512 \
        --joiner-dim 512 \
        --num-encoder-layers 2,2,3,4,3,2 \
        --feedforward-dim 512,768,1024,1536,1024,768 \
        --encoder-dim 192,256,384,512,384,256 \
        --encoder-unmasked-dim 192,192,256,256,256,192 \
        --causal 1
    

  Here, exp is the model folder downloaded from the official website, which can be renamed as desired. After successful model conversion, the output log will print: Export decoder to path/decoder-epoch-99-avg-1-chunk-16-left-128.onnx Export encoder to path/encoder-epoch-99-avg-1-chunk-16-left-128.onnx Export jointer to path/jointer-epoch-99-avg-1-chunk-16-left-128.onnx

  At this point, the ONNX model can be exported, but it cannot yet be deployed to our platform. Some operators need to be modified.

• Model Modification

  A patch for code modification is provided in the model conversion code. You can apply it directly by running the following commands:

  $cp 0001-modify-export-onnx-streaming.py-for-fixed-inference.patch ./
  $git am 0001-modify-export-onnx-streaming.py-for-fixed-inference.patch
  

  After modification, run the Model Export step again to generate the deployable zipformer.onnx model. - encoder_optimized-epoch-30-avg-1-chunk-16-left-128.onnx - decoder-epoch-30-avg-1-chunk-16-left-128.onnx - joiner-epoch-30-avg-1-chunk-16-left-128.onnx

2.2 Offline Model Conversion

2.2.1 Offline Model Conversion Workflow

Note: 1) OpenDLAModel corresponds to the smodel file extracted from the image-dev_model_convert.tar archive. 2) Unlike other algorithms, this algorithm does not need to enter Docker for conversion.

• Copy the ONNX models to the conversion code directory

  $cp path/encoder_optimized-epoch-30-avg-1-chunk-16-left-128.onnx OpenDLAModel/asr/zipformer/onnx
  $cp path/decoder-epoch-30-avg-1-chunk-16-left-128.onnx OpenDLAModel/asr/zipformer/onnx
  $cp path/joiner-epoch-30-avg-1-chunk-16-left-128.onnx OpenDLAModel/asr/zipformer/onnx
  

• Conversion command

  # Enter the OpenDLAModel directory
  $cd /work/SGS_XXX/OpenDLAModel
  $bash convert.sh -a asr/zipformer -c config/asr_zipformer.cfg -p SGS_IPU_Toolchain(absolute path) -s false
  

• Final generated model path

  output/${chip}_${time}/zipformer_encoder.img
  output/${chip}_${time}/zipformer_decoder.img
  output/${chip}_${time}/zipformer_joiner.img
  

2.2.2 Pre-processing & Post-processing Description

-   Pre-processing

    The input to this model is fbank features. The fbank extraction workflow: for 16kHz raw audio, perform pre-emphasis, framing (frame length 25ms, frame shift 10ms), apply a Hamming window, and compute the amplitude spectrum via FFT. Then pass it through an 80-dimensional Mel filterbank and take the logarithm to obtain the 80-dimensional log-Mel filterbank features, which serve as the input to the encoder.

-   Post-processing

    This model supports multiple post-processing (decoding) methods. Two of them are introduced here: prefix_beam_search and greedy search.

    -   Greedy Search
        At each step, the token corresponding to the maximum logit value is directly selected as output. The path is unique, the speed is fast, and the computation is small.

    -   Prefix Beam Search
        -   Maintains multiple candidate sequences (prefixes). At each time step, all candidates are expanded with tokens, and the top-N (beam size, corresponding to the `-n` parameter) paths are retained based on cumulative probability.
        -   Scores of identical prefixes are merged to avoid duplicate paths. Finally, the sequence with the highest cumulative score is output.

        This decoding method offers higher accuracy and robustness; the trade-off is greater computation and memory overhead, and slower speed.

2.2.3 Key Script Parameter Description

• encoder_config.ini

  [INPUT_CONFIG]
  inputs=x,encoder_states_0,encoder_states_1,encoder_states_2,encoder_states_3,encoder_states_4,encoder_states_5,embed_states,processed_lens;
  input_formats=RAWDATA_F32_NHWC,RAWDATA_F32_NHWC,RAWDATA_F32_NHWC,RAWDATA_F32_NHWC,RAWDATA_F32_NHWC,RAWDATA_F32_NHWC,RAWDATA_F32_NHWC,RAWDATA_F32_NHWC,RAWDATA_S16_NHWC;
  quantizations=TRUE,TRUE,TRUE,TRUE,TRUE,TRUE,TRUE,TRUE,TRUE;
  
  [OUTPUT_CONFIG]
  outputs=encoder_out,new_encoder_states_0,new_encoder_states_1,new_encoder_states_2,new_encoder_states_3,new_encoder_states_4,new_encoder_states_5,new_embed_states,new_processed_lens;
  dequantizations=TRUE,TRUE,TRUE,TRUE,TRUE,TRUE,TRUE,TRUE,TRUE;
  
  [OPTIMIZE_CONFIG]
  optimize_layernorm_precision=TRUE;
  
  [CONV_CONFIG]
  input_format=ALL_FP16;              # For int16 quantization, chips after mhera all use ALL_FP16
  

• decoder_config.ini

  [INPUT_CONFIG]
  inputs=y;
  input_formats=RAWDATA_S16_NHWC;
  quantizations=TRUE;
  
  [OUTPUT_CONFIG]
  outputs=decoder_out;
  dequantizations=TRUE;
  
  [OPTIMIZE_CONFIG]
  optimize_layernorm_precision=TRUE;
  
  [CONV_CONFIG]
  input_format=ALL_FP16;              # For int16 quantization, chips after mhera all use ALL_FP16; otherwise, use ALL_INT16
  

• joiner_config.ini

  [INPUT_CONFIG]
  inputs=encoder_out,decoder_out;
  input_formats=RAWDATA_F32_NHWC,RAWDATA_F32_NHWC;
  quantizations=TRUE,TRUE;
  
  [OUTPUT_CONFIG]
  outputs=logit;
  dequantizations=TRUE;
  
  [OPTIMIZE_CONFIG]
  optimize_layernorm_precision=TRUE;
  
  [CONV_CONFIG]
  input_format=ALL_FP16;              # For int16 quantization, chips after mhera all use ALL_FP16
  

2.3 Model Simulation

• Obtain float/fixed/offline model outputs

  $bash convert.sh -a asr/zipformer -c config/asr_zipformer.cfg -p SGS_IPU_Toolchain(absolute path) -s true
  

  After executing the above command, inference will be performed on float, fixed, and offline models according to the input parameters. You can add prints in the code to obtain the model output tensors.

• Model accuracy comparison

  python asr/zipformer/sim_onnx_zipformer.py \
      --encoder output/${chip}_${time}/zipformer_decoder.img \
      --decoder output/${chip}_${time}/zipformer_decoder.img \
      --joiner output/${chip}_${time}/zipformer_joiner \
      --tokens asr/zipformer/tokens.txt \
      --wav asr/zipformer/test_wavs/1089-134686-0001.wav \
      --sample-rate 16000 \
      --feature-dim 80 \
      --optimized
  

  After executing the above command, the inference results of the ONNX model will be output. You can add prints in between to obtain the model output tensors.

3 Board-side Deployment

3.1 Program Compilation

Before compiling the example program, you need to select a defconfig for SDK full-package compilation based on the board (nand/nor/emmc, DDR model, etc.). For details, please refer to the alkaid SDK sigdoc "Development Environment Setup" document.

• Compile the board-side zipformer example.

  $cd sdk/verify/opendla
  $make clean && make source/asr/zipformer -j8
  

• Final generated executable file path

  sdk/verify/opendla/out/${AARCH}/app/prog_asr_zipformer
  

3.2 Runtime Files

When running the program, you need to copy the following files to the board first:

• prog_asr_zipformer
• 1188-133604-0000.wav
• tokens.txt
• tokenizer_config.json
• preprocessor_config.json
• zipformer_decoder.img
• zipformer_encoder.img
• zipformer_joiner.img

3.3 Running Instructions

• Usage: ./prog_asr_zipformer -d model -w wav -b use_beam_search -n beam_size (executable usage command)

• Required Input:

  • d: model path
  • e: encoder model name
  • D: decoder model name
  • j: joiner model name
  • w: audio path
  • b: whether to use prefix_beam_search
  • n: number of beams used

• Typical Output:

  ./prog_asr_zipformer -d models/zipformer/ -e zipformer_encoder.img -D zipformer_decoder.img -j zipformer_joiner.img  -w models/zipformer/1188-133604-0000.wav -b true -n 5
      Using models - encoder: zipformer_encoder.img, decoder: zipformer_decoder.img, jointer: zipformer_joiner.img
      load dict from models/zipformer/tokens.txt
      Loading Zipformer model from: models/zipformer/
      init zipformer
      init decoder
      init encoder
      init jointer
      Model loaded successfully.
  
      Opened file: models/zipformer/1188-133604-0000.wav
      feats shape = [85680]
      zipformer response is :  YOU WILL FIND ME CONTINUALLY SPEAKING OF FOUR MEN TITIAN HOBINE TURNER AND TINKORAT IN ALMOST THE SAME TERMS
  

---