Skip to content

fastvlm

1 Overview

1.1 Background

FastVLM is a multimodal large model open-sourced by the Apple team. The open source project address is:

https://github.com/apple/ml-fastvlm

The model download address is:

https://huggingface.co/lhuanyu/fastvlm_1.5b

2 Model Conversion

2.1 Offline Model Conversion

For model conversion, please refer to the detailed large model conversion process documented in: doc/SDK_Doc_Release/M60/module/LLMConverter/llm_index.html under the SGS_IPU_Toolchain path.

2.2 Offline Model

The converted model is split into:

  • decoder_model_fixed.sim_sgsimg.img

  • norm_head_fixed.sim_sgsimg.img

  • FastVLMVision_fixed.sim_sgsimg.img

  • token_embedding_weight.npy

  • cos_weight.npy

  • sin_weight.npy

Generally, the head and embedding weights of large models are shared, so the converted model will be in tie_embedding mode, with parameters converted to token_embedding_weight.npy.

Model Name Model Description
decoder_model_fixed.sim_sgsimg.img Weights and graph of the model decoder part
norm_head_fixed.sim_sgsimg.img Graph of the model head part
token_embedding_weight.npy Weights of the model embedding part
FastVLMVision_fixed.sim_sgsimg.img Weights and graph of the model vision part
cos_weight.npy Weights of the cos part of model positional encoding
sin_weight.npy Weights of the sin part of model positional encoding

2.3 Model Implementation

  • sdk/verify/opendla/source/llm/modular/vision_decoder.cpp

    Implements the loading of decoder_model_fixed.sim_sgsimg.img, token_embedding.npy, cos_weight.npy, sin_weight.npy, management of subgraph and kv_cache memory, and the complete decoder model inference process.

    The decoder_model_fixed.sim_sgsimg.img contains multiple subgraphs for prefill and decode processes. The prefill subgraph generally sets the number of tokens inferred each time to 128, and kv_cache corresponds to the maximum token_length gradually increasing, such as 512/1024/2048, etc. The decode subgraph infers 1 token at a time, with maximum token_length corresponding to prefill. When calling the decoder_model_fixed.sim_sgsimg.img model, different subgraphs are called according to different stages and inputs. For details, please refer to sdk/verify/opendla/source/vlm/fastvlm/fastvlm.cpp

  • sdk/verify/opendla/source/llm/modular/norm_head.cpp

    Implements the loading and inference process of the norm_head_fixed.sim_sgsimg.img model.

  • sdk/verify/opendla/source/llm/modular/vision.cpp

    Implements the loading and inference process of the FastVLMVision_fixed.sim_sgsimg.img model.

3 Board Deployment

3.1 Program Compilation

Before compiling the sample program, you need to select the defconfig for SDK full package compilation based on the board specifications (nand/nor/emmc, DDR model, etc.). For details, please refer to the "Development Environment Setup" document in the Alkaid SDK sigdoc.

  • Compile the on-board FastVLM example.

    $cd sdk/verify/opendla
$make clean && make source/vlm/fastvlm -j8

  • The final generated executable file address:

    sdk/verify/opendla/out/${AARCH}/app/prog_vlm_fastvlm

  • Note: This demo can also be used for other large model inference, as long as the model is consistent with FastVLM

3.2 Runtime Files

When running the program, the following files need to be prepared:

  • prog_vlm_fastvlm
  • fastvlm offline model
  • apple.jpg stored in sdk/verify/opendla/source/resource/apple.jpg

3.3 Runtime Instructions

  • Usage: ./prog_vlm_fastvlm -d ./fastvlm/ -m ./apple.jpg -p "Briefly describe the content of the picture" (command to execute the file)

    • -d points to the converted FastVLM model, which contains the converted model files and corresponding JSON files
    • -m points to the image path
    • -p refers to the prompt, the demo currently passes "Briefly describe the content of the picture"

  • Typical Output:

    ./prog_vlm_fastvlm -d ./fastvlm/ -m ./apple.jpg -p "Briefly describe the content of the picture"

Assistant:  image features a highly detailed, close-up photograph of a red apple with a green leaf attached to its stem. The apple, which is round and predominantly red with a few yellowish spots, is positioned centrally against a white background. The lighting is bright, casting a subtle shadow beneath the apple, enhancing its three-dimensional appearance. The leaf, with its green stem and veins, is attached to the top of the apple, adding a touch of natural realism. The image is of high quality, with a focus on the apple's texture and color, making it suitable for use in a variety of contexts, such as a stock photo for a website or a stock image for a company's website.