Skip to content

LLM Environment Setup

LLM Model Conversion & Runtime Environment Setup

📝 IPU_Toolchain is developed based on AVX2 instruction set. Please use a processor that supports AVX2.

LLM models have large parameter counts and high memory consumption during conversion. Please use the recommended configuration.

Recommended Configuration

🖥️💻 ⚙️
CPU Intel® Xeon® Gold 6242 or Higher
RAM 256G or Higher

1. Install Docker Development Environment

Method 1:Pull the docker image from DockerHub(recommended).

docker pull docker.sigmastar.com.cn/sigmastar/ipu_docker:v1.11.1
docker tag docker.sigmastar.com.cn/sigmastar/ipu_docker:v1.11.1 sigmastar/ipu_docker:v1.11.1
docker run -it --net=host --privileged --name SGS_IPU_ToolChain --ulimit core=-1 -v /:/work/SGS_IPU_ToolChain sigmastar/ipu_docker:v1.11.1  env LANG=C.UTF-8 /bin/bash

Method 2:Load the SGS docker image file (LLM conversion requires v1.11.1 or higher).

tar -Jxvf sgs_docker_v1.11.1.tar.xz
docker load < sgs_docker_v1.11.1.tar

Run SGS docker.

./run_docker.sh

Usage

Explanation of the docker run parameters in run_docker.sh. Users can modify parameter configurations according to their needs:

  • -i / --interactive: Keep the container's standard input (stdin) open, allowing users to interact with the container.
  • -t / --tty: Allocate a pseudo-TTY, typically used together with -i to obtain an interactive shell in the container.
  • --net=host: Use the host machine's network stack.
  • --privileged: Enable privileged mode for the container, giving it access to host devices and capabilities such as mounting devices and using kernel modules.
  • --name: Assign a name to this container for easier reference in future docker operations.
  • --ulimit core=-1: Set the container's core file generation limit. -1 means no limit, allowing core files of any size.
  • -v /:/work: Mount the host's root directory / to the container's /work directory. If /work does not exist in the container, it will be created automatically.
  • env LANG=C.UTF-8: Set the environment variable LANG=C.UTF-8 in the container to specify language and character encoding, ensuring applications in the container correctly support UTF-8 encoding.
  • /bin/bash: The command executed after the container starts, i.e., launching a Bash shell. -it ensures interaction with this shell.

Usage

For more Docker-related questions, please refer to this link.

2. SGS_IPU_Toolchain Environment Setup

Usage

Newly created docker containers do not contain the IPU Toolchain. Please obtain the IPU Toolchain from the mounted path inside the container (the mount path is specified by the -v parameter in run_docker.sh), or use the docker cp command to copy the IPU Toolchain into the container.

Enter the SGS_IPU_Toolchain directory and set up the environment.

cd /path/to/SGS_IPU_Toolchain
source cfg_env.sh