PCIe User Guide

---

REVISION HISTORY

Revision No.	Description	Date
1.0	Initial release	01/04/2026

---

1. Overview

PCIe (Peripheral Component Interconnect Express) is a high-speed serial computer expansion bus standard used to connect various peripheral devices to computer systems. PCIe uses point-to-point serial connections and supports full-duplex communication through lanes. Each lane contains two pairs of differential signal lines for transmitting and receiving, which can operate at different rates independently.

The SGS PCIe controller complies with PCI Express base specifications and supports various link widths and rate configurations. The controller supports Root Complex (RC) and Endpoint (EP) modes, functioning as a root complex to connect downstream devices or as an endpoint device to connect to an upstream root complex. It supports multiple rate specifications including Gen1 and Gen2, providing high-performance data transfer capabilities.

---

2. Key Terms

• Lane

  The basic unit of a PCIe link, containing a pair of differential transmit signal lines and a pair of differential receive signal lines, supporting full-duplex communication. PCIe links can be configured as x1, x2, and other widths.

• Root Complex (RC)

  The core component in PCIe topology, connecting CPU and memory subsystems, responsible for initiating and receiving PCIe transactions. RC can connect multiple Switch and Endpoint devices.

• Endpoint (EP)

  The terminal device in PCIe topology, participating in PCIe transactions as a requester or completer. EP can be various peripherals such as network cards, storage controllers, graphics cards, etc.

• Gen1/Gen2 (Generation)

  Different rate specifications of PCIe. Gen1: 2.5GT/s, Gen2: 5GT/s. The theoretical bandwidth per lane increases with the rate.

• TLP (Transaction Layer Packet)

  The packet format used by the PCIe transaction layer to transmit information between requesters and completers. Includes Memory Read/Write, IO Read/Write, Configuration Read/Write, and other types.

• BAR (Base Address Register)

  Used to configure the memory or IO mapped addresses of PCIe devices. Through BAR, the CPU can access internal registers and memory space of PCIe devices.

• MSI/MSI-X (Message Signaled Interrupts)

  The interrupt mechanism used by PCIe devices, triggering interrupts by writing to specific memory addresses, replacing traditional pin-based interrupt methods. MSI-X supports more interrupt vectors.

• LTSSM (Link Training and Status State Machine)

  The training and management state machine of PCIe links, responsible for link initialization, rate negotiation, width negotiation, and fault recovery.

• Completion Timeout

  The timeout mechanism for PCIe transactions, triggered when the requester does not receive a completion packet within the specified time.

---

3. Feature Description

3.1 PCIe Resources

The MHERA PCIe controller provides 2 independent PCIe controllers (PCIe0 and PCIe1), each configurable as RC (Root Complex) or EP (Endpoint) mode.

The register address mapping for each PCIe controller is as follows (based on mhera.dtsi):

Register Region	PCIe0	PCIe1	Description
DBI	0x1F356000 (4KB)	0x1F35a000 (4KB)	Data Bus Interface, configuration space
DBI2	0x1F357000 (512B)	0x1F35b000 (512B)	DBI extension registers
ATU	0x1F358000 (4KB)	0x1F35c000 (4KB)	Address Translation Unit
CFG	0x10000000 (4KB)	0x40000000 (4KB)	Configuration space (RC mode)
Configuration Space	0x10000000 (1GB)	0x40000000 (1GB)	EP mode configuration space
MAC	0x1F2C7800 (512B)	0x1F2C7a00 (512B)	MAC registers
MAC2	0x1F2CD400 (512B)	0x1F2CD600 (512B)	MAC extension registers
DMA	0x1F359000 (4KB)	0x1F35d000 (4KB)	DMA registers

Note: The MHERA chip is configured with 2 PCIe controllers, both supporting x2 link width and Gen2 rate.

3.2 Feature Support

The following table shows the feature support of each PCIe controller

Feature	RC Mode	EP Mode	Link Width	Max Rate	MSI	MSI-X
Support Status	Supported	Supported	x1, x2	Gen2 (5GT/s)	Supported	Supported

Link rate support is shown in the following table:

PCIe Generation	Encoding	Raw Rate	Single Lane Bandwidth(x1)
Gen1	8b/10b	2.5 GT/s	250 MB/s
Gen2	8b/10b	5.0 GT/s	500 MB/s

Link width configuration support is shown below:

Link Width	Lane Count	Theoretical Bandwidth(Gen2)
x1	1	500 MB/s
x2	2	1.0 GB/s

PCIe Topology:

PCIe topology adopts a tree hierarchy with Root Complex (RC) as the core, connecting multiple Endpoint devices through Switches. RC connects to CPU and memory subsystems, responsible for initiating and receiving PCIe transactions.

3.3 Considerations

• Differential Signal Routing

  PCIe uses differential signal pairs, requiring strict control of impedance matching (typically 100Ω differential impedance), and the two lines of the same differential pair need equal length and equal spacing routing.

• Interrupt Configuration

  MSI-X interrupts support more interrupt vectors than MSI interrupts, so it is recommended to prioritize MSI-X in high-performance scenarios.

3.4 Bandwidth Calculation

The effective bandwidth of PCIe depends on link rate, link width, and coding efficiency. The theoretical bandwidth calculation formula is as follows:

Theoretical Bandwidth = Link Rate × Link Width × Coding Efficiency

For Gen2 x2 link:

• Raw Rate: 5 GT/s
• Link Width: 2 Lanes
• Coding Efficiency: 8/10 = 80%
• Theoretical Bandwidth: 5 GT/s × 2 × 0.8 = 8.0 Gb/s ≈ 1.0 GB/s

Link Rate Configuration

The link rate can be configured through the max-link-speed property in the device tree (DTS), and can also be viewed and adjusted at runtime using PCIe tools.

View current link rate:

 lspci -vv -s <device_id> | grep LnkCap

---

4. Hardware Connection

1. PCIe connections use high-speed differential signal pairs. Each lane contains a pair of transmit differential lines (TXP/TXN) and a pair of receive differential lines (RXP/RXN). Transmission and reception achieve full-duplex communication through independent differential pairs.

2. PCIe link width can be configured as x1, x2, etc. Different widths correspond to different numbers of differential pairs. For example, x2 link requires 2 pairs of transmit differential lines and 2 pairs of receive differential lines.

3. The PCIe hardware configuration of the MHERA chip is as follows:

   PCIe0 Configuration:

   • Link Width: x2
   • Supported Rate: Gen1, Gen2
   • Differential Signal Pairs: PCIe0_TX0_P/N, PCIe0_TX1_P/N, PCIe0_RX0_P/N, PCIe0_RX1_P/N

   PCIe1 Configuration:

   • Link Width: x2
   • Supported Rate: Gen1, Gen2
   • Differential Signal Pairs: PCIe1_TX0_P/N, PCIe1_TX1_P/N, PCIe1_RX0_P/N, PCIe1_RX1_P/N

PCIe Connection Description: PCIe implements point-to-point serial connections through high-speed differential signal pairs. Each lane contains a pair of transmit differential lines (TXP/TXN) and a pair of receive differential lines (RXP/RXN), supporting full-duplex communication. Differential signals require strict control of impedance matching (100Ω) and equal-length routing.

---

5. Kernel Usage Guide

5.1 Kernel Config Configuration

When compiling the Kernel, you need to enable PCIe-related configurations through menuconfig. The method to enter the kernel configuration interface:

make menuconfig

Then follow the path below to enable PCIe drivers:

Device Drivers --->

    Sgs SoC platform drivers --->

        Sgs PCIe Support --->

            [*] Sgs PCIe Controller - RC mode    # PCIe RC mode support
            [*] Sgs PCIe Controller - EP mode    # PCIe EP mode support
            [*] Sgs PCIe HOTPLUG                # PCIe hotplug support
                Sgs PCIe Rpmsg-lite  --->        # Rpmsg-lite communication support (optional)
                Sgs PCIe Verify  --->           # PCIe verification function (optional)

Configuration Item Description:

• Sgs PCIe Controller - RC mode: Enable PCIe Root Complex mode, allowing the chip to connect downstream PCIe devices as a root complex
• Sgs PCIe Controller - EP mode: Enable PCIe Endpoint mode, allowing the chip to connect to an upstream Root Complex as an endpoint device
• Sgs PCIe HOTPLUG: Enable PCIe hotplug function, supporting dynamic addition/removal of PCIe devices at runtime
• Sgs PCIe Rpmsg-lite: Communication protocol based on RPMsg for inter-device communication over PCIe (optional feature)
• Sgs PCIe Verify: PCIe verification function for debugging and testing (optional feature)

Note: The above configuration items are enabled in the default defconfig (RC, EP, and HOTPLUG). If modifications are needed, follow the above path for configuration.

After configuration is complete, save and recompile the kernel:

make -j$(nproc)

5.2 DTS Definition

In mhera.dtsi, the pcie0 and pcie1 node sections are as follows:

pcie0: pcie@1F356000 {
        compatible = "sgs,dw-pcie-earlyparam"; /* [rc] "sgs,dw-pcie" [ep] "sgs,dw-pcie-ep" [bootargs] "sgs,dw-pcie-earlyparam" */
        reg = <0x0 0x1f356000  0x1000>,     /* 4K DBI */
              <0x0 0x1f357000  0x200>,      /* DBI2 */
              <0x0 0x1f358000  0x1000>,     /* atu */
              <0x1 0x00000000  0x1000>,     /* [only rc] 4K CFG */
              <0x1 0x00000000  0x40000000>,  /* [only ep] Configuration space */
              <0x0 0x1f2c7800  0x200>,     /* mac */
              <0x0 0x1f2cd400  0x200>,     /* mac2 */
              <0x0 0x1f359000  0x1000>;    /* dma */
        reg-names = "dbi", "dbi2", "atu", "config", "addr_space", "mac", "mac2", "dma";
        portid = <0>;
        #address-cells = <3>;
        #size-cells = <2>;
        /* properties for RC */
        device_type = "pci";
        bus-range = <0x00 0xff>;
        /* ranges:   [flags]   [pci addr]    [cpu addr]        [size] */
        ranges = <0x81000000 0 0x00001000   0x1 0x00001000  0 0x0000f000   /* downstream I/O */
                  0x82000000 0 0x00010000   0x1 0x00010000  0 0x3fff0000>;/* non-prefetchable memory */
        linux,pci-domain = <0>;
        num-lanes = <2>;
        max-link-speed = <2>;
        interrupts = <GIC_SPI INT_IRQ_PCIE0_OUT IRQ_TYPE_LEVEL_HIGH>,
                     <GIC_SPI INT_IRQ_PCIE0_OUT IRQ_TYPE_LEVEL_HIGH>,
                     <GIC_SPI INT_IRQ_PCIE0_OUT IRQ_TYPE_LEVEL_HIGH>,
                     <GIC_SPI INT_IRQ_PCIE0_SUP_PHY_P0  IRQ_TYPE_LEVEL_HIGH>,
                     <GIC_SPI INT_IRQ_PCIE0_SUP_PHY_P1  IRQ_TYPE_LEVEL_HIGH>;
        interrupt-names = "msi", "mac", "hdma", "sup_phy0", "sup_phy1";
        #interrupt-cells = <1>;
        dma_read_channel_number=<8>;
        dma_write_channel_number=<8>;
        clocks = <&CLK_pcie0>;
        clk_source = <0>; /*0: internal, 1:internal_2, 2: external, 3: wire */
        status = "okay";
    };

pcie1: pcie@1F35a000 {
        compatible = "sgs,dw-pcie-earlyparam"; /* [rc] "sgs,dw-pcie" [ep] "sgs,dw-pcie-ep" [bootargs] "sgs,dw-pcie-earlyparam" */
        reg = <0x0 0x1f35a000  0x1000>,     /* DBI */
              <0x0 0x1f35b000  0x200>,      /* DBI2 */
              <0x0 0x1f35c000  0x1000>,     /* atu */
              <0x1 0x40000000  0x1000>,     /* [only rc] 4K CFG */
              <0x1 0x40000000  0x40000000>,  /* [only ep] Configuration space */
              <0x0 0x1f2c7a00  0x200>,     /* mac */
              <0x0 0x1f2cd600  0x200>,     /* mac2 */
              <0x0 0x1f35d000  0x1000>;    /* dma */
        reg-names = "dbi", "dbi2", "atu", "config", "addr_space", "mac", "mac2", "dma";
        portid = <1>;
        #address-cells = <3>;
        #size-cells = <2>;
        /* properties for RC */
        device_type = "pci";
        bus-range = <0x00 0xff>;
        /* ranges:   [flags]   [pci addr]    [cpu addr]        [size] */
        ranges = <0x81000000 0 0x00001000   0x1 0x40000000  0 0x00020000   /* downstream I/O */
                  0x82000000 0 0x00100000   0x1 0x40100000  0 0x3ff00000>; /* non-prefetchable memory */
        linux,pci-domain = <1>;
        num-lanes = <2>;
        max-link-speed = <2>;
        interrupts = <GIC_SPI INT_IRQ_PCIE1_OUT IRQ_TYPE_LEVEL_HIGH>,
                     <GIC_SPI INT_IRQ_PCIE1_OUT IRQ_TYPE_LEVEL_HIGH>,
                     <GIC_SPI INT_IRQ_PCIE1_OUT IRQ_TYPE_LEVEL_HIGH>,
                     <GIC_SPI INT_IRQ_PCIE1_SUP_PHY_P0  IRQ_TYPE_LEVEL_HIGH>,
                     <GIC_SPI INT_IRQ_PCIE1_SUP_PHY_P1  IRQ_TYPE_LEVEL_HIGH>;
        interrupt-names = "msi", "mac", "hdma", "sup_phy0", "sup_phy1";
        #interrupt-cells = <1>;
        dma_read_channel_number=<8>;
        dma_write_channel_number=<8>;
        clocks = <&CLK_pcie1>;
        clk_source = <0>; /*0: internal, 1:internal_2, 2: external, 3: wire */
        status = "okay";
    };

In the above sections, the address space of the PCIe controller includes DBI, DBI2, ATU, configuration space, MAC, MAC2, and DMA register space.

The description of the properties in the node is as follows:

Property	Description	Remarks
compatible	Used for driver matching	"sgs,dw-pcie" (RC), "sgs,dw-pcie-ep" (EP), "sgs,dw-pcie-earlyparam"
reg	Register address and size	Including DBI, DBI2, ATU, CFG, configuration space, MAC, MAC2, DMA
reg-names	Register region names	"dbi", "dbi2", "atu", "config", "addr_space", "mac", "mac2", "dma"
portid	Port ID	0: pcie0, 1: pcie1
#address-cells	Address cells	Do not modify
#size-cells	Size cells	Do not modify
device_type	Device type	Must be "pci"
bus-range	PCIe bus number range	Configure according to actual settings
ranges	Address mapping range	Configure according to actual settings
linux,pci-domain	PCIe domain number	Distinguish different PCIe controllers
num-lanes	Link width	1, 2
max-link-speed	Maximum link rate	1=Gen1, 2=Gen2
interrupts	Interrupt number	Including msi, mac, hdma, sup_phy0, sup_phy1
interrupt-names	Interrupt names	"msi", "mac", "hdma", "sup_phy0", "sup_phy1"
dma_read_channel_number	DMA read channel count	Default 8
dma_write_channel_number	DMA write channel count	Default 8
clocks	Clock node	Do not modify
clk_source	Clock source selection	0: internal, 1: internal_2, 2: external, 3: wire
status	Node enable	"okay": enable "disabled": disable

5.3 Module Usage Introduction

The basic workflow for user-space applications is:

1. Scan for PCIe devices

2. Access PCIe device configuration space

3. Map PCIe device BAR space for read/write operations

lspci Tool Description:

The lspci command included in the system is a simplified version with relatively limited functionality. If you need the complete version of the lspci tool (with more features and detailed options), please download it from the pciutils official website:

• Official Website: https://pci-ids.ucw.cz/git/
• Download Path: git://git.kernel.org/pub/scm/utils/pciutils/pciutils.git
• Compilation Method:

  make
  make install
  

The complete lspci tool provides more detailed device information parsing, more query options, and better compatibility.

Example: Using lspci tool to view PCIe device information

/ # lspci -vv
00:00.0 PCI bridge: SGS Technologies PCIe Controller (rev 01)
    Control: I/O+ Mem+ BusMaster+
    Status: Cap+ 66MHz- UDF- FastB2B- OrdErr-
    LnkCap: Port #0, Speed 5GT/s, Width x2
    LnkSta: Speed 5GT/s, Width x2

/ # lspci -s 00:01.0 -vv
00:01.0 Ethernet controller: Ethernet Controller X
    Region 0: Memory at 0x40000000 (32-bit, prefetchable)
    Region 2: Memory at 0x40100000 (32-bit, non-prefetchable)

5.4 Sample code

This demo provides a reference for accessing PCIe device configuration space and BAR space, and can be modified according to requirements.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <errno.h>
#include <string.h>

#define PCI_CONFIG_SPACE_SIZE 256
#define PCI_BAR0_SIZE 0x10000

/* Read PCIe configuration space */
int pci_read_config(int fd, int offset, unsigned char *buf, int len)
{
    if (lseek(fd, offset, SEEK_SET) == -1) {
        perror("lseek");
        return -1;
    }

    if (read(fd, buf, len) != len) {
        perror("read config");
        return -1;
    }

    return 0;
}

/* Write to PCIe configuration space */
int pci_write_config(int fd, int offset, unsigned char *buf, int len)
{
    if (lseek(fd, offset, SEEK_SET) == -1) {
        perror("lseek");
        return -1;
    }

    if (write(fd, buf, len) != len) {
        perror("write config");
        return -1;
    }

    return 0;
}

/* Map BAR space and access */
void* pci_map_bar(int bar_fd, size_t size)
{
    void *mapped_addr = mmap(NULL, size, PROT_READ | PROT_WRITE,
                             MAP_SHARED, bar_fd, 0);
    if (mapped_addr == MAP_FAILED) {
        perror("mmap");
        return NULL;
    }

    return mapped_addr;
}

int main(int argc, char **argv)
{
    int config_fd = -1;
    int bar_fd = -1;
    unsigned char config_buf[PCI_CONFIG_SPACE_SIZE];
    void *bar_base = NULL;
    char device_path[256];
    unsigned int vendor_id, device_id;

    /* Open PCIe device configuration space */
    snprintf(device_path, sizeof(device_path),
             "/sys/bus/pci/devices/0000:00:01.0/config");

    config_fd = open(device_path, O_RDWR);
    if (config_fd < 0) {
        perror("Failed to open PCI device config");
        return -1;
    }

    /* Read Vendor ID and Device ID */
    pci_read_config(config_fd, 0x00, config_buf, 4);
    vendor_id = config_buf[1] << 8 | config_buf[0];
    device_id = config_buf[3] << 8 | config_buf[2];

    printf("PCIe Device Information:\n");
    printf("  Vendor ID: 0x%04x\n", vendor_id);
    printf("  Device ID: 0x%04x\n", device_id);

    /* Read Class Code */
    pci_read_config(config_fd, 0x08, config_buf, 4);
    printf("  Class Code: 0x%02x%02x%02x\n",
           config_buf[2], config_buf[1], config_buf[0]);

    /* Read BAR0 address */
    pci_read_config(config_fd, 0x10, config_buf, 4);
    unsigned int bar0 = config_buf[3] << 24 | config_buf[2] << 16 |
                       config_buf[1] << 8 | config_buf[0];
    printf("  BAR0: 0x%08x\n", bar0);

    /* Open BAR0 resource file for memory mapping */
    snprintf(device_path, sizeof(device_path),
             "/sys/bus/pci/devices/0000:00:01.0/resource0");

    bar_fd = open(device_path, O_RDWR | O_SYNC);
    if (bar_fd < 0) {
        perror("Failed to open BAR0 resource");
        goto out;
    }

    /* Map BAR space */
    bar_base = pci_map_bar(bar_fd, PCI_BAR0_SIZE);
    if (!bar_base) {
        fprintf(stderr, "Failed to map BAR0\n");
        goto out;
    }

    printf("\nAccessing BAR0 registers:\n");
    printf("  BAR0 mapped at: %p\n", bar_base);

    /* Read register at BAR0 offset 0x00 */
    unsigned int reg_value = *(unsigned int*)(bar_base + 0x00);
    printf("  Register at 0x00: 0x%08x\n", reg_value);

out:
    if (bar_base && bar_base != MAP_FAILED) {
        munmap(bar_base, PCI_BAR0_SIZE);
    }

    if (bar_fd >= 0) {
        close(bar_fd);
    }

    if (config_fd >= 0) {
        close(config_fd);
    }

    return 0;
}

6. FAQ

Q1: PCIe Link Training Failure

1. Reproduce the failure where PCIe link cannot be established normally, dmesg shows link training failure:

   sgs-pcie 1f35a000.pcie: PCIe1: mode=RC, clk_source=internal, ssc_enable=0
   sgs-pcie 1f35a000.pcie: PCIe1: lanes=2, mac=ffffffc008e25a00, mac2=ffffffc008e2d600, dbi=ffffffc008e35000
   sgs-pcie 1f35a000.pcie: sgs_pcie_phy_init232
   sgs-pcie 1f35a000.pcie: host bridge /soc/pcie@1F35a000 ranges:
   sgs-pcie 1f35a000.pcie:       IO 0x0140000000..0x014001ffff -> 0x0000001000
   sgs-pcie 1f35a000.pcie:      MEM 0x0140100000..0x017fffffff -> 0x0000100000
   sgs-pcie 1f35a000.pcie: iATU unroll: enabled
   sgs-pcie 1f35a000.pcie: iATU regions: 8 ob, 8 ib, align 4K, limit 4G
   sgs-pcie 1f35a000.pcie: Phy link never came up
   

2. Root Cause Analysis

   Common causes of PCIe link training failure include:

   1. Hardware connection issues: differential signal line impedance mismatch, incorrect wire sequence, or poor contact;

   2. Clock configuration issues: incorrect reference clock frequency or poor clock signal quality;

   3. Device tree configuration issues: max-link-speed and num-lanes configuration do not match actual hardware.

3. Solutions

   Check hardware connections and device tree configuration to ensure:

   1. Use an oscilloscope or logic analyzer to check differential signal quality;

   2. Check max-link-speed and num-lanes configuration in device tree;

   3. Try reducing link rate or link width for testing.

Q2: PCIe Device Cannot Enumerate

1. PCIe device is not recognized by the system after startup, lspci command cannot see the device:

   / # lspci
   / #
   

2. Root Cause Analysis

   Possible reasons why PCIe device cannot enumerate:

   1. PCIe controller driver is not loaded correctly;

   2. Device tree configuration error, PCIe controller is not enabled;

   3. PERST# signal of PCIe device is not reset correctly;

   4. PCIe device power supply is not working properly.

   You can check the initialization status of the PCIe controller through dmesg. Use the dmesg | grep pci command to view initialization logs and error information of the PCIe controller.

3. Solutions

   1. Check whether the PCIe controller driver is loaded correctly;

   2. Ensure the status property of the PCIe controller in device tree is "ok";

   3. Check the timing of PERST# signal in hardware design to ensure it meets PCIe specification requirements;

   4. Use a multimeter to check whether the power supply voltage of the PCIe device is normal.

Q3: PCIe Bandwidth Below Expected

1. PCIe link is established, but actual bandwidth is far below theoretical value:

   / # lspci -vv -s 00:01.0 | grep LnkSta LnkSta: Speed 2.5GT/s, Width x1

   Although configured as Gen2 x2, it actually works at Gen1 x1.

2. Description

   PCIe link rate and width will be automatically negotiated. The actual rate depends on:

   1. The maximum rate supported by both devices;

   2. Whether signal quality meets high-speed transmission requirements;

   3. Whether errors are detected during link training.

3. Solutions

   1. Check device tree configuration to confirm max-link-speed and num-lanes are set correctly;

   2. Check hardware design to ensure impedance matching and routing quality of differential signal lines;

   3. Check whether the peer device supports the configured rate and width;

   4. Check dmesg logs to confirm error messages during link training.