2. NVMe PCI Endpoint Function Target

Author:

Damien Le Moal <dlemoal@kernel.org>

The NVMe PCI endpoint function target driver implements a NVMe PCIe controller using a NVMe fabrics target controller configured with the PCI transport type.

2.1. Overview

The NVMe PCI endpoint function target driver allows exposing a NVMe target controller over a PCIe link, thus implementing an NVMe PCIe device similar to a regular M.2 SSD. The target controller is created in the same manner as when using NVMe over fabrics: the controller represents the interface to an NVMe subsystem using a port. The port transfer type must be configured to be “pci”. The subsystem can be configured to have namespaces backed by regular files or block devices, or can use NVMe passthrough to expose to the PCI host an existing physical NVMe device or a NVMe fabrics host controller (e.g. a NVMe TCP host controller).

The NVMe PCI endpoint function target driver relies as much as possible on the NVMe target core code to parse and execute NVMe commands submitted by the PCIe host. However, using the PCI endpoint framework API and DMA API, the driver is also responsible for managing all data transfers over the PCIe link. This implies that the NVMe PCI endpoint function target driver implements several NVMe data structure management and some NVMe command parsing.

  1. The driver manages retrieval of NVMe commands in submission queues using DMA if supported, or MMIO otherwise. Each command retrieved is then executed using a work item to maximize performance with the parallel execution of multiple commands on different CPUs. The driver uses a work item to constantly poll the doorbell of all submission queues to detect command submissions from the PCIe host.

  2. The driver transfers completion queues entries of completed commands to the PCIe host using MMIO copy of the entries in the host completion queue. After posting completion entries in a completion queue, the driver uses the PCI endpoint framework API to raise an interrupt to the host to signal the commands completion.

  3. For any command that has a data buffer, the NVMe PCI endpoint target driver parses the command PRPs or SGLs lists to create a list of PCI address segments representing the mapping of the command data buffer on the host. The command data buffer is transferred over the PCIe link using this list of PCI address segments using DMA, if supported. If DMA is not supported, MMIO is used, which results in poor performance. For write commands, the command data buffer is transferred from the host into a local memory buffer before executing the command using the target core code. For read commands, a local memory buffer is allocated to execute the command and the content of that buffer is transferred to the host once the command completes.

2.1.1. Controller Capabilities

The NVMe capabilities exposed to the PCIe host through the BAR 0 registers are almost identical to the capabilities of the NVMe target controller implemented by the target core code. There are some exceptions.

  1. The NVMe PCI endpoint target driver always sets the controller capability CQR bit to request “Contiguous Queues Required”. This is to facilitate the mapping of a queue PCI address range to the local CPU address space.

  2. The doorbell stride (DSTRB) is always set to be 4B

  3. Since the PCI endpoint framework does not provide a way to handle PCI level resets, the controller capability NSSR bit (NVM Subsystem Reset Supported) is always cleared.

  4. The boot partition support (BPS), Persistent Memory Region Supported (PMRS) and Controller Memory Buffer Supported (CMBS) capabilities are never reported.

2.1.2. Supported Features

The NVMe PCI endpoint target driver implements support for both PRPs and SGLs. The driver also implements IRQ vector coalescing and submission queue arbitration burst.

The maximum number of queues and the maximum data transfer size (MDTS) are configurable through configfs before starting the controller. To avoid issues with excessive local memory usage for executing commands, MDTS defaults to 512 KB and is limited to a maximum of 2 MB (arbitrary limit).

2.1.3. Mimimum number of PCI Address Mapping Windows Required

Most PCI endpoint controllers provide a limited number of mapping windows for mapping a PCI address range to local CPU memory addresses. The NVMe PCI endpoint target controllers uses mapping windows for the following.

  1. One memory window for raising MSI or MSI-X interrupts

  2. One memory window for MMIO transfers

  3. One memory window for each completion queue

Given the highly asynchronous nature of the NVMe PCI endpoint target driver operation, the memory windows as described above will generally not be used simultaneously, but that may happen. So a safe maximum number of completion queues that can be supported is equal to the total number of memory mapping windows of the PCI endpoint controller minus two. E.g. for an endpoint PCI controller with 32 outbound memory windows available, up to 30 completion queues can be safely operated without any risk of getting PCI address mapping errors due to the lack of memory windows.

2.1.4. Maximum Number of Queue Pairs

Upon binding of the NVMe PCI endpoint target driver to the PCI endpoint controller, BAR 0 is allocated with enough space to accommodate the admin queue and multiple I/O queues. The maximum of number of I/O queues pairs that can be supported is limited by several factors.

  1. The NVMe target core code limits the maximum number of I/O queues to the number of online CPUs.

  2. The total number of queue pairs, including the admin queue, cannot exceed the number of MSI-X or MSI vectors available.

  3. The total number of completion queues must not exceed the total number of PCI mapping windows minus 2 (see above).

The NVMe endpoint function driver allows configuring the maximum number of queue pairs through configfs.

2.1.5. Limitations and NVMe Specification Non-Compliance

Similar to the NVMe target core code, the NVMe PCI endpoint target driver does not support multiple submission queues using the same completion queue. All submission queues must specify a unique completion queue.

2.2. User Guide

This section describes the hardware requirements and how to setup an NVMe PCI endpoint target device.

2.2.1. Kernel Requirements

The kernel must be compiled with the configuration options CONFIG_PCI_ENDPOINT, CONFIG_PCI_ENDPOINT_CONFIGFS, and CONFIG_NVME_TARGET_PCI_EPF enabled. CONFIG_PCI, CONFIG_BLK_DEV_NVME and CONFIG_NVME_TARGET must also be enabled (obviously).

In addition to this, at least one PCI endpoint controller driver should be available for the endpoint hardware used.

To facilitate testing, enabling the null-blk driver (CONFIG_BLK_DEV_NULL_BLK) is also recommended. With this, a simple setup using a null_blk block device as a subsystem namespace can be used.

2.2.2. Hardware Requirements

To use the NVMe PCI endpoint target driver, at least one endpoint controller device is required.

To find the list of endpoint controller devices in the system:

# ls /sys/class/pci_epc/
 a40000000.pcie-ep

If PCI_ENDPOINT_CONFIGFS is enabled:

# ls /sys/kernel/config/pci_ep/controllers
 a40000000.pcie-ep

The endpoint board must of course also be connected to a host with a PCI cable with RX-TX signal swapped. If the host PCI slot used does not have plug-and-play capabilities, the host should be powered off when the NVMe PCI endpoint device is configured.

2.2.3. NVMe Endpoint Device

Creating an NVMe endpoint device is a two step process. First, an NVMe target subsystem and port must be defined. Second, the NVMe PCI endpoint device must be setup and bound to the subsystem and port created.

2.2.4. Creating a NVMe Subsystem and Port

Details about how to configure a NVMe target subsystem and port are outside the scope of this document. The following only provides a simple example of a port and subsystem with a single namespace backed by a null_blk device.

First, make sure that configfs is enabled:

# mount -t configfs none /sys/kernel/config

Next, create a null_blk device (default settings give a 250 GB device without memory backing). The block device created will be /dev/nullb0 by default:

# modprobe null_blk
# ls /dev/nullb0
/dev/nullb0

The NVMe PCI endpoint function target driver must be loaded:

# modprobe nvmet_pci_epf
# lsmod | grep nvmet
nvmet_pci_epf          32768  0
nvmet                 118784  1 nvmet_pci_epf
nvme_core             131072  2 nvmet_pci_epf,nvmet

Now, create a subsystem and a port that we will use to create a PCI target controller when setting up the NVMe PCI endpoint target device. In this example, the port is created with a maximum of 4 I/O queue pairs:

# cd /sys/kernel/config/nvmet/subsystems
# mkdir nvmepf.0.nqn
# echo -n "Linux-pci-epf" > nvmepf.0.nqn/attr_model
# echo "0x1b96" > nvmepf.0.nqn/attr_vendor_id
# echo "0x1b96" > nvmepf.0.nqn/attr_subsys_vendor_id
# echo 1 > nvmepf.0.nqn/attr_allow_any_host
# echo 4 > nvmepf.0.nqn/attr_qid_max

Next, create and enable the subsystem namespace using the null_blk block device:

# mkdir nvmepf.0.nqn/namespaces/1
# echo -n "/dev/nullb0" > nvmepf.0.nqn/namespaces/1/device_path
# echo 1 > "nvmepf.0.nqn/namespaces/1/enable"

Finally, create the target port and link it to the subsystem:

# cd /sys/kernel/config/nvmet/ports
# mkdir 1
# echo -n "pci" > 1/addr_trtype
# ln -s /sys/kernel/config/nvmet/subsystems/nvmepf.0.nqn \
        /sys/kernel/config/nvmet/ports/1/subsystems/nvmepf.0.nqn

2.2.5. Creating a NVMe PCI Endpoint Device

With the NVMe target subsystem and port ready for use, the NVMe PCI endpoint device can now be created and enabled. The NVMe PCI endpoint target driver should already be loaded (that is done automatically when the port is created):

# ls /sys/kernel/config/pci_ep/functions
nvmet_pci_epf

Next, create function 0:

# cd /sys/kernel/config/pci_ep/functions/nvmet_pci_epf
# mkdir nvmepf.0
# ls nvmepf.0/
baseclass_code    msix_interrupts   secondary
cache_line_size   nvme              subclass_code
deviceid          primary           subsys_id
interrupt_pin     progif_code       subsys_vendor_id
msi_interrupts    revid             vendorid

Configure the function using any device ID (the vendor ID for the device will be automatically set to the same value as the NVMe target subsystem vendor ID):

# cd /sys/kernel/config/pci_ep/functions/nvmet_pci_epf
# echo 0xBEEF > nvmepf.0/deviceid
# echo 32 > nvmepf.0/msix_interrupts

If the PCI endpoint controller used does not support MSI-X, MSI can be configured instead:

# echo 32 > nvmepf.0/msi_interrupts

Next, let’s bind our endpoint device with the target subsystem and port that we created:

# echo 1 > nvmepf.0/nvme/portid
# echo "nvmepf.0.nqn" > nvmepf.0/nvme/subsysnqn

The endpoint function can then be bound to the endpoint controller and the controller started:

# cd /sys/kernel/config/pci_ep
# ln -s functions/nvmet_pci_epf/nvmepf.0 controllers/a40000000.pcie-ep/
# echo 1 > controllers/a40000000.pcie-ep/start

On the endpoint machine, kernel messages will show information as the NVMe target device and endpoint device are created and connected.

null_blk: disk nullb0 created
null_blk: module loaded
nvmet: adding nsid 1 to subsystem nvmepf.0.nqn
nvmet_pci_epf nvmet_pci_epf.0: PCI endpoint controller supports MSI-X, 32 vectors
nvmet: Created nvm controller 1 for subsystem nvmepf.0.nqn for NQN nqn.2014-08.org.nvmexpress:uuid:2ab90791-2246-4fbb-961d-4c3d5a5a0176.
nvmet_pci_epf nvmet_pci_epf.0: New PCI ctrl "nvmepf.0.nqn", 4 I/O queues, mdts 524288 B

2.2.6. PCI Root-Complex Host

Booting the PCI host will result in the initialization of the PCIe link (this may be signaled by the PCI endpoint driver with a kernel message). A kernel message on the endpoint will also signal when the host NVMe driver enables the device controller:

nvmet_pci_epf nvmet_pci_epf.0: Enabling controller

On the host side, the NVMe PCI endpoint function target device will is discoverable as a PCI device, with the vendor ID and device ID as configured:

# lspci -n
0000:01:00.0 0108: 1b96:beef

An this device will be recognized as an NVMe device with a single namespace:

# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1     259:0    0   250G  0 disk

The NVMe endpoint block device can then be used as any other regular NVMe namespace block device. The nvme command line utility can be used to get more detailed information about the endpoint device:

# nvme id-ctrl /dev/nvme0
NVME Identify Controller:
vid       : 0x1b96
ssvid     : 0x1b96
sn        : 94993c85650ef7bcd625
mn        : Linux-pci-epf
fr        : 6.13.0-r
rab       : 6
ieee      : 000000
cmic      : 0xb
mdts      : 7
cntlid    : 0x1
ver       : 0x20100
...

2.3. Endpoint Bindings

The NVMe PCI endpoint target driver uses the PCI endpoint configfs device attributes as follows.

vendorid

Ignored (the vendor id of the NVMe target subsystem is used)

deviceid

Anything is OK (e.g. PCI_ANY_ID)

revid

Do not care

progif_code

Must be 0x02 (NVM Express)

baseclass_code

Must be 0x01 (PCI_BASE_CLASS_STORAGE)

subclass_code

Must be 0x08 (Non-Volatile Memory controller)

cache_line_size

Do not care

subsys_vendor_id

Ignored (the subsystem vendor id of the NVMe target subsystem is used)

subsys_id

Anything is OK (e.g. PCI_ANY_ID)

msi_interrupts

At least equal to the number of queue pairs desired

msix_interrupts

At least equal to the number of queue pairs desired

interrupt_pin

Interrupt PIN to use if MSI and MSI-X are not supported

The NVMe PCI endpoint target function also has some specific configurable fields defined in the nvme subdirectory of the function directory. These fields are as follows.

mdts_kb

Maximum data transfer size in KiB (default: 512)

portid

The ID of the target port to use

subsysnqn

The NQN of the target subsystem to use