繁體中文 00.01.NVMe over PCIe Transport explains how NVMe queues, properties, and notifications operate over PCIe. This note connects memory-mapped I/O, DMA, and interrupts to the transfer of an NVMe command.NVMeNon-Volatile Memory Express, the specification family for a host interface to a non-volatile-memory subsystem.PCIePCI Express, the transport and device interconnect used by an NVMe memory-based controller.I/OInput/Output, the class of data operations performed on a namespace. The main ideas 01Interface locations01-01Use BARs and configuration space to locate the NVMe interface and capabilities. 02Commands and notifications02-01Distinguish queue data, doorbells, and interrupts. 03Platform behavior03-01Understand the scope of resets, power, error reporting, and link measurements....
read more
NVMe over PCIe Transport 1.4: Complete Transport Binding
00.01.NVMe over PCIe Transport explains how NVMe queues, properties, and notifications operate over PCIe. This note connects memory-mapped I/O, DMA, and interrupts to the transfer of an NVMe command.
- NVMe
- Non-Volatile Memory Express, the specification family for a host interface to a non-volatile-memory subsystem.
- PCIe
- PCI Express, the transport and device interconnect used by an NVMe memory-based controller.
- I/O
- Input/Output, the class of data operations performed on a namespace.
The main ideas
Interface locations
01-01Use BARs and configuration space to locate the NVMe interface and capabilities.
Commands and notifications
02-01Distinguish queue data, doorbells, and interrupts.
Platform behavior
03-01Understand the scope of resets, power, error reporting, and link measurements.
00.02.The Base specification defines the common NVMe command and queue model; PCIe Transport supplies the local PCIe binding. Accessing a queue entry in memory and accessing a device MMIO register are different kinds of access.
- MMIO
- Memory-Mapped I/O, access to device registers through CPU memory operations.
Connecting the main ideas
00.03.The submission/completion model defined by Base needs concrete addresses, memory accesses, and notifications over PCIe. Locate the controller register space, follow SQ/CQ exchanges, and then interpret interrupts and PCIe status.
- controller
- Controller, the entity that implements the NVMe interface, fetches commands, and reports completions.
- CQ
- Completion Queue, the queue into which a controller posts command completions.
- SQ
- Submission Queue, the queue into which the host places commands.
00.04.Configuration Space, error records, and eye measurements describe device presentation, transport events, and receiver measurement layouts at different levels. They are distinct from NVMe command completion status. The aim is to explain the roles of data, doorbells, and interrupts in an exchange.
01 How NVMe uses PCIe
01.01.Figure 1 shows document applicability and Figure 2 separates protocol responsibility. Engineering analysis separates command semantics from the way host memory, MMIO, configuration space, and interrupts carry the operation. The Transport does not rewrite Base when the two conflict.
- Host
- The system running the operating system and issuing NVMe commands.
01.02.The PCIe Transport supplements the Base Specification with PCIe-specific structures, extensions, requirements, and behavior; common NVMe behavior remains in Base. In a conflict, Base has higher precedence than a Transport Specification.
Sources: PCIe Transport 1.4 §1.2
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §1.2, printed pages 6, PDF pages 6
| Specification layer | Behavior defined | Use in this report |
|---|---|---|
| Base | Common command and completion semantics | Highest-precedence NVMe definition |
| PCIe Transport | Address, register, doorbell, and interrupt binding | Adds PCIe-specific requirements |
| PCI-SIG specifications | Native PCIe capability and transaction semantics | This report covers only NVMe-specific statements present in the supplied source |
02 BARs, MMIO, and doorbell addresses
02.01.NVMe controller registers reside in the memory space designated by BAR0/BAR1. Doorbells begin at 1000h; SQ-tail and CQ-head registers for queue y are spaced using CAP.DSTRD. Figures 3-6 form one address derivation rather than four independent register tables.
- DSTRD
- Doorbell Stride, the CAP field determining spacing between adjacent doorbell registers.
- CAP
- Controller Capabilities, the controller property at offset 00h that reports queue, page-size, timeout, and other capabilities.
02.02.NVMe controller registers reside in memory space identified by BAR0/BAR1. The host shall use native-width or aligned 32-bit accesses and shall not issue locked accesses; violation produces undefined behavior.
Sources: PCIe Transport 1.4 §3.1
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.1, printed pages 9-10, PDF pages 9-10
| Doorbell or written value | Address or value calculation | Host notification conveyed |
|---|---|---|
| SQ y tail | 1000h + (2y) x (4 << DSTRD) | Host publishes a new SQ tail |
| CQ y head | 1000h + (2y+1) x (4 << DSTRD) | Host publishes a consumed CQ head |
| Doorbell value | Queue pointer | Does not contain the SQE or CQE body |
- CQE
- Completion Queue Entry, one completion-result structure in a CQ.
03 Command exchange between host and controller
03.01.SQE creation, doorbell write, controller fetch, CQE posting, interrupt delivery, and CQ-head update are not names for one event; they are successive ownership handoffs between host and controller. Their order governs both memory ordering and resource reuse.
- The host writes a command to the Submission Queue (SQ).
- The host updates the SQ Tail Doorbell to announce new work.
- The controller retrieves and executes the command, then writes its result to the Completion Queue (CQ).
- The host reads the CQE and updates the CQ Head Doorbell to release consumed entries.
03.02.The command flow writes an SQE, updates the SQ-tail doorbell, lets the controller fetch and execute, posts a CQE, optionally interrupts, processes the CQE, and updates the CQ-head doorbell. A doorbell conveys a pointer, not the command body.
Sources: PCIe Transport 1.4 §3.4
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.4, printed pages 12-13, PDF pages 12-13
| Resource to reuse | When prior use ends | Host observation |
|---|---|---|
| SQ-slot reuse | Controller has consumed the SQE | Completion SQHD assists tracking |
| Command-buffer reuse | Command completed and data is visible | Check command and data direction |
| CQ-slot release | Host completely consumed the CQE | Then write the CQ-head doorbell |
04 Interrupt modes and notification behavior
04.01.Pin-based, single-message MSI, multiple-message MSI, and MSI-X differ in more than performance. They provide different vector counts, masking locations, and capability structures; interrupt coalescing separately controls when multiple completions produce a notification. Figure 9 and Figures 34-46 belong with queue-to-vector mapping.
04.02.Modes are pin-based, single-message MSI, multiple-message MSI, and MSI-X. The specification recommends MSI-X. Coalescing can reduce interrupt rate at the cost of latency, and Admin-CQ interrupts should not be delayed.
- Admin
- Administrative, the control path used to create, configure, query, or manage controllers and queues.
Sources: PCIe Transport 1.4 §3.5
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.5, printed pages 13-16, PDF pages 13-16
| Interrupt mechanism | Notification and masking arrangement | Resources or limits affected |
|---|---|---|
| Pin-based | Legacy shared signaling | Sharing and masking differ |
| Single MSI | One message/vector | Multiple CQs may share a service path |
| Multiple MSI | A set of contiguous messages | Constrained by MME/MMC capability |
| MSI-X | Table-based vectors with independent masks | Preferred by the specification |
05 Configuration space and PCIe error reporting
05.01.Figures 10-67 traverse the Type 0 header, Power Management, MSI/MSI-X, PCIe capability, and AER. Find the capability or extended-capability base before applying offsets. AER status, mask, severity, and header log form one diagnostic set rather than isolated error bits.
- AER
- Advanced Error Reporting, the PCIe capability for classifying, masking, and logging link or transaction errors.
05.02.Section 3.8 defines additional NVMe-controller requirements for the PCI header, Power Management, MSI/MSI-X, PCIe capability, and AER. Original PCI/PCIe field semantics remain governed by PCI-SIG specifications.
Sources: PCIe Transport 1.4 §3.8.1-3.8.7
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.8.1-3.8.7, printed pages 16-35, PDF pages 16-35
| Reporting layer or capability | Event or resource described | Information needed alongside it |
|---|---|---|
| NVMe CQE status | Command execution result | Read the fields in NVMe command context |
| PCIe Device Status | PCIe Function status summary | Found in PCIe capability |
| AER | Correctable/uncorrectable transport errors | Read status, mask, severity, and header together |
| Power state | Slot limit and device power control | Never choose an NVMe state above the slot power limit |
06 Receiver eye-opening measurement data layout
06.01.The receiver eye-opening measurement log has variable length. Its header describes the whole dataset, while lane descriptors describe individual lanes. Structure levels and length units identify which lane each measurement belongs to.
06.02.The Physical Interface Receiver Eye Opening Measurement log page reports measurements through a header, lane descriptors, and EOM data. The host checks support and size before parsing lanes and parameters.
- EOM
- Eye Opening Measurement, the procedure and log data for measuring a PCIe receiver eye opening.
Sources: PCIe Transport 1.4 §3.9
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.9, printed pages 39-46, PDF pages 39-46
| Measurement region | Measurement or location described | Checks before comparison |
|---|---|---|
| Specific parameter | Selects measurement action and quality/state | Establish request context first |
| Specific identifier | Selects lane/test context | Prevents mixing different measurements |
| Header | Global length and layout | Base for every later offset |
| Lane descriptor | Per-lane boundaries and status | Walk only within returned buffer |
Where to continue in the specification
07.01.Use the flow above to frame the problem, then open the corresponding sections for fields and full conditions. The Chinese tutorial also explains every in-scope figure with its takeaway, example, and details.
| Concept to explain | Specification sections |
|---|---|
| How NVMe uses PCIe | PCIe Transport 1.4 §1.2 · PCIe Transport 1.4 §1.3 · PCIe Transport 1.4 §2 |
| BARs, MMIO, and doorbell addresses | PCIe Transport 1.4 §3.1 · PCIe Transport 1.4 §3.1.2.1-3.1.2.2 |
| Command exchange between host and controller | PCIe Transport 1.4 §3.4 · PCIe Transport 1.4 §3.2 |
| Interrupt modes and notification behavior | PCIe Transport 1.4 §3.5 · PCIe Transport 1.4 §3.2 · PCIe Transport 1.4 §Annex A |
| Configuration space and PCIe error reporting | PCIe Transport 1.4 §3.8.1-3.8.7 · PCIe Transport 1.4 §3.7 · PCIe Transport 1.4 §3.6 |
| Receiver eye-opening measurement data layout | PCIe Transport 1.4 §3.9 · PCIe Transport 1.4 §Annex A |
Check your understanding
1. What roles do PCIe MMIO properties and host-memory queues play?
08.01.Properties provide controller configuration, status, and queue notification interfaces. Queues carry command and completion entries. Separating the control interface from those structures explains how submission works.
Sources
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.1, printed pages 9-10, PDF pages 9-10
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.2, printed pages 11, PDF pages 11
2. Why is it unsafe to assume adjacent doorbells are always 4 bytes apart?
08.02.CAP.DSTRD determines the spacing: stride is 2^(2+DSTRD) bytes. It is 4 bytes only when DSTRD=0; queue ID also determines the SQ Tail and CQ Head locations.
Sources
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.1.2.1-3.1.2.2, printed pages 10-11, PDF pages 10-11
3. Why must the host inspect the CQ after an interrupt?
08.03.The interrupt is a notification; CQEs contain command identity and completion status. One notification need not correspond to one completion entry, so the host processes valid entries according to queue progress.
Sources
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.5, printed pages 13-16, PDF pages 13-16
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.4, printed pages 12-13, PDF pages 12-13
4. Why distinguish NVMe command status from PCIe error reporting?
08.04.The former describes command processing; the latter describes transport and device-level errors. They can be related, but they address different objects. Success in one layer does not establish the state of every layer.
Sources
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.7, printed pages 16, PDF pages 16
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.8.1-3.8.7, printed pages 16-35, PDF pages 16-35
Comments