An NVMe technical report covering the main ideas, mechanisms, conditions, and examples.
read more
NVMe over PCIe Transport 1.4: Complete Transport Binding
NVM Express Base 2.4NVM Command Set 1.3NVMe over PCIe Transport 1.4
Start with one command
00.01.The host writes a command to the SQ (Submission Queue), then updates the SQ Tail Doorbell to announce the new queue position. The controller fetches and executes the command, then places its result in the CQ (Completion Queue). The doorbell does not carry the command; the host reads the CQE to obtain the result.
- controller
- Controller, the entity that implements the NVMe interface, fetches commands, and reports completions.
- Host
- The system running the operating system and issuing NVMe commands.
- CQE
- Completion Queue Entry, one completion-result structure in a CQ.
- CQ
- Completion Queue, the queue into which a controller posts command completions.
- SQ
- Submission Queue, the queue into which the host places commands.
00.02.Base 2.4 defines the shared command/queue model; this report supplies PCIe addresses, accesses, and notifications. Transport 1.4 §1.5 (p.7) references PCIe Base 6.2, while this site’s PCIe course uses 6.0. Check the applicable version when reading PCIe-specific details.
- PCIe
- PCI Express, the transport and device interconnect used by an NVMe memory-based controller.
Continue along this reading route
First: where does the notification go?
00.03.Follow §3.1 into BAR0/BAR1 and access rules, then §3.1.2 for DSTRD and queue-ID offsets.
- DSTRD
- Doorbell Stride, the CAP field determining spacing between adjacent doorbell registers.
Next: how do command and result travel?
00.04.Follow §3.4 through SQE, doorbell, fetch, CQE, and CQ-head update. SQE/CQE are submission/completion entries.
- CQE
- Completion Queue Entry, one completion-result structure in a CQ.
- SQE
- Submission Queue Entry, one command structure in an SQ.
- CQ
- Completion Queue, the queue into which a controller posts command completions.
Then: what follows an interrupt?
00.05.Continue into interrupt mode and completion processing, keeping PCIe errors separate from NVMe command status. Receiver eye measurements are a separate diagnostic path.
- NVMe
- Non-Volatile Memory Express, the specification family for a host interface to a non-volatile-memory subsystem.
- PCIe
- PCI Express, the transport and device interconnect used by an NVMe memory-based controller.
Sources: PCIe Transport 1.4 §1.2 · PCIe Transport 1.4 §3.1 · PCIe Transport 1.4 §3.1.2.1-3.1.2.2 · PCIe Transport 1.4 §3.4
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §1.2, printed pages 6, PDF pages 6
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.1, printed pages 9-10, PDF pages 9-10
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.1.2.1-3.1.2.2, printed pages 10-11, PDF pages 10-11
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.4, printed pages 12-13, PDF pages 12-13
01 How NVMe uses PCIe
01.01.Figure 1 shows document applicability and Figure 2 separates protocol responsibility. Engineering analysis separates command semantics from the way host memory, MMIO, configuration space, and interrupts carry the operation. The Transport does not rewrite Base when the two conflict.
- Host
- The system running the operating system and issuing NVMe commands.
- MMIO
- Memory-Mapped I/O, access to device registers through CPU memory operations.
01.02.The PCIe Transport supplements the Base Specification with PCIe-specific structures, extensions, requirements, and behavior; common NVMe behavior remains in Base. In a conflict, Base has higher precedence than a Transport Specification.
- NVMe
- Non-Volatile Memory Express, the specification family for a host interface to a non-volatile-memory subsystem.
Sources: PCIe Transport 1.4 §1.2
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §1.2, printed pages 6, PDF pages 6
| Specification layer | Behavior defined | Use in this report |
|---|---|---|
| Base | Common command and completion semantics | Highest-precedence NVMe definition |
| PCIe Transport | Address, register, doorbell, and interrupt binding | Adds PCIe-specific requirements |
| PCI-SIG specifications | Native PCIe capability and transaction semantics | This report covers only NVMe-specific statements present in the supplied source |
02 BARs, MMIO, and doorbell addresses
02.01.NVMe controller registers reside in the memory space designated by BAR0/BAR1. Doorbells begin at 1000h; SQ-tail and CQ-head registers for queue y are spaced using CAP.DSTRD. Figures 3-6 form one address derivation rather than four independent register tables.
- controller
- Controller, the entity that implements the NVMe interface, fetches commands, and reports completions.
- DSTRD
- Doorbell Stride, the CAP field determining spacing between adjacent doorbell registers.
- CAP
- Controller Capabilities, the controller property at offset 00h that reports queue, page-size, timeout, and other capabilities.
- SQ
- Submission Queue, the queue into which the host places commands.
02.02.NVMe controller registers reside in memory space identified by BAR0/BAR1. The host shall use native-width or aligned 32-bit accesses and shall not issue locked accesses; violation produces undefined behavior.
Sources: PCIe Transport 1.4 §3.1
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.1, printed pages 9-10, PDF pages 9-10
| Doorbell or written value | Address or value calculation | Host notification conveyed |
|---|---|---|
| SQ y tail | 1000h + (2y) x (4 << DSTRD) | Host publishes a new SQ tail |
| CQ y head | 1000h + (2y+1) x (4 << DSTRD) | Host publishes a consumed CQ head |
| Doorbell value | Queue pointer | Does not contain the SQE or CQE body |
03 Command exchange between host and controller
03.01.SQE creation, doorbell write, controller fetch, CQE posting, interrupt delivery, and CQ-head update are not names for one event; they are successive ownership handoffs between host and controller. Their order governs both memory ordering and resource reuse.
- The host writes a command to the Submission Queue (SQ).
- The host updates the SQ Tail Doorbell to announce new work.
- The controller retrieves and executes the command, then writes its result to the Completion Queue (CQ).
- The host reads the CQE and updates the CQ Head Doorbell to release consumed entries.
03.02.The command flow writes an SQE, updates the SQ-tail doorbell, lets the controller fetch and execute, posts a CQE, optionally interrupts, processes the CQE, and updates the CQ-head doorbell. A doorbell conveys a pointer, not the command body.
Sources: PCIe Transport 1.4 §3.4
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.4, printed pages 12-13, PDF pages 12-13
| Resource to reuse | When prior use ends | Host observation |
|---|---|---|
| SQ-slot reuse | Controller has consumed the SQE | Completion SQHD assists tracking |
| Command-buffer reuse | Command completed and data is visible | Check command and data direction |
| CQ-slot release | Host completely consumed the CQE | Then write the CQ-head doorbell |
04 Interrupt modes and notification behavior
04.01.Pin-based, single-message MSI, multiple-message MSI, and MSI-X differ in more than performance. They provide different vector counts, masking locations, and capability structures; interrupt coalescing separately controls when multiple completions produce a notification. Figure 9 and Figures 34-46 belong with queue-to-vector mapping.
- MSI-X
- MSI-X, an extended message-signaled-interrupt mechanism with more vectors, per-vector masking, and a table.
- MSI
- Message Signaled Interrupt, a PCI mechanism that delivers an interrupt through a memory-write message.
04.02.Modes are pin-based, single-message MSI, multiple-message MSI, and MSI-X. The specification recommends MSI-X. Coalescing can reduce interrupt rate at the cost of latency, and Admin-CQ interrupts should not be delayed.
- Admin
- Administrative, the control path used to create, configure, query, or manage controllers and queues.
Sources: PCIe Transport 1.4 §3.5
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.5, printed pages 13-16, PDF pages 13-16
| Interrupt mechanism | Notification and masking arrangement | Resources or limits affected |
|---|---|---|
| Pin-based | Legacy shared signaling | Sharing and masking differ |
| Single MSI | One message/vector | Multiple CQs may share a service path |
| Multiple MSI | A set of contiguous messages | Constrained by MME/MMC capability |
| MSI-X | Table-based vectors with independent masks | Preferred by the specification |
05 Configuration space and PCIe error reporting
05.01.Figures 10-67 traverse the Type 0 header, Power Management, MSI/MSI-X, PCIe capability, and AER. Find the capability or extended-capability base before applying offsets. AER status, mask, severity, and header log form one diagnostic set rather than isolated error bits.
- AER
- Advanced Error Reporting, the PCIe capability for classifying, masking, and logging link or transaction errors.
05.02.Section 3.8 defines additional NVMe-controller requirements for the PCI header, Power Management, MSI/MSI-X, PCIe capability, and AER. Original PCI/PCIe field semantics remain governed by PCI-SIG specifications.
- AER
- Advanced Error Reporting, the PCIe capability for classifying, masking, and logging link or transaction errors.
Sources: PCIe Transport 1.4 §3.8.1-3.8.7
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.8.1-3.8.7, printed pages 16-35, PDF pages 16-35
| Reporting layer or capability | Event or resource described | Information needed alongside it |
|---|---|---|
| NVMe CQE status | Command execution result | Read the fields in NVMe command context |
| PCIe Device Status | PCIe Function status summary | Found in PCIe capability |
| AER | Correctable/uncorrectable transport errors | Read status, mask, severity, and header together |
| Power state | Slot limit and device power control | Never choose an NVMe state above the slot power limit |
06 Receiver eye-opening measurement data layout
06.01.The receiver eye-opening measurement log has variable length. Its header describes the whole dataset, while lane descriptors describe individual lanes. Structure levels and length units identify which lane each measurement belongs to.
06.02.The Physical Interface Receiver Eye Opening Measurement log page reports measurements through a header, lane descriptors, and EOM data. The host checks support and size before parsing lanes and parameters.
- EOM
- Eye Opening Measurement, the procedure and log data for measuring a PCIe receiver eye opening.
Sources: PCIe Transport 1.4 §3.9
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.9, printed pages 39-46, PDF pages 39-46
| Measurement region | Measurement or location described | Checks before comparison |
|---|---|---|
| Specific parameter | Selects measurement action and quality/state | Establish request context first |
| Specific identifier | Selects lane/test context | Prevents mixing different measurements |
| Header | Global length and layout | Base for every later offset |
| Lane descriptor | Per-lane boundaries and status | Walk only within returned buffer |
- offset
- offset: a displacement measured from a stated start. It answers “how far from the start,” unlike an index.
Where to continue in the specification
07.01.Use the flow above to frame the problem, then open the corresponding sections for fields and full conditions. The Chinese tutorial also explains every in-scope figure with its takeaway, example, and details.
| Concept to explain | Specification sections |
|---|---|
| How NVMe uses PCIe | PCIe Transport 1.4 §1.2 · PCIe Transport 1.4 §1.3 · PCIe Transport 1.4 §2 |
| BARs, MMIO, and doorbell addresses | PCIe Transport 1.4 §3.1 · PCIe Transport 1.4 §3.1.2.1-3.1.2.2 |
| Command exchange between host and controller | PCIe Transport 1.4 §3.4 · PCIe Transport 1.4 §3.2 |
| Interrupt modes and notification behavior | PCIe Transport 1.4 §3.5 · PCIe Transport 1.4 §3.2 · PCIe Transport 1.4 §Annex A |
| Configuration space and PCIe error reporting | PCIe Transport 1.4 §3.8.1-3.8.7 · PCIe Transport 1.4 §3.7 · PCIe Transport 1.4 §3.6 |
| Receiver eye-opening measurement data layout | PCIe Transport 1.4 §3.9 · PCIe Transport 1.4 §Annex A |
- MMIO
- Memory-Mapped I/O, access to device registers through CPU memory operations.
Connect the ideas in this article
1. What roles do PCIe MMIO properties and host-memory queues play?
08.01.Properties provide controller configuration, status, and queue notification interfaces. Queues carry command and completion entries. Separating the control interface from those structures explains how submission works.
Sources
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.1, printed pages 9-10, PDF pages 9-10
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.2, printed pages 11, PDF pages 11
2. Why is it unsafe to assume adjacent doorbells are always 4 bytes apart?
08.02.CAP.DSTRD determines the spacing: stride is 2^(2+DSTRD) bytes. It is 4 bytes only when DSTRD=0; queue ID also determines the SQ Tail and CQ Head locations.
Sources
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.1.2.1-3.1.2.2, printed pages 10-11, PDF pages 10-11
3. Why must the host inspect the CQ after an interrupt?
08.03.The interrupt is a notification; CQEs contain command identity and completion status. One notification need not correspond to one completion entry, so the host processes valid entries according to queue progress.
Sources
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.5, printed pages 13-16, PDF pages 13-16
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.4, printed pages 12-13, PDF pages 12-13
4. Why distinguish NVMe command status from PCIe error reporting?
08.04.The former describes command processing; the latter describes transport and device-level errors. They can be related, but they address different objects. Success in one layer does not establish the state of every layer.
Sources
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.7, printed pages 16, PDF pages 16
Source: NVME-PCIE-TRANSPORT-1.4, Rev. 1.4, §3.8.1-3.8.7, printed pages 16-35, PDF pages 16-35
Comments