NVMe Self-Study Bank: Doorbells, full queues and wrap-around
NVMe · Q1–328
Q22–Q30
Doorbells, full queues and wrap-around
Doorbells communicate pointer positions; phase tells the host whether a CQ slot belongs to the next completion generation. Together they distinguish wrap-around from an invalid movement and new completions from stale ones.
- CQ
- Completion Queue
Each question is followed by its explanation. Start with the volume overview, then follow the field interpretations, comparisons or processes. Essential conditions and the normal path are visible from the start. All numerical examples are hypothetical. Status is written SCT/SC; h indicates hexadecimal.
- SCT
- Status Code Type
An eight-entry CQ: wrap-around and phase together
- CQ
- Completion Queue
Returning to slot 0 does not make old data valid again; the host must also compare the expected phase.
| Point in time | Host next slot / expected phase | Read result and action |
|---|---|---|
| Initialization | Slot 0, expect P=1; initialize memory P to 0 | No new completion yet |
| End of first traversal | Slot 7, expect P=1 | P=1: consume CQE, wrap to 0, change expectation to 0 |
| Start of second traversal | Slot 0, expect P=0 | Ignore stale P=1; consume only new P=0 |
| Report consumption | CQ Head advances from 7 to 2 | Consumed slots 7, 0, 1: 3 entries, not a retreat by 5 |
- CQE
- Completion Queue Entry
Worked interpretation: Modular distance is (2−7+8) mod 8=3. The doorbell reports position 2, not “free two more slots.” Phase establishes CQE validity; the head doorbell reports consumption to the controller.
- CQE
- Completion Queue Entry
Sources: Base 2.4 §3.3.1 · Base 2.4 §4.2.1, 4.2.3–4.2.4 · PCIe Transport 1.4 §3.1–3.4
Review the terminology when needed
- Controller / namespace
- A controller receives commands and manages access. A namespace is a logical storage space that commands can address. An NVM subsystem contains controllers and nonvolatile storage resources.
- SQ / CQ / SQE / CQE
- Submission and Completion Queues carry command entries (SQEs) and completion entries (CQEs). QID identifies a queue, CID distinguishes outstanding commands in one SQ, and NSID identifies a namespace.
- Register / Identify / Feature / Log
- A register exposes control or state. Identify queries capabilities and attributes; features query or configure operation; log pages report specific state or records. FID, LID, CNS and CSI select features, logs, Identify structures and command sets.
- index / offset / zero-based
- An index selects an entry, usually starting at 0; an offset measures distance from an origin in specified units. A zero-based count encodes count−1, but not every zero-valued field is a count. A Dword is 4 bytes; a byte is 8 bits.
- Scope / reset / retention
- Scope names the affected objects; retention means preserving state. Controller Reset (clearing CC.EN) is one form of Controller Level Reset, or CLR. Different CLR triggers can retain different registers.
Questions in this volume
- Q22 · When should the host update SQ Tail and CQ Head doorbells?
- Q23 · What happens with repeated, apparently backward, out-of-range or wrong-QID doorbells?
- Q24 · How are SQ/CQ doorbell addresses calculated from CAP.DSTRD?
- Q25 · May the host access doorbells of uncreated/deleted queues or a disabled controller?
- Q26 · How should SQ Tail and CQ Head wrap around?
- Q27 · How does the CQ phase change across wrap, and how does the host identify a new completion?
- Q28 · How does CQ Full arise, and may related SQ processing continue?
- Q29 · How does processing resume when the host releases CQ space?
- Q30 · How does the host identify completions from SQs sharing a CQ?
Q22 When should the host update SQ Tail and CQ Head doorbells?
- SQ
- Submission Queue
Follow the explanation
Notify the controller of completed host memory work: SQ Tail submits commands, while CQ Head releases completion slots.
- SQ
- Submission Queue
1. Prepare the operation
Each SQ/CQ has its own doorbell; updating one does not update another.
Verify a valid queue, a ready controller, sufficient free slots and the register address derived from CAP.DSTRD.
SQ Tail points to the next insertion slot; CQ Head points to the next completion to consume. Writes contain new pointer values, not increments.
2. Sequence and completion conditions
Make complete SQEs visible using the platform's memory-ordering requirements before updating Tail. Consume valid CQEs before updating Head; batching is allowed.
The controller learns which commands can be fetched and which CQ slots can be reused. Advancing CQ Head does not submit new I/O.
3. Handle unmet conditions
Releasing unconsumed CQEs risks overwrite. Submitting incomplete SQEs violates usage ordering and has no guaranteed single error CQE.
Build a timeline of SQE writes, Tail updates, CQE phase changes and Head updates; they have distinct roles.
- SQE
- Submission Queue Entry
Sources: Base 2.4 §3.3.1 · PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1
Source locations for this question
Sources: Base 2.4 §3.3.1 · PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1
Q23 What happens with repeated, apparently backward, out-of-range or wrong-QID doorbells?
Follow the explanation
Validate pointer movement without treating a ring index as a monotonically increasing counter.
1. Distinguish the failure conditions
Errors affect the target queue and possibly SQs sharing its CQ. Access to a nonexistent queue differs from an invalid value on an existing queue.
Valid indices lie in 0..size−1; a numerically smaller value may be a valid wrap. Do not assume that rewriting the same value is a harmless no-op: Base 2.4 Figure 153 lists it as a possible cause of Invalid Doorbell Write Value.
For an invalid value on an existing queue with an outstanding Asynchronous Event Request (AER), the controller reports Invalid Doorbell Write Value through that AER. The event information is not a CQE status for the MMIO write. Without an outstanding AER, this rule alone does not establish notification. A write to a nonexistent queue doorbell has undefined behavior.
- MMIO
- Memory-Mapped I/O, mapped device-register access
2. Establish the cause from evidence
Need size, previous pointer, available/consumed slots and QID mapping. The new value alone is insufficient.
Teaching example: for size=8, Tail 6→1 advances three slots, provided three are free and populated. A 6→6 write cannot announce a full ring and may be an invalid write.
3. Outcome and follow-up checks
A valid movement communicates the correct count. Doorbell readback is vendor-specific and is not a verification method.
Compare old pointer, ring size and available space, not just numeric ordering. Correlate the error event with affected-queue command consumption.
First calculate (new−old+size) modulo size and verify that the resulting advance is legal.
4. Check notifications and log records separately
Base §3.3.1.2 specifies an error event for an invalid doorbell value with an outstanding Asynchronous Event Request. An affected SQ may complete consumed commands but consumes no new ones; the host deletes and recreates the queue.
Follow the Error event’s indicated log for additional information; the MMIO write itself has no More bit. Do not invent mandatory logging for undefined nonexistent-QID accesses.
- MMIO
- Memory-Mapped I/O, mapped device-register access
Sources: Base 2.4 §3.3.1 · PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §5.2.2 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1
Source locations for this question
Sources: Base 2.4 §3.3.1 · PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §5.2.2 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1
Q24 How are SQ/CQ doorbell addresses calculated from CAP.DSTRD?
Follow the explanation
Derive the unique doorbell address from the NVMe register base and QID.
1. Establish the source and scope
Offsets are relative to the controller's MMIO base, not a queue's memory address.
PCI BAR0/BAR1 locate the NVMe register space; CAP.DSTRD provides adjacent-doorbell stride.
2. Fields, units and worked interpretation
stride=4×2^DSTRD; SQy offset=1000h+2y×stride; CQy offset=1000h+(2y+1)×stride. Each doorbell itself remains 32 bits.
Obtain the mapped base, calculate stride, select the SQ/CQ expression for QID, then access with a legal MMIO width.
DSTRD=2 and QID=3 give SQ offset 1060h and CQ offset 1070h. Add the controller base for full addresses.
3. Conditions that change the interpretation
Address calculation has no status. A wrong address can select another valid queue or an undefined location; the controller cannot infer host intent.
Match results against created QIDs and the mapped BAR range. DSTRD is an exponent, not a byte count or a direct QID multiplier.
- BAR
- Base Address Register
Sources: Base 2.4 §3.1.4 (CAP, VS) · PCIe Transport 1.4 §3.1–3.4 · PCIe Transport 1.4 §3.8.1 (NVMe configuration access) · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1
Source locations for this question
Sources: Base 2.4 §3.1.4 (CAP, VS) · PCIe Transport 1.4 §3.1–3.4 · PCIe Transport 1.4 §3.8.1 (NVMe configuration access) · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1
Q25 May the host access doorbells of uncreated/deleted queues or a disabled controller?
Follow the explanation
Respect queue lifetime: an address in the doorbell region does not mean a queue currently exists.
1. Establish the queue lifetime
Use each queue after successful creation and retire it after successful deletion or controller reset.
Check host queue-lifetime records, CC.EN and CSTS.RDY; doorbell readback cannot establish queue existence.
SQyTDBL/CQyHDBL announce pointers for valid queues. Admin Q0's environment comes from AQA/ASQ/ACQ and initialization.
Create successfully, use, stop submissions, successfully delete, then stop all use. Reused QIDs after reset still require reconstruction.
2. No fixed response for a nonexistent queue
PCIe §3.1.2 defines writes to nonexistent SQ/CQ doorbells as undefined. RDY=0 violates submission prerequisites. No guaranteed Invalid Queue Identifier or AER follows.
Use Create/Delete completion times to establish the valid interval and check doorbell timestamps against it.
Sources: PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §3.3.1 · Base 2.4 §3.3.3–3.4.1 · Base 2.4 §5.3.3–5.3.4 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1
Source locations for this question
Sources: PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §3.3.1 · Base 2.4 §3.3.3–3.4.1 · Base 2.4 §5.3.3–5.3.4 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1
Q26 How should SQ Tail and CQ Head wrap around?
Follow the explanation
Reuse fixed storage as a ring without recreating the queue each turn.
1. Prepare the operation
The host advances SQ Tail and CQ Head; the controller maintains SQ Head and CQ Tail.
Use the actual N=QSIZE+1 slots, not CAP.MQES as every queue's size.
Advance with (index+1) modulo N. Tail is the next insertion position, Head the next consumption position; equality means empty.
2. Sequence and completion conditions
Wrap N−1 to zero while tracking the real count. Do not overfill the SQ or release unconsumed CQ entries.
With N=8, CQ Head 7→2 consumes slots 7, 0 and 1: three entries, not a backward movement of five.
3. Handle unmet conditions
Valid wrap is not an error. Writing N as an index is out of range; invalid-pointer handling follows Question 23.
Validate modulo relationships among Head, Tail and size. A decreasing doorbell value alone is not a violation.
Sources: Base 2.4 §3.3.1 · PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1
Source locations for this question
Sources: Base 2.4 §3.3.1 · PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1
Q27 How does the CQ phase change across wrap, and how does the host identify a new completion?
Follow the explanation
Let the host distinguish a newly posted completion from data left in the same slot by a previous pass.
1. Prepare the operation
Phase belongs to the CQ ring lifetime; it is neither CID success/failure nor an SQ pointer.
- CID
- Command Identifier, unique among outstanding commands in one SQ
This is a basic PCIe CQE mechanism, not an optional feature. Initialize every slot's phase to zero before creation and initially expect one.
CQE DW3 bit 16 is P. Each new posting in a slot inverts that slot's previous phase.
2. Sequence and completion conditions
Assume a four-entry CQ; the host initially expects P=1.
- Inspect P at the current head
If P does not match, wait: the slot has no new result for this traversal. Do not decode its stale CID or status.
- P matches → process the CQE
Base 2.4 §4.2.1 requires P to be updated in the final write if the controller constructs a CQE with multiple writes. The host also follows platform memory-access rules when reading the other fields. Head visits 0, 1, 2 and 3; new CQEs in this traversal use P=1.
- Head wraps from 3 to 0 → invert expected phase
Expect P=0 in the next traversal. Moving from slot 1 to 2 does not change the expected phase.
Recreating the CQ requires reinitializing phase bits and the host expectation. Four entries are used only to illustrate wrapping.
- CID
- Command Identifier, unique among outstanding commands in one SQ
3. Handle unmet conditions
A phase mismatch means no new completion at that position, not an error status. Do not decode stale CID/SC as a new result.
Exercise at least one wrap and compare Head, expected P and actual P. Reinitialize after queue recreation rather than inheriting the old phase.
Sources: Base 2.4 §4.2.1, 4.2.3–4.2.4 · Base 2.4 §3.3.1 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §4.2.3 · Base 2.4 §5.2.13.1.2 · Base 2.4 §5.2.2 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events)
Source locations for this question
Sources: Base 2.4 §4.2.1, 4.2.3–4.2.4 · Base 2.4 §3.3.1 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §4.2.3 · Base 2.4 §5.2.13.1.2 · Base 2.4 §5.2.2 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events)
Q28 How does CQ Full arise, and may related SQ processing continue?
Follow the explanation
Flow control prevents overwriting unconsumed completions. Fullness usually means completions arrive faster than the host releases slots.
1. Mechanism and scope
It affects the CQ and associated SQs. Processing must continue for SQs not associated with that full CQ.
Full means (Tail+1) modulo N = Head, leaving one slot unused. Tail is controller-internal, not directly readable by the host.
2. Understand it through actions and results
Inspect actual CQ size, Head updates and pending completions, not Number of Queues.
The host validates phases, processes results and advances Head. The controller shall not post another completion to this CQ until a slot is available.
The controller may stop processing further associated SQ entries. This does not require all consumed commands either to stop instantly or to continue unconditionally.
3. Avoid a misleading conclusion
CQ Full is a capacity state, not a required error status. Overwriting unacknowledged entries to make room violates correctness.
If only its associated SQs stall, inspect consumption first. If independent SQs stop because of this full CQ, check the requirement to continue their processing.
Sources: Base 2.4 §3.3.1 · Base 2.4 §3.4.1–3.4.5 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §4.2.3 · Base 2.4 §5.2.13.1.2 · Base 2.4 §5.2.2 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events)
Source locations for this question
Sources: Base 2.4 §3.3.1 · Base 2.4 §3.4.1–3.4.5 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §4.2.3 · Base 2.4 §5.2.13.1.2 · Base 2.4 §5.2.2 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events)
Q29 How does processing resume when the host releases CQ space?
Follow the explanation
Resume work after capacity becomes available rather than treating ordinary CQ Full as a recreation error.
1. Prepare the operation
Released slots serve completions from associated SQs; queue allocations and namespace capabilities do not change.
Verify a live queue and no independent stop condition such as reset, CFS or Processing Paused.
The host writes the new CQ Head, allowing the controller to reuse released slots with correct phases for subsequent completions.
2. Sequence and completion conditions
Consume CQEs, update Head, allow slots to become available, then post waiting completions and continue affected work under normal scheduling and other constraints.
Released slots are reusable without recreation. There is no universal fixed latency from a Head update to the next CQE.
3. Handle unmet conditions
Correct release has no special success CQE. An invalid Head remains a doorbell-usage issue, not a Create Queue error.
Verify legal reuse of released slots and compare progress on associated and independent SQs.
Sources: Base 2.4 §3.3.1 · PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §3.4.1–3.4.5 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §4.2.3 · Base 2.4 §5.2.13.1.2 · Base 2.4 §5.2.2 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events)
Source locations for this question
Sources: Base 2.4 §3.3.1 · PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §3.4.1–3.4.5 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §4.2.3 · Base 2.4 §5.2.13.1.2 · Base 2.4 §5.2.2 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events)
Q30 How does the host identify completions from SQs sharing a CQ?
Follow the explanation
Deliver completion to the correct request. CID uniqueness is required among outstanding commands of the same SQ.
1. Establish the source and scope
Matching includes controller, SQ lifetime, SQID and CID. Controller/lifetime are host context, not additional CQE fields.
- SQID
- Submission Queue Identifier
Use the creation-time SQ-to-CQ map and outstanding table, not Identify for per-command locations.
2. Fields, units and worked interpretation
SQID identifies the SQ and CID its command. SQHD reports consumption progress, not the original slot of that command.
- SQID
- Submission Queue Identifier
Validate phase first, match SQID/CID, process status/data, update that SQ's head information, then release the CQ slot.
SQ1/CID 9 and SQ2/CID 9 are matched correctly even if they complete in reverse order.
3. Conditions that change the interpretation
Reusing an outstanding CID in one SQ may return Command ID Conflict (0/03h). The conflict-search extent is implementation-specific; host uniqueness obligations remain.
Check that SQID is associated with this CQ and CID belongs to its current queue lifetime.
Sources: Base 2.4 §4.2.1, 4.2.3–4.2.4 · Base 2.4 §3.3.1 · Base 2.4 §4.1.1 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §4.2.3 · Base 2.4 §5.2.13.1.2 · Base 2.4 §5.2.2 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events)
Source locations for this question
Sources: Base 2.4 §4.2.1, 4.2.3–4.2.4 · Base 2.4 §3.3.1 · Base 2.4 §4.1.1 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §4.2.3 · Base 2.4 §5.2.13.1.2 · Base 2.4 §5.2.2 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events)
Source locations and existing figure guides
Base printed page = PDF page−26; the other two use identical numbers. Locations follow the supplied PDF body and retain figure numbers. Shared pages contribute only the relevant definitions, excluding Fabrics and PCIe link/packet content.
- Base 2.4 · §3.1.4 (CAP, VS)
Printed pages 54–59 · PDF 80–85 · Figure 36–37 - Base 2.4 · §3.3.1
Printed pages 88–91 · PDF 114–117 · Figure 73–74 - Base 2.4 · §3.4.1–3.4.5
Printed pages 101–105 · PDF 127–131 · Figure 80–81 - Base 2.4 · §3.3.3–3.4.1
Printed pages 101 · PDF 127 - Base 2.4 · §3.7.1–3.7.4
Printed pages 120–124 · PDF 146–150 - Base 2.4 · §4.1.1
Printed pages 139–142 · PDF 165–168 · Figure 92–93 - Base 2.4 · §4.2.1, 4.2.3–4.2.4
Printed pages 144–157 · PDF 170–183 · Figure 97–105, 109 - Base 2.4 · §4.2.3
Printed pages 145–155 · PDF 171–181 · Figure 101–105 - Base 2.4 · §5.2.2
Printed pages 183–190 · PDF 209–216 · Figure 150–156 - Base 2.4 · §5.2.13.1.2
Printed pages 218–220 · PDF 244–246 · Figure 212 - Base 2.4 · §5.2.13.1.14 (header, reset, hardware, Set Feature events)
Printed pages 244–256, 258, 262–264 · PDF 270–282, 284, 288–290 · Figure 232–244, 246, 252–253 - Base 2.4 · §5.3.3–5.3.4
Printed pages 531–532 · PDF 557–558 · Figure 580–583 - Base 2.4 · §9.1–9.6.1
Printed pages 825–826 · PDF 851–852 - PCIe Transport 1.4 · §3.1–3.4
Printed pages 9–13 · PDF 9–13 · Figure 3–8 - PCIe Transport 1.4 · §3.8.1 (NVMe configuration access)
Printed pages 16–18 · PDF 16–18 · Figure 10–17
When you need a field guide
Existing figure explanations have canonical locations; use these links instead of duplicating the same guide.
- Base 2.4 Figure 101 · Completion Queue Entry: Status Field
- Base 2.4 Figure 104 · Status Code – Command Specific Status Values
- Base 2.4 Figure 36 · Offset 0h: CAP – Controller Capabilities
- Base 2.4 Figure 93 · Common Command Format
- Base 2.4 Figure 97 · Common Completion Queue Entry Layout – Admin and All I/O Command Sets
Original documents used
- NVM Express Base Specification · Revision 2.4 · 2026-07-31
NVM-Express-Base-Specification-Revision-2.4-Ratified-2026.07.31.pdf - NVM Express NVM Command Set Specification · Revision 1.3 · 2026-07-31
NVM-Express-NVM-Command-Set-Specification-Revision-1.3-Ratified-2026.07.31.pdf - NVM Express NVMe over PCIe Transport Specification · Revision 1.4 · 2026-07-31
NVM-Express-NVMe-over-PCIe-Transport-Specification-Revision-1.4-Ratified-2026.07.31.pdf


Comments