NVMe Self-Study Bank: Doorbells, full queues and wrap-around

NVMe · Q1–328

NVM Express Base 2.4NVM Command Set 1.3NVMe over PCIe Transport 1.4

Q22–Q30

Doorbells, full queues and wrap-around

Doorbells communicate pointer positions; phase tells the host whether a CQ slot belongs to the next completion generation. Together they distinguish wrap-around from an invalid movement and new completions from stale ones.

CQ
Completion Queue

Each question is followed by its explanation. Start with the volume overview, then follow the field interpretations, comparisons or processes. Essential conditions and the normal path are visible from the start. All numerical examples are hypothetical. Status is written SCT/SC; h indicates hexadecimal.

SCT
Status Code Type

An eight-entry CQ: wrap-around and phase together

CQ
Completion Queue

Returning to slot 0 does not make old data valid again; the host must also compare the expected phase.

Point in timeHost next slot / expected phaseRead result and action
InitializationSlot 0, expect P=1; initialize memory P to 0No new completion yet
End of first traversalSlot 7, expect P=1P=1: consume CQE, wrap to 0, change expectation to 0
Start of second traversalSlot 0, expect P=0Ignore stale P=1; consume only new P=0
Report consumptionCQ Head advances from 7 to 2Consumed slots 7, 0, 1: 3 entries, not a retreat by 5
CQE
Completion Queue Entry

Worked interpretation: Modular distance is (2−7+8) mod 8=3. The doorbell reports position 2, not “free two more slots.” Phase establishes CQE validity; the head doorbell reports consumption to the controller.

CQE
Completion Queue Entry

Sources: Base 2.4 §3.3.1 · Base 2.4 §4.2.1, 4.2.3–4.2.4 · PCIe Transport 1.4 §3.1–3.4

Review the terminology when needed
Controller / namespace
A controller receives commands and manages access. A namespace is a logical storage space that commands can address. An NVM subsystem contains controllers and nonvolatile storage resources.
SQ / CQ / SQE / CQE
Submission and Completion Queues carry command entries (SQEs) and completion entries (CQEs). QID identifies a queue, CID distinguishes outstanding commands in one SQ, and NSID identifies a namespace.
Register / Identify / Feature / Log
A register exposes control or state. Identify queries capabilities and attributes; features query or configure operation; log pages report specific state or records. FID, LID, CNS and CSI select features, logs, Identify structures and command sets.
index / offset / zero-based
An index selects an entry, usually starting at 0; an offset measures distance from an origin in specified units. A zero-based count encodes count−1, but not every zero-valued field is a count. A Dword is 4 bytes; a byte is 8 bits.
Scope / reset / retention
Scope names the affected objects; retention means preserving state. Controller Reset (clearing CC.EN) is one form of Controller Level Reset, or CLR. Different CLR triggers can retain different registers.

Questions in this volume

  1. Q22 · When should the host update SQ Tail and CQ Head doorbells?
  2. Q23 · What happens with repeated, apparently backward, out-of-range or wrong-QID doorbells?
  3. Q24 · How are SQ/CQ doorbell addresses calculated from CAP.DSTRD?
  4. Q25 · May the host access doorbells of uncreated/deleted queues or a disabled controller?
  5. Q26 · How should SQ Tail and CQ Head wrap around?
  6. Q27 · How does the CQ phase change across wrap, and how does the host identify a new completion?
  7. Q28 · How does CQ Full arise, and may related SQ processing continue?
  8. Q29 · How does processing resume when the host releases CQ space?
  9. Q30 · How does the host identify completions from SQs sharing a CQ?

Q22 When should the host update SQ Tail and CQ Head doorbells?

SQ
Submission Queue
Follow the explanation

Notify the controller of completed host memory work: SQ Tail submits commands, while CQ Head releases completion slots.

SQ
Submission Queue

1. Prepare the operation

Each SQ/CQ has its own doorbell; updating one does not update another.

Verify a valid queue, a ready controller, sufficient free slots and the register address derived from CAP.DSTRD.

SQ Tail points to the next insertion slot; CQ Head points to the next completion to consume. Writes contain new pointer values, not increments.

2. Sequence and completion conditions

Make complete SQEs visible using the platform's memory-ordering requirements before updating Tail. Consume valid CQEs before updating Head; batching is allowed.

The controller learns which commands can be fetched and which CQ slots can be reused. Advancing CQ Head does not submit new I/O.

3. Handle unmet conditions

Releasing unconsumed CQEs risks overwrite. Submitting incomplete SQEs violates usage ordering and has no guaranteed single error CQE.

Build a timeline of SQE writes, Tail updates, CQE phase changes and Head updates; they have distinct roles.

SQE
Submission Queue Entry

Sources: Base 2.4 §3.3.1 · PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1

Back to questions

Q23 What happens with repeated, apparently backward, out-of-range or wrong-QID doorbells?

Follow the explanation

Validate pointer movement without treating a ring index as a monotonically increasing counter.

1. Distinguish the failure conditions

Errors affect the target queue and possibly SQs sharing its CQ. Access to a nonexistent queue differs from an invalid value on an existing queue.

Valid indices lie in 0..size−1; a numerically smaller value may be a valid wrap. Do not assume that rewriting the same value is a harmless no-op: Base 2.4 Figure 153 lists it as a possible cause of Invalid Doorbell Write Value.

For an invalid value on an existing queue with an outstanding Asynchronous Event Request (AER), the controller reports Invalid Doorbell Write Value through that AER. The event information is not a CQE status for the MMIO write. Without an outstanding AER, this rule alone does not establish notification. A write to a nonexistent queue doorbell has undefined behavior.

MMIO
Memory-Mapped I/O, mapped device-register access

2. Establish the cause from evidence

Need size, previous pointer, available/consumed slots and QID mapping. The new value alone is insufficient.

Teaching example: for size=8, Tail 6→1 advances three slots, provided three are free and populated. A 6→6 write cannot announce a full ring and may be an invalid write.

3. Outcome and follow-up checks

A valid movement communicates the correct count. Doorbell readback is vendor-specific and is not a verification method.

Compare old pointer, ring size and available space, not just numeric ordering. Correlate the error event with affected-queue command consumption.

First calculate (new−old+size) modulo size and verify that the resulting advance is legal.

4. Check notifications and log records separately

Base §3.3.1.2 specifies an error event for an invalid doorbell value with an outstanding Asynchronous Event Request. An affected SQ may complete consumed commands but consumes no new ones; the host deletes and recreates the queue.

Follow the Error event’s indicated log for additional information; the MMIO write itself has no More bit. Do not invent mandatory logging for undefined nonexistent-QID accesses.

MMIO
Memory-Mapped I/O, mapped device-register access

Sources: Base 2.4 §3.3.1 · PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §5.2.2 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1

Back to questions

Q24 How are SQ/CQ doorbell addresses calculated from CAP.DSTRD?

Follow the explanation

Derive the unique doorbell address from the NVMe register base and QID.

1. Establish the source and scope

Offsets are relative to the controller's MMIO base, not a queue's memory address.

PCI BAR0/BAR1 locate the NVMe register space; CAP.DSTRD provides adjacent-doorbell stride.

2. Fields, units and worked interpretation

stride=4×2^DSTRD; SQy offset=1000h+2y×stride; CQy offset=1000h+(2y+1)×stride. Each doorbell itself remains 32 bits.

Obtain the mapped base, calculate stride, select the SQ/CQ expression for QID, then access with a legal MMIO width.

DSTRD=2 and QID=3 give SQ offset 1060h and CQ offset 1070h. Add the controller base for full addresses.

3. Conditions that change the interpretation

Address calculation has no status. A wrong address can select another valid queue or an undefined location; the controller cannot infer host intent.

Match results against created QIDs and the mapped BAR range. DSTRD is an exponent, not a byte count or a direct QID multiplier.

BAR
Base Address Register

Sources: Base 2.4 §3.1.4 (CAP, VS) · PCIe Transport 1.4 §3.1–3.4 · PCIe Transport 1.4 §3.8.1 (NVMe configuration access) · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1

Back to questions

Q25 May the host access doorbells of uncreated/deleted queues or a disabled controller?

Follow the explanation

Respect queue lifetime: an address in the doorbell region does not mean a queue currently exists.

1. Establish the queue lifetime

Use each queue after successful creation and retire it after successful deletion or controller reset.

Check host queue-lifetime records, CC.EN and CSTS.RDY; doorbell readback cannot establish queue existence.

SQyTDBL/CQyHDBL announce pointers for valid queues. Admin Q0's environment comes from AQA/ASQ/ACQ and initialization.

Create successfully, use, stop submissions, successfully delete, then stop all use. Reused QIDs after reset still require reconstruction.

2. No fixed response for a nonexistent queue

PCIe §3.1.2 defines writes to nonexistent SQ/CQ doorbells as undefined. RDY=0 violates submission prerequisites. No guaranteed Invalid Queue Identifier or AER follows.

Use Create/Delete completion times to establish the valid interval and check doorbell timestamps against it.

Sources: PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §3.3.1 · Base 2.4 §3.3.3–3.4.1 · Base 2.4 §5.3.3–5.3.4 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1

Back to questions

Q26 How should SQ Tail and CQ Head wrap around?

Follow the explanation

Reuse fixed storage as a ring without recreating the queue each turn.

1. Prepare the operation

The host advances SQ Tail and CQ Head; the controller maintains SQ Head and CQ Tail.

Use the actual N=QSIZE+1 slots, not CAP.MQES as every queue's size.

Advance with (index+1) modulo N. Tail is the next insertion position, Head the next consumption position; equality means empty.

2. Sequence and completion conditions

Wrap N−1 to zero while tracking the real count. Do not overfill the SQ or release unconsumed CQ entries.

With N=8, CQ Head 7→2 consumes slots 7, 0 and 1: three entries, not a backward movement of five.

3. Handle unmet conditions

Valid wrap is not an error. Writing N as an index is out of range; invalid-pointer handling follows Question 23.

Validate modulo relationships among Head, Tail and size. A decreasing doorbell value alone is not a violation.

Sources: Base 2.4 §3.3.1 · PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events) · Base 2.4 §9.1–9.6.1

Back to questions

Q27 How does the CQ phase change across wrap, and how does the host identify a new completion?

Follow the explanation

Let the host distinguish a newly posted completion from data left in the same slot by a previous pass.

1. Prepare the operation

Phase belongs to the CQ ring lifetime; it is neither CID success/failure nor an SQ pointer.

CID
Command Identifier, unique among outstanding commands in one SQ

This is a basic PCIe CQE mechanism, not an optional feature. Initialize every slot's phase to zero before creation and initially expect one.

CQE DW3 bit 16 is P. Each new posting in a slot inverts that slot's previous phase.

2. Sequence and completion conditions

Phase changes per traversal, not per CQE

Assume a four-entry CQ; the host initially expects P=1.

  1. Inspect P at the current head

    If P does not match, wait: the slot has no new result for this traversal. Do not decode its stale CID or status.

  2. P matches → process the CQE

    Base 2.4 §4.2.1 requires P to be updated in the final write if the controller constructs a CQE with multiple writes. The host also follows platform memory-access rules when reading the other fields. Head visits 0, 1, 2 and 3; new CQEs in this traversal use P=1.

  3. Head wraps from 3 to 0 → invert expected phase

    Expect P=0 in the next traversal. Moving from slot 1 to 2 does not change the expected phase.

Recreating the CQ requires reinitializing phase bits and the host expectation. Four entries are used only to illustrate wrapping.

CID
Command Identifier, unique among outstanding commands in one SQ

3. Handle unmet conditions

A phase mismatch means no new completion at that position, not an error status. Do not decode stale CID/SC as a new result.

Exercise at least one wrap and compare Head, expected P and actual P. Reinitialize after queue recreation rather than inheriting the old phase.

Sources: Base 2.4 §4.2.1, 4.2.3–4.2.4 · Base 2.4 §3.3.1 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §4.2.3 · Base 2.4 §5.2.13.1.2 · Base 2.4 §5.2.2 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events)

Back to questions

Q28 How does CQ Full arise, and may related SQ processing continue?

Follow the explanation

Flow control prevents overwriting unconsumed completions. Fullness usually means completions arrive faster than the host releases slots.

1. Mechanism and scope

It affects the CQ and associated SQs. Processing must continue for SQs not associated with that full CQ.

Full means (Tail+1) modulo N = Head, leaving one slot unused. Tail is controller-internal, not directly readable by the host.

2. Understand it through actions and results

Inspect actual CQ size, Head updates and pending completions, not Number of Queues.

The host validates phases, processes results and advances Head. The controller shall not post another completion to this CQ until a slot is available.

The controller may stop processing further associated SQ entries. This does not require all consumed commands either to stop instantly or to continue unconditionally.

3. Avoid a misleading conclusion

CQ Full is a capacity state, not a required error status. Overwriting unacknowledged entries to make room violates correctness.

If only its associated SQs stall, inspect consumption first. If independent SQs stop because of this full CQ, check the requirement to continue their processing.

Sources: Base 2.4 §3.3.1 · Base 2.4 §3.4.1–3.4.5 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §4.2.3 · Base 2.4 §5.2.13.1.2 · Base 2.4 §5.2.2 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events)

Back to questions

Q29 How does processing resume when the host releases CQ space?

Follow the explanation

Resume work after capacity becomes available rather than treating ordinary CQ Full as a recreation error.

1. Prepare the operation

Released slots serve completions from associated SQs; queue allocations and namespace capabilities do not change.

Verify a live queue and no independent stop condition such as reset, CFS or Processing Paused.

The host writes the new CQ Head, allowing the controller to reuse released slots with correct phases for subsequent completions.

2. Sequence and completion conditions

Consume CQEs, update Head, allow slots to become available, then post waiting completions and continue affected work under normal scheduling and other constraints.

Released slots are reusable without recreation. There is no universal fixed latency from a Head update to the next CQE.

3. Handle unmet conditions

Correct release has no special success CQE. An invalid Head remains a doorbell-usage issue, not a Create Queue error.

Verify legal reuse of released slots and compare progress on associated and independent SQs.

Sources: Base 2.4 §3.3.1 · PCIe Transport 1.4 §3.1–3.4 · Base 2.4 §3.4.1–3.4.5 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §4.2.3 · Base 2.4 §5.2.13.1.2 · Base 2.4 §5.2.2 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events)

Back to questions

Q30 How does the host identify completions from SQs sharing a CQ?

Follow the explanation

Deliver completion to the correct request. CID uniqueness is required among outstanding commands of the same SQ.

1. Establish the source and scope

Matching includes controller, SQ lifetime, SQID and CID. Controller/lifetime are host context, not additional CQE fields.

SQID
Submission Queue Identifier

Use the creation-time SQ-to-CQ map and outstanding table, not Identify for per-command locations.

2. Fields, units and worked interpretation

SQID identifies the SQ and CID its command. SQHD reports consumption progress, not the original slot of that command.

SQID
Submission Queue Identifier

Validate phase first, match SQID/CID, process status/data, update that SQ's head information, then release the CQ slot.

SQ1/CID 9 and SQ2/CID 9 are matched correctly even if they complete in reverse order.

3. Conditions that change the interpretation

Reusing an outstanding CID in one SQ may return Command ID Conflict (0/03h). The conflict-search extent is implementation-specific; host uniqueness obligations remain.

Check that SQID is associated with this CQ and CID belongs to its current queue lifetime.

Sources: Base 2.4 §4.2.1, 4.2.3–4.2.4 · Base 2.4 §3.3.1 · Base 2.4 §4.1.1 · Base 2.4 §3.7.1–3.7.4 · Base 2.4 §4.2.3 · Base 2.4 §5.2.13.1.2 · Base 2.4 §5.2.2 · Base 2.4 §5.2.13.1.14 (header, reset, hardware, Set Feature events)

Back to questions

Source locations and existing figure guides

Base printed page = PDF page−26; the other two use identical numbers. Locations follow the supplied PDF body and retain figure numbers. Shared pages contribute only the relevant definitions, excluding Fabrics and PCIe link/packet content.

  • Base 2.4 · §3.1.4 (CAP, VS)
    Printed pages 54–59 · PDF 80–85 · Figure 36–37
  • Base 2.4 · §3.3.1
    Printed pages 88–91 · PDF 114–117 · Figure 73–74
  • Base 2.4 · §3.4.1–3.4.5
    Printed pages 101–105 · PDF 127–131 · Figure 80–81
  • Base 2.4 · §3.3.3–3.4.1
    Printed pages 101 · PDF 127
  • Base 2.4 · §3.7.1–3.7.4
    Printed pages 120–124 · PDF 146–150
  • Base 2.4 · §4.1.1
    Printed pages 139–142 · PDF 165–168 · Figure 92–93
  • Base 2.4 · §4.2.1, 4.2.3–4.2.4
    Printed pages 144–157 · PDF 170–183 · Figure 97–105, 109
  • Base 2.4 · §4.2.3
    Printed pages 145–155 · PDF 171–181 · Figure 101–105
  • Base 2.4 · §5.2.2
    Printed pages 183–190 · PDF 209–216 · Figure 150–156
  • Base 2.4 · §5.2.13.1.2
    Printed pages 218–220 · PDF 244–246 · Figure 212
  • Base 2.4 · §5.2.13.1.14 (header, reset, hardware, Set Feature events)
    Printed pages 244–256, 258, 262–264 · PDF 270–282, 284, 288–290 · Figure 232–244, 246, 252–253
  • Base 2.4 · §5.3.3–5.3.4
    Printed pages 531–532 · PDF 557–558 · Figure 580–583
  • Base 2.4 · §9.1–9.6.1
    Printed pages 825–826 · PDF 851–852
  • PCIe Transport 1.4 · §3.1–3.4
    Printed pages 9–13 · PDF 9–13 · Figure 3–8
  • PCIe Transport 1.4 · §3.8.1 (NVMe configuration access)
    Printed pages 16–18 · PDF 16–18 · Figure 10–17

When you need a field guide

Existing figure explanations have canonical locations; use these links instead of duplicating the same guide.

Original documents used
  • NVM Express Base Specification · Revision 2.4 · 2026-07-31
    NVM-Express-Base-Specification-Revision-2.4-Ratified-2026.07.31.pdf
  • NVM Express NVM Command Set Specification · Revision 1.3 · 2026-07-31
    NVM-Express-NVM-Command-Set-Specification-Revision-1.3-Ratified-2026.07.31.pdf
  • NVM Express NVMe over PCIe Transport Specification · Revision 1.4 · 2026-07-31
    NVM-Express-NVMe-over-PCIe-Transport-Specification-Revision-1.4-Ratified-2026.07.31.pdf
Jia-Chang

Jia-Chang

Human

Comments

  Write a comment ...