PCIe Chapter 17: Data Link Initialization, Flow Control, and Replay

FC (Flow Control) tracks buffer space at the adjacent receiver. Link replay handles transmission integrity, while a Completion (transaction response) belongs to the request itself. We place all three along one path so their participants and guarantees stay clear.

Author-drawn teaching diagram based on PCI Express Base 6.0. Exchanges unrelated to this lesson are omitted.

TLP: Transaction Layer Packet

Does link-level communication mean TLP transmission can start?

TLP: Transaction Layer Packet

The selected VC must finish Flow Control initialization. Start with Figure 3-2 (p.265): after the Physical Link becomes operational, DL_Feature exchanges features when applicable, then DL_Init initializes default VC0. Chapter 16 covers software enablement of additional VCs.

VC: Virtual Channel

Read the DL_Init rules in section 3.2.1 (p.267): FC_INIT1 reports DL_Down, while FC_INIT2 already reports DL_Up. Nevertheless, section 3.4.1 (p.273) still blocks TLP transmission on that VC throughout FC_INIT2 until initialization exits. Follow the FC state; the DL_Up output alone does not establish permission to transmit.

VC: Virtual Channel
FC: Flow Control

In Non-Flit Mode, InitFC1 and InitFC2 each exchange Posted, Non-Posted, and Completion resource information. The first phase records advertised credits; the second completes confirmation. Other VCs that have already initialized may keep operating under their own conditions. The following example starts with VC0 initialized and both sides DL_Active.

Reading route: §3.2 Data Link Control and Management State Machine, Figure 3-2 Data Link Control and Management State Machine, pp.264–265; §3.2.1, pp.266–267; §3.4 Flow Control Initialization Protocol and §3.4.1 Flow Control Initialization State Machine Rules, pp.270–273; §2.6.1, p.241. Document and PDF pages match.

Which credit class does this packet need?

Table 2-44 (p.237) separates Posted, Non-Posted, and Completion resources into Header and Data, producing six classes. PH means Posted Header Credit and PD means Posted Data Credit; Non-Posted and Completion use NPH/NPD and CplH/CplD.

Then use Table 2-45 on the same page to identify the packet type. Data presence alone is insufficient: I/O Write and ordinary Memory Write both carry data, but the former is Non-Posted and the latter is Posted.

Packet	Resources	Response relationship
Memory Write Request	1 PH + n PD	No ordinary Completion
Memory Read Request	1 NPH	Returned data uses a separate Completion
Completion with Data	1 CplH + n CplD	Check these resources in the return direction

A basic Data credit is 4 DW, or 16 bytes. Round the individual TLP's data length upward to obtain n. The following 20-byte write therefore needs 1 PH and 2 PD. These are protocol accounting units, not a required physical buffer layout.

DW: Double Word, a four-byte unit

Reading route: §2.6 Ordering and Receive Buffer Flow Control, pp.235–236; §2.6.1 Flow Control (FC) Rules, Table 2-44 Flow Control Credit Types and Table 2-45 TLP Flow Control Credit Consumption, p.237. This is ordinary Non-Flit accounting; Flit Shared FC has additional granularity rules.

Why can a smaller UpdateFC value provide more available credit?

CREDIT_LIMIT is the cumulative limit advertised by the receiver; CREDITS_CONSUMED is the transmitter's cumulative use. Both wrap at their field widths. The latest UpdateFC value alone is not the available balance.

【模擬資料】Teaching assumptions, not measurements of a particular device or platform. Root Port 00:01.0 is preparing a 20-byte Memory Write to Endpoint 01:00.0 on TC0/VC0. The Link operates at 8.0 GT/s in Non-Flit Mode. Neither side supports Scaled Flow Control. Address, BAR, Memory Space Enable, and other transmission conditions are established; the receiver's Rx_MPS_Limit is 128 bytes.

BAR: Base Address Register

During initialization, the Endpoint advertised HdrFC=08h and DataFC=008h for Posted traffic in this direction: both finite. All other required classes also initialized. Start after substantial prior traffic, with no other Posted traffic changing the following counters during this interval. These are specification conceptual counters that a complete trace can reconstruct, not mandated readable Configuration Space registers.

Select the field width before decoding the update

The no-Scaled-Flow-Control row of Table 2-47 (p.241) uses 8-bit Header and 12-bit Data counters, with moduli 100h and 1000h. Locate VC, HdrFC, and DataFC in Figure 3-11 (p.284). Establish that this is UpdateFC-P for this direction's VC0, not another VC or the reverse-direction account.

Then track section 2.6.1.1's two quantities. The consumed column is CREDITS_CONSUMED; the limit column is CREDIT_LIMIT. Their modular difference gives available credit within the specification's outstanding-unused-credit bounds.

Observation	Consumed counter	Latest limit	Available credit
Waiting to send	`PH=FEh; PD=FFEh`	`PH=FFh; PD=FFEh`	1 PH / 0 PD
Update A received	`PH=FEh; PD=FFEh`	`PH=00h; PD=000h`	2 PH / 2 PD
New TLP passes the FC gate	`PH=FFh; PD=000h`	`PH=00h; PD=000h`	1 PH / 0 PD
Valid Ack covers this TLP	`PH=FFh; PD=000h`	`PH=00h; PD=000h`	1 PH / 0 PD
Update B received	`PH=FFh; PD=000h`	`PH=01h; PD=002h`	2 PH / 2 PD

The first row has 1 PH and 0 PD, so the write waits. Update A gives (000h − FFEh) mod 1000h = 002h, or 2 PD, and (00h − FEh) mod 100h = 02h, or 2 PH. The smaller zero-valued fields reflect cumulative-counter wrap, not withdrawal of previously advertised space.

What do Ack and UpdateFC change after normal transmission?

The new TLP consumes 1 PH and 2 PD once when it passes the FC gate. Consumed PH becomes FFh; consumed PD wraps from FFEh plus 2 to 000h. The available balance becomes 1 PH / 0 PD. A later valid Ack permits release of acknowledged replay copies but does not change the FC counters.

Author-created normal timeline. Vertical spacing has no time scale. This example places Ack before Update B; the mechanisms do not require that order. Posted Memory Write has no ordinary Completion.

When the receiver processes the data and makes the corresponding space available, it advances CREDITS_ALLOCATED and advertises Update B. Only after the transmitter receives that valid update does its limit become PH=01h, PD=002h, restoring 2 PH / 2 PD. Section 2.6.1.1 expressly forbids interpreting credit return as transaction completion or system visibility.

How does a zero UpdateFC differ from Infinite Credit?

The initialization advertisement 08h/008h established finite accounting here; subsequent zeros remain cumulative-counter values. Infinite Credit is a different state established by its initialization encoding. It prevents throttling for that credit type, while other transmission conditions remain. It does not imply an infinite physical receive buffer.

Connected sources: Table 2-46 Minimum Initial Flow Control Advertisements, pp.238–239; §2.6.1, pp.240–241; §2.6.1.1 FC Information Tracked by Transmitter, pp.242–246, Equation 2-1 CREDITS_CONSUMED and Equations 2-10/2-11 for transmission gating; §2.6.1.2 FC Information Tracked by Receiver, pp.246–247; §3.5, pp.280–281, Figure 3-11 Data Link Layer Packet Format for UpdateFC, p.284. Document and PDF pages match.

Does Replay consume a second set of credits?

No. In an alternative to the normal example, suppose the first transmission fails LCRC and a Nak causes replay. The Retry Buffer resends the same TLP, which already passed the FC gate. Section 2.6 states that FC accounting is unaware of repeated Data Link transmissions.

LCRC: Link Cyclic Redundancy Check

Replay itself leaves CREDITS_CONSUMED at PH=FFh, PD=000h, without consuming another 1 PH and 2 PD. A valid concurrent UpdateFC can still change CREDIT_LIMIT. Section 3.6.2.1 also explains that Retry Buffer entries have already been allocated receive space by the transmitter's FC logic. A new write generated by the upper layer is a different TLP and needs a new credit check and consumption.

Neither Ack nor UpdateFC substitutes for Completion. The example's Posted Memory Write has no ordinary Completion. For a Memory Read, NPH covers the Request, while CplH/CplD are checked in the return direction. The requester verifies Requester ID, Tag, Status, and returned data coverage. See Chapter 5 for matching and completeness, and Chapter 8 for detailed Replay rules.

LCRC (Link Cyclic Redundancy Check) and sequence numbers protect an adjacent Link. ECRC (End-to-End CRC) adds integrity checking across intermediate components under its support/enable conditions. Data Poisoning marks known-unreliable data without repairing it. These mechanisms address different responsibilities from receive-buffer accounting.

ECRC: End-to-End Cyclic Redundancy Check

Reading route: §2.6, p.236; §3.6.2 LCRC, Sequence Number, and Retry Management (TLP Transmitter), p.288; §3.6.2.1, pp.293–294; §3.6.2.2 Handling of Received DLLPs (Non-Flit Mode), pp.297–298; §3.6.3 LCRC and Sequence Number (TLP Receiver) (Non-Flit Mode), pp.300–301; §2.7 End-to-End Data Integrity, p.252.

What changes with Scaled FC or Flit Mode?

Scaled FC (Scaled Flow Control) activation is established through Data Link Feature Exchange; initialization supplies HdrScale/DataScale. Table 2-47 changes effective counter widths with scaling. The example's fixed 8-bit/12-bit modular arithmetic cannot simply be retained.

Flit Mode adds Shared FC and dedicated credits, with additional rules for Completion merging, accounting granularity across contiguous TLPs, and the applicable gate. One transmitted Flit does not mean one credit.

Protection also moves to Flit-level CRC, FEC (Forward Error Correction), and Replay. Section 3.1 calls the Flit link CRC LCRC; section 3.6.2.3 (p.300) explicitly excludes Ack/Nak DLLPs in Flit Mode. Do not copy the example's return-packet formats into a Flit trace. Continue to Chapter 18 for full Flit carriage and protection.

Reading route: §3.3 Data Link Feature Exchange, pp.268–269; Table 2-47, p.241; §2.6, pp.235–236; §2.6.1, pp.241–242; §3.1 Data Link Layer Overview, pp.263–264; §3.6.2.3, p.300.

How does Flit Mode indicate which credit a TLP uses?

First confirm the Link is operating in Flit Mode, then read Figure 2-77 in §2.2.10.3. The 4-byte Flit Mode Local TLP Prefix precedes a TLP. Byte 0 is 8Dh; byte 1 bit 7 is TLP Uses Dedicated Credits. This Prefix is permitted only in Flit Mode.

Teaching example: Assume this Link uses Shared Flow Control (Shared FC). The following are three independent TLP cases. Each still has to pass Flow Control checks for its type, length, and the credits available at that time.

On a phone, scroll the table horizontally to see all columns.

Local Prefix	Bit 7	Credit used by this TLP
`8D 80 00 00`	1	Dedicated credit
`8D 00 00 00`	0	Shared credit
No such Prefix	Absent	Shared credit

All other bits in the illustrated Prefixes are zero. The specification permits this Prefix on Flit Mode TLPs but strongly recommends applying it only where specifically required. Section 2.6.1 separately permits a transmitter to choose dedicated credit even when shared credit is sufficient; it indicates that choice with bit 7.

Applying this Prefix to a Non-Flit TLP is an error. Handling it as Malformed is mandatory for a component that supports Flit Mode and strongly recommended for other components. Component support and the Link's current operating mode are separate conditions.

PCI Express Base 6.0 §2.2.10.3, Figure 2-77, printed/PDF pp.200–201; §2.6.1, p.236; MUST@FLIT in Terms and Acronyms, p.88. Non-Flit Prefix decoding is covered in Chapter 14.

Why can the transmitter wait after receive space is freed?

Separate three events: the receiver processes TLPs and advances CREDITS_ALLOCATED; it schedules and sends UpdateFC; the transmitter receives a valid update and changes CREDIT_LIMIT. The first can finish while later events remain pending. An update whose DLLP integrity check fails does not take effect.

DLLP: Data Link Layer Packet

The section 2.6.1.2 Non-Flit Mode Flow Control Update Latency note measures from freed receive space to the first Symbol transmitted for the corresponding UpdateFC. Tables 2-48–2-50 are recommended latency guidelines, not the entire transmission-plus-processing delay. Required update-scheduling triggers and periodic-update rules must be read separately.

Appendix H explains the derivation and is explicitly informational. For persistently zero PD, locate the missing event along that chain. A wait caused by Replay, ordering, or arbitration belongs to its own mechanism.

Reading route: §2.6.1.2 FC Information Tracked by Receiver, pp.246–249; Implementation Note: Non-Flit Mode Flow Control Update Latency, pp.251–252; §3.5, p.281; Appendix H Flow Control Update Latency and ACK Update Latency Calculations and §H.1 Flow Control Update Latency, pp.1581–1582.

Edition and sources

Authority: PCI Express Base Specification Revision 6.0 (6.0-1.0-PUB). Printed document and PDF viewer pages match. Counters, configurations, and the timeline are author-created, not hardware measurements. This revision teaches finite, unscaled Non-Flit Posted credit; full Scaled/Shared FC, all Replay rules, and Appendix H formulas remain for deeper coverage.

Complete source locations and purpose

§3.2 Data Link Control and Management State Machine and §3.2.1 Data Link Control and Management State Machine Rules, pp.264–267; Figure 3-2, p.265; §3.4 Flow Control Initialization Protocol and §3.4.1 Flow Control Initialization State Machine Rules, pp.270–273: DL_Up versus completed FC initialization.
§2.6 Ordering and Receive Buffer Flow Control, pp.235–236; Figure 2-89 Relationship Between Requester and Ultimate Completer, p.235; §2.6.1 Flow Control (FC) Rules, Tables 2-44 Flow Control Credit Types and 2-45 TLP Flow Control Credit Consumption, p.237; Table 2-46 Minimum Initial Flow Control Advertisements, pp.238–240: classes, units, initial advertisements, and Link scope.
§2.6.1, pp.240–241; Table 2-47 [Field Size] Values, p.241; §2.6.1.1 FC Information Tracked by Transmitter, pp.242–246; Equation 2-1 CREDITS_CONSUMED, p.242; Equation 2-10 CUMULATIVE_CREDITS_REQUIRED, p.245; Equation 2-11 Transmitter Gate, p.246: widths, wrap, finite/infinite accounting, and transmission conditions.
§2.6.1.2 FC Information Tracked by Receiver, pp.246–249; Equation 2-12 CREDITS_ALLOCATED, p.247: release and return; Non-Flit Mode Flow Control Update Latency note and Tables 2-48–2-50, pp.251–252: measurement endpoints and recommendations.
§3.5 Data Link Layer Packets (DLLPs), pp.280–281; Figure 3-11 Data Link Layer Packet Format for UpdateFC, p.284: VC, HdrFC, DataFC, and update validity.
§3.6.2 LCRC, Sequence Number, and Retry Management (TLP Transmitter), p.288; §3.6.2.1, pp.293–294; §3.6.2.2 Handling of Received DLLPs (Non-Flit Mode), pp.297–298; §3.6.3 LCRC and Sequence Number (TLP Receiver) (Non-Flit Mode), pp.300–301: retry storage, Ack, and no second credit consumption.
§2.7 End-to-End Data Integrity, p.252; §3.1 Data Link Layer Overview, pp.263–264; §3.3 Data Link Feature Exchange, pp.268–269; §3.6.2.3, p.300: integrity, features, and mode boundaries.
Appendix H Flow Control Update Latency and ACK Update Latency Calculations and §H.1 Flow Control Update Latency, pp.1581–1582: informational derivation.

Jia-Chang

Human

Comments

Write a comment ...