PCIe Chapter 14: TLP Headers and Flit Packet Layouts

Why start with the format?

In Non-Flit Mode, locate Fmt/Type in Figure 2-4, then consult Tables 2-2 and 2-3. Fmt distinguishes header size and data presence; Type also determines routing. A Memory Read requests data without carrying that read data in its request. For example, a separate Non-Flit 32-bit-address Memory Read with Length=2 and no Prefix or Digest has only a 12-byte Request Header; its data returns in Completion packets. The Type/format determines whether Length describes data present in this packet.

For Flit Mode, restart with Figure 2-6 and Table 2-5. The eight-bit Type is a different encoding: NOP is 00h, while a 32-bit-address Memory Read is 03h. Reusing the Non-Flit decoding table misidentifies the packet.

How do extensions affect the data position?

OHC (Orthogonal Header Content, optional header extensions) is indicated in Figure 2-6. Read the presence/order rules on p.131, then Table 2-6 for the applicable A format. A Memory Request with explicit Byte Enables uses OHC-A1 in Figure 2-7. OHC follows the Header Base in A-B-C-E order.

TS (Trailer Size) describes the trailer separately from Length. Non-Flit Mode instead has its own prefix and TD (TLP Digest presence) rules. Determine the entire packet size before locating the next packet.

Where do we start when a Prefix precedes the Header?

Start in Non-Flit Mode. A TLP Prefix is one DW (4 bytes) ahead of the underlying Header. In byte 0, Fmt[2:0]=100b identifies a Prefix; Type[4]=0 selects a Local Prefix and Type[4]=1 an End-End Prefix. An underlying Header is required: a received Prefix without one is handled as a Malformed TLP.

DW

Double Word, a four-byte unit

End-End Prefixes accompany a TLP from source to destination; the size, routing and Flow Control of Local Prefixes depend on their type. When the underlying TLP is protected by ECRC (End-to-End CRC), its End-End Prefixes are covered but Local Prefixes are not. Support and egress blocking still need to be checked along the path.

ECRC

End-to-End Cyclic Redundancy Check

Sections 2.2.10.2.1 and 2.2.10.4.1 require explicit vendor-specific enablement before a component sends vendor-defined Local or End-End Prefixes. A component supporting vendor-defined Local Prefixes must also support 3-bit Fmt and set Device Capabilities 2.Extended Fmt Field Supported. Each section recommends configurability for either of its two vendor encodings; this is a recommendation, and Local Prefix encoding need not be symmetric between the two ends.

If a Receiver does not support a received Local Prefix type and Extended Fmt Field Supported=1, the TLP is generally Malformed unless another specification explicitly says otherwise. With that bit clear, behavior is device-specific. Flit Mode uses its own Type and Prefix rules; do not apply the byte decoding above directly.

PCI Express Base 6.0 §§2.2.10.1–2.2.10.2.1 and Table 2-38, pp.199–200; §2.2.10.4 and Table 2-39, pp.201–202; §2.2.10.4.1, p.203; §7.5.3.15, Figure 7-36 and Table 7-33, pp.989 and 992. Printed and PDF pages match.

Why can a supporting Switch still block a prefixed Request?

Separate receiver support, Prefix count, and egress control. Section 2.2.10.4 permits at most four End-End Prefixes in a Non-Flit TLP. Receivers supporting TLP Prefixes must check that limit and treat a detected violation as Malformed. A Switch supporting End-End Prefixes must forward up to four, but each egress has its own blocking control.

Check the destination Function's limit separately. Except for the special Root Port rules, a Function receiving more End-End Prefixes than its advertised maximum must treat the TLP as Malformed. A Root Port may advertise less than its hardware implements; error handling still uses the advertised value. Within the four-Prefix limit, for a Request exceeding the Root Port's advertised value, UR handling is recommended, otherwise Malformed is required. For a Completion, Unexpected Completion (UC) is recommended, otherwise Malformed is required. The error is associated with the Ingress Port for a TLP received there, or the Egress Port for a TLP received internally for transmission.

A Receiver that does not support End-End Prefixes must treat a TLP containing one as Malformed. For a multi-Function Upstream Port, if one Function sets End-End TLP Prefix Supported, all Functions handle an unsupported type addressed to them as UR for Requests or UC for Completions. Software should ensure prefixed TLPs are not sent to unsupported components.

PCI Express Base 6.0 §2.2.10.4 and Table 2-39, printed/PDF pp.201–202; §7.5.3.16, Figure 7-37 and Table 7-34, pp.994 and 997. The preceding section explains Prefix format; the next covers P2P forwarding between Root Ports.

Do two supporting Root Ports guarantee Prefix forwarding between them?

No. An End-End Prefix accompanies a TLP across the fabric, but §2.2.10.4.2 makes peer-to-peer (P2P) forwarding between Root Ports optional and implementation-dependent. When the Root Complex supports such forwarding, each associated Root Port must advertise support; two set support bits alone do not establish that this particular Root Port pair can forward the TLP.

The same section requires the Root Port support bit when that port forwards prefixed TLPs initiated by host software or a Root Complex Integrated Endpoint (RCiEP), or forwards prefixed TLPs received on its Ingress Port to RCiEPs. If a Root Complex splits a TLP into smaller TLPs while forwarding P2P between Root Ports, it must replicate the original End-End Prefixes in each smaller TLP.

PCI Express Base 6.0 §§2.2.10.4 and 2.2.10.4.2, printed/PDF pp.201–203; §7.5.3.15, Figure 7-36 and Table 7-33, pp.989 and 992. The example topology and capability values are assumptions.

How is End-End Prefix content read in Flit Mode?

Use Figure 2-6's OHC[4:0] field to find OHC-E. Orthogonal Header Content (OHC) follows the Header Base; when A, B, C and E are present, they appear in that order. Section 2.2.11 Figures 2-78, 2-79 and 2-80 define OHC-E1, E2 and E4 as 1, 2 and 4 DW. OHC-E conveys content corresponding to End-End Prefix types 0011b–1111b; consult each mechanism's Flit Mode rules for other types. This range does not make every type usable: Base 6.0 Table 2-39 reserves 0011b–1101b, while this example uses the vendor-defined 1110b and 1111b.

DW

Double Word, a four-byte unit

Section 2.2.11 requires entries to start in the first DW and be populated in order; unused DWs indicate No Entry. The transmitter must choose the smallest OHC-E that accommodates the content and avoid unnecessary No Entry DWs: one entry uses E1, while three use E4 with the remaining DW marked No Entry. A byte 0 high nibble of 0000b means No Entry and the remaining bits are reserved; 0010b–1111b is Reserved and Receivers must handle it as No Entry, not as a valid entry.

Finally check destination OHC-E support and the whole path. Root Complex P2P forwarding of TLPs containing OHC-E between Root Ports is optional and implementation-dependent; associated Root Ports must advertise End-End TLP Prefix Supported when the Root Complex supports that forwarding. This parallels the Non-Flit path check above, but decoding follows the active mode.

PCI Express Base 6.0 §2.2.1.2 and Figure 2-6, printed/PDF pp.123 and 131; §2.2.10.4 and Table 2-39, p.201; §2.2.11 and Figures 2-78–2-80, pp.203–204; §7.5.3.15 and Table 7-33, p.992. Vendor data values are assumptions.

Follow the same write intent through both formats

40 00 00 02
03 00 00 3C
80 00 10 00
A0 A1 A2 A3
B0 B1 B2 B3

40 01 00 02
03 00 00 00
80 00 10 00
00 00 00 3C
A0 A1 A2 A3
B0 B1 B2 B3

Section 2.2.5.2 explains why the extension is needed for this intent: without OHC-A1, a two-DWORD Memory Request implies First and Last DW BE=1111b. All eight bytes would be enabled, changing the intended middle-four-byte write. OHC presence affects the access performed by the receiver.

A Non-Flit decoder applied to the second trace could misread byte 7=0x00 as both BE fields and mistake OHC bytes 00 00 00 3C for the payload start, shifting data by four bytes. Correct the analyzer mode first; these artifacts do not establish that an erroneous packet actually appeared on the Link.

Why do eight payload bytes update only four bytes?

Continue the same simulated writes. Section 2.2.5.1 and Table 2-10 map each BE bit to a data byte: First DW BE bit 0 selects the lowest-address byte in the first DW; Last DW BE bit 0 selects the lowest-address byte in the last DW. First=1100b selects offsets +2 and +3; Last=0011b selects +4 and +5.

The selected range is 0x80001002–0x80001005, totaling four bytes, while each TLP still carries eight payload bytes. Length remains 2 DW; disabled bytes are not removed from the packet. This example stays within a 4-KB boundary and selects contiguous bytes. Other lengths, alignments or non-contiguous patterns require the full §2.2.5.1 rules.

In the Non-Flit example, changing to a one-DWORD Request requires Last DW BE= 0000b; the example's 0011b cannot simply be retained. Read the conditions following Figure 2-26 together with the field locations.

Does TD=1 prove that the receiver checked ECRC?

ECRC

End-to-End Cyclic Redundancy Check

It first indicates a TLP Digest at the end of a Non-Flit TLP. Whether ECRC (End-to-End CRC) is checked also depends on receiver capability and controls. In an author-created variation of the preceding Non-Flit Memory Write, a 12-byte Header and 8-byte Payload with TD=1 and no Prefix have a further 4-byte Digest, for a 24-byte logical TLP.

Figure 2-5 locates TD and Length; Length describes the Payload and excludes the Digest. Under §2.2.3 a mismatch between TD and the observed TLP size is a Malformed TLP, reported in association with the receiving Port. A receiver without ECRC checking support must ignore the Digest; an intermediate receiver forwarding a Multicast TLP through an MC_Overlay-enabled egress has a specific exception in §6.14.5. For supported checking, consult the conditions in §2.7.1.

TD=1 is not a record of a successful check. Do not apply this Non-Flit TD format directly to Flit Mode. Determine actual enablement and error records from the receiving component and its AER controls.

AER

Advanced Error Reporting

Base 6.0 §2.2.1.1, Figure 2-5 (PDF/printed p.120); §§2.2.2–2.2.3 TLP Digest Rules - Non-Flit Mode Only (pp.142, 144); further conditions in §§2.7.1 and 6.14.5.

Must a device use the same transmit and receive limit?

Not necessarily. MPS (Maximum Payload Size) limits TLP data payloads; determine the transmitting and receiving Functions' limits separately. Read Max_Payload_Size Supported, Rx_MPS_Fixed, and Mixed_MPS_Supported in Device Capabilities and the Max_Payload_Size setting in Device Control, then apply §2.2.2 to the particular device and direction.

For Tx_MPS_Limit, a single-Function device uses its own setting; an ARI (Alternative Routing-ID Interpretation) device uses Function 0's setting. A non-ARI Multi-Function Device uses the common setting when all Functions agree; otherwise an implementation-selected Function supplies it. Software should not configure differing settings without knowing the implementation.

If Mixed_MPS_Supported=1, an implementation-specific mechanism can choose a target-specific value above or below Tx_MPS_Limit, up to the Function's Supported value. Host-memory targets still use Tx_MPS_Limit. Consult implementation information to determine a target-specific value.

For receive, Rx_MPS_Fixed=1 makes Rx_MPS_Limit the receiving Function's Supported value. With it clear, apply §2.2.2's single-Function, ARI, or applicable non-ARI multi-Function rules. A receiver must check payloads against its Rx_MPS_Limit; a detected violation is handled as a Malformed TLP and reported in association with the receiving Port.

ARI

Alternative Routing-ID Interpretation

MPS limits data carried in a TLP. A Memory Read Request carries no requested data, so its Length is not restricted by MPS; consult Maximum Read Request Size and other applicable rules. For Flit-capable implementations, Table 7-20 requires at least 512-byte Supported and Rx_MPS_Fixed=1; these requirements do not apply uniformly to all Non-Flit devices.

Are non-contiguous Byte Enables always an error?

Continue the Memory Write example, changing Length, alignment and BE under §2.2.5.1. These simulated Non-Flit Mode Requests show why non-contiguous enables must be evaluated with Length and alignment.

On a phone, scroll the table horizontally to see all columns.

Figure 2-26 locates the fields in Non-Flit Mode; Table 2-10 maps each bit to a data byte. The surrounding text supplies the legality conditions. A Receiver may check individual BE rules; when an implemented check detects a violation, the TLP is Malformed. No error record alone does not prove valid BEs.

In Flit Mode, first consult §2.2.5.2. A Memory Request without OHC-A1 implies First DW BE=1111b; for Length at least 2 DW, Last DW BE is also implied 1111b. Other BE values require OHC-A1. I/O and Configuration Requests have their own OHC requirements; do not apply the Memory Request omission rule to them.

Base 6.0 §§2.2.5.1–2.2.5.2, Figure 2-26 and Table 2-10, pp.149–152; QW is 8 bytes.

When is the same Header byte no longer Byte Enables?

Return to Figure 2-26. In Non-Flit Mode, when a Memory Read or Deferrable Memory Write (DMWr) has TH=1, the positions normally holding First/Last DW BE instead carry Steering Tag ST[7:0]. Do not decode that byte directly as two BE fields.

Teaching example: for an applicable Memory Read with TH=1 and Length=1 DW, First DW BE is implied 1111b and Last DW BE 0000b. For Length greater than 1 DW, both are implied 1111b. TH may be set only when it is acceptable to complete the Request as if all requested bytes were enabled; §2.2.5.1 also discusses reads with side effects in Non-Prefetchable space.

This example concerns Non-Flit Mode field reuse. For a Flit Mode trace, use that mode's Header and OHC rules instead of applying Figure 2-26's byte position.

Base 6.0 §2.2.5.1, Figure 2-26, pp.149–150; additional TH/Steering Tag requirements: §2.2.7.1.1.

Why does a Read receive data when no byte is enabled?

Teaching example: a Non-Flit Mode Memory Read has Length=1 DW, First DW BE=0000b and Last DW BE=0000b. Section 2.2.5.1 requires the corresponding Completion to have Length=1 DW and a 1-DW Data Payload. Its contents are unspecified; do not treat the returned value as device data.

The zero-length Read implementation note describes a use as a flush: a Requester can use it to establish that earlier Posted Writes completed at their PCIe destination. For that use to be effective in all cases, the Read must target the same device as those Writes; the specification recommends using the address of one of those Writes. The flush applies only to traffic in the same Traffic Class as the Read. It does not establish that application data is durable on nonvolatile media.

For example, send a TC0 Posted Write, then a TC0 zero-length Read to the same device and observe its normal Completion. That supports the specification's conclusion about the earlier Write at the PCIe destination. A TC1 Write is not covered by this TC0 Read. Account for the path and applicable Attributes; issuing the Read is not the same observation as receiving its response.

Base 6.0 §2.2.5.1, zero-length Write/Read and implementation notes, pp.151–152; detailed Ordering conditions: §2.4.

Continue into receiver handling

Section 2.2.9 identifies requests requiring a Completion (transaction response). A Memory Read receives data in its completion; a normal posted Memory Write has no corresponding successful completion. Chapter 15 follows receiver checks and the resulting response.

Edition and sources

This chapter uses PCI Express Base Specification Revision 6.0 (6.0-1.0-PUB). Document and PDF page numbers match. Example configurations and values are author assumptions, not device measurements.