PCIe Chapter 26: Multifunction, ARI, SR-IOV, and Shadow Functions

Identify the participants and assumptions

Use the Port/Link model in Chapter 1 and allocation background in Chapter 4. Root Port → Switch → Endpoint has two Links. The single Endpoint Port serves one PF and three VFs to be created. Configuration, Transaction identity and resources belong to individual Functions; speed, width and LTSSM belong to their shared Link.

LTSSM

Link Training and Status State Machine

Simulated data: topology, fields and timings are author-created, not hardware measurements. PF RID=0300h is the lowest-numbered PF and is in D0. Software is ARI-aware; sufficient Bus Numbers and upstream resource Windows are available, with no conflicting Function identities. Initially VF Enable=0 and VF MSE=0. This case has no VF Resizable BAR, FRS support, or shorter reported/platform VF Enable Time.

BAR

Base Address Register

FRS

Function Readiness Status

Base 6.0 Chapter 9 already defines SR-IOV architecture, configuration and lifecycle. VM assignment policy, IOMMU isolation and internal device scheduling still require platform/device information.

SR-IOV

Single Root I/O Virtualization

Does 0314h mean Device 2, Function 4, or Function 20?

Section 6.13 divides a conventional 16-bit RID (Routing ID) into Bus [15:8], Device [7:3] and Function [2:0]. ARI (Alternative Routing-ID Interpretation) instead interprets the low byte as an 8-bit Function Number without increasing RID width. Thus 0314h conventionally displays as Bus 03h/Device 02h/Function 04h, but means Bus 03h/Function 14h, decimal20, for this ARI Device. ARI does not itself create VFs; creation remains controlled through the associated PF's SR-IOV Control.

Figure 6-13 identifies the Switch Downstream Port immediately above the Endpoint. Check its Device Capabilities 2.ARI Forwarding Supported [5] and Device Control 2.ARI Forwarding Enable [5]; both are set here. An ARI Endpoint below the Switch does not require enabling every ancestor Port.

In the Device, find ARI Extended Capability ID=000Eh, Version=1. Table 7-157's Next Function Number links non-VF Functions from Function 0. For VFs that field is undefined: use First VF Offset/VF Stride instead. Shadow Functions do not enter this list either.

The PF's ARI Capable Hierarchy is a separate hint that the immediately upstream Downstream Port has ARI enabled. Software should match it to that Port's ARI Forwarding Enable; it is set here. It resides in the lowest-numbered PF and affects the Device. Do not change it while any PF has VF Enable set; it does not replace the Port's forwarding control.

Base 6.0 §6.13, Figure 6-13, pp.762–765; §7.8.8, Figures 7-181–7-184, Tables 7-156–7-158, pp.1146–1148; §7.5.3.15, p.990; §7.5.3.16, p.996; §9.3.3.3.5, pp.1457–1458.

How does an ARI Function obtain the Bus Number for its Requests?

The preceding section splits 0314h into ARI Bus 03h and Function Number 14h. Section 2.2.6.2 requires Functions in an ARI Device to capture only the Bus Number from a Type 0 Configuration Write. The low eight bits of an ARI Requester ID contain the Function Number, not a separate Device Number.

This example uses Non-Flit Mode and does not cover Flit Mode Segment fields. For initial configuration and an ordinary non-ARI Function, see Chapter 3. Updating Bus/Function identity is separate from enabling VFs.

PCI Express Base 6.0 §2.2.6.2, printed/PDF pp.156–157, especially the ARI footnotes; §6.13 Alternative Routing-ID Interpretation, pp.760–762; §6.4 Device Synchronization, pp.717–718.

Read the fields that precede VF creation

In the PF, locate SR-IOV Extended Capability ID=0010h, Version=1, with base S. Figure 9-10 locates fields with different access widths. Counts and RID offsets below are 16-bit; Page Sizes are 32-bit:

TotalVFs is capacity; NumVFs is the selected count. NumVFs initially has an undefined value, so software explicitly sets it. Here 3≤8. Write NumVFs only while VF Enable is clear; changing it while enabled or selecting more than TotalVFs has undefined results.

System Page Size=00000010h selects bit4, hence 2⁴⁺¹²=65,536 bytes=64 KiB, not 16 bytes. Select exactly one supported bit: zero, multiple bits or an unsupported bit have undefined results. The reported bitmap includes the six required page sizes. Write System Page Size only while VF Enable is clear; changing it leaves all VF BARn contents indeterminate, requiring fresh sizing/allocation.

NumVFs and ARI Capable Hierarchy can affect First VF Offset/VF Stride. Configure those inputs before reading the offsets rather than reusing a snapshot from another count.

Base 6.0 §§9.2.1.1–9.2.1.2, pp.1445–1447; §9.3.3.1, Figures 9-10/9-11, Table 9-2, pp.1451–1452; §9.3.3.7, p.1459; §§9.3.3.12–9.3.3.14, pp.1462–1463.

Find the third VF's identity

Table 9-1 defines VF N RID=PF RID+First VF Offset+(N−1)×VF Stride, using unsigned 16-bit arithmetic with carries dropped. With NumVFs=3, PF 0300h, Offset 0010h and Stride0002h:

VF numbering here starts at 1 as in Table 9-1; reconcile a tool's zero-based VF index first. A stride of2 increments RID, not a byte address. The legacy column is another display of the same bits, while this case uses ARI Function Numbers.

These VFs remain on Bus 03h. Other configurations may need additional Bus Numbers reserved before enabling. Every PF/VF RID must be unique; a VF must not be on a numerically lower Bus than its PF. Modulo arithmetic does not authorize a conflicting or lower-Bus assignment.

Base 6.0 §9.2.1.2, Table 9-1, pp.1446–1447; §§9.3.3.9–9.3.3.10, p.1461.

Find VF 3's memory range independently

Figure 9-9 separates the PF's ordinary BAR from the VF BAR in its SR-IOV Capability. A VF's own Type0 Header BARs are hardwired zero. Configure the group through the PF's VF BARs, not by programming a separate Header BAR in each VF.

After selecting the 64 KiB System Page Size, sizing the 64-bit Prefetchable VF BAR0/1 pair returns low DW=FFFF000Ch and high DW=FFFFFFFFh. Remove the low four attribute bits to obtain address mask FFFFFFFFFFFF0000h. Invert across 64 bits and add 1: each VF aperture is 10000h=64 KiB.

DW

Double Word, a four-byte unit

Allocate first-VF base 90000000h, represented by VF BAR0=9000000Ch and VF BAR1=00000000h. Section 9.2.1.1.1 derives later starts as base+(VF number−1)×per-VF aperture:

VF 3 starts at 90020000h and occupies 90020000h–9002FFFFh. Three VFs map 30000h=192 KiB using NumVFs=3, not maximum TotalVFs=8. Software may reserve extra space as a separate future-allocation policy. Upstream Windows must cover the mapped ranges, and each aperture's size/alignment must satisfy System Page Size.

If implemented, VF Resizable BAR ID=0024h, Version=1 resides in the PF, never in a VF. It is independent of the PF's ordinary Resizable BAR. Clear VF MSE before changing VF BAR Size, select a supported size, reprogram the now-undefined VF BAR, then enable decoding as needed. Hardware reflects the greater requirement of System Page Size and VF BAR Size. This optional mechanism is not used in the numeric case.

Base 6.0 §9.2.1.1.1, Figure 9-9, pp.1445–1446; §9.3.3.14, pp.1462–1463; §7.5.1.2.1, Figures 7-10/7-11, Table 7-9, pp.928–930; §7.8.7, Figures 7-178–7-180, Tables 7-154/7-155, pp.1142–1145.

VF Enable does not permit immediate BAR access

Separate initialization waiting, Configuration readiness and Memory decoding. Assume unlisted Control bits are zero and keep ARI Capable Hierarchy [4] set: 0010h is disabled, 0011h has VF Enable=1/VF MSE=0, and 0019h additionally sets VF MSE [3]. In the diagram, RRS (Request Retry Status) follows the retry path; SC (Successful Completion) proceeds to the next configuration step.

FRS

Function Readiness Status

After enabling, Requests to the VFs must wait until at least 100 ms, a PF FRS (Function Readiness Status) Message with reason VF Enabled, or the applicable reported/platform VF Enable Time. The case has no early alternatives, so no Configuration Request is sent at 50 ms.

At 120 ms the VF returns Configuration RRS (Request Retry Status). Such retry status is permitted within the 1.0 s interval after VF Enable, and the RC/software must allow at least 1.0 s before declaring a VF broken for not returning Successful Completion to a valid Configuration Request. Each RRS does not restart that interval.

At 180 ms SC (Successful Completion) establishes one Memory-readiness condition. VF MSE remains clear, so Memory access is still disabled. At 181 ms Control=0019h enables decoding; with Function-specific initialization also complete, the example accesses VF 3 at 90020010h. Other specified FRS,1.0 s or applicable reported/platform VF Enable Time paths can establish Memory readiness. Simply waiting100 ms is not equivalent to those conditions.

After a VF has successfully responded to a Request, it must not return Configuration RRS without an intervening disable or other valid Reset. Clearing VF Enable destroys VFs; no field in the PF's SR-IOV Capability may then be accessed until at least 1.0 s has elapsed or a PF FRS Message with reason VF Disabled is received. Do not immediately poll Control or rewrite NumVFs.

Base 6.0 §9.3.3.3, Figure 9-13/Table 9-4, pp.1454–1455; §9.3.3.3.1, pp.1455–1457; §9.3.3.3.4, p.1457.

FFFFh Vendor ID and clear MSE have VF-specific meanings

Unlike ordinary Function probing, §§7.5.1.1.1–7.5.1.1.2 require a VF's raw Vendor ID and Device ID to return FFFFh. Virtualization software should present its PF's Vendor ID and the PF's VF Device ID field respectively. At the 180 ms read, first inspect the SC Completion, then interpret FFFFh using the VF-specific rule rather than an ordinary absence shortcut.

A VF's Command.Memory Space Enable is ROZ (Read Only Zero). The PF's VF MSE controls Memory decoding for the associated Active VFs. I/O Space Enable is also hardwired zero; VFs do not support I/O Space. A VF's own Bus Master Enable instead controls its ability to initiate Memory Requests. That direction differs from host access to VF resources; PF BME does not represent every VF.

Once their individual Bus Master Enable and device conditions are satisfied, two VFs may issue Memory Reads with Tag 05h. Match returned Completions using Requester IDs0310h and 0312h as well as Tag; equal Tags do not make one Transaction. A healthy shared Link does not establish each Function's resource, BME, Transaction or device-initialization state.

For interrupt details return to Chapter 7. With allocated resources VFs may use MSI, MSI-X or both, but must not implement INTx. Internal allocation, scheduling and VM assignment remain platform/device policy.

MSI

Message Signaled Interrupt

MSI-X

Message Signaled Interrupts Extension

Base 6.0 §§7.5.1.1.1–7.5.1.1.3, Table 7-4, pp.917–918; §7.5.1.2.1, p.928; §9.2.1.4, p.1449; PF/VF scope and identities, pp.1440–1447.

Distinguish VF Reset, PF Reset and recreation

New VFs enter D0uninitialized when the PF is in D0; creating them while the PF is in a lower power state has undefined behavior. Conventional Reset also clears VF Enable and removes VFs. The shared Link remaining in L0 does not describe these Function lifecycle changes.

Base 6.0 §§9.2.2–9.2.2.3, p.1449; §9.2.3, p.1450; §9.3.3.3.1, p.1456.

Why can a Shadow identity issue Requests without independent Configuration?

Shadow Functions borrow Requester ID/Tag resources of otherwise unimplemented Functions to extend a Requester's Outstanding Request capacity. Characteristics come from the associated Function; this does not create another assignable VF. Locate ID=002Dh, Version=1, then use Figure 7-365 and Tables 7-320–7-322.

Continuing the simulated case: assume PF 0300h implements one Shadow Function and 0320h is otherwise unused. Number of Shadow Functions=0 is N−1 encoded, hence one Instance. The first entry atH+0Ch contains Shadow Function Number=0020h, a16-bit BDF offset. Add 0300h+0020h=0320h; it is not an absolute Function Number replacement.

BDF

Bus/Device/Function

Assume translation/access-control resources have been prepared, Phantom Functions Enabled=0, and no Outstanding Non-Posted Requests before enabling Shadow Functions. Changing its Enable while such Requests are outstanding, or enabling it together with Phantom Functions, has undefined behavior.

Requester ID 0320h may now issue Memory Read. Ordinary Posted Requests are also permitted, but MSI/MSI-X Requests must not use Shadow Transaction ID resources. Configuration access to 0320h, absent errors with other behavior, must return UR rather than independent Configuration Space. This differs from a created VF returning SC with FFFFh ID fields.

MSI

Message Signaled Interrupt

MSI-X

Message Signaled Interrupts Extension

A Shadow Function Number greater than 7 requires ARI; without ARI the Shadow must remain in the same Device. Software must update translation resources before Requests, otherwise behavior is undefined. Relate the identity back to its PF/Capability while still considering its separate RID throughout the path.

Base 6.0 §7.9.25 and §§7.9.25.1–7.9.25.4, pp.1299–1303; Figures 7-365–7-369, Tables 7-319–7-322. Use the field definitions for first-entry offset and N−1 count.

Edition, source route and platform boundaries

PCI Express Base Specification Revision 6.0 (6.0-1.0-PUB) supplies the rules; printed and PDF-viewer pages match. All topology, fields, RID/BAR values and timings are author-created. No hardware was configured. No external website supplies technical claims.