Add routines to trigger a function level reset (FLR) of a PCI-express
device via the PCI-express device control register. This also includes
support routines to wait for pending transactions to complete as well
as calculating the maximum completion timeout permitted by a device.
Change the ppt(4) driver to reset pass through devices before attaching
to a VM during startup and before detaching from a VM during shutdown.
Sponsored by: Chelsio Communications
Add accessor methods to fetch the BAR holding the MSI-X table and PBA.
While here, explicitly note the requirement that the BAR(s) must be
allocated prior to calling pci_alloc_msix().
Add a new -B flag for use with list mode (-l) that lists details about
bridges. Currently this includes information about what resources a
bridge decodes on the upstream side for use by downstream devices including
bus numbers, I/O port resources, and memory resources. Windows and bus
ranges are enumerated for both PCI-PCI bridges and PCI-CardBus bridges.
To simplify the implementation, all enumeration is done by reading the
appropriate config space registers directly rather than querying the
bridge driver in the kernel via new ioctls. This does result in a few
limitations.
First, an unimplemented window in a PCI-PCI bridge cannot be accurately
detected as accurate detection requires writing to the window base
register. That is not safe for pciconf(8). Instead, this assumes that
any window where both the base and limit read as all zeroes is
unimplemented.
Second, the PCI-PCI bridge driver in a tree has a few quirks for
PCI-PCI bridges that use subtractive decoding but do not indicate that
via the progif config register. The list of quirks is duplicated in
pciconf's source.
This helps reduce the diff in pci(4) between head and stable/10 to help pave
the way for bringing in IOV/nv(9) more cleanly
Differential Revision: https://reviews.freebsd.org/D4728
Relnotes: yes
Reviewed by: hselasky (ofed piece), royger (overall change)
Sponsored by: EMC / Isilon Storage Division
r270212 (by royger):
pci: make MSI(-X) enable and disable methods of the PCI bus
Make the functions pci_disable_msi, pci_enable_msi and pci_enable_msix
methods of the newbus PCI bus. This code should not include any
functional change.
Sponsored by: Citrix Systems R&D
Reviewed by: imp, jhb
Differential Revision: https://reviews.freebsd.org/D354
dev/pci/pci.c:
- Convert the mentioned functions to newbus methods.
- Fix the callers of the converted functions.
sys/dev/pci/pci_private.h:
dev/pci/pci_if.m:
- Declare the new methods.
dev/pci/pcivar.h:
- Add helpers to call the newbus methods.
ofed/include/linux/pci.h:
- Add define to prevent the ofed version of pci_enable_msix from
clashing with the FreeBSD native version.
r270332 (by royger):
pci: add a new pci_child_added newbus method.
This is needed so when running under Xen the calls to pci_child_added
can be intercepted and a custom Xen method can be used to register
those devices with Xen. This should not include any functional
change, since the Xen implementation will be added in a following
patch and the native implementation is a noop.
Sponsored by: Citrix Systems R&D
Reviewed by: jhb
dev/pci/pci.c:
dev/pci/pci_if.m:
dev/pci/pci_private.h:
dev/pci/pcivar.h:
- Add the pci_child_added newbus method.
Additional PCI helper functions.
290414:
Add helper routines for PCI device drivers to read, write, and modify
PCI-Express capability registers (that is, PCI config registers in the
standard PCI config space belonging to the PCI-Express capability
register set).
Note that all of the current PCI-e registers are either 16 or 32-bits,
so only widths of 2 or 4 bytes are supported.
290415:
Add a new helper function for PCI devices to locate the upstream
PCI-express root port of a given PCI device.
MFC (281874). It broke suspend and resume on several Thinkpads (though not
all) in 10 even though it works fine on the same laptops in HEAD.
PR: 201239
Reported by: Kevin Oberman and several others
Various fixes for suspend and resume of PCI to PCI and PCI to Cardbus
bridges.
274633:
Remove stray empty comment. The code is adequately explained in the
block comment above, so there's nothing to add here.
274639:
Modernize comments about BIOSes being lame since in this detail they
aren't lame, the rules changed along the way. Catch up to 1999 or so
with the new rules.
274663:
Fix typo pointed out by avg@ and Joerg Sonnenberger. Add a clarifying
sentence too.
277233:
Suspend and resume were the only two functions not to follow the brdev
convention here, so fix that.
277234:
Move the suspsned and resume functions to the bus attachment. They
were accessing PCI config registers, which won't work for the ISA
version.
277235:
Always enable I/O, memory and dma cycles. Some BIOSes don't enable
them, sometimes they are reset for power state transitions or during
whatever happens while suspended. Also, it is good practice to always
do this.
281870:
Cosmetic change: use PCIR_SECLAT_2 rather than PCIR_SECLAT_1.
281871:
The minimim grant and maximum latency PCI config registers are only valid
for type 0 devices, not type 1 or 2 bridges. Don't read them for bridge
devices during bus scans and return an error when attempting to read them
as ivars for bridge devices.
281873:
Don't explicitly manage power states for PCI-PCI bridge devices in the
driver's suspend and resume routines. These have been redundant no-ops
since r214065 changed the PCI bus driver to manage power states for
all devices (including type 1/2 bridge devices) during suspend and resume.
281874:
Update the pci_cfg_save/restore routines to operate on bridge devices
(type 1 and type 2) as well as leaf devices (type 0). In particular,
this allows the existing PCI bus logic to save and restore capability
registers such as MSI and PCI-express work for bridge devices rather than
requiring that code to be duplicated in bridge drivers. It also means
that bridge drivers no longer need to save and restore basic registers
such as the PCI command register or BARs nor manage powerstates for the
bridge device.
While here, pci_setup_secbus() has been changed to initialize the 'sec'
and 'sub' fields in the 'secbus' structure instead of requiring the pcib
and pccbb drivers to do this in the NEW_PCIB + PCI_RES_BUS case.
MFC support for PCI Alternate RID Interpretation. ARI is an optional PCIe
feature that allows PCI devices to present up to 256 functions on a bus.
This is effectively a prerequisite for PCI SR-IOV support.
r264007:
Add a method to get the PCI RID for a device.
Reviewed by: kib
MFC after: 2 months
Sponsored by: Sandvine Inc.
r264008:
Re-implement the DMAR I/O MMU code in terms of PCI RIDs
Under the hood the VT-d spec is really implemented in terms of
PCI RIDs instead of bus/slot/function, even though the spec makes
pains to convert back to bus/slot/function in examples. However
working with bus/slot/function is not correct when PCI ARI is
in use, so convert to using RIDs in most cases. bus/slot/function
will only be used when reporting errors to a user.
Reviewed by: kib
MFC after: 2 months
Sponsored by: Sandvine Inc.
r264009:
Re-write bhyve's I/O MMU handling in terms of PCI RID.
Reviewed by: neel
MFC after: 2 months
Sponsored by: Sandvine Inc.
r264011:
Add support for PCIe ARI
PCIe Alternate RID Interpretation (ARI) is an optional feature that
allows devices to have up to 256 different functions. It is
implemented by always setting the PCI slot number to 0 and
re-purposing the 5 bits used to encode the slot number to instead
contain the function number. Combined with the original 3 bits
allocated for the function number, this allows for 256 functions.
This is enabled by default, but it's expected to be a no-op on currently
supported hardware. It's a prerequisite for supporting PCI SR-IOV, and
I want the ARI support to go in early to help shake out any bugs in it.
ARI can be disabled by setting the tunable hw.pci.enable_ari=0.
Reviewed by: kib
MFC after: 2 months
Sponsored by: Sandvine Inc.
r264012:
Print status of ARI capability in pciconf -c
Teach pciconf how to print out the status (enabled/disabled) of the ARI
capability on PCI Root Complexes and Downstream Ports.
MFC after: 2 months
Sponsored by: Sandvine Inc.
r264013:
Add missing copyright date.
MFC after: 2 months
Add support for displaying VPD for PCI devices via pciconf.
- Store the length of each read-only VPD value since not all values are
guaranteed to be ASCII values (though most are).
- Add a new pciio ioctl to fetch VPD for a single PCI device. The values
are returned as a list of variable length records, one for the device
name and each keyword.
- Add a new -V flag to pciconf's list mode which displays VPD data for
each device.
Here are two new functions to map and unmap the Video BIOS:
void * vga_pci_map_bios(device_t dev, size_t *size);
void vga_pci_unmap_bios(device_t dev, void *bios);
The BIOS is either taken from the shadow copy made by the System BIOS at
boot time if the given device was used for the default display (i386,
amd64 and ia64 only), or from the PCI expansion ROM.
Additionally, one can determine if a given device was the default
display at boot time using the following new function:
void vga_pci_unmap_bios(device_t dev, void *bios);
VMware up to at least ESXi 5.1. Actually, using INTx in that case instead
may still result in interrupt storms, with MSI being the only working
option in some configurations. So introduce a PCI_QUIRK_DISABLE_MSIX quirk
which only blacklists MSI-X but not also MSI and use it for the VMware
PCI-PCI-bridges. Note that, currently, we still assume that if MSI doesn't
work, MSI-X won't work either - but that's part of the internal logic and
not guaranteed as part of the API contract. While at it, add and employ
a pci_has_quirk() helper.
Reported and tested by: Paul Bucher
- Use NULL instead of 0 for pointers.
Submitted by: jhb (mostly)
Approved by: jhb
MFC after: 3 days
Expand pci_save_state and pci_restore_state to save more of
the config state for PCI Express and PCI-X devices. Various
writable control registers are present in PCI Express that
can potentially be lost over suspend/resume cycle.
This change is modeled after similar functionality in Linux.
Reviewed by: wlosh,jhb
MFC after: 1 month
- pci_find_extcap() is repurposed to be used for fetching PCI-express
extended capabilities (PCIZ_* constants in <dev/pci/pcireg.h>).
- pci_find_htcap() can be used to locate a specific HyperTransport
capability (PCIM_HTCAP_* constants in <dev/pci/pcireg.h>).
- Cache the starting location of the PCI-express capability for PCI-express
devices in PCI device ivars.
pci_cfg_save() and pci_cfg_restore() for device drivers to use when
saving and restoring state (e.g. to handle device-specific resets).
Reviewed by: imp
MFC after: 2 weeks
bus driver will now remember the size of a BAR obtained during the initial
bus scan and use that size when doing lazy resource allocation rather than
resizing the BAR. The bus driver will now also report unallocated BARs to
userland for display by 'pciconf -lb'. Psuedo-resources that are not BARs
(such as the implicit I/O port resources for master/slave ATA controllers)
will no longer be listed as BARs in 'pciconf -lb'. During resume, BARs are
restored from their new saved state instead of having the raw registers
saved and restored across resume. This also fixes restoring BARs at
unusual loactions if said BAR has been allocated by a driver.
Add a constant for the offset of the ROM BIOS BAR in PCI-PCI bridges and
properly handle ROM BIOS BARs in PCI-PCI bridges. The PCI bus now also
properly handles the lack of a ROM BIOS BAR in a PCI-Cardbus bridge.
Tested by: jkim
"extended capabilities" to refer to the new set of capability structures
starting at offset 0x100 in config space for PCI-express devices. For now
both function names will still work. I will merge this to older branches
to ease driver portability, but 9.0 will ship with a new pci_find_extcap()
function that locates extended capabilities instead.
Reviewed by: imp
MFC after: 1 week
- Always enable the HyperTransport MSI mapping window for HyperTransport
to PCI bridges (these show up as HyperTransport slave devices).
The mapping windows in PCI-PCI bridges are enabled by existing code
in the PCI-PCI bridge driver as MSI requests propagate up the device
tree, but Host-PCI bridges don't really show up in that tree.
- If the PCI device at domain 0 bus 0 slot 0 function 0 is not a
HyperTransport device, then blacklist MSI on any other HT devices in
the system. Linux has a similar quirk.
PR: kern/155442
Tested by: Zack Dannar zdannar of gmail
MFC after: 1 week
Specification Rev. 1.2. Rename pp_pcmcsr field of PM capabilities to pp_bse
to avoid further confusions and adjust some comments accordingly. The real
PMCSR (Power Management Control/Status Register) is PCIR_POWER_STATUS and
it is actually BSE (PCI-to-PCI Bridge Support Extensions) register.
used but MSI to HyperTransport IRQ mapping is enabled, and would act as
if MSI is turned on, resulting in interrupt loss.
This commit will,
1. enable MSI mapping on a device only when MSI is enabled for that
device and the MSI address matches the HT mapping window.
2. enable MSI mapping on a bridge only when a downstream device is
allocated an MSI address in the mapping window
PR: kern/118842
Reviewed by: jhb
MFC after: 1 week
support machines having multiple independently numbered PCI domains
and don't support reenumeration without ambiguity amongst the
devices as seen by the OS and represented by PCI location strings.
This includes introducing a function pci_find_dbsf(9) which works
like pci_find_bsf(9) but additionally takes a domain number argument
and limiting pci_find_bsf(9) to only search devices in domain 0 (the
only domain in single-domain systems). Bge(4) and ofw_pcibus(4) are
changed to use pci_find_dbsf(9) instead of pci_find_bsf(9) in order
to no longer report false positives when searching for siblings and
dupe devices in the same domain respectively.
Along with this change the sole host-PCI bridge driver converted to
actually make use of PCI domain support is uninorth(4), the others
continue to use domain 0 only for now and need to be converted as
appropriate later on.
Note that this means that the format of the location strings as used
by pciconf(8) has been changed and that consumers of <sys/pciio.h>
potentially need to be recompiled.
Suggested by: jhb
Reviewed by: grehan, jhb, marcel
Approved by: re (kensmith), jhb (PCI maintainer hat)
- Simplify the amount of work that has be done for each architecture by
pushing more of the truly MI code down into the PCI bus driver.
- Don't bind MSI-X indicies to IRQs so that we can allow a driver to map
multiple MSI-X messages into a single IRQ when handling a message
shortage.
The changes include:
- Add a new pcib_if method: PCIB_MAP_MSI() which is called by the PCI bus
to calculate the address and data values for a given MSI/MSI-X IRQ.
The x86 nexus drivers map this into a call to a new 'msi_map()' function
in msi.c that does the mapping.
- Retire the pcib_if method PCIB_REMAP_MSIX() and remove the 'index'
parameter from PCIB_ALLOC_MSIX(). MD code no longer has any knowledge
of the MSI-X index for a given MSI-X IRQ.
- The PCI bus driver now stores more MSI-X state in a child's ivars.
Specifically, it now stores an array of IRQs (called "message vectors" in
the code) that have associated address and data values, and a small
virtual version of the MSI-X table that specifies the message vector
that a given MSI-X table entry uses. Sparse mappings are permitted in
the virtual table.
- The PCI bus driver now configures the MSI and MSI-X address/data
registers directly via custom bus_setup_intr() and bus_teardown_intr()
methods. pci_setup_intr() invokes PCIB_MAP_MSI() to determine the
address and data values for a given message as needed. The MD code
no longer has to call back down into the PCI bus code to set these
values from the nexus' bus_setup_intr() handler.
- The PCI bus code provides a callout (pci_remap_msi_irq()) that the MD
code can call to force the PCI bus to re-invoke PCIB_MAP_MSI() to get
new values of the address and data fields for a given IRQ. The x86
MSI code uses this when an MSI IRQ is moved to a different CPU, requiring
a new value of the 'address' field.
- The x86 MSI psuedo-driver loses a lot of code, and in fact the separate
MSI/MSI-X pseudo-PICs are collapsed down into a single MSI PIC driver
since the only remaining diff between the two is a substring in a
bootverbose printf.
- The PCI bus driver will now restore MSI-X state (including programming
entries in the MSI-X table) on device resume.
- The interface for pci_remap_msix() has changed. Instead of accepting
indices for the allocated vectors, it accepts a mini-virtual table
(with a new length parameter). This table is an array of u_ints, where
each value specifies which allocated message vector to use for the
corresponding MSI-X message. A vector of 0 forces a message to not
have an associated IRQ. The device may choose to only use some of the
IRQs assigned, in which case the unused IRQs must be at the "end" and
will be released back to the system. This allows a driver to use the
same remap table for different shortage values. For example, if a driver
wants 4 messages, it can use the same remap table (which only uses the
first two messages) for the cases when it only gets 2 or 3 messages and
in the latter case the PCI bus will release the 3rd IRQ back to the
system.
MFC after: 1 month
it via pci_get_vpd_*() rather than always reading it for each device during
boot. I've left the tunable so that it can still be turned off if a device
driver causes a lockup via a query to a broken device, but devices whose
drivers do not use VPD (the vast majority) should no longer result in
lockups during boot, and most folks should not need to tweak the tunable
now.
Tested on: bge(4)
Silence from: jmg
- First off, device drivers really do need to know if they are allocating
MSI or MSI-X messages. MSI requires allocating powerof2() messages for
example where MSI-X does not. To address this, split out the MSI-X
support from pci_msi_count() and pci_alloc_msi() into new driver-visible
functions pci_msix_count() and pci_alloc_msix(). As a result,
pci_msi_count() now just returns a count of the max supported MSI
messages for the device, and pci_alloc_msi() only tries to allocate MSI
messages. To get a count of the max supported MSI-X messages, use
pci_msix_count(). To allocate MSI-X messages, use pci_alloc_msix().
pci_release_msi() still handles both MSI and MSI-X messages, however.
As a result of this change, drivers using the existing API will only
use MSI messages and will no longer try to use MSI-X messages.
- Because MSI-X allows for each message to have its own data and address
values (and thus does not require all of the messages to have their
MD vectors allocated as a group), some devices allow for "sparse" use
of MSI-X message slots. For example, if a device supports 8 messages
but the OS is only able to allocate 2 messages, the device may make the
best use of 2 IRQs if it enables the messages at slots 1 and 4 rather
than default of using the first N slots (or indicies) at 1 and 2. To
support this, add a new pci_remap_msix() function that a driver may call
after a successful pci_alloc_msix() (but before allocating any of the
SYS_RES_IRQ resources) to allow the allocated IRQ resources to be
assigned to different message indices. For example, from the earlier
example, after pci_alloc_msix() returned a value of 2, the driver would
call pci_remap_msix() passing in array of integers { 1, 4 } as the
new message indices to use. The rid's for the SYS_RES_IRQ resources
will always match the message indices. Thus, after the call to
pci_remap_msix() the driver would be able to access the first message
in slot 1 at SYS_RES_IRQ rid 1, and the second message at slot 4 at
SYS_RES_IRQ rid 4. Note that the message slots/indices are 1-based
rather than 0-based so that they will always correspond to the rid
values (SYS_RES_IRQ rid 0 is reserved for the legacy INTx interrupt).
To support this API, a new PCIB_REMAP_MSIX() method was added to the
pcib interface to change the message index for a single IRQ.
Tested by: scottl
work:
- A new PCI quirk (PCI_QUIRK_DISABLE_MSI) is added to the quirk table.
- A new pci_msi_device_blacklisted() determines if a passed in device
matches an MSI quirk in the quirk table. This can be overridden (all
quirks ignored) by setting the hw.pci.honor_msi_blacklist to 0.
- A global blacklist check is performed in the MI PCI bus code by checking
to see if the device at 0:0:0 is blacklisted.
Tested by: jdp
- Add 3 new functions to the pci_if interface along with suitable wrappers
to provide the device driver visible API:
- pci_alloc_msi(dev, int *count) backed by PCI_ALLOC_MSI(). '*count'
here is an in and out parameter. The driver stores the desired number
of messages in '*count' before calling the function. On success,
'*count' holds the number of messages allocated to the device. Also on
success, the driver can access the messages as SYS_RES_IRQ resources
starting at rid 1. Note that the legacy INTx interrupt resource will
not be available when using MSI. Note that this function will allocate
either MSI or MSI-X messages depending on the devices capabilities and
the 'hw.pci.enable_msix' and 'hw.pci.enable_msi' tunables. Also note
that the driver should activate the memory resource that holds the
MSI-X table and pending bit array (PBA) before calling this function
if the device supports MSI-X.
- pci_release_msi(dev) backed by PCI_RELEASE_MSI(). This function
releases the messages allocated for this device. All of the
SYS_RES_IRQ resources need to be released for this function to succeed.
- pci_msi_count(dev) backed by PCI_MSI_COUNT(). This function returns
the maximum number of MSI or MSI-X messages supported by this device.
MSI-X is preferred if present, but this function will honor the
'hw.pci.enable_msix' and 'hw.pci.enable_msi' tunables. This function
should return the largest value that pci_alloc_msi() can return
(assuming the MD code is able to allocate sufficient backing resources
for all of the messages).
- Add default implementations for these 3 methods to the pci_driver generic
PCI bus driver. (The various other PCI bus drivers such as for ACPI and
OFW will inherit these default implementations.) This default
implementation depends on 4 new pcib_if methods that bubble up through
the PCI bridges to the MD code to allocate IRQ values and perform any
needed MD setup code needed:
- PCIB_ALLOC_MSI() attempts to allocate a group of MSI messages.
- PCIB_RELEASE_MSI() releases a group of MSI messages.
- PCIB_ALLOC_MSIX() attempts to allocate a single MSI-X message.
- PCIB_RELEASE_MSIX() releases a single MSI-X message.
- Add default implementations for these 4 methods that just pass the
request up to the parent bus's parent bridge driver and use the
default implementation in the various MI PCI bridge drivers.
- Add MI functions for use by MD code when managing MSI and MSI-X
interrupts:
- pci_enable_msi(dev, address, data) programs the MSI capability address
and data registers for a group of MSI messages
- pci_enable_msix(dev, index, address, data) initializes a single MSI-X
message in the MSI-X table
- pci_mask_msix(dev, index) masks a single MSI-X message
- pci_unmask_msix(dev, index) unmasks a single MSI-X message
- pci_pending_msix(dev, index) returns true if the specified MSI-X
message is currently pending
- Save the MSI capability address and data registers in the pci_cfgreg
block in a PCI devices ivars and restore the values when a device is
resumed. Note that the MSI-X table is not currently restored during
resume.
- Add constants for MSI-X register offsets and fields.
- Record interesting data about any MSI-X capability blocks we come
across in the pci_cfgreg block in the ivars for PCI devices.
Tested on: em (i386, MSI), bce (amd64/i386, MSI), mpt (amd64, MSI-X)
Reviewed by: scottl, grehan, jfv
MFC after: 2 months
only those bars that had addresses assigned by the BIOS and where the
bridges were properly programmed. Now even unprogrammed ones work.
This was needed for sun4v. We still only implement up to 2GB memory
ranges, even for 64-bit bars. PCI standards at least through 2.2 say
that this is the max (or 1GB is, I only know it is < 32bits).
o Always define pci_addr_t as uint64_t. A pci address is always 64-bits,
but some hosts can't address all of them.
o Preserve the upper half of the 64-bit word during resource probing.
o Test to make sure that 64-bit values can fit in a u_long (true on some
platforms, but not others). Don't use those that can't.
o minor pedantry about data sizes.
o Better bridge resource reporting in bootverbose case.
o Minor formatting changes to cope with different data types on different
platforms.
Submitted by: jmg, with many changes by me to fully support 64-bit
addresses.
to search for a specific extended capability. If the specified capability
is found for the given device, then the function returns success and
optionally returns the offset of that capability. If the capability is
not found, the function returns an error.
buses support querying the MAC address in a standard-for-that-bus way.
The base pci bus returns NULL for this IVAR always.
Submitted by: sam
Approved by: re (blanket for NEWCARD)
