-------- Original Message --------
Subject: Re: [PATCH 0/5]PCI: x86 MMCONFIG
Date: Wed, 19 Dec 2007 19:44:13 -0500
From: Tony Camuso <[email protected]>
Reply-To: [email protected]
To: Robert Hancock <[email protected]>
References: <[email protected]>
<[email protected]> <[email protected]>
Robert Hancock wrote:
> Greg KH wrote:
>
> I think we need more details on why this patch is needed. Also, we
> already have something like this in arch/x86/pci/mmconfig-shared.c, in
> the unreachable_devices function. This attempts to detect devices where
> MMCONFIG cannot access the configuration space (one of these would be at
> least one device in the AMD K8 built-in northbridge). If this is not
> sufficient, I would suggest expanding that mechanism instead of adding
> all this new code.
>
We thought of doing that. But IIRC, unreachable_devices() only covers
the first 16 buses on segment 0. What happens to the bitmap when you
want to cover all possible buses on all possible segments?
We felt that we had to come up with a more in-line way of doing this.
Tony Camuso wrote:
> Greg KH wrote:
>>
>> Sure, I realize this, but it solves the problem in one way for broken
>> hardware, such that it at least allows it to work, right? It also
>> provides a better incentive for the manufacturer to fix their bios,
>> which as you are on-site at HP, it would seem odd that they would just
>> not do that instead of trying to work around this in the kernel...
>>
>> thanks,
>>
>> greg k-h
>
> I don't think that many OEMs have that much control over the BIOS in
> their "value lines".
> :)
>
> And the MMCONFIG problem with enterprise systems and workstations, where
> we do control the BIOS (for the most part), is due to known bugs in
> certain versions of certain chipsets, HT1000, AMD8132, among them, not
> the BIOS.
>
> Anyway, we are devising better ways to deal with these anomalies
> than blacklists and telling customers to use "pci=nommconf"
>
> And we're bringing them to the community for discussion, improvement,
> and, we hope, acceptance.
First off, I would like to see confirmation from the horses's mouths
here (namely AMD, ServerWorks/Broadcom, and whoever else) that there is
no other way to get around this problem than disabling MMCONFIG for
accesses behind those chips.
The case of the device built into the K8 northbridge that's unreachable
by MMCONFIG kind of makes sense, since the northbridge is what's
translating the MMCONFIG memory access into config accesses. It seems
bizarre to me that a bridge chip could possibly have such a problem. The
MMCONFIG access should get translated into a configuration space access
in the northbridge and from that point on there's no difference between
an MMCONFIG and type1 access.
Look at MSI for another example, we recently had a patch from NVIDIA
posted to enable Hypertransport MSI mapping bits on some chipsets so
that MSI would function, because the BIOS failed to set them up
properly. Are we sure there's not a similar BIOS configuration issue
that could ideally be fixed in the BIOS, or else fixed up in the kernel?
--
Robert Hancock Saskatoon, SK, Canada
To email, remove "nospam" from [email protected]
Home Page: http://www.roberthancock.com/
Robert Hancock wrote:
> First off, I would like to see confirmation from the horses's mouths
> here (namely AMD, ServerWorks/Broadcom, and whoever else) that there is
> no other way to get around this problem than disabling MMCONFIG for
> accesses behind those chips.
>
I happen to have this one stored in my desktop.
From AMD-8132TM HyperTransportTM
PCI-X?2.0 Tunnel
Revision Guide
http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/30801.pdf
79 AMD-8132TM Tunnel Lacks Extended Configuration
Space Memory-Mapped I/O Base Address Register
Description
Current AMD processors do not natively support PCI-defined extended configuration space. A memory
mapped I/O base address register (MMIO BAR) is required in chipset devices to support extended
configuration space. The AMD-8132 does not have this MMIO BAR.
Potential Effect On System
The AMD-8132 is a PCI-X? Mode 2 capable device and requires the MMIO BAR to support extended
configuration space. Using a device which does have this MMIO BAR and an AMD-8132 on the same
HyperTransportTM link of the processor may cause firmware/software problems.
The base configuration space of the AMD-8132 and PCI(-X) devices attached to it are accessible using only
the mechanism defined in PCI 2.3. Registers of PCI-X Mode 2 devices attached to the AMD-8132 in the
extended configuration space are not accessible. The AMD-8132 has no registers in the extended
configuration space.
Suggested Workaround
It is strongly recommended that system designers do not connect the AMD-8132 and devices that use extended
configuration space MMIO BARs (ex: HyperTransport-to-PCI Express? bridges) to the same processor
HyperTransport link.
Fix Planned
No
> Robert Hancock wrote:
>
> First off, I would like to see confirmation from the horses's mouths
> here (namely AMD, ServerWorks/Broadcom, and whoever else) that there
> is no other way to get around this problem than disabling MMCONFIG for
> accesses behind those chips.
>
>
And here are the excerpts from that page of the spec which are salient
to the present discussion:
----------------------------------------------------------------------
The base configuration space of the AMD-8132 and PCI(-X) devices attached to it are accessible using only
the mechanism defined in PCI 2.3. Registers of PCI-X Mode 2 devices attached to the AMD-8132 in the
extended configuration space are not accessible. The AMD-8132 has no registers in the extended
configuration space.
Fix Planned
No
Robert Hancock wrote:
> The case of the device built into the K8 northbridge that's unreachable
> by MMCONFIG kind of makes sense, since the northbridge is what's
> translating the MMCONFIG memory access into config accesses. It seems
> bizarre to me that a bridge chip could possibly have such a problem. The
> MMCONFIG access should get translated into a configuration space access
> in the northbridge and from that point on there's no difference between
> an MMCONFIG and type1 access.
>
Robert's point is well taken.
Only northbridge chips can give us this kind of trouble, and the only
chips mentioned in the present discussion as not being mmconf-compliant
are northbridges (8132, ht1000).
The patch is aware of this, so once a root bus has been programmed for
legacy pci config access, all descendent buses automatically inherit
this access mechanism and are therefore not probed by the patch.
Tony Camuso wrote:
> Robert Hancock wrote:
>
>> First off, I would like to see confirmation from the horses's mouths
>> here (namely AMD, ServerWorks/Broadcom, and whoever else) that there
>> is no other way to get around this problem than disabling MMCONFIG for
>> accesses behind those chips.
>>
>
> I happen to have this one stored in my desktop.
>
> From AMD-8132TM HyperTransportTM
> PCI-X?2.0 Tunnel
> Revision Guide
>
> http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/30801.pdf
>
>
> 79 AMD-8132TM Tunnel Lacks Extended Configuration
> Space Memory-Mapped I/O Base Address Register
>
> Description
>
> Current AMD processors do not natively support PCI-defined extended
> configuration space. A memory
> mapped I/O base address register (MMIO BAR) is required in chipset
> devices to support extended
> configuration space. The AMD-8132 does not have this MMIO BAR.
> Potential Effect On System
>
> The AMD-8132 is a PCI-X? Mode 2 capable device and requires the MMIO BAR
> to support extended
> configuration space. Using a device which does have this MMIO BAR and an
> AMD-8132 on the same
> HyperTransportTM link of the processor may cause firmware/software
> problems.
>
> The base configuration space of the AMD-8132 and PCI(-X) devices
> attached to it are accessible using only
> the mechanism defined in PCI 2.3. Registers of PCI-X Mode 2 devices
> attached to the AMD-8132 in the
> extended configuration space are not accessible. The AMD-8132 has no
> registers in the extended
> configuration space.
>
> Suggested Workaround
>
> It is strongly recommended that system designers do not connect the
> AMD-8132 and devices that use extended
> configuration space MMIO BARs (ex: HyperTransport-to-PCI Express?
> bridges) to the same processor
> HyperTransport link.
>
> Fix Planned
> No
That does sound fairly definitive. I have to wonder why certain system
designers then didn't follow their strong recommendation..
--
Robert Hancock Saskatoon, SK, Canada
To email, remove "nospam" from [email protected]
Home Page: http://www.roberthancock.com/
Robert Hancock wrote:
> I have to wonder why certain system
> designers then didn't follow their strong recommendation..
>
I don't think I want to go there.
I used to be a hardware/firmware guy.
:D :D
On 12/20/2007 9:15 PM, Robert Hancock wrote:
>>
>> Suggested Workaround
>>
>> It is strongly recommended that system designers do not connect the
>> AMD-8132 and devices that use extended
>> configuration space MMIO BARs (ex: HyperTransport-to-PCI Express?
>> bridges) to the same processor
>> HyperTransport link.
>>
>> Fix Planned
>> No
>
> That does sound fairly definitive. I have to wonder why certain system
> designers then didn't follow their strong recommendation..
Just curious, do you know of any system where that recommendation was
not followed? On all motherboards where I have seen a AMD-8131 or a
AMD-8132, they were alone on their hypertransport link, and other
"northbridges" (more precisely hypertransport to pci-express or
pci-whatever, often nvidia) with a "MMCONFIG BAR" where on one of the
other available hypertransport links in the system.
Loic
Loic Prylli wrote:
>
> Just curious, do you know of any system where that recommendation was
> not followed? On all motherboards where I have seen a AMD-8131 or a
> AMD-8132, they were alone on their hypertransport link, and other
> "northbridges" (more precisely hypertransport to pci-express or
> pci-whatever, often nvidia) with a "MMCONFIG BAR" where on one of the
> other available hypertransport links in the system.
>
>
> Loic
>
Here is the PCI configuration of the HP DL585G2.
You can see two nVidia CK804 PCIE root ports at bus 0 and bus 0x40.
Each of them has an 8132 connected as a subordinate bridge.
[root@hp-dl585g2-01 ~]# lspci -vt
-+-[0000:40]-+-00.0 nVidia Corporation CK804 Memory Controller
| +-01.0 nVidia Corporation CK804 Memory Controller
| +-0b.0-[0000:4f-51]--
| +-0c.0-[0000:4c-4e]--
| +-0d.0-[0000:49-4b]--
| +-0e.0-[0000:46-48]--
| +-10.0-[0000:41]--+-01.0 Broadcom Corporation NetXtreme II BCM5706 Gigabit Ethernet
| | \-02.0 Broadcom Corporation NetXtreme II BCM5706 Gigabit Ethernet
| +-10.1 Advanced Micro Devices [AMD] AMD-8132 PCI-X IOAPIC
| +-11.0-[0000:42-45]--
| \-11.1 Advanced Micro Devices [AMD] AMD-8132 PCI-X IOAPIC
\-[0000:00]-+-00.0 nVidia Corporation CK804 Memory Controller
+-01.0 nVidia Corporation CK804 ISA Bridge
+-02.0 nVidia Corporation CK804 USB Controller
+-02.1 nVidia Corporation CK804 USB Controller
+-06.0 nVidia Corporation CK804 IDE
+-09.0-[0000:01]--+-03.0 ATI Technologies Inc ES1000
| +-04.0 Compaq Computer Corporation Integrated Lights Out Controller
| +-04.2 Compaq Computer Corporation Integrated Lights Out Processor
| +-04.4 Hewlett-Packard Company Proliant iLO2 virtual USB controller
| \-04.6 Hewlett-Packard Company Proliant iLO2 virtual UART
+-0c.0-[0000:08-0a]----00.0 Hewlett-Packard Company Smart Array Controller
+-0d.0-[0000:05-07]--
+-0e.0-[0000:02-04]--
+-18.0 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
+-18.1 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
+-18.2 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
+-18.3 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
+-19.0 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
+-19.1 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
+-19.2 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
+-19.3 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
+-1a.0 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
+-1a.1 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
+-1a.2 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
+-1a.3 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
+-1b.0 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
+-1b.1 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
+-1b.2 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
\-1b.3 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
[root@hp-dl585g2-01 ~]#
Robert Hancock <[email protected]> writes:
> First off, I would like to see confirmation from the horses's mouths
> here (namely AMD,
AMD publicly releases errata sheets/data sheets for their PCI bridges
(check their website). I haven't checked the 8132 errata for this
though. Not sure it implements MMCONFIG at all.
However the PCI Express chipsets typically do implement
their own MMCONFIG aperture.
> The case of the device built into the K8 northbridge that's
> unreachable by MMCONFIG kind of makes sense,
The internal northbridge devices on K8 are not reachable through
mmconfig. While BIOS are supposed to express this in MCFG by excluding
that bus many don't. That was the original reason I added the
type1<->mcfg sanity check. It catches the K8 case fine.
> since the northbridge is
> what's translating the MMCONFIG memory access into config accesses.
The way it works on K8 systems is that the CPU internal northbridge
knows nothing about MMCONFIG, but that the external chipsets
implement an MMCONFIG aperture on their own outside the northbridge.
If you have multiple bridges like some SLI K8 setups that could
be multiple ones.
This has changed on the Quad Core Fam10h CPUs BTW -- there the
NB can deliver an single mmconfig aperture that is translated
to appropiate transactions on the Hypertransport link.
What might happen with K8 and 8132 (I'm speculating here) is that
they got a PCI Express chipset that implements an MMCONFIG
aperture for its devices, but the system also has a PCI-X 8132
bridge and the MMCONFIG aperture inside the chipset doesn't
support talking to the 8132 which might be "upstream" in the HT
topology. And the BIOS' MCFG doesn't tell Linux that.
-Andi
Tony Camuso wrote:
>> Robert Hancock wrote:
>>
>> First off, I would like to see confirmation from the horses's mouths
>> here (namely AMD, ServerWorks/Broadcom, and whoever else) that there
>> is no other way to get around this problem than disabling MMCONFIG
>> for accesses behind those chips.
>>
>>
>
> And here are the excerpts from that page of the spec which are salient
> to the present discussion:
>
> ----------------------------------------------------------------------
>
> The base configuration space of the AMD-8132 and PCI(-X) devices
> attached to it are accessible using only
> the mechanism defined in PCI 2.3. Registers of PCI-X Mode 2 devices
> attached to the AMD-8132 in the
> extended configuration space are not accessible. The AMD-8132 has no
> registers in the extended
> configuration space.
>
> Fix Planned
> No
>
>
Prarit, Tony et al, this is as official as it gets. It's pretty
clearly stated.
Bhavana
Loic Prylli wrote:
> On 12/20/2007 6:21 PM, Tony Camuso wrote:
>> And the MMCONFIG problem with enterprise systems and workstations, where
>> we do control the BIOS (for the most part), is due to known bugs in
>> certain versions of certain chipsets, HT1000, AMD8132, among them, not
>> the BIOS.
>
>
>
> The lack of MMCONFIG support is indeed because some hypertransport
> chipsets lack that support. But there are some BIOSes out there that are
> advertising support for all busses in their MCFG acpi attribute (even
> the busses managed by some amd8131 in a mixed nvidia-ck804/amd8131
> motherboard), and the BIOS seems at least faulty for advertising a
> capability that does not exist.
This didn't really occur to me before for some reason. But yes, the MCFG
table lists the buses to which each MMCONFIG region is applicable. If
there are entire buses which MMCONFIG cannot access, it should not be
indicating they are accessible via MMCONFIG in the ACPI MCFG table. If
it is, then it's truly a BIOS bug.
Unless of course Linux isn't handling what the MCFG table is indicating
properly. Then it's our bug. It would be good to verify this on one of
the systems involved..
One of the things this patch (currently in -mm) does is dump out the
segment and starting/ending buses for each MCFG configuration listed.
The dmesg from this patch applied on such a system would tell you which
is the case:
http://git.kernel.org/?p=linux/kernel/git/x86/linux-2.6-x86.git;a=commit;h=e18c985289ee356f06dbc953281a3c140a02fbb3
--
Robert Hancock Saskatoon, SK, Canada
To email, remove "nospam" from [email protected]
Home Page: http://www.roberthancock.com/
I need to resubmit this patch.
Resubmission will be in my next email.
It was a minor problem whereby the patch would advise the user
If a device isn't working, try "pci=nommconf".
even when "pci=mmconf" was used at the command line.
One additional line of logic fixed it, but I would like to resubmit
the patch rather than patch the patch.
Loic Prylli wrote:
> I just realized one thing: the bar sizing code in pci_read_bases() (that
> writes 0xffffffff in the bars) does not seem to disable the
> PCI_COMMAND_MEM/PCI_COMMAND_IO bits in the cmd register before
> manipulating the BARs. And it seems nobody else ensures they are
> disabled at this point either (or am I missing something?).
No you're not missing anything. This problem causes many machines to
break horribly when MMCONFIG is enabled. There's a patch in -mm to fix
this. (It special-cases the case of host bridges and doesn't disable the
decode bits for those, since some are known to do crazy things if you
do that.)
http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.24-rc6/2.6.24-rc6-mm1/broken-out/pci-disable-decoding-during-sizing-of-bars.patch
>
> Touching the bars while they are enabled would be buggy behaviour from
> our part, and something trivial to fix. And it might well fix that
> particular problem (it's fair play from the machine to crash if we
> create a decoding conflict, simply disabling the cmd bits in
> pci_read_bases() should remove that conflict).
>
> FWIW, to partially answer your last question, Windows does disable
> mem-space and/or IO-space when sizing the bars of a device (I have some
> traces of configuration-space-access taken on a window machine for one
> of the PCI busses).
Good to know. There was some speculation that it did not.
--
Robert Hancock Saskatoon, SK, Canada
To email, remove "nospam" from [email protected]
Home Page: http://www.roberthancock.com/