[+ Yinghai]
On Mon, Feb 29, 2016 at 02:03:45PM -0500, Sinan Kaya wrote:
> On 2/16/2016 8:53 AM, Tomasz Nowicki wrote:
> > From the functionality point of view this series might be split into the
> > following logic parts:
> > 1. Make MMCONFIG code arch-agnostic which allows all architectures to collect
> > PCI config regions and used when necessary.
> > 2. Move non-arch specific bits to the core code.
> > 3. Use MMCONFIG code and implement generic ACPI based PCI host controller driver.
> > 4. Enable above driver on ARM64
> >
> > Patches has been built on top of 4.5-rc3 and can be found here:
> > [email protected]:semihalf-nowicki-tomasz/linux.git (pci-acpi-v5)
> >
> > NOTE, this patch set depends on Lorenzo's fixes:
> > https://patchwork.ozlabs.org/patch/576450/
> > which can be found in pci-acpi-v5 branch.
> >
> > This has been tested on Cavium ThunderX server, JunoR2, HP RX2660 IA64, x86,
> > Hip05, X-Gene and QEMU-aarch64. Any help in reviewing and testing is very appreciated.
> >
> > v4 -> v5
> > - dropped MCFG refactoring group patches 1-6 from series v4 and integrated Jayachandran's patch
> > https://patchwork.ozlabs.org/patch/575525/
> > - rewrite PCI legacy IRQs allocation
> > - squashed two patches 11 and 12 from series v4, fixed bisection issue
> > - changelog improvements
> > - rebased to 4.5-rc3
> >
> > v3 -> v4
> > - dropped Jiang's fix http://lkml.iu.edu/hypermail/linux/kernel/1601.1/04318.html
> > - added Lorenzo's fix patch 19/24
> > - ACPI PCI bus domain number assigning cleanup
> > - changed resource management, we now claim and reassign resources
> > - improvements for applying quirks
> > - dropped Matthew's http://www.spinics.net/lists/linux-pci/msg45950.html dependency
> > - rebased to 4.5-rc1
> >
>
> Having tested v4 and v5, I'm seeing some resource assignment problems
> and address conflicts. And problems booting QEMU.
I asked Tomasz to add resource claiming code in v4 to make sure that,
if FW has left resources in a reasonable set-up, we reuse it as-is.
Now, I was and I am aware this could trigger resource allocation
issues (in particular in relation to bridges apertures sizing),
that can be nonetheless solved by forcing the kernel to reallocate
resources (pci=realloc, that's exactly what's there for, release
the bridge apertures, resize the busses downstream and reassign
the respective hierarchy).
I am not entirely aware of how consistently pci=realloc was used on
x86, what I am aware of is the panoply of pci=* command line parameters
defined for x86 and I would certainly avoid that.
The decision on whether we claim resources before reassigning them
is either dictacted by the boot method (ie ACPI->claim resources by
default) or we should control it via a FW option or a command
line option, PCI standard (PCI FW revision 3.1, 3.5 "Device State
at Firmware/Operating System Handoff) IIUC does not stricly mandate
FW configuring the whole PCI hierarchy (and to be 100% compliant
we should check the device IO/MEM enable bits before claiming, as x86 does
- see pcibios_allocate_dev_resources() in arch/x86/pci/i386.c).
x86 and IA64 claim PCI resources on boot and live with that (well, minus
the gazillions x86 pci= parameters that change the PCI resources assignment
one way or another), comments very welcome in particular on the pci=realloc
option and its usage.
What's certain is, if we do not claim resources by default we will *never*
be able to do it, it will certainly trigger regressions.
Lorenzo
On 3/3/2016 6:23 AM, Lorenzo Pieralisi wrote:
> x86 and IA64 claim PCI resources on boot and live with that (well, minus
> the gazillions x86 pci= parameters that change the PCI resources assignment
> one way or another), comments very welcome in particular on the pci=realloc
> option and its usage.
I have been working with Linux PCIe over 3 years. I never used pci=realloc argument.
The v5 series minus [PATCH V5 11/15] drivers: pci: add generic code to claim bus resources
is working just fine and is ready to go upstream in my opinion. It passed my internal
testing with different types of endpoints.
The inclusion of this patch is now requiring everybody to add pci=realloc argument
otherwise the resources assigned by the UEFI BIOS are not working.
I think there is still some work to be done in this patch and is too early to be included
into the series. It is blocking progress of the series which is sitting on review over 1
year already.
[ 0.752916] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x80360800000-0x8037fffffff 64bit pref] (contains BAR2 for 63 VFs)
[ 0.771799] pci 0000:00:00.0: PCI bridge to [bus 01-06]
[ 0.777054] pci 0000:00:00.0: root [mem 0x80100100000-0x8013fffffff window] res [mem 0x8013ff00000-0x8013fffffff] nr 14
[ 0.787846] pci 0000:00:00.0: pci_claim_bridge_resource:714 1: i:14
[ 0.794135] pci 0000:00:00.0: root [mem 0x80300000000-0x8037fffffff window] res [mem 0x80360000000-0x8037fffffff 64bit pref] nr 15
[ 0.805881] pci 0000:00:00.0: pci_claim_bridge_resource:714 1: i:15
[ 0.812155] pci 0000:01:00.0: root [mem 0x8013ff00000-0x8013fffffff] res [mem 0x8013ff00000-0x8013fffffff 64bit] nr 0
[ 0.822773] pci 0000:01:00.0: root [mem 0x80360000000-0x8037fffffff 64bit pref] res [mem 0x80360000000-0x803607fffff 64bit pref] nr 2
[ 0.834778] pci 0000:01:00.0: root [mem 0x80360000000-0x8037fffffff 64bit pref] res [mem 0x8037ff00000-0x8037fffffff pref] nr 6
[ 0.846265] pci 0000:01:00.0: can't claim BAR 9 [mem 0x80360800000-0x8037fffffff 64bit pref]: address conflict with 0000:01:00.0 [mem 0x8037ff00000-0x8037fffffff pref]
[ 0.861237] pci 0000:01:00.0: BAR 9: no space for [mem size 0x1f800000 64bit pref]
[ 0.868811] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x1f800000 64bit pref]
I keep saying this but the type of CPU is not important when it comes to PCIe. Both PCIe and
ACPI are governed by specs. If it is working for x86 and i64; it needs to work for ARM64 as
well.
Even ARM64 has the luxury to omit the old BIOS behaviors. Most ARM64 systems use tianocore based
UEFI BIOS.
This is pointing to an implementation problem in arm64 adaptation. Need to figure out
what is different.
--
Sinan Kaya
Qualcomm Technologies, Inc. on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
On Thu, Mar 03, 2016 at 09:24:56AM -0500, Sinan Kaya wrote:
> On 3/3/2016 6:23 AM, Lorenzo Pieralisi wrote:
> > x86 and IA64 claim PCI resources on boot and live with that (well, minus
> > the gazillions x86 pci= parameters that change the PCI resources assignment
> > one way or another), comments very welcome in particular on the pci=realloc
> > option and its usage.
>
> I have been working with Linux PCIe over 3 years. I never used
> pci=realloc argument.
>
> The v5 series minus [PATCH V5 11/15] drivers: pci: add generic code to
> claim bus resources is working just fine and is ready to go upstream
> in my opinion. It passed my internal testing with different types of
> endpoints.
>
> The inclusion of this patch is now requiring everybody to add
> pci=realloc argument otherwise the resources assigned by the UEFI BIOS
> are not working.
>
> I think there is still some work to be done in this patch and is too
> early to be included into the series. It is blocking progress of the
> series which is sitting on review over 1 year already.
First off, I think that's specious, patch 11 is not blocking anything,
if you and Tomasz want to drop it go ahead and take responsibility
of the consequences.
I am not saying patch 11 is perfect, it is there to review, if you
spot bugs point them out.
If you are interested and willing to make an effort to understand why I
asked Tomasz to integrate it, a bit of background here:
http://permalink.gmane.org/gmane.linux.kernel.pci/44830
If we want to drop patch 11, we are going to discard whatever FW
set-up at FW/OS hand-off and reassign everything. Want to do it ?
Go ahead.
I wrote it in my previous email, probably it was not clear, so, here we
go again.
If we want to at least consider the FW PCI configuration at FW/OS
handoff, we should read the PCI bridge apertures and claim them, when
that fails reassign the corresponding PCI bus hierarchy (which means
releasing the bridge resources and downstream devices and reassign
them), that's what pci=realloc does.
I think that it is a command line option since it has to be a choice,
ie overriding FW set-up should be an option, not a default.
Patch 11 does what x86 does in arch code arch/x86/pci/i386.c,
pcibios_resource_survey()
and that works for them (of course, minus quirks that do exist).
I could integrate the code implementing pci=realloc in patch 11 so
that we realloc by default all resources claimed that failed (which
means that bridges are resized accordingly and you won't be forced
to use pci=realloc on command line).
> [ 0.752916] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x80360800000-0x8037fffffff 64bit pref] (contains BAR2 for 63 VFs)
> [ 0.771799] pci 0000:00:00.0: PCI bridge to [bus 01-06]
> [ 0.777054] pci 0000:00:00.0: root [mem 0x80100100000-0x8013fffffff window] res [mem 0x8013ff00000-0x8013fffffff] nr 14
> [ 0.787846] pci 0000:00:00.0: pci_claim_bridge_resource:714 1: i:14
> [ 0.794135] pci 0000:00:00.0: root [mem 0x80300000000-0x8037fffffff window] res [mem 0x80360000000-0x8037fffffff 64bit pref] nr 15
> [ 0.805881] pci 0000:00:00.0: pci_claim_bridge_resource:714 1: i:15
> [ 0.812155] pci 0000:01:00.0: root [mem 0x8013ff00000-0x8013fffffff] res [mem 0x8013ff00000-0x8013fffffff 64bit] nr 0
> [ 0.822773] pci 0000:01:00.0: root [mem 0x80360000000-0x8037fffffff 64bit pref] res [mem 0x80360000000-0x803607fffff 64bit pref] nr 2
> [ 0.834778] pci 0000:01:00.0: root [mem 0x80360000000-0x8037fffffff 64bit pref] res [mem 0x8037ff00000-0x8037fffffff pref] nr 6
> [ 0.846265] pci 0000:01:00.0: can't claim BAR 9 [mem 0x80360800000-0x8037fffffff 64bit pref]: address conflict with 0000:01:00.0 [mem 0x8037ff00000-0x8037fffffff pref]
> [ 0.861237] pci 0000:01:00.0: BAR 9: no space for [mem size 0x1f800000 64bit pref]
> [ 0.868811] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x1f800000 64bit pref]
>
>
> I keep saying this but the type of CPU is not important when it comes
> to PCIe. Both PCIe and ACPI are governed by specs. If it is working
> for x86 and i64; it needs to work for ARM64 as well.
That's theory. In practice there is massive legacy there and PCI resource
assignment is carried out in an arch specific way (otherwise there would
be no pci claiming/assignment code in arch/* right ?) and the resource
claiming/assignment strictly depends on FW set-up, like it or lump it,
that's the way it *currently* is.
I wrote in my previous email, the status of PCI resources at OS/FW
handoff is not strictly mandated by the PCI standard AFAIK (it is
covered by 3.5 "Device state at Firmware/Operating System Handoff" in
the PCI FW spec revision 3.1), so what I suggest above is the only option
we have (or you just discard FW configuration altogether, that's what
happens if all PCI resources are reassigned, it is a choice to be made
and it is neither correct nor wrong, I wish it would).
> Even ARM64 has the luxury to omit the old BIOS behaviors. Most ARM64
> systems use tianocore based UEFI BIOS.
>
> This is pointing to an implementation problem in arm64 adaptation.
> Need to figure out what is different.
Look no further, firmware is different. How do we want to proceed ?
Lorenzo
On 04.03.2016 11:55, Lorenzo Pieralisi wrote:
> On Thu, Mar 03, 2016 at 09:24:56AM -0500, Sinan Kaya wrote:
>> >On 3/3/2016 6:23 AM, Lorenzo Pieralisi wrote:
>>> > >x86 and IA64 claim PCI resources on boot and live with that (well, minus
>>> > >the gazillions x86 pci= parameters that change the PCI resources assignment
>>> > >one way or another), comments very welcome in particular on the pci=realloc
>>> > >option and its usage.
>> >
>> >I have been working with Linux PCIe over 3 years. I never used
>> >pci=realloc argument.
>> >
>> >The v5 series minus [PATCH V5 11/15] drivers: pci: add generic code to
>> >claim bus resources is working just fine and is ready to go upstream
>> >in my opinion. It passed my internal testing with different types of
>> >endpoints.
>> >
>> >The inclusion of this patch is now requiring everybody to add
>> >pci=realloc argument otherwise the resources assigned by the UEFI BIOS
>> >are not working.
>> >
>> >I think there is still some work to be done in this patch and is too
>> >early to be included into the series. It is blocking progress of the
>> >series which is sitting on review over 1 year already.
> First off, I think that's specious, patch 11 is not blocking anything,
> if you and Tomasz want to drop it go ahead and take responsibility
> of the consequences.
>
> I am not saying patch 11 is perfect, it is there to review, if you
> spot bugs point them out.
>
> If you are interested and willing to make an effort to understand why I
> asked Tomasz to integrate it, a bit of background here:
>
> http://permalink.gmane.org/gmane.linux.kernel.pci/44830
>
> If we want to drop patch 11, we are going to discard whatever FW
> set-up at FW/OS hand-off and reassign everything. Want to do it ?
> Go ahead.
>
> I wrote it in my previous email, probably it was not clear, so, here we
> go again.
>
> If we want to at least consider the FW PCI configuration at FW/OS
> handoff, we should read the PCI bridge apertures and claim them, when
> that fails reassign the corresponding PCI bus hierarchy (which means
> releasing the bridge resources and downstream devices and reassign
> them), that's what pci=realloc does.
>
> I think that it is a command line option since it has to be a choice,
> ie overriding FW set-up should be an option, not a default.
>
> Patch 11 does what x86 does in arch code arch/x86/pci/i386.c,
>
> pcibios_resource_survey()
>
> and that works for them (of course, minus quirks that do exist).
>
> I could integrate the code implementing pci=realloc in patch 11 so
> that we realloc by default all resources claimed that failed (which
> means that bridges are resized accordingly and you won't be forced
> to use pci=realloc on command line).
>
I agree with Lorenzo. Just because v3 works it does not mean we want to
go this way. Also, I think we should realloc all resources claimed that
failed, w/o need to use pci=realloc on command line.
Tomasz
Posting on top of Tomasz's email...
On 3/4/2016 7:01 AM, Tomasz Nowicki wrote:
> On 04.03.2016 11:55, Lorenzo Pieralisi wrote:
>> On Thu, Mar 03, 2016 at 09:24:56AM -0500, Sinan Kaya wrote:
>>> >On 3/3/2016 6:23 AM, Lorenzo Pieralisi wrote:
>>>> > >x86 and IA64 claim PCI resources on boot and live with that (well, minus
>>>> > >the gazillions x86 pci= parameters that change the PCI resources assignment
>>>> > >one way or another), comments very welcome in particular on the pci=realloc
>>>> > >option and its usage.
>>> >
>>> >I have been working with Linux PCIe over 3 years. I never used
>>> >pci=realloc argument.
>>> >
>>> >The v5 series minus [PATCH V5 11/15] drivers: pci: add generic code to
>>> >claim bus resources is working just fine and is ready to go upstream
>>> >in my opinion. It passed my internal testing with different types of
>>> >endpoints.
>>> >
>>> >The inclusion of this patch is now requiring everybody to add
>>> >pci=realloc argument otherwise the resources assigned by the UEFI BIOS
>>> >are not working.
>>> >
>>> >I think there is still some work to be done in this patch and is too
>>> >early to be included into the series. It is blocking progress of the
>>> >series which is sitting on review over 1 year already.
>> First off, I think that's specious, patch 11 is not blocking anything,
>> if you and Tomasz want to drop it go ahead and take responsibility
>> of the consequences.
>>
>> I am not saying patch 11 is perfect, it is there to review, if you
>> spot bugs point them out.
>>
>> If you are interested and willing to make an effort to understand why I
>> asked Tomasz to integrate it, a bit of background here:
>>
>> http://permalink.gmane.org/gmane.linux.kernel.pci/44830
>>
>> If we want to drop patch 11, we are going to discard whatever FW
>> set-up at FW/OS hand-off and reassign everything. Want to do it ?
>> Go ahead.
Yes, we should ideally reuse the BAR addresses assigned by FW. And, it is not
working as I said. It happens to work on Intel architectures. I'm saying that
this patch needs some more work not that it is right or wrong.
How long it takes to figure this out is the question? It could have been dealt
with separately. If you can come up with a solution in the near future,
I'm ready to test.
There was some big push on v3 to get it tested by multiple vendors. I was under
the impression that we are trying to get some version accepted.
Since I'm the only one complaining right now, I guess nobody else is testing
v4 and v5.
>>
>> I wrote it in my previous email, probably it was not clear, so, here we
>> go again.
>>
>> If we want to at least consider the FW PCI configuration at FW/OS
>> handoff, we should read the PCI bridge apertures and claim them, when
>> that fails reassign the corresponding PCI bus hierarchy (which means
>> releasing the bridge resources and downstream devices and reassign
>> them), that's what pci=realloc does.
>>
>> I think that it is a command line option since it has to be a choice,
>> ie overriding FW set-up should be an option, not a default.
>>
>> Patch 11 does what x86 does in arch code arch/x86/pci/i386.c,
>>
>> pcibios_resource_survey()
>>
>> and that works for them (of course, minus quirks that do exist).
>>
>> I could integrate the code implementing pci=realloc in patch 11 so
>> that we realloc by default all resources claimed that failed (which
>> means that bridges are resized accordingly and you won't be forced
>> to use pci=realloc on command line).
>>
>
> I agree with Lorenzo. Just because v3 works it does not mean we want to go this way. Also, I think we should realloc all resources claimed that failed, w/o need to use pci=realloc on command line.
>
Let's give this a try. I have seen the kernel messages with and without realloc option
too. I don't want to see any kind of error messages if it is actually working.
I don't want to get a support request that PCIe is broken even though it is not just
because of some error message in boot log that eventually got corrected by the
architecture.
> Tomasz
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Sinan Kaya
Qualcomm Technologies, Inc. on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
On Fri, Mar 04, 2016 at 09:52:17AM -0500, Sinan Kaya wrote:
[...]
> >> I could integrate the code implementing pci=realloc in patch 11 so
> >> that we realloc by default all resources claimed that failed (which
> >> means that bridges are resized accordingly and you won't be forced
> >> to use pci=realloc on command line).
> >>
> >
> > I agree with Lorenzo. Just because v3 works it does not mean we want to go this way. Also, I think we should realloc all resources claimed that failed, w/o need to use pci=realloc on command line.
> >
>
> Let's give this a try. I have seen the kernel messages with and
> without realloc option too. I don't want to see any kind of error
> messages if it is actually working.
I agree, claiming resources failures are too noisy, it is a pet-peeve
of mine too. The code to realloc resources is in the kernel already,
it is just a matter of defining how to use it (ie trigger it by default
without command line option - actually the kernel can be already
compiled to enable realloc by default, see CONFIG_PCI_REALLOC_ENABLE_AUTO),
that's why I added Yinghai to the thread, Bjorn and him have more
insights on how this has been used on current systems and I am really keen
on getting their opinion, they have more visibility into this than I do,
writing the patch itself should be simple enough.
Thanks !
Lorenzo