2022-09-13 13:59:22

by Xiaochun Lee

[permalink] [raw]
Subject: [PATCH v1] PCI: Set no io resource for bridges that behind VMD controller

From: Xiaochun Lee <[email protected]>

When enable VMDs on Intel CPUs, VMD controllers(8086:28c0) be
recognized by VMD driver and there are many failed messages of
BAR 13 when scan the bridges and assign IO resource behind it
as listed below, the bridge wants to get 0x6000 as its IO
resource, but there is no IO resources on the host bridge.

VMD host bridge resources:
vmd 0000:64:00.5: PCI host bridge to bus 10000:80
pci_bus 10000:80: root bus resource [bus 80-9f]
pci_bus 10000:80: root bus resource [mem 0xe0000000-0xe1ffffff]
pci_bus 10000:80: root bus resource [mem 0x24ffff02010-0x24fffffffff 64bit]

Failed messages of BAR#13:
pci 10000:80:02.0: BAR 13: no space for [io size 0x1000]
pci 10000:80:02.0: BAR 13: failed to assign [io size 0x1000]
pci 10000:80:03.0: BAR 13: no space for [io size 0x1000]
pci 10000:80:03.0: BAR 13: failed to assign [io size 0x1000]

VMD-enabled root ports use
Enhanced Configuration Access Mechanism (ECAM) access
PCI Express configuration space, and offer VMD_CFGBAR as
base of PCI Express configuration space for the bridges
behind it. The configuration space includes IO resources,
but these IO resources are not actually used on X86,
especially the NVMes as device connected on this hot plug
bridges, and it can result in BAR#13 assign IO resource
failed. So we clear IO resources by setting an IO base value
greater than limit to these bridges. Hence, we can leverage
kernel parameter "pci=hpiosize=0KB" to avoid this failed
messages show out.

Signed-off-by: Xiaochun Lee <[email protected]>
---
drivers/pci/quirks.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 57 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 4944798..f8a37f0 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -5956,3 +5956,60 @@ static void aspm_l1_acceptable_latency(struct pci_dev *dev)
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency);
#endif
+
+#if defined(CONFIG_X86_64) || defined(CONFIG_X86)
+/*
+ * VMD-enabled root ports use Enhanced Configuration Access Mechanism (ECAM)
+ * access PCI Express configuration space, and offer VMD_CFGBAR as
+ * base of PCI Express configuration space for the bridges behind it.
+ * The configuration space includes IO resources, but these IO
+ * resources are not actually used on X86, especially the NVMes as
+ * device connnected on this hot plug bridges, and it can result
+ * in BAR#13 assign IO resource failed. So we clear IO resources
+ * by setting an IO base value greater than limit to these bridges.
+ * Hence, append kernel parameter "pci=hpiosize=0KB" can avoid
+ * this BAR#13 failed messages show out.
+ */
+static void quirk_vmd_no_iosize(struct pci_dev *bridge)
+{
+ u8 io_base_lo, io_limit_lo;
+ u16 io_low;
+ u32 io_upper16;
+ unsigned long io_mask, base, limit;
+
+ io_mask = PCI_IO_RANGE_MASK;
+ if (bridge->io_window_1k)
+ io_mask = PCI_IO_1K_RANGE_MASK;
+
+ /* VMD Domain */
+ if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
+ pci_read_config_byte(bridge, PCI_IO_BASE, &io_base_lo);
+ pci_read_config_byte(bridge, PCI_IO_LIMIT, &io_limit_lo);
+ base = (io_base_lo & io_mask) << 8;
+ limit = (io_limit_lo & io_mask) << 8;
+ if (limit >= base) {
+ /* if there are defined io ports behind the bridge on x86,
+ * we clear it, since there is only 64KB IO resource on it,
+ * beyond that, hotplug io bridges don't needs IO port resource,
+ * such as NVMes attach on it. So the corresponding range must be
+ * turned off by writing base value greater than limit to the
+ * bridge's base/limit registers.
+ */
+
+ /* Clear upper 16 bits of I/O base/limit */
+ io_upper16 = 0;
+ /* set base value greater than limit */
+ io_low = 0x00f0;
+
+ /* Temporarily disable the I/O range before updating PCI_IO_BASE */
+ pci_write_config_dword(bridge, PCI_IO_BASE_UPPER16, 0x0000ffff);
+ /* Update lower 16 bits of I/O base/limit */
+ pci_write_config_word(bridge, PCI_IO_BASE, io_low);
+ /* Update upper 16 bits of I/O base/limit */
+ pci_write_config_dword(bridge, PCI_IO_BASE_UPPER16, io_upper16);
+ }
+ }
+}
+DECLARE_PCI_FIXUP_CLASS_HEADER(PCI_ANY_ID, PCI_ANY_ID,
+ PCI_CLASS_BRIDGE_PCI, 8, quirk_vmd_no_iosize);
+#endif
--
1.8.3.1


2022-09-23 15:33:26

by kernel test robot

[permalink] [raw]
Subject: Re: [PATCH v1] PCI: Set no io resource for bridges that behind VMD controller

Hi Xiaochun,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on helgaas-pci/next]
[also build test ERROR on linus/master v6.0-rc6 next-20220923]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url: https://github.com/intel-lab-lkp/linux/commits/Xiaochun-Lee/PCI-Set-no-io-resource-for-bridges-that-behind-VMD-controller/20220913-213745
base: https://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git next
config: um-allmodconfig (https://download.01.org/0day-ci/archive/20220924/[email protected]/config)
compiler: gcc-11 (Debian 11.3.0-5) 11.3.0
reproduce (this is a W=1 build):
# https://github.com/intel-lab-lkp/linux/commit/f97a8ba561d7cf5a755c8f42421138e8b1073cf9
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Xiaochun-Lee/PCI-Set-no-io-resource-for-bridges-that-behind-VMD-controller/20220913-213745
git checkout f97a8ba561d7cf5a755c8f42421138e8b1073cf9
# save the config file
mkdir build_dir && cp config build_dir/.config
make W=1 O=build_dir ARCH=um SHELL=/bin/bash

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>

All errors (new ones prefixed by >>):

drivers/pci/quirks.c: In function 'quirk_vmd_no_iosize':
>> drivers/pci/quirks.c:5985:13: error: implicit declaration of function 'is_vmd' [-Werror=implicit-function-declaration]
5985 | if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
| ^~~~~~
cc1: some warnings being treated as errors


vim +/is_vmd +5985 drivers/pci/quirks.c

5959
5960 #if defined(CONFIG_X86_64) || defined(CONFIG_X86)
5961 /*
5962 * VMD-enabled root ports use Enhanced Configuration Access Mechanism (ECAM)
5963 * access PCI Express configuration space, and offer VMD_CFGBAR as
5964 * base of PCI Express configuration space for the bridges behind it.
5965 * The configuration space includes IO resources, but these IO
5966 * resources are not actually used on X86, especially the NVMes as
5967 * device connnected on this hot plug bridges, and it can result
5968 * in BAR#13 assign IO resource failed. So we clear IO resources
5969 * by setting an IO base value greater than limit to these bridges.
5970 * Hence, append kernel parameter "pci=hpiosize=0KB" can avoid
5971 * this BAR#13 failed messages show out.
5972 */
5973 static void quirk_vmd_no_iosize(struct pci_dev *bridge)
5974 {
5975 u8 io_base_lo, io_limit_lo;
5976 u16 io_low;
5977 u32 io_upper16;
5978 unsigned long io_mask, base, limit;
5979
5980 io_mask = PCI_IO_RANGE_MASK;
5981 if (bridge->io_window_1k)
5982 io_mask = PCI_IO_1K_RANGE_MASK;
5983
5984 /* VMD Domain */
> 5985 if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {

--
0-DAY CI Kernel Test Service
https://01.org/lkp

2022-09-25 10:22:42

by Xiaochun XC17 Li

[permalink] [raw]
Subject: RE: [External] Re: [PATCH v1] PCI: Set no io resource for bridges that behind VMD controller

Hi,

> -----Original Message-----
> From: kernel test robot <[email protected]>
> Sent: Friday, September 23, 2022 11:21 PM
> To: Xiaochun Lee <[email protected]>;
> [email protected]; [email protected]
> Cc: [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected]; linux-
> [email protected]; Xiaochun XC17 Li <[email protected]>
> Subject: [External] Re: [PATCH v1] PCI: Set no io resource for bridges that
> behind VMD controller
>
> Hi Xiaochun,
>
> Thank you for the patch! Yet something to improve:
>
> [auto build test ERROR on helgaas-pci/next] [also build test ERROR on
> linus/master v6.0-rc6 next-20220923] [If your patch is applied to the wrong
> git tree, kindly drop us a note.
Hi, thanks for your reply, this patch is only expected to be applied
on x86 or x64, actually function "is_vmd()" is just defined in
arch/x86/include/asm/pci.h, do we need to support it on arch um?

> And when submitting patch, we suggest to use '--base' as documented in
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit-
> scm.com%2Fdocs%2Fgit-format-
> patch%23_base_tree_information&amp;data=05%7C01%7Clixc17%40lenov
> o.com%7C63bd7990b33f4dd9b4bc08da9d77696c%7C5c7d0b28bdf8410caa9
> 34df372b16203%7C0%7C0%7C637995433448231363%7CUnknown%7CTWF
> pbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> VCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=TGZ2LnHsHCrJLW7WBBMZ
> lFxv8SIaQNmCGC1bEv2BEDE%3D&amp;reserved=0]
>
> url:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
> b.com%2Fintel-lab-lkp%2Flinux%2Fcommits%2FXiaochun-Lee%2FPCI-Set-
> no-io-resource-for-bridges-that-behind-VMD-controller%2F20220913-
> 213745&amp;data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b33f4d
> d9b4bc08da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0
> %7C637995433448231363%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&amp;sdata=3NULvkgYg0fWTopgYY5EYHsLIBPdTEcMxbz5Tj12D0E%
> 3D&amp;reserved=0
> base:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.k
> ernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fhelgaas%2Fpci.git&a
> mp;data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b33f4dd9b4bc08
> da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C63799
> 5433448231363%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&a
> mp;sdata=ZTZ7kKlwOk7IB3MN0rRwqHaNGGl4OiwrR7AjImBbTqc%3D&amp;
> reserved=0 next
> config: um-allmodconfig
> (https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdow
> nload.01.org%2F0day-
> ci%2Farchive%2F20220924%2F202209240714.ronvmf1X-
> lkp%40intel.com%2Fconfig&amp;data=05%7C01%7Clixc17%40lenovo.com%
> 7C63bd7990b33f4dd9b4bc08da9d77696c%7C5c7d0b28bdf8410caa934df372
> b16203%7C0%7C0%7C637995433448231363%7CUnknown%7CTWFpbGZsb3
> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> %3D%7C3000%7C%7C%7C&amp;sdata=6xMM7Fh9bKGnzBlmeu8kki3yvGGQ
> 2k%2FKCp%2BqvoLt4zU%3D&amp;reserved=0)
> compiler: gcc-11 (Debian 11.3.0-5) 11.3.0 reproduce (this is a W=1 build):
> #
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
> b.com%2Fintel-lab-
> lkp%2Flinux%2Fcommit%2Ff97a8ba561d7cf5a755c8f42421138e8b1073cf9&
> amp;data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b33f4dd9b4bc0
> 8da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C6379
> 95433448387557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi
> LCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&
> amp;sdata=4mjCA86AEbwN9eIiw1%2BLUPlf0%2BV2juqoDo7ojKSMweY%3D
> &amp;reserved=0
> git remote add linux-review
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
> b.com%2Fintel-lab-
> lkp%2Flinux&amp;data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b3
> 3f4dd9b4bc08da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%
> 7C0%7C637995433448387557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000
> %7C%7C%7C&amp;sdata=YusFni1tvPN89d2e%2B8yADL3jbwjuydLs0RD3NSjj
> 2MA%3D&amp;reserved=0
> git fetch --no-tags linux-review Xiaochun-Lee/PCI-Set-no-io-resource-
> for-bridges-that-behind-VMD-controller/20220913-213745
> git checkout f97a8ba561d7cf5a755c8f42421138e8b1073cf9
> # save the config file
> mkdir build_dir && cp config build_dir/.config
> make W=1 O=build_dir ARCH=um SHELL=/bin/bash
>
> If you fix the issue, kindly add following tag where applicable
> | Reported-by: kernel test robot <[email protected]>
>
> All errors (new ones prefixed by >>):
>
> drivers/pci/quirks.c: In function 'quirk_vmd_no_iosize':
> >> drivers/pci/quirks.c:5985:13: error: implicit declaration of function
> >> 'is_vmd' [-Werror=implicit-function-declaration]
> 5985 | if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
> | ^~~~~~
> cc1: some warnings being treated as errors
>
>
> vim +/is_vmd +5985 drivers/pci/quirks.c
>
> 5959
> 5960 #if defined(CONFIG_X86_64) || defined(CONFIG_X86)
> 5961 /*
> 5962 * VMD-enabled root ports use Enhanced Configuration Access
> Mechanism (ECAM)
> 5963 * access PCI Express configuration space, and offer VMD_CFGBAR
> as
> 5964 * base of PCI Express configuration space for the bridges behind it.
> 5965 * The configuration space includes IO resources, but these IO
> 5966 * resources are not actually used on X86, especially the NVMes as
> 5967 * device connnected on this hot plug bridges, and it can result
> 5968 * in BAR#13 assign IO resource failed. So we clear IO resources
> 5969 * by setting an IO base value greater than limit to these bridges.
> 5970 * Hence, append kernel parameter "pci=hpiosize=0KB" can avoid
> 5971 * this BAR#13 failed messages show out.
> 5972 */
> 5973 static void quirk_vmd_no_iosize(struct pci_dev *bridge)
> 5974 {
> 5975 u8 io_base_lo, io_limit_lo;
> 5976 u16 io_low;
> 5977 u32 io_upper16;
> 5978 unsigned long io_mask, base, limit;
> 5979
> 5980 io_mask = PCI_IO_RANGE_MASK;
> 5981 if (bridge->io_window_1k)
> 5982 io_mask = PCI_IO_1K_RANGE_MASK;
> 5983
> 5984 /* VMD Domain */
> > 5985 if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
>
> --
> 0-DAY CI Kernel Test Service
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2F01.or
> g%2Flkp&amp;data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b33f4
> dd9b4bc08da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0
> %7C637995433448387557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&amp;sdata=4qYtyE86mw9VyFbB3QKJqtldq6relV2Cunv9STAmmqA
> %3D&amp;reserved=0

2022-09-25 13:40:51

by Bjorn Helgaas

[permalink] [raw]
Subject: Re: [External] Re: [PATCH v1] PCI: Set no io resource for bridges that behind VMD controller

On Sun, Sep 25, 2022 at 09:52:21AM +0000, Xiaochun XC17 Li wrote:
> > -----Original Message-----
> > From: kernel test robot <[email protected]>
> > Sent: Friday, September 23, 2022 11:21 PM
> > To: Xiaochun Lee <[email protected]>;
> > [email protected]; [email protected]
> > Cc: [email protected]; [email protected]; [email protected];
> > [email protected]; [email protected]; [email protected]; linux-
> > [email protected]; Xiaochun XC17 Li <[email protected]>
> > Subject: [External] Re: [PATCH v1] PCI: Set no io resource for bridges that
> > behind VMD controller
> >
> > Hi Xiaochun,
> >
> > Thank you for the patch! Yet something to improve:
> >
> > [auto build test ERROR on helgaas-pci/next] [also build test ERROR on
> > linus/master v6.0-rc6 next-20220923] [If your patch is applied to the wrong
> > git tree, kindly drop us a note.
> Hi, thanks for your reply, this patch is only expected to be applied
> on x86 or x64, actually function "is_vmd()" is just defined in
> arch/x86/include/asm/pci.h, do we need to support it on arch um?

Yes, all possible Kconfig configurations must build cleanly. You
may have to add additional ifdef tests or an is_vmd() stub.

For future reference, your email reply doesn't follow the usual Linux
mailing list style, so it is unnecessarily hard to read. In
particular, it lacks the line that shows what you're responding to.
It would look something like this:

On Fri, Sep 23, 2022 at 11:21PM +0800, kernel test robot wrote:

Here's a sample:
https://lore.kernel.org/r/[email protected]

More background:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style