From: Xiaochun Lee <[email protected]>
When enable VMDs on Intel CPUs, VMD controllers(8086:28c0) be
recognized by VMD driver and there are many failed messages of
BAR 13 when scan the bridges and assign IO resource behind it
as listed below, the bridge wants to get 0x6000 as its IO
resource, but there is no IO resources on the host bridge.
VMD host bridge resources:
vmd 0000:64:00.5: PCI host bridge to bus 10000:80
pci_bus 10000:80: root bus resource [bus 80-9f]
pci_bus 10000:80: root bus resource [mem 0xe0000000-0xe1ffffff]
pci_bus 10000:80: root bus resource [mem 0x24ffff02010-0x24fffffffff 64bit]
Failed messages of BAR#13:
pci 10000:80:02.0: BAR 13: no space for [io size 0x1000]
pci 10000:80:02.0: BAR 13: failed to assign [io size 0x1000]
pci 10000:80:03.0: BAR 13: no space for [io size 0x1000]
pci 10000:80:03.0: BAR 13: failed to assign [io size 0x1000]
VMD-enabled root ports use
Enhanced Configuration Access Mechanism (ECAM) access
PCI Express configuration space, and offer VMD_CFGBAR as
base of PCI Express configuration space for the bridges
behind it. The configuration space includes IO resources,
but these IO resources are not actually used on X86,
especially the NVMes as device connected on this hot plug
bridges, and it can result in BAR#13 assign IO resource
failed. So we clear IO resources by setting an IO base value
greater than limit to these bridges. Hence, we can leverage
kernel parameter "pci=hpiosize=0KB" to avoid this failed
messages show out.
Signed-off-by: Xiaochun Lee <[email protected]>
---
drivers/pci/quirks.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 57 insertions(+)
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 4944798..f8a37f0 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -5956,3 +5956,60 @@ static void aspm_l1_acceptable_latency(struct pci_dev *dev)
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency);
#endif
+
+#if defined(CONFIG_X86_64) || defined(CONFIG_X86)
+/*
+ * VMD-enabled root ports use Enhanced Configuration Access Mechanism (ECAM)
+ * access PCI Express configuration space, and offer VMD_CFGBAR as
+ * base of PCI Express configuration space for the bridges behind it.
+ * The configuration space includes IO resources, but these IO
+ * resources are not actually used on X86, especially the NVMes as
+ * device connnected on this hot plug bridges, and it can result
+ * in BAR#13 assign IO resource failed. So we clear IO resources
+ * by setting an IO base value greater than limit to these bridges.
+ * Hence, append kernel parameter "pci=hpiosize=0KB" can avoid
+ * this BAR#13 failed messages show out.
+ */
+static void quirk_vmd_no_iosize(struct pci_dev *bridge)
+{
+ u8 io_base_lo, io_limit_lo;
+ u16 io_low;
+ u32 io_upper16;
+ unsigned long io_mask, base, limit;
+
+ io_mask = PCI_IO_RANGE_MASK;
+ if (bridge->io_window_1k)
+ io_mask = PCI_IO_1K_RANGE_MASK;
+
+ /* VMD Domain */
+ if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
+ pci_read_config_byte(bridge, PCI_IO_BASE, &io_base_lo);
+ pci_read_config_byte(bridge, PCI_IO_LIMIT, &io_limit_lo);
+ base = (io_base_lo & io_mask) << 8;
+ limit = (io_limit_lo & io_mask) << 8;
+ if (limit >= base) {
+ /* if there are defined io ports behind the bridge on x86,
+ * we clear it, since there is only 64KB IO resource on it,
+ * beyond that, hotplug io bridges don't needs IO port resource,
+ * such as NVMes attach on it. So the corresponding range must be
+ * turned off by writing base value greater than limit to the
+ * bridge's base/limit registers.
+ */
+
+ /* Clear upper 16 bits of I/O base/limit */
+ io_upper16 = 0;
+ /* set base value greater than limit */
+ io_low = 0x00f0;
+
+ /* Temporarily disable the I/O range before updating PCI_IO_BASE */
+ pci_write_config_dword(bridge, PCI_IO_BASE_UPPER16, 0x0000ffff);
+ /* Update lower 16 bits of I/O base/limit */
+ pci_write_config_word(bridge, PCI_IO_BASE, io_low);
+ /* Update upper 16 bits of I/O base/limit */
+ pci_write_config_dword(bridge, PCI_IO_BASE_UPPER16, io_upper16);
+ }
+ }
+}
+DECLARE_PCI_FIXUP_CLASS_HEADER(PCI_ANY_ID, PCI_ANY_ID,
+ PCI_CLASS_BRIDGE_PCI, 8, quirk_vmd_no_iosize);
+#endif
--
1.8.3.1
Hi Xiaochun,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on helgaas-pci/next]
[also build test ERROR on linus/master v6.0-rc6 next-20220923]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Xiaochun-Lee/PCI-Set-no-io-resource-for-bridges-that-behind-VMD-controller/20220913-213745
base: https://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git next
config: um-allmodconfig (https://download.01.org/0day-ci/archive/20220924/[email protected]/config)
compiler: gcc-11 (Debian 11.3.0-5) 11.3.0
reproduce (this is a W=1 build):
# https://github.com/intel-lab-lkp/linux/commit/f97a8ba561d7cf5a755c8f42421138e8b1073cf9
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Xiaochun-Lee/PCI-Set-no-io-resource-for-bridges-that-behind-VMD-controller/20220913-213745
git checkout f97a8ba561d7cf5a755c8f42421138e8b1073cf9
# save the config file
mkdir build_dir && cp config build_dir/.config
make W=1 O=build_dir ARCH=um SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
All errors (new ones prefixed by >>):
drivers/pci/quirks.c: In function 'quirk_vmd_no_iosize':
>> drivers/pci/quirks.c:5985:13: error: implicit declaration of function 'is_vmd' [-Werror=implicit-function-declaration]
5985 | if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
| ^~~~~~
cc1: some warnings being treated as errors
vim +/is_vmd +5985 drivers/pci/quirks.c
5959
5960 #if defined(CONFIG_X86_64) || defined(CONFIG_X86)
5961 /*
5962 * VMD-enabled root ports use Enhanced Configuration Access Mechanism (ECAM)
5963 * access PCI Express configuration space, and offer VMD_CFGBAR as
5964 * base of PCI Express configuration space for the bridges behind it.
5965 * The configuration space includes IO resources, but these IO
5966 * resources are not actually used on X86, especially the NVMes as
5967 * device connnected on this hot plug bridges, and it can result
5968 * in BAR#13 assign IO resource failed. So we clear IO resources
5969 * by setting an IO base value greater than limit to these bridges.
5970 * Hence, append kernel parameter "pci=hpiosize=0KB" can avoid
5971 * this BAR#13 failed messages show out.
5972 */
5973 static void quirk_vmd_no_iosize(struct pci_dev *bridge)
5974 {
5975 u8 io_base_lo, io_limit_lo;
5976 u16 io_low;
5977 u32 io_upper16;
5978 unsigned long io_mask, base, limit;
5979
5980 io_mask = PCI_IO_RANGE_MASK;
5981 if (bridge->io_window_1k)
5982 io_mask = PCI_IO_1K_RANGE_MASK;
5983
5984 /* VMD Domain */
> 5985 if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
--
0-DAY CI Kernel Test Service
https://01.org/lkp
Hi,
> -----Original Message-----
> From: kernel test robot <[email protected]>
> Sent: Friday, September 23, 2022 11:21 PM
> To: Xiaochun Lee <[email protected]>;
> [email protected]; [email protected]
> Cc: [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected]; linux-
> [email protected]; Xiaochun XC17 Li <[email protected]>
> Subject: [External] Re: [PATCH v1] PCI: Set no io resource for bridges that
> behind VMD controller
>
> Hi Xiaochun,
>
> Thank you for the patch! Yet something to improve:
>
> [auto build test ERROR on helgaas-pci/next] [also build test ERROR on
> linus/master v6.0-rc6 next-20220923] [If your patch is applied to the wrong
> git tree, kindly drop us a note.
Hi, thanks for your reply, this patch is only expected to be applied
on x86 or x64, actually function "is_vmd()" is just defined in
arch/x86/include/asm/pci.h, do we need to support it on arch um?
> And when submitting patch, we suggest to use '--base' as documented in
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit-
> scm.com%2Fdocs%2Fgit-format-
> patch%23_base_tree_information&data=05%7C01%7Clixc17%40lenov
> o.com%7C63bd7990b33f4dd9b4bc08da9d77696c%7C5c7d0b28bdf8410caa9
> 34df372b16203%7C0%7C0%7C637995433448231363%7CUnknown%7CTWF
> pbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> VCI6Mn0%3D%7C3000%7C%7C%7C&sdata=TGZ2LnHsHCrJLW7WBBMZ
> lFxv8SIaQNmCGC1bEv2BEDE%3D&reserved=0]
>
> url:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
> b.com%2Fintel-lab-lkp%2Flinux%2Fcommits%2FXiaochun-Lee%2FPCI-Set-
> no-io-resource-for-bridges-that-behind-VMD-controller%2F20220913-
> 213745&data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b33f4d
> d9b4bc08da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0
> %7C637995433448231363%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&sdata=3NULvkgYg0fWTopgYY5EYHsLIBPdTEcMxbz5Tj12D0E%
> 3D&reserved=0
> base:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.k
> ernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fhelgaas%2Fpci.git&a
> mp;data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b33f4dd9b4bc08
> da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C63799
> 5433448231363%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&a
> mp;sdata=ZTZ7kKlwOk7IB3MN0rRwqHaNGGl4OiwrR7AjImBbTqc%3D&
> reserved=0 next
> config: um-allmodconfig
> (https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdow
> nload.01.org%2F0day-
> ci%2Farchive%2F20220924%2F202209240714.ronvmf1X-
> lkp%40intel.com%2Fconfig&data=05%7C01%7Clixc17%40lenovo.com%
> 7C63bd7990b33f4dd9b4bc08da9d77696c%7C5c7d0b28bdf8410caa934df372
> b16203%7C0%7C0%7C637995433448231363%7CUnknown%7CTWFpbGZsb3
> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> %3D%7C3000%7C%7C%7C&sdata=6xMM7Fh9bKGnzBlmeu8kki3yvGGQ
> 2k%2FKCp%2BqvoLt4zU%3D&reserved=0)
> compiler: gcc-11 (Debian 11.3.0-5) 11.3.0 reproduce (this is a W=1 build):
> #
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
> b.com%2Fintel-lab-
> lkp%2Flinux%2Fcommit%2Ff97a8ba561d7cf5a755c8f42421138e8b1073cf9&
> amp;data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b33f4dd9b4bc0
> 8da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C6379
> 95433448387557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi
> LCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&
> amp;sdata=4mjCA86AEbwN9eIiw1%2BLUPlf0%2BV2juqoDo7ojKSMweY%3D
> &reserved=0
> git remote add linux-review
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
> b.com%2Fintel-lab-
> lkp%2Flinux&data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b3
> 3f4dd9b4bc08da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%
> 7C0%7C637995433448387557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000
> %7C%7C%7C&sdata=YusFni1tvPN89d2e%2B8yADL3jbwjuydLs0RD3NSjj
> 2MA%3D&reserved=0
> git fetch --no-tags linux-review Xiaochun-Lee/PCI-Set-no-io-resource-
> for-bridges-that-behind-VMD-controller/20220913-213745
> git checkout f97a8ba561d7cf5a755c8f42421138e8b1073cf9
> # save the config file
> mkdir build_dir && cp config build_dir/.config
> make W=1 O=build_dir ARCH=um SHELL=/bin/bash
>
> If you fix the issue, kindly add following tag where applicable
> | Reported-by: kernel test robot <[email protected]>
>
> All errors (new ones prefixed by >>):
>
> drivers/pci/quirks.c: In function 'quirk_vmd_no_iosize':
> >> drivers/pci/quirks.c:5985:13: error: implicit declaration of function
> >> 'is_vmd' [-Werror=implicit-function-declaration]
> 5985 | if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
> | ^~~~~~
> cc1: some warnings being treated as errors
>
>
> vim +/is_vmd +5985 drivers/pci/quirks.c
>
> 5959
> 5960 #if defined(CONFIG_X86_64) || defined(CONFIG_X86)
> 5961 /*
> 5962 * VMD-enabled root ports use Enhanced Configuration Access
> Mechanism (ECAM)
> 5963 * access PCI Express configuration space, and offer VMD_CFGBAR
> as
> 5964 * base of PCI Express configuration space for the bridges behind it.
> 5965 * The configuration space includes IO resources, but these IO
> 5966 * resources are not actually used on X86, especially the NVMes as
> 5967 * device connnected on this hot plug bridges, and it can result
> 5968 * in BAR#13 assign IO resource failed. So we clear IO resources
> 5969 * by setting an IO base value greater than limit to these bridges.
> 5970 * Hence, append kernel parameter "pci=hpiosize=0KB" can avoid
> 5971 * this BAR#13 failed messages show out.
> 5972 */
> 5973 static void quirk_vmd_no_iosize(struct pci_dev *bridge)
> 5974 {
> 5975 u8 io_base_lo, io_limit_lo;
> 5976 u16 io_low;
> 5977 u32 io_upper16;
> 5978 unsigned long io_mask, base, limit;
> 5979
> 5980 io_mask = PCI_IO_RANGE_MASK;
> 5981 if (bridge->io_window_1k)
> 5982 io_mask = PCI_IO_1K_RANGE_MASK;
> 5983
> 5984 /* VMD Domain */
> > 5985 if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
>
> --
> 0-DAY CI Kernel Test Service
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2F01.or
> g%2Flkp&data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b33f4
> dd9b4bc08da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0
> %7C637995433448387557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&sdata=4qYtyE86mw9VyFbB3QKJqtldq6relV2Cunv9STAmmqA
> %3D&reserved=0
On Sun, Sep 25, 2022 at 09:52:21AM +0000, Xiaochun XC17 Li wrote:
> > -----Original Message-----
> > From: kernel test robot <[email protected]>
> > Sent: Friday, September 23, 2022 11:21 PM
> > To: Xiaochun Lee <[email protected]>;
> > [email protected]; [email protected]
> > Cc: [email protected]; [email protected]; [email protected];
> > [email protected]; [email protected]; [email protected]; linux-
> > [email protected]; Xiaochun XC17 Li <[email protected]>
> > Subject: [External] Re: [PATCH v1] PCI: Set no io resource for bridges that
> > behind VMD controller
> >
> > Hi Xiaochun,
> >
> > Thank you for the patch! Yet something to improve:
> >
> > [auto build test ERROR on helgaas-pci/next] [also build test ERROR on
> > linus/master v6.0-rc6 next-20220923] [If your patch is applied to the wrong
> > git tree, kindly drop us a note.
> Hi, thanks for your reply, this patch is only expected to be applied
> on x86 or x64, actually function "is_vmd()" is just defined in
> arch/x86/include/asm/pci.h, do we need to support it on arch um?
Yes, all possible Kconfig configurations must build cleanly. You
may have to add additional ifdef tests or an is_vmd() stub.
For future reference, your email reply doesn't follow the usual Linux
mailing list style, so it is unnecessarily hard to read. In
particular, it lacks the line that shows what you're responding to.
It would look something like this:
On Fri, Sep 23, 2022 at 11:21PM +0800, kernel test robot wrote:
Here's a sample:
https://lore.kernel.org/r/[email protected]
More background:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
On Sun, Sep 25, 2022 at 8:57 PM +0800, Bjorn Helgaas <[email protected]> wrote:
> On Sun, Sep 25, 2022 at 09:52:21AM +0000, Xiaochun XC17 Li wrote:
> > > -----Original Message-----
> > > From: kernel test robot <[email protected]>
> > > Sent: Friday, September 23, 2022 11:21 PM
> > > To: Xiaochun Lee <[email protected]>;
> > > [email protected]; [email protected]
> > > Cc: [email protected]; [email protected]; [email protected];
> > > [email protected]; [email protected]; [email protected];
> linux-
> > > [email protected]; Xiaochun XC17 Li <[email protected]>
> > > Subject: [External] Re: [PATCH v1] PCI: Set no io resource for
> > > bridges that behind VMD controller
> > >
> > > Hi Xiaochun,
> > >
> > > Thank you for the patch! Yet something to improve:
> > >
> > > [auto build test ERROR on helgaas-pci/next] [also build test ERROR
> > > on linus/master v6.0-rc6 next-20220923] [If your patch is applied to
> > > the wrong git tree, kindly drop us a note.
> > Hi, thanks for your reply, this patch is only expected to be applied
> > on x86 or x64, actually function "is_vmd()" is just defined in
> > arch/x86/include/asm/pci.h, do we need to support it on arch um?
>
> Yes, all possible Kconfig configurations must build cleanly. You may have
> to add additional ifdef tests or an is_vmd() stub.
Thanks, I'll fix it and complier it base arch um, the patch v2 will be sent out.
>
> For future reference, your email reply doesn't follow the usual Linux
> mailing list style, so it is unnecessarily hard to read. In particular, it lacks
> the line that shows what you're responding to.
> It would look something like this:
>
> On Fri, Sep 23, 2022 at 11:21PM +0800, kernel test robot wrote:
OK, thanks!
I did a test just now, it seems to meet the format requirements.
I sent emails via outlook, and I've tried to set the reply format,
unfortunately, the time zone and date time are not resolved correctly.
>
> Here's a sample:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Fr%2FYyGghUdcrOdrR0ep%40smile.fi.intel.com&data=05
> %7C01%7Clixc17%40lenovo.com%7C27c59eea4afb46c8e24608da9ef5752d%
> 7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C6379970750209852
> 91%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM
> zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=Y8
> O9DzJ%2FW8JlAz0LM8o6LgnQcWUbYlxseM1wWbLvUKI%3D&reserved
> =0
>
> More background:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.
> wikipedia.org%2Fwiki%2FPosting_style%23Interleaved_style&data=05
> %7C01%7Clixc17%40lenovo.com%7C27c59eea4afb46c8e24608da9ef5752d%
> 7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C6379970750209852
> 91%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM
> zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=3Q
> oamhkXi78AOmFMuCRLnL52Ed8yEZ79Vx72pEpE4cs%3D&reserved=0
On Tue, Sep 27, 2022 at 02:40:39AM +0000, Xiaochun XC17 Li wrote:
> On Sun, Sep 25, 2022 at 8:57 PM +0800, Bjorn Helgaas <[email protected]> wrote:
> > On Sun, Sep 25, 2022 at 09:52:21AM +0000, Xiaochun XC17 Li wrote:
> > > > -----Original Message-----
> > > > From: kernel test robot <[email protected]>
> > > > Sent: Friday, September 23, 2022 11:21 PM
> > > > To: Xiaochun Lee <[email protected]>;
> > > > [email protected]; [email protected]
> > > > Cc: [email protected]; [email protected]; [email protected];
> > > > [email protected]; [email protected]; [email protected];
> > linux-
> > > > [email protected]; Xiaochun XC17 Li <[email protected]>
> > > > Subject: [External] Re: [PATCH v1] PCI: Set no io resource for
> > > > bridges that behind VMD controller
> > ...
> > For future reference, your email reply doesn't follow the usual Linux
> > mailing list style, so it is unnecessarily hard to read. In particular, it lacks
> > the line that shows what you're responding to.
> > It would look something like this:
> >
> > On Fri, Sep 23, 2022 at 11:21PM +0800, kernel test robot wrote:
>
> OK, thanks!
> I did a test just now, it seems to meet the format requirements.
> I sent emails via outlook, and I've tried to set the reply format,
> unfortunately, the time zone and date time are not resolved correctly.
Much better, thanks! It's much easier to read without all the
redundant headers.
Bjorn
On Tue, Sep 13, 2022 at 09:24:45PM +0800, Xiaochun Lee wrote:
> From: Xiaochun Lee <[email protected]>
>
> When enable VMDs on Intel CPUs, VMD controllers(8086:28c0) be
> recognized by VMD driver and there are many failed messages of
> BAR 13 when scan the bridges and assign IO resource behind it
> as listed below, the bridge wants to get 0x6000 as its IO
> resource, but there is no IO resources on the host bridge.
>
> VMD host bridge resources:
> vmd 0000:64:00.5: PCI host bridge to bus 10000:80
> pci_bus 10000:80: root bus resource [bus 80-9f]
> pci_bus 10000:80: root bus resource [mem 0xe0000000-0xe1ffffff]
> pci_bus 10000:80: root bus resource [mem 0x24ffff02010-0x24fffffffff 64bit]
>
> Failed messages of BAR#13:
> pci 10000:80:02.0: BAR 13: no space for [io size 0x1000]
> pci 10000:80:02.0: BAR 13: failed to assign [io size 0x1000]
> pci 10000:80:03.0: BAR 13: no space for [io size 0x1000]
> pci 10000:80:03.0: BAR 13: failed to assign [io size 0x1000]
>
> VMD-enabled root ports use
> Enhanced Configuration Access Mechanism (ECAM) access
> PCI Express configuration space, and offer VMD_CFGBAR as
> base of PCI Express configuration space for the bridges
> behind it. The configuration space includes IO resources,
> but these IO resources are not actually used on X86,
> especially the NVMes as device connected on this hot plug
> bridges, and it can result in BAR#13 assign IO resource
> failed. So we clear IO resources by setting an IO base value
> greater than limit to these bridges. Hence, we can leverage
> kernel parameter "pci=hpiosize=0KB" to avoid this failed
> messages show out.
>
> Signed-off-by: Xiaochun Lee <[email protected]>
Some of the discussion here got lost because of email issues. Lore
has some:
https://lore.kernel.org/all/[email protected]/T/#u,
and patchwork has a v2 with a little more discussion:
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
But the v2 patch doesn't seem to have made it to the mailing lists or
to lore (https://lore.kernel.org/all/?q=f%3Alixc17) and I don't apply
things until they appear on the mailing list.
I *would* like to get rid of those "no space" and "failed to assign"
messages. This is an issue for platforms other than VMD, too. Just
an FYI that you need to follow up on this if we want make progress.
Bjorn
On Sat, Nov 11, 2022 at 07:54:25, "Bjorn Helgaas" <[email protected]> wrote:
> On Tue, Sep 13, 2022 at 09:24:45PM +0800, Xiaochun Lee wrote:
> > From: Xiaochun Lee <[email protected]>
> >
> > When enable VMDs on Intel CPUs, VMD controllers(8086:28c0) be
> > recognized by VMD driver and there are many failed messages of BAR 13
> > when scan the bridges and assign IO resource behind it as listed
> > below, the bridge wants to get 0x6000 as its IO resource, but there is
> > no IO resources on the host bridge.
> >
> > VMD host bridge resources:
> > vmd 0000:64:00.5: PCI host bridge to bus 10000:80 pci_bus 10000:80:
> > root bus resource [bus 80-9f] pci_bus 10000:80: root bus resource [mem
> > 0xe0000000-0xe1ffffff] pci_bus 10000:80: root bus resource [mem
> > 0x24ffff02010-0x24fffffffff 64bit]
> >
> > Failed messages of BAR#13:
> > pci 10000:80:02.0: BAR 13: no space for [io size 0x1000] pci
> > 10000:80:02.0: BAR 13: failed to assign [io size 0x1000] pci
> > 10000:80:03.0: BAR 13: no space for [io size 0x1000] pci
> > 10000:80:03.0: BAR 13: failed to assign [io size 0x1000]
> >
> > VMD-enabled root ports use
> > Enhanced Configuration Access Mechanism (ECAM) access PCI Express
> > configuration space, and offer VMD_CFGBAR as base of PCI Express
> > configuration space for the bridges behind it. The configuration space
> > includes IO resources, but these IO resources are not actually used on
> > X86, especially the NVMes as device connected on this hot plug
> > bridges, and it can result in BAR#13 assign IO resource failed. So we
> > clear IO resources by setting an IO base value greater than limit to
> > these bridges. Hence, we can leverage kernel parameter
> > "pci=hpiosize=0KB" to avoid this failed messages show out.
> >
> > Signed-off-by: Xiaochun Lee <[email protected]>
>
> Some of the discussion here got lost because of email issues. Lore has
> some:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Fall%2F1663075485-20591-1-git-send-email-
> lixiaochun.2888%40163.com%2FT%2F%23u&data=05%7C01%7Clixc17
> %40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bdf
> 8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknown
> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
> WwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JzWYkIsaEfINofiqo
> XyjEh43VjXO3HZw2JLSsmhpUiQ%3D&reserved=0,
> and patchwork has a v2 with a little more discussion:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc
> hwork.kernel.org%2Fproject%2Flinux-pci%2Fpatch%2F1664288166-7432-1-
> git-send-email-
> lixiaochun.2888%40163.com%2F&data=05%7C01%7Clixc17%40lenovo.
> com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bdf8410caa934
> df372b16203%7C0%7C0%7C638038076734438158%7CUnknown%7CTWFpb
> GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> 6Mn0%3D%7C3000%7C%7C%7C&sdata=y%2BtIsepTpyLeoHW6CrgkZz2
> tiMjY0TONfK7zNCKXQ90%3D&reserved=0
>
> But the v2 patch doesn't seem to have made it to the mailing lists or to
> lore
> (https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
> e.kernel.org%2Fall%2F%3Fq%3Df%253Alixc17&data=05%7C01%7Clixc1
> 7%40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bd
> f8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknow
> n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h
> aWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zxGz1hZOD2tvQP
> EsbxQTzjHwQvXvqeO%2FUd6I9S%2Fj314%3D&reserved=0) and I don't
> apply things until they appear on the mailing list.
>
> I *would* like to get rid of those "no space" and "failed to assign"
> messages. This is an issue for platforms other than VMD, too. Just an FYI
> that you need to follow up on this if we want make progress.
>
> Bjorn
Thanks for your comments, so do you mean we'd better come up with a
solution to avoid messages like "no space" and "failed to assign" using
a common way, for both VMD and other platforms?
Xiaochun
On Wed, Nov 16, 2022 at 03:34:34AM +0000, Xiaochun XC17 Li wrote:
> On Sat, Nov 11, 2022 at 07:54:25, "Bjorn Helgaas" <[email protected]> wrote:
> > On Tue, Sep 13, 2022 at 09:24:45PM +0800, Xiaochun Lee wrote:
> > > From: Xiaochun Lee <[email protected]>
> > >
> > > When enable VMDs on Intel CPUs, VMD controllers(8086:28c0) be
> > > recognized by VMD driver and there are many failed messages of BAR 13
> > > when scan the bridges and assign IO resource behind it as listed
> > > below, the bridge wants to get 0x6000 as its IO resource, but there is
> > > no IO resources on the host bridge.
> > >
> > > VMD host bridge resources:
> > > vmd 0000:64:00.5: PCI host bridge to bus 10000:80 pci_bus 10000:80:
> > > root bus resource [bus 80-9f] pci_bus 10000:80: root bus resource [mem
> > > 0xe0000000-0xe1ffffff] pci_bus 10000:80: root bus resource [mem
> > > 0x24ffff02010-0x24fffffffff 64bit]
> > >
> > > Failed messages of BAR#13:
> > > pci 10000:80:02.0: BAR 13: no space for [io size 0x1000] pci
> > > 10000:80:02.0: BAR 13: failed to assign [io size 0x1000] pci
> > > 10000:80:03.0: BAR 13: no space for [io size 0x1000] pci
> > > 10000:80:03.0: BAR 13: failed to assign [io size 0x1000]
> > >
> > > VMD-enabled root ports use
> > > Enhanced Configuration Access Mechanism (ECAM) access PCI Express
> > > configuration space, and offer VMD_CFGBAR as base of PCI Express
> > > configuration space for the bridges behind it. The configuration space
> > > includes IO resources, but these IO resources are not actually used on
> > > X86, especially the NVMes as device connected on this hot plug
> > > bridges, and it can result in BAR#13 assign IO resource failed. So we
> > > clear IO resources by setting an IO base value greater than limit to
> > > these bridges. Hence, we can leverage kernel parameter
> > > "pci=hpiosize=0KB" to avoid this failed messages show out.
> > >
> > > Signed-off-by: Xiaochun Lee <[email protected]>
> >
> > Some of the discussion here got lost because of email issues. Lore has
> > some:
> > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> > kernel.org%2Fall%2F1663075485-20591-1-git-send-email-
> > lixiaochun.2888%40163.com%2FT%2F%23u&data=05%7C01%7Clixc17
> > %40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bdf
> > 8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknown
> > %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
> > WwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JzWYkIsaEfINofiqo
> > XyjEh43VjXO3HZw2JLSsmhpUiQ%3D&reserved=0,
> > and patchwork has a v2 with a little more discussion:
> > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc
> > hwork.kernel.org%2Fproject%2Flinux-pci%2Fpatch%2F1664288166-7432-1-
> > git-send-email-
> > lixiaochun.2888%40163.com%2F&data=05%7C01%7Clixc17%40lenovo.
> > com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bdf8410caa934
> > df372b16203%7C0%7C0%7C638038076734438158%7CUnknown%7CTWFpb
> > GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> > 6Mn0%3D%7C3000%7C%7C%7C&sdata=y%2BtIsepTpyLeoHW6CrgkZz2
> > tiMjY0TONfK7zNCKXQ90%3D&reserved=0
> >
> > But the v2 patch doesn't seem to have made it to the mailing lists or to
> > lore
> > (https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
> > e.kernel.org%2Fall%2F%3Fq%3Df%253Alixc17&data=05%7C01%7Clixc1
> > 7%40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bd
> > f8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknow
> > n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h
> > aWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zxGz1hZOD2tvQP
> > EsbxQTzjHwQvXvqeO%2FUd6I9S%2Fj314%3D&reserved=0) and I don't
> > apply things until they appear on the mailing list.
> >
> > I *would* like to get rid of those "no space" and "failed to assign"
> > messages. This is an issue for platforms other than VMD, too. Just an FYI
> > that you need to follow up on this if we want make progress.
>
> Thanks for your comments, so do you mean we'd better come up with a
> solution to avoid messages like "no space" and "failed to assign" using
> a common way, for both VMD and other platforms?
I tried to say two separate things:
1) It doesn't seem like a VMD-specific thing, so it would be ideal if
the solution were generic instead of being VMD-specific.
2) Some of your previous patches didn't make it to the mailing list,
so we couldn't really do anything with them.
Bjorn
On Wed, Nov 16, 2022 at 12:10 PM +0800, "Bjorn Helgaas" <[email protected]> wrote:
> On Wed, Nov 16, 2022 at 03:34:34AM +0000, Xiaochun XC17 Li wrote:
> > On Sat, Nov 11, 2022 at 07:54:25, "Bjorn Helgaas" <[email protected]>
> wrote:
> > > On Tue, Sep 13, 2022 at 09:24:45PM +0800, Xiaochun Lee wrote:
> > > > From: Xiaochun Lee <[email protected]>
> > > >
> > > > When enable VMDs on Intel CPUs, VMD controllers(8086:28c0) be
> > > > recognized by VMD driver and there are many failed messages of BAR
> > > > 13 when scan the bridges and assign IO resource behind it as
> > > > listed below, the bridge wants to get 0x6000 as its IO resource,
> > > > but there is no IO resources on the host bridge.
> > > >
> > > > VMD host bridge resources:
> > > > vmd 0000:64:00.5: PCI host bridge to bus 10000:80 pci_bus 10000:80:
> > > > root bus resource [bus 80-9f] pci_bus 10000:80: root bus resource
> > > > [mem 0xe0000000-0xe1ffffff] pci_bus 10000:80: root bus resource
> > > > [mem 0x24ffff02010-0x24fffffffff 64bit]
> > > >
> > > > Failed messages of BAR#13:
> > > > pci 10000:80:02.0: BAR 13: no space for [io size 0x1000] pci
> > > > 10000:80:02.0: BAR 13: failed to assign [io size 0x1000] pci
> > > > 10000:80:03.0: BAR 13: no space for [io size 0x1000] pci
> > > > 10000:80:03.0: BAR 13: failed to assign [io size 0x1000]
> > > >
> > > > VMD-enabled root ports use
> > > > Enhanced Configuration Access Mechanism (ECAM) access PCI
> Express
> > > > configuration space, and offer VMD_CFGBAR as base of PCI Express
> > > > configuration space for the bridges behind it. The configuration
> > > > space includes IO resources, but these IO resources are not
> > > > actually used on X86, especially the NVMes as device connected on
> > > > this hot plug bridges, and it can result in BAR#13 assign IO
> > > > resource failed. So we clear IO resources by setting an IO base
> > > > value greater than limit to these bridges. Hence, we can leverage
> > > > kernel parameter "pci=hpiosize=0KB" to avoid this failed messages
> show out.
> > > >
> > > > Signed-off-by: Xiaochun Lee <[email protected]>
> > >
> > > Some of the discussion here got lost because of email issues. Lore
> > > has
> > > some:
> > >
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> > > kernel.org%2Fall%2F1663075485-20591-1-git-send-email-
> > >
> lixiaochun.2888%40163.com%2FT%2F%23u&data=05%7C01%7Clixc17
> > > %40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28
> bdf
> > >
> 8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknown
> > > %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
> ha
> > >
> WwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JzWYkIsaEfINofiqo
> > > XyjEh43VjXO3HZw2JLSsmhpUiQ%3D&reserved=0,
> > > and patchwork has a v2 with a little more discussion:
> > >
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpa
> > > tc
> > > hwork.kernel.org%2Fproject%2Flinux-pci%2Fpatch%2F1664288166-
> 7432-1-
> > > git-send-email-
> > >
> lixiaochun.2888%40163.com%2F&data=05%7C01%7Clixc17%40lenovo.
> > >
> com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bdf8410caa934
> > >
> df372b16203%7C0%7C0%7C638038076734438158%7CUnknown%7CTWFpb
> > >
> GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> > >
> 6Mn0%3D%7C3000%7C%7C%7C&sdata=y%2BtIsepTpyLeoHW6CrgkZz2
> > > tiMjY0TONfK7zNCKXQ90%3D&reserved=0
> > >
> > > But the v2 patch doesn't seem to have made it to the mailing lists
> > > or to lore
> > >
> (https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
> > > or
> > >
> e.kernel.org%2Fall%2F%3Fq%3Df%253Alixc17&data=05%7C01%7Clixc1
> > >
> 7%40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bd
> > >
> f8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknow
> > >
> n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h
> > >
> aWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zxGz1hZOD2tvQP
> > > EsbxQTzjHwQvXvqeO%2FUd6I9S%2Fj314%3D&reserved=0) and I
> don't
> > > apply things until they appear on the mailing list.
> > >
> > > I *would* like to get rid of those "no space" and "failed to assign"
> > > messages. This is an issue for platforms other than VMD, too. Just
> > > an FYI that you need to follow up on this if we want make progress.
> >
> > Thanks for your comments, so do you mean we'd better come up with a
> > solution to avoid messages like "no space" and "failed to assign"
> > using a common way, for both VMD and other platforms?
>
> I tried to say two separate things:
>
> 1) It doesn't seem like a VMD-specific thing, so it would be ideal if the
> solution were generic instead of being VMD-specific.
OK, I'll try to find out a generic approach.
>
> 2) Some of your previous patches didn't make it to the mailing list, so we
> couldn't really do anything with them.
>
> Bjorn
Actually, the V2 patch didn't following the mailing of V1 patch,
I sent it out separately like the first time for V1 patch, I don't know if there
is a mistake in the way I sent the V2 patch. I was wondering if I was going to
send out V2 patches based on the mailing list of V1 patch, if it is yes, I must
study how to summit the V2 patch, so would you please correct me?
Xiaochun
On Wed, Nov 16, 2022 at 04:26:02AM +0000, Xiaochun XC17 Li wrote:
> On Wed, Nov 16, 2022 at 12:10 PM +0800, "Bjorn Helgaas" <[email protected]> wrote:
> > 2) Some of your previous patches didn't make it to the mailing list, so we
> > couldn't really do anything with them.
> >
> > Bjorn
>
> Actually, the V2 patch didn't following the mailing of V1 patch, I
> sent it out separately like the first time for V1 patch, I don't
> know if there is a mistake in the way I sent the V2 patch. I was
> wondering if I was going to send out V2 patches based on the mailing
> list of V1 patch, if it is yes, I must study how to summit the V2
> patch, so would you please correct me?
I was mistaken, the V2 patch *was* on the mailing list:
https://lore.kernel.org/r/[email protected]
I guess I didn't find it because I was searching for the email address
"[email protected]", and v2 was posted by "[email protected]".
Sorry, my mistake! It looks like the sending of v2 was just fine.
Bjorn