Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 24 Apr 2023 20:50:14 +0200
From:   Igor Mammedov <imammedo@redhat.com>
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     "Rafael J. Wysocki" <rafael@kernel.org>,
        linux-kernel@vger.kernel.org, mst@redhat.com, lenb@kernel.org,
        bhelgaas@google.com, linux-acpi@vger.kernel.org,
        linux-pci@vger.kernel.org,
        Mika Westerberg <mika.westerberg@linux.intel.com>
Subject: Re: [PATCH] pci: acpiphp: try to reassign resources on bridge if
 necessary
Message-ID: <20230424205014.2c213437@imammedo.users.ipa.redhat.com>
In-Reply-To: <20230418163114.GA134491@bhelgaas>
References: <CAJZ5v0gWKwOiACmK9=ru5W15Kydv6JqKJ8d4ngzKC7jqAjjcpQ@mail.gmail.com>
        <20230418163114.GA134491@bhelgaas>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Tue, 18 Apr 2023 11:31:14 -0500
Bjorn Helgaas <helgaas@kernel.org> wrote:

> [+cc Mika, who made previous changes in this area]
>=20
> On Tue, Apr 18, 2023 at 05:38:15PM +0200, Rafael J. Wysocki wrote:
> > On Tue, Apr 18, 2023 at 4:17=E2=80=AFPM Igor Mammedov <imammedo@redhat.=
com> wrote: =20
> > > On Tue, 18 Apr 2023 14:55:29 +0200
> > > "Rafael J. Wysocki" <rafael@kernel.org> wrote: =20
> > > > On Tue, Apr 18, 2023 at 10:50=E2=80=AFAM Igor Mammedov <imammedo@re=
dhat.com> wrote: =20
> > > > >
> > > > > When using ACPI PCI hotplug, hotplugging a device with
> > > > > large BARs may fail if bridge windows programmed by
> > > > > firmware are not large enough.
> > > > >
> > > > > Reproducer:
> > > > >   $ qemu-kvm -monitor stdio -M q35  -m 4G \
> > > > >       -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=3Don \
> > > > >       -device id=3Drp1,pcie-root-port,bus=3Dpcie.0,chassis=3D4 \
> > > > >       disk_image
> > > > >
> > > > >  wait till linux guest boots, then hotplug device
> > > > >    (qemu) device_add qxl,bus=3Drp1
> > > > >
> > > > >  hotplug on guest side fails with:
> > > > >    pci 0000:01:00.0: [1b36:0100] type 00 class 0x038000
> > > > >    pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x03ffffff]
> > > > >    pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
> > > > >    pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00001fff]
> > > > >    pci 0000:01:00.0: reg 0x1c: [io  0x0000-0x001f]
> > > > >    pci 0000:01:00.0: BAR 0: no space for [mem size 0x04000000]
> > > > >    pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x04000000]
> > > > >    pci 0000:01:00.0: BAR 1: no space for [mem size 0x04000000]
> > > > >    pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x04000000]
> > > > >    pci 0000:01:00.0: BAR 2: assigned [mem 0xfe800000-0xfe801fff]
> > > > >    pci 0000:01:00.0: BAR 3: assigned [io  0x1000-0x101f]
> > > > >    qxl 0000:01:00.0: enabling device (0000 -> 0003)
> > > > >    Unable to create vram_mapping
> > > > >    qxl: probe of 0000:01:00.0 failed with error -12
> > > > >
> > > > > However when using native PCIe hotplug
> > > > >   '-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=3Doff'
> > > > > it works fine, since kernel attempts to reassign unused resources.
> > > > > Use the same machinery as native PCIe hotplug to (re)assign resou=
rces. =20
>=20
> Thanks for the nice reproducer and logs!
>=20
> > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com> =20
> > > >
> > > > Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> > > >
> > > > or please let me know if you want me to pick this up. =20
> > >
> > > It would be nice if you could pick it up. =20
> >=20
> > OK, I'll do that unless Bjorn tells me that he prefers to take it via
> > the PCI tree. =20
>=20
> It's OK with me if you pick this up, but please update the subject to
> use the style of previous commits, e.g.,
>=20
>   PCI: acpiphp: Reassign resources on bridge if necessary
>=20
> Previous changes involving pci_assign_unassigned_bridge_resources() in
> enable_slot() (these are from Mika, so I cc'd him in case he wants to
> comment):
>=20
>   84c8b58ed3ad ("ACPI / hotplug / PCI: Don't scan bridges managed by nati=
ve hotplug")
>   77adf9355304 ("ACPI / hotplug / PCI: Allocate resources directly under =
the non-hotplug bridge")
>=20
> > > > > ---
> > > > > tested in QEMU with Q35 machine on PCIE root port and also
> > > > > with nested conventional bridge attached to root port.
> > > > > ---
> > > > >  drivers/pci/hotplug/acpiphp_glue.c | 2 +-
> > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/drivers/pci/hotplug/acpiphp_glue.c b/drivers/pci/hot=
plug/acpiphp_glue.c
> > > > > index 5b1f271c6034..9aebde28a92f 100644
> > > > > --- a/drivers/pci/hotplug/acpiphp_glue.c
> > > > > +++ b/drivers/pci/hotplug/acpiphp_glue.c
> > > > > @@ -517,7 +517,7 @@ static void enable_slot(struct acpiphp_slot *=
slot, bool bridge) =20
>=20
> Previous context:
>=20
>                                              __pci_bus_size_bridges(dev->=
subordinate,
>                                                                     &add_=
list);
>=20
> > > > >                                 }
> > > > >                         }
> > > > >                 }
> > > > > -               __pci_bus_assign_resources(bus, &add_list, NULL);
> > > > > +               pci_assign_unassigned_bridge_resources(bus->self)=
; =20
>=20
> "add_list" is now used only for __pci_bus_size_bridges(), which
> *looks* unnecessary unless there's some obscure side-effect of that
> path when that parameter is non-NULL.
>=20
> If "add_list" is unnecessary, you would probably use
> pci_bus_size_bridges() above instead of __pci_bus_size_bridges().

pci_assign_unassigned_bridge_resources() calls __pci_bus_size_bridges()
so original one is not needed anymore, in addition it might leak entries
added to add_list.
I'll remove __pci_bus_size_bridges() and respin patch (incl. updated subjec=
t)

=20
> After this patch, we have:
>=20
>   if (bridge && bus->self && hotplug_is_native(bus->self)) {
>     for_each_pci_bridge(dev, bus)
>       acpiphp_native_scan_bridge(dev);
>   } else {
>     ...
>     pci_assign_unassigned_bridge_resources(bus->self);
>   }
>=20
> We do not do pci_assign_unassigned_bridge_resources() in the "then"
> part of the "if".  Per the comment, that case may be used for adding
> Thunderbolt controllers.  Is there a reason we do not want
> pci_assign_unassigned_bridge_resources() in that path,
> or should it be
> in both cases?
acpiphp_native_scan_bridge() looks very similar to 'else'
branch modulo skip native hp bridge condition.
Otherwise both branches look similar,=20
but I don't have means to test that usecase,
so I'd avoid touching something nobody complains about.

>=20
> > > > >         }
> > > > >
> > > > >         acpiphp_sanitize_bus(bus); =20
>=20