Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <20201218031703.3053753-6-saravanak@google.com>
 <20210117230134.32042-1-michael@walle.cc> <CAGETcx9=6fPAHLuMyfxfXTGxeSUO8FwHVU_F4bfqLTfK6c+eXw@mail.gmail.com>
 <4b9ae679b6f76d2f7e340e2ec229dd53@walle.cc>
In-Reply-To: <4b9ae679b6f76d2f7e340e2ec229dd53@walle.cc>
From:   Saravana Kannan <saravanak@google.com>
Date:   Tue, 19 Jan 2021 16:00:13 -0800
Message-ID: <CAGETcx-s1Hc8iiForpoeqP6Tf-syOYvxHMui+3vRJFmqArL+-A@mail.gmail.com>
Subject: Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default
To:     Michael Walle <michael@walle.cc>
Cc:     Jisheng Zhang <Jisheng.Zhang@synaptics.com>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        John Stultz <john.stultz@linaro.org>,
        Android Kernel Team <kernel-team@android.com>,
        Kevin Hilman <khilman@baylibre.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Marc Zyngier <maz@kernel.org>,
        Nicolas Saenz Julienne <nsaenzjulienne@suse.de>,
        "Rafael J. Wysocki" <rafael@kernel.org>, minghuan.Lian@nxp.com,
        mingkai.hu@nxp.com, roy.zang@nxp.com,
        Linux PCI <linux-pci@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Tue, Jan 19, 2021 at 2:41 AM Michael Walle <michael@walle.cc> wrote:
>
> Am 2021-01-18 22:01, schrieb Saravana Kannan:
> > On Sun, Jan 17, 2021 at 3:01 PM Michael Walle <michael@walle.cc> wrote:
> >> > Cyclic dependencies in some firmware was one of the last remaining
> >> > reasons fw_devlink=on couldn't be set by default. Now that cyclic
> >> > dependencies don't block probing, set fw_devlink=on by default.
> >> >
> >> > Setting fw_devlink=on by default brings a bunch of benefits (currently,
> >> > only for systems with device tree firmware):
> >> > * Significantly cuts down deferred probes.
> >> > * Device probe is effectively attempted in graph order.
> >> > * Makes it much easier to load drivers as modules without having to
> >> >   worry about functional dependencies between modules (depmod is still
> >> >   needed for symbol dependencies).
> >> >
> >> > If this patch prevents some devices from probing, it's very likely due
> >> > to the system having one or more device drivers that "probe"/set up a
> >> > device (DT node with compatible property) without creating a struct
> >> > device for it.  If we hit such cases, the device drivers need to be
> >> > fixed so that they populate struct devices and probe them like normal
> >> > device drivers so that the driver core is aware of the devices and their
> >> > status. See [1] for an example of such a case.
> >> >
> >> > [1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/
> >> > Signed-off-by: Saravana Kannan <saravanak@google.com>
> >>
> >> This breaks (at least) probing of the PCIe controllers of my board.
> >> The
> >> driver in question is
> >>   drivers/pci/controller/dwc/pci-layerscape.c
> >> I've also put the maintainers of this driver on CC. Looks like it uses
> >> a
> >> proper struct device. But it uses builtin_platform_driver_probe() and
> >> apparently it waits for the iommu which uses module_platform_driver().
> >> Dunno if that will work together.
> >
> > Yeah, the builtin vs module doesn't matter. I've had fw_devlink work
> > multiple times with the consumer driver being built in and the
> > supplier actually loaded as a module. Making that work is one of the
> > goals of fw_devlink.
>
> Ok.

Hi Michael,

My bad, I spoke too soon. I thought you were talking about builtin_ vs
module_. My response is correct in that context. But the problem here
is related to builtin_platform_driver_probe(). That macro expects the
device (PCI) to be added and ready to probe by the time it's called.
If not, it just gives up and frees the code. That's why it's not
getting called after the first attempt. Can you please convert it into
builtin_platform_driver()? It should be a pretty trivial change.

-Saravana

>
> >> The board device tree can be found here:
> >>   arch/arm64/boot/dts/freescale/fsl-ls1028a-kontron-sl28-var3-ads2.dts
> >>
> >> Attached is the log with enabled "probe deferral" messages enabled.
> >
> > I took a look at the logs. As you said, pci seems to be waiting on
> > iommu, but it's not clear why the iommu didn't probe by then. Can you
> > add initcall_debug=1 and enable the logs in device_link_add()? Btw, I
> > realize one compromise on the logs is to send them as an attachment
> > instead of inline. That way, it's still archived in the list, but I
> > don't have to deal with log lines getting wrapped, etc.
> >
> > Thanks for reporting the issues. Also, could you try picking up all of
> > these changes and giving it a shot. It's unlikely to help, but I want
> > to rule out issues related to fixes in progress.
> >
> > https://lore.kernel.org/lkml/20210116011412.3211292-1-saravanak@google.com/
> > https://lore.kernel.org/lkml/20210115210159.3090203-1-saravanak@google.com/
> > https://lore.kernel.org/lkml/20201218210750.3455872-1-saravanak@google.com/
>
> Did pick them up, the last one had a conflict due some superfluous
> lines.
> Maybe they got reordered in that arrray.
>
> Issue still persist. I've enabled the debug in device_link_add(), in
> device_links_check_suppliers() and booted with initcall_debug. Please
> see attached log. Lets see how that goes ;)
>
> [    0.132687] calling  ls_pcie_driver_init+0x0/0x38 @ 1
> [    0.132762] platform 3400000.pcie: probe deferral - supplier
> 5000000.iommu not ready
> [    0.132777] platform 3500000.pcie: probe deferral - supplier
> 5000000.iommu not ready
> [    0.132818] initcall ls_pcie_driver_init+0x0/0x38 returned -19 after
> 119 usecs
>
> After that, ls_pcie_driver_init() is never called again.
>
> -michael