Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <20201218031703.3053753-6-saravanak@google.com> <20210117230134.32042-1-michael@walle.cc>
In-Reply-To: <20210117230134.32042-1-michael@walle.cc>
From:   Saravana Kannan <saravanak@google.com>
Date:   Mon, 18 Jan 2021 13:01:22 -0800
Message-ID: <CAGETcx9=6fPAHLuMyfxfXTGxeSUO8FwHVU_F4bfqLTfK6c+eXw@mail.gmail.com>
Subject: Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default
To:     Michael Walle <michael@walle.cc>
Cc:     Jisheng Zhang <Jisheng.Zhang@synaptics.com>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        John Stultz <john.stultz@linaro.org>,
        Android Kernel Team <kernel-team@android.com>,
        Kevin Hilman <khilman@baylibre.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Marc Zyngier <maz@kernel.org>,
        Nicolas Saenz Julienne <nsaenzjulienne@suse.de>,
        "Rafael J. Wysocki" <rafael@kernel.org>, minghuan.Lian@nxp.com,
        mingkai.hu@nxp.com, roy.zang@nxp.com,
        Linux PCI <linux-pci@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Sun, Jan 17, 2021 at 3:01 PM Michael Walle <michael@walle.cc> wrote:
>
> Hi Saravana, again ;)

Hi again! :)

>
> > Cyclic dependencies in some firmware was one of the last remaining
> > reasons fw_devlink=on couldn't be set by default. Now that cyclic
> > dependencies don't block probing, set fw_devlink=on by default.
> >
> > Setting fw_devlink=on by default brings a bunch of benefits (currently,
> > only for systems with device tree firmware):
> > * Significantly cuts down deferred probes.
> > * Device probe is effectively attempted in graph order.
> > * Makes it much easier to load drivers as modules without having to
> >   worry about functional dependencies between modules (depmod is still
> >   needed for symbol dependencies).
> >
> > If this patch prevents some devices from probing, it's very likely due
> > to the system having one or more device drivers that "probe"/set up a
> > device (DT node with compatible property) without creating a struct
> > device for it.  If we hit such cases, the device drivers need to be
> > fixed so that they populate struct devices and probe them like normal
> > device drivers so that the driver core is aware of the devices and their
> > status. See [1] for an example of such a case.
> >
> > [1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/
> > Signed-off-by: Saravana Kannan <saravanak@google.com>
>
> This breaks (at least) probing of the PCIe controllers of my board. The
> driver in question is
>   drivers/pci/controller/dwc/pci-layerscape.c
> I've also put the maintainers of this driver on CC. Looks like it uses a
> proper struct device. But it uses builtin_platform_driver_probe() and
> apparently it waits for the iommu which uses module_platform_driver().
> Dunno if that will work together.

Yeah, the builtin vs module doesn't matter. I've had fw_devlink work
multiple times with the consumer driver being built in and the
supplier actually loaded as a module. Making that work is one of the
goals of fw_devlink.

> The board device tree can be found here:
>   arch/arm64/boot/dts/freescale/fsl-ls1028a-kontron-sl28-var3-ads2.dts
>
> Attached is the log with enabled "probe deferral" messages enabled.

I took a look at the logs. As you said, pci seems to be waiting on
iommu, but it's not clear why the iommu didn't probe by then. Can you
add initcall_debug=1 and enable the logs in device_link_add()? Btw, I
realize one compromise on the logs is to send them as an attachment
instead of inline. That way, it's still archived in the list, but I
don't have to deal with log lines getting wrapped, etc.

Thanks for reporting the issues. Also, could you try picking up all of
these changes and giving it a shot. It's unlikely to help, but I want
to rule out issues related to fixes in progress.

https://lore.kernel.org/lkml/20210116011412.3211292-1-saravanak@google.com/
https://lore.kernel.org/lkml/20210115210159.3090203-1-saravanak@google.com/
https://lore.kernel.org/lkml/20201218210750.3455872-1-saravanak@google.com/

Thanks,
Saravana