Received-SPF: pass (google.com: domain of linux-kernel+bounces-97357-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Fri, 8 Mar 2024 11:12:48 -0600
From: Bjorn Helgaas <helgaas@kernel.org>
To: Krishna Chaitanya Chundru <quic_krichai@quicinc.com>
Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>,
	Jingoo Han <jingoohan1@gmail.com>,
	Gustavo Pimentel <gustavo.pimentel@synopsys.com>,
	Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>,
	Lorenzo Pieralisi <lpieralisi@kernel.org>,
	Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
	Rob Herring <robh@kernel.org>, Bjorn Helgaas <bhelgaas@google.com>,
	linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
	quic_vbadigan@quicinc.com, quic_ramkri@quicinc.com,
	quic_nitegupt@quicinc.com, quic_skananth@quicinc.com,
	quic_parass@quicinc.com
Subject: Re: [PATCH v2] PCI: dwc: Enable runtime pm of the host bridge
Message-ID: <20240308171248.GA685266@bhelgaas>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <e83ed3e5-0c31-cfae-6f75-211709e79aa5@quicinc.com>

On Fri, Mar 08, 2024 at 08:38:52AM +0530, Krishna Chaitanya Chundru wrote:
> On 3/8/2024 3:25 AM, Bjorn Helgaas wrote:
> > [+to Rafael, sorry, another runtime PM question, beginning of thread:
> > https://lore.kernel.org/r/20240305-runtime_pm_enable-v2-1-a849b74091d1@quicinc.com]
> > 
> > On Thu, Mar 07, 2024 at 07:28:54AM +0530, Krishna Chaitanya Chundru wrote:
> > > On 3/6/2024 1:27 AM, Bjorn Helgaas wrote:
> > > > On Tue, Mar 05, 2024 at 03:19:01PM +0530, Krishna chaitanya chundru wrote:
> > > > > The Controller driver is the parent device of the PCIe host bridge,
> > > > > PCI-PCI bridge and PCIe endpoint as shown below.
> > > > > 
> > > > > 	PCIe controller(Top level parent & parent of host bridge)
> > > > > 			|
> > > > > 			v
> > > > > 	PCIe Host bridge(Parent of PCI-PCI bridge)
> > > > > 			|
> > > > > 			v
> > > > > 	PCI-PCI bridge(Parent of endpoint driver)
> > > > > 			|
> > > > > 			v
> > > > > 		PCIe endpoint driver
> > > > > 
> > > > > Since runtime PM is disabled for host bridge, the state of the child
> > > > > devices under the host bridge is not taken into account by PM framework
> > > > > for the top level parent, PCIe controller. So PM framework, allows
> > > > > the controller driver to enter runtime PM irrespective of the state
> > > > > of the devices under the host bridge.
> > > > 
> > > > IIUC this says that we runtime suspend the controller even though
> > > > runtime PM is disabled for the host bridge?  I have a hard time
> > > > parsing this; can you cite a function that does this or some relevant
> > > > documentation about how this part of runtime PM works?
> > > > 
> > > Generally controller should go to runtime suspend when endpoint client
> > > drivers and pci-pci host bridge drivers goes to runtime suspend as the
> > > controller driver is the parent, but we are observing controller driver
> > > goes to runtime suspend even when client drivers and PCI-PCI bridge are
> > > in active state.
> > 
> > It surprises me that a device could be suspended while children are
> > active.  A PCI-PCI bridge must be in D0 for any devices below it to be
> > active.  The controller is a platform device, not a PCI device, but I
> > am similarly surprised that we would suspend it when children are
> > active, which makes me think we didn't set the hierarchy up correctly.
> > 
> > It doesn't seem like we should need to enable runtime PM for a parent
> > to keep it from being suspended when children are active.
>
> Here we are not enabling runtime PM of the controller device, we are
> enabling runtime PM for the bridge device which is maintained by the
> PCIe framework. The bridge device is the parent of the PCI-PCI
> bridge and child of the controller device. As the bridge device's
> runtime PM is not enabled the PM framework is ignoring the child's
> runtime status.

OK, it's the host bridge, not the controller.

I'm still surprised that the PM framework will runtime suspend a
device when child devices are active.

And further confused about managing the host bridge runtime PM in a
controller driver.  Which other callers of pci_alloc_host_bridge() or
devm_pci_alloc_host_bridge() will need similar changes?

> > > > > --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> > > > > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> > > > > @@ -16,6 +16,7 @@
> > > > >   #include <linux/of_pci.h>
> > > > >   #include <linux/pci_regs.h>
> > > > >   #include <linux/platform_device.h>
> > > > > +#include <linux/pm_runtime.h>
> > > > > 
> > > > >   #include "../../pci.h"
> > > > >   #include "pcie-designware.h"
> > > > > @@ -505,6 +506,9 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
> > > > >          if (pp->ops->post_init)
> > > > >                  pp->ops->post_init(pp);
> > > > > 
> > > > > +       pm_runtime_set_active(&bridge->dev);
> > > > > +       pm_runtime_enable(&bridge->dev);
> > > > > +
> > > > >          return 0;
> > > > > 
> > > > >   err_stop_link: