Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 3 Jun 2021 12:23:13 -0500
From:   Bjorn Helgaas <helgaas@kernel.org>
To:     Florian Fainelli <f.fainelli@gmail.com>
Cc:     Jim Quinlan <jim2101024@gmail.com>, linux-pci@vger.kernel.org,
        Bjorn Helgaas <bhelgaas@google.com>,
        Nicolas Saenz Julienne <nsaenzjulienne@suse.de>,
        bcm-kernel-feedback-list@broadcom.com, james.quinlan@broadcom.com,
        Nicolas Saenz Julienne <nsaenz@kernel.org>,
        Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
        Rob Herring <robh@kernel.org>,
        "moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" 
        <linux-rpi-kernel@lists.infradead.org>,
        "moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" 
        <linux-arm-kernel@lists.infradead.org>,
        open list <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v1 4/4] PCI: brcmstb: add shutdown call to driver
Message-ID: <20210603172313.GA2123252@bjorn-Precision-5520>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <2c046f34-8bf1-97ff-3440-7351c7b2d528@gmail.com>
Precedence: bulk

On Wed, May 26, 2021 at 10:03:47AM -0700, Florian Fainelli wrote:
> On 5/25/21 2:18 PM, Bjorn Helgaas wrote:
> > On Tue, Apr 27, 2021 at 01:51:39PM -0400, Jim Quinlan wrote:
> >> The shutdown() call is similar to the remove() call except the former does
> >> not need to invoke pci_{stop,remove}_root_bus(), and besides, errors occur
> >> if it does.
> > 
> > This doesn't explain why shutdown() is necessary.  "errors occur"
> > might be a hint, except that AFAICT, many similar drivers do invoke
> > pci_stop_root_bus() and pci_remove_root_bus() (several of them while
> > holding pci_lock_rescan_remove()), without implementing .shutdown().
> 
> We have to implement .shutdown() in order to meet a certain power budget
> while the chip is being put into S5 (soft off) state and still support
> Wake-on-WLAN, for our latest chips this translates into roughly 200mW of
> power savings at the wall. We could probably add a word or two in a v2
> that indicates this is done for power savings.

"Saving power" is a great reason to do this.  But we still need to
connect this to the driver model and the system-level behavior
somehow.

The pci_driver comment says @shutdown is to "stop idling DMA
operations" and it hooks into reboot_notifier_list in kernel/sys.c.
That's incorrect or at least incomplete because reboot_notifier_list
isn't mentioned at all in kernel/sys.c, and I don't see the connection
between @shutdown and reboot_notifier_list.

AFAICT, @shutdown is currently used in this path:

  kernel_restart_prepare or kernel_shutdown_prepare
    device_shutdown
      dev->bus->shutdown
        pci_device_shutdown                     # pci_bus_type.shutdown
          drv->shutdown

so we're going to either reboot or halt/power-off the entire system,
and we're not going to use this device again until we're in a
brand-new kernel and we re-enumerate the device and re-register the
driver.

I'm not quite sure how either of those fits into the power-saving
reason.  I guess going to S5 is probably via the kernel_power_off()
path and that by itself doesn't turn off as much power to the PCIe
controller as it could?  And this new .shutdown() method will get
called in that path and will turn off more power, but will still leave
enough for wake-on-LAN to work?  And when we *do* wake from S5,
obviously that means a complete boot with a new kernel.

Bjorn