Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 14 Aug 2023 17:06:58 -0500
From:   Bjorn Helgaas <helgaas@kernel.org>
To:     Jim Quinlan <james.quinlan@broadcom.com>
Cc:     Bjorn Helgaas <bhelgaas@google.com>,
        "open list:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS" 
        <linux-pci@vger.kernel.org>,
        Nicolas Saenz Julienne <nsaenz@kernel.org>,
        Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
        Cyril Brulebois <kibi@debian.org>,
        Phil Elwell <phil@raspberrypi.com>,
        "maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE" 
        <bcm-kernel-feedback-list@broadcom.com>,
        Florian Fainelli <f.fainelli@gmail.com>,
        Jim Quinlan <jim2101024@gmail.com>,
        Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
        Lorenzo Pieralisi <lpieralisi@kernel.org>,
        Rob Herring <robh@kernel.org>,
        "moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" 
        <linux-rpi-kernel@lists.infradead.org>,
        "moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" 
        <linux-arm-kernel@lists.infradead.org>,
        open list <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v6 3/5] PCI: brcmstb: Set higher value for internal bus
 timeout
Message-ID: <20230814220658.GA193031@bhelgaas>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CA+-6iNzKB4vMkM4Qt6781GN-WwzCCdUYY3o=stgPSjQyJPr6vQ@mail.gmail.com>
Precedence: bulk

On Mon, Aug 14, 2023 at 03:30:07PM -0400, Jim Quinlan wrote:
> On Fri, Jul 28, 2023 at 12:15 PM Jim Quinlan <james.quinlan@broadcom.com> wrote:
> > On Thu, Jul 27, 2023, 10:44 PM Lorenzo Pieralisi <lpieralisi@kernel.org> wrote:
> >> On Fri, Jun 23, 2023 at 10:40:56AM -0400, Jim Quinlan wrote:
> >> > During long periods of the PCIe RC HW being in an L1SS sleep state, there
> >> > may be a timeout on an internal bus access, even though there may not be
> >> > any PCIe access involved.  Such a timeout will cause a subsequent CPU
> >> > abort.
> >> >
> >> > So, when "brcm,enable-l1ss" is observed, we increase the timeout value to
> >> > four seconds instead of using its HW default.
> >> >
> >> > Signed-off-by: Jim Quinlan <james.quinlan@broadcom.com>
> >> > Tested-by: Florian Fainelli <f.fainelli@gmail.com>
> >> > ---
> >> >  drivers/pci/controller/pcie-brcmstb.c | 16 ++++++++++++++++
> >> >  1 file changed, 16 insertions(+)
> >> >
> >> > diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
> >> > index d30636a725d7..fe0415a98c63 100644
> >> > --- a/drivers/pci/controller/pcie-brcmstb.c
> >> > +++ b/drivers/pci/controller/pcie-brcmstb.c
> >> > @@ -1034,6 +1034,21 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
> >> >       return 0;
> >> >  }
> >> >
> >> > +/*
> >> > + * This extends the timeout period for an access to an internal bus.  This
> >> > + * access timeout may occur during L1SS sleep periods even without the
> >> > + * presence of a PCIe access.
> >> > + */
> >> > +static void brcm_extend_rbus_timeout(struct brcm_pcie *pcie)
> >> > +{
> >> > +     /* TIMEOUT register is two registers before RGR1_SW_INIT_1 */
> >> > +     const unsigned int REG_OFFSET = PCIE_RGR1_SW_INIT_1(pcie) - 8;
> >>
> >> Nit: you could define an offset for the TIMEOUT register, if that makes
> >> it any cleaner, up to you.
> >>
> >> > +     u32 timeout_us = 4000000; /* 4 seconds, our setting for L1SS */
> >>
> >> It would be useful to describe why this has to be 4 seconds in case
> >> someone in the future will have to change it.
> >
> > IIRC our customer requested 2s and we doubled it.  Bjorn, can you
> > please add this comment or a paraphrase of it before applying --
> > I'm currently on vacation.
> 
> Is the above request okay with you?  What is the status of these
> commits -- will they be applied to pci-next in the near future?

The "brcm,enable-l1ss" DT property is either unnecessary or an
indication of a hardware defect in the controller.

Requiring the property is a terrible user experience, completely
antithetical to the PCI compatibility story, and per the conversation
at [1], there are no known problems that would occur if we ignored
"brcm,enable-l1ss" and always configured mode (c) ("Bidirectional
CLKREQ# for L1SS capable devices").

Even when configured as mode (c), L1SS is not *always* enabled.  It's
certainly not enabled before ASPM init, and users can always disable
L1SS whenever they desire via the sysfs interfaces or pcie_aspm=off,
so if there's some problem with running in mode (c) with L1SS
disabled, we're still likely to see it.

But if you want to require the DT property, I guess it's mainly an
issue for you and your customers.

So to answer your question, yes, I'm OK with this series.

Bjorn

[1] https://lore.kernel.org/r/20230428223500.23337-2-jim2101024@gmail.com

> >> > +     /* Each unit in timeout register is 1/216,000,000 seconds */
> >> > +     writel(216 * timeout_us, pcie->base + REG_OFFSET);
> >> > +}
> >> > +
> >> >  static void brcm_config_clkreq(struct brcm_pcie *pcie)
> >> >  {
> >> >       bool l1ss = of_property_read_bool(pcie->np, "brcm,enable-l1ss");
> >> > @@ -1059,6 +1074,7 @@ static void brcm_config_clkreq(struct brcm_pcie *pcie)
> >> >                * of 400ns, as specified in 3.2.5.2.2 of the PCI Express
> >> >                * Mini CEM 2.0 specification.
> >> >                */
> >> > +             brcm_extend_rbus_timeout(pcie);
> >> >               clkreq_set |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK;
> >> >               dev_info(pcie->dev, "bi-dir CLKREQ# for L1SS power savings");
> >> >       } else {