Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230411165919.23955-3-jim2101024@gmail.com> <20230414202720.GA215111@bhelgaas>
In-Reply-To: <20230414202720.GA215111@bhelgaas>
From:   Jim Quinlan <jim2101024@gmail.com>
Date:   Fri, 14 Apr 2023 19:14:02 -0400
Message-ID: <CANCKTBuBZ5j+MqD9_uudvESO5WGC5hqCK1Sd4vD-CBzHsv+0zA@mail.gmail.com>
Subject: Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     linux-pci@vger.kernel.org,
        Nicolas Saenz Julienne <nsaenz@kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
        Cyril Brulebois <kibi@debian.org>,
        Phil Elwell <phil@raspberrypi.com>,
        bcm-kernel-feedback-list@broadcom.com, james.quinlan@broadcom.com,
        Florian Fainelli <f.fainelli@gmail.com>,
        Lorenzo Pieralisi <lpieralisi@kernel.org>,
        =?UTF-8?Q?Krzysztof_Wilczy=C5=84ski?= <kw@linux.com>,
        Rob Herring <robh@kernel.org>,
        "moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" 
        <linux-rpi-kernel@lists.infradead.org>,
        "moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" 
        <linux-arm-kernel@lists.infradead.org>,
        open list <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Fri, Apr 14, 2023 at 4:27=E2=80=AFPM Bjorn Helgaas <helgaas@kernel.org> =
wrote:
>
> This subject line no verb.  Can you add a leading verb to suggest what
> this patch does?
>
> s/accomodations/accommodations/
>
> On Tue, Apr 11, 2023 at 12:59:17PM -0400, Jim Quinlan wrote:
> > The Broadcom STB/CM PCIe HW core, which is also used in RPi SOCs, must =
be
> > deliberately set by the probe() into one of three mutually exclusive mo=
des:
> >
> >   (a) No CLKREQ# expected or required, refclk is always available.
> >   (b) CLKREQ# is expected to be driven by downstream device when needed=
.
> >   (c) Bidirectional CLKREQ# for L1SS capable devices.
> >
> > Previously, only (b) was supported by the driver, as almost all STB/CM
> > boards operate in this mode.  But now there is interest in activating L=
1SS
> > power savings from STB/CM customers, and also interest in accomodating =
mode
> > (a) for designs such as the RPi CM4 with IO board.
>
> accommodating
>
> > The HW+driver is able to tell us when mode (a) mode is needed.  But the=
re
> > is no easy way to tell if L1SS mode should be configured.  In certain
> > situations, getting this wrong may cause a panic during boot time.  So =
we
> > rely on the DT prop "brcm,enable-l1ss" to tell us when mode (c) is desi=
red.
> > Using this mode only makes sense when the downstream device is L1SS-cap=
able
> > and the OS has been configured to activate L1SS
> > (e.g. policy=3D=3Dpowersupersave).
>
> I'm really concerned about the user experience here.  I assume users
> do not want to edit the DT based on what device they plug in.  They
> shouldn't need to (and probably won't) know whether the device
> supports L1SS.
>
> I hate kernel/module parameters, but I think even that would be better
> then having to edit the DT.
>
> There's obviously a period of time when L1SS is supported but not yet
> enabled, so I'm *guessing* the "OS has been configured to activate
> L1SS" is not actually a requirement, and choosing (c) really just
> opens the possibility that L1SS can be used?

Yes.  Before this patch series we had two panic scenarios:

(a) Endpoint devices with no CLKREQ# connection
(b) Endpoints that are L1SS-capable

Even without  the "brcm,enable-l1ss" property present, both (a) and
(b) should be eliminated.
The reason (b) is eliminated is because the RC driver now unadvertises
 RC L1SS by default; subsequently, Linux does
not turn it on.  So the default setting should be fine for all devices.

For those folks who have L1SS capable devices and desire L1SS power
savings, they can add
the brcm,enable-l1ss property.  But everyone should have functionality
 w/o doing anything.

As I am typing this I realize that my comments and dev_info()s  are not
aligned with what I am saying so I will change them in V3.  Sorry
about the confusion.

>
> Would be nice to have a hint (maybe a line or two of the panic
> message) to help users find the fix for a problem they're seeing.SS
>
> Obviously the ideal would be if we could use (c) in all cases, so I
> assume that's where a panic might happen.  What situation would that
> be?  An endpoint that doesn't support L1SS?  One that supports L1SS
> but it's not enabled?  Maybe if L1SS isn't configured correctly, e.g.,
> LTR values programmed wrong?

Let me test everything on Monday and get back to you before I make any
statements.


Regards,
Jim Quinilan
Broadcom STB

>
> Bjorn