Received-SPF: pass (google.com: domain of linux-kernel+bounces-173931-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Wed, 8 May 2024 18:33:02 -0500
From: Bjorn Helgaas <helgaas@kernel.org>
To: Jim Quinlan <james.quinlan@broadcom.com>
Cc: linux-pci@vger.kernel.org, Nicolas Saenz Julienne <nsaenz@kernel.org>,
	Bjorn Helgaas <bhelgaas@google.com>,
	Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
	Cyril Brulebois <kibi@debian.org>,
	Phil Elwell <phil@raspberrypi.com>,
	bcm-kernel-feedback-list@broadcom.com,
	Florian Fainelli <florian.fainelli@broadcom.com>,
	Jim Quinlan <jim2101024@gmail.com>,
	Lorenzo Pieralisi <lpieralisi@kernel.org>,
	Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
	Rob Herring <robh@kernel.org>,
	"moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" <linux-rpi-kernel@lists.infradead.org>,
	"moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" <linux-arm-kernel@lists.infradead.org>,
	open list <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v9 4/4] PCI: brcmstb: Configure HW CLKREQ# mode
 appropriate for downstream device
Message-ID: <20240508233302.GA1792067@bhelgaas>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CA+-6iNxEZm=axRAeeAKwxemjEdjjmJYTUs8nThp_NDohXcV5Jg@mail.gmail.com>

On Wed, May 08, 2024 at 01:55:24PM -0400, Jim Quinlan wrote:
> On Mon, May 6, 2024 at 7:20 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> ...

> > As a user, how do I determine which setting to use?
>
> Using the "safe" mode will always work.  In fact I considered making
> this the default mode.

> As I said, I cannot enumerate all of the reasons why one mode works
> and one does not for a particular device+board+connector combo.  The
> HW folks have not really been forthcoming on the reasons as well.
> 
> > Trial and error?  If so, how do I identify the errors?
>
> Either PCIe link-up is not happening, or it is happening but the
> device driver is non-functional and boot typically  hangs.

What I'm hearing is that it's trial and error. 

If we can't tell users how to figure out which mode to use, I think we
have to explicitly say "try the modes in this order until you find one
that works."

That sucks, but if it's all we can do, I guess we don't have much
choice, and we should just own up to it.

There's no point in telling users "if your card drives CLKREQ# use X,
but if not and it can tolerate out-of-spec T_CLRon timing, use Y"
because nobody knows how to figure that out.

And we can say which features are enabled in each mode so they aren't
surprised, e.g., something like this:

  "default" -- The Root Port supports ASPM L0s, L1, L1 Substates, and
    Clock Power Management.  This provides the best power savings but
    some devices may not work correctly because the Root Port doesn't
    comply with T_CLRon timing required for PCIe Mini Cards [1].

  "no-l1ss" -- The Root Port supports ASPM L0s, L1 (but not L1
    Substates), and Clock Power Management.  [I assume there's some
    other Root Port defect that causes issues with some devices in
    this mode; I dunno.  If we don't know exactly what it is, I guess
    we can't really say anything.]

  "safe" -- The Root Port supports ASPM L0, L1, L1 Substates, but not
    Clock Power Management.  All devices should work in this mode.

[1] PCIe Mini CEM r2.1, sec 3.2.5.2.2

(I'm not sure which features are *actually* enabled in each mode, I
just guessed.)

Bjorn