Received: by 2002:ac0:da4c:0:0:0:0:0 with SMTP id a12csp691389imi; Thu, 21 Jul 2022 09:05:41 -0700 (PDT) X-Google-Smtp-Source: AGRyM1vy0mLI5NzvwsiWLyS93RF3BCnj2rcaEAz98cyjt1Zub2p+H92BqzGui8zGZlkeBK1wAVy6 X-Received: by 2002:aca:3c06:0:b0:33a:4951:84fe with SMTP id j6-20020aca3c06000000b0033a495184femr5095158oia.198.1658419540920; Thu, 21 Jul 2022 09:05:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1658419540; cv=none; d=google.com; s=arc-20160816; b=ePm0Tlr3zNCaUtUhGOqG8CbWjwdsFi++BZepsDurFf78iAlxqlUL0vNf89I1DD6A/o msxkx/6mDiq/IM4xeJuUm7ghwv6XhLYYsYpRD8CCfN9O1+e9XFqoh8kIJZRLqs7rfTvx osxqlAkm/RJR+RJMofj8vCa3m4+V84ACKlXesC4HhA+/a1HSvpGyuh3yRPWvV0dymJdL ABTkpnf8turpECB/FqI/s+JbhW9wSxKh+rKu6NRG8Uj6fpeKn5lQtq3tYH/cdzvQTykM B1OqzwqDs48LGxH/cMge49840lhoeqOC4g90K1HEZfiulpzKM/vBH/uM2m8hkmmOrkr9 JU4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :message-id:subject:cc:to:from:date:dkim-signature; bh=YzO7TMZdixn17o4Fkn5NmafQf/uou+3QKtdv2kWS7Zc=; b=fv+ofPMgQ8KXJGKHL/6hAj4roPPbFvy85LR0tlU4sigtUyDlbSMymtOxqxv/JmQr7E tziOx5myevqbeoflIvLIqEBi+X337v39j4PwQqfQ3N1Q4Hw/viBRomlToIPclw6E1F9+ Bvg6BUIoB8sEW6H+tX5/nF9IftqtIcobkZB+v0DlCMwZkkVBhhdeJ7Yvbi67NhyRG2i8 yDfPTnCS8uZSaQP0ke0JI1K7TqHvRD1M+sR5SdLTg7r/Gj3Dl3F/LsmZX5eQxOcJDtux kyCTxDSm4WIh6HZJ88v3mfE8FoJ4ztbrWEy9bpsGIyy3O+iVhj5Sk+4gyHnzRxow7UwV KXtg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="XGWFT/85"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x8-20020a056870e38800b00101a2f2ddd3si1882245oad.191.2022.07.21.09.05.27; Thu, 21 Jul 2022 09:05:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="XGWFT/85"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233493AbiGUPsU (ORCPT + 99 others); Thu, 21 Jul 2022 11:48:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233097AbiGUPsH (ORCPT ); Thu, 21 Jul 2022 11:48:07 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2866289A8F; Thu, 21 Jul 2022 08:46:32 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id CD0B4B8253C; Thu, 21 Jul 2022 15:46:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EC5A7C341C0; Thu, 21 Jul 2022 15:46:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1658418389; bh=PXrSX6TqOGgu/mUPbX1MSmgTXFQ9SHdxh/ZWILLOrWM=; h=Date:From:To:Cc:Subject:In-Reply-To:From; b=XGWFT/851segzOr4xlR49pzv4LvKUjk8gig6g8kUKwDDgcAfTJIc0R03rIpTCY3aV 1KS2DiK3JMESQACTiJYmZnlYMzrquaz7yK9Ez8VrJ/LZQaJgf2rfBdSgOhwbUqkzpG D2pmFlqhbcZ+QcwhmIyeTcq2fZX2aRJQB4L5M27vI9F/cn/a7Ml03p2IFh2BhHstQY NKaz8J6QOMgt1His78n/IsY6GxYY/VWBhcWsosInI/8e3MsKz0I8NbQ3AAOrOvAXSX cmmhojATSEUiEIMJnHwSnQwTuUsqO49oryt5KKAbQk+sjHW9S6KS3PuTrCOx089Q5B KugxgcpzPBFXQ== Date: Thu, 21 Jul 2022 10:46:27 -0500 From: Bjorn Helgaas To: Jim Quinlan Cc: Jim Quinlan , "open list:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS" , Nicolas Saenz Julienne , Bjorn Helgaas , Lorenzo Pieralisi , Cyril Brulebois , "maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE" , Florian Fainelli , Lorenzo Pieralisi , Rob Herring , Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= , "moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" , "moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" , open list Subject: Re: [PATCH v2 3/6] PCI: brcmstb: Add "refusal mode" to preclude PCIe-induced CPU aborts Message-ID: <20220721154627.GA1722428@bhelgaas> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-7.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Jul 21, 2022 at 10:53:54AM -0400, Jim Quinlan wrote: > https://lore.kernel.org/linux-pci/20171215201434.GY30595@bhelgaas-glaptop.roam.corp.google.com/ > On Wed, Jul 20, 2022 at 6:06 PM Bjorn Helgaas wrote: > > On Sat, Jul 16, 2022 at 06:24:50PM -0400, Jim Quinlan wrote: > > > Our PCIe RC HW has an atypical behavior: if it does not have PCIe link > > > established between itself and downstream, any subsequent config space > > > access causes a CPU abort. This commit sets a "refusal mode" if the PCIe > > > link-up fails, and this has our pci_ops map_bus function returning a NULL > > > address, which in turn precludes the access from happening. > > > > > > Right now, "refusal mode" is window dressing. It will become relevant > > > in a future commit when brcm_pcie_start_link() is invoked during > > > enumeration instead of before it. > > > > > > Signed-off-by: Jim Quinlan > > > --- > > > drivers/pci/controller/pcie-brcmstb.c | 24 ++++++++++++++++++++++++ > > > 1 file changed, 24 insertions(+) > > > > > > diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c > > > index c026446d5830..72219a4f3964 100644 > > > --- a/drivers/pci/controller/pcie-brcmstb.c > > > +++ b/drivers/pci/controller/pcie-brcmstb.c > > > @@ -255,6 +255,7 @@ struct brcm_pcie { > > > u32 hw_rev; > > > void (*perst_set)(struct brcm_pcie *pcie, u32 val); > > > void (*bridge_sw_init_set)(struct brcm_pcie *pcie, u32 val); > > > + bool refusal_mode; > > > }; > > > > > > static inline bool is_bmips(const struct brcm_pcie *pcie) > > > @@ -687,6 +688,19 @@ static void __iomem *brcm_pcie_map_conf(struct pci_bus *bus, unsigned int devfn, > > > if (pci_is_root_bus(bus)) > > > return PCI_SLOT(devfn) ? NULL : base + where; > > > > > > + if (pcie->refusal_mode) { > > > + /* > > > + * At this point we do not have PCIe link-up. If there is > > > + * a config read or write access besides those targeting > > > + * the host bridge, our PCIe HW throws a CPU abort. To > > > + * prevent this we return the NULL address. The calling > > > + * functions -- pci_generic_config_*() -- will notice this > > > + * and not perform the access, and if it is a read access, > > > + * 0xffffffff is returned. > > > + */ > > > + return NULL; > > > + } > > > > Is this any different from all the other .map_bus() implementations > > that return NULL when the link is down? > > Not really, but long ago I submitted code that gated the config spec > access based on link status and was advised not to do it [1]. > I'll be happy to make it look like the others. > > [1] https://lore.kernel.org/linux-pci/20171215201434.GY30595@bhelgaas-glaptop.roam.corp.google.com/ My point there was that if you can deal with the abort cleanly, that's the best approach. Apparently brcmstb can't recover cleanly, so you have to settle for the 99% solution. The refusal_mode approach has the same race as checking *_pcie_link_up(), since the link may go down between the time brcm_pcie_start_link() sees that it is up and the time somebody does a config access: brcm_pcie_start_link pcie->refusal_mode = false brcm_pcie_map_conf if (pcie->refusal_mode) # still false So there's no advantage in making the code look different. Checking for link-up in the config access path can never completely remove the window, but it does make it smaller than using refusal_mode. Bjorn