Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 1 Jun 2023 11:11:56 +0300
From:   Vladimir Oltean <vladimir.oltean@nxp.com>
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     linux-pci@vger.kernel.org, netdev@vger.kernel.org,
        Bjorn Helgaas <bhelgaas@google.com>,
        Rob Herring <robh@kernel.org>,
        Claudiu Manoil <claudiu.manoil@nxp.com>,
        Michael Walle <michael@walle.cc>, linux-kernel@vger.kernel.org,
        Liu Peibao <liupeibao@loongson.cn>,
        Binbin Zhou <zhoubinbin@loongson.cn>,
        Huacai Chen <chenhuacai@loongson.cn>
Subject: Re: [PATCH pci] PCI: don't skip probing entire device if first fn OF
 node has status = "disabled"
Message-ID: <20230601081156.zyymihd565fscuha@skbuf>
References: <20230531165819.phx7uwlgtvnt3tvb@skbuf>
 <ZHetDo5PozWdtrxP@bhelgaas>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZHetDo5PozWdtrxP@bhelgaas>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?0vN4br9sQRWld9dGsUGbryiCTnjqU42L3od8TKm5llxaWeP+8da/U56/UPc9?=
 =?us-ascii?Q?x2dwe7t5rV0RAMer98Lb9XLPhksbbkQbznFrCJgDdHU9Gc3TCYvpuWB8mn8E?=
 =?us-ascii?Q?rxNGchjVj75on291OXpl90ktWmJ3lM6m4qR+hMWQ5b59UWPL9qfeuoWWFD7e?=
 =?us-ascii?Q?+tjZjbihOMqwExv9lOOB5wodHph2ar5omad6mwrhgv05LS+Qk9o4I0mYr501?=
 =?us-ascii?Q?uLJQaLZQpTFkHQNzmbVfLPMJa6CORB0mqBDlCHlJeJEtX7dGLJONMRBosmYF?=
 =?us-ascii?Q?cnFnVjybBugWC/H8zz5nFPhIRrmAhvasRUWeREbEvAE9tUxZf3WIR9e466yf?=
 =?us-ascii?Q?u4tfABA4GrGy2IMX9q3iFYJIEpSbtN7sQtQUQ1kPxnjhJ30WfWQ9005HoqAI?=
 =?us-ascii?Q?FXmvpeFqih3vZyJyAnPaCixowyJwPqDAmDOMKVGcrs7cpxT3mbTD3ga7G2Se?=
 =?us-ascii?Q?mZpBczZvz7mHfpUanvvKVS/ELZlUe/vTrz+3RJqsFH2uVMCzIJ8sLvBMEu5y?=
 =?us-ascii?Q?a1W911yoJDzMeksjqFC9MfTltQt+Fw3K98v/VPqKycH8a0lBYZqrzqGPSZyV?=
 =?us-ascii?Q?/BzYflHOCEt/so8blVYe6nztvqRgbA7GZtMFtd38kiWIcMec62L/6kOu/gYm?=
 =?us-ascii?Q?UBVnI6x1SvuwPpFPOordhXQpaWNX0noHDq9+W0qhImjozmuveiic/G6iVHHn?=
 =?us-ascii?Q?Q+w6/xm8u72nHpoinl9IJifzEZ+slnv3Pf5ayDLv131zELBSrAisDRAcpCln?=
 =?us-ascii?Q?y9sQVV42LQ7BdRm0gcxsXmAEXuwvHKMHd+FEJ0mgYn7kw/xdBJY6h0750ZZa?=
 =?us-ascii?Q?p61xWhagpztM3oBvKsAP0owIAjWFh5g4hRhfB21rO7MDjYQO7MQYXId3qstb?=
 =?us-ascii?Q?7Mxh8JE8RuNYuxcy8wGIBo9+cr2PCLFrx3jKNEfVbGWz8C4R4Az4zwJwIXyH?=
 =?us-ascii?Q?ARV4pwLwdAxrJxlB+tyoGEqEtNL4QyEdA+jXMK7R4JPnop6WCcgFIfxQY2fq?=
 =?us-ascii?Q?OPwiFE5EmXazajaMr9zAJ7xFg/Qhf8O0EqMAQGftP2hnE4l2VehxJ1PUaioI?=
 =?us-ascii?Q?tkFQgBrIEd5hpPEBxBeoqaqvhPqKO49KLgZ9ur2TSJWFHDnW2boU3NfbYgeE?=
 =?us-ascii?Q?EmwJd5CGDQduj0OD+Ti+L6ZLlRWWj7PZRF3F6xO5W1RW/R6XW3nTLSCqgxe+?=
 =?us-ascii?Q?xo+6ahLtZjffmREQQMZ1LA1uYlbTJrQPEcqOwbnlMW6dat7BtdcW3nk1Yl6Y?=
 =?us-ascii?Q?0EYglDcbz01jf8/6oSkXweJR8qnu4sbYe40krFDNi5zOwxZ4gSICUtBF0tPJ?=
 =?us-ascii?Q?KGlF6Mye+uT2c40FJpPv2sXs2WjGoovGgp7mWeA/hGN3Y1yF68w3Wkhy9viw?=
 =?us-ascii?Q?ytthETNZeBnUMfWmq2Cylh/pNSca2tvas/3U7FfEq2PaYDsADllA4TvauoOF?=
 =?us-ascii?Q?6vjFdwVhoz34NHkDQTBJPk5eM7y5py5HV2/F5rwkojq3XIelYeZz6Av8u8mU?=
 =?us-ascii?Q?uRfuh09taCE0XApoZPZq/88BNFru2Tnql5GomSu5g8TuIqRMRK4zCI8Q3a3x?=
 =?us-ascii?Q?Pesnvm/cDr5qEXTAVDEtR8mrtp8wbCtenwDUVOp4eDG5y4Tg8Q8rzL1TZALG?=
 =?us-ascii?Q?sA=3D=3D?=
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cd8df577-436d-4dd7-566b-08db6277dfea
X-MS-Exchange-CrossTenant-AuthSource: AM0PR04MB6452.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Jun 2023 08:12:01.3156
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 2mwEiYylBgTrneKSI4Gpqo57he/KkFL/UMCXj2UjKrxRfxcBshlDgMuWpnEAzewIupyG41bREBq8/pTkMvbAfQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBAPR04MB7399
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, May 31, 2023 at 03:24:46PM -0500, Bjorn Helgaas wrote:
> I guess I should have asked "what bad things happen without this patch
> and without the DT 'disabled' status"?

Well, now that you put it this way, I do realize that things are not so
ideal for me.

Our drivers for the functions of this device were already checking for
of_device_is_available() during probe. So, reverting the core PCIe
patch, they would still not register a network interface, which is good.

However (and this is the bad part), multiple functions of this PCIe
device unfortunately share a common memory, which is not zeroized by
hardware, and so, to avoid multi-bit ECC errors, it must be zeroized by
software, using some memory space accesses from all functions that have
access to that shared memory (every function zeroizes its piece of it).
This, sadly, includes functions which have status = "disabled". See
commit 3222b5b613db ("net: enetc: initialize RFS/RSS memories for unused
ports too").

What we used to do was start probing a bit in enetc_pf_probe(), enable
the memory space, zeroize our part of the shared memory, then check
of_device_is_available() and finally, we disable the memory space again
and exit probing with -ENODEV.

That is not possible anymore with the core patch, because the PCIe core
will not probe our disabled functions at all anymore.

The ENETC is not a hot-pluggable PCIe device. It uses Enhanced Allocation
to essentially describe on-chip memory spaces, which are always present.
So presumably, a different system-level solution to initialize those
shared memories (U-Boot?) may be chosen, if implementing this workaround
in Linux puts too much pressure on the PCIe core and the way in which it
does things. Initially I didn't want to do this in prior boot stages
because we only enable the RCEC in Linux, nothing is broken other than
the spurious AER messages, and, you know.. the kernel may still run
indefinitely on top of bootloaders which don't have the workaround applied.
So working around it in Linux avoids one dependency.