Received-SPF: pass (google.com: domain of linux-kernel+bounces-70403-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
From: Minda Chen <minda.chen@starfivetech.com>
To: Conor Dooley <conor@kernel.org>, =?iso-8859-2?Q?Krzysztof_Wilczy=F1ski?=
	<kw@linux.com>, Rob Herring <robh+dt@kernel.org>, Bjorn Helgaas
	<bhelgaas@google.com>, Lorenzo Pieralisi <lpieralisi@kernel.org>, Daire
 McNamara <daire.mcnamara@microchip.com>, Emil Renner Berthing
	<emil.renner.berthing@canonical.com>, Krzysztof Kozlowski
	<krzysztof.kozlowski+dt@linaro.org>, Kevin Hilman <khilman@baylibre.com>
CC: "devicetree@vger.kernel.org" <devicetree@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-riscv@lists.infradead.org" <linux-riscv@lists.infradead.org>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>, Paul Walmsley
	<paul.walmsley@sifive.com>, Palmer Dabbelt <palmer@dabbelt.com>, Albert Ou
	<aou@eecs.berkeley.edu>, Philipp Zabel <p.zabel@pengutronix.de>, Mason Huo
	<mason.huo@starfivetech.com>, Kevin Xie <kevin.xie@starfivetech.com>
Subject: Re: [PATCH v15 22/23] PCI: starfive: Offload the NVMe timeout
 workaround to host drivers.
Thread-Topic: [PATCH v15 22/23] PCI: starfive: Offload the NVMe timeout
 workaround to host drivers.
Thread-Index: AQHaYlP4rHGw8vZ4EUmOCs3FPdp69rEP5FTw
Date: Sun, 18 Feb 2024 10:23:20 +0000
Message-ID:
 <SHXPR01MB0863B1FE2434D02FC221FF73E652A@SHXPR01MB0863.CHNPR01.prod.partner.outlook.cn>
References: <20240218101921.113528-1-minda.chen@starfivetech.com>
 <20240218101921.113528-7-minda.chen@starfivetech.com>
In-Reply-To: <20240218101921.113528-7-minda.chen@starfivetech.com>
Accept-Language: en-US
Content-Language: zh-CN
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?iso-8859-2?Q?WtwyofcqpL7czQU4qlGOj1oHFgdt1rxH37HuvX1Tw24R5w8wqrsphx9Bnt?=
 =?iso-8859-2?Q?9Eq0y2jNMJhtrmlnF4fFO0bYIMK2mXSk09x0/3aA3BwGVXu8iotn3rUMub?=
 =?iso-8859-2?Q?z+VnCVyRzaHDZNaBxtHE4oTFyjXcVS+EogmYoZLIMjCda2sdSr35dMVfwm?=
 =?iso-8859-2?Q?ySWfsUvFiMRnUwV+WyHKgmAnJf1Sap6ApqGTubU+TA+BLU9dCAfXhOLZyH?=
 =?iso-8859-2?Q?631WzmAd/ydWWFzF0hiLI+PQs/GaPUZEeMJSq2oGVuHAaiQa9Q6CXjjFXM?=
 =?iso-8859-2?Q?3xHztmme4PMr1ZKJHSpu55ZQlTcV/0sfShk4T20Oi0PXWTPZde+WEKbRAl?=
 =?iso-8859-2?Q?akfPsIx1WLbCC5+sJme5s3WDQG/4k+FTNY5eZB9marMS+X7IuwyIp67n5H?=
 =?iso-8859-2?Q?dJOEzNQ2vjk1v2iLHBAvm15G5XcmvrymJHtFT57aYXOElcvtHlhqcUZEC5?=
 =?iso-8859-2?Q?Cojhpam8cCX5cgN5iUcn2c8GV4VstwwtHoWsxDr7nNLlY47H2o+9BYdfvJ?=
 =?iso-8859-2?Q?oRS1ItWxi3HI2TdEpnLfxcx+Xmvo3Lq0A2m4szMICb63H33iB4UmsBrut9?=
 =?iso-8859-2?Q?+jZQ01pTPbF0ieAM292FgQw+CquVQsW4w+/0xsSAZdELilDWgkz8N0L+P7?=
 =?iso-8859-2?Q?jXUPVzNhdgLy0a+Bjb/Sx0QnjXyKygL9OqSILweWc4cXvCEWihHrCslTsJ?=
 =?iso-8859-2?Q?W6SJ2FTgswIhioeYXwJsnbMu2x/fP4pieIxeFi4Hq7jWD69aG9fbuVeAF6?=
 =?iso-8859-2?Q?kJEdVS6bHw7A5D8Ls+YwUaQDCXaTjSgWV0LMU7juqtADiU7Ar32O1mUnNq?=
 =?iso-8859-2?Q?7FVoBBVz52g2Ogbnujv+JXChDzAXpKidYqdC9mxQ22oSoaK99mpmzsMke/?=
 =?iso-8859-2?Q?M9pWyvF1O8GutaJH+4i49Fl17TjdNNzpx17VvaU7VC9VDJXncchGlE1gw+?=
 =?iso-8859-2?Q?xBlbJ5RfiWzoz1LB2oqN9cKAxIv0lo3/pvn21cuqfYR47ONYOrVUuG3P8T?=
 =?iso-8859-2?Q?6GQcD6saIRjTqBoHxuq6/3mo5CRZQztHn69fdFCYPOvqilgAOeiu0OtpKL?=
 =?iso-8859-2?Q?AylK1X/QTL6ANR4k/EYWXQReYkD+Z9jSMCVgietchfvUhjhHsmsNfco+Gz?=
 =?iso-8859-2?Q?qiZQBo2uJFRzQUJm35AoJeEw83kfr6QY9bCIvfa4mVGUV9iv87mpE1HI9q?=
 =?iso-8859-2?Q?7h30aU9bP9mY22rrnqS7RZE2jZx3aBDs54nNpidJm53UUtYDLgrYkMD7Nd?=
 =?iso-8859-2?Q?4T+m50wy/P8hCk4tB9hk/RXsDfODLkld9ZaeSBPcBxRgS1KjrfRpzktmZg?=
 =?iso-8859-2?Q?uVUtD8J39KpGNsYmbttSD7yOGWlXY1DJdusIohRXiVx1qhag9M33vik4PA?=
 =?iso-8859-2?Q?qvcBa+J97fOV0mcCevW7QR17aLAfY+vY9O3NVSkCmdjQuoNjOc7vJ91LRX?=
 =?iso-8859-2?Q?uqbrNV4WTYGiJK9lwHn4qvqRdpS5zI/gPs3JLs8twJAvq0VGCmzlOI7W78?=
 =?iso-8859-2?Q?yJRCtfTAGaPmZ9QiKS4RObSseiyAnHp+gAkrLt6+kOcMJlLneG8T1CnXga?=
 =?iso-8859-2?Q?OI0G0yp7MaLjvFaZMBHUeRD76ZH2MkI3+dptrWiixw0nhdAPTmhbGqIvnD?=
 =?iso-8859-2?Q?mWp3psZRcCp7Do3d78ZxHYoQcX+BfkMSbUH1l26bOtPRMR6wZJau7zRg?=
 =?iso-8859-2?Q?=3D=3D?=
Content-Type: text/plain; charset="iso-8859-2"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-OriginatorOrg: starfivetech.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SHXPR01MB0863.CHNPR01.prod.partner.outlook.cn
X-MS-Exchange-CrossTenant-Network-Message-Id: 2a26c880-8617-4841-e51c-08dc306ba164
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Feb 2024 10:23:20.6638
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 06fe3fa3-1221-43d3-861b-5a4ee687a85c
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: cKSAaozEglzH42xwzucXHBr2bxtVJxt0t0z+5TdsumeO9BZnx7e7vxBo3uHhjSaaXNJb2tzoLQZO+9i5ZZmmI1fAHMYnYL9kKXOfCyQzyq4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SHXPR01MB0464


>=20
> From: Kevin Xie <kevin.xie@starfivetech.com>
>=20
> As the Starfive JH7110 hardware can't keep two inbound post write in orde=
r all
> the time, such as MSI messages and NVMe completions. If the NVMe
> completion update later than the MSI, an NVMe IRQ handle will miss.
>=20
> As a workaround, we will wait a while before going to the generic handle =
here.
>=20
> Verified with NVMe SSD, USB SSD, R8169 NIC.
> The performance are stable and even higher after this patch.
>=20
> Signed-off-by: Kevin Xie <kevin.xie@starfivetech.com>
> Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
> ---
>  drivers/pci/controller/plda/pcie-plda-host.c | 12 ++++++++++++
>  drivers/pci/controller/plda/pcie-plda.h      |  1 +
>  drivers/pci/controller/plda/pcie-starfive.c  |  1 +
>  3 files changed, 14 insertions(+)
>=20
> diff --git a/drivers/pci/controller/plda/pcie-plda-host.c
> b/drivers/pci/controller/plda/pcie-plda-host.c
> index a18923d7cea6..9e077ddf45c0 100644
> --- a/drivers/pci/controller/plda/pcie-plda-host.c
> +++ b/drivers/pci/controller/plda/pcie-plda-host.c
> @@ -13,6 +13,7 @@
>  #include <linux/msi.h>
>  #include <linux/pci_regs.h>
>  #include <linux/pci-ecam.h>
> +#include <linux/delay.h>
>=20
>  #include "pcie-plda.h"
>=20
> @@ -44,6 +45,17 @@ static void plda_handle_msi(struct irq_desc *desc)
>  			       bridge_base_addr + ISTATUS_LOCAL);
>  		status =3D readl_relaxed(bridge_base_addr + ISTATUS_MSI);
>  		for_each_set_bit(bit, &status, msi->num_vectors) {
> +			/*
> +			 * As the Starfive JH7110 hardware can't keep two
> +			 * inbound post write in order all the time, such as
> +			 * MSI messages and NVMe completions.
> +			 * If the NVMe completion update later than the MSI,
> +			 * an NVMe IRQ handle will miss.
> +			 * As a workaround, we will wait a while before
> +			 * going to the generic handle here.
> +			 */
> +			if (port->msi_quirk_delay_us)
> +				udelay(port->msi_quirk_delay_us);
>  			ret =3D generic_handle_domain_irq(msi->dev_domain, bit);
>  			if (ret)
>  				dev_err_ratelimited(dev, "bad MSI IRQ %d\n", diff --git
> a/drivers/pci/controller/plda/pcie-plda.h
> b/drivers/pci/controller/plda/pcie-plda.h
> index 04e385758a2f..feccf285dfe8 100644
> --- a/drivers/pci/controller/plda/pcie-plda.h
> +++ b/drivers/pci/controller/plda/pcie-plda.h
> @@ -186,6 +186,7 @@ struct plda_pcie_rp {
>  	int msi_irq;
>  	int intx_irq;
>  	int num_events;
> +	u16 msi_quirk_delay_us;
>  };
>=20
>  struct plda_event {
> diff --git a/drivers/pci/controller/plda/pcie-starfive.c
> b/drivers/pci/controller/plda/pcie-starfive.c
> index 9bb9f0e29565..5cfc30572b7f 100644
> --- a/drivers/pci/controller/plda/pcie-starfive.c
> +++ b/drivers/pci/controller/plda/pcie-starfive.c
> @@ -391,6 +391,7 @@ static int starfive_pcie_probe(struct platform_device
> *pdev)
>=20
>  	plda->host_ops =3D &sf_host_ops;
>  	plda->num_events =3D PLDA_MAX_EVENT_NUM;
> +	plda->msi_quirk_delay_us =3D 1;
>  	/* mask doorbell event */
>  	plda->events_bitmap =3D GENMASK(PLDA_INT_EVENT_NUM - 1, 0)
>  			     & ~BIT(PLDA_AXI_DOORBELL)
> --
> 2.17.1

Hi Kevin Hilman
   Will you try this series patch? Maybe this patch can fix the NVMe timeou=
t print issue.
If this patch can fix it, please notify us. Thanks