Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <a25455eac6a02eeb9710d9204dfe0b91938f61a1.camel@linux.ibm.com>
Subject: Re: [PATCH] net/mlx5: stop waiting for PCI link if reset is required
From:   Niklas Schnelle <schnelle@linux.ibm.com>
To:     Leon Romanovsky <leon@kernel.org>,
        Bjorn Helgaas <helgaas@kernel.org>
Cc:     Saeed Mahameed <saeedm@nvidia.com>,
        "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Gerd Bayer <gbayer@linux.ibm.com>,
        Alexander Schmidt <alexs@linux.ibm.com>,
        netdev@vger.kernel.org, linux-rdma@vger.kernel.org,
        linux-kernel@vger.kernel.org
Date:   Tue, 04 Apr 2023 17:27:35 +0200
In-Reply-To: <20230403182105.GC4514@unreal>
References: <20230403075657.168294-1-schnelle@linux.ibm.com>
         <20230403182105.GC4514@unreal>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.46.4 (3.46.4-1.fc37) 
MIME-Version: 1.0
Precedence: bulk

On Mon, 2023-04-03 at 21:21 +0300, Leon Romanovsky wrote:
> On Mon, Apr 03, 2023 at 09:56:56AM +0200, Niklas Schnelle wrote:
> > after an error on the PCI link, the driver does not need to wait
> > for the link to become functional again as a reset is required. Stop
> > the wait loop in this case to accelerate the recovery flow.
> >=20
> > Co-developed-by: Alexander Schmidt <alexs@linux.ibm.com>
> > Signed-off-by: Alexander Schmidt <alexs@linux.ibm.com>
> > Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
> > ---
> >  drivers/net/ethernet/mellanox/mlx5/core/health.c | 12 ++++++++++--
> >  1 file changed, 10 insertions(+), 2 deletions(-)
> >=20
> > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers=
/net/ethernet/mellanox/mlx5/core/health.c
> > index f9438d4e43ca..81ca44e0705a 100644
> > --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c
> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c
> > @@ -325,6 +325,8 @@ int mlx5_health_wait_pci_up(struct mlx5_core_dev *d=
ev)
> >  	while (sensor_pci_not_working(dev)) {
>=20
> According to the comment in sensor_pci_not_working(), this loop is
> supposed to wait till PCI will be ready again. Otherwise, already in
> first iteration, we will bail out with pci_channel_offline() error.
>=20
> Thanks

Well yes. The problem is that this works for intermittent errors
including when the card resets itself which seems to be the use case in
mlx5_fw_reset_complete_reload() and mlx5_devlink_reload_fw_activate().
If there is a PCI error that requires a link reset though we see some
problems though it does work after running into the timeout.

As I understand it and as implemented at least on s390,
pci_channel_io_frozen is only set for fatal errors that require a reset
while non fatal errors will have pci_channel_io_normal (see also
Documentation/PCI/pcieaer-howto.rst) thus I think pci_channel_offline()
should only be true if a reset is required or there is a permanent
error. Furthermore in the pci_channel_io_frozen state the PCI function
may be isolated and the reads will not reach the endpoint, this is the
case at least on s390.  Thus for errors requiring a reset the loop
without pci_channel_offline() will run until the reset is performed or
the timeout is reached. In the mlx5_health_try_recover() case during
error recovery we will then indeed always loop until timeout, because
the loop blocks mlx5_pci_err_detected() from returning thus blocking
the reset (see Documentation/PCI/pci-error-recovery.rst). Adding Bjorn,
maybe he can confirm or correct my assumptions here.

Thanks,
Niklas

>=20
> >  		if (time_after(jiffies, end))
> >  			return -ETIMEDOUT;
> > +		if (pci_channel_offline(dev->pdev))
> > +			return -EIO;
> >  		msleep(100);
> >  	}
> >  	return 0;
> > @@ -332,10 +334,16 @@ int mlx5_health_wait_pci_up(struct mlx5_core_dev =
*dev)
> > =20
> >  static int mlx5_health_try_recover(struct mlx5_core_dev *dev)
> >  {
> > +	int rc;
> > +
> >  	mlx5_core_warn(dev, "handling bad device here\n");
> >  	mlx5_handle_bad_state(dev);
> > -	if (mlx5_health_wait_pci_up(dev)) {
> > -		mlx5_core_err(dev, "health recovery flow aborted, PCI reads still no=
t working\n");
> > +	rc =3D mlx5_health_wait_pci_up(dev);
> > +	if (rc) {
> > +		if (rc =3D=3D -ETIMEDOUT)
> > +			mlx5_core_err(dev, "health recovery flow aborted, PCI reads still n=
ot working\n");
> > +		else
> > +			mlx5_core_err(dev, "health recovery flow aborted, PCI channel offli=
ne\n");
> >  		return -EIO;
> >  	}
> >  	mlx5_core_err(dev, "starting health recovery flow\n");
> >=20
> > base-commit: 7e364e56293bb98cae1b55fd835f5991c4e96e7d
> > --=20
> > 2.37.2
> >=20