Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Sun, 9 Apr 2023 11:54:25 +0300
From:   Leon Romanovsky <leon@kernel.org>
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     Niklas Schnelle <schnelle@linux.ibm.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Gerd Bayer <gbayer@linux.ibm.com>,
        Alexander Schmidt <alexs@linux.ibm.com>,
        netdev@vger.kernel.org, linux-rdma@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH] net/mlx5: stop waiting for PCI link if reset is required
Message-ID: <20230409085425.GC14869@unreal>
References: <a25455eac6a02eeb9710d9204dfe0b91938f61a1.camel@linux.ibm.com>
 <20230405210613.GA3638573@bhelgaas>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230405210613.GA3638573@bhelgaas>
Precedence: bulk

On Wed, Apr 05, 2023 at 04:06:13PM -0500, Bjorn Helgaas wrote:
> On Tue, Apr 04, 2023 at 05:27:35PM +0200, Niklas Schnelle wrote:
> > On Mon, 2023-04-03 at 21:21 +0300, Leon Romanovsky wrote:
> > > On Mon, Apr 03, 2023 at 09:56:56AM +0200, Niklas Schnelle wrote:
> > > > after an error on the PCI link, the driver does not need to wait
> > > > for the link to become functional again as a reset is required. Stop
> > > > the wait loop in this case to accelerate the recovery flow.
> > > > 
> > > > Co-developed-by: Alexander Schmidt <alexs@linux.ibm.com>
> > > > Signed-off-by: Alexander Schmidt <alexs@linux.ibm.com>
> > > > Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
> > > > ---
> > > >  drivers/net/ethernet/mellanox/mlx5/core/health.c | 12 ++++++++++--
> > > >  1 file changed, 10 insertions(+), 2 deletions(-)
> > > > 
> > > > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c
> > > > index f9438d4e43ca..81ca44e0705a 100644
> > > > --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c
> > > > +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c
> > > > @@ -325,6 +325,8 @@ int mlx5_health_wait_pci_up(struct mlx5_core_dev *dev)
> > > >  	while (sensor_pci_not_working(dev)) {
> > > 
> > > According to the comment in sensor_pci_not_working(), this loop is
> > > supposed to wait till PCI will be ready again. Otherwise, already in
> > > first iteration, we will bail out with pci_channel_offline() error.
> > 
> > Well yes. The problem is that this works for intermittent errors
> > including when the card resets itself which seems to be the use case in
> > mlx5_fw_reset_complete_reload() and mlx5_devlink_reload_fw_activate().
> > If there is a PCI error that requires a link reset though we see some
> > problems though it does work after running into the timeout.
> > 
> > As I understand it and as implemented at least on s390,
> > pci_channel_io_frozen is only set for fatal errors that require a reset
> > while non fatal errors will have pci_channel_io_normal (see also
> > Documentation/PCI/pcieaer-howto.rst)
> 
> Yes, I think that's true, see handle_error_source().
> 
> > thus I think pci_channel_offline()
> > should only be true if a reset is required or there is a permanent
> > error.
> 
> Yes, I think pci_channel_offline() will only be true when a fatal
> error has been reported via AER or DPC (or a hotplug driver says the
> device has been removed).  The driver resetting the device should not
> cause such a fatal error.

Thank you for an explanation and confirmation.