Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <20210715080822.14575-1-justin.he@arm.com> <CAJ2QiJLWgxw0X8rkMhKgAGgiFS5xhrhMF5Dct_J791Kt-ys7QQ@mail.gmail.com>
 <AM6PR08MB4376894A46B47B024F50FBB3F7E19@AM6PR08MB4376.eurprd08.prod.outlook.com>
In-Reply-To: <AM6PR08MB4376894A46B47B024F50FBB3F7E19@AM6PR08MB4376.eurprd08.prod.outlook.com>
From:   Prabhakar Kushwaha <prabhakar.pkin@gmail.com>
Date:   Mon, 19 Jul 2021 20:20:54 +0530
Message-ID: <CAJ2QiJJ8=jkbRVscnXM2m_n2RX2pNdJG4iA3tYiNGDYefb-hjA@mail.gmail.com>
Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
To:     Justin He <Justin.He@arm.com>
Cc:     Ariel Elior <aelior@marvell.com>,
        "GR-everest-linux-l2@marvell.com" <GR-everest-linux-l2@marvell.com>,
        "David S. Miller" <davem@davemloft.net>,
        Jakub Kicinski <kuba@kernel.org>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        nd <nd@arm.com>, Shai Malin <malin1024@gmail.com>,
        Shai Malin <smalin@marvell.com>,
        Prabhakar Kushwaha <pkushwaha@marvell.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

Hi Justin,

On Mon, Jul 19, 2021 at 6:47 PM Justin He <Justin.He@arm.com> wrote:
>
> Hi Prabhakar
>
> > -----Original Message-----
> > From: Prabhakar Kushwaha <prabhakar.pkin@gmail.com>
> > Sent: Monday, July 19, 2021 6:36 PM
> > To: Justin He <Justin.He@arm.com>
> > Cc: Ariel Elior <aelior@marvell.com>; GR-everest-linux-l2@marvell.com;
> > David S. Miller <davem@davemloft.net>; Jakub Kicinski <kuba@kernel.org>;
> > netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
> > kernel@vger.kernel.org>; nd <nd@arm.com>; Shai Malin <malin1024@gmail.com>;
> > Shai Malin <smalin@marvell.com>; Prabhakar Kushwaha <pkushwaha@marvell.com>
> > Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
> > _qed_mcp_cmd_and_union()
> >
> > Hi Jia,
> >
> > On Thu, Jul 15, 2021 at 2:28 PM Jia He <justin.he@arm.com> wrote:
> > >
> > > Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
> > > QL41000 ethernet controller:
> > >  BUG: scheduling while atomic: kworker/0:4/531/0x00000200
> > >   [qed_probe:488()]hw prepare failed
> > >   kernel BUG at mm/vmalloc.c:2355!
> > >   Internal error: Oops - BUG: 0 [#1] SMP
> > >   CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-
> > Ubuntu
> > >   pstate: 00400009 (nzcv daif +PAN -UAO)
> > >  Call trace:
> > >   vunmap+0x4c/0x50
> > >   iounmap+0x48/0x58
> > >   qed_free_pci+0x60/0x80 [qed]
> > >   qed_probe+0x35c/0x688 [qed]
> > >   __qede_probe+0x88/0x5c8 [qede]
> > >   qede_probe+0x60/0xe0 [qede]
> > >   local_pci_probe+0x48/0xa0
> > >   work_for_cpu_fn+0x24/0x38
> > >   process_one_work+0x1d0/0x468
> > >   worker_thread+0x238/0x4e0
> > >   kthread+0xf0/0x118
> > >   ret_from_fork+0x10/0x18
> > >
> > > In this case, qed_hw_prepare() returns error due to hw/fw error, but in
> > > theory work queue should be in process context instead of interrupt.
> > >
> > > The root cause might be the unpaired spin_{un}lock_bh() in
> > > _qed_mcp_cmd_and_union(), which causes botton half is disabled
> > incorrectly.
> > >
> > > Reported-by: Lijian Zhang <Lijian.Zhang@arm.com>
> > > Signed-off-by: Jia He <justin.he@arm.com>
> > > ---
> >
> > This patch is adding additional spin_{un}lock_bh().
> > Can you please enlighten about the exact flow causing this unpaired
> > spin_{un}lock_bh.
> >
> For instance:
> _qed_mcp_cmd_and_union()
>   In while loop
>     spin_lock_bh()
>     qed_mcp_has_pending_cmd() (assume false), will break the loop

I agree till here.

>   if (cnt >= max_retries) {
> ...
>     return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
>   }
>

Because of break, cnt has not been increased.
   - cnt is still less than max_retries.
  - if (cnt >= max_retries) will not be *true*, leading to spin_unlock_bh().
Hence pairing completed.

I am not seeing any issue here.

> > Also,
> > as per description, looks like you are not sure actual the root-cause.
> > does this patch really solved the problem?
>
> I don't have that ThunderX2 to verify the patch.
> But I searched all the spin_lock/unlock_bh and spin_lock_irqsave/irqrestore
> under driver/.../qlogic, this is the only problematic point I could figure
> out. And this might be possible code path of qed_probe().
>

Without testing and proper root-cause,  it is tough to accept the suggested fix.

--pk