Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Benjamin Coddington" <bcodding@redhat.com>
To:     "Trond Myklebust" <trondmy@hammerspace.com>
Cc:     linux-nfs@vger.kernel.org
Subject: Re: client call_decode WARN_ON memcmp race since 72691a269f0b
Date:   Wed, 31 Aug 2022 18:58:56 -0400
Message-ID: <7A73C3D7-BB74-4DEA-B8D1-F337FB68C689@redhat.com>
In-Reply-To: <14AB2A51-CF42-4CA8-9627-7837BD0D584A@redhat.com>
References: <6806AB48-F7D6-4639-8D03-E31BA25C4CF3@redhat.com>
 <24350372a2a5cb244bc843faef569404088f9278.camel@hammerspace.com>
 <14AB2A51-CF42-4CA8-9627-7837BD0D584A@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 31 Aug 2022, at 18:44, Benjamin Coddington wrote:

> On 31 Aug 2022, at 17:34, Trond Myklebust wrote:
>
>> On Wed, 2022-08-31 at 16:20 -0400, Benjamin Coddington wrote:
>>> Hey Trond,
>>>
>>> Since "72691a269f0b SUNRPC: Don't reuse bvec on retransmission of the
>>> request" I can sometimes pop the WARN_ON() in call_decode(), usually
>>> on
>>> generic/642.
>>>
>>> I think there's a kworker halfway through
>>>
>>> xprt_complete_rqst() ->
>>>        xprt_request_dequeue_receive_locked()
>>>
>>> between these two linse:
>>>          xdr_free_bvec(&req->rq_rcv_buf);
>>>          req->rq_private_buf.bvec = NULL;
>>>
>>> And at the same time the task is doing the WARN_ON(memcmp()) in
>>> call_decode.
>>>
>>> I'm not sure of a good fix - perhaps we can fixup the other paths
>>> that
>>> require the NULL check in xdr_alloc_bvec() so we don't have to set
>>> bvec =
>>> NULL.
>>>
>>> Any thoughts?
>>>
>>
>> How about this?
>
> I think that will fix it.  I'll test it overnight and see what happens.

On a second look, I'm not sure this will fix it, because I don't think its a
memory-barrier problem, rather its that setting the two buffer's bvec = NULL
is not atomic.

Meanwhile, I am testing your patch.

Ben