Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH v1 1/8] xprtrdma: Avoid Receive Queue wrapping
To:     Chuck Lever <chuck.lever@oracle.com>, linux-rdma@vger.kernel.org,
        linux-nfs@vger.kernel.org
References: <161721926778.515226.9805598788670386587.stgit@manet.1015granger.net>
 <161721936504.515226.14877637433211331378.stgit@manet.1015granger.net>
From:   Tom Talpey <tom@talpey.com>
Message-ID: <818b4215-d16a-532f-88cf-b65763f50341@talpey.com>
Date:   Wed, 31 Mar 2021 16:05:38 -0400
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101
 Thunderbird/68.12.1
MIME-Version: 1.0
In-Reply-To: <161721936504.515226.14877637433211331378.stgit@manet.1015granger.net>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 3/31/2021 3:36 PM, Chuck Lever wrote:
> Commit e340c2d6ef2a ("xprtrdma: Reduce the doorbell rate (Receive)")
> increased the number of Receive WRs that are posted by the client,
> but did not increase the size of the Receive Queue allocated during
> transport set-up.
> 
> This is usually not an issue because RPCRDMA_BACKWARD_WRS is defined
> as (32) when SUNRPC_BACKCHANNEL is defined. In cases where it isn't,
> there is a real risk of Receive Queue wrapping.
> 
> Fixes: e340c2d6ef2a ("xprtrdma: Reduce the doorbell rate (Receive)")
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
>   net/sunrpc/xprtrdma/frwr_ops.c |    1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c
> index 766a1048a48a..132df9b59ab4 100644
> --- a/net/sunrpc/xprtrdma/frwr_ops.c
> +++ b/net/sunrpc/xprtrdma/frwr_ops.c
> @@ -257,6 +257,7 @@ int frwr_query_device(struct rpcrdma_ep *ep, const struct ib_device *device)
>   	ep->re_attr.cap.max_send_wr += 1; /* for ib_drain_sq */
>   	ep->re_attr.cap.max_recv_wr = ep->re_max_requests;
>   	ep->re_attr.cap.max_recv_wr += RPCRDMA_BACKWARD_WRS;
> +	ep->re_attr.cap.max_recv_wr += RPCRDMA_MAX_RECV_BATCH;

Naively, it seems to me this should be max(BACKWARD, BATCH).
But, extra WR slots are cheap and safe.

Reviewed-By: Tom Talpey <tom@talpey.com>

>   	ep->re_attr.cap.max_recv_wr += 1; /* for ib_drain_rq */
>   
>   	ep->re_max_rdma_segs =
> 
> 
>