Subject: Re: [PATCH v2] xprtrdma: Make sure Send CQ is allocated on an
 existing compvec
To:     Nicolas Morey-Chaisemartin <nmoreychaisemartin@suse.com>,
        linux-rdma@vger.kernel.org
Cc:     linux-nfs@vger.kernel.org, chuck.lever@oracle.com
References: <164839c0-399e-d01b-dafc-5c873e650bdb@suse.com>
From:   Tom Talpey <tom@talpey.com>
Message-ID: <0e2e8737-9ec7-f3b5-ff0c-f178330b826c@talpey.com>
Date:   Thu, 24 Jan 2019 08:04:23 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.4.0
MIME-Version: 1.0
In-Reply-To: <164839c0-399e-d01b-dafc-5c873e650bdb@suse.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

On 1/23/2019 11:02 PM, Nicolas Morey-Chaisemartin wrote:
> Make sure the device has at least 2 completion vectors before allocating to compvec#1

I agree this fixes the bug, but wouldn't it be better to attempt
to steer the affinity to chosen cores, rather than unconditionally
using static values? At least make some determination that the
device, and the memory that was just allocated, were pointed to
slightly optimal NUMA node to handle its completions.

Does the verbs API not support this? I thought that was discussed
a while back.

Tom.

> Fixes: a4699f5647f3 (xprtrdma: Put Send CQ in IB_POLL_WORKQUEUE mode)
> Signed-off-by: Nicolas Morey-Chaisemartin <nmoreychaisemartin@suse.com>
> ---
> Fixes since v1:
>   - Use num_comp_vector instead online_cpus
> 
>   net/sunrpc/xprtrdma/verbs.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index b725911c0f3f..db913bcef984 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -546,7 +546,7 @@ rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia,
>   
>   	sendcq = ib_alloc_cq(ia->ri_device, NULL,
>   			     ep->rep_attr.cap.max_send_wr + 1,
> -			     1, IB_POLL_WORKQUEUE);
> +			     ia->ri_device->num_comp_vectors > 1 ? 1 : 0, IB_POLL_WORKQUEUE);
>   	if (IS_ERR(sendcq)) {
>   		rc = PTR_ERR(sendcq);
>   		dprintk("RPC:       %s: failed to create send CQ: %i\n",
>