Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\))
Subject: Re: [PATCH] xprtrdma: Make sure Send CQ is allocated on an existing
 CPU
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <1dd8b5a5-4ce7-a53e-9510-bd122039bcfb@suse.de>
Date:   Wed, 23 Jan 2019 12:30:42 -0500
Cc:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        linux-rdma <linux-rdma@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <7D9897D4-F7E6-46A2-BA50-BB2D5220F162@oracle.com>
References: <3f5da1e4-99f1-8376-a291-e50a7d52c303@suse.com>
 <BF1A6DC0-FF34-4B80-8602-6B122AD5ABFD@oracle.com>
 <18bbe249-7d03-2475-d943-2f7d386a3797@suse.de>
 <1dd8b5a5-4ce7-a53e-9510-bd122039bcfb@suse.de>
To:     Nicolas Morey-Chaisemartin <nmoreychaisemartin@suse.de>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk


> On Jan 23, 2019, at 12:07 PM, Nicolas Morey-Chaisemartin =
<nmoreychaisemartin@suse.de> wrote:
>=20
>=20
>=20
> On 1/23/19 6:06 PM, Nicolas Morey-Chaisemartin wrote:
>>=20
>> On 1/23/19 5:51 PM, Chuck Lever wrote:
>>> Hi Nicolas-
>>>=20
>>>> On Jan 23, 2019, at 8:12 AM, Nicolas Morey-Chaisemartin =
<nmoreychaisemartin@suse.com> wrote:
>>>>=20
>>>> Make sure host has at least 2 CPU before allocating to CPU#1
>>> The fourth parameter of ib_alloc_cq() is not a CPU number,
>>> it's a completion vector number. What failure did you see
>>> that prompted this patch?
>> When trying to mount, I get this:
>> + mount -o rdma,port=3D20049 192.168.20.15:/tmp/RAM /tmp/RAM
>> mount.nfs: mounting 192.168.20.15:/tmp/RAM failed, reason given by =
server: No such file or directory
>>=20
>> Digging a bit into the code, it appears that the cq allocation here =
returns a ENOENT which come from mlx5_vector2eqn.
>> On my system (VM with a mlx5 card with SRIOV), the comp_eqs_list only =
contains one entry with index =3D=3D 0
>>=20
>> Nicolas
>>=20
>=20
> Also, adding a 2nd core to my VM fixes the issue (thus my =
understanding that it was a CPU number)

Fair enough. The 2nd CPU adds a 2nd compvec. Instead of
num_cpus_online() you want ib_device::num_comp_vectors.

I suspect there's a spiffier way to go about this these
days thanks to ib_get_vector_affinity, but you've found
a longstanding bug. So let's get something that can be
comfortably backported to stable.


--
Chuck Lever