Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
IronPort-SDR: 2DizwvBUhuWmJ8EyQBSdx4z8c408FNlxtJmDQLPXlQOLaYn7eYhD4G1Q+Ojd3eRzW2bDaVJKlB
 ykZ/Bq+T7IaQ==
IronPort-SDR: FIF5WfCWETsAWReYXNH+IIVmcS0AuT+c154fIAwTkaCjCiLGCgs3vaP9O/ffQx6b5mKKrZqUTJ
 Afvm+IYQUmPw==
Subject: Re: [PATCH] IB/hfi1: Avoid allocing memory on memoryless numa node
To:     Jason Gunthorpe <jgg@ziepe.ca>
Cc:     Xianting Tian <tian.xianting@h3c.com>,
        mike.marciniszyn@cornelisnetworks.com, dledford@redhat.com,
        linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org
References: <20201010085732.20708-1-tian.xianting@h3c.com>
 <9ba33073-044c-9da6-a90d-4626e6441793@cornelisnetworks.com>
 <20201016141152.GC36674@ziepe.ca>
From:   Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
Message-ID: <c69d2429-25cc-75dc-d4de-3ee66e9aaa08@cornelisnetworks.com>
Date:   Fri, 16 Oct 2020 11:14:09 -0400
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
 Thunderbird/68.12.1
MIME-Version: 1.0
In-Reply-To: <20201016141152.GC36674@ziepe.ca>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 10/16/2020 10:11 AM, Jason Gunthorpe wrote:
> On Mon, Oct 12, 2020 at 08:36:57AM -0400, Dennis Dalessandro wrote:
>> On 10/10/2020 4:57 AM, Xianting Tian wrote:
>>> In architecture like powerpc, we can have cpus without any local memory
>>> attached to it. In such cases the node does not have real memory.
>>>
>>> Use local_memory_node(), which is guaranteed to have memory.
>>> local_memory_node is a noop in other architectures that does not support
>>> memoryless nodes.
>>>
>>> Signed-off-by: Xianting Tian <tian.xianting@h3c.com>
>>>    drivers/infiniband/hw/hfi1/file_ops.c | 2 +-
>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c
>>> index 8ca51e43c..79fa22cc7 100644
>>> +++ b/drivers/infiniband/hw/hfi1/file_ops.c
>>> @@ -965,7 +965,7 @@ static int allocate_ctxt(struct hfi1_filedata *fd, struct hfi1_devdata *dd,
>>>    	 */
>>>    	fd->rec_cpu_num = hfi1_get_proc_affinity(dd->node);
>>>    	if (fd->rec_cpu_num != -1)
>>> -		numa = cpu_to_node(fd->rec_cpu_num);
>>> +		numa = local_memory_node(cpu_to_node(fd->rec_cpu_num));
>>>    	else
>>>    		numa = numa_node_id();
>>>    	ret = hfi1_create_ctxtdata(dd->pport, numa, &uctxt);
>>>
>>
>> The hfi1 driver depends on X86_64. I'm not sure what this patch buys, can
>> you expand a bit?
> 
> Yikes, that is strongly discouraged.

Hmm. This was never raised as an issue before. Regardless I can't recall 
why we did this in the first place. I'll do some digging, try to jog my 
memory.

-Denny