In architecture like powerpc, we can have cpus without any local memory
attached to it. In such cases the node does not have real memory.
Use local_memory_node(), which is guaranteed to have memory.
local_memory_node is a noop in other architectures that does not support
memoryless nodes.
Signed-off-by: Xianting Tian <[email protected]>
---
drivers/infiniband/hw/hfi1/file_ops.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c
index 8ca51e43c..79fa22cc7 100644
--- a/drivers/infiniband/hw/hfi1/file_ops.c
+++ b/drivers/infiniband/hw/hfi1/file_ops.c
@@ -965,7 +965,7 @@ static int allocate_ctxt(struct hfi1_filedata *fd, struct hfi1_devdata *dd,
*/
fd->rec_cpu_num = hfi1_get_proc_affinity(dd->node);
if (fd->rec_cpu_num != -1)
- numa = cpu_to_node(fd->rec_cpu_num);
+ numa = local_memory_node(cpu_to_node(fd->rec_cpu_num));
else
numa = numa_node_id();
ret = hfi1_create_ctxtdata(dd->pport, numa, &uctxt);
--
2.17.1
On 10/10/2020 4:57 AM, Xianting Tian wrote:
> In architecture like powerpc, we can have cpus without any local memory
> attached to it. In such cases the node does not have real memory.
>
> Use local_memory_node(), which is guaranteed to have memory.
> local_memory_node is a noop in other architectures that does not support
> memoryless nodes.
>
> Signed-off-by: Xianting Tian <[email protected]>
> ---
> drivers/infiniband/hw/hfi1/file_ops.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c
> index 8ca51e43c..79fa22cc7 100644
> --- a/drivers/infiniband/hw/hfi1/file_ops.c
> +++ b/drivers/infiniband/hw/hfi1/file_ops.c
> @@ -965,7 +965,7 @@ static int allocate_ctxt(struct hfi1_filedata *fd, struct hfi1_devdata *dd,
> */
> fd->rec_cpu_num = hfi1_get_proc_affinity(dd->node);
> if (fd->rec_cpu_num != -1)
> - numa = cpu_to_node(fd->rec_cpu_num);
> + numa = local_memory_node(cpu_to_node(fd->rec_cpu_num));
> else
> numa = numa_node_id();
> ret = hfi1_create_ctxtdata(dd->pport, numa, &uctxt);
>
The hfi1 driver depends on X86_64. I'm not sure what this patch buys,
can you expand a bit?
-Denny
Hi Dennis
Thanks for the comments
If it depends on x86_64, I think this issue doesn't exist.
Sorry to disturb you.
-----Original Message-----
From: Dennis Dalessandro [mailto:[email protected]]
Sent: Monday, October 12, 2020 8:37 PM
To: tianxianting (RD) <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]
Subject: Re: [PATCH] IB/hfi1: Avoid allocing memory on memoryless numa node
On 10/10/2020 4:57 AM, Xianting Tian wrote:
> In architecture like powerpc, we can have cpus without any local
> memory attached to it. In such cases the node does not have real memory.
>
> Use local_memory_node(), which is guaranteed to have memory.
> local_memory_node is a noop in other architectures that does not
> support memoryless nodes.
>
> Signed-off-by: Xianting Tian <[email protected]>
> ---
> drivers/infiniband/hw/hfi1/file_ops.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/infiniband/hw/hfi1/file_ops.c
> b/drivers/infiniband/hw/hfi1/file_ops.c
> index 8ca51e43c..79fa22cc7 100644
> --- a/drivers/infiniband/hw/hfi1/file_ops.c
> +++ b/drivers/infiniband/hw/hfi1/file_ops.c
> @@ -965,7 +965,7 @@ static int allocate_ctxt(struct hfi1_filedata *fd, struct hfi1_devdata *dd,
> */
> fd->rec_cpu_num = hfi1_get_proc_affinity(dd->node);
> if (fd->rec_cpu_num != -1)
> - numa = cpu_to_node(fd->rec_cpu_num);
> + numa = local_memory_node(cpu_to_node(fd->rec_cpu_num));
> else
> numa = numa_node_id();
> ret = hfi1_create_ctxtdata(dd->pport, numa, &uctxt);
>
The hfi1 driver depends on X86_64. I'm not sure what this patch buys, can you expand a bit?
-Denny
On 10/16/2020 10:11 AM, Jason Gunthorpe wrote:
> On Mon, Oct 12, 2020 at 08:36:57AM -0400, Dennis Dalessandro wrote:
>> On 10/10/2020 4:57 AM, Xianting Tian wrote:
>>> In architecture like powerpc, we can have cpus without any local memory
>>> attached to it. In such cases the node does not have real memory.
>>>
>>> Use local_memory_node(), which is guaranteed to have memory.
>>> local_memory_node is a noop in other architectures that does not support
>>> memoryless nodes.
>>>
>>> Signed-off-by: Xianting Tian <[email protected]>
>>> drivers/infiniband/hw/hfi1/file_ops.c | 2 +-
>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c
>>> index 8ca51e43c..79fa22cc7 100644
>>> +++ b/drivers/infiniband/hw/hfi1/file_ops.c
>>> @@ -965,7 +965,7 @@ static int allocate_ctxt(struct hfi1_filedata *fd, struct hfi1_devdata *dd,
>>> */
>>> fd->rec_cpu_num = hfi1_get_proc_affinity(dd->node);
>>> if (fd->rec_cpu_num != -1)
>>> - numa = cpu_to_node(fd->rec_cpu_num);
>>> + numa = local_memory_node(cpu_to_node(fd->rec_cpu_num));
>>> else
>>> numa = numa_node_id();
>>> ret = hfi1_create_ctxtdata(dd->pport, numa, &uctxt);
>>>
>>
>> The hfi1 driver depends on X86_64. I'm not sure what this patch buys, can
>> you expand a bit?
>
> Yikes, that is strongly discouraged.
Hmm. This was never raised as an issue before. Regardless I can't recall
why we did this in the first place. I'll do some digging, try to jog my
memory.
-Denny
On Mon, Oct 12, 2020 at 08:36:57AM -0400, Dennis Dalessandro wrote:
> On 10/10/2020 4:57 AM, Xianting Tian wrote:
> > In architecture like powerpc, we can have cpus without any local memory
> > attached to it. In such cases the node does not have real memory.
> >
> > Use local_memory_node(), which is guaranteed to have memory.
> > local_memory_node is a noop in other architectures that does not support
> > memoryless nodes.
> >
> > Signed-off-by: Xianting Tian <[email protected]>
> > drivers/infiniband/hw/hfi1/file_ops.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c
> > index 8ca51e43c..79fa22cc7 100644
> > +++ b/drivers/infiniband/hw/hfi1/file_ops.c
> > @@ -965,7 +965,7 @@ static int allocate_ctxt(struct hfi1_filedata *fd, struct hfi1_devdata *dd,
> > */
> > fd->rec_cpu_num = hfi1_get_proc_affinity(dd->node);
> > if (fd->rec_cpu_num != -1)
> > - numa = cpu_to_node(fd->rec_cpu_num);
> > + numa = local_memory_node(cpu_to_node(fd->rec_cpu_num));
> > else
> > numa = numa_node_id();
> > ret = hfi1_create_ctxtdata(dd->pport, numa, &uctxt);
> >
>
> The hfi1 driver depends on X86_64. I'm not sure what this patch buys, can
> you expand a bit?
Yikes, that is strongly discouraged.
Jason