Hello,
Recently I noticed the mlx5_alloc_priv_descs() do many allocations from
kmalloc-8k. It's called from following chain:
cma_cm_event_handler => nvme_rdma_cm_handler => nvme_rdma_create_queue_ib
=> ib_mr_pool_init => ib_alloc_mr => __mlx5_ib_alloc_mr
=> _mlx5_alloc_mkey_descs
When the nvme-rdma asks to allocate memory regions that support maximum 257
sg entries, the real MR memory allocation size is slightly bigger than 2048.
The MR memory allocation seems adding "add_size" to guarantee that there
is enough space to find the memory aligned to 2048 (MLX5_UMR_ALIGN).
The requested size is bigger than 4096 after adding the extra bytes.
Then kmalloc uses 8192 as the actual allocation size.
For power-of-two sizes, the alignment can be guaranteed by kmalloc()
according to commit 59bb47985c1d ("mm, sl[aou]b: guarantee natural
alignment for kmalloc(power-of-two)").
Here comes a draft patch that uses the lesser of existing requested
size or the next power-of-two size.
I'm not sure if it's better to simply allocate roundup_pow_of_two(),
or if it's true to assume that slab will not have intermediate size
between 2 power-of-two sizes. Also not sure if a generic memalign()
will be the better approach.
Can you please let me know what's the preferred fix, or fix it directly
to let the allocation use the smaller slab if possible? Thanks
Yuanyuan Zhong (1):
RDMA/mlx5: align MR mem allocation size to power-of-two
drivers/infiniband/hw/mlx5/mr.c | 5 +++++
1 file changed, 5 insertions(+)
--
2.34.1
On Thu, 29 Jun 2023 15:32:47 -0600, Yuanyuan Zhong wrote:
> Recently I noticed the mlx5_alloc_priv_descs() do many allocations from
> kmalloc-8k. It's called from following chain:
> cma_cm_event_handler => nvme_rdma_cm_handler => nvme_rdma_create_queue_ib
> => ib_mr_pool_init => ib_alloc_mr => __mlx5_ib_alloc_mr
> => _mlx5_alloc_mkey_descs
>
> When the nvme-rdma asks to allocate memory regions that support maximum 257
> sg entries, the real MR memory allocation size is slightly bigger than 2048.
>
> [...]
Applied, thanks!
[1/1] RDMA/mlx5: align MR mem allocation size to power-of-two
https://git.kernel.org/rdma/rdma/c/52b4bdd28c861e
Best regards,
--
Leon Romanovsky <[email protected]>