Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Yuanyuan Zhong <yzhong@purestorage.com>
To:     leon@kernel.org, jgg@ziepe.ca
Cc:     cachen@purestorage.com, linux-rdma@vger.kernel.org,
        linux-kernel@vger.kernel.org,
        Yuanyuan Zhong <yzhong@purestorage.com>
Subject: [PATCH 0/1] RDMA/mlx5: align MR mem allocation size to power-of-two
Date:   Thu, 29 Jun 2023 15:32:47 -0600
Message-Id: <20230629213248.3184245-1-yzhong@purestorage.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

Hello,

Recently I noticed the mlx5_alloc_priv_descs() do many allocations from
kmalloc-8k. It's called from following chain:
cma_cm_event_handler => nvme_rdma_cm_handler => nvme_rdma_create_queue_ib
=> ib_mr_pool_init => ib_alloc_mr => __mlx5_ib_alloc_mr
=> _mlx5_alloc_mkey_descs

When the nvme-rdma asks to allocate memory regions that support maximum 257
sg entries, the real MR memory allocation size is slightly bigger than 2048.

The MR memory allocation seems adding "add_size" to guarantee that there
is enough space to find the memory aligned to 2048 (MLX5_UMR_ALIGN).

The requested size is bigger than 4096 after adding the extra bytes.
Then kmalloc uses 8192 as the actual allocation size.

For power-of-two sizes, the alignment can be guaranteed by kmalloc()
according to commit 59bb47985c1d ("mm, sl[aou]b: guarantee natural
alignment for kmalloc(power-of-two)").

Here comes a draft patch that uses the lesser of existing requested
size or the next power-of-two size.

I'm not sure if it's better to simply allocate roundup_pow_of_two(),
or if it's true to assume that slab will not have intermediate size
between 2 power-of-two sizes. Also not sure if a generic memalign()
will be the better approach.

Can you please let me know what's the preferred fix, or fix it directly
to let the allocation use the smaller slab if possible? Thanks


Yuanyuan Zhong (1):
  RDMA/mlx5: align MR mem allocation size to power-of-two

 drivers/infiniband/hw/mlx5/mr.c | 5 +++++
 1 file changed, 5 insertions(+)

-- 
2.34.1