Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Praveen Kannoju <praveen.kannoju@oracle.com>
To:     Praveen Kannoju <praveen.kannoju@oracle.com>,
        "leon@kernel.org" <leon@kernel.org>,
        "dledford@redhat.com" <dledford@redhat.com>,
        "jgg@ziepe.ca" <jgg@ziepe.ca>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
CC:     Rajesh Sivaramasubramaniom 
        <rajesh.sivaramasubramaniom@oracle.com>,
        Rama Nichanamatlu <rama.nichanamatlu@oracle.com>,
        Aruna Ramakrishna <aruna.ramakrishna@oracle.com>,
        Jeffery Yoder <jeffery.yoder@oracle.com>
Subject: RE: [PATCH v2] IB/mlx5: Reduce max order of memory allocated for xlt
 update
Thread-Topic: [PATCH v2] IB/mlx5: Reduce max order of memory allocated for xlt
 update
Thread-Index: AQHXGmWJp8i+xPdCxUaGXGoNHWnqKKqRnxOA
Date:   Tue, 23 Mar 2021 13:42:41 +0000
Message-ID: <CY4PR10MB14483428B986BAF6CC6B89118C649@CY4PR10MB1448.namprd10.prod.outlook.com>
References: <1615900141-14012-1-git-send-email-praveen.kannoju@oracle.com>
In-Reply-To: <1615900141-14012-1-git-send-email-praveen.kannoju@oracle.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: f6029229-2d3a-408d-2dd4-08d8ee018827
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Mar 2021 13:42:41.7653
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: QrY0naWUdtzOzM2AUBET4oemnDGskWJfhwtVsuVWHdb+YPFYdHzYWhjdek7X8rUTcUBb36JigHj/knHfr6T1RvzKOs0P0X8cTyUVdGbw46M=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR10MB1239
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9931 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999
 malwarescore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2009150000 definitions=main-2103230101
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9931 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0
 lowpriorityscore=0 suspectscore=0 clxscore=1011 priorityscore=1501
 spamscore=0 adultscore=0 impostorscore=0 mlxlogscore=999 mlxscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2009150000 definitions=main-2103230101
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Ping.
Request the reviewers to go through the patch and let us know if you have a=
ny queries with respect to it.

-
Praveen Kumar Kannoju.=20


-----Original Message-----
From: Praveen Kannoju [mailto:praveen.kannoju@oracle.com]=20
Sent: 16 March 2021 06:39 PM
To: leon@kernel.org; dledford@redhat.com; jgg@ziepe.ca; linux-rdma@vger.ker=
nel.org; linux-kernel@vger.kernel.org
Cc: Rajesh Sivaramasubramaniom <rajesh.sivaramasubramaniom@oracle.com>; Ram=
a Nichanamatlu <rama.nichanamatlu@oracle.com>; Aruna Ramakrishna <aruna.ram=
akrishna@oracle.com>; Jeffery Yoder <jeffery.yoder@oracle.com>; Praveen Kan=
noju <praveen.kannoju@oracle.com>
Subject: [PATCH v2] IB/mlx5: Reduce max order of memory allocated for xlt u=
pdate

To update xlt (during mlx5_ib_reg_user_mr()), the driver can request up to
1 MB (order-8) memory, depending on the size of the MR. This costly allocat=
ion can sometimes take very long to return (a few seconds), especially if t=
he system is fragmented and does not have any free chunks for orders >=3D 3=
. This causes the calling application to hang for a long time. To avoid the=
se long latency spikes, limit max order of allocation to order 3, and reuse=
 that buffer to populate_xlt() for that MR. This will increase the latency =
slightly (in the order of microseconds) for each
mlx5_ib_update_xlt() call, especially for larger MRs (since were making mul=
tiple calls to populate_xlt()), but its a small price to pay to avoid the l=
arge latency spikes with higher order allocations. The flag __GFP_NORETRY i=
s used while fetching the free pages to ensure that there are no long compa=
ction stalls when the system's memory is in fragmented condition.

Signed-off-by: Praveen Kumar Kannoju <praveen.kannoju@oracle.com>
---
 drivers/infiniband/hw/mlx5/mr.c | 22 +++-------------------
 1 file changed, 3 insertions(+), 19 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/m=
r.c index db05b0e..dac19f0 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1004,9 +1004,7 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct i=
b_pd *pd,
 	return mr;
 }
=20
-#define MLX5_MAX_UMR_CHUNK ((1 << (MLX5_MAX_UMR_SHIFT + 4)) - \
-			    MLX5_UMR_MTT_ALIGNMENT)
-#define MLX5_SPARE_UMR_CHUNK 0x10000
+#define MLX5_SPARE_UMR_CHUNK 0x8000
=20
 /*
  * Allocate a temporary buffer to hold the per-page information to transfe=
r to @@ -1028,30 +1026,16 @@ static void *mlx5_ib_alloc_xlt(size_t *nents, =
size_t ent_size, gfp_t gfp_mask)
 	 */
 	might_sleep();
=20
-	gfp_mask |=3D __GFP_ZERO;
+	gfp_mask |=3D __GFP_ZERO | __GFP_NORETRY;
=20
-	/*
-	 * If the system already has a suitable high order page then just use
-	 * that, but don't try hard to create one. This max is about 1M, so a
-	 * free x86 huge page will satisfy it.
-	 */
 	size =3D min_t(size_t, ent_size * ALIGN(*nents, xlt_chunk_align),
-		     MLX5_MAX_UMR_CHUNK);
+		     MLX5_SPARE_UMR_CHUNK);
 	*nents =3D size / ent_size;
 	res =3D (void *)__get_free_pages(gfp_mask | __GFP_NOWARN,
 				       get_order(size));
 	if (res)
 		return res;
=20
-	if (size > MLX5_SPARE_UMR_CHUNK) {
-		size =3D MLX5_SPARE_UMR_CHUNK;
-		*nents =3D get_order(size) / ent_size;
-		res =3D (void *)__get_free_pages(gfp_mask | __GFP_NOWARN,
-					       get_order(size));
-		if (res)
-			return res;
-	}
-
 	*nents =3D PAGE_SIZE / ent_size;
 	res =3D (void *)__get_free_page(gfp_mask);
 	if (res)
--
1.8.3.1