Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Praveen Kannoju <praveen.kannoju@oracle.com>
To:     Jason Gunthorpe <jgg@nvidia.com>,
        Aruna Ramakrishna <aruna.ramakrishna@oracle.com>
CC:     "leon@kernel.org" <leon@kernel.org>,
        "dledford@redhat.com" <dledford@redhat.com>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Rajesh Sivaramasubramaniom 
        <rajesh.sivaramasubramaniom@oracle.com>,
        Rama Nichanamatlu <rama.nichanamatlu@oracle.com>,
        Jeffery Yoder <jeffery.yoder@oracle.com>
Subject: RE: [PATCH v2] IB/mlx5: Reduce max order of memory allocated for xlt
 update
Thread-Topic: [PATCH v2] IB/mlx5: Reduce max order of memory allocated for xlt
 update
Thread-Index: AQHXGmWJp8i+xPdCxUaGXGoNHWnqKKqRyJQAgAA7xICAADsYgIAAV88AgAI9RwCACaQeAIABcEqQ
Date:   Thu, 1 Apr 2021 15:56:37 +0000
Message-ID: <CY4PR10MB1448C03F77B4DDF8833489058C7B9@CY4PR10MB1448.namprd10.prod.outlook.com>
References: <1615900141-14012-1-git-send-email-praveen.kannoju@oracle.com>
 <20210323160756.GE2710221@ziepe.ca>
 <80966C8E-341B-4F5D-9DCA-C7D82AB084D5@oracle.com>
 <20210323231321.GF2710221@ziepe.ca>
 <0DFF7518-8818-445B-94AC-8EB2096446BE@oracle.com>
 <20210325143928.GM2710221@ziepe.ca> <20210331175312.GA1531363@nvidia.com>
In-Reply-To: <20210331175312.GA1531363@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 9c098f3e-46a6-4b72-c28e-08d8f526bb7d
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Apr 2021 15:56:37.4759
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: TtEucn7irk6sxLlGE5qE+dvgl5yQVKI8VP5WsNghyxryRiHEMzqSmS3kxsGLdx5pFwdhzOcl/S7U7dw2Fh1vCg7gv/Axd6p6DnP1mtR+tr8=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR10MB1368
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9941 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999
 adultscore=0 bulkscore=0 mlxscore=0 spamscore=0 malwarescore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2103310000 definitions=main-2104010105
X-Proofpoint-GUID: asGQy_dk67aNyNBDOp6qD0UDcy8oLQeK
X-Proofpoint-ORIG-GUID: asGQy_dk67aNyNBDOp6qD0UDcy8oLQeK
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9941 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999 mlxscore=0
 lowpriorityscore=0 suspectscore=0 priorityscore=1501 phishscore=0
 clxscore=1011 impostorscore=0 malwarescore=0 bulkscore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103310000
 definitions=main-2104010105
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jason,
We will go ahead with just adding the "__GFP_NORETRY " flag to reduce the t=
ime it takes to fail the higher order memory allocations in case higher ord=
er pages are not available.

Will send out the corresponding patch.

Thank you very much for your inputs.

-
Praveen Kumar Kannoju.


-----Original Message-----
From: Jason Gunthorpe [mailto:jgg@nvidia.com]=20
Sent: 31 March 2021 11:23 PM
To: Aruna Ramakrishna <aruna.ramakrishna@oracle.com>
Cc: Praveen Kannoju <praveen.kannoju@oracle.com>; leon@kernel.org; dledford=
@redhat.com; linux-rdma@vger.kernel.org; linux-kernel@vger.kernel.org; Raje=
sh Sivaramasubramaniom <rajesh.sivaramasubramaniom@oracle.com>; Rama Nichan=
amatlu <rama.nichanamatlu@oracle.com>; Jeffery Yoder <jeffery.yoder@oracle.=
com>
Subject: Re: [PATCH v2] IB/mlx5: Reduce max order of memory allocated for x=
lt update

On Thu, Mar 25, 2021 at 11:39:28AM -0300, Jason Gunthorpe wrote:
> On Tue, Mar 23, 2021 at 09:27:38PM -0700, Aruna Ramakrishna wrote:
>=20
> > > Do you have benchmarks that show the performance of the high order=20
> > > pages is not relavent? I'm a bit surprised to hear that
> > >=20
> >=20
> > I guess my point was more to the effect that an order-8 alloc will=20
> > fail more often than not, in this flow. For instance, when we were=20
> > debugging the latency spikes here, this was the typical buddyinfo=20
> > output on that system:
> >=20
> > Node 0, zone      DMA      0      1      1      2      3      0      1 =
     0      1      1      3=20
> > Node 0, zone    DMA32      7      7      7      6     10      2      6 =
     7      6      2    306=20
> > Node 0, zone   Normal   3390  51354  17574   6556   1586     26      2 =
     1      0      0      0=20
> > Node 1, zone   Normal  11519  23315  23306   9738     73      2      0 =
     1      0      0      0=20
> >=20
> > I think this level of fragmentation is pretty normal on long running=20
> > systems. Here, in the reg_mr flow, the first try (order-8) alloc=20
> > will probably fail 9 times out of 10 (esp. after the addition of=20
> > GFP_NORETRY flag), and then as fallback, the code tries to allocate=20
> > a lower order, and if that too fails, it allocates a page. I think=20
> > it makes sense to just avoid trying an order-8 alloc here.
>=20
> But a system like this won't get THPs either, so I'm not sure it is=20
> relevant. The function was designed as it is to consume a "THP" if it=20
> is available.

So can we do this with just the addition of __GFP_NORETRY ?

Jason