Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.17\))
Subject: Re: [PATCH v2] IB/mlx5: Reduce max order of memory allocated for xlt
 update
From:   Aruna Ramakrishna <aruna.ramakrishna@oracle.com>
In-Reply-To: <20210323231321.GF2710221@ziepe.ca>
Date:   Tue, 23 Mar 2021 21:27:38 -0700
Cc:     Praveen Kumar Kannoju <praveen.kannoju@oracle.com>,
        leon@kernel.org, dledford@redhat.com, linux-rdma@vger.kernel.org,
        linux-kernel@vger.kernel.org,
        Rajesh Sivaramasubramaniom 
        <rajesh.sivaramasubramaniom@oracle.com>,
        Rama Nichanamatlu <rama.nichanamatlu@oracle.com>,
        Jeffery Yoder <jeffery.yoder@oracle.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <0DFF7518-8818-445B-94AC-8EB2096446BE@oracle.com>
References: <1615900141-14012-1-git-send-email-praveen.kannoju@oracle.com>
 <20210323160756.GE2710221@ziepe.ca>
 <80966C8E-341B-4F5D-9DCA-C7D82AB084D5@oracle.com>
 <20210323231321.GF2710221@ziepe.ca>
To:     Jason Gunthorpe <jgg@ziepe.ca>
Precedence: bulk


> On Mar 23, 2021, at 4:13 PM, Jason Gunthorpe <jgg@ziepe.ca> wrote:
>=20
> On Tue, Mar 23, 2021 at 12:41:51PM -0700, Aruna Ramakrishna wrote:
>>   There is a far greater possibility of an order-8 allocation =
failing,
>>   esp. with the addition of __GFP_NORETRY , and the code would have =
to
>>   fall back to a lower order allocation more often than not (esp. on =
a
>>   long running system). Unless the performance gains from using =
order-8
>>   pages is significant (and it does not seem that way to me), we can =
just
>>   skip this step and directly go to the lower order allocation.
>=20
> Do not send HTML mails.

I apologize; I=E2=80=99ve fixed the setting now.

>=20
> Do you have benchmarks that show the performance of the high order
> pages is not relavent? I'm a bit surprised to hear that
>=20

I guess my point was more to the effect that an order-8 alloc will fail =
more often than not, in this flow. For instance, when we were debugging =
the latency spikes here, this was the typical buddyinfo output on that =
system:

Node 0, zone      DMA      0      1      1      2      3      0      1   =
   0      1      1      3=20
Node 0, zone    DMA32      7      7      7      6     10      2      6   =
   7      6      2    306=20
Node 0, zone   Normal   3390  51354  17574   6556   1586     26      2   =
   1      0      0      0=20
Node 1, zone   Normal  11519  23315  23306   9738     73      2      0   =
   1      0      0      0=20

I think this level of fragmentation is pretty normal on long running =
systems. Here, in the reg_mr flow, the first try (order-8) alloc will =
probably fail 9 times out of 10 (esp. after the addition of GFP_NORETRY =
flag), and then as fallback, the code tries to allocate a lower order, =
and if that too fails, it allocates a page. I think it makes sense to =
just avoid trying an order-8 alloc here.

Thanks,
Aruna


> This code really needs some attention to use a proper
> scatter/gather. I understand the chip can do it, just some of the
> software layers need to be stripped away so it can form the right SGL
> in the HW.
>=20
> Jason