Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 25 Mar 2021 11:39:28 -0300
From:   Jason Gunthorpe <jgg@ziepe.ca>
To:     Aruna Ramakrishna <aruna.ramakrishna@oracle.com>
Cc:     Praveen Kumar Kannoju <praveen.kannoju@oracle.com>,
        leon@kernel.org, dledford@redhat.com, linux-rdma@vger.kernel.org,
        linux-kernel@vger.kernel.org,
        Rajesh Sivaramasubramaniom 
        <rajesh.sivaramasubramaniom@oracle.com>,
        Rama Nichanamatlu <rama.nichanamatlu@oracle.com>,
        Jeffery Yoder <jeffery.yoder@oracle.com>
Subject: Re: [PATCH v2] IB/mlx5: Reduce max order of memory allocated for xlt
 update
Message-ID: <20210325143928.GM2710221@ziepe.ca>
References: <1615900141-14012-1-git-send-email-praveen.kannoju@oracle.com>
 <20210323160756.GE2710221@ziepe.ca>
 <80966C8E-341B-4F5D-9DCA-C7D82AB084D5@oracle.com>
 <20210323231321.GF2710221@ziepe.ca>
 <0DFF7518-8818-445B-94AC-8EB2096446BE@oracle.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <0DFF7518-8818-445B-94AC-8EB2096446BE@oracle.com>
Precedence: bulk

On Tue, Mar 23, 2021 at 09:27:38PM -0700, Aruna Ramakrishna wrote:

> > Do you have benchmarks that show the performance of the high order
> > pages is not relavent? I'm a bit surprised to hear that
> > 
> 
> I guess my point was more to the effect that an order-8 alloc will
> fail more often than not, in this flow. For instance, when we were
> debugging the latency spikes here, this was the typical buddyinfo
> output on that system:
> 
> Node 0, zone      DMA      0      1      1      2      3      0      1      0      1      1      3 
> Node 0, zone    DMA32      7      7      7      6     10      2      6      7      6      2    306 
> Node 0, zone   Normal   3390  51354  17574   6556   1586     26      2      1      0      0      0 
> Node 1, zone   Normal  11519  23315  23306   9738     73      2      0      1      0      0      0 
> 
> I think this level of fragmentation is pretty normal on long running
> systems. Here, in the reg_mr flow, the first try (order-8) alloc
> will probably fail 9 times out of 10 (esp. after the addition of
> GFP_NORETRY flag), and then as fallback, the code tries to allocate
> a lower order, and if that too fails, it allocates a page. I think
> it makes sense to just avoid trying an order-8 alloc here.

But a system like this won't get THPs either, so I'm not sure it is
relevant. The function was designed as it is to consume a "THP" if it
is available.

Jason