Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6;
From:   "Huang, Ying" <ying.huang@intel.com>
To:     Mel Gorman <mgorman@techsingularity.net>
Cc:     <linux-mm@kvack.org>, <linux-kernel@vger.kernel.org>,
        Arjan Van De Ven <arjan@linux.intel.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Vlastimil Babka <vbabka@suse.cz>,
        David Hildenbrand <david@redhat.com>,
        Johannes Weiner <jweiner@redhat.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Michal Hocko <mhocko@suse.com>,
        Pavel Tatashin <pasha.tatashin@soleen.com>,
        Matthew Wilcox <willy@infradead.org>,
        "Christoph Lameter" <cl@linux.com>
Subject: Re: [PATCH 03/10] mm, pcp: reduce lock contention for draining
 high-order pages
References: <20230920061856.257597-1-ying.huang@intel.com>
        <20230920061856.257597-4-ying.huang@intel.com>
        <20231011124900.sp22hoxoitrslbia@techsingularity.net>
Date:   Thu, 12 Oct 2023 20:11:19 +0800
In-Reply-To: <20231011124900.sp22hoxoitrslbia@techsingularity.net> (Mel
        Gorman's message of "Wed, 11 Oct 2023 13:49:00 +0100")
Message-ID: <87cyxkf3bs.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=ascii
Precedence: bulk

Mel Gorman <mgorman@techsingularity.net> writes:

> On Wed, Sep 20, 2023 at 02:18:49PM +0800, Huang Ying wrote:
>> In commit f26b3fa04611 ("mm/page_alloc: limit number of high-order
>> pages on PCP during bulk free"), the PCP (Per-CPU Pageset) will be
>> drained when PCP is mostly used for high-order pages freeing to
>> improve the cache-hot pages reusing between page allocating and
>> freeing CPUs.
>> 
>> On system with small per-CPU data cache, pages shouldn't be cached
>> before draining to guarantee cache-hot.  But on a system with large
>> per-CPU data cache, more pages can be cached before draining to reduce
>> zone lock contention.
>> 
>> So, in this patch, instead of draining without any caching, "batch"
>> pages will be cached in PCP before draining if the per-CPU data cache
>> size is more than "4 * batch".
>> 
>> On a 2-socket Intel server with 128 logical CPU, with the patch, the
>> network bandwidth of the UNIX (AF_UNIX) test case of lmbench test
>> suite with 16-pair processes increase 72.2%.  The cycles% of the
>> spinlock contention (mostly for zone lock) decreases from 45.8% to
>> 21.2%.  The number of PCP draining for high order pages
>> freeing (free_high) decreases 89.8%.  The cache miss rate keeps 0.3%.
>> 
>> Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
>
> Acked-by: Mel Gorman <mgorman@techsingularity.net>
>
> However, the flag should also have been documented to make it clear that
> it preserves some pages on the PCP if the cache is large enough.

Sure.  Will do this.

> Similar
> to the previous patch, it would have been easier to reason about in the
> general case if the decision had only been based on the LLC without
> having to worry if any intermediate layer has a meaningful impact that
> varies across CPU implementations.

Sure.  Will do this.

--
Best Regards,
Huang, Ying