Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <e314748f-b4df-c65d-7acc-45c21abf31ce@intel.com>
Date:   Thu, 11 May 2023 07:23:51 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.10.0
Subject: Re: [RFC 0/6] mm: improve page allocator scalability via splitting
 zones
Content-Language: en-US
To:     Huang Ying <ying.huang@intel.com>, linux-mm@kvack.org
Cc:     linux-kernel@vger.kernel.org,
        Arjan Van De Ven <arjan@linux.intel.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Mel Gorman <mgorman@techsingularity.net>,
        Vlastimil Babka <vbabka@suse.cz>,
        David Hildenbrand <david@redhat.com>,
        Johannes Weiner <jweiner@redhat.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Michal Hocko <mhocko@suse.com>,
        Pavel Tatashin <pasha.tatashin@soleen.com>,
        Matthew Wilcox <willy@infradead.org>
References: <20230511065607.37407-1-ying.huang@intel.com>
From:   Dave Hansen <dave.hansen@intel.com>
In-Reply-To: <20230511065607.37407-1-ying.huang@intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 5/10/23 23:56, Huang Ying wrote:
> To improve the scalability of the page allocation, in this series, we
> will create one zone instance for each about 256 GB memory of a zone
> type generally.  That is, one large zone type will be split into
> multiple zone instances. 

A few anecdotes for why I think _some_ people will like this:

Some Intel hardware has a "RAM" caching mechanism.  It either caches
DRAM in High-Bandwidth Memory or Persistent Memory in DRAM.  This cache
is direct-mapped and can have lots of collisions.  One way to prevent
collisions is to chop up the physical memory into cache-sized zones and
let users choose to allocate from one zone.  That fixes the conflicts.

Some other Intel hardware a ways to chop a NUMA node representing a
single socket into slices.  Usually one slice gets a memory controller
and its closest cores.  Intel calls these approaches Cluster on Die or
Sub-NUMA Clustering and users can select it from the BIOS.

In both of these cases, users have reported scalability improvements.
We've gone as far as to suggest the socket-splitting options to folks
today who are hitting zone scalability issues on that hardware.

That said, those _same_ users sometimes come back and say something
along the lines of: "So... we've got this app that allocates a big hunk
of memory.  It's going slower than before."  They're filling up one of
the chopped-up zones, hitting _some_ kind of undesirable reclaim
behavior and they want their humpty-dumpty zones put back together again
... without hurting scalability.  Some people will never be happy. :)

Anyway, _if_ you do this, you might also consider being able to
dynamically adjust a CPU's zonelists somehow.  That would relieve
pressure on one zone for those uneven allocations.  That wasn't an
option in the two cases above because users had ulterior motives for
sticking inside a single zone.  But, in your case, the zones really do
have equivalent performance.