Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <20210401183216.443C4443@viggo.jf.intel.com> <20210401183219.DC1928FA@viggo.jf.intel.com>
 <CAAPL-u8Jk-i-9-iSnU7_nb-k2ZMqdRk5c88d-M6Bi1rfv4kSLQ@mail.gmail.com>
 <20210414080849.GA20886@linux> <CAAPL-u-w_WShb0RyXhs8koihTOPvFK_dCwB22RhzA=f9kRyqqQ@mail.gmail.com>
 <6215a690-d14a-de7e-72cb-1aa4e2822f2e@intel.com>
In-Reply-To: <6215a690-d14a-de7e-72cb-1aa4e2822f2e@intel.com>
From:   Wei Xu <weixugc@google.com>
Date:   Thu, 15 Apr 2021 13:25:08 -0700
Message-ID: <CAAPL-u87JKZWzHLigh4KiGAocR8ZDzbi8MPU2WNm9e_h+F1yHw@mail.gmail.com>
Subject: Re: [PATCH 02/10] mm/numa: automatically generate node migration order
To:     Dave Hansen <dave.hansen@intel.com>
Cc:     Oscar Salvador <osalvador@suse.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Linux MM <linux-mm@kvack.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Yang Shi <shy828301@gmail.com>,
        David Rientjes <rientjes@google.com>,
        Huang Ying <ying.huang@intel.com>,
        Dan Williams <dan.j.williams@intel.com>,
        David Hildenbrand <david@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Thu, Apr 15, 2021 at 8:35 AM Dave Hansen <dave.hansen@intel.com> wrote:
> > This can help enable more flexible demotion policies to be
> > configured, such as to allow a cgroup to allocate from all fast tier
> > nodes, but only demote to a local slow tier node.  Such a policy can
> > reduce memory stranding at the fast tier (compared to if memory
> > hardwall is used) and still allow demotion from all fast tier nodes
> > without incurring the expensive random accesses to the demoted pages
> > if they were demoted to remote slow tier nodes.
>
> Could you explain this stranding effect in a bit more detail?  I'm not
> quite following.

By memory stranding, I mean that memory on a machine (or a NUMA node)
cannot be utilized even under extremely high work loads.  Memory
stranding happens usually due to mismatches between job/machine
shapes as well as resource fragmentation resulted from bin-packing
scheduling.  It is an important problem for cloud resource efficiency.

If NUMA hardwalling is used, we effectively split a single machine
into multiple smaller machines based on NUMA nodes.  This changes the
machine shapes and also makes memory more fragmented, which can lead
to more memory being stranded.

Here is a simple example:  Suppose that each machine has 2 NUMA nodes,
each with 4 cores and 5GB RAM, and all the jobs have the shape of
2 CPUs and 3GB memory.  Without NUMA memory hardwalling, we can pack
3 jobs onto each machine, which leaves 1GB memory and 2 cores in
stranding.  However, with NUMA memory hardwalling enabled, we can then
only pack 2 jobs onto each machine (one job on each NUMA node), which
increases the resource stranding to 4GB memory and 4 cores.