Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 15 Dec 2022 09:22:33 +0100
From:   Johannes Weiner <hannes@cmpxchg.org>
To:     "Huang, Ying" <ying.huang@intel.com>
Cc:     Michal Hocko <mhocko@suse.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Yang Shi <shy828301@gmail.com>, Wei Xu <weixugc@google.com>,
        Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: memcg reclaim demotion wrt. isolation
Message-ID: <Y5rZSRxcgQzQQVbS@cmpxchg.org>
References: <Y5idFucjKVbjatqc@dhcp22.suse.cz>
 <Y5ik+CCmvapf87Mb@cmpxchg.org>
 <Y5maoIUuH79KrfJt@dhcp22.suse.cz>
 <87edt1dwd2.fsf@yhuang6-desk2.ccr.corp.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87edt1dwd2.fsf@yhuang6-desk2.ccr.corp.intel.com>
Precedence: bulk

On Thu, Dec 15, 2022 at 02:17:13PM +0800, Huang, Ying wrote:
> Michal Hocko <mhocko@suse.com> writes:
> 
> > On Tue 13-12-22 17:14:48, Johannes Weiner wrote:
> >> On Tue, Dec 13, 2022 at 04:41:10PM +0100, Michal Hocko wrote:
> >> > Hi,
> >> > I have just noticed that that pages allocated for demotion targets
> >> > includes __GFP_KSWAPD_RECLAIM (through GFP_NOWAIT). This is the case
> >> > since the code has been introduced by 26aa2d199d6f ("mm/migrate: demote
> >> > pages during reclaim"). I suspect the intention is to trigger the aging
> >> > on the fallback node and either drop or further demote oldest pages.
> >> > 
> >> > This makes sense but I suspect that this wasn't intended also for
> >> > memcg triggered reclaim. This would mean that a memory pressure in one
> >> > hierarchy could trigger paging out pages of a different hierarchy if the
> >> > demotion target is close to full.
> >> 
> >> This is also true if you don't do demotion. If a cgroup tries to
> >> allocate memory on a full node (i.e. mbind()), it may wake kswapd or
> >> enter global reclaim directly which may push out the memory of other
> >> cgroups, regardless of the respective cgroup limits.
> >
> > You are right on this. But this is describing a slightly different
> > situaton IMO. 
> >
> >> The demotion allocations don't strike me as any different. They're
> >> just allocations on behalf of a cgroup. I would expect them to wake
> >> kswapd and reclaim physical memory as needed.
> >
> > I am not sure this is an expected behavior. Consider the currently
> > discussed memory.demote interface when the userspace can trigger
> > (almost) arbitrary demotions. This can deplete fallback nodes without
> > over-committing the memory overall yet push out demoted memory from
> > other workloads. From the user POV it would look like a reclaim while
> > the overall memory is far from depleted so it would be considered as
> > premature and a warrant a bug report.
> >
> > The reclaim behavior would make more sense to me if it was constrained
> > to the allocating memcg hierarchy so unrelated lruvecs wouldn't be
> > disrupted.
> 
> When we reclaim/demote some pages from a memcg proactively, what is our
> goal?  To free up some memory in this memcg for other memcgs to use?  If
> so, it sounds reasonable to keep the pages of other memcgs as many as
> possible.

The goal of proactive aging is to free up any resources that aren't
needed to meet the SLAs (e.g. end-to-end response time of webserver).
Meaning, to run things as leanly as possible within spec. Into that
free space, another container can then be co-located.

This means that the goal is to free up as many resources as possible,
starting with the coveted hightier. If a container has been using
all-hightier memory but is able demote to lowtier, there are 3 options
for existing memory in the lower tier:

1) Colder/stale memory - should be displaced

2) Memory that can be promoted once the hightier is free -
   reclaim/demotion of the coldest pages needs to happen at least
   temporarily, or the tierswap is in stale mate.

3) Equally hot memory - if this exceeds capacity of the lower tier,
   the hottest overall pages should stay, the excess demoted/reclaimed.

You can't know what scenario you're in until you put the demoted pages
in direct LRU competition with what's already there. And in all three
scenarios, direct LRU competition also produces the optimal outcome.