Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 26 Oct 2022 20:20:01 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     Michal Hocko <mhocko@suse.com>
CC:     Aneesh Kumar K V <aneesh.kumar@linux.ibm.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Tejun Heo <tj@kernel.org>, Zefan Li <lizefan.x@bytedance.com>,
        Waiman Long <longman@redhat.com>,
        "Huang, Ying" <ying.huang@intel.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "Hansen, Dave" <dave.hansen@intel.com>,
        "Chen, Tim C" <tim.c.chen@intel.com>,
        "Yin, Fengwei" <fengwei.yin@intel.com>
Subject: Re: [PATCH] mm/vmscan: respect cpuset policy during page demotion
Message-ID: <Y1kl8VbPE0RYdyEB@feng-clx>
References: <20221026074343.6517-1-feng.tang@intel.com>
 <dc453287-015d-fd1c-fe7f-6ee45772d6aa@linux.ibm.com>
 <Y1jpDfwBQId3GkJC@feng-clx>
 <Y1j7tsj5M0Md/+Er@dhcp22.suse.cz>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <Y1j7tsj5M0Md/+Er@dhcp22.suse.cz>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?URu1t9CES64yjxWrL5wv3/LUbX6Cbj6FLSx6uv+r6O2LiIds7RPgj7CNZQkn?=
 =?us-ascii?Q?/sA/OjA6Bev2Z60S7he28DcZC+NVZdsXVQBkO8nKiv66AQW8wOyZGLWCp3Up?=
 =?us-ascii?Q?TarUDWr6GhD3RZYyFBabQEcYb7WJ2qvpU56MLtWuFo83QdQXuimnYIfF//9K?=
 =?us-ascii?Q?Nm+yBot1xY3pb7kHHxgTx2nXNCUiOd4cdTNtM0BwNMg86A/9StOyIawMAXJL?=
 =?us-ascii?Q?Yz6M7a8F4YH563HwGsZfpYFleEd9kyq3oJIpItexTI2FEjbAWSO5PrdQJumK?=
 =?us-ascii?Q?lOkkQdcxDsCxPRqDfE9j0pF9FCkLdPM5MrW5HhRumqudNng/yR9rkav2BtUl?=
 =?us-ascii?Q?05B6Z9mb5aKa3w3mK2+2lO+0oof2OkzcDSYkw68H1bo+dAiC35DPX65x/UCt?=
 =?us-ascii?Q?EMxsWkPNHJly8brYm7BwQFrZ/EDK9EDoxnrAk4hKiQt6JTlXDMU8XvHADES7?=
 =?us-ascii?Q?b5BXa1QhJGomBJJrEdmaSQkDLebWgm9jOJbOjWiHkbj2PCnrQdnJ9Zp4EXXT?=
 =?us-ascii?Q?Fcy5P9GAzQ1a0HaMvR0uJBRdxT6m2vRBgqlc01ppUnSwx5356fFfowGXN/CH?=
 =?us-ascii?Q?5Q7ro4bc4FfacLVWWCxeyU/52Yl8vGhgQC5K2n7heMEN9rM74rYmu2Hu4m0V?=
 =?us-ascii?Q?0oexGjsabUzTq+l9jg3HcQavSC0uxUW17MHQXVxPzhJLyJkMH/tbU90Dllp5?=
 =?us-ascii?Q?9YhBb+E/nAeKgxB4snJ1/S05/eYtueDBHu9gVG6IWDV00sJlXsUr5cohunXG?=
 =?us-ascii?Q?J/PDMvmCYdOUD77NeQ4DRnTGVSou3exYxQ/LmuUd+xbdy/FbAE2AGiVxeBNE?=
 =?us-ascii?Q?t6qe6kloZw0rJFUgKuKM80V+ksIrTL3zc9dwLTAIc31mZlIGmXPditQrB/yi?=
 =?us-ascii?Q?yzEfshsgCcDRVPBa6/GAUhN+uuqUuTJlCu6kAwkoq6H553FkRJ8XDTty1PuK?=
 =?us-ascii?Q?pzDDY7aSk0/SQuRBlp1kgWEQeDAq9qG/Wp2SFOTAgsvlIlc2q8yqq5qUx0Fm?=
 =?us-ascii?Q?v3/WHCiX63us/f+ZG2mmQEZlCBBCL5k7KxXW/wM5FmdFzSCSVJR20ttg4mFK?=
 =?us-ascii?Q?rVZB7oXqKY4eevUb6i0U/TkVzpzNhg6jfRFXIqgz+gkDekKf4MkyZiV8RNYi?=
 =?us-ascii?Q?THHZY6OG/l6AHFvRQ59bFWDNd5+wRaDX1hu4iOskKdL+lt2XtoKQ4ys9MUzq?=
 =?us-ascii?Q?Q+Py1yGCNDgP14/vIAM4cBOj8JouMZRdhSiB70s2uOZkz/fajPCJ8bhmr/To?=
 =?us-ascii?Q?YTsGMYDeP31zkw5cuSCitFEBvBCHgKYxM8cQJJHqkO2gbQJpiaakW1OkiAW5?=
 =?us-ascii?Q?WUc+1o8w7gvGl6dMewgdvup0rsebQhqrMuTG/o0UQt25GEAs+mpvDq2jRFaT?=
 =?us-ascii?Q?6w5o+SBYaJGgNbaIv2WqKnOTMZ9uFUpiBtKBzHjroBKD0FFxAkAtarubEH5j?=
 =?us-ascii?Q?Um1LMkYltyLgN0IOOmKXBkNnKl+Pidt7tcViJAaTmbds38QGRiE5v8cHQmDr?=
 =?us-ascii?Q?NIZ07Sd8y2azEN/9U8Wp2HH50qV9+sA15kp/KMCm2EZQ3I/Ck1mR6g1xy/bX?=
 =?us-ascii?Q?2oDunfjshKMgSoBMg61zen2eAbcgLKv8Zfjiz9i/?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 573e305a-cd81-41ba-8c8c-08dab74c6da1
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Oct 2022 12:20:11.2610
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mhTvmNZcaV6yt+r6N9YiOx6nGooSJpy/CNx57nA0FliOpAW3m5W5bPdVAd3hDOBZu/XMt6L5SvOwb5lHahrcIg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5940
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Oct 26, 2022 at 05:19:50PM +0800, Michal Hocko wrote:
> On Wed 26-10-22 16:00:13, Feng Tang wrote:
> > On Wed, Oct 26, 2022 at 03:49:48PM +0800, Aneesh Kumar K V wrote:
> > > On 10/26/22 1:13 PM, Feng Tang wrote:
> > > > In page reclaim path, memory could be demoted from faster memory tier
> > > > to slower memory tier. Currently, there is no check about cpuset's
> > > > memory policy, that even if the target demotion node is not allowd
> > > > by cpuset, the demotion will still happen, which breaks the cpuset
> > > > semantics.
> > > > 
> > > > So add cpuset policy check in the demotion path and skip demotion
> > > > if the demotion targets are not allowed by cpuset.
> > > > 
> > > 
> > > What about the vma policy or the task memory policy? Shouldn't we respect
> > > those memory policy restrictions while demoting the page? 
> >  
> > Good question! We have some basic patches to consider memory policy
> > in demotion path too, which are still under test, and will be posted
> > soon. And the basic idea is similar to this patch.
> 
> For that you need to consult each vma and it's owning task(s) and that
> to me sounds like something to be done in folio_check_references.
> Relying on memcg to get a cpuset cgroup is really ugly and not really
> 100% correct. Memory controller might be disabled and then you do not
> have your association anymore.
 
You are right, for cpuset case, the solution depends on 'CONFIG_MEMCG=y',
and the bright side is most of distribution have it on.

> This all can get quite expensive so the primary question is, does the
> existing behavior generates any real issues or is this more of an
> correctness exercise? I mean it certainly is not great to demote to an
> incompatible numa node but are there any reasonable configurations when
> the demotion target node is explicitly excluded from memory
> policy/cpuset?

We haven't got customer report on this, but there are quite some customers
use cpuset to bind some specific memory nodes to a docker (You've helped
us solve a OOM issue in such cases), so I think it's practical to respect
the cpuset semantics as much as we can.

Your concern about the expensive cost makes sense! Some raw ideas are:
* if the shrink_folio_list is called by kswapd, the folios come from
  the same per-memcg lruvec, so only one check is enough
* if not from kswapd, like called form madvise or DAMON code, we can
  save a memcg cache, and if the next folio's memcg is same as the
  cache, we reuse its result. And due to the locality, the real
  check is rarely performed.

Thanks,
Feng

> -- 
> Michal Hocko
> SUSE Labs
>