Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 7 Nov 2022 16:05:37 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     Michal Hocko <mhocko@suse.com>
CC:     "Huang, Ying" <ying.huang@intel.com>,
        Aneesh Kumar K V <aneesh.kumar@linux.ibm.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Tejun Heo <tj@kernel.org>, Zefan Li <lizefan.x@bytedance.com>,
        Waiman Long <longman@redhat.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "Hansen, Dave" <dave.hansen@intel.com>,
        "Chen, Tim C" <tim.c.chen@intel.com>,
        "Yin, Fengwei" <fengwei.yin@intel.com>
Subject: Re: [PATCH] mm/vmscan: respect cpuset policy during page demotion
Message-ID: <Y2i8UbaOjGyqwJQ6@feng-clx>
References: <87wn8lkbk5.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <Y1ou5DGHrEsKnhri@dhcp22.suse.cz>
 <87o7txk963.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <Y1o63SWD2KmQkT3v@dhcp22.suse.cz>
 <87fsf9k3yg.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <Y1p5vaN1AWhpNWZx@dhcp22.suse.cz>
 <87bkpwkg24.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <Y1+J7+1V1nJXF+3b@dhcp22.suse.cz>
 <Y1/XC+witPxFj04T@feng-clx>
 <Y1/cgrgdVP+KdYzf@dhcp22.suse.cz>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <Y1/cgrgdVP+KdYzf@dhcp22.suse.cz>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?lDVXzRdKmM9ZbwlN6C5Fn/vdCGtYPH2jMgUFhokNx53RDFyN1+m/oei2VnXO?=
 =?us-ascii?Q?Sxn9kMgbdZ5lZijp4m/Ib7Y21Pd0PaPLsAjTbpn4bUSgK6Oc0aiaNse55DNt?=
 =?us-ascii?Q?3TDqGwKfK4HUR1OFqGxRiTc/A2icXI1hWvHj3rCVMf9/QvTuk3jeI/lW/+8L?=
 =?us-ascii?Q?lTLnxEhYvfwqZGeuG4p+DgAW+hiy5sWuD8YikV1hLzFQhO5SSfz15FiEkbUo?=
 =?us-ascii?Q?D7ArJC19MM/OG0JKxvey4nnlkAinEbDTZAL6m479KajLjiJJPMzY37KXIa91?=
 =?us-ascii?Q?I4Xw/E+0jSIPbsZE1uusrsEHkJAcIssIaexWu0EAj8bRN93rNgmPIq0sh8gD?=
 =?us-ascii?Q?cH8kcbaNkNLF6cyDgHMFI+mzXUscal14ges9KkGK0Qgqzp2e8TLPfWuUAd6P?=
 =?us-ascii?Q?/uDSl2r+EL5CP3rvKoN7V9vnPZVNwcBc8VBFdPi7VxyHPPInH6lGqK5V27Cb?=
 =?us-ascii?Q?Omh2iI3fhtN52gfZg58k9rHzmzUEpBAHniL6HHQ68f23bdxwxpUYtGMeuJ16?=
 =?us-ascii?Q?USrmMCpxs7C9JW9eB5Ja+7Zo/JzHIn5VvtCvjY0NApMPKUo8hSuNhdy9VK8R?=
 =?us-ascii?Q?cRBJi1uw/2330gDZR1eTSEDBXTgbwMZ8zlZsR9Y4ldfiu+GJV+xKAN1hdk4/?=
 =?us-ascii?Q?R83eoYdgaHNSodr5EkhO+NGCCpWxDNgRV8aUDeBH9dYxJauiFPBAGmRAZhhb?=
 =?us-ascii?Q?hSNxhkmkrxQHhBaLmbQ1l71J28ZNlF7NyLi56LCUhDuWPzqw3xeM6MUoRaik?=
 =?us-ascii?Q?c18t/YOtjnJnY16EyLUYPJkrEKvrwB5jJs3sJ9I3CJmjrj8Oaj3RvGYFHnep?=
 =?us-ascii?Q?9yiS93hs+w04Zcag93hjBNqgwSIwE8aoQVxoWbNYOTkaTzayMBabriO/5Rgk?=
 =?us-ascii?Q?5yYhfLdN3awsJVQEQU2DLl6SeZaWR2P1HJuMLRJh/5eo1xcRWookko0xpt3O?=
 =?us-ascii?Q?9vKJEuxVKyxG+A3/cvESIgxv/0Cww68mNXIL/ykz7prpaaM68eGOzOvL+6Qa?=
 =?us-ascii?Q?jKeha6CT6bNna9veihGhJlVr+OUuIzSPo/C6BhWx0q/54IWp6cDssgfQuFUE?=
 =?us-ascii?Q?KeVWn0kSbF/x7BDdIZU/B1vJX2lRuBGDxZcvVJZmoaCfXONgBhso7RcK2Hsn?=
 =?us-ascii?Q?+b9dIwEMK/bYZYN7LdxppP7C8L5E4ByudYKo0DKALN+NHP/7xykVTd/QIOOx?=
 =?us-ascii?Q?GMDLJZ8ax3RwnlhLQgndTYhjarXTPway7jbuKANndaeh2D+bbtC9/FOhYdGb?=
 =?us-ascii?Q?34hf9eHgBLL/eEBgh9IRdltwaxyNj7D3fdnzg3NMCC/jxFh0DE6Urm467s1Y?=
 =?us-ascii?Q?x3bNKDD4mZ4arFZLfR5eoQOmXNrvUnuBilx/JIvE6HMifrfc+Qxm5tNElfua?=
 =?us-ascii?Q?Ah/1BzjYJODcjNvRP5Kwl5CTkYa3rZILENezmDeVBlAkIKtZCKiAZwscm19I?=
 =?us-ascii?Q?KqwDnnIzrTYwk1VJMEuOhkq9oljKvf1A/scXVaY+GoT1If9yoeo6DxKhEUQ/?=
 =?us-ascii?Q?6ZYeude9BnHPEpJj9ouf0nV8d/HPT3hyPIYBwnu5GsTsIodfuDVr6IvvpEg9?=
 =?us-ascii?Q?WDZWnfn8tyxBoDJc1SmTzp1CqvpvzKWjNqMo15Zo?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5d3ee87c-41b9-47d4-2c0b-08dac0975071
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2022 08:08:55.0409
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 2VdIcJghSfnFXY6ma87f67ufaGIo2FFB616nxIm/2UobPojI1WEQ68jIN2NkdbHpoo4y/3PvALH38tVW3EabSw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB7471
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Oct 31, 2022 at 03:32:34PM +0100, Michal Hocko wrote:
> > > OK, then let's stop any complicated solution right here then. Let's
> > > start simple with a per-mm flag to disable demotion of an address space.
> > > Should there ever be a real demand for a more fine grained solution
> > > let's go further but I do not think we want a half baked solution
> > > without real usecases.
> > 
> > Yes, the concern about the high cost for mempolicy from you and Yang is
> > valid. 
> > 
> > How about the cpuset part?
> 
> Cpusets fall into the same bucket as per task mempolicies wrt costs. Geting a
> cpuset requires knowing all tasks associated with a page. Or am I just
> missing any magic? And no memcg->cpuset association is not a proper
> solution at all.

No, you are not missing anything. It's really difficult to find a
solution for all holes. And the patch is actually a best-efforts
approach, trying to cover cgroup v2 + memory controller enabled case,
which we think is a common user case for newer platforms with tiering
memory.
 
> > We've got bug reports from different channels
> > about using cpuset+docker to control meomry placement in memory tiering
> > system, leading to 2 commits solving them:
> > 
> > 2685027fca38 ("cgroup/cpuset: Remove cpus_allowed/mems_allowed setup in
> > cpuset_init_smp()") 
> > https://lore.kernel.org/all/20220419020958.40419-1-feng.tang@intel.com/
> > 
> > 8ca1b5a49885 ("mm/page_alloc: detect allocation forbidden by cpuset and
> > bail out early")
> > https://lore.kernel.org/all/1632481657-68112-1-git-send-email-feng.tang@intel.com/
> > 
> > >From these bug reports, I think it's reasonable to say there are quite
> > some real world users using cpuset+docker+memory-tiering-system.
> 
> I don't think anybody is questioning existence of those usecases. The
> primary question is whether any of them really require any non-trivial
> (read nodemask aware) demotion policies. In other words do we know of
> cpuset policy setups where demotion fallbacks are (partially) excluded?

For cpuset numa memory binding, there are possible usercases:

* User wants cpuset to bind some important containers to faster
  memory tiers for better latency/performance (where simply disabling
  demotion should work, like your per-mm flag solution)

* User wants to bind to a set of physically closer nodes (like faster
  CPU+DRAM node and slower PMEM node). With initial demotion code,
  our HW will have 1:1 demotion/promotion pair for one DRAM node and
  its closer PMEM node, and user's binding can work fine. And there
  are many other types of memory tiering system from other vendors,
  like many CPU-less DRAM nodes in system, and Aneesh's patchset[1]
  created a more general tiering interface, where IIUC each tier has
  a nodemask, and an upper tier can demote to the whole lower tier,
  where the demotion path is N:N mapping. And for this, fine-tuning
  cpuset nodes binding needs this handling.

[1]. https://lore.kernel.org/lkml/20220818131042.113280-1-aneesh.kumar@linux.ibm.com/

Thanks,
Feng

> -- 
> Michal Hocko
> SUSE Labs