Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 19 Aug 2022 15:40:31 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     Kefeng Wang <wangkefeng.wang@huawei.com>
CC:     huang ying <huang.ying.caritas@gmail.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        <linux-mm@kvack.org>, <linux-kernel@vger.kernel.org>,
        Liu Shixin <liushixin2@huawei.com>,
        Huang Ying <ying.huang@intel.com>
Subject: Re: [PATCH RFC] mm, proc: add PcpFree to meminfo
Message-ID: <Yv8+bxiZ8+Rw+wu3@ziqianlu-Dell-Optiplex7000>
References: <20220816084426.135528-1-wangkefeng.wang@huawei.com>
 <CAC=cRTN=vbrgOsH7PNNMV+Coyabp=H_XF99MUL00kfET=K-32w@mail.gmail.com>
 <578084e8-bdc3-6e4e-f390-4202a8bc7c1d@huawei.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <578084e8-bdc3-6e4e-f390-4202a8bc7c1d@huawei.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?YDVMooLLCJ7si5GBk70Nd1hqhJsXhdYMhvBHjJmtvQ9DrYo+ofcmAzQO0R4o?=
 =?us-ascii?Q?FZ0k8hTgZHHrws80F3m0eMPWGr0f5qRKYYOInKwTSIRE+4H8ojOgbDJtSK80?=
 =?us-ascii?Q?ueSFYKIgRbhhNxclRC8i0SyeABEeSPQpniKsK3x7Af1cpio1sCnM3uKpm9v2?=
 =?us-ascii?Q?q3tXIDTKOJM08UTQZAtk+YYER3iMIvnYaXKItL2JxDTdnoKSWdmn8qmXJ/Ej?=
 =?us-ascii?Q?Hx5+s4OMkssAomfS9WRTYycZmw/rmfKDz5dBC8AUwFHiPzyzUmo9FnwCcc9U?=
 =?us-ascii?Q?CxWMLQUnAJY1spg3nALTN5A+YBKYjNfAxpUTmrkmyQGSwe1iAigkfzv91DNx?=
 =?us-ascii?Q?sQ6Mwc/Lzi03HGKfL/zBS5HoZ4d3YuVYnQEOxltR851cRjEXR45RY8yJ31TI?=
 =?us-ascii?Q?sbL0MrWTFIro8/aV/h6Lr6IMSD9KeaobWbZhp0PZbzpQyawAYi2N+4FDA6fD?=
 =?us-ascii?Q?5uhNy4ppP6onAZ8OdYw5NDoXzdX8WD8Bt/XbDzEp+WSBsxGB2nyUJRixAzAO?=
 =?us-ascii?Q?DkkC6bU9XcEOy5qbA32J3z6qSnyYUIKYCYjx8QDtpzDGMoMNvHF9h3NM4yYi?=
 =?us-ascii?Q?4bXkKi8yN40748md4r25CUe7zOa3Lk21bO8oFgjboostxWEO3ie1SI5P7Fb0?=
 =?us-ascii?Q?boojzorJipUmRb05N2j9q0ocu+2FMXfs2BEw133YGvIqDcOBXG7eLozSfRtM?=
 =?us-ascii?Q?RN0E5OpK5cPzanT5c0oblpzOxfiNCngSl6DNczG0p9ng76mezR7DmT5oXxv/?=
 =?us-ascii?Q?LF06ol9IZAWTJ3+vYHcIbzJZqdczzzSlP8A4fUiTdyJyNr1z3cnTfL3NFnRZ?=
 =?us-ascii?Q?2Xfdfs+h4SkTUsJ3UklEj/tq5NiWJUJicktL403WQbCF9Q0/H5NMak7xrzHZ?=
 =?us-ascii?Q?arhgTYgcEfpIuPAUSxPUdrS9xge4o/mpl8ZdTsfzgo4PwyHeTUFmZY/7jjTs?=
 =?us-ascii?Q?CGzfIKIlcHjO9f7efGDgR6iW58VIR/VIUoZ06QO2hQcvhKXKeRdChhLwiZ9U?=
 =?us-ascii?Q?UzEBWjRT47YKeR3slNt9J4zTpq6NBTVjQftWCCMC4RjHnwz1x5/HKmvOoane?=
 =?us-ascii?Q?7NxqyAw234xNJ42Jf2V9Kp4XtmS9mvoTe1xOayBZdYmPkQOhWKHWsFO7saap?=
 =?us-ascii?Q?xzLZO/DYMoxS+8bpkbU82YgfEXI+1fRbWWEGtc6kCK9bXpkmpygrnYoJOECN?=
 =?us-ascii?Q?Muzbd+2icOhFSiXve339503f2Q4wxci08g9Q35VYeGYOzw+g4H25zwYwpxdd?=
 =?us-ascii?Q?CIR/gT93NVhqGDFiVKeM2wVT42rDCWAPXitKLCnINJYbZRR+aTD6q4i6Zs37?=
 =?us-ascii?Q?rltRoqWfxaktiRUWmQw03DB2J0TvWusxk6+m9VZ+skIo7/Myutgwl9Oz4V1Z?=
 =?us-ascii?Q?uygm/Q+a4ZXzI9o4NQG+fKS2xWFW3XQz4OohNgjA10dEUFkhX7N2WtJ7aRQ5?=
 =?us-ascii?Q?bDg5c93mVvM+oIbG6GvTbv7CLBB6ZlZuwPzZ9pL/uY6uWmgfAhAyVnKxVHjK?=
 =?us-ascii?Q?M5l9451dmpY/VaFjNzXz2AFn5Uzx10/zmisVi1XkPJcHMe/ysDpzFUNg3JKM?=
 =?us-ascii?Q?zC6ZLhrAvoKCumhWbhP107HHfeTXVdCXAwPU8gWG?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 7c772bd5-7173-4f58-5aca-08da81b624f5
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Aug 2022 07:40:53.4811
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: t5vO96HO01wtiSOYWnbYW0mjz2sgcvYfabWNt4sG0+tJodPcMMMOfHw507/+R7EZvgs2J8cpWMTsXdPSAlQCsA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB2876
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Aug 16, 2022 at 05:24:07PM +0800, Kefeng Wang wrote:
> 
> On 2022/8/16 16:48, huang ying wrote:
> > On Tue, Aug 16, 2022 at 4:38 PM Kefeng Wang <wangkefeng.wang@huawei.com> wrote:
> > > From: Liu Shixin <liushixin2@huawei.com>
> > > 
> > > The page on pcplist could be used, but not counted into memory free or
> > > avaliable, and pcp_free is only showed by show_mem(). Since commit
> > > d8a759b57035 ("mm, page_alloc: double zone's batchsize"), there is a
> > > significant decrease in the display of free memory, with a large number
> > > of cpus and nodes, the number of pages in the percpu list can be very
> > > large, so it is better to let user to know the pcp count.
> > Can you show some data?
> 
> 80M+ with 72cpus/2node
> 

80M+ for a 2 node system doesn't sound like a significant number.

> > 
> > Another choice is to count PCP free pages in MemFree.  Is that OK for
> > your use case too?
> 
> Yes, the user will make policy according to MemFree, we think count PCP free
> pages
> 
> in MemFree is better, but don't know whether it is right way.
> 

Is there a real problem where user makes a sub-optimal policy due to the
not accounted 80M+ free memory?

Counting PCP pages as free seems natural, since they are indeed free
pages. One concern is, there might be much more calls of
__mod_zone_freepage_state() if you do free page counting for PCP pages,
not sure if that would hurt performance. Also, you will need to
differentiate in __free_one_page() whether counting for free pages are
still needed since some pages are freed through PCP(and thus already
counted) while some are not.

BTW, since commit b92ca18e8ca59("mm/page_alloc: disassociate the pcp->high
from pcp->batch"), pcp size is no longer associated with batch size. Is
it that you are testing on an older kernel?

Thanks,
Aaron

> > > Signed-off-by: Liu Shixin <liushixin2@huawei.com>
> > > Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
> > > ---
> > >   drivers/base/node.c | 14 +++++++++++++-
> > >   fs/proc/meminfo.c   |  9 +++++++++
> > >   2 files changed, 22 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/drivers/base/node.c b/drivers/base/node.c
> > > index eb0f43784c2b..846864e45db6 100644
> > > --- a/drivers/base/node.c
> > > +++ b/drivers/base/node.c
> > > @@ -375,6 +375,9 @@ static ssize_t node_read_meminfo(struct device *dev,
> > >          struct sysinfo i;
> > >          unsigned long sreclaimable, sunreclaimable;
> > >          unsigned long swapcached = 0;
> > > +       unsigned long free_pcp = 0;
> > > +       struct zone *zone;
> > > +       int cpu;
> > > 
> > >          si_meminfo_node(&i, nid);
> > >          sreclaimable = node_page_state_pages(pgdat, NR_SLAB_RECLAIMABLE_B);
> > > @@ -382,9 +385,17 @@ static ssize_t node_read_meminfo(struct device *dev,
> > >   #ifdef CONFIG_SWAP
> > >          swapcached = node_page_state_pages(pgdat, NR_SWAPCACHE);
> > >   #endif
> > > +       for_each_populated_zone(zone) {
> > > +               if (zone_to_nid(zone) != nid)
> > > +                       continue;
> > > +               for_each_online_cpu(cpu)
> > > +                       free_pcp += per_cpu_ptr(zone->per_cpu_pageset, cpu)->count;
> > > +       }
> > > +
> > >          len = sysfs_emit_at(buf, len,
> > >                              "Node %d MemTotal:       %8lu kB\n"
> > >                              "Node %d MemFree:        %8lu kB\n"
> > > +                           "Node %d PcpFree:        %8lu kB\n"
> > >                              "Node %d MemUsed:        %8lu kB\n"
> > >                              "Node %d SwapCached:     %8lu kB\n"
> > >                              "Node %d Active:         %8lu kB\n"
> > > @@ -397,7 +408,8 @@ static ssize_t node_read_meminfo(struct device *dev,
> > >                              "Node %d Mlocked:        %8lu kB\n",
> > >                              nid, K(i.totalram),
> > >                              nid, K(i.freeram),
> > > -                           nid, K(i.totalram - i.freeram),
> > > +                           nid, K(free_pcp),
> > > +                           nid, K(i.totalram - i.freeram - free_pcp),
> > >                              nid, K(swapcached),
> > >                              nid, K(node_page_state(pgdat, NR_ACTIVE_ANON) +
> > >                                     node_page_state(pgdat, NR_ACTIVE_FILE)),
> > > diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c
> > > index 6e89f0e2fd20..672c784dfc8a 100644
> > > --- a/fs/proc/meminfo.c
> > > +++ b/fs/proc/meminfo.c
> > > @@ -38,6 +38,9 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
> > >          unsigned long pages[NR_LRU_LISTS];
> > >          unsigned long sreclaimable, sunreclaim;
> > >          int lru;
> > > +       unsigned long free_pcp = 0;
> > > +       struct zone *zone;
> > > +       int cpu;
> > > 
> > >          si_meminfo(&i);
> > >          si_swapinfo(&i);
> > > @@ -55,8 +58,14 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
> > >          sreclaimable = global_node_page_state_pages(NR_SLAB_RECLAIMABLE_B);
> > >          sunreclaim = global_node_page_state_pages(NR_SLAB_UNRECLAIMABLE_B);
> > > 
> > > +       for_each_populated_zone(zone) {
> > > +               for_each_online_cpu(cpu)
> > > +                       free_pcp += per_cpu_ptr(zone->per_cpu_pageset, cpu)->count;
> > > +       }
> > > +
> > >          show_val_kb(m, "MemTotal:       ", i.totalram);
> > >          show_val_kb(m, "MemFree:        ", i.freeram);
> > > +       show_val_kb(m, "PcpFree:        ", free_pcp);
> > >          show_val_kb(m, "MemAvailable:   ", available);
> > >          show_val_kb(m, "Buffers:        ", i.bufferram);
> > >          show_val_kb(m, "Cached:         ", cached);
> > > --
> > > 2.35.3
> > > 
> > > 
> > .