Received-SPF: pass (google.com: domain of linux-kernel+bounces-492-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Precedence: bulk
MIME-Version: 1.0
References: <CAOUHufZeADNp_y=Ng+acmMMgnTR=ZGFZ7z-m6O47O=CmJauWjw@mail.gmail.com>
 <20231208071235.17812-1-henry.hj@antgroup.com>
In-Reply-To: <20231208071235.17812-1-henry.hj@antgroup.com>
From: Yu Zhao <yuzhao@google.com>
Date: Thu, 14 Dec 2023 23:46:29 -0700
Message-ID: <CAOUHufaBfziNTwypP=dxZXYZi4nniT6aYQZiZxzyQjSa3Zmaow@mail.gmail.com>
Subject: Re: [RFC v2] mm: Multi-Gen LRU: fix use mm/page_idle/bitmap
To: Henry Huang <henry.hj@antgroup.com>
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, 
	=?UTF-8?B?6LCI6Ym06ZSL?= <henry.tjf@antgroup.com>, 
	=?UTF-8?B?5pyx6L6JKOiMtuawtCk=?= <teawater@antgroup.com>, 
	akpm@linux-foundation.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Dec 8, 2023 at 12:12=E2=80=AFAM Henry Huang <henry.hj@antgroup.com>=
 wrote:
>
> Thanks for replying this RFC.
>
> > 1. page_idle/bitmap isn't a capable interface at all -- yes, Google
> > proposed the idea [1], but we don't really use it anymore because of
> > its poor scalability.
>
> In our environment, we use /sys/kernel/mm/page_idle/bitmap to check
> pages whether were accessed during a peroid of time.

Is it a production environment? If so, what's your
1. scan interval
2. memory size

I'm trying to understand why scalability isn't a problem for you. On
an average server, there are hundreds of millions of PFNs, so it'd be
very expensive to use that ABI even for a time interval of minutes.

> We manage all pages
> idle time in userspace. Then use a prediction algorithm to select pages
> to reclaim. These pages would more likely be idled for a long time.

"There is a system in place now that is based on a user-space process
that reads a bitmap stored in sysfs, but it has a high CPU and memory
overhead, so a new approach is being tried."
https://lwn.net/Articles/787611/

Could you elaborate how you solved this problem?

> We only need kernel to tell use whether a page is accessed, a boolean
> value in kernel is enough for our case.

How do you define "accessed"? I.e., through page tables or file
descriptors or both?

> > 2. PG_idle/young, being a boolean value, has poor granularity. If
> > anyone must use page_idle/bitmap for some specific reason, I'd
> > recommend exporting generation numbers instead.
>
> Yes, at first time, we try using multi-gen LRU proactvie scan and
> exporting generation&refs number to do the same thing.
>
> But there are serveral problems:
>
> 1. multi-gen LRU only care about self-memcg pages. In our environment,
> it's likely to see that different memcg's process share pages.

This is related to my question above: are those pages mapped into
different memcgs or not?

> multi-gen LRU only update gen of pages in current memcg. It's hard to
> judge a page whether is accessed depends on gen update.

This depends. I'd be glad to elaborate after you clarify the above.

> We still have no ideas how to solve this problem.
>
> 2. We set swappiness 0, and use proactive scan to select cold pages
> & proactive reclaim to swap anon pages. But we can't control passive
> scan(can_swap =3D false), which would make anon pages cold/hot inversion
> in inc_min_seq.

There is an option to prevent the inversion, IIUC, the force_scan
option is what you are looking for.