Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
MIME-Version: 1.0
References: <20220309021230.721028-1-yuzhao@google.com> <20220309021230.721028-8-yuzhao@google.com>
 <CAGsJ_4wVA6G42y4Oj7ToaCoroZTRbS-tU606ELwra3_Sfrgo7w@mail.gmail.com>
 <CAOUHufbUdoWiBF5x2ZeK104jT39zAuuU0xXKZ51eaU6P8SNUyg@mail.gmail.com> <CAGsJ_4z=vkub4e5J5ggsTN=YN4qsqmXSOpf_gu993oOGreLjPg@mail.gmail.com>
In-Reply-To: <CAGsJ_4z=vkub4e5J5ggsTN=YN4qsqmXSOpf_gu993oOGreLjPg@mail.gmail.com>
From:   Yu Zhao <yuzhao@google.com>
Date:   Thu, 7 Apr 2022 17:51:50 -0600
Message-ID: <CAOUHufbU=Uov54da8u9AqEuP0Y-1T2Khgd5Br-AhTfP=jbM1mA@mail.gmail.com>
Subject: Re: [PATCH v9 07/14] mm: multi-gen LRU: exploit locality in rmap
To:     Barry Song <21cnbao@gmail.com>
Cc:     Andrew Morton <akpm@linux-foundation.org>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Andi Kleen <ak@linux.intel.com>,
        Aneesh Kumar <aneesh.kumar@linux.ibm.com>,
        Catalin Marinas <catalin.marinas@arm.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Hillf Danton <hdanton@sina.com>, Jens Axboe <axboe@kernel.dk>,
        Jesse Barnes <jsbarnes@google.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Jonathan Corbet <corbet@lwn.net>,
        Matthew Wilcox <willy@infradead.org>,
        Mel Gorman <mgorman@suse.de>,
        Michael Larabel <Michael@michaellarabel.com>,
        Michal Hocko <mhocko@kernel.org>,
        Mike Rapoport <rppt@kernel.org>,
        Rik van Riel <riel@surriel.com>,
        Vlastimil Babka <vbabka@suse.cz>,
        Will Deacon <will@kernel.org>,
        Ying Huang <ying.huang@intel.com>,
        LAK <linux-arm-kernel@lists.infradead.org>,
        Linux Doc Mailing List <linux-doc@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Linux-MM <linux-mm@kvack.org>,
        Kernel Page Reclaim v2 <page-reclaim@google.com>,
        x86 <x86@kernel.org>, Brian Geffon <bgeffon@google.com>,
        Jan Alexander Steffens <heftig@archlinux.org>,
        Oleksandr Natalenko <oleksandr@natalenko.name>,
        Steven Barrett <steven@liquorix.net>,
        Suleiman Souhlal <suleiman@google.com>,
        Daniel Byrne <djbyrne@mtu.edu>,
        Donald Carr <d@chaos-reins.com>,
        =?UTF-8?Q?Holger_Hoffst=C3=A4tte?= <holger@applied-asynchrony.com>,
        Konstantin Kharlamov <Hi-Angel@yandex.ru>,
        Shuang Zhai <szhai2@cs.rochester.edu>,
        Sofia Trinh <sofia.trinh@edi.works>,
        Vaibhav Jain <vaibhav@linux.ibm.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Wed, Apr 6, 2022 at 9:46 PM Barry Song <21cnbao@gmail.com> wrote:
>
> On Thu, Apr 7, 2022 at 3:04 PM Yu Zhao <yuzhao@google.com> wrote:
> >
> > On Wed, Apr 6, 2022 at 8:29 PM Barry Song <21cnbao@gmail.com> wrote:
> > >
> > > On Wed, Mar 9, 2022 at 3:48 PM Yu Zhao <yuzhao@google.com> wrote:
> > > >
> > > > Searching the rmap for PTEs mapping each page on an LRU list (to test
> > > > and clear the accessed bit) can be expensive because pages from
> > > > different VMAs (PA space) are not cache friendly to the rmap (VA
> > > > space). For workloads mostly using mapped pages, the rmap has a high
> > > > CPU cost in the reclaim path.
> > > >
> > > > This patch exploits spatial locality to reduce the trips into the
> > > > rmap. When shrink_page_list() walks the rmap and finds a young PTE, a
> > > > new function lru_gen_look_around() scans at most BITS_PER_LONG-1
> > > > adjacent PTEs. On finding another young PTE, it clears the accessed
> > > > bit and updates the gen counter of the page mapped by this PTE to
> > > > (max_seq%MAX_NR_GENS)+1.
> > >
> > > Hi Yu,
> > > It seems an interesting feature to save the cost of rmap. but will it lead to
> > > possible judging of cold pages as hot pages?
> > > In case a page is mapped by 20 processes,  and it has been accessed
> > > by 5 of them, when we look around one of the 5 processes, the page
> > > will be young and this pte is cleared. but we still have 4 ptes which are not
> > > cleared. then we don't access the page for a long time, but the 4 uncleared
> > > PTEs will still make the page "hot" since they are not cleared, we will find
> > > the page is hot either due to look-arounding the 4 processes or rmapping
> > > the page later?
> >
> > Why are the remaining 4 accessed PTEs skipped? The rmap should check
> > all the 20 PTEs.
>
> for example page A is the neighbour of page B in process 1, when we do rmap
> for B, we look-around and clear A's pte in process 1. but A's ptes are
> still set in
> process 2,3,4,5.

It makes no difference because it's too insignificant. The goal is not
to give several million pages unique timestamps and sort them; it's to
partition pages on the orders one tenth to a few seconds and quickly
find some reasonable candidates. Temporal locality gets weaker
exponentially over time. Even on small systems, the difference is not
measurable if several thousand pages used in the last few seconds are
chosen over another several thousand pages used in the last minute.