Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Sun, 14 Mar 2021 19:04:45 -0600
From:   Yu Zhao <yuzhao@google.com>
To:     Zi Yan <ziy@nvidia.com>
Cc:     Matthew Wilcox <willy@infradead.org>, linux-mm@kvack.org,
        Alex Shi <alex.shi@linux.alibaba.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Hillf Danton <hdanton@sina.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Joonsoo Kim <iamjoonsoo.kim@lge.com>,
        Mel Gorman <mgorman@suse.de>, Michal Hocko <mhocko@suse.com>,
        Roman Gushchin <guro@fb.com>, Vlastimil Babka <vbabka@suse.cz>,
        Wei Yang <richard.weiyang@linux.alibaba.com>,
        Yang Shi <shy828301@gmail.com>,
        Ying Huang <ying.huang@intel.com>,
        linux-kernel@vger.kernel.org, page-reclaim@google.com
Subject: Re: [PATCH v1 06/14] mm, x86: support the access bit on non-leaf PMD
 entries
Message-ID: <YE6yrQC1Ps195wPw@google.com>
References: <20210313075747.3781593-1-yuzhao@google.com>
 <20210313075747.3781593-7-yuzhao@google.com>
 <74C83FFE-DC78-40CD-B6BE-00614DC8F125@nvidia.com>
 <20210314225103.GQ2577561@casper.infradead.org>
 <YE6kacDMLga/kfvf@google.com>
 <3465DA3A-B735-4657-A1E0-89CF5DE0D4FF@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <3465DA3A-B735-4657-A1E0-89CF5DE0D4FF@nvidia.com>
Precedence: bulk

On Sun, Mar 14, 2021 at 08:27:29PM -0400, Zi Yan wrote:
> On 14 Mar 2021, at 20:03, Yu Zhao wrote:
> 
> > On Sun, Mar 14, 2021 at 10:51:03PM +0000, Matthew Wilcox wrote:
> >> On Sun, Mar 14, 2021 at 06:12:42PM -0400, Zi Yan wrote:
> >>> On 13 Mar 2021, at 2:57, Yu Zhao wrote:
> >>>
> >>>> Some architectures support the accessed bit on non-leaf PMD entries
> >>>> (parents) in addition to leaf PTE entries (children) where pages are
> >>>> mapped, e.g., x86_64 sets the accessed bit on a parent when using it
> >>>> as part of linear-address translation [1]. Page table walkers who are
> >>>> interested in the accessed bit on children can take advantage of this:
> >>>> they do not need to search the children when the accessed bit is not
> >>>> set on a parent, given that they have previously cleared the accessed
> >>>> bit on this parent in addition to its children.
> >>>>
> >>>> [1]: Intel 64 and IA-32 Architectures Software Developer's Manual
> >>>>      Volume 3 (October 2019), section 4.8
> >>>
> >>> Just curious. Does this also apply to non-leaf PUD entries? Do you
> >>> mind sharing which sentence from the manual gives the information?
> >>
> >> The first few sentences from 4.8:
> >>
> >> : For any paging-structure entry that is used during linear-address
> >> : translation, bit 5 is the accessed flag. For paging-structure
> >> : entries that map a page (as opposed to referencing another paging
> >> : structure), bit 6 is the dirty flag. These flags are provided for
> >> : use by memory-management software to manage the transfer of pages and
> >> : paging structures into and out of physical memory.
> >>
> >> : Whenever the processor uses a paging-structure entry as part of
> >> : linear-address translation, it sets the accessed flag in that entry
> >> : (if it is not already set).
> 
> Matthew, thanks for the pointer.
> 
> >
> > As far as I know x86 is the one that supports this.
> >
> >> The way they differentiate between the A and D bits makes it clear to
> >> me that the A bit is set at each level of the tree, but the D bit is
> >> only set on leaf entries.
> >
> > And the difference makes perfect sense (to me). Kudos to Intel.
> 
> Hi Yu,
> 
> You only introduced HAVE_ARCH_PARENT_PMD_YOUNG but no HAVE_ARCH_PARENT_PUD_YOUNG.
> Is it PUD granularity too large to be useful for multigenerational LRU algorithm?

Oh, sorry. I overlooked this part of the question.

Yes, you are right. We found no measurable performance difference
between using and not using the accessed bit on non-leaf PUD entries.

For the PMD case, the difference is tiny but still measurable on small
systems, e.g., laptops with 4GB memory. It's clear (a few percent in
kswapd) on servers with tens of GBs of 4KB pages.