Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <ecb2ab76-8b48-13a6-559a-21f73fcf6cea@intel.com>
Date:   Thu, 5 May 2022 07:24:26 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.7.0
Subject: Re: RFC: Memory Tiering Kernel Interfaces
Content-Language: en-US
To:     Wei Xu <weixugc@google.com>
Cc:     Alistair Popple <apopple@nvidia.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Andrew Morton <akpm@linux-foundation.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Huang Ying <ying.huang@intel.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Yang Shi <shy828301@gmail.com>, Linux MM <linux-mm@kvack.org>,
        Greg Thelen <gthelen@google.com>,
        "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
        Jagdish Gediya <jvgediya@linux.ibm.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Michal Hocko <mhocko@kernel.org>,
        Baolin Wang <baolin.wang@linux.alibaba.com>,
        Brice Goglin <brice.goglin@gmail.com>,
        Feng Tang <feng.tang@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>
References: <CAAPL-u9sVx94ACSuCVN8V0tKp+AMxiY89cro0japtyB=xNfNBw@mail.gmail.com>
 <20220501175813.tvytoosygtqlh3nn@offworld> <87o80eh65f.fsf@nvdebian.thelocal>
 <e47bf89c-cee7-3006-5c1b-c76f640c3e23@intel.com>
 <87mtfygoxs.fsf@nvdebian.thelocal>
 <9fb22767-54de-d316-7e6b-5aac375c9c49@intel.com>
 <CAAPL-u9UkG1WMQm5u1eAe+4-d04aqVAXc9H0gLSegYoHctkCnw@mail.gmail.com>
 <52541497-c097-5a51-4718-feed13660255@intel.com>
 <CAAPL-u_G8dzXCjx=xcnemgXGQxwnVUFEy_NJzHsJOHapUm-9Ew@mail.gmail.com>
From:   Dave Hansen <dave.hansen@intel.com>
In-Reply-To: <CAAPL-u_G8dzXCjx=xcnemgXGQxwnVUFEy_NJzHsJOHapUm-9Ew@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 5/4/22 23:35, Wei Xu wrote:
> On Wed, May 4, 2022 at 10:02 AM Dave Hansen <dave.hansen@intel.com> wrote:
>> That means a lot of page table and EPT walks to map those linear
>> addresses back to physical.  That adds to the inefficiency.
> 
> That's true if the tracking is purely based on physical pages.  For
> hot page tracking from PEBS, we can consider tracking in
> virtual/linear addresses.  We don't need to maintain the history for
> all linear page addresses nor for an indefinite amount of time.  After
> all, we just need to identify pages accessed frequently recently and
> promote them.

Except that you don't want to promote on *every* access.  That might
lead to too much churn.

You're also assuming that all accesses to a physical page are via a
single linear address, which ignores shared memory mapped at different
linear addresses.  Our (maybe wrong) assumption has been that shared
memory is important enough to manage that it can't be ignored.

>> In the end, you get big PEBS buffers with lots of irrelevant data that
>> needs significant post-processing to make sense of it.
> 
> I am curious about what are "lots of irrelevant data" if PEBS data is
> filtered on data sources (e.g. DRAM vs PMEM) by hardware.  If we need
> to have different policies for the pages from the same data source,
> then I agree that the software has to do a lot of filtering work.

Perhaps "irrelevant" was a bad term to use.  I meant that you can't just
take the PEBS data and act directly on it.  It has to be post-processed
and you will see things in there like lots of adjacent accesses to a
page.  Those additional accesses can be interesting but at some point
you have all the weight you need to promote the page and the _rest_ are
irrelevant.

>> The folks at Intel that tried this really struggled to take this mess and turn it into a successful hot-page tracking.
>>
>> Maybe someone else will find a better way to do it, but we tried and
>> gave up.
> 
> It might be challenging to use PEBS as the only and universal hot page
> tracking hardware mechanism. For example, there are challenges to use
> PEBS to sample KVM guest accesses from the host.

Yep, agreed.  This aspect of the hardware is very painful at the moment.

> On the other hand, PEBS with hardware-based data source filtering can
> be a useful mechanism to improve hot page tracking in conjunction
> with other techniques.

Rather than "can", I'd say: "might".  Backing up to what I said originally:

> So, in practice, these events (PEBS) weren't very useful
> for driving memory tiering.

By "driving" I really meant solely driving.  Like, can PEBS be used as
the one and only mechanism?  We couldn't make it work.  But, the
hardware _is_ sitting there mostly unused.  It might be great to augment
what is there, and nobody should be discouraged from looking at it again.