Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CAAPL-u9sVx94ACSuCVN8V0tKp+AMxiY89cro0japtyB=xNfNBw@mail.gmail.com>
 <20220501175813.tvytoosygtqlh3nn@offworld> <87o80eh65f.fsf@nvdebian.thelocal>
 <e47bf89c-cee7-3006-5c1b-c76f640c3e23@intel.com> <87mtfygoxs.fsf@nvdebian.thelocal>
 <9fb22767-54de-d316-7e6b-5aac375c9c49@intel.com>
In-Reply-To: <9fb22767-54de-d316-7e6b-5aac375c9c49@intel.com>
From:   Wei Xu <weixugc@google.com>
Date:   Tue, 3 May 2022 18:31:30 -0700
Message-ID: <CAAPL-u9UkG1WMQm5u1eAe+4-d04aqVAXc9H0gLSegYoHctkCnw@mail.gmail.com>
Subject: Re: RFC: Memory Tiering Kernel Interfaces
To:     Dave Hansen <dave.hansen@intel.com>
Cc:     Alistair Popple <apopple@nvidia.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Andrew Morton <akpm@linux-foundation.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Huang Ying <ying.huang@intel.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Yang Shi <shy828301@gmail.com>, Linux MM <linux-mm@kvack.org>,
        Greg Thelen <gthelen@google.com>,
        "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
        Jagdish Gediya <jvgediya@linux.ibm.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Michal Hocko <mhocko@kernel.org>,
        Baolin Wang <baolin.wang@linux.alibaba.com>,
        Brice Goglin <brice.goglin@gmail.com>,
        Feng Tang <feng.tang@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Tue, May 3, 2022 at 4:54 PM Dave Hansen <dave.hansen@intel.com> wrote:
>
> On 5/3/22 15:35, Alistair Popple wrote:
> > Not entirely true. The GPUs on POWER9 have performance counters capable of
> > collecting this kind of information for memory accessed from the GPU. I will
> > admit though that sadly most people probably don't have a P9 sitting under their
> > desk :)
>
> Well, x86 CPUs have performance monitoring hardware that can
> theoretically collect physical access information too.  But, this
> performance monitoring hardware wasn't designed for this specific use
> case in mind.  So, in practice, these events (PEBS) weren't very useful
> for driving memory tiering.

The PEBS events without any filtering might not be useful for memory
tiering, but the PEBS events with hardware-based data source filtering
can be useful in driving promotions in memory tiering. Certainly,
because these events are not designed for this specific use case in
mind, there are inefficiencies using them for memory tiering, e.g.
instead of just getting a heat counter for each hot page, we can get
events repeatedly on the hot pages.

> Are you saying that the GPUs on POWER9 have performance counters that
> can drive memory tiering in practice?  I'd be curious if there's working
> code to show how they get used.  Maybe the hardware is better than the
> x86 PMU or the software consuming it is more clever than what we did.
> But, I'd love to see it either way.