Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <9fb22767-54de-d316-7e6b-5aac375c9c49@intel.com>
Date:   Tue, 3 May 2022 16:54:34 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.7.0
Subject: Re: RFC: Memory Tiering Kernel Interfaces
Content-Language: en-US
To:     Alistair Popple <apopple@nvidia.com>
Cc:     Davidlohr Bueso <dave@stgolabs.net>, Wei Xu <weixugc@google.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Huang Ying <ying.huang@intel.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Yang Shi <shy828301@gmail.com>, Linux MM <linux-mm@kvack.org>,
        Greg Thelen <gthelen@google.com>,
        "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
        Jagdish Gediya <jvgediya@linux.ibm.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Michal Hocko <mhocko@kernel.org>,
        Baolin Wang <baolin.wang@linux.alibaba.com>,
        Brice Goglin <brice.goglin@gmail.com>,
        Feng Tang <feng.tang@intel.com>, Jonathan.Cameron@huawei.com
References: <CAAPL-u9sVx94ACSuCVN8V0tKp+AMxiY89cro0japtyB=xNfNBw@mail.gmail.com>
 <20220501175813.tvytoosygtqlh3nn@offworld> <87o80eh65f.fsf@nvdebian.thelocal>
 <e47bf89c-cee7-3006-5c1b-c76f640c3e23@intel.com>
 <87mtfygoxs.fsf@nvdebian.thelocal>
From:   Dave Hansen <dave.hansen@intel.com>
In-Reply-To: <87mtfygoxs.fsf@nvdebian.thelocal>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 5/3/22 15:35, Alistair Popple wrote:
> Not entirely true. The GPUs on POWER9 have performance counters capable of
> collecting this kind of information for memory accessed from the GPU. I will
> admit though that sadly most people probably don't have a P9 sitting under their
> desk :)

Well, x86 CPUs have performance monitoring hardware that can
theoretically collect physical access information too.  But, this
performance monitoring hardware wasn't designed for this specific use
case in mind.  So, in practice, these events (PEBS) weren't very useful
for driving memory tiering.

Are you saying that the GPUs on POWER9 have performance counters that
can drive memory tiering in practice?  I'd be curious if there's working
code to show how they get used.  Maybe the hardware is better than the
x86 PMU or the software consuming it is more clever than what we did.
But, I'd love to see it either way.