Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <e47bf89c-cee7-3006-5c1b-c76f640c3e23@intel.com>
Date:   Tue, 3 May 2022 10:47:20 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.7.0
Subject: Re: RFC: Memory Tiering Kernel Interfaces
Content-Language: en-US
To:     Alistair Popple <apopple@nvidia.com>,
        Davidlohr Bueso <dave@stgolabs.net>
Cc:     Wei Xu <weixugc@google.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Huang Ying <ying.huang@intel.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Yang Shi <shy828301@gmail.com>, Linux MM <linux-mm@kvack.org>,
        Greg Thelen <gthelen@google.com>,
        "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
        Jagdish Gediya <jvgediya@linux.ibm.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Michal Hocko <mhocko@kernel.org>,
        Baolin Wang <baolin.wang@linux.alibaba.com>,
        Brice Goglin <brice.goglin@gmail.com>,
        Feng Tang <feng.tang@intel.com>, Jonathan.Cameron@huawei.com
References: <CAAPL-u9sVx94ACSuCVN8V0tKp+AMxiY89cro0japtyB=xNfNBw@mail.gmail.com>
 <20220501175813.tvytoosygtqlh3nn@offworld> <87o80eh65f.fsf@nvdebian.thelocal>
From:   Dave Hansen <dave.hansen@intel.com>
In-Reply-To: <87o80eh65f.fsf@nvdebian.thelocal>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 5/3/22 10:14, Alistair Popple wrote:
> I would certainly be interested in figuring out how HW could provide some sort
> of heatmap to identify which pages are hot and which processing unit is using
> them. Currently for these systems users have to manually assign memory policy to
> get any reasonable performance, both to disable NUMA balancing and make sure
> memory is allocated on the right node.

Autonuma-induced page faults are a total non-starter for lots of
workloads, even ignoring GPUs.  Basically anyone who is latency
sensitive stays far, far away from autonuma.

As for improving on page faults for data collection...

*Can* hardware provide this information?  Definitely.

Have hardware vendors been motivated enough to add hardware to do this?
 Nope, not yet.

Do you know anyone that works for any hardware companies? ;)

Seriously, though.  Folks at Intel _are_ thinking about this problem.
I'm hoping we have hardware some day to help lend a hand.  The more
hardware vendors that do this, the more likely it is that we'll have
good kernel code to consume data from the hardware.