Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
References: <20230721012932.190742-1-ying.huang@intel.com>
 <20230721012932.190742-2-ying.huang@intel.com>
 <87r0owzqdc.fsf@nvdebian.thelocal>
 <87r0owy95t.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <87sf9cxupz.fsf@nvdebian.thelocal>
 <878rb3xh2x.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-agent: mu4e 1.8.13; emacs 28.2
From:   Alistair Popple <apopple@nvidia.com>
To:     "Huang, Ying" <ying.huang@intel.com>
Cc:     Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org,
        nvdimm@lists.linux.dev, linux-acpi@vger.kernel.org,
        "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>,
        Wei Xu <weixugc@google.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Michal Hocko <mhocko@kernel.org>,
        Yang Shi <shy828301@gmail.com>,
        Rafael J Wysocki <rafael.j.wysocki@intel.com>,
        Dave Jiang <dave.jiang@intel.com>
Subject: Re: [PATCH RESEND 1/4] memory tiering: add abstract distance
 calculation algorithms management
Date:   Thu, 27 Jul 2023 13:42:15 +1000
In-reply-to: <878rb3xh2x.fsf@yhuang6-desk2.ccr.corp.intel.com>
Message-ID: <87351axbk6.fsf@nvdebian.thelocal>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?4xpDh6mKPCHPqauXQ8uVlb3A3RAOaod2KaP5DsOpCGaMbRrIixctj4cY1g6c?=
 =?us-ascii?Q?/BDZqWtqjwsz8ASV1kAdo6lnvuBotTR9Hc1VlB78Fhu+RZmjvKuMauU7hTnj?=
 =?us-ascii?Q?FTk7ieYWnYN1Cy2Jfuywk/9WA9EkuYPTAysttA709b02z1+XJcH0FDGYiOlb?=
 =?us-ascii?Q?VZ94TimJsRXnL3KNu+pEqREHiZQ5UlJMx2knAX0DRypcfeNRYzN/0tdwbWTi?=
 =?us-ascii?Q?B9MZTMFHH/HFcWZnt3WNC9CkNU+qIGDnIzBgVAOh9Jx3qiL5rzh+sQ9L2d8z?=
 =?us-ascii?Q?9sSTvn1QVjoICP8fUc69c9wEL7/3QwIBflFT/qSMQQ5h5iIwaaDlfheFZaqm?=
 =?us-ascii?Q?WGbkKtIXiaYKkQiuvR8XXv+Q4cSNRvmk5JccT+PL20K+8Dn48JvEs/XhTzX+?=
 =?us-ascii?Q?QgLbkaOzrAo/pxhhH2KPwWxfB+QZh5tDzv3mEtOWGZiSlMrl+aziDCvs/AGa?=
 =?us-ascii?Q?EIRt+wfCHfPYvBGgtc5R0896DSE2Q26lE5Ozed6Dh1LEwWh4XoNDorccbz44?=
 =?us-ascii?Q?Vp1IP7WVsYfAX9aZBUSYdNvclc5W/zIfkanr85+2n3OIr515ulHf+roia/jE?=
 =?us-ascii?Q?4sGdUDkBuRRVnQuLOLXaC7zWZ0oyMDPl1eSoGjIlSqpGrQ7Pte+QJa6vHWHp?=
 =?us-ascii?Q?FO+VEfdO1Q+c/Qzjjd6IP/rojNJTttg5rm/UxmzvV7hBZJ8kTD5/akdMvVsl?=
 =?us-ascii?Q?+S7P+zGgtL0du/++uUqShdRlCKFaeCAWsFkQTK9LTdXNVcXkPVg3xy0Drlui?=
 =?us-ascii?Q?pzwCDdTofcuFCJF6BsNev9dfqSqLBwpib/AqjxYr5FK0zkT1x8KMOJIzC0DF?=
 =?us-ascii?Q?jxfWBGORInLF7/iccuMcMizcLTsxgfWMjCMqDSSKnUNmZzk4l/qsLCd0Q6Yv?=
 =?us-ascii?Q?DplzDakhMAcJIO8E1k1g2bw0OeH6j6rl+IC8pLG7exmbyQGX0T/kiPe23S5m?=
 =?us-ascii?Q?YAA39R0Dlk015noR+7Je8n3esMaNhsqbbAcFTjd7N9asulhtuZuPnsx0lfvY?=
 =?us-ascii?Q?CldKENHdFWOnPkG72umu993+Md/X7TPHUoXtlsLO0OKOnrRw+FxKubKuI+aq?=
 =?us-ascii?Q?BfI6zlhMQlryOARWP9wqk+b5FrkowiJjFzi3wC6n3xy4M2gF90oKxNWvQY2z?=
 =?us-ascii?Q?/dItk1JdBoKKBYVQLD+tL8mbYCSRhxnAqtpJHgOOgKB2oucEnqShwuLSTD5i?=
 =?us-ascii?Q?EZ6TzNVoetH4yTxHyeLLdQJ0M4PeMUgRgcQ6BImPk2QK8396T4HcpCaWYcbp?=
 =?us-ascii?Q?91SgsXqxxjZ/uwj/Dl81XyukL6ghueA03EnpEKtA6HfipldsiEB6NMvCtgNd?=
 =?us-ascii?Q?FWKLQPAVzppJb7bcdQSDPlfi4sAHpjbdp8B4ANTP4I7szG9GW5Tp/s5vtoBn?=
 =?us-ascii?Q?WW+1I34suIcg13gH1jI5lkewJAojhzr66bGooOdmjUv5brWvtoGGJMTByjrF?=
 =?us-ascii?Q?W45Dyv9GR8Lfo4NtphW+oNiQj3WI0FTHohtibZkz/wT6mhVmC443qwVqdXJp?=
 =?us-ascii?Q?4ebobsLrOgfjkhqWMoPeMw5Cs/feKP74oZ8N52Mpuk5/SpsPZgSkx/qyL3e3?=
 =?us-ascii?Q?oHByVaLIHCXmjaOcNgVgTS9BKZEeKzWkgzlNlLNE?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f807bd33-a63c-4c4a-6527-08db8e53dc44
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jul 2023 03:45:03.3497
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Z3OBLFhEfs10b8EeheWxo9TH8l1SQtVimSu8g5UroeZzgZhAVI1LlFzFcg6svMOAPpk5NFBjYIm2J8O+Trj2QA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7837
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


"Huang, Ying" <ying.huang@intel.com> writes:

>>> The other way (suggested by this series) is to make dax/kmem call a
>>> notifier chain, then CXL CDAT or ACPI HMAT can identify the type of
>>> device and calculate the distance if the type is correct for them.  I
>>> don't think that it's good to make dax/kem to know every possible
>>> types of memory devices.
>>
>> Do we expect there to be lots of different types of memory devices
>> sharing a common dax/kmem driver though? Must admit I'm coming from a
>> GPU background where we'd expect each type of device to have it's own
>> driver anyway so wasn't expecting different types of memory devices to
>> be handled by the same driver.
>
> Now, dax/kmem.c is used for
>
> - PMEM (Optane DCPMM, or AEP)
> - CXL.mem
> - HBM (attached to CPU)

Thanks a lot for the background! I will admit to having a faily narrow
focus here.

>>> And, I don't think that we are forced to use the general notifier
>>> chain interface in all memory device drivers.  If the memory device
>>> driver has better understanding of the memory device, it can use other
>>> way to determine abstract distance.  For example, a CXL memory device
>>> driver can identify abstract distance by itself.  While other memory
>>> device drivers can use the general notifier chain interface at the
>>> same time.
>>
>> Whilst I think personally I would find that flexibility useful I am
>> concerned it means every driver will just end up divining it's own
>> distance rather than ensuring data in HMAT/CDAT/etc. is correct. That
>> would kind of defeat the purpose of it all then.
>
> But we have no way to enforce that too.

Enforce that HMAT/CDAT/etc. is correct? Agree we can't enforce it, but
we can influence it. If drivers can easily ignore the notifier chain and
do their own thing that's what will happen.

>>> While other memory device drivers can use the general notifier chain
>>> interface at the same time.

How would that work in practice though? The abstract distance as far as
I can tell doesn't have any meaning other than establishing preferences
for memory demotion order. Therefore all calculations are relative to
the rest of the calculations on the system. So if a driver does it's own
thing how does it choose a sensible distance? IHMO the value here is in
coordinating all that through a standard interface, whether that is HMAT
or something else.

 - Alistair