Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <07912a0d-eb91-a6ef-2b9d-74593805f29e@amd.com>
Date:   Fri, 28 Oct 2022 19:23:33 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.11.0
Subject: Re: [RFC] memory tiering: use small chunk size and more tiers
Content-Language: en-US
To:     "Huang, Ying" <ying.huang@intel.com>
Cc:     Aneesh Kumar K V <aneesh.kumar@linux.ibm.com>, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org,
        Andrew Morton <akpm@linux-foundation.org>,
        Alistair Popple <apopple@nvidia.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Hesham Almatary <hesham.almatary@huawei.com>,
        Jagdish Gediya <jvgediya.oss@gmail.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Michal Hocko <mhocko@kernel.org>,
        Tim Chen <tim.c.chen@intel.com>, Wei Xu <weixugc@google.com>,
        Yang Shi <shy828301@gmail.com>
References: <20221027065925.476955-1-ying.huang@intel.com>
 <578c9b89-10eb-1e23-8868-cdd6685d8d4e@linux.ibm.com>
 <877d0kk5uf.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <59291b98-6907-0acf-df11-6d87681027cc@linux.ibm.com>
 <8735b8jy9k.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <0d938c9f-c810-b10a-e489-c2b312475c52@amd.com>
 <87tu3oibyr.fsf@yhuang6-desk2.ccr.corp.intel.com>
From:   Bharata B Rao <bharata@amd.com>
In-Reply-To: <87tu3oibyr.fsf@yhuang6-desk2.ccr.corp.intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UTYwT1BHWnJ6aTNmREJ6VjJnK2k2NlpncnF6YmRnYlY2NTh2QzFFbHd3em9h?=
 =?utf-8?B?bFpJbjVPTEFCeWNUdHpYQkwyWUxIYVFUbWU2Wm5MN2liUUtLZFJxZlkzOUZQ?=
 =?utf-8?B?aFFDUDZiL2hadmJ2NllhelBXYnVpdVdLZ0lJRFA0djdZUWJBRkh3cldCTVVp?=
 =?utf-8?B?em9BV3pEc1lndXNVRGlmcWNrVWx5cTYvK2tLckFod2pTMVl5blhWN0tBNEpa?=
 =?utf-8?B?d0FsNzdOSkpCSXh0VjRaTlZVVjA1WGhicXFtbFdsbjFrOTdIY3YrZ1NvRkh0?=
 =?utf-8?B?RzhIRDZGMnIvWUFRNlpDRXROUUVzOG5JSStGMVE1VU1xMHgxdGd6L0FDY2xs?=
 =?utf-8?B?T0RxQUxFbmdHSU1wU2JMUEZ1Tmg3QzZoUmZ2cVBIeW9oZnF6c1FockhMcXcz?=
 =?utf-8?B?a0VkQXlNdDBpZnJMaXdCeW9hOEtHakxwRTNPbEhVeTd1TTE5NEtuUGZaUTM2?=
 =?utf-8?B?VEdGNDhyRUM1V2t3dGI0U1k2OW1WQWJMSlNDUDFjbFRVNXltamdSZlB2TWpx?=
 =?utf-8?B?cGVtSkp5NmlOK1oyVWZvaDJlYXRQbnJ3WDU5QTduMkl1Vlk4WlVVYkVUQW43?=
 =?utf-8?B?Vk1raVB0ay9CeklzRlYrYUdPSytveHRockt6SUhlbXN4TzJHRk50VmFvZ3Jz?=
 =?utf-8?B?RHVCb2VCMU5tV2RWY3RNRDgrV1JDdG1TN1BiVzdjNDlKaC9xSEw4NmJodUdG?=
 =?utf-8?B?Qlowd1dkY1BJTzV3bEh6OUVvTjJ0OHlyWXlmWGFwZEwrcjNjQUxtRzBWL1hp?=
 =?utf-8?B?cW9KbnNTSGUrbWNpSVBldlhLWm56YnF3UldUT290Y0UrcUVJS0wzVTVuUmhu?=
 =?utf-8?B?Wml1M1o0QlAybThmN1l4ZE9hRGtGdlNyNnhZNE03UmRaRnp4cnExMXlhVmJU?=
 =?utf-8?B?SjY5MkhLdWMyeUI3SDd6SU41dTBaYVNOVVk0VSsvUHpQKzNqYW5BWTlGZ0JF?=
 =?utf-8?B?clg3QytBbm40VkE4a01rU2lIRlpGRnJTa2JFZW5mb0lLeUk0V0hjdXJWTm1F?=
 =?utf-8?B?ZVZRUDVJQVY5Z0FIUTJjbU9qQzRXSHB5bGFPUzB6WDdTZHB5ZWNWYkl2VWVD?=
 =?utf-8?B?eGZ6c08xYkFiR21sU0NFbTFHVFlmK3p2dVNBTnhFODFyMStFWWFtRkQ5ZHpq?=
 =?utf-8?B?RFNFRnBXQmNDSk5FbHUra0lVYzFndndSdjdKN1owdy9ZY2dQZEVGb08vWWc1?=
 =?utf-8?B?ZnJCLy9NeFhtNEV5RXh0WlFIM0hJMkVFMjl6TnpoWHFaVTIyUHdpaExocFRC?=
 =?utf-8?B?V1BIZjFYK2hxWWt2YVlvbFBvRDI0VkhDSjk5d1V4MDhJcHFlNHVHUlcyOEdq?=
 =?utf-8?B?VHk3TUtUTXJ3WUhjbFlHdWZIa0dmcEU5ZmpraUErVDBHWDNHdVRhdytTcGlU?=
 =?utf-8?B?ZGsvNi9uT2xxeWd6MnZoYW5Ca1JRaFBUelVDN2pIZzRKRlo3aHB1NFljL2VN?=
 =?utf-8?B?RHpjU25zK0R6Y0lyMEZlWG9EcmN2ODJvMUhJcjRnbjFxcWRhejlhUC9XaWYy?=
 =?utf-8?B?OVF4ZGJzQ3JETTEwSWRmSzc5WXR1R3dkakplU0w2a1FVN1VjRFJZd1ZaSkdq?=
 =?utf-8?B?SnFyZ0lXNHluSmlNdGlSSnhLZjNZVXIycFpRaFhkUTMrd2FLckNES0RVa1Zp?=
 =?utf-8?B?M291eXl0V1MyeTIzRC9oMlhnK3NrVXFxd1VDSmxqaUY5cFdvS2Y0OHJiN1Zj?=
 =?utf-8?B?K0xhQzMyZU5wcithL3VLbnJLRGlZcGNuUWpFdmhXSDd2aEFpanArdG1PMCsx?=
 =?utf-8?B?Qmc3a1l0b01ucUFOWmF0N2pWMWRhYitBUmgrMEM1clh3YjUvM2dEc1BBWmVN?=
 =?utf-8?B?Qzd4WTFsQndXelRxNzRiSkVWYVA3MmhodVNOVHZEenQwWittMCtlZXZHd0R4?=
 =?utf-8?B?d1VVeXk2VFdIVlBnMjB2TVR0SFNWNmJYVjlyWVFQV2Q0RkJSVTRnNnl5ck9C?=
 =?utf-8?B?R01PZVBwaUZQOHZSNFJ2RkI0dkg2ZWZGUncxM2I5Q2lFY3VvYlJ2dE5qb1hL?=
 =?utf-8?B?ck5idEZJV3E1WFErcXhqZlRtbi94THErZjJwaEdIUUUzKzNuR1E0RzdnZGc4?=
 =?utf-8?B?OXZESlRCV09HYm4xNFBMZWNRVlJUVVFKaXJlSHN5WHhHdWptQkQrQmlSZjNk?=
 =?utf-8?Q?d24tT+Bz7aoct9Hj32dLrnjK0?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 158eee12-faf5-4cd6-97c9-08dab8ebd592
X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB6434.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Oct 2022 13:53:46.9307
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: aN3z2uuag52FqeVkq2ZIZHUlRm4gYcknz/TICTfycIWlP9m09cLXFVUj4surZ6vJgrQvRLxxPIi2Sspf3KAsrQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6267
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 10/28/2022 2:03 PM, Huang, Ying wrote:
> Bharata B Rao <bharata@amd.com> writes:
> 
>> On 10/28/2022 11:16 AM, Huang, Ying wrote:
>>> If my understanding were correct, you think the latency / bandwidth of
>>> these NUMA nodes will near each other, but may be different.
>>>
>>> Even if the latency / bandwidth of these NUMA nodes isn't exactly same,
>>> we should deal with that in memory types instead of memory tiers.
>>> There's only one abstract distance for each memory type.
>>>
>>> So, I still believe we will not have many memory tiers with my proposal.
>>>
>>> I don't care too much about the exact number, but want to discuss some
>>> general design choice,
>>>
>>> a) Avoid to group multiple memory types into one memory tier by default
>>>    at most times.
>>
>> Do you expect the abstract distances of two different types to be
>> close enough in real life (like you showed in your example with
>> CXL - 5000 and PMEM - 5100) that they will get assigned into same tier
>> most times?
>>
>> Are you foreseeing that abstract distance that get mapped by sources
>> like HMAT would run into this issue?
> 
> Only if we set abstract distance chunk size large.  So, I think that
> it's better to set chunk size as small as possible to avoid potential
> issue.  What is the downside to set the chunk size small?

I don't see anything in particular. However

- With just two memory types (default_dram_type and dax_slowmem_type
with adistance values of 576 and 576*5 respectively) defined currently,
- With no interface yet to set/change adistance value of a memory type,
- With no defined way to convert the performance characteristics info
(bw and latency) from sources like HMAT into a adistance value,

I find it a bit difficult to see how a chunk size of 10 against the
existing 128 could be more useful.

Regards,
Bharata.