Received-SPF: pass (google.com: domain of linux-kernel+bounces-38977-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <f441b3d1-e8f3-4cae-b4c1-d7829d4d52c5@intel.com>
Date: Thu, 25 Jan 2024 08:57:08 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 1/3] x86/resctrl: Display the number of available CLOSIDs
Content-Language: en-US
To: Haifeng Xu <haifeng.xu@shopee.com>
CC: <fenghua.yu@intel.com>, <babu.moger@amd.com>, <peternewman@google.com>,
	<x86@kernel.org>, <linux-kernel@vger.kernel.org>
References: <20240123092024.1271882-1-haifeng.xu@shopee.com>
 <6fb89a6b-031c-451e-80f7-58c277eda265@intel.com>
 <7e30ea7b-60ac-4c9e-b676-f354e9228ad1@shopee.com>
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <7e30ea7b-60ac-4c9e-b676-f354e9228ad1@shopee.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YUZBNXVTRnlwYzM0U1Z1dGlTbVNOM0pQN0xxWWJYZXNUWFpJOWV0Rnh6aGMz?=
 =?utf-8?B?ZW9jR3JaOW16WHBnS0wrRzRRNEQ3VDRVdUxnbmtZNFlyeE0rZEhpYkNxbVR4?=
 =?utf-8?B?WWNYTGo5eWVvYklyV0R3QWpISXExQjBrQUttOWw3Z3pFejkwbUwwM0xMQk1Q?=
 =?utf-8?B?T1RKRGNQQ2lteGVTUTJPbDZ0cW5HbCtJVWpTU1ZGSUN3MWljYUhpa3ljN1I5?=
 =?utf-8?B?ZExtY2w3RjR5TFUyL0pyb1FNaHljNVlQVUVvNUJBNjBOelRGTWtqdUpkWmpa?=
 =?utf-8?B?OWF6bXk5ZkpEeGJrbkVXV0lqQnRhNlNmVnoxNS9hQ01HWDF5UGtQTHBRZ0dU?=
 =?utf-8?B?SmM3T3ZUUmI3aGgwb0toV1NoNVdpM3RULzdrM0FuWTZxNHFYL1RGM1JHQkRr?=
 =?utf-8?B?MWk1bmJqOHJjMVZpc01HVllSZEFXajBPTm5YTFBvMXppUkhST2l3RVRhQ0tM?=
 =?utf-8?B?VXpvNlVPeG9hR2xrM0I1R2xYQzdNVHA1SHB4OU9jTkxURHpROHpuS2k2eTV5?=
 =?utf-8?B?TkhLeWkxZXZEVFBtL3YwL09aYlBEL0RNN0xoeWtPWmtlRHhuN0ZVeW1LVFpJ?=
 =?utf-8?B?VURkbVJIbzdoU3Z0VGkzdmJiaWw0SXpuc0tEWk1IQ0t0SlJMMVBCZ3MycEJE?=
 =?utf-8?B?TkpwT05wV0VTUERjMXZ1aFpHNEpGMWFDb21zNXE4WkE4UlVGUzczWUg2YURj?=
 =?utf-8?B?dFp0Wkx6TWpZTnpvVXk1R3d3K29qTFJxYXVEYlZ1SjBPbzVMcW1JR1JXb3J4?=
 =?utf-8?B?dmRPOXNTbjlGdG1zbXZCU05MRG43L25pVVYvL3lHSkxhRjkyaEVRY1BCK2h0?=
 =?utf-8?B?enY4THNHbys5Rm5aekIvKzUwQUVJekJzUHZYNHdvWkh3c05SVkdLUEE0d1Fz?=
 =?utf-8?B?SzlQbm40SzFsWEpuNENzRHRla2dOYitMY0VIa2RFZytFcE8vNkpNanFEeUxQ?=
 =?utf-8?B?R0dwRlplZU5SWXcyN3NOTi8rQ0RsV3hCRCtMd1BkTFpocGZna0p5ZTVGY3c3?=
 =?utf-8?B?ZDl3cmtPUVNUQUd4TlhQbnRqYXhKb1c1UzRwYmpKMlhra1BCSVVlS1pnT0Nm?=
 =?utf-8?B?blhadzBnbUMrUENSd2ExbjdFWWViNmRMdjhkUkRFSEtMdUJNUC81SFNLZzNl?=
 =?utf-8?B?elNpZU9JbHl6b29KdFovd2M5R1hOS2RJS00vUklaRCtXcFFlamViQnEzQjJF?=
 =?utf-8?B?LzhZS1U3bE9KMldpMkVUR0dsWmxWQ2RLdGdCM1d4SkJPU1NZcThSYjFaRzk1?=
 =?utf-8?B?NldFeVI2VzNPWEdJMFJoSFdCVjVraDEwY0k0RjcwZkNkK2lCQTZTNm1jZVFF?=
 =?utf-8?B?bktZNlhxc2J4OHEvcFhKdUlRT1I0U2FmcUVqeENYYWZsTG4rWXdrYmtUVWdr?=
 =?utf-8?B?aFd5aG1RSEt2REpRbE9RRXIzOVU0SnhIenB2bDE3bXc1cFljSHczMEs3aE9R?=
 =?utf-8?B?bGZnb2pMcFNxRGlSQlBmQjByWXlYOUpWUjk3c2NaUGN1YkJkRWMvRmxSempi?=
 =?utf-8?B?R2gwU1NEY0dTb2pCS1lrTjVnWDRYNmZUcks2MHVFcXJMOVZhdXNJVGVYWmw5?=
 =?utf-8?B?S3hTOUZmS295dnd0REdKWFM2czJOMnRZelcycDF0R2NmeVJvN3k1clNLbEty?=
 =?utf-8?B?QVdXK0p2eVhZV3gwMWp3NE5GRlJFSjRSY0EybU5QT1pkdVNsMjlhK3A5NnpB?=
 =?utf-8?B?YWR6dkFTZVJsdnJRd09DYW5nZWRQZHJ6UXJaQWsxUFh2a3hUUmJYUDVCUzdn?=
 =?utf-8?B?Q0UxM3RqY09hdm85N0NyQzcrWmdnR1gxQWR4T2ZRRzA3aHliS2VqRXZDcUVT?=
 =?utf-8?B?N0NCdUpQS0szRmVtazQ3QzZiRWVqbU5vZXBqY3kzUG43NnFoa1FlSnZuK0o2?=
 =?utf-8?B?K2EzYnZXMUZQYlpnbFdlOHJaZ0JNcWk2N3JBZXdiL3NFQUx2bnRheDQrYVZG?=
 =?utf-8?B?cXpOSi8vOXIyTEZWSGlYeTJ5MEg0czNnQkwvdUp1OXdLZWRMMkRVTWtUbWxH?=
 =?utf-8?B?YkpxM25QaW94NFRwVWp5VmtKR1lEaE41ajFJeERPZS8vSXpLZ0ZyMDkvcTRW?=
 =?utf-8?B?TTVDK2xJUjZXN0NvLzhqYUxnYS83cEVWOExrTzdVd2FkVDh0R3IxeFZ0V1Iz?=
 =?utf-8?B?Sk43N0thTjBYSmRJZlUybXZKSGhrMGl4VU0xZUdwM1QzVWlEM1o3MWg1bU9o?=
 =?utf-8?B?OVE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c6dbcd0b-754c-4735-6e83-08dc1dc6ac21
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2024 16:57:10.9569
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: E/Q4eNmdQQme+jcAK5YrKvKgiZLeLBbSx2YQATN48r5ghc7YsS/6/vVVu7sSmUiHorSQ6w6yREprqMbvknMzP7qjVX5wPCFKX6S3NBa7uy8=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB5375
X-OriginatorOrg: intel.com

Hi Haifeng,

On 1/24/2024 11:44 PM, Haifeng Xu wrote:
> On 2024/1/25 06:23, Reinette Chatre wrote:
>> On 1/23/2024 1:20 AM, Haifeng Xu wrote:
>>> We can know the number of CLOSIDs for each rdt resource, for example:
>>>
>>> cat /sys/fs/resctrl/info/L3/num_closids
>>> cat /sys/fs/resctrl/info/MB/num_closids
>>> ...
>>>
>>> The number of available CLOSIDs is the minimal value of them. When users
>>> try to create new control groups, to avoid running out of CLOSIDs, they
>>> have to traverse /sys/fs/resctrl/ and count the number of directories.
>>>
>>> To make things more easier, add a RFTYPE_TOP_INFO file 'free_closids'
>>> that tells users how many free closids are left.
>>
>> I do not see this as a change that benefits the kernel or user space.
>> It sounds to me as though user space is planning some behavior based
>> on what it knows about the current kernel internals and requesting
>> more information to peek into these internals to make it easier
>> to do so. The kernel can always choose to do things different
>> internally, but it is required to maintain a consistent interface to
>> user space. We should thus always take great care with new interfaces.
>>
>> From what I can tell user space intends to use this "free_closids"
>> to mean "how many more control resource groups can be created". This
>> is not a contract that I think we should enter into. There has been
>> discussions aiming to disconnect the number of resource groups
>> from the number of closids (effectively letting resource groups
>> with the same resource allocations share a closid). 
> 
> Is this feature merged into latest kernel or just discussions?
> Could you please provide more details?
> Last time, you mentioned that a monitoring group can be moved
> from one control group to another.

It was the original proposal that started a discussion [1]. The discussion
about the problem needing solved instead led to the feature where a monitoring
group can be moved from one control group to another. This solution only works
for AMD and Intel though.

> This is something
>> that the kernel may still do at some point but sharing "free_closids"
>> knowing that user space intends to use it as a "number of resource groups
>> remaining" counter would make future enhancements like this difficult.
> 
> OK, thanks.
> 
>>
>> Could you please provide more detail in why this is required? User
>> space should not need to keep track to know how many groups can be
>> created, creating a new group will fail with ENOSPC if no more
>> groups can be created.
>>
> 
> User space reports alerts when failing to create new groups. If no one tell them that
> closids aren't enough, they will keep trying to create new groups and the number of alerts
> could be very high.

When user space receives the "no space available" it should be expected that
any more attempts to create new groups will also fail (until a resource group is
removed). It is futile for user space to keep trying in this scenario.

> 
> So we want to know how many closids are available, if it's zero, we give up creating
> new control groups and those alerts will disappear.
> 
> Maybe user behavoirs can be ajusted. There is no need to create too many groups, especially
> for those groups with same resource. Or as you mentioned above, we can reuse closid.

Reinette

[1] https://lore.kernel.org/lkml/CALPaoCj-zav4x6H3ffXo_O+RFan8Qb-uLy-DdtkaQTfuxY4a0w@mail.gmail.com/