Received-SPF: pass (google.com: domain of linux-kernel+bounces-168233-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <330e3391-b917-4a88-bae3-bdcbb8cfd6f4@intel.com>
Date: Fri, 3 May 2024 14:16:33 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH v3 17/17] x86/resctrl: Introduce interface to modify
 assignment states of the groups
To: <babu.moger@amd.com>, Peter Newman <peternewman@google.com>
CC: <corbet@lwn.net>, <fenghua.yu@intel.com>, <tglx@linutronix.de>,
	<mingo@redhat.com>, <bp@alien8.de>, <dave.hansen@linux.intel.com>,
	<x86@kernel.org>, <hpa@zytor.com>, <paulmck@kernel.org>,
	<rdunlap@infradead.org>, <tj@kernel.org>, <peterz@infradead.org>,
	<yanjiewtw@gmail.com>, <kim.phillips@amd.com>, <lukas.bulwahn@gmail.com>,
	<seanjc@google.com>, <jmattson@google.com>, <leitao@debian.org>,
	<jpoimboe@kernel.org>, <rick.p.edgecombe@intel.com>,
	<kirill.shutemov@linux.intel.com>, <jithu.joseph@intel.com>,
	<kai.huang@intel.com>, <kan.liang@linux.intel.com>,
	<daniel.sneddon@linux.intel.com>, <pbonzini@redhat.com>,
	<sandipan.das@amd.com>, <ilpo.jarvinen@linux.intel.com>,
	<maciej.wieczor-retman@intel.com>, <linux-doc@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <eranian@google.com>, <james.morse@arm.com>
References: <cover.1711674410.git.babu.moger@amd.com>
 <f7dac996d87b4144e4c786178a7fd3d218eaebe8.1711674410.git.babu.moger@amd.com>
 <CALPaoCihU+mat2A-wNtTm=qbpya8ZqhDURsfZfjuHitch0WrLA@mail.gmail.com>
 <9d59d38b-af1a-46d8-81c4-b426d47d4ed6@amd.com>
 <CALPaoCgFEybS5MhsPx4EaJmsfBe8Es_6QwWBXpoctdaNf0NcMQ@mail.gmail.com>
 <80c56417-3d74-7d68-1228-9cd944567b4e@amd.com>
 <c8a23c54-237c-4ebb-9c88-39606b9ae1ab@intel.com>
 <4cfe9c47-4736-b08c-50a3-5d7a11246b5b@amd.com>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <4cfe9c47-4736-b08c-50a3-5d7a11246b5b@amd.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?WkdYWVFTa09nUVg0RWI2aUJaWlZFUlN1bjdaVEVxaUNWNkZHUTRwNlI3Y25X?=
 =?utf-8?B?aVh2NlpRREhDR0tiUW1EUXVIRWF0OCtseTdqaHkzNkxHdDkxWk9iejViK2ow?=
 =?utf-8?B?UWpBQ2xvZm8xOERMbjltTEwybEpGb04vN3lrbyt1ODFOZXZsWlRCcDRmM1Zo?=
 =?utf-8?B?MEVOZWVwbXArMFYwWmpkcW5ZMW03YTZoajdVamlTWlptblY1MDlTYUtTYkYz?=
 =?utf-8?B?bjYraGF2UlVZRlA2K0svYVZGalVROEhnMExKQ2JPamVIeGJvYkN1YTJGeEp6?=
 =?utf-8?B?d3FxNHFubTRnK2RZU3J6RmRwWWNobkZuQ3c1U1BBbGR4TFFCMytQNWJYaWFi?=
 =?utf-8?B?RHVnbGlheUVycCtxTkJEbkJRNXBEcjlmWUhobjRhZXY1OGFuNlh3ODBlOWRa?=
 =?utf-8?B?L2swNW11TDQ2ZWNhZmVpSXEwcHZmL095cE9YeWxnS1BHZ0RnK3NxV2JWUlF6?=
 =?utf-8?B?QmtHNkZidWVWODlDN0hJNVpIL2gzSmVoRTQyYUxkMEhEUzhQcjFtUlhZYVY5?=
 =?utf-8?B?WG1sY3cwc3ZTN2FmVVpCQVZUQkp5Q1ZPN25tUE9NSWkya3NUZVRsUFNRYmNI?=
 =?utf-8?B?MytJclVhMnFXdXpWY0tweklmODRUbUdFdXdPbmYwaFQyUFJ5UWNFVkVYekM1?=
 =?utf-8?B?SDN1S2NmSVZsdHB1VHMrSDUwV0w5MFhWWjY5UTdVbzFIcThjNXFuWlFHWi8w?=
 =?utf-8?B?VWFPeERuYjYvaVJpN2lwVHhpNTd5SFZGQWVQZ0wxMzN2Y1dwdE5nWVJOZi84?=
 =?utf-8?B?bUtQKzgwQVlUdXB0SDNMNWlPN2FQOE5xcS9pNW9oM3FyTHdFK1ZoSW5wcUlE?=
 =?utf-8?B?eE92QnFsVmFNNlJOTWtBSFZzT0g2VkZ5MUp2VlVmNVNKTXV0R2dwUkdVOWdH?=
 =?utf-8?B?NE5OL2pYVGRpVnlVSVNFSk5PaG8zVDNjTlZpV25RV0wrTzdYNThaY1lBOTdZ?=
 =?utf-8?B?QTIxQ0xWQ0FVcHZmU0xrNVZ5R2FNb1d6Zjc1QkRKejJUSWdnOFBaSk9sV0l5?=
 =?utf-8?B?dE56TUFnWUp5cStJQVBwdmhSSjA4amVZeEZvcEF0TUpHRllEUitJTDFoMEN6?=
 =?utf-8?B?SFRIeE04bU9aQ01aNUlZWEErcXVFVTZ3OWNMSzdXcmNGMllCNWgrNEZjcDkx?=
 =?utf-8?B?d2hqTUtBVmhSVUU2N1UrSkJLbnUxbU42cFQvVmJaL2dpc3k3ZUVNTHp5dVpa?=
 =?utf-8?B?UE1QcHVBR3pSWktQeCtGQ3ptT3pUbHAvbDYvaUU1ajZ3d1hvcFczS3BvMERM?=
 =?utf-8?B?UEZIRjJkZ0hnS3kwM0ltWHV6WjJYNDZyK3lTS2VIdU9xMWZyc0dvMC9FY1R0?=
 =?utf-8?B?RDd6SElYMVlhb3dDek43dXoyRjBkMlNhTFN5SnN3SVhqZC9hV1pZQWUxR2hJ?=
 =?utf-8?B?US9xWlNyck1ORm42NFJ3S1hUL2tNaWZPeWlUVm5hZVc2Wk9ENEJGWi9RTmtN?=
 =?utf-8?B?SHQrT3RZRzEyczlBZEhhUUgwL2oxQk9aMFUrb1kxVzh6dUkxOTFESnhhRWFt?=
 =?utf-8?B?VU5oVitKSlBDVklNRGQ0aWNCdTZ2NFpPRVEwZ1NuWFNWTDZ2djFpMU5qTTBx?=
 =?utf-8?B?NnYwYUJ0NCtSaWd0WnM0TG5Uem1kNUpYZ0hqTE1Hc3pvY3FhazZUVnN2cDNF?=
 =?utf-8?B?alZxcGVsaXJHS1h6SlYwOVVzeDRJVU50UW04cUtuK0xnS1NSenpzdHFJM3Ir?=
 =?utf-8?B?RXRVbTBDamlFVlUzZktoVzVTUTZHWHR2TTlEU3JIMjlqcE4yeUJ5ZWpOYnU3?=
 =?utf-8?B?b0lqZFM3TFplUWR6UDVVd3kxWUpuSXc2NTUxMFBFSWUxS3N3OW1EYzlKRU9U?=
 =?utf-8?B?SWtGT1hCb3p3R3JSclhVeVZkQUQ5OGs0cmtUUEhQNm9WYkFFYzVobVU4M3J4?=
 =?utf-8?B?bjRpblpkV3puS3phcDlKYWVPdDlSbFZnRy9IWUxNOTlMR1l3VnZzMGZaMC9W?=
 =?utf-8?B?YkRIZVJ2Vkk2dCtyNmpPRENZY0I1alArWXN4U21KNm5uMGNwekJMQmRFM1hV?=
 =?utf-8?B?bDMwS3JKRzh0U3ZRRE5mb2JxMGFNNS9WZkF0cEsxSXZUaFNYb1JJekt1aTNT?=
 =?utf-8?B?NjNXRXNON2h5WmZwZHh3bFZkV2YyTFFYZmhET3QwQ2dObXhUcDFtWmJIaFV3?=
 =?utf-8?B?MFA5WGJZT244SWR1QklmS3RuUTdqUy9FRFNERmFISm9BbnJOdThGL1VSUGlo?=
 =?utf-8?B?b0E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5ada43eb-8dc5-4526-b3a9-08dc6bb650d7
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 May 2024 21:16:36.5444
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +QnBfSOwV9K0+kdepmOQtPoa2yoByCTMezci0t33mqpNRU9o+/4faBeWwCvLBpsuqpXxfWaIBn5DHRvW0Z3ZZiFWBl9oZ2Q0KmeDw7NImFg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB6220
X-OriginatorOrg: intel.com

Hi Babu,

On 5/3/2024 9:14 AM, Moger, Babu wrote:
> On 5/2/2024 6:00 PM, Reinette Chatre wrote:
>> On 4/17/2024 3:52 PM, Moger, Babu wrote:
>>> On 4/17/2024 3:56 PM, Peter Newman wrote:
>>>> On Wed, Apr 17, 2024 at 12:39 PM Moger, Babu <babu.moger@amd.com> wrote:
>>>>> On 4/17/24 12:45, Peter Newman wrote:
>>>>>> On Thu, Mar 28, 2024 at 6:10 PM Babu Moger <babu.moger@amd.com> wrote:
>>>>>>> diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
>>>>>>> index 2d96565501ab..64ec70637c66 100644
>>>>>>> --- a/Documentation/arch/x86/resctrl.rst
>>>>>>> +++ b/Documentation/arch/x86/resctrl.rst
>>>>>>> @@ -328,6 +328,77 @@ with the following files:
>>>>>>>            None of events are assigned on this mon group. This is a child
>>>>>>>            monitor group of the non default control mon group.
>>>>>>>
>>>>>>> +       Assignment state can be updated by writing to this interface.
>>>>>>> +
>>>>>>> +       NOTE: Assignment on one domain applied on all the domains. User can
>>>>>>> +       pass one valid domain and assignment will be updated on all the
>>>>>>> +       available domains.
>>>>>> How would different assignments to different domains work? If the
>>>>>> allocations are global, then the allocated monitor ID is available to
>>>>>> all domains whether they use it or not.
>>>>> That is correct.
>>>>> [A] Hardware counters(max 2 per group) are allocated at the group level.
>>>>> So, those counters are available to all the domains on that group. I will
>>>>> maintain a bitmap at the domain level. The bitmap will be set on the
>>>>> domains where assignment is applied and IPIs are sent. IPIs will not be
>>>>> sent to other domains.
>>>> Unless the monitor allocation is scoped at the domain level, I don't
>>>> see much point in implementing the per-domain parsing today, as the
>>>> only benefit is avoiding IPIs to domains whose counters you don't plan
>>>> to read.
>>>
>>> In that case lets remove the domain specific assignments. We can avoid some code complexity.
>>>
>>
>> As I understand counters are scoped at the domain level and it is
>> an implementation choice to make the allocation global. (Similar to
>> the decision to make CLOSIDs global.)
>>
>> Could you please elaborate how you plan to remove domain specific
>> assignments? I do think it needs to remain as part of the user interface
>> so I wonder if this may look like only "*=<flags>" is supported on
>> these systems and attempting to assign an individual domain may fail
>> with "not supported".
> 
> This series applies the assignment to all the domains.
> 
> For example:
> 
> # echo "//0=t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
> 
> User here wants to assign a monitor to total event on domain 0.
> But this series applies monitor to all the domains in the system. IPIs will be sent to all the domains.

I would like to recommend against this. (a) this is not what the API
says will happen, (b) behavior like this may result in users having scripts
with syntax like above expecting changes to all domains and when/if
AMD or another architecture decides to implement per-domain assignment
it will break user space.

> Basically this is equivalent to
> 
> # echo "//*=t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
> 
> 
> I was thinking of adding domain specific assignment in next version.
> That involves adding a new field in rdt_domain to keep track of
> assignment. Peter suggested it may not be much of a value add for his
> usage model.

I do not have insight into how all users will end up using this.

Reinette