Received-SPF: pass (google.com: domain of linux-kernel+bounces-170279-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <91ccdf56-1a33-437e-9a55-ae2e1e89fa8d@amd.com>
Date: Mon, 6 May 2024 13:09:29 -0500
User-Agent: Mozilla Thunderbird
Reply-To: babu.moger@amd.com
Subject: Re: [RFC PATCH v3 17/17] x86/resctrl: Introduce interface to modify
 assignment states of the groups
To: Reinette Chatre <reinette.chatre@intel.com>,
 Peter Newman <peternewman@google.com>
Cc: corbet@lwn.net, fenghua.yu@intel.com, tglx@linutronix.de,
 mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org,
 hpa@zytor.com, paulmck@kernel.org, rdunlap@infradead.org, tj@kernel.org,
 peterz@infradead.org, yanjiewtw@gmail.com, kim.phillips@amd.com,
 lukas.bulwahn@gmail.com, seanjc@google.com, jmattson@google.com,
 leitao@debian.org, jpoimboe@kernel.org, rick.p.edgecombe@intel.com,
 kirill.shutemov@linux.intel.com, jithu.joseph@intel.com,
 kai.huang@intel.com, kan.liang@linux.intel.com,
 daniel.sneddon@linux.intel.com, pbonzini@redhat.com, sandipan.das@amd.com,
 ilpo.jarvinen@linux.intel.com, maciej.wieczor-retman@intel.com,
 linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, eranian@google.com,
 james.morse@arm.com
References: <cover.1711674410.git.babu.moger@amd.com>
 <f7dac996d87b4144e4c786178a7fd3d218eaebe8.1711674410.git.babu.moger@amd.com>
 <CALPaoCihU+mat2A-wNtTm=qbpya8ZqhDURsfZfjuHitch0WrLA@mail.gmail.com>
 <9d59d38b-af1a-46d8-81c4-b426d47d4ed6@amd.com>
 <CALPaoCgFEybS5MhsPx4EaJmsfBe8Es_6QwWBXpoctdaNf0NcMQ@mail.gmail.com>
 <80c56417-3d74-7d68-1228-9cd944567b4e@amd.com>
 <c8a23c54-237c-4ebb-9c88-39606b9ae1ab@intel.com>
 <4cfe9c47-4736-b08c-50a3-5d7a11246b5b@amd.com>
 <330e3391-b917-4a88-bae3-bdcbb8cfd6f4@intel.com>
Content-Language: en-US
From: "Moger, Babu" <babu.moger@amd.com>
In-Reply-To: <330e3391-b917-4a88-bae3-bdcbb8cfd6f4@intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?UGhFcDh6UXJCNUJTSyt1cVNYTmJrMTkxOFgvT2xlajBBT2huajlsSUNSaERN?=
 =?utf-8?B?ZlEyaVlHN0ptMWsrOTV0S09EU2xmUEtqV2p1dTYyK3duaVBxdUh3WUhKK1Uz?=
 =?utf-8?B?TGRUV2p1U1ZwYmIrY2NJNkVodk1MNWtybG1jM1VhYWJEQ1ZmVVZIQVo4V2RW?=
 =?utf-8?B?ZWVZQXN3bVlkRWJEWU5vNnJwVTFWek9WKzRNRmkvbVp0Q3RYWW1zeno5RTNx?=
 =?utf-8?B?b2k1VlQ3bjY5SnFYcERtVGxsc2xvRHgxbDBtU0RxbXVkWXYxNXN6cUs4Rko2?=
 =?utf-8?B?RnVhb0hiM0RCQnJNQkRjbTQxV04waXprM2xaYnR1bXlRTUtMVnpzSUFlTTF6?=
 =?utf-8?B?QlNLNXh2NUl4ZXpxeldJeHliMXMzOHJCTXcyL0Q0MXhqVWx4NFZEdlk1dG92?=
 =?utf-8?B?enJXKzJyUHppcTRtQWljZktFRkJPK2g3VmtXd2Z1VitRYVNBTmltSlF4VkNT?=
 =?utf-8?B?TVhYUEh3eVhlY2tkNXAvZzdqWHNJOVFod3lPNzRkYVFUMTVJaUQzeVdkNGY0?=
 =?utf-8?B?ZnoyT2ZMU3VXMzhhY0I4NmRHZ0g3V2dkNmR2NjRLM1QrRlZWdjFQdENob2tT?=
 =?utf-8?B?SFk0amZIb2l5Ky8xSzVQT2pTWjF3YUhIdjF3QWU4U0NibzZjLzdTZ2NhSUhN?=
 =?utf-8?B?cGRjSXRSalFpRVk1cGhQSko2U0pYSEwzMzFIcTdRYmdCSzVvV2k5ZndpblNt?=
 =?utf-8?B?emszS1owdXA2RWJ6ZkxLcFI2elRDdjBmRVV5OVYxQmgwS1NXcUdYMWJINGl2?=
 =?utf-8?B?bVBQM1UxYTdVbmNTMFRic056cERhM3BTUGxnVXhSditqL0dKUGx4NmVzQ2No?=
 =?utf-8?B?N0NObkh4MUFkRHdzWVJ0TzE5TUIwTytJY0FHUGs5aGhFYTJaM2VYRUo0ODdW?=
 =?utf-8?B?Y0MvUmNnWE1zZlUyZGtscURIeXBxaVJ3d0RlWktoNUFicVEzM3pRNDhIemxE?=
 =?utf-8?B?QVBubCtTY3BGUTNoNS9QQjBNVVpNUFRPWUFNU3g5TDVWSFpyUzFnZy9QN0Jy?=
 =?utf-8?B?YkxKUnduRGtIVGt1R3ZMaG1TSmwvemtVNCtmdVVTYVhET2JvZ2p0czUybGVD?=
 =?utf-8?B?ODNxSmxoQ2FhbHpqLzVPNzg2eWYvSjV1Ym0vaEtpLzRwVjRBcnJiY29idHFK?=
 =?utf-8?B?K0hJM283bWg2bE9OK29VZldyOVlXcXF2OWlDd2FJdER5TkZYR2ZVZGYvOEgz?=
 =?utf-8?B?WFVqME5NcEhSck5Fb2U4OXh0QjJPQVBpQkFPcmFuZFQ2dlRhREw5cTVOTVVZ?=
 =?utf-8?B?RHcySGZINEhBMXpyU2ZLOUZWRXdndVRNeDNTMzZKazlFQlltSzRnT3hOTkVi?=
 =?utf-8?B?YlB0Q0FzeERQWmJOekI5RUV0VDh4RjU1b01sekhNa0VpTUcvdUxUOFdha0Zl?=
 =?utf-8?B?YzhrMGxBWitYeTRzWXJvd0hRSFZOUWlKdDJPc0hXcHFQVnMveXoyaXo3UkFM?=
 =?utf-8?B?b0Ywc1dEY0VOc3JqTlRoRHRpcVJTQ2hnN1ZlVHZoeVltbVhDNmlxaDIxVGI4?=
 =?utf-8?B?VUt6U0ppdkZyZUNVU3VuSnBVUFp5eFlQamdQaFVheUM2K2oxbnV2d2UrajFp?=
 =?utf-8?B?TFRLeVl0UU5zTlhWdzlJSEJkOG1BUE4rMjBmKy9oUDZ1RTlXbzB6ZThQOWpS?=
 =?utf-8?B?RER1dXNMTUg0Ukd5TUZtQVhXSHFaT1VBTzhOaGViL3ZUbUZpMWZVTWNoU0JD?=
 =?utf-8?B?b0UzWGhkZ0swcUpsa0xkSmFIUnFRYUJXQ0RXMzZidEtJR2FzaWU5MklZWm9a?=
 =?utf-8?B?S253K1RzUTFFNzc4UFZUZHNqU05lZDZJdDRneDlxdnYyYklpWkZqazIzb3VJ?=
 =?utf-8?B?WHFQemlYYTdpUEM3N0l1U1FxaXJ5c3lCZVp0c052S1VUZnB2bWk1MG5zVkwx?=
 =?utf-8?B?VXZteisrTjBJM3JOQUI3aDcybyt1YjBvc3FMdXZCWXF1UUtWbnd1V2h6NmVy?=
 =?utf-8?B?dTAxS2lDZS9NR1lOTFhVd0NsWW9pdndyY2hOSVVTS0tFM0ppTGp5bWt6UDB3?=
 =?utf-8?B?M0hHNStldnA0R0duRDV4bDd6TkU5UHlwUkZReXJZZ2VXN2RZTHNGWTNiTGtX?=
 =?utf-8?B?UE9rWXhoVUdtVmtsNERvM25FZVF4ay9FSEZ3OHY4SWNnNXJxNzFYLzZvdzBB?=
 =?utf-8?Q?T1ZU=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4b104d36-87cb-457c-603c-08dc6df7aedb
X-MS-Exchange-CrossTenant-AuthSource: MW3PR12MB4553.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 May 2024 18:09:33.9607
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: X9YgIt0afUq8YseVQNSzavNzn3kZoSIkDHNU5tx/vLix7iPls3YQjpXL9GnEWeYp
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6269

Hi Reinette,

On 5/3/24 16:16, Reinette Chatre wrote:
> Hi Babu,
> 
> On 5/3/2024 9:14 AM, Moger, Babu wrote:
>> On 5/2/2024 6:00 PM, Reinette Chatre wrote:
>>> On 4/17/2024 3:52 PM, Moger, Babu wrote:
>>>> On 4/17/2024 3:56 PM, Peter Newman wrote:
>>>>> On Wed, Apr 17, 2024 at 12:39 PM Moger, Babu <babu.moger@amd.com> wrote:
>>>>>> On 4/17/24 12:45, Peter Newman wrote:
>>>>>>> On Thu, Mar 28, 2024 at 6:10 PM Babu Moger <babu.moger@amd.com> wrote:
>>>>>>>> diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
>>>>>>>> index 2d96565501ab..64ec70637c66 100644
>>>>>>>> --- a/Documentation/arch/x86/resctrl.rst
>>>>>>>> +++ b/Documentation/arch/x86/resctrl.rst
>>>>>>>> @@ -328,6 +328,77 @@ with the following files:
>>>>>>>>            None of events are assigned on this mon group. This is a child
>>>>>>>>            monitor group of the non default control mon group.
>>>>>>>>
>>>>>>>> +       Assignment state can be updated by writing to this interface.
>>>>>>>> +
>>>>>>>> +       NOTE: Assignment on one domain applied on all the domains. User can
>>>>>>>> +       pass one valid domain and assignment will be updated on all the
>>>>>>>> +       available domains.
>>>>>>> How would different assignments to different domains work? If the
>>>>>>> allocations are global, then the allocated monitor ID is available to
>>>>>>> all domains whether they use it or not.
>>>>>> That is correct.
>>>>>> [A] Hardware counters(max 2 per group) are allocated at the group level.
>>>>>> So, those counters are available to all the domains on that group. I will
>>>>>> maintain a bitmap at the domain level. The bitmap will be set on the
>>>>>> domains where assignment is applied and IPIs are sent. IPIs will not be
>>>>>> sent to other domains.
>>>>> Unless the monitor allocation is scoped at the domain level, I don't
>>>>> see much point in implementing the per-domain parsing today, as the
>>>>> only benefit is avoiding IPIs to domains whose counters you don't plan
>>>>> to read.
>>>>
>>>> In that case lets remove the domain specific assignments. We can avoid some code complexity.
>>>>
>>>
>>> As I understand counters are scoped at the domain level and it is
>>> an implementation choice to make the allocation global. (Similar to
>>> the decision to make CLOSIDs global.)
>>>
>>> Could you please elaborate how you plan to remove domain specific
>>> assignments? I do think it needs to remain as part of the user interface
>>> so I wonder if this may look like only "*=<flags>" is supported on
>>> these systems and attempting to assign an individual domain may fail
>>> with "not supported".
>>
>> This series applies the assignment to all the domains.
>>
>> For example:
>>
>> # echo "//0=t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>>
>> User here wants to assign a monitor to total event on domain 0.
>> But this series applies monitor to all the domains in the system. IPIs will be sent to all the domains.
> 
> I would like to recommend against this. (a) this is not what the API
> says will happen, (b) behavior like this may result in users having scripts
> with syntax like above expecting changes to all domains and when/if
> AMD or another architecture decides to implement per-domain assignment
> it will break user space.

Sure. We keep per-domain specific assignment.

"//0=t"   This will apply assignment only domain 0.
"//*=t"   This will apply assignment one all the domains.

Hope it clarifies.

> 
>> Basically this is equivalent to
>>
>> # echo "//*=t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>>
>>
>> I was thinking of adding domain specific assignment in next version.
>> That involves adding a new field in rdt_domain to keep track of
>> assignment. Peter suggested it may not be much of a value add for his
>> usage model.
> 
> I do not have insight into how all users will end up using this.
> 
> Reinette
> 

-- 
Thanks
Babu Moger