Received-SPF: pass (google.com: domain of linux-kernel+bounces-166897-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <6edffe1b-e9a9-4995-8172-353efc189666@amd.com>
Date: Thu, 2 May 2024 15:14:33 -0500
User-Agent: Mozilla Thunderbird
Reply-To: babu.moger@amd.com
Subject: Re: [RFC PATCH v3 00/17] x86/resctrl : Support AMD Assignable
 Bandwidth Monitoring Counters (ABMC)
To: Peter Newman <peternewman@google.com>
Cc: corbet@lwn.net, fenghua.yu@intel.com, reinette.chatre@intel.com,
 tglx@linutronix.de, mingo@redhat.com, bp@alien8.de,
 dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com,
 paulmck@kernel.org, rdunlap@infradead.org, tj@kernel.org,
 peterz@infradead.org, yanjiewtw@gmail.com, kim.phillips@amd.com,
 lukas.bulwahn@gmail.com, seanjc@google.com, jmattson@google.com,
 leitao@debian.org, jpoimboe@kernel.org, rick.p.edgecombe@intel.com,
 kirill.shutemov@linux.intel.com, jithu.joseph@intel.com,
 kai.huang@intel.com, kan.liang@linux.intel.com,
 daniel.sneddon@linux.intel.com, pbonzini@redhat.com, sandipan.das@amd.com,
 ilpo.jarvinen@linux.intel.com, maciej.wieczor-retman@intel.com,
 linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, eranian@google.com,
 james.morse@arm.com
References: <cover.1711674410.git.babu.moger@amd.com>
 <CALPaoCjZ3oLdKymJjASt0aqtd0GGOme7LavvYOtPYTb_rA-mYQ@mail.gmail.com>
 <b35dc4e9-7e8b-42ed-9a51-ae50d521cf4b@amd.com>
 <CALPaoChxYoJx8eR48EkSKf-hu2p2myQJLZEhj_Pq6O4R15-=5A@mail.gmail.com>
Content-Language: en-US
From: "Moger, Babu" <babu.moger@amd.com>
In-Reply-To: <CALPaoChxYoJx8eR48EkSKf-hu2p2myQJLZEhj_Pq6O4R15-=5A@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?NVV6TGpLZzJkRXFla2RIV1hkTCtzUFJOS3IvcXFMaW53RmMzcE5zTFR3UVRk?=
 =?utf-8?B?OUllK1E4VTJmVjI3Y0pIZk5LSW9LWElRRmJXdmRNNDZtd3ZIUVpJd3Z6bUdz?=
 =?utf-8?B?QXRzL2FJaTVrYVpFeG96MlFCTlJCTUcrVXZleFdSaDV3UjFSYURGbjY3TEU4?=
 =?utf-8?B?aHhRTE5ibjJacExobjA4TzlQd3VGREcyd3AyN1dPUnFpOTJOMXgxWkRYcHVI?=
 =?utf-8?B?T1UwSmZSb2IvRWx2RCtieW1zV01OUG5UaWptRjNwTDFCMDBGUTYrWlF6Szls?=
 =?utf-8?B?dmdvQjlMYlJHb1RZcUtUQVZLWUNDbG1nd2ovbUVaNnZ2NUxudnJxU2lJQzBC?=
 =?utf-8?B?ZDFtZm5WZ1g0b3J6SzZRM0w2TzdDRGljSW0yNlpaUlpxNE5yeW9zUmJhSTRl?=
 =?utf-8?B?TlRndi9MOGxLR2ZtUmhqN2tNMnBNdm9YOXB4a1BENlVrYWdWeWJkbFYzRmtu?=
 =?utf-8?B?OXAyUUVtcEhQd2VLU2FJNG90SHJQNWVRQzI2M1R0dGJVTjdRZDQrMnpTa0xo?=
 =?utf-8?B?Y1U1elB3VjZxSWZLVTJOY1plWTBmL2U2SFNQd09ESm9BekRzYituYmxCbkRP?=
 =?utf-8?B?bU5nc2t6OURFcEhxM1pBYkNzSFlxQlkrTEcrMWFYREVrSG51d3NzZUlxVEFU?=
 =?utf-8?B?RzZXQUpIcGpTY3BJTHBwMEs5TndWcHRIaVVQSE5xbTYyVjZyaW9iYjROWHJS?=
 =?utf-8?B?Um9Pa3B6bVcrWnJBcGNPMS9sSnFwSm8rS3plWlljL0YrR09uMHBaVFJOL3ZM?=
 =?utf-8?B?RVVyQmR4eEczVVdjb2pvem1Ib094WXhJdlM5MEdBZ3c1UGZmNERWcnFDMzJP?=
 =?utf-8?B?bjRWUnBNNm9HT0NjdEhGdkRXNmJNcVYvVk8rcWxxSkoyYnZHODM0YzJHRnQz?=
 =?utf-8?B?VGpnSllCaDdjNDRsdnZKSi9aZWp1Q2lkYThac2NOMFI5Z08zeU1FT0RlV2lu?=
 =?utf-8?B?azIvVzBSbjY2bG9BTkZQTGRHeGpJTnRGSWtWVmozamtLL2d5UGt6YjhkUmJG?=
 =?utf-8?B?L1NJeWhWbnhQVUFXSDlGWmtvZlhCWUVjYnhseDVqMEJXUmJBbFI0S1NsdXR3?=
 =?utf-8?B?TDNXZStNbG0wQzVUOXJVbzhFZjRUL0xWUVc1NjEwbWxLRWZ2K1IyRHFxL2Uz?=
 =?utf-8?B?NURlcWxGOEJLWThCRUcxajR1VnIzc2VWZG5jMFZVZkFGbTB1RERjd2t5bjk1?=
 =?utf-8?B?ZGN5bzdvQnJsR1lwMEZRUXBLRldudHFFaUd1N3pEQm1pejRXaGYwMXR2VVhu?=
 =?utf-8?B?R3JRUFl6R0FMU2RhWnUwdm1jd1BteEZGYk1CbTZTVEtBaVpEL3lBL0EzZmhr?=
 =?utf-8?B?RlJjclhTQjJIN29SNW5kY0JtVkNlV2ZGa09XY2piQktlZUFpMmc1R3VxVkZt?=
 =?utf-8?B?YUM1TkFnY1hKYWJtVFlJQy9oM1ZRblVjYjdpZVZycE82cVZzUzlMZ0FCa2ZK?=
 =?utf-8?B?SmhVWmU4SURSS0JoSk9pRUFyRGh5RE5vd21DcnpQT1dwK1dGRUhkMkZwa3Qv?=
 =?utf-8?B?NmQ2NFZJWmNHaWM1ei9IY1RZWmhtdEIyMkx0c1dNY1V6bVI1UUsxcmtzMjUv?=
 =?utf-8?B?aUlqVjB3bi9ON25DTnRwMVFPanhvUFdtTThKdlNMTmZqRTVlWTE3RkUyOW1O?=
 =?utf-8?B?VXlVT2pUK1pHTS9XMnFmVmM2cXo2MXFmNTZVajMxL1gwdURnTjVLd0FlaTA4?=
 =?utf-8?B?SWNZZmh6NnBWQVFtQmx4b0hVV2VWZ3Z4YkZBQ1NqR3NVclp2NVFQZDFSd0M0?=
 =?utf-8?B?RDErMmNCMWJUUDE3L2tJSkVLTUdkU2hFcDhsdnhETUJITElqZWlkdTlsaXlr?=
 =?utf-8?B?TDYvbVl4dzRsVEpyKzJhZ3ZqdjBJQ0ZEZHIrdmVhZ21haW8wMklDYjJqK0Zn?=
 =?utf-8?B?OGkvM2pueDBpK1VvbktDeEFUSEdtMHdyUGNVN1VPYTBHdlZ3VVFRMXhyZGt0?=
 =?utf-8?B?cVFQYjNMZXErWjd2eXliaVRZelZuZjhIZC9BdzhIWlc1SVRFcnQ5OHd4TEdw?=
 =?utf-8?B?b1BLbzRXeWhleHYvVG5UcVNvSEVxOVNkRDBlQTVlUWJTcTVIWGszdTJlYnUx?=
 =?utf-8?B?WXlQVTQxNXdtdUFySi9KclEwUTNxQ1dGUEJLNmMwenhlNThKYnB5SCt5dWNo?=
 =?utf-8?Q?D+lk=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 275c24c0-b58f-4fa2-008d-08dc6ae47dec
X-MS-Exchange-CrossTenant-AuthSource: MW3PR12MB4553.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 May 2024 20:14:37.9040
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: xDQH1ahkrWGC1MdMerZQ0Xi+dxMBDwcxPu9aqzaUp4VIWO5t2Bvh+JQwlG6J56QZ
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6550

Hi Peter,

On 5/2/24 12:50, Peter Newman wrote:
> Hi Babu,
> 
> On Thu, May 2, 2024 at 9:25 AM Moger, Babu <babu.moger@amd.com> wrote:
>> On 5/1/24 12:48, Peter Newman wrote:
>>> The FS layer is informed by the arch layer (through rdt_resource
>>> fields) how many assignable monitors are available and whether a
>>> monitor is assigned to an entire group or a single event in a group.
>>> Also, the FS layer can assume that monitors are indexed contiguously,
>>> allowing it to host the data structures managing FS-level view of
>>> monitor usage.
>>>
>>> I used the following resctrl_arch-interfaces to propagate assignments
>>> to the implementation:
>>>
>>> void resctrl_arch_assign_monitor(struct rdt_domain *d, u32 mon_id, u32
>>> closid, u32 rmid, int evtid);
>>
>> Sure. I can add these in next version.
>>
>> Few comments..
>>
>> AMD does not need closid for assignment. I assume ARM requires closid.
> 
> Correct, MPAM needs a CLOSID+RMID (PARTID+PMG) to identify a
> monitoring group. The CLOSID parameter is ignored on x86.
> 
>>
>> What is mon_id here?
> 
> On ABMC, the value is programmed into L3_QOS_ABMC_CFG.CtrID

ok.

> 
> 
>>
>>> void resctrl_arch_unassign_monitor(struct rdt_domain *d, u32 mon_id);
>>
>> We need rmid and evtid for unassign interface here.
> 
> From my reading of the ABMC specification, it does not look necessary
> to program BwSrc or BwType when changing L3_QOS_ABMC_CFG.CtrEn to 0
> for a particular CtrID. This interface only disables a counter, so it
> should not need to know about how it was previously used when assign
> is able to reassign, as assign will always reset the arch_mbm data.

Yes. That is correct. We may not need to set BwSrc or BwType for unassign.

But, we need evtid to update the monitor state of the rdtgroup.
> 
> I do not see any harm in the arch_mbm data being stale while the
> counter is unassigned, because the data is not accessed when reading
> the hardware counter fails. In general, resctrl_arch_rmid_read()
> cannot return any information if the hardware counter is not readable
> at the time it is called.

Ok. Le me check about keeping the stale arch_mbm data after unassign.
It may be okay.


> 
>>
>>
>>>
>>> I chose to allow reassigning an assigned monitor without calling
>>> unassign first. This is important when monitors are unassigned and
>>> assigned in a single write to mbm_assign_control, as it allows all
>>> updates to be performed in a single round of parallel IPIs to the
>>> domains.
>>
>> Yes. It is not required to call unassign before assign. Hardware(AMD)
>> supports it.
>>
>> But, we only have 32 counters. We need to know which counter we are going
>> to use for assignment. If all the counters already assigned, then we can't
>> figure out the counter id without calling unassigm first. Using the random
>> counter will overwrite the already assigned counter.
> 
> I made the caller of resctrl_arch_assign_monitor() responsible for
> selecting which monitor to assign. As long as the user orders the
> unassign operations before the assign operations in a write to
> mbm_assign_control, the FS code will be able to find an available
> monitor ID.

How does assign_resctrl_arch_assign_monitor() selects the monitor id (or
counter id) if all of them are assigned already.

In this series the monitor ids are allocated using assign_cntrs_alloc.
rdtgroup_assign_abmc()  calls assign_cntrs_alloc() to get monitor id. It
reports error if it cannot get free monitor id.

Expectation is the user to unassign an event from another group(or the
same group) before calling assign.

Are you expecting something else here?

> 
> 
>>> I chose to make this a mount option to simplify the management of the
>>> monitor tracking data structures. They are simply allocated at mount
>>> time and deallocated and unmount.
>>
>> Initially I added it as an mount option.
>> Based on our earlier discussion, we decided to use the assign feature by
>> default if hardware supports it. Users don't have to worry about the details.
>>>
>>> I called the option "mon_assign": The mount option parser calls
>>> resctrl_arch_mon_assign_enable() to determine whether the
>>> implementation supports assignment in some form. If it returns an
>>> error, the mount fails. When successful, the assignable monitor count
>>> is made non-zero in the appropriate rdt_resource, triggering the
>>> behavior change in the FS layer.
>>>
>>> I'm still not sure if it's a good idea to enable monitor assignment by
>>> default. This would be a major disruption in the MBM usage model
>>> triggered by moving software between AMD CPU models. I thought the
>>
>> Why will it be a disruption? Why do you think mount option will solve the
>> problem? As always, there will be option to go back to legacy mode. right?
>>
>>> safest option was to disallow creating more monitoring groups than
>>> monitors unless the option is selected. Given that nobody else
>>
>> Current code allows to create more groups, but it will report "Monitor
>> assignment failed" when it runs out of monitors.
> 
> Ok that should be fine then.
> 
> However, I don't think it's necessary to support dynamically changing
> the usage model of monitoring groups without remounting. I believe it
> makes it more difficult for the FS code to generically manage monitor
> assignment.

Are you suggesting to enable ABMC by default when available?

Then provide the mount option switch back to legacy mode?
I am fine with that if we all agree on that.
-- 
Thanks
Babu Moger