Received-SPF: pass (google.com: domain of linux-kernel+bounces-168229-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <ea56c630-80f4-4564-beb3-2b61e810a558@intel.com>
Date: Fri, 3 May 2024 14:14:57 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH v3 00/17] x86/resctrl : Support AMD Assignable
 Bandwidth Monitoring Counters (ABMC)
To: Peter Newman <peternewman@google.com>
CC: <babu.moger@amd.com>, <corbet@lwn.net>, <fenghua.yu@intel.com>,
	<tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>,
	<dave.hansen@linux.intel.com>, <x86@kernel.org>, <hpa@zytor.com>,
	<paulmck@kernel.org>, <rdunlap@infradead.org>, <tj@kernel.org>,
	<peterz@infradead.org>, <yanjiewtw@gmail.com>, <kim.phillips@amd.com>,
	<lukas.bulwahn@gmail.com>, <seanjc@google.com>, <jmattson@google.com>,
	<leitao@debian.org>, <jpoimboe@kernel.org>, <rick.p.edgecombe@intel.com>,
	<kirill.shutemov@linux.intel.com>, <jithu.joseph@intel.com>,
	<kai.huang@intel.com>, <kan.liang@linux.intel.com>,
	<daniel.sneddon@linux.intel.com>, <pbonzini@redhat.com>,
	<sandipan.das@amd.com>, <ilpo.jarvinen@linux.intel.com>,
	<maciej.wieczor-retman@intel.com>, <linux-doc@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <eranian@google.com>, <james.morse@arm.com>
References: <cover.1711674410.git.babu.moger@amd.com>
 <CALPaoCjZ3oLdKymJjASt0aqtd0GGOme7LavvYOtPYTb_rA-mYQ@mail.gmail.com>
 <b35dc4e9-7e8b-42ed-9a51-ae50d521cf4b@amd.com>
 <CALPaoChxYoJx8eR48EkSKf-hu2p2myQJLZEhj_Pq6O4R15-=5A@mail.gmail.com>
 <6edffe1b-e9a9-4995-8172-353efc189666@amd.com>
 <ab2a6a4b-3740-47c6-9443-e6bb7a0c1adb@intel.com>
 <CALPaoCiYFKeASPMDwzzaHLw4JiMtBB6DTyVPgt0Voe3c3Tav_A@mail.gmail.com>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <CALPaoCiYFKeASPMDwzzaHLw4JiMtBB6DTyVPgt0Voe3c3Tav_A@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MjZQMi9YMU0vcjJMWVkzYXlUam1YTGdnZjViclQwR1cyK0tZL1NxMWFKaWNC?=
 =?utf-8?B?ekJReWpEaitiTWpRYlhsd2RCRGxjaFRiWXYzTGxQcllPdDRHa21xbXlpbDlz?=
 =?utf-8?B?R3o5ck1aREFQejFmbGc0TVJLclo4TW03SXd3RGp4MmRiQjZHcTRGVjduZ0Ft?=
 =?utf-8?B?S091S0NBNGYwUDBOeTVDVHc5UWpUMU80Q2wxODV3VW96S1gybUxkSHhGQ3FS?=
 =?utf-8?B?N1NhMmpnenMzY0xHUEkzRVduZTB6dHVOd1JubFlLNjErNFFMMERyTUFpakUv?=
 =?utf-8?B?czNXeTNLWDh0Rk4vWnVMaEpIM04yRjBkWlMrcGsvKyt5OUFZb1doYURrMTE0?=
 =?utf-8?B?MHdsczlMRmhPVHdVTXFSNkpCVHhwdXJmT1RHeGtXWEFZQlRxU2NCb2x1RzVP?=
 =?utf-8?B?NmJxRk5LaEU2dHl5MHhucEZkcWViQzRIQ2xJNWNwaWx3amczTUZXWVZHV0Zx?=
 =?utf-8?B?c1QvZzRQWEhEbk53aG10S3M0SkliWEJSc2pkcU9IQ0JKcXVsUnUyRlpBY3pY?=
 =?utf-8?B?S0ZiTlVQV0pQa0hjcXBNNlM1R2RVdkhEdGpkejEwWkhqZVJMQVZIVTdoakVM?=
 =?utf-8?B?aEFEelZHQVgyYXUvTmVOV0hzT2RRYmI5WGVWMXBkS2ZWUTJtWkxuV3czUHRC?=
 =?utf-8?B?cWtpcFVoRmZCNHJMSmIyOTlCSG44VUZGQnhxTzJSM0pGOHA0ZGtUZjNzRU1B?=
 =?utf-8?B?OVkwTC9rcU1tYlZ3cHpvVGYrT2hJWTBZdDJNVlhSdlRYRE9pV1BCT1FBV1RM?=
 =?utf-8?B?eEhIc2tudUo3cm1iN3V5RERTK2hmbjN4T2RqVTU3NUdEMy91c2pTVmFHeGJh?=
 =?utf-8?B?VXJaSnlXOVhvOEdFQ1FSREZEMlAwcThEMGVNbFlUdnIyVHdhRUZ0b3FYY09k?=
 =?utf-8?B?a2RpK3lDOCtzRlRMSXBQSjAwQUFiVVg0Q3JCYVU3amtLRW9RaWZVWjc2bU5U?=
 =?utf-8?B?RFR4SFFGeG5vODgzYjJQQXcxYklUdkRQdlN5WXgyS1BaYlhiRUZRcnVSNXVt?=
 =?utf-8?B?NEJSSXI2MHQ4UmY1UW5vMjVwSUpneW0rb3BqQkIzbk9HeHhyRTdTTXpEL0J3?=
 =?utf-8?B?TW1jRFZTbzhWRVZjblh0SVo5cC9TekhUVjk3c3NrcnpFSG1mbzRXeFRVQWs2?=
 =?utf-8?B?Wk95aU1KTzIrRVNCTDRmNU1VWE94N3BmeGhwS09GR1M0QVMycjRhOXVyS09z?=
 =?utf-8?B?anB3TkRpOTJVejFnMk9nWC9ieWZwU3F1UjVTZ3d4SU5mU0dmOWJCbWtBZ3Qr?=
 =?utf-8?B?Q2I2QmVXTUF1a2huVXFqbUZnaC9zVGFROTF2ZGZ3Y1FQbTNKSmFTOUlHSUdy?=
 =?utf-8?B?cHdWdGRjOWhrWnZkMDJnVFV0NmRIa2l1UTJhQW1JR3ZvUkk1eDcyQUN4SHJF?=
 =?utf-8?B?NERFcjIzOGpPc2thbVVjL2lmREwvbWtRakNVVXFKWHkwNzJsYmhpL1VQSWF0?=
 =?utf-8?B?NVdYekQyTUpLeVY2YU9kS3YvWXcvY2Nzak5zNTlyYk56S0FkamsyenpnTENX?=
 =?utf-8?B?N1licjJlMlhneGNzbTdETno3OCs4QWw0SWhESGVmYVFIcUNwWHJ4RW5YL1RQ?=
 =?utf-8?B?ZzRaZ0xQRGt4ME0rWkdGcEhWZ0NqZVFZaGlKM0d4OURJMTl0OW9YVFdHV0VI?=
 =?utf-8?B?VHVhM3FsZE80WXBxTmg0T0ZsRU54bENvYjdsSllwNUpOOFVhbkxRWVVsZ282?=
 =?utf-8?B?RzNCODkydWVNdEF0WlZiNUg1ZC9kR1NvVktUbWIxMmxUNFFGUW9WWS92Z2pu?=
 =?utf-8?B?eXNha0pwMDM0MDJhcWJpZFVwY2N6RGVGT09UVG1PRlNiWUZucHhoN1hKaGhI?=
 =?utf-8?B?WU9ud0pJUExmdnFrbWIvTG9ObTFVVnZpR0JLOWZHT0FLc3ZuUUtFYVJuNXBB?=
 =?utf-8?B?cUNZOENLdzdvM0FHN3pOTyt1Q24rNzdBbjl6REVoWG1SMXJRVDgrdnYzSjVr?=
 =?utf-8?B?YnhGK1AvZ0dXVTJPdzlFYnlvSXBjVmh6Y2ZaK2xkTmdhaDhaemljZXhZTTVl?=
 =?utf-8?B?QTJqd25kU0RMNFlId09STEpIZmxIaTRTK0tXQmxveDBGY3hSVE1ycmp5cmRu?=
 =?utf-8?B?bFg5NzByZllNRkV0eEU0ZnFCb1Z4eG9SbHBQeWRVUzhNRzFXbVJZaE5leTJW?=
 =?utf-8?B?Yis1TTN6SVE1bXptN3ZZajJjVHVjR2lYN211YzZoaGsxaHFQa0tKZC9UbDVk?=
 =?utf-8?B?K3c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 7a120cdc-15d2-4d10-90e4-08dc6bb617f4
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 May 2024 21:15:01.1490
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: W6ZSeXgyjfi6lh19QgjrRXQU7SqKyAk0aeuzXymOwKoER3qKFs7UHkwrntJrn6fspqb/HWhHmfL9MxMkq9QfhxkLsvMIj78jytIFX6Zy9s0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB6218
X-OriginatorOrg: intel.com

Hi Peter,

On 5/2/2024 5:57 PM, Peter Newman wrote:
> Hi Reinette,
> 
> On Thu, May 2, 2024 at 4:21 PM Reinette Chatre
> <reinette.chatre@intel.com> wrote:
>>
>> Hi Peter and Babu,
>>
>> On 5/2/2024 1:14 PM, Moger, Babu wrote:
>>> Are you suggesting to enable ABMC by default when available?
>>
>> I do think ABMC should be enabled by default when available and it looks
>> to be what this series aims to do [1]. The way I reason about this is
>> that legacy user space gets more reliable monitoring behavior without
>> needing to change behavior.
> 
> I don't like that for a monitor assignment-aware user, following the
> creation of new monitoring groups, there will be less monitors
> available for assignment. If the user wants precise control over where
> monitors are allocated, they would need to manually unassign the
> automatically-assigned monitor after creating new groups.
> 
> It's an annoyance, but I'm not sure if it would break any realistic
> usage model. Maybe if the monitoring agent operates independently of
> whoever creates monitoring groups it could result in brief periods
> where less monitors than expected are available because whoever just
> created a new monitoring group hasn't given the automatically-assigned
> monitors back yet.
> 

I will respond in other thread.


>>
>> I thought there was discussion about communicating to user space
>> when an attempt is made to read data from an event that does not
>> have a counter assigned. Something like below but I did not notice this
>> in this series.
>>
>> # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
>> Unassigned
>>
>>>
>>> Then provide the mount option switch back to legacy mode?
>>> I am fine with that if we all agree on that.
>>
>> Why is a mount option needed? I think we should avoid requiring a remount
>> unless required and I would like to understand why it is required here.
>>
>> Peter: could you please elaborate what you mean with it makes it more
>> difficult for the FS code to generically manage monitor assignment?
>>
>> Why would user space be required to recreate all control and monitor
>> groups if wanting to change how memory bandwidth monitoring is done?
> 
> I was looking at this more from the perspective of whether it's
> necessary to support the live transition of the groups' configuration
> back and forth between programming models.  I find it very unlikely
> for the userspace controller software to change its mind about the
> programming model for monitoring in a running system, so I thought
> this would be in the same category as choosing at mount time whether
> or not to use CDP or the MBA software controller.

This seems reasonable to me if only considering ABMC and legacy. When
also taking into account soft-RMID it is no longer obvious to me. I do
still have an impression that the soft-RMID solution impacts context switch
duration so I am considering the scenario where user space may want to
use soft-RMID for portions of time to get an idea of workload behavior and
then dynamically move to less accurate measurements to not impact the
workloads all the time.

In this case perhaps more like how user space can dynamically change power
saving mode based on requirements of responsiveness etc.


> Also, in the software implementation of monitor assignment for older
> AMD processors, which is based on allocating a subset of RMIDs, I'm
> concerned that the context switch handler would want to read the
> monitors associated with the incoming thread's current group to
> determine whether it should use one of the tracked RMIDs. I believe it
> would be cleaner if the lifetime of the generic monitor-tracking
> structures would last until the static branches gating
> __resctrl_sched_in() could be disabled.

Yes, this falls under the umbrella of needing to understand the impact
of switching between mechanisms that is not obvious to me.

> 
>>
>> From this implementation it has been difficult to understand the impact
>> of switching between ABMC and legacy.
> 
> I'll see if there's a good way to share my software monitor assignment
> prototype so it's clearer how the user interface would interact with
> diverse implementations. Unfortunately, it's difficult to see the
> required abstraction boundaries without the fs/resctrl refactoring
> changes[1] applied. It would also require my changes[2] for reading a
> thread's RMID from the FS structures to prevent monitor assignments
> from forcing an update of all task_structs in the system.
> 
> -Peter
> 
> [1] https://lore.kernel.org/lkml/20240426150537.8094-1-Dave.Martin@arm.com/
> [2] https://lore.kernel.org/lkml/20240325172707.73966-1-peternewman@google.com/