Received-SPF: pass (google.com: domain of linux-kernel+bounces-175221-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <6a6418f4-9167-4da0-bbd1-740b226f4c39@intel.com>
Date: Thu, 9 May 2024 19:47:23 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH v3 00/17] x86/resctrl : Support AMD Assignable
 Bandwidth Monitoring Counters (ABMC)
To: <babu.moger@amd.com>, Peter Newman <peternewman@google.com>
CC: <corbet@lwn.net>, <fenghua.yu@intel.com>, <tglx@linutronix.de>,
	<mingo@redhat.com>, <bp@alien8.de>, <dave.hansen@linux.intel.com>,
	<x86@kernel.org>, <hpa@zytor.com>, <paulmck@kernel.org>,
	<rdunlap@infradead.org>, <tj@kernel.org>, <peterz@infradead.org>,
	<yanjiewtw@gmail.com>, <kim.phillips@amd.com>, <lukas.bulwahn@gmail.com>,
	<seanjc@google.com>, <jmattson@google.com>, <leitao@debian.org>,
	<jpoimboe@kernel.org>, <rick.p.edgecombe@intel.com>,
	<kirill.shutemov@linux.intel.com>, <jithu.joseph@intel.com>,
	<kai.huang@intel.com>, <kan.liang@linux.intel.com>,
	<daniel.sneddon@linux.intel.com>, <pbonzini@redhat.com>,
	<sandipan.das@amd.com>, <ilpo.jarvinen@linux.intel.com>,
	<maciej.wieczor-retman@intel.com>, <linux-doc@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <eranian@google.com>, <james.morse@arm.com>
References: <cover.1711674410.git.babu.moger@amd.com>
 <CALPaoCjZ3oLdKymJjASt0aqtd0GGOme7LavvYOtPYTb_rA-mYQ@mail.gmail.com>
 <b35dc4e9-7e8b-42ed-9a51-ae50d521cf4b@amd.com>
 <CALPaoChxYoJx8eR48EkSKf-hu2p2myQJLZEhj_Pq6O4R15-=5A@mail.gmail.com>
 <6edffe1b-e9a9-4995-8172-353efc189666@amd.com>
 <ab2a6a4b-3740-47c6-9443-e6bb7a0c1adb@intel.com>
 <CALPaoCiYFKeASPMDwzzaHLw4JiMtBB6DTyVPgt0Voe3c3Tav_A@mail.gmail.com>
 <b725e4ca-8602-eb26-9d47-914526621f52@amd.com>
 <380a91e1-248a-7d35-4da2-2bad2ddc6005@amd.com>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <380a91e1-248a-7d35-4da2-2bad2ddc6005@amd.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZmM3TWdGM1BGd05PT0l2YXBJTnZ4eENXaGc3cnNkb2RqQUkrQlFYQU4zNWlB?=
 =?utf-8?B?b3RCTW5IUmIrS3JKVDlmT25vM3NydWh0T3RnQ3NaaG5HaG5vMEY2MTBOUmhl?=
 =?utf-8?B?WklPZE51QVJWVzUwdjQ4SHZqZ1k1aWtmamVTQUNNN3drVlBDelFEUzJZWFln?=
 =?utf-8?B?U3RXM1l4SHV5ZnQzVmNLTXUwSFAxR3RYKy9WV1hhdHNZeVdVODg4eWZnRCtr?=
 =?utf-8?B?bGp5OEROQUcxUnQ3K0xNalY0Vnhmc1FGRG5OVEdsei9FYU1BS3B1Nk5QaTBN?=
 =?utf-8?B?NjluRUQyMzhJMTBIdUw0N2NCS2ZmV2tvM3pGeEk5OC9uZFg3R2JOaUR1UlMx?=
 =?utf-8?B?TWpyam54WjViYklsUUtxUS9ROWxlN2VRVGpobWJwbU5aemxZZ2ZRdTdZN0Qw?=
 =?utf-8?B?U2psUUpaMDdlbWFIWUYxNDU4dGduZWZDN2ZrT1VINkhQZExNWmRtUzM1L0M5?=
 =?utf-8?B?V1Z5MjVvOXFRQUFtRnpYL1JYdkp2KzRhWXpZSHovRHA3U1dSTkNuZ3huMU11?=
 =?utf-8?B?TTNLWEEzeWZUa0hCOEFyUHdoU1k2dWdsSnRXY0FtcUo4WlZSWjRick9rdThB?=
 =?utf-8?B?bXlZd2NNdStRajFiSWZzcll3bkFVS1V1bUFqZG1TZ1p0NGI0dlhTeGVPeXhx?=
 =?utf-8?B?eDNud3dwYVFkbnBOQzU1TUR4cjVTS0dLelhxdDY1UnMxWnhpd0NyWmV3NEdh?=
 =?utf-8?B?aUhwekJnUHJYL3pna3hwZHlYUldtcjRRaXhXM2p0TGJxdERzMDIyR2JVWWxr?=
 =?utf-8?B?NEZtU1RkTVJFYW9jQWlRbEFCYnZ1UWxVR1VmZnJnT0UwVW80c0pvdEdFc1Ju?=
 =?utf-8?B?UnBjenlraUtJNjJGendNaHQ1UE9VNFZ1d1RuZEdGRmlqcDQxczlFVmZYM0kw?=
 =?utf-8?B?dzJpbEtKckZ0NmE1OUdhQ00rditVNFlxNGU4dk5pZXhhSUN2aExPdldnaUVH?=
 =?utf-8?B?Mk5aSmpTZDBJcHlyVlpTclpKTy9vd2U2MFpyaHliMWdnellvNkJZK29Xand4?=
 =?utf-8?B?a05YbDNGcG8wSTVCTjNobk1iV3ovQUoyQWE5SUNiQzhyVVp1M0VHaTJCdU95?=
 =?utf-8?B?Z3JZaVpKYjNqcmdqWDQyY2Z6Z0h3TDF4RzJQTmJDYU1zVmJoVGdTMVlIYTBD?=
 =?utf-8?B?cHhiSmR3THN1aFZ2c25CNmFxeGs0WHZCTlM1RHFXNmZSbFp2Z3NxVjVocXpO?=
 =?utf-8?B?VE0reC9xVmZybWU4ZHFST1E3eHhpMktYZkpOVldCNldkL2UzQlBiNTVNY2Vq?=
 =?utf-8?B?UTBrcE1nbkdzWHVnR00xU0oyQmpMYjdJeHRNeEJXUjc3cDRjYUVkR3VvV0xL?=
 =?utf-8?B?Z2IvTDlhMzh5WUtIUVNIM0dGQldrS0dCdXlNRzdNandld2ZpVkdmN0cxZEh2?=
 =?utf-8?B?bzJ3RUNCRlVFbmRMUHdyS1ZoWkhZdWk3Y28xQ3RSWWdkLzFUYW8wMU1NSy9m?=
 =?utf-8?B?cEJpOWZQZHRHS2ppOHZ0dmF4VCs0V3RhOWZ6emdhaW5TcjZEeFZxN3dKM3Np?=
 =?utf-8?B?STlIbDV2b01YUldvQVFzbC9OOUtuUVFKMllCOTZueUVsMEcvZHlabnZDNy8v?=
 =?utf-8?B?TzVscVNBVk1BM2ZUaGNjNXFHRVQ2UGhCeFZZcjJ3UmZ4T1htU1NTZEc1UGU4?=
 =?utf-8?B?WE9SNlc0ZXJ3NWF2bE1nMFhHU1pJdmVTM1dZT1BiY3N2S1lDN3A5aTFiQ2E2?=
 =?utf-8?B?ZzFGdzN6ajJmbEZIZ0ZTZ244K1NlTStNc3UvcGE5K1IvS3hyY3BxRG1HTkpC?=
 =?utf-8?B?dVpGMDZEeE5vcmR3S3lHMFVEVlJ0OTJXa0RscC9QUjlUdDNaRkxPUjF5ejRy?=
 =?utf-8?B?dkdObUNxNnpSWlRUelI4YmNFMUNnZXZZcWUzaTZMZWpxbWMwMUxZNXVXcUlt?=
 =?utf-8?B?WmNHNDdxR0FQZjU4aEREUzhndmQ0YjdnbktWT1BPSzhyQ0w3RXNIcThMQW5I?=
 =?utf-8?B?WVlIUktSQ0tQdTVvZUpDT2ZTVkV1c2JVZllwSFRIUVAzQ0hZODNucUFyMXo3?=
 =?utf-8?B?QXpMSTJIUjRZMkhsSFNjUGxMU3lrTFVZT0hYdlJDdWoxOWRIZTE3MkNYRTZz?=
 =?utf-8?B?Zll1YlZYSCtoR1RMdWFRbVdFRkdiRFV5WVdodDd2ZXFTaGpOdG11U2dBWC9G?=
 =?utf-8?B?akhPTkVYOUxOVDh2Qzd1SlRIaU1YOUtPVExTZ1VtNEVFSjBTU0RMQ05IK0xZ?=
 =?utf-8?B?OWc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 277cd638-bd9d-48b4-4a96-08dc709b8741
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 May 2024 02:47:27.2471
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: fcYCLYI8tQEHepuAc5zStNjQdkL08IqoySB2A/ijzwNHoD4NgMyaBqgGdgiCSiviIRcNZncv7uh+NDJeRCS8H0UmgtuD4jBTIxTpV4LOKiE=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB5147
X-OriginatorOrg: intel.com


On 5/9/2024 5:57 PM, Moger, Babu wrote:
> On 5/3/2024 3:44 PM, Moger, Babu wrote:
>> On 5/2/2024 7:57 PM, Peter Newman wrote:
>>> On Thu, May 2, 2024 at 4:21 PM Reinette Chatre
>>> <reinette.chatre@intel.com> wrote:
>>>> On 5/2/2024 1:14 PM, Moger, Babu wrote:
>>>>> Are you suggesting to enable ABMC by default when available?
>>>>
>>>> I do think ABMC should be enabled by default when available and it looks
>>>> to be what this series aims to do [1]. The way I reason about this is
>>>> that legacy user space gets more reliable monitoring behavior without
>>>> needing to change behavior.
>>>
>>> I don't like that for a monitor assignment-aware user, following the
>>> creation of new monitoring groups, there will be less monitors
>>> available for assignment. If the user wants precise control over where
>>> monitors are allocated, they would need to manually unassign the
>>> automatically-assigned monitor after creating new groups.
>>>
>>> It's an annoyance, but I'm not sure if it would break any realistic
>>> usage model. Maybe if the monitoring agent operates independently of
>>
>> Yes. Its annoyance.
>>
>> But if you think about it, normal users don't create too many groups.
>> They wont have to worry about assign/unassign headache if we enable monitor assignment automatically. Also there is pqos tool which uses this interface. It does not have to know about assign/unassign stuff.
>>
>>
>>> whoever creates monitoring groups it could result in brief periods
>>> where less monitors than expected are available because whoever just
>>> created a new monitoring group hasn't given the automatically-assigned
>>> monitors back yet.
>>>
>>>>
>>>> I thought there was discussion about communicating to user space
>>>> when an attempt is made to read data from an event that does not
>>>> have a counter assigned. Something like below but I did not notice this
>>>> in this series.
>>>>
>>>> # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
>>>> Unassigned
>>>>
>>>>>
>>>>> Then provide the mount option switch back to legacy mode?
>>>>> I am fine with that if we all agree on that.
>>>>
>>>> Why is a mount option needed? I think we should avoid requiring a remount
>>>> unless required and I would like to understand why it is required here.
>>>>
>>>> Peter: could you please elaborate what you mean with it makes it more
>>>> difficult for the FS code to generically manage monitor assignment?
>>>>
>>>> Why would user space be required to recreate all control and monitor
>>>> groups if wanting to change how memory bandwidth monitoring is done?
>>>
>>> I was looking at this more from the perspective of whether it's
>>> necessary to support the live transition of the groups' configuration
>>> back and forth between programming models.  I find it very unlikely
>>> for the userspace controller software to change its mind about the
>>> programming model for monitoring in a running system, so I thought
>>> this would be in the same category as choosing at mount time whether
>>> or not to use CDP or the MBA software controller.
>>
>> Good point about the mount option is, we don't create extra files for monitor assignment in /sys/fs/resctrl when we mount with legacy option.
> 
> I think we still have not decided about the "mount" option for
> switching to legacy monitoring. Mount option seems safe at this
> point. 

I have not heard back after sending [1] so I do still believe that users
may want a way to not have soft-RMID running all the time without impacting
monitor and control groups.


> We don't have to deal with extra files in resctrl filesystem
> with dynamic switching.
Reinette

[1] https://lore.kernel.org/lkml/ea56c630-80f4-4564-beb3-2b61e810a558@intel.com/