Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6;
Message-ID: <d8734783-f310-4f8b-837a-78f838669281@intel.com>
Date:   Fri, 8 Dec 2023 12:09:38 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 00/15] x86/resctrl : Support AMD QoS RMID Pinning feature
Content-Language: en-US
To:     Peter Newman <peternewman@google.com>
CC:     Babu Moger <babu.moger@amd.com>, <corbet@lwn.net>,
        <fenghua.yu@intel.com>, <tglx@linutronix.de>, <mingo@redhat.com>,
        <bp@alien8.de>, <dave.hansen@linux.intel.com>,
        James Morse <james.morse@arm.com>, <x86@kernel.org>,
        <hpa@zytor.com>, <paulmck@kernel.org>, <rdunlap@infradead.org>,
        <tj@kernel.org>, <peterz@infradead.org>, <seanjc@google.com>,
        <kim.phillips@amd.com>, <jmattson@google.com>,
        <ilpo.jarvinen@linux.intel.com>, <jithu.joseph@intel.com>,
        <kan.liang@linux.intel.com>, <nikunj@amd.com>,
        <daniel.sneddon@linux.intel.com>, <pbonzini@redhat.com>,
        <rick.p.edgecombe@intel.com>, <rppt@kernel.org>,
        <maciej.wieczor-retman@intel.com>, <linux-doc@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <eranian@google.com>,
        <dhagiani@amd.com>
References: <20231201005720.235639-1-babu.moger@amd.com>
 <d97cbeba-af6d-4b64-b5c8-32dc437a67b6@intel.com>
 <CALPaoCjNgv4rFpbovyayNynuAqYGP0rVLv=djVnDO1LR+zU55g@mail.gmail.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <CALPaoCjNgv4rFpbovyayNynuAqYGP0rVLv=djVnDO1LR+zU55g@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UzQzV1g0bWFhSWRZdkx0dmJZSjNCUWp5alFnV3ZXcXF6bTlIaG01d0lXbzJs?=
 =?utf-8?B?ZWo1SGJ1WE1uNVFPZEkxZ216L2tJV1ozWnBiM21wZUdxSkE0QlE1VDBJRkJm?=
 =?utf-8?B?ZUJYOXBXdzlqTC8va3RhMmpGQlhnc3pkQnp0alAzWk1IMFBGTGp5OTltYitR?=
 =?utf-8?B?WDNqRnorTlBJR1JJMzBKWmIzQXhPTEliMlBWNC9FYS9RdVJCb0V3L2dOMUV6?=
 =?utf-8?B?TkVkSU4wQUkzMFByT090OFRaQ3BoNlU1VTNqMW9RQkt4eUk2YjJvd2hiMloy?=
 =?utf-8?B?VGNTdWIzbm1pK285dWNUcmdEV0NBcldKWVgrVk1ncmdqNnRsUGNKclFxbEJi?=
 =?utf-8?B?bzRwKzhkTHpGVTJQTnZkbVdoam1Sa2c3a3dlOVRDbWFWcmhWRWxMZ3NIM0Fm?=
 =?utf-8?B?aU10K1JRYU1ZR3VmNDIzUEJ4OTI4V2JZdDZDZVcvZGR2bDVMdVVrZERiM3Ns?=
 =?utf-8?B?Z1EwbHF0T0hzaklWQVhMOTIxQm14cWZCdXBwSTJYVk5wTG1QT25OMExxK1RO?=
 =?utf-8?B?eTdTNTlqMWNBR1FlT2VJYWwxcmNURU5ZOFlWZGZtbk5WMCsyQ09OaExFTDFG?=
 =?utf-8?B?WTlBdEloZDBTTjVxbUdobmFtVTFqZFg2K3RoRDhNN09EYml2SnVld3VvOE5L?=
 =?utf-8?B?enhvVDhlcnFsVTRQQkZka3Zqc0Y1c3QraXFTdzVSTGJqdVlXbEZWelN0REtM?=
 =?utf-8?B?SUh3NVQ3Y3UvTkFEUXpIL3NXb0l5MXczdFNHaTdGbXBkU3IyUkI2d1RlU1F2?=
 =?utf-8?B?ejRQcEhCRnJTNWtlQk84eVpJY3JLaDdMdHpuOGhmaGZKcnNWK3JNcEVTTFdz?=
 =?utf-8?B?Q1YvWC9xaVBld1J1cHNyS0Nnb2FqdzA4YnJ4Vkd1QkMrdlJOS0VDNWJTQWJS?=
 =?utf-8?B?dUttNVlZRXNHWFhvRnE2Y3BKUWhsVFQreFM1djBodVZmdzN0bzdHZEpadkdT?=
 =?utf-8?B?M2wrMXo3L2V1SDdxRDhlbWFoMDZFY3VPWCtqc1BLZUhiTDV1THlFLzE0cVh4?=
 =?utf-8?B?YjJhekxMZXRhQ3hra1ZjZW03K1J4aks0N2RkcXRSWE53eEFwUms1RWd3cmlB?=
 =?utf-8?B?NHZLbDN6aDBuSFpGcE5vSGhoUEtwMXFCZmlhbVRTbTdLcGhHN25lNUtyUEtJ?=
 =?utf-8?B?ZGJiRE1SSHFIbmxqZ0JXSktjRWg3NSs1WVlCMEZlYVJjZEtKcU1ub2NXdXNG?=
 =?utf-8?B?Z0NKV1A5VDZVRVk1RzA2bkJEVGZMT1JpRm9jVmZlckNVV0lFeElnc1pPWTEr?=
 =?utf-8?B?dUl2TE9XbzFJOGk5VElvZUJSRXhrSnVuNWFyREdSYkloQUhIZWx3bHJsazBs?=
 =?utf-8?B?S1dpSlJUN21zUTY2dm44bjVSTmc1UmtmakZVWkZRM1VVeW45cmJyQ1FqeC9T?=
 =?utf-8?B?MjNpK0JRKzBaYlZvbUFob3puRVJVcXNkRElrVUNCUVRKOTRjYW5rVlVCcXVy?=
 =?utf-8?B?cjB2dTdINEFxUVhjTnFCektpVTRtSzMyZkFHb000TTRnenF4TlFBcktvR3pQ?=
 =?utf-8?B?Um1FMFJVY1ozcExvRXVQLzBhYU94ZUsyZlJqc2xiOHhFWmVxQUJ1cXNQT0N4?=
 =?utf-8?B?elRqTHEyWS8rNlpDWllMRS94eHdSQTdaSS9LR0dScmNZZFovQnl1WnpBQnRU?=
 =?utf-8?B?amdBVDBiblZoTXdwblFPK2NWYVR3UGdpT3JlRFJrWWxBcExLMTJ2Mi85NUhY?=
 =?utf-8?B?L1gxenRIMVhlVGVWd3RNYnJqODNndXZwVUFmby9hd0xyVHM2bWZoV0RwYTZp?=
 =?utf-8?B?UmpDOExZa05EcTgzbWpjdzZpR1U0aThna0grVlNySUlhdXFlR2tURkRJL2Qw?=
 =?utf-8?B?UUtQd2t6TS83dHJqem1sb1dzQjNaL0oweTU4UkpScjJlTmNDTVZ0am13VWpE?=
 =?utf-8?B?Y1lIb29GUlR1T1VldGhMTUt6ZHg4MnhjenBNZmk5RXRmZXAzM1BrWTVxamRi?=
 =?utf-8?B?andSd2I2SlZQdTJUMjFaTVhxWGxTd2xINU16Z1pNcXNHR1RIbG1kMFQyblJu?=
 =?utf-8?B?d1l0cWc2ZGVIcW9WcEVOYmg4NzZEVXRXYjM0NHNrbU85dE5xeWEwdEZEUTR6?=
 =?utf-8?B?R2tjS0syUUpQdUY0TWI0cGdpeklTa2FGdDU1czRxRndCN1VBV1NOK3RRdm02?=
 =?utf-8?B?YXpWNE1ZYnd4a2FPQWk4YjArU1k3Yks1WTNSK0M4ck1pdEpaRnMzS2xXVEFN?=
 =?utf-8?B?enc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: db17fa4f-73e6-4ca0-8ce4-08dbf8299e24
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Dec 2023 20:09:43.9426
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ZLFJGj1ZV/rkI+xCRPE2uIxqEesU0s1rh0Y/S+nUBsf+IpE0Q9kRnUNUMcuJSdp7BTdMVj9WRjzioUd0ae/wIK5IZGvgAE10xSZEASp25Gg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB6309
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 08 Dec 2023 12:10:07 -0800 (PST)

Hi Peter,

On 12/8/2023 11:45 AM, Peter Newman wrote:
> On Tue, Dec 5, 2023 at 3:17 PM Reinette Chatre
> <reinette.chatre@intel.com> wrote:
>> On 11/30/2023 4:57 PM, Babu Moger wrote:
>>> c. Read the monitor states. There will be new file "monitor_state"
>>>    for each monitor group when ABMC feature is enabled. By default,
>>>    both total and local MBM events are in "unassign" state.
>>>
>>>       #cat /sys/fs/resctrl/monitor_state
>>>       total=unassign;local=unassign
>>>
>>> d. Read the event mbm_total_bytes and mbm_local_bytes. Note that MBA
>>>    events are not available until the user assigns the events explicitly.
>>>    Users need to assign the counters to monitor the events in this mode.
>>>
>>>       #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
>>>       Unavailable
>>
>> How is the llc_occupancy event impacted when ABMC is enabled? Can all RMIDs
>> still be used to track cache occupancy?
>>
>>>
>>>       #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes
>>>       Unavailable
>>
>> I believe that "Unavailable" already has an accepted meaning within current
>> interface and is associated with temporary failure. Even the AMD spec states "This
>> is generally a temporary condition and subsequent reads may succeed". In the
>> scenario above there is no chance that this counter would produce a value later.
>> I do not think it is ideal to overload existing interface with different meanings
>> associated with a new hardware specific feature ... something like "Disabled" seems
>> more appropriate.
> 
> Could we hide event counter files if they're not enabled? Is there
> value in displaying the value of a non-running counter that will be
> reset the next time it's enabled?

It may be possible to hide the counter file when it is disabled but in this
case it is not clear to me how to communicate to user space that it is
an available counter that can be enabled and by hiding the file one mechanism
to actually enable the counter is lost. It is not required to
display a stale value when a counter is disabled, text like "Disabled"
can be used.

>> Considering this we may even consider using these files themselves as a
>> way to enable the counters if they are disabled. For example, just
>> "echo 1 > /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes" can be used
>> to enable this counter. No need for a new "monitor_state". Please note that this
>> is not an official proposal since there are two other use cases that still need to
>> be considered as we await James's feedback on how this may work for MPAM and
>> also how this may be useful on AMD hardware that does not support ABMC but
>> users may want to get similar benefits ([1])
> 
> We plan to use the ABMC counters as a window to sample the MB/s rate
> of a very large number of groups, so there's a serious concern about
> the number of write syscalls this will take, as they will add up
> quickly for a large RMID*domain count.
> 
> To that end, the ideal would be the ability to re-assign all ABMC
> counters on all domains in a single system call.

Understood. I've already pointed out that this is a use case needing
to be considered. Please see [1] - search for "global enable/disable".

Reinette

[1] https://lore.kernel.org/lkml/e36699cf-c73e-401b-b770-63eba708df38@intel.com/