Received-SPF: pass (google.com: domain of linux-kernel+bounces-206754-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <0e6ab67e-bdff-440a-8772-8d462b71cb42@intel.com>
Date: Fri, 7 Jun 2024 14:08:36 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v19 16/20] x86/resctrl: Make resctrl_arch_rmid_read()
 handle sum over domains
To: "Luck, Tony" <tony.luck@intel.com>
CC: "Yu, Fenghua" <fenghua.yu@intel.com>, "Wieczor-Retman, Maciej"
	<maciej.wieczor-retman@intel.com>, Peter Newman <peternewman@google.com>,
	James Morse <james.morse@arm.com>, Babu Moger <babu.moger@amd.com>, "Drew
 Fustini" <dfustini@baylibre.com>, Dave Martin <Dave.Martin@arm.com>,
	"x86@kernel.org" <x86@kernel.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "patches@lists.linux.dev"
	<patches@lists.linux.dev>
References: <20240528222006.58283-1-tony.luck@intel.com>
 <20240528222006.58283-17-tony.luck@intel.com>
 <9818c304-9056-4d79-acbe-2b35cb847ecd@intel.com>
 <Zl5OmVWFpBtA509j@agluck-desk3.sc.intel.com>
 <03ab405b-d7e4-4c35-8a3f-546ffd590954@intel.com>
 <SJ1PR11MB6083EE537A820BB7CD42D2BEFCFB2@SJ1PR11MB6083.namprd11.prod.outlook.com>
From: Reinette Chatre <reinette.chatre@intel.com>
Content-Language: en-US
In-Reply-To: <SJ1PR11MB6083EE537A820BB7CD42D2BEFCFB2@SJ1PR11MB6083.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cWRKRUwwdTNPaUxZRHBjQ0VaQzBiblFsblp6OGlTdXNwR1hGUTdYUlZjYmxa?=
 =?utf-8?B?eGdZZTBlTUwyNURua2pwQlAydSthTml2cm9xYmRkcXV5QmlOOC9vdzJXMVF3?=
 =?utf-8?B?TTM1QW4ycXNmMzAzNzNtcmxxSzBuRFBuWjVPSnVFbGxtdnc4clZXUnlJdklw?=
 =?utf-8?B?NWl0Uy9jSzRIMExzUlowNURBSHNqTWlNY0M5RFQxOWpBdUFqOENrQmJWdi9i?=
 =?utf-8?B?VGJSMkdTdHRYMHl2bk5KK3BuV3RrZTV0YjNvTzJHd3h6OTNaSzQ1U3dLcy9S?=
 =?utf-8?B?S3lUK0JPZU0xbldHSldQMDRUWUZxRUpxQkhtVWtCVHIxVlN0TE5IS0kweWJT?=
 =?utf-8?B?N0hZcGdMMUEvdFlwM3o2Y2JwUW8wUTIxa3huZXY3WlZlbVFXWUY0bm5hQW5E?=
 =?utf-8?B?SnZWOHcreU9VQXB6WnNIZWhwSU9Fam5aclR1Q2g3c3RFSUtEbTJzb29rYmdn?=
 =?utf-8?B?UTVOYUhKVkRZWjIzTTQrVVJpYThFMkZpYTN2UkFuL242VkhyTGUvZElYMEwy?=
 =?utf-8?B?aS9UdHMreWQ3T3dRT0ptSFdGb1NOTWlaUEZQSy85YVNhUE96L0NkWjRGTjdV?=
 =?utf-8?B?b1Jmdk54Z3RkblZKM243RXVVMUQ3cE1PTTdDd3hhT3JlL3pabERmSU1uNy85?=
 =?utf-8?B?Um13aXVWdFY5V3ZSWUJLVGtESytyR2tzTXFDaEdRNk45S0IxM1ZrbjBmdC9n?=
 =?utf-8?B?ejJOcGFGK2ZzOTdBTWhlNXZVbnFMT2xFb2RWdU5LL2UvVm5teVlNLzZNQkY2?=
 =?utf-8?B?aFFsdm9yQWp5TDd4VEIvMXB2Y0xmK01pMWorTkRCdVNoRVRvZENNbWV5RE5a?=
 =?utf-8?B?VjhiT2tjM0tNazFrdm9CR3BocU50bXlVTU9iVXF4Sk1DNmMwaVo1WmVBbGZB?=
 =?utf-8?B?T0hlZEV1RklyU2JmYUFZWE51dDRiUThCbXpZV29iY2o1VFFjY08wUGJ0R09Y?=
 =?utf-8?B?NnRFb1JwMzVLOUVLNGw1S0pXUmFHMS9LMkQ0dDhkcFhJMTFSbCsrV2daOWln?=
 =?utf-8?B?T0hMei90VHpiak5YNGFxbHpZam9Xd04xVklLVVZDNEs1WWdNUit0SklKWklU?=
 =?utf-8?B?UHR5dzBNR3o0RG5uWFBUTzg0ekU2N2EwaUFONUlXVHhCQlo2SUpYSnc4VlUv?=
 =?utf-8?B?TGV2ZzVDVHBWd2dkWlJLNkxOK0RvWlltVFpqYUY1K3VJT3dxWmRkZnVqOHJ1?=
 =?utf-8?B?b1YxZjQ5dlAzQnZSUFBjU1dTRW13eDgvRHQ5TEVxWTVuN2xSZDZtNklNN0N2?=
 =?utf-8?B?ZGZJRjFCdlRMTkRHTHV0alA5UFRycm9iQThhQUN5MTBrV1BoOUZWYnE1WUh0?=
 =?utf-8?B?NFhrZjFpWno5ajlTQTdsYk5YZkV6ZHNnUkZZUVdkUVoxaDZQeW5aT0NXSlpo?=
 =?utf-8?B?UmM4VmxpOUdSaDNrR28raG1mbnJkbSs1d0xmQk9IOFN3dEJqRC9WL2tKOFd6?=
 =?utf-8?B?RFFMMGpwZU1KS0R1SjJETU9Ldjk2SVhTTU4rakJlWHBJNTU3OFJVSFdzMzBQ?=
 =?utf-8?B?emZxQVo2NDl3U0ppVWtLRTM1K3A0clJEVWxHVFFYSGNHdkgra3MvVHIxdWpJ?=
 =?utf-8?B?RmNMUUtMYVNVYWlQM2lrQnpIZHlPWlF1MlJEY0VRVVkrODRBckMxY0VieUJ6?=
 =?utf-8?B?OTlGSDBXQWw4SDFxUmtqRXA4eTJOYlJoVzJvUW1na2xUOVNKMEF3SjVqaWpH?=
 =?utf-8?B?RjlPSXQ3aVJtV255dXgrOWhib2xia25TOWVYWUNHdUtMbXFEbjNvNHMvMkov?=
 =?utf-8?B?VVNtMnF5Y0s5YTFmM0JNT2JYdlA3cTRCMXFrdVdiOHc3b3lRK0NiS0hyS3BX?=
 =?utf-8?B?MjIxYnZHU3oxME5mYUwrZ1kwdS9JQTJiajI5MFdNVUZZTSs5UXBkdGNJRGEy?=
 =?utf-8?B?SW9QZ25TdlhBUUQxQXVzUDJhdktiUTloOVJBbkc4WEcwNTQ5c1RuMm1tRTdT?=
 =?utf-8?B?RjdERU9zYzlNZmEyTzZCVVBDa0xuZnU1ZmI4U2JxMXAzclRkcDlpZE5WQ0V1?=
 =?utf-8?B?Zkw1RU54RmtubVhiTVM3bGwrU1gvL0tJL2RaVHJTNlNwdC9UdTlpZ2JYUzZz?=
 =?utf-8?B?MGdBNlNrc0NwWVhtYjM2V3h3N3VDRW56K2RZRlFNeEhoZTFCQktteEVCbTFU?=
 =?utf-8?B?Z0Nza3JNSjMxejEvRkErMldFS2FaOHJjZ3AzRGlMSkNzSUtYOE1ZcGZldG1L?=
 =?utf-8?B?enc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: fb9b3bb4-453b-43ad-de95-08dc873600d5
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jun 2024 21:08:39.2806
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 17cLw3f56Xyc/arbt7QSa7yjtn8DvJviVWGEFmfv5FxxLny167rjpwzKEAEqG08brz6/GXJHU/AtzeFqGuAOn0561tFm17GkW3OITaGj+E8=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB7608
X-OriginatorOrg: intel.com

Hi Tony,

On 6/7/24 12:51 PM, Luck, Tony wrote:
>> Hi Tony,
>>
>> On 6/3/24 4:15 PM, Tony Luck wrote:
>>> On Thu, May 30, 2024 at 01:24:57PM -0700, Reinette Chatre wrote:
>>>> On 5/28/24 3:20 PM, Tony Luck wrote:
>> ...
>>
>>>>> +
>>>>> + return 0;
>>>>> +}
>>>>> +
>>>>> +int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
>>>>> +                    u32 unused, u32 rmid, enum resctrl_event_id eventid,
>>>>> +                    u64 *val, bool sum, struct cacheinfo *ci, void *ignored)
>>>>
>>>> This is not architecture specific code.
>>>
>>> Can you explain further? I've dropped the "sum" argument. As you pointed
>>> out elsewhere this can be inferred from "d == NULL".  But I do need the
>>> cacheinfo information in resctrl_arch_rmid_read() to:
>>> 1) determine which domains to sum (those that match ci->id).
>>> 2) sanity check code is executing on a CPU in ci->shared_cpu_map.
>>>
>>
>> "resctrl_arch_*" is the prefix of functions needed to be implemented
>> by every architecture. As I understand there is nothing architecture
>> specific about what this function does and every architecture's function
>> would thus end up looking identical. I expected the cacheinfo
>> information to be available from all architectures. If this is not
>> the case then it does not belong in struct rdt_mon_domain but should
>> instead be moved to struct rdt_hw_mon_domain ... but since cacheinfo
>> has already made its way into the filesystem code it is not clear
>> to me how you envision the arch/fs split.
> 
> Hi Reinette,
> 
> Files in resctrl that sum over resources are going to be a necessary feature

Sum over resources? That is something entirely different from SNC, no?

> for backwards compatibility. I'm doing it for the first time here for SNC, but
> I know of another platform topology change on the horizon that could also
> benefit from this.
> 
> Looking at the end-point of the James Morse/Dave Martin patch series to
> split out the arch independent layer to fs/resctrl/ I see that fs/monitor.c
> makes calls to resctrl_arch_rmid_read(). The x86 version of this remains
> in arch/x86/kernel/cpu/resctrl/monitor.c (I don't see an MPAM version).
> 
> James already added two arguments that MPAM needs and x86 doesn't
> (hence "u32 unused" and "void *ignored" in the argument list). I confess
> that my thought had been "If he can pad out the argument list for MPAM,
> then I can do it too for x86". But that leads to madness, so time to reconsider.
> 
> I can see a couple of paths.
> 
> 1) MPAM/others will also want to have files that sum things, so maybe they want
> an extra argument that shows what to add up. Though even if they do, their
> requirement may not be met by a "cacheinfo" pointer.
> 
> 2) Only x86 (Intel) will use this. Maybe in this case the answer is to do some
> renaming so the "void *unused" argument can be used to pass architecture
> specific information.

By creating the new "sub" monitor files the sum of data has become a feature
of resctrl fs.

By including a pointer to struct cacheinfo in struct rdt_mon_domain as well as
struct rmid_read it surely is not just for Intel. If you want to make this just for
Intel then the whole solution needs to be changed.

> 
> Sketch for option #2. Currently the code does:
> ---------------------
> 
> That void *argument is currently supplied by a call. E.g.
> 
> 	void *arch_mon_ctx;
> 
> 	arch_mon_ctx = resctrl_arch_mon_ctx_alloc(r, QOS_L3_OCCUP_EVENT_ID);
> 
> 
> 	resctrl_arch_rmid_read(r, d, entry->closid, entry->rmid,
> 				QOS_L3_OCCUP_EVENT_ID, &val,
> 				arch_mon_ctx);
> 
> 	resctrl_arch_mon_ctx_free(r, QOS_L3_OCCUP_EVENT_ID, arch_mon_ctx);
> 
> The x86 version of resctrl_arch_mon_ctx_alloc() just does "might_sleep(); return NULL;"
> and resctrl_arch_mon_ctx_free() does nothing.
> 
> New version makes these changes:
> ---------------------
> 
> Add rdt_mon_domain * as a new argument to resctrl_arch_mon_ctx_alloc() (which MPAM can
> ignore).
> 
> x86 alloc function becomes:
> 
> void *resctrl_arch_mon_ctx_alloc(struct rdt_resource *r, struct rdt_mon_domain *d, int evtid)
> {
> 	might_sleep();
> 
> 	return d->ci;
> }
> 
> resctrl_arch_mon_ctx_free() remains an empty stub.
> 
> 
> Is this a reasonable way to split the independent fs layer code from the architecture specific?

Why is this necessary at all? The new variant of resctrl_arch_rmid_read() introduced in this
patch does not contain anything that is architecture specific and thus it is filesystem code.
It is this code that uses the information in struct cacheinfo to set the correct domain, if
needed. In this patch you rewrite resctrl_arch_rmid_read() as something architecture specific
but I cannot see what is architecture specific about it at all. Why not just call this new
function resctrl_rmid_read() that stays in filesystem code and then what you have in this
patch as resctrl_arch_rmid_read_one() should be what is already known as the architecture
specific resctrl_arch_rmid_read(). It is the architecture specific RMID read function that
does not need struct cacheinfo.

Reinette