Received-SPF: pass (google.com: domain of linux-kernel+bounces-64537-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <db3b27c5-e66e-4dd4-a56f-9e6928e26618@intel.com>
Date: Tue, 13 Feb 2024 15:26:56 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v9 00/24] x86/resctrl: monitored closid+rmid together,
 separate arch/fs locking
Content-Language: en-US
To: James Morse <james.morse@arm.com>, <x86@kernel.org>,
	<linux-kernel@vger.kernel.org>, Borislav Petkov <bp@alien8.de>
CC: Fenghua Yu <fenghua.yu@intel.com>, Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@redhat.com>, H Peter Anvin <hpa@zytor.com>, Babu Moger
	<Babu.Moger@amd.com>, <shameerali.kolothum.thodi@huawei.com>, "D Scott
 Phillips OS" <scott@os.amperecomputing.com>, <carl@os.amperecomputing.com>,
	<lcherian@marvell.com>, <bobo.shaobowang@huawei.com>,
	<tan.shaopeng@fujitsu.com>, <baolin.wang@linux.alibaba.com>, Jamie Iles
	<quic_jiles@quicinc.com>, Xin Hao <xhao@linux.alibaba.com>,
	<peternewman@google.com>, <dfustini@baylibre.com>, <amitsinght@marvell.com>,
	David Hildenbrand <david@redhat.com>, "Luck, Tony" <tony.luck@intel.com>
References: <20240213184438.16675-1-james.morse@arm.com>
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20240213184438.16675-1-james.morse@arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cDhjRUVMUlcrRjNGZTUwZCt4OUtiOVBJS2h1ZWJsdWQ3Y2hhL3J6dDFtU2g0?=
 =?utf-8?B?SU5YQ24rbkIrMTI5eGY0eitkSWYzaVdjTUpwQW0xUE9Da3IrTGtLMlFDUlZw?=
 =?utf-8?B?cWhHa0x3S2ZzWnlwaThGUEluVWVldFJOSkliQUZnSkVMZHpjSWNudTlSMXl2?=
 =?utf-8?B?UnFpSVZabThiN0JWcGJISk8yMWJiOHR5elRpZkZmN2FpdVM1VjBjR3FVdFp6?=
 =?utf-8?B?NHpCQUlCZm5Vc2EzdHhTK3VIQ1YvN1JsR1pWc21RODFXL3pEZS8vTzgyVnFX?=
 =?utf-8?B?TFgrMTcxald4V29oTkN6bFlNNjg5TEF1SW1GSFVwanZuWWFIcVFvSDBBYmdy?=
 =?utf-8?B?Q0duMUlhaG1OMTk4elB4UE5Ldk1xWlYvcFVoSjBzM3FnRCtrTTdEOUY5dTc4?=
 =?utf-8?B?NUkwRzFTYm1kMDFWRXZoY3FQaGdqbXVUYVF6eWRTS1QrbWs3NWlsM2hCNWYv?=
 =?utf-8?B?NUhMNFU3Q3lNSGROQlZUSnVtUE5nZktYUHRSanVZb1V1dmVEYi9mZDdJUWVU?=
 =?utf-8?B?NzBUcUNTWjk1cXN5M0ExbVBGV0l0L1B2NDEwYXgwYy8rLzNDcDJVQXhBTkhu?=
 =?utf-8?B?UHQ1Z1Z4Q1hhcGlOWTRNeDV4c3RjVC9LVzFIcmFFM3Q1T2pwbXRuL2llOTMw?=
 =?utf-8?B?NDhISndnQytHMUt4YjZJaHh3SThaemdLS3RDblV3YWQ1V0NFdU1kMG80czJs?=
 =?utf-8?B?OEs4OEpWS0J6OTFFUXhQZStINS9iYUw5ZGkxWExhMStkdmwzNkNUUGVoZUpu?=
 =?utf-8?B?TTlRWjZ6QlpGVnFNRkszTWxzZGNuU0dGNnZjNVEyYWI4Qk1IcHhPOUZzK2x2?=
 =?utf-8?B?SXFCdlYvK1FxRzgwamZzck1mVUJnYkc4RjVxSERXcWhHVHA1RTJ6L251YWcr?=
 =?utf-8?B?aWYzbmJuKzhxZ3VhdEpWOTFzVUJnVnU1a0tQc2JOQU10d3MrVXNOSnJQanZv?=
 =?utf-8?B?OUtLMkswOWFFLzVOeXArVzN4YnFOazJvOXVTRk9uUThnL0E3YjhNbkxSMXRP?=
 =?utf-8?B?NUs0dktmVnRIUUFVUVRzK3VSNXEzQ2h4aHJFN3MxbjVxTDJJVjJxL2k4UlZq?=
 =?utf-8?B?Vi8zU0J3SVpFNmpDUklNSnRHQkpsWDJpcmxmNWZabUtBK0RNejdsVjNFWGRy?=
 =?utf-8?B?M2pKdm9jZm9ta1laQU80TG5nbTZEWmlFNjNuQmtPanhmSnljd0pIRTFsZW1i?=
 =?utf-8?B?OGI4ellYc0h6U0dCNnlXdW5jRTd0cS9BaFFXTThMTEMyUlRCMFlNTTJqRW1L?=
 =?utf-8?B?Und5bkF4TDMxdTVuQkxHU0I0SXhmcnh2SHBLdHFIVTkvZ1p6Zi9uaWt1MDhw?=
 =?utf-8?B?UXh5NGx4dWZud20vSm1IRUk2Z1Flenppdnh6NE43UW43NS8xOWQ3dzdJRFJp?=
 =?utf-8?B?T1ZYejBobWljc011NkJTaENVOXpuMG9BRFg5R2pWWHVyQjhDaFhvNUZzcXJ5?=
 =?utf-8?B?MTlFU2piOFBBTHJKak5LV3JxcGdCaGFnSXZUTS96WmtaQlFnN1pvVk5DS0ND?=
 =?utf-8?B?cUd5c0F4SWhFTWp0VTlDN2IvaXd6TmF0NTRCWjA0UGJ6cDljWGdSaVRTVkxP?=
 =?utf-8?B?dnFsNHpmT0xVR2RzVVFNeHpWQ0RqZHVWOW1BWUhOaW5kZ0x4NXlsVUJmQ1RI?=
 =?utf-8?B?dGs2OG51cE53TEEzQWs1VXBZRFBna084UDRaTktMT2JKdkIyWWpIdi8rY2FM?=
 =?utf-8?B?SEdQWndBZ3lIcVcwTWxMWjd6cktzLytkb2tRWE9QQzdtSmtES0J4Q0RiRk9h?=
 =?utf-8?B?M2hmN3hQaDlmZUpIa2RXUzZ6anlmVTNmVGhjdGhEeEZuTU1zUjJadGNzcSs2?=
 =?utf-8?B?ODR4NlZuK0Vhek52VWFCbS95SjlkL0YwbnFLZkExRWhxVlVIR3lIRFU1MFAv?=
 =?utf-8?B?MmVuZ0prZ0ZvS1NPMzhURURiMUpJVmRxMDMyTm1NKzUzcWJCa1ZPVmk3RTB0?=
 =?utf-8?B?b0M1Q3praUJRWCtONkx0bVNpd1laQVVZY004cDFKTG9xY1loaDI3SGYxKzl4?=
 =?utf-8?B?ZnpwTytRcjdXSFVDc2IvcVBKOXlTZjJZU01UYVR2VmczeVdiQVMxN2lmSWV3?=
 =?utf-8?B?b1hrT3RoSXRtRTIrNDROSHVaNVAvdm53MUZaYVZhU21weG0ycjBlL0F1ZVhW?=
 =?utf-8?B?RTUrbm1DdlV3d0JORERpbVk0TFFUL2JjWUYvQlZ1TDJ3WnYvdEdYcjB0emhQ?=
 =?utf-8?B?YlE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c2b656dc-98bb-49be-0768-08dc2ceb466b
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Feb 2024 23:26:59.1244
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Ocl0WH3hRZrKTkRW+URQxOt17V6K0fETAsj1m5z7DJHgULn3ExmW3GvOWPZ5co4tMCqpJCCIWsz3f8e/GVShd8Rfxp5MOHg83+G7g73YKMw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7366
X-OriginatorOrg: intel.com

(+Tony)

Hi Boris,

Could you please consider this series for inclusion?

Thank you very much.

Reinette

On 2/13/2024 10:44 AM, James Morse wrote:
> Hello!
> 
> It's been back and forth for whether this series should be rebased onto Tony's
> SNC series. This version isn't, its based on tip/x86/cache.
> (I have the rebased-and-tested versions if anyone needs them)
> 
> This version just has minor cleanup from the previous one.
> * An unusued parameter in unused code has gone,
> * I've added a comment about the sizeing of the index array - it only matters on arm64.
> 
> Changes are also noted on each patch.
> 
> ~
> 
> This series does two things, it changes resctrl to call resctrl_arch_rmid_read()
> in a way that works for MPAM, and it separates the locking so that the arch code
> and filesystem code don't have to share a mutex. I tried to split this as two
> series, but these touch similar call sites, so it would create more work.
> 
> (What's MPAM? See the cover letter of the first series. [1])
> 
> On x86 the RMID is an independent number. MPAMs equivalent is PMG, but this
> isn't an independent number - it extends the PARTID (same as CLOSID) space
> with bits that aren't used to select the configuration. The monitors can
> then be told to match specific PMG values, allowing monitor-groups to be
> created.
> 
> But, MPAM expects the monitors to always monitor by PARTID. The
> Cache-storage-utilisation counters can only work this way.
> (In the MPAM spec not setting the MATCH_PARTID bit is made CONSTRAINED
> UNPREDICTABLE - which is Arm's term to mean portable software can't rely on
> this)
> 
> It gets worse, as some SoCs may have very few PMG bits. I've seen the
> datasheet for one that has a single bit of PMG space.
> 
> To be usable, MPAM's counters always need the PARTID and the PMG.
> For resctrl, this means always making the CLOSID available when the RMID
> is used.
> 
> To ensure RMID are always unique, this series combines the CLOSID and RMID
> into an index, and manages RMID based on that. For x86, the index and RMID
> would always be the same.
> 
> 
> Currently the architecture specific code in the cpuhp callbacks takes the
> rdtgroup_mutex. This means the filesystem code would have to export this
> lock, resulting in an ill-defined interface between the two, and the possibility
> of cross-architecture lock-ordering head aches.
> 
> The second part of this series adds a domain_list_lock to protect writes to the
> domain list, and protects the domain list with RCU - or cpus_read_lock().
> 
> Use of RCU is to allow lockless readers of the domain list. To get MPAMs monitors
> working, its very likely they'll need to be plumbed up to perf. An uncore PMU
> driver would need to be a lockless reader of the domain list.
> 
> This series is based on tip/x86/caches's commit fc747eebef73, and can be retrieved from:
> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/monitors_and_locking/v9
> 
> Bugs welcome,
> 
> Thanks,
> 
> James
> 
> [1] https://lore.kernel.org/lkml/20210728170637.25610-1-james.morse@arm.com/
> [v1] https://lore.kernel.org/all/20221021131204.5581-1-james.morse@arm.com/
> [v2] https://lore.kernel.org/lkml/20230113175459.14825-1-james.morse@arm.com/
> [v3] https://lore.kernel.org/r/20230320172620.18254-1-james.morse@arm.com 
> [v4] https://lore.kernel.org/r/20230525180209.19497-1-james.morse@arm.com
> [v5] https://lore.kernel.org/lkml/20230728164254.27562-1-james.morse@arm.com/
> [v6] https://lore.kernel.org/all/20230914172138.11977-1-james.morse@arm.com/
> [v7] https://lore.kernel.org/r/20231025180345.28061-1-james.morse@arm.com/
> [v8] https://lore.kernel.org/lkml/20231215174343.13872-1-james.morse@arm.com/
> 
> James Morse (24):
>   tick/nohz: Move tick_nohz_full_mask declaration outside the #ifdef
>   x86/resctrl: kfree() rmid_ptrs from resctrl_exit()
>   x86/resctrl: Create helper for RMID allocation and mondata dir
>     creation
>   x86/resctrl: Move rmid allocation out of mkdir_rdt_prepare()
>   x86/resctrl: Track the closid with the rmid
>   x86/resctrl: Access per-rmid structures by index
>   x86/resctrl: Allow RMID allocation to be scoped by CLOSID
>   x86/resctrl: Track the number of dirty RMID a CLOSID has
>   x86/resctrl: Use __set_bit()/__clear_bit() instead of open coding
>   x86/resctrl: Allocate the cleanest CLOSID by searching
>     closid_num_dirty_rmid
>   x86/resctrl: Move CLOSID/RMID matching and setting to use helpers
>   x86/resctrl: Add cpumask_any_housekeeping() for limbo/overflow
>   x86/resctrl: Queue mon_event_read() instead of sending an IPI
>   x86/resctrl: Allow resctrl_arch_rmid_read() to sleep
>   x86/resctrl: Allow arch to allocate memory needed in
>     resctrl_arch_rmid_read()
>   x86/resctrl: Make resctrl_mounted checks explicit
>   x86/resctrl: Move alloc/mon static keys into helpers
>   x86/resctrl: Make rdt_enable_key the arch's decision to switch
>   x86/resctrl: Add helpers for system wide mon/alloc capable
>   x86/resctrl: Add CPU online callback for resctrl work
>   x86/resctrl: Allow overflow/limbo handlers to be scheduled on any-but
>     cpu
>   x86/resctrl: Add CPU offline callback for resctrl work
>   x86/resctrl: Move domain helper migration into resctrl_offline_cpu()
>   x86/resctrl: Separate arch and fs resctrl locks
> 
>  arch/x86/include/asm/resctrl.h            |  90 +++++
>  arch/x86/kernel/cpu/resctrl/core.c        | 102 ++---
>  arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  48 ++-
>  arch/x86/kernel/cpu/resctrl/internal.h    |  67 +++-
>  arch/x86/kernel/cpu/resctrl/monitor.c     | 463 +++++++++++++++++-----
>  arch/x86/kernel/cpu/resctrl/pseudo_lock.c |  15 +-
>  arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 359 ++++++++++++-----
>  include/linux/resctrl.h                   |  48 ++-
>  include/linux/tick.h                      |   9 +-
>  9 files changed, 921 insertions(+), 280 deletions(-)
>