Received-SPF: pass (google.com: domain of linux-kernel+bounces-73792-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <d409c32d-0f9a-4773-bc25-9d39de3c9e9b@intel.com>
Date: Tue, 20 Feb 2024 14:58:03 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v9 00/24] x86/resctrl: monitored closid+rmid together,
 separate arch/fs locking
Content-Language: en-US
To: Tony Luck <tony.luck@intel.com>, Thomas Gleixner <tglx@linutronix.de>
CC: Borislav Petkov <bp@alien8.de>, James Morse <james.morse@arm.com>,
	<x86@kernel.org>, <linux-kernel@vger.kernel.org>, Fenghua Yu
	<fenghua.yu@intel.com>, Ingo Molnar <mingo@redhat.com>, H Peter Anvin
	<hpa@zytor.com>, Babu Moger <Babu.Moger@amd.com>,
	<shameerali.kolothum.thodi@huawei.com>, D Scott Phillips OS
	<scott@os.amperecomputing.com>, <carl@os.amperecomputing.com>,
	<lcherian@marvell.com>, <bobo.shaobowang@huawei.com>,
	<tan.shaopeng@fujitsu.com>, <baolin.wang@linux.alibaba.com>, Jamie Iles
	<quic_jiles@quicinc.com>, Xin Hao <xhao@linux.alibaba.com>,
	<peternewman@google.com>, <dfustini@baylibre.com>, <amitsinght@marvell.com>,
	David Hildenbrand <david@redhat.com>
References: <20240213184438.16675-1-james.morse@arm.com>
 <20240217105543.GAZdCQr_nosDP4tGuO@fat_crate.local> <87zfvwieli.ffs@tglx>
 <ZdUSwOM9UUNpw84Y@agluck-desk3>
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <ZdUSwOM9UUNpw84Y@agluck-desk3>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NFhVdm53L1V1bVVaNDA2dU04ZnFqaWk3ZUVhd1lMUTJMMEpHRFVpeDI5bGVj?=
 =?utf-8?B?Z0lUNVBFVG8xekRhNmlkVHA4N25ST2xOa2xDblowYkpUWEFSbmwwOXhHOFdi?=
 =?utf-8?B?Mk9LdFd0Yk5aUjRXUFVSNzY0WnBEYVpTK0YwSFpCcFh1M21haGpLYVIwMkt3?=
 =?utf-8?B?LytKSlE0cW1vQkhlcjVDMFg4OEw0d1loT1d1TExKcGY1ZWMydmxSTzJydGgr?=
 =?utf-8?B?Z0dxM0cxWjZ3UzNjZkJFczlGaEpnZjRJVTd6S1krTU5EL3h1TkxIT2J1c0kw?=
 =?utf-8?B?K2wwRHNXamx1eVFMWlhIeTVoK3kxZ25ya2VCcFl5cktZUTNQVXlMekg0b01k?=
 =?utf-8?B?cFJoMXp3V2IxdlFGOFRpMEY2a2Zob0twRkRpWU02OW5wN3NXelFSTnhTWWpn?=
 =?utf-8?B?aXArV1NPV0IxOFlIcUVlZy8vT09XZ1FmUE56ck50N1Q4QVlqTzJscmNWdXdX?=
 =?utf-8?B?TnAyMmc2eml1VFplZWlHczR4bFFVOUw1eDNBZkoyNGE0VHc2Y3RmMEFFcG1V?=
 =?utf-8?B?TGhaRlc5WUN5Nm8vNUVVblRTN0kxVmdXcS9VbERuQ3kwWHUxaVlickpVQWRU?=
 =?utf-8?B?Nkw0R0x2dTFhSkJ0VzV2U1RnYWQrckc3bXozWkNXQXdZZm9KdFhJemwvcUxG?=
 =?utf-8?B?MlRNSW5IRnVjRlRncHdxSVFNdHI0VE9TemZNcWcxZlhxdzZRK2FJWDZrM3I0?=
 =?utf-8?B?WnZwVkdDWTNUSDZtR0pDRTR3dEdaRXRlN291SGxlbmpvcnFvaEZrVUNVV3hN?=
 =?utf-8?B?WmNlakl4RW1ibmczaVh5RG1TbmFPeVRJQVdyWmN1Y2hpbVJLSVIxTUUwZUxP?=
 =?utf-8?B?NVZzdjc4ZVdLRjJwK3J5Ri9MUG1CSllEUE5WZFFmRzZGVy9SbFYrN0ZLSXBX?=
 =?utf-8?B?akg5bEE0c0xxUUZ0UUIwOGErcDh5dFhpRDc3cWJqYXRYVmJqR3Z0NTV2Yjg3?=
 =?utf-8?B?cDBNeUk1RzhOaU9ORVFEMzRLV1Z3YXQrcnpJbzhyRGM4dUt3eUJiM05ocmN6?=
 =?utf-8?B?NEpmZC9iWGZiWlNieGpGckdQeFZ3ZGI3cGc1OGxOMk9LNHJ5SVg3bXNyRStn?=
 =?utf-8?B?dE1ER1hOajJ1bk5VR2NwQWZ6cHpPakJ5ZHhZMStETnZYais2akhWTkhvcDRE?=
 =?utf-8?B?VW8yaWNKQU1FR2Q5ZnF5TDJ6WmtIK3BNdFNjY29GQld2MEJJUE1YM3daUWpR?=
 =?utf-8?B?VFFjcUsyYjl1TTNVSitETkt2ZlkzUDRCVHZ6VDBFL1NMV21WL2pkeWt0ZmF4?=
 =?utf-8?B?TmpleklnZDBhZU5ZUzBEbUtTMENpNHowQ2dLSlI5VmhCaFNUYTJrKzBxRWFp?=
 =?utf-8?B?MWp6OEI3eWdRMXVMdi9rV28zY0YweEJBRk9NeTJxSXVwbk1IdnFMZDZzRTli?=
 =?utf-8?B?WUVuaHlJMk1LZzE2L01uYzcvbkZJUi9Xb0c5OUdJblBXc2RPQjA2VXZZV1h0?=
 =?utf-8?B?cDVNdVZhWXhSeWpqRlozL2dQTjZFM0dGRHVEUG5pS1VVQ1Z2SVphMDcyU1pR?=
 =?utf-8?B?Uit3UmNlOU9GQTBINjVLUUM0TzEzbnJJeTNBakRRSVNhbWhnT2tJNmJjNHJ0?=
 =?utf-8?B?OGh6UkJWS1JjZU9QMElaakVWTnBVNk5nQUJSZVV1Q1dYUEU2MzI0V2l3bnZB?=
 =?utf-8?B?dzl1Mm1UMytMR3dXaEFoZnJWaEdQejRDamRpU0EwSUdHd25DVjNWNFQxSmtL?=
 =?utf-8?B?YjBsa2xvSHFkZWdXOEFlalhjV1g0bGNDNzBmTzBIaDA1OFNWdFV5bmVzcU16?=
 =?utf-8?B?amF2UG91ZDVPSnh2eFFyUnp2bXR4ckVxQUFZeU9ZVWxwaVdwRjRsYWxCekFH?=
 =?utf-8?B?Y3cwVEQ0Qm9mS3pXMHJYSW9uc2VvaGtVSWVybFRKbUp6ZmRJVGhSeWxheWtH?=
 =?utf-8?B?WE9UM0U4Y1NRNHR3OUJmc2pWYjRWOVdsZi9BRE0zejl4UHJjSU82NmprR2tU?=
 =?utf-8?B?a2xyU0FLVWYxc3E3NGZ2Z1IvSnFJMHlXajA2YWVxQXlsV2FtNHFkY3BhQlhM?=
 =?utf-8?B?SDI4NzRLM0RYcU9lWE9lOTJEa2FHT3NFR0FjQUlSMUFjZGEyS0dNTkhmZ1Ew?=
 =?utf-8?B?MSsxSTdSVWtSa1F2c2pEQ1U3b0dsT2hhVURkR0JiaDduRE5zQ1ZHNkNET2tR?=
 =?utf-8?B?dkswaC90S3h1MC9McUM2QmFmR3AzS1FxYXJsTmxsMGRNZkRxdDcvWEFJbUQ4?=
 =?utf-8?B?cWc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5d6f9e4c-1a3f-4583-d019-08dc326766d0
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2024 22:58:06.9168
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: MQbvgztxrBV/6EpZr7Frpe9n5D+30tyjBrWiWzWEdkr6oNCfbae8aChfvUGtwJqLDZu48bdTKJK9gehBwUeoGIzMKjFb0O/Cr1PBVo4SDWU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB8131
X-OriginatorOrg: intel.com


On 2/20/2024 12:59 PM, Tony Luck wrote:
> On Mon, Feb 19, 2024 at 05:49:29PM +0100, Thomas Gleixner wrote:
>> On Sat, Feb 17 2024 at 11:55, Borislav Petkov wrote:
>>
>>> On Tue, Feb 13, 2024 at 06:44:14PM +0000, James Morse wrote:
>>>> Hello!
>>>>
>>>> It's been back and forth for whether this series should be rebased onto Tony's
>>>> SNC series. This version isn't, its based on tip/x86/cache.
>>>> (I have the rebased-and-tested versions if anyone needs them)
>>>
>>> The set applied ontop of tip:x86/cache gives:
>>>
>>> vmlinux.o: in function `get_domain_from_cpu':
>>> (.text+0x150f33): undefined reference to `lockdep_is_cpus_held'
>>> ld: vmlinux.o: in function `rdt_ctrl_update':
>>> (.text+0x150fbc): undefined reference to `lockdep_is_cpus_held'
>>
>> Wants to be folded into patch 24.
>>
>> Thanks,
>>
>>         tglx
>> ---
>> --- a/arch/x86/kernel/cpu/resctrl/core.c
>> +++ b/arch/x86/kernel/cpu/resctrl/core.c
>> @@ -368,8 +368,8 @@ struct rdt_domain *get_domain_from_cpu(i
>>  	 * about locks this thread holds will lead to false positives. Check
>>  	 * someone is holding the CPUs lock.
>>  	 */
>> -	if (IS_ENABLED(CONFIG_LOCKDEP))
>> -		lockdep_is_cpus_held();
>> +	if (IS_ENABLED(CONFIG_HOTPLUG_CPU) && IS_ENABLED(CONFIG_LOCKDEP))
>> +		WARN_ON_ONCE(!lockdep_is_cpus_held());
>>  
>>  	list_for_each_entry(d, &r->domains, list) {
>>  		/* Find the domain that contains this CPU */
> 
> Testing tip x86/cache that WARN fires while running
> tools/tests/selftests/resctrl/resctrl_test.
> 
> Everthing runs OK if I drop the top commit:
>   fb700810d30b ("x86/resctrl: Separate arch and fs resctrl locks")

The new WARN_ON_ONCE() is why this encountered. The comment notes that
lockdep_is_cpus_held() is used to determine if "someone is holding the
CPUs lock" but it seems that lockdep_is_cpus_held() still only checks
if "current" is holding cpu_hotplug_lock and that is not possible
when running the code via IPI.

The trace that Tony shared notes that this is triggered by get_domain_from_cpu()
called via rdt_ctrl_update(). rdt_ctrl_update() is only run via IPI:

	resctrl_arch_update_domains() {
		...
		lockdep_assert_cpus_held();
		...
		on_each_cpu_mask(cpu_mask, rdt_ctrl_update, &msr_param, 1);
		...
	}

and

	reset_all_ctrls() {
		...
		lockdep_assert_cpus_held();
		...
		on_each_cpu_mask(cpu_mask, rdt_ctrl_update, &msr_param, 1);
	}

I sprinkled some debug_show_held_locks(current) to confirm and encountered
the following when reproducing the trace using the resctrl tests:

[  202.914334] resctrl_arch_update_domains:355
[  202.919971] 4 locks held by resctrl_tests/3330:
[  202.925169]  #0: ff11001086e09408 (sb_writers#15){.+.+}-{0:0}, at: ksys_write+0x69/0x100
[  202.934375]  #1: ff110010bb653688 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0xf7/0x200
[  202.944348]  #2: ffffffff8346c890 (cpu_hotplug_lock){++++}-{0:0}, at: rdtgroup_kn_lock_live+0x4c/0xa0
[  202.954774]  #3: ffffffff8344ae68 (rdtgroup_mutex){+.+.}-{3:3}, at: rdtgroup_kn_lock_live+0x5a/0xa0
[  202.965030] get_domain_from_cpu:366
[  202.969087] no locks held by swapper/0/0.
[  202.973697] ------------[ cut here ]------------
[  202.978979] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/cpu/resctrl/core.c:375 get_domain_from_cpu+0x6f/0x80
<SNIP>
[  203.200095] Call Trace:
[  203.202947]  <TASK>
[  203.205406]  ? __warn+0x84/0x180
[  203.209123]  ? get_domain_from_cpu+0x6f/0x80
[  203.214011]  ? report_bug+0x1c7/0x1e0
[  203.218214]  ? handle_bug+0x3c/0x80
[  203.222230]  ? exc_invalid_op+0x18/0x80
[  203.227198]  ? asm_exc_invalid_op+0x1a/0x20
[  203.232529]  ? __pfx_rdt_ctrl_update+0x20/0x20
[  203.238146]  ? get_domain_from_cpu+0x6f/0x80
[  203.243548]  rdt_ctrl_update+0x26/0x80
<SNIP>


So even though it is confirmed via lockdep_assert_cpus_held() that
resctrl_arch_update_domains() holds cpu_hotplug_lock, it does not seem possible
to have a similar lockdep check in the function called by it (resctrl_arch_update_domains())
via IPI. It thus does not look like that lockdep checking within
get_domain_from_cpu() can be accurate and I cannot see what it can be replaced with
to make it accurate. Any guidance will be appreciated. Perhaps we should just drop (but
with detailed context comments remaining) the lockdep check in get_domain_from_cpu()? 

Reinette