Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2;
Message-ID: <0022df64-b7f5-43b4-87ed-5df5d47c5c6a@amd.com>
Date:   Tue, 24 Oct 2023 13:30:59 -0500
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 2/2] perf/x86/amd: Don't allow pre-emption in
 amd_pmu_lbr_reset()
Content-Language: en-US
To:     Ingo Molnar <mingo@kernel.org>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Borislav Petkov <bp@alien8.de>,
        Thomas Gleixner <tglx@linutronix.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Sandipan Das <sandipan.das@amd.com>,
        "H . Peter Anvin" <hpa@zytor.com>, linux-kernel@vger.kernel.org,
        x86@kernel.org, linux-pm@vger.kernel.org, rafael@kernel.org,
        pavel@ucw.cz, linux-perf-users@vger.kernel.org,
        Ingo Molnar <mingo@redhat.com>,
        Arnaldo Carvalho de Melo <acme@kernel.org>,
        Mark Rutland <mark.rutland@arm.com>,
        Alexander Shishkin <alexander.shishkin@linux.intel.com>,
        Jiri Olsa <jolsa@kernel.org>,
        Namhyung Kim <namhyung@kernel.org>,
        Ian Rogers <irogers@google.com>,
        Adrian Hunter <adrian.hunter@intel.com>
References: <20231023160018.164054-1-mario.limonciello@amd.com>
 <20231023160018.164054-3-mario.limonciello@amd.com>
 <ZTd6BYr17ycdHR2a@gmail.com> <ZTf2IxAVPUFq91F4@gmail.com>
From:   Mario Limonciello <mario.limonciello@amd.com>
In-Reply-To: <ZTf2IxAVPUFq91F4@gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OFU0bnFLMVB1RnpJZHc2V2QvczJlS21VS1RvZElBNnMzRzdKb0Uvayt5OWJa?=
 =?utf-8?B?R0dJbkN0VEJIcko1WFJ4RWFXVG1JeXN1dW5vcHluRnY5VWhidk1Lbm1IOEMw?=
 =?utf-8?B?b1Y0eFFnazMwVmlVWkwrTGxiL25wODAySmt0OS9URmpaTmNHV2hvOTdkTElJ?=
 =?utf-8?B?Mm1sSEhuTWo5UHNYczFCdjhhcFl0TzZEekcvZmlFdWh0YmMwa2JVZVFxVXFn?=
 =?utf-8?B?ekZrSUVQd3NKY2pQY2dmTGdHczZQVmdUNHdFSjFHcys1cjVRVnoyRW96YmVa?=
 =?utf-8?B?SFpuQUVDcklMU2QreUJOTkRScHZvVFlJUTN5OUM1SFZqZzdPajE5MVU3TSs2?=
 =?utf-8?B?c1VNMXlnMHBveEtNTnp1em5Dc1Z1VDEyMTBFRTNyazYwTkx6RHMxYjdiRnJV?=
 =?utf-8?B?K2xKOVQ2NS96UzdZcXZhWEZUWDJyRjhvRDdEN08xbXBXcmhOY0YrR09RMzRL?=
 =?utf-8?B?SDZKU1kwNUZTOC94TVBTMEpERmZJNk04WWQ4cVVIeDFJVVF1UDJpNVZGTkNW?=
 =?utf-8?B?N2tPK3k5TVNKT09LTDZRbjFydkx6K3J3WGRCd21kWGdUcEw1NXpmUFBXQTJE?=
 =?utf-8?B?b1hOM1Y3LzM3TkJ1UWJkemFGSk1JTVJSUnlobGdaek9MQzBjWVBoNmV4K0FB?=
 =?utf-8?B?SVU4VUtxSlFjb29SMHdLZGNnWHJWWFZUMWE5bkdRSVhnMGpRZkFxcnIwekh2?=
 =?utf-8?B?bW42NzF2NjU0OGJEb2JCNTRyelQ1M3FBeE91Sld3YUNJUndLYnRlaTMrVVFm?=
 =?utf-8?B?THZxcHQ0VW5LU3RkTjF3ZFg2aUNKblFCTlRxSFNJY3pxcXJOWGgrZ25iN0pJ?=
 =?utf-8?B?OGdlMEpab1JTZ3Q2QTIrZ3NKUlpHMWpvcXJ0cFJRZUxaY2VpNlZXaTlmYWp0?=
 =?utf-8?B?MjNsK1Y0S3lZbS9vU2ZMdHJoMmJPSG1EcVNYcGFsRzV6SXpYVFV0NXI1emUz?=
 =?utf-8?B?YzdqdGppZTQ1ZVA2ZktnR2JmVlRReFlDaFlIcHFHSnVNZitvWGF1SENGUDF2?=
 =?utf-8?B?YWRiaUtTNjdlK2FueElJVGxXOXc3QUhnZ1MzcjZPdXR4TnN1WCsxSmZZOEc2?=
 =?utf-8?B?VWZad3VqcTdwcHBTclJnNE80SFd3RGFzRU5sd1JTYmcwR0xUckxMckdjbGlm?=
 =?utf-8?B?QUR0c2pNY2J3S05Rc1dsUEJKalgwQlFwaVpUYjdGVzBKUFJCU21LTzlPQXNY?=
 =?utf-8?B?L1l6SW5Pb0pOUFNEOWVCMS9MQ3RwVEtOcHZXamVrOURUQk8vZXVTb2lMdXpt?=
 =?utf-8?B?dEpuS0p1SkFwRFh6MzZTVWpNYjNYVSs3Z0x4cXZHc2s4eGpmY1FhODVNY21Q?=
 =?utf-8?B?WnV6V1dPcEJ0d3VQb21uS0ZxNEsrZ3BjQzBIYm5TbjIrSWxsdlpRcnp3M1VY?=
 =?utf-8?B?RHNlYkpSV3Q0Vngrajl2S0lDRlZoeFhsSXFkS3REQVp5QnRvcnllZjZSQ0tl?=
 =?utf-8?B?UktTSk5SZzZIcC9vbnNycGE0SlhmeTZiVjFKVSs0cFkzaldTNmczbWY0R25Q?=
 =?utf-8?B?NndWc2Q3QmVlR0FhSHB1a0lsMFdVbko4SUtNOHZFazRDZnVpaDRUdWlvak0z?=
 =?utf-8?B?QTNpVXVrd29kdWNuWHFyZ0R1eGpobVEvcFBab3dyN1d3LzE2UkpJWnM0N1o1?=
 =?utf-8?B?NlQ0L2RzQ29INlpkVzI0Y2kwUmx5T2hZcE9WOEpBV2E3YkVHYldUTzdBaFJp?=
 =?utf-8?B?L2ViTXl4WUovRUJodkNKanVLb083aXd0czJQeDB4TjRRTUNmV3NoQngxaW1o?=
 =?utf-8?B?RVlRQ29RalJlbTVXeS85RnBYOU9mZlhmc250MitaRlNGOHpSK3ZrK2RLZjVE?=
 =?utf-8?B?NEw1NDh2YUh4ekZOYVd1MWtEOUdFT0tFUU1zSk1qeWxrc2k5VFVkZFlBeDEw?=
 =?utf-8?B?Q21mUWh5cG90WVJRWFYwdHR4TzhNUGl6RjhzQXhCN0h2QmpEMzc3UytyRGVw?=
 =?utf-8?B?UmpnT1dzd1VhUjdqUG1OdEpOTE1aMUVISTBiUXZIb1JzTEhBK29XMGpBaWFY?=
 =?utf-8?B?djlNVUFPdFVJWUt1UkdLQVd3OExWUm5kVndZbk1rd3NTM0pyZ0Z1TWx5dW5R?=
 =?utf-8?B?VVpYVm9lY0RqQkJLKy9XVmRyQnFyWW1xV1FHYmlKeUFCSStvQm5aRGdiUmE1?=
 =?utf-8?Q?S+N0p6dXC37xGipGCn59ZC5+v?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2577aea7-c0e7-41d7-9e83-08dbd4bf60c1
X-MS-Exchange-CrossTenant-AuthSource: MN0PR12MB6101.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Oct 2023 18:31:03.2449
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LbdfaiaaVur5ldaNSoihWHbwEHVGW0nuEPXlfm5IEKrXxv3sgKMaoMNdMWuWIl1yq6xHmFc5fYSkVaoXnwPJlg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5456
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Tue, 24 Oct 2023 11:31:15 -0700 (PDT)

On 10/24/2023 11:51, Ingo Molnar wrote:
> 
> * Ingo Molnar <mingo@kernel.org> wrote:
> 
>>
>> * Mario Limonciello <mario.limonciello@amd.com> wrote:
>>
>>> Fixes a BUG reported during suspend to ram testing.
>>>
>>> ```
>>> [  478.274752] BUG: using smp_processor_id() in preemptible [00000000] code: rtcwake/2948
>>> [  478.274754] caller is amd_pmu_lbr_reset+0x19/0xc0
>>> ```
>>>
>>> Cc: stable@vger.kernel.org # 6.1+
>>> Fixes: ca5b7c0d9621 ("perf/x86/amd/lbr: Add LbrExtV2 branch record support")
>>> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
>>> ---
>>>   arch/x86/events/amd/lbr.c | 3 ++-
>>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
>>> index eb31f850841a..5b98e8c7d8b7 100644
>>> --- a/arch/x86/events/amd/lbr.c
>>> +++ b/arch/x86/events/amd/lbr.c
>>> @@ -321,7 +321,7 @@ int amd_pmu_lbr_hw_config(struct perf_event *event)
>>>   
>>>   void amd_pmu_lbr_reset(void)
>>>   {
>>> -	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
>>> +	struct cpu_hw_events *cpuc = get_cpu_ptr(&cpu_hw_events);
>>>   	int i;
>>>   
>>>   	if (!x86_pmu.lbr_nr)
>>> @@ -335,6 +335,7 @@ void amd_pmu_lbr_reset(void)
>>>   
>>>   	cpuc->last_task_ctx = NULL;
>>>   	cpuc->last_log_id = 0;
>>> +	put_cpu_ptr(&cpu_hw_events);
>>>   	wrmsrl(MSR_AMD64_LBR_SELECT, 0);
>>>   }
>>
>> Weird, amd_pmu_lbr_reset() is called from these places:
>>
>>    - amd_pmu_lbr_sched_task(): during task sched-in during
>>      context-switching, this should already have preemption disabled.
>>
>>    - amd_pmu_lbr_add(): this gets indirectly called by amd_pmu::add
>>      (amd_pmu_add_event()), called by event_sched_in(), which too should have
>>      preemption disabled.
>>
>> I clearly must have missed some additional place it gets called in.
> 
> Just for completeness, the additional place I missed is
> amd_pmu_cpu_reset():
> 
>                  static_call(amd_pmu_branch_reset)();
> 
> ... and the amd_pmu_branch_reset static call is set up with
> amd_pmu_lbr_reset, which is why git grep missed it.
> 
> Anyway, amd_pmu_cpu_reset() is very much something that should run
> non-preemptable to begin with, so your patch only papers over the real
> problem AFAICS.
> 
> Thanks,
> 
> 	Ingo

In that case - should preemption be disabled for all of 
x86_pmu_dying_cpu() perhaps?

For good measure x86_pmu_starting_cpu() too?