Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp2477385yba; Mon, 15 Apr 2019 12:28:05 -0700 (PDT) X-Google-Smtp-Source: APXvYqzV49Q7L1pSmNGSb+YvmpHJf/zC2L5ea1fMyhu5FebNEEaRoSVNjeP21pyH2zthH4hracSh X-Received: by 2002:a62:2587:: with SMTP id l129mr77927507pfl.151.1555356485039; Mon, 15 Apr 2019 12:28:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555356485; cv=none; d=google.com; s=arc-20160816; b=Lj/2j869WNY1hbJUW+DjzSGJi6YA1ZKyFLFreJsAisXBU40LqZgQ43MsvuHWVlGj/g 6aNJqLh/WvQs0eVIXJ3KD3g/RAIpISPFSyEDu76TFDL+vpTLqKnkMg+kbuzcESXGj7/7 F5/UCla25oV7m/kN64PImDBSCh0LnFgbOhCHSQ4NcqnqHmfgGAX+fEtlDF6f6GUlHiPM LTviKxMfGfBZkbpBpKcUYJ0pnbProrS3+DU65RZtRtN/s6Z1eNHYauzFvrLdcL6wu/oP bM/BUHdqMtHfRTr8OvNm9Q+gJ0c0dgyAAOfN6Hmprn1aVmqgOBRmNm306gpKE3vV0JdK ww2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=fMkMZ8/OOMiiU9FhnMkr14Lbz1A6KscuJvJYZGyizVs=; b=VIv7pzB6tF/T/BVYbEEHivqrNlvKp+eVdSENnhxY3V/Y8mhPc3F3XKR/BJUaLhhVJV 8fGgYvW3kpLTLah8L+fpRRgYtcL8kAAQfzoPzpZ14hGXWibzfrQRVe0Zq6uuCyWSf8uu 3Jv7AIG5+6bqSZf9Gr1F8/fkYy0+kkxJFeS7V9uuQF9vkPkIuYrHlejk/k/yXs1REHyA IXJnffDmVociEQsN/duEtC35zWzR2g7IIlcLpdFvjyhTpaKRrt5+wjxli16lCyYTFRC2 aB+WBjnlKFsPX+P3KYeYJgwOsEnWAZPYR7Sk0DQ61Ohpwo92kOVVWcg9uIffjSQA7XzP s69Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=uHI5XZVc; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 19si44451614pgl.402.2019.04.15.12.27.48; Mon, 15 Apr 2019 12:28:05 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=uHI5XZVc; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729588AbfDOTEm (ORCPT + 99 others); Mon, 15 Apr 2019 15:04:42 -0400 Received: from mail.kernel.org ([198.145.29.99]:37698 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729569AbfDOTEi (ORCPT ); Mon, 15 Apr 2019 15:04:38 -0400 Received: from localhost (83-86-89-107.cable.dynamic.v4.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id BFF3D20880; Mon, 15 Apr 2019 19:04:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1555355077; bh=KPYYhPZqSI/FP4Dxd7uiGmGTlPzzp5DZCgYuGDd8qOs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uHI5XZVcI6DgTYOcrfh5xEKcyNn6ufWmjmD4B7DzZ1kgSe4MTFfL/sU+6Tvnmv59r kfhsaObi4QmVAb9QfWwHPvDcWPecsXGCBUj7xDxcoc2EVtHNaLEMPEjNrepzRhy19g 2OrlIrUcouNaPUlh0bxMYSOECoKjHc9jd5eqVHwg= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Tom Lendacky , "Peter Zijlstra (Intel)" , Alexander Shishkin , Arnaldo Carvalho de Melo , Arnaldo Carvalho de Melo , Borislav Petkov , Jiri Olsa , Linus Torvalds , Namhyung Kim , Stephane Eranian , Thomas Gleixner , Vince Weaver , Ingo Molnar Subject: [PATCH 4.14 64/69] x86/perf/amd: Resolve NMI latency issues for active PMCs Date: Mon, 15 Apr 2019 20:59:22 +0200 Message-Id: <20190415183736.083652040@linuxfoundation.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190415183726.036654568@linuxfoundation.org> References: <20190415183726.036654568@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Lendacky, Thomas commit 6d3edaae16c6c7d238360f2841212c2b26774d5e upstream. On AMD processors, the detection of an overflowed PMC counter in the NMI handler relies on the current value of the PMC. So, for example, to check for overflow on a 48-bit counter, bit 47 is checked to see if it is 1 (not overflowed) or 0 (overflowed). When the perf NMI handler executes it does not know in advance which PMC counters have overflowed. As such, the NMI handler will process all active PMC counters that have overflowed. NMI latency in newer AMD processors can result in multiple overflowed PMC counters being processed in one NMI and then a subsequent NMI, that does not appear to be a back-to-back NMI, not finding any PMC counters that have overflowed. This may appear to be an unhandled NMI resulting in either a panic or a series of messages, depending on how the kernel was configured. To mitigate this issue, add an AMD handle_irq callback function, amd_pmu_handle_irq(), that will invoke the common x86_pmu_handle_irq() function and upon return perform some additional processing that will indicate if the NMI has been handled or would have been handled had an earlier NMI not handled the overflowed PMC. Using a per-CPU variable, a minimum value of the number of active PMCs or 2 will be set whenever a PMC is active. This is used to indicate the possible number of NMIs that can still occur. The value of 2 is used for when an NMI does not arrive at the LAPIC in time to be collapsed into an already pending NMI. Each time the function is called without having handled an overflowed counter, the per-CPU value is checked. If the value is non-zero, it is decremented and the NMI indicates that it handled the NMI. If the value is zero, then the NMI indicates that it did not handle the NMI. Signed-off-by: Tom Lendacky Signed-off-by: Peter Zijlstra (Intel) Cc: # 4.14.x- Cc: Alexander Shishkin Cc: Arnaldo Carvalho de Melo Cc: Arnaldo Carvalho de Melo Cc: Borislav Petkov Cc: Jiri Olsa Cc: Linus Torvalds Cc: Namhyung Kim Cc: Peter Zijlstra Cc: Stephane Eranian Cc: Thomas Gleixner Cc: Vince Weaver Link: https://lkml.kernel.org/r/Message-ID: Signed-off-by: Ingo Molnar Signed-off-by: Greg Kroah-Hartman --- arch/x86/events/amd/core.c | 56 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 55 insertions(+), 1 deletion(-) --- a/arch/x86/events/amd/core.c +++ b/arch/x86/events/amd/core.c @@ -4,10 +4,13 @@ #include #include #include +#include #include #include "../perf_event.h" +static DEFINE_PER_CPU(unsigned int, perf_nmi_counter); + static __initconst const u64 amd_hw_cache_event_ids [PERF_COUNT_HW_CACHE_MAX] [PERF_COUNT_HW_CACHE_OP_MAX] @@ -488,6 +491,57 @@ static void amd_pmu_disable_all(void) } } +/* + * Because of NMI latency, if multiple PMC counters are active or other sources + * of NMIs are received, the perf NMI handler can handle one or more overflowed + * PMC counters outside of the NMI associated with the PMC overflow. If the NMI + * doesn't arrive at the LAPIC in time to become a pending NMI, then the kernel + * back-to-back NMI support won't be active. This PMC handler needs to take into + * account that this can occur, otherwise this could result in unknown NMI + * messages being issued. Examples of this is PMC overflow while in the NMI + * handler when multiple PMCs are active or PMC overflow while handling some + * other source of an NMI. + * + * Attempt to mitigate this by using the number of active PMCs to determine + * whether to return NMI_HANDLED if the perf NMI handler did not handle/reset + * any PMCs. The per-CPU perf_nmi_counter variable is set to a minimum of the + * number of active PMCs or 2. The value of 2 is used in case an NMI does not + * arrive at the LAPIC in time to be collapsed into an already pending NMI. + */ +static int amd_pmu_handle_irq(struct pt_regs *regs) +{ + struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); + int active, handled; + + /* + * Obtain the active count before calling x86_pmu_handle_irq() since + * it is possible that x86_pmu_handle_irq() may make a counter + * inactive (through x86_pmu_stop). + */ + active = __bitmap_weight(cpuc->active_mask, X86_PMC_IDX_MAX); + + /* Process any counter overflows */ + handled = x86_pmu_handle_irq(regs); + + /* + * If a counter was handled, record the number of possible remaining + * NMIs that can occur. + */ + if (handled) { + this_cpu_write(perf_nmi_counter, + min_t(unsigned int, 2, active)); + + return handled; + } + + if (!this_cpu_read(perf_nmi_counter)) + return NMI_DONE; + + this_cpu_dec(perf_nmi_counter); + + return NMI_HANDLED; +} + static struct event_constraint * amd_get_event_constraints(struct cpu_hw_events *cpuc, int idx, struct perf_event *event) @@ -680,7 +734,7 @@ static ssize_t amd_event_sysfs_show(char static __initconst const struct x86_pmu amd_pmu = { .name = "AMD", - .handle_irq = x86_pmu_handle_irq, + .handle_irq = amd_pmu_handle_irq, .disable_all = amd_pmu_disable_all, .enable_all = x86_pmu_enable_all, .enable = x86_pmu_enable_event,