Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp2522452yba; Mon, 15 Apr 2019 13:30:46 -0700 (PDT) X-Google-Smtp-Source: APXvYqxn5mxHYefAwF7jJWKwbf4ZH8Ex0up/TFaP7Lg9p+KQn9I/ARPMDevjtGN/Yrz4946pzhZa X-Received: by 2002:a63:1d45:: with SMTP id d5mr50612051pgm.184.1555360246091; Mon, 15 Apr 2019 13:30:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555360246; cv=none; d=google.com; s=arc-20160816; b=Y2G5ohNaeUxv0AuZo48lAze0oAcClUXid9Rm6+5Gvx4YqE9AXrUfrY050c3eZvAP6D B+++J5siW5gNT0PjhACxmTlerW/JZQGsQj3zYoa3WL4WVfE70FJVL8FIDUQ5EonLzVVV 8Dz35vEjUZQ6bvQILnN6VwjbKz1fay87NunSGRhwraTk5vDlJfVwv8cOjcA4yZaPSXI4 6LmEBZvK+hyXZ/QYheLwJPw/kSZYLvbk7JxozN+0VjXpXYgTgYa3lQC8tmZhlQYmDFen lD4SH5yBDOeJ6gjxzGbduNDh/EXehBICKz1vBZ8N631lEaAnlX7vN9dFUk5RghK2AR59 Sydw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=fMkMZ8/OOMiiU9FhnMkr14Lbz1A6KscuJvJYZGyizVs=; b=EOLCF7n081M3agCUNLonZ5TJw14EhS/Rs+lW9Wbdfhurpa366+wO4aKyd8ja7XfJhU 90mjr4tPOgIxm8bAim+PoNmNCzcuEutgdcklJRt6kIyYO/OEwyBhMUI1qlV/NX+2k7rq GQYAtQp9xgl7nhEhy5T5pVyaw7F7k4StzsunApV0H67kOuqTuz2arSUcsXbHq2GWDrNL Qy9cIMh9xCmnUp1w7iEEV/j4GfNUOCXIVg5sMdHiLDuXb73zCKRpNdi/86E/zNWeMbf/ Nm+lLW+WS1ZEjeWwSBwrtiaevlML+Ky4/1DJEz7kAcOdp5d7MS+63li7mBRpVBQf/PhG I4YA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=S6JFHlrR; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e6si45822004pgb.435.2019.04.15.13.30.29; Mon, 15 Apr 2019 13:30:46 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=S6JFHlrR; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730480AbfDOTIa (ORCPT + 99 others); Mon, 15 Apr 2019 15:08:30 -0400 Received: from mail.kernel.org ([198.145.29.99]:43878 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730461AbfDOTI2 (ORCPT ); Mon, 15 Apr 2019 15:08:28 -0400 Received: from localhost (83-86-89-107.cable.dynamic.v4.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E0E1C218DA; Mon, 15 Apr 2019 19:08:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1555355307; bh=KPYYhPZqSI/FP4Dxd7uiGmGTlPzzp5DZCgYuGDd8qOs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=S6JFHlrRlQWBP/lX2X/9pVCLykhOoLlWwayD+VIKMWsYYyRcoOWF5E2mlboN3A3W4 sBAgMRQbORBeJBNlUob5444U9ksh65mvj7QY+8Ht8hJnNTRboHaYNB0/NtCW4INXSm ca5WhLU+qz3V4iTQnqK6D8ybKiydnMigZn2Oz7K4= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Tom Lendacky , "Peter Zijlstra (Intel)" , Alexander Shishkin , Arnaldo Carvalho de Melo , Arnaldo Carvalho de Melo , Borislav Petkov , Jiri Olsa , Linus Torvalds , Namhyung Kim , Stephane Eranian , Thomas Gleixner , Vince Weaver , Ingo Molnar Subject: [PATCH 4.19 089/101] x86/perf/amd: Resolve NMI latency issues for active PMCs Date: Mon, 15 Apr 2019 20:59:27 +0200 Message-Id: <20190415183745.085762799@linuxfoundation.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190415183740.341577907@linuxfoundation.org> References: <20190415183740.341577907@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Lendacky, Thomas commit 6d3edaae16c6c7d238360f2841212c2b26774d5e upstream. On AMD processors, the detection of an overflowed PMC counter in the NMI handler relies on the current value of the PMC. So, for example, to check for overflow on a 48-bit counter, bit 47 is checked to see if it is 1 (not overflowed) or 0 (overflowed). When the perf NMI handler executes it does not know in advance which PMC counters have overflowed. As such, the NMI handler will process all active PMC counters that have overflowed. NMI latency in newer AMD processors can result in multiple overflowed PMC counters being processed in one NMI and then a subsequent NMI, that does not appear to be a back-to-back NMI, not finding any PMC counters that have overflowed. This may appear to be an unhandled NMI resulting in either a panic or a series of messages, depending on how the kernel was configured. To mitigate this issue, add an AMD handle_irq callback function, amd_pmu_handle_irq(), that will invoke the common x86_pmu_handle_irq() function and upon return perform some additional processing that will indicate if the NMI has been handled or would have been handled had an earlier NMI not handled the overflowed PMC. Using a per-CPU variable, a minimum value of the number of active PMCs or 2 will be set whenever a PMC is active. This is used to indicate the possible number of NMIs that can still occur. The value of 2 is used for when an NMI does not arrive at the LAPIC in time to be collapsed into an already pending NMI. Each time the function is called without having handled an overflowed counter, the per-CPU value is checked. If the value is non-zero, it is decremented and the NMI indicates that it handled the NMI. If the value is zero, then the NMI indicates that it did not handle the NMI. Signed-off-by: Tom Lendacky Signed-off-by: Peter Zijlstra (Intel) Cc: # 4.14.x- Cc: Alexander Shishkin Cc: Arnaldo Carvalho de Melo Cc: Arnaldo Carvalho de Melo Cc: Borislav Petkov Cc: Jiri Olsa Cc: Linus Torvalds Cc: Namhyung Kim Cc: Peter Zijlstra Cc: Stephane Eranian Cc: Thomas Gleixner Cc: Vince Weaver Link: https://lkml.kernel.org/r/Message-ID: Signed-off-by: Ingo Molnar Signed-off-by: Greg Kroah-Hartman --- arch/x86/events/amd/core.c | 56 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 55 insertions(+), 1 deletion(-) --- a/arch/x86/events/amd/core.c +++ b/arch/x86/events/amd/core.c @@ -4,10 +4,13 @@ #include #include #include +#include #include #include "../perf_event.h" +static DEFINE_PER_CPU(unsigned int, perf_nmi_counter); + static __initconst const u64 amd_hw_cache_event_ids [PERF_COUNT_HW_CACHE_MAX] [PERF_COUNT_HW_CACHE_OP_MAX] @@ -488,6 +491,57 @@ static void amd_pmu_disable_all(void) } } +/* + * Because of NMI latency, if multiple PMC counters are active or other sources + * of NMIs are received, the perf NMI handler can handle one or more overflowed + * PMC counters outside of the NMI associated with the PMC overflow. If the NMI + * doesn't arrive at the LAPIC in time to become a pending NMI, then the kernel + * back-to-back NMI support won't be active. This PMC handler needs to take into + * account that this can occur, otherwise this could result in unknown NMI + * messages being issued. Examples of this is PMC overflow while in the NMI + * handler when multiple PMCs are active or PMC overflow while handling some + * other source of an NMI. + * + * Attempt to mitigate this by using the number of active PMCs to determine + * whether to return NMI_HANDLED if the perf NMI handler did not handle/reset + * any PMCs. The per-CPU perf_nmi_counter variable is set to a minimum of the + * number of active PMCs or 2. The value of 2 is used in case an NMI does not + * arrive at the LAPIC in time to be collapsed into an already pending NMI. + */ +static int amd_pmu_handle_irq(struct pt_regs *regs) +{ + struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); + int active, handled; + + /* + * Obtain the active count before calling x86_pmu_handle_irq() since + * it is possible that x86_pmu_handle_irq() may make a counter + * inactive (through x86_pmu_stop). + */ + active = __bitmap_weight(cpuc->active_mask, X86_PMC_IDX_MAX); + + /* Process any counter overflows */ + handled = x86_pmu_handle_irq(regs); + + /* + * If a counter was handled, record the number of possible remaining + * NMIs that can occur. + */ + if (handled) { + this_cpu_write(perf_nmi_counter, + min_t(unsigned int, 2, active)); + + return handled; + } + + if (!this_cpu_read(perf_nmi_counter)) + return NMI_DONE; + + this_cpu_dec(perf_nmi_counter); + + return NMI_HANDLED; +} + static struct event_constraint * amd_get_event_constraints(struct cpu_hw_events *cpuc, int idx, struct perf_event *event) @@ -680,7 +734,7 @@ static ssize_t amd_event_sysfs_show(char static __initconst const struct x86_pmu amd_pmu = { .name = "AMD", - .handle_irq = x86_pmu_handle_irq, + .handle_irq = amd_pmu_handle_irq, .disable_all = amd_pmu_disable_all, .enable_all = x86_pmu_enable_all, .enable = x86_pmu_enable_event,