Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp2459733yba; Mon, 15 Apr 2019 12:06:11 -0700 (PDT) X-Google-Smtp-Source: APXvYqwv4rNSbHwKq2QzVK/tO4u8AeybHgbxbEqTFFBXjOOI1upMg8AaCxSc5Sy7Gw/PG1MhJBmn X-Received: by 2002:a17:902:294b:: with SMTP id g69mr28304259plb.57.1555355171738; Mon, 15 Apr 2019 12:06:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555355171; cv=none; d=google.com; s=arc-20160816; b=B9gBxv+KIVYJ1gQd7ZRPwxehbwvx9zYDaLC/ivBPQniebvGoCdGFYIHLLK689Gedsd /QYMM2FcRqdEAoUhRZZzIbH9DeP2D9CKStnLSmVWdC/Jf+G35MbSwdh6bMklVnOCNTye 2Kzo24M/heEIcudX+fzkmUiqJxb0iNi8m1CNB1v+dRJJOBRNDbAW1LtQAENl9+K77SXD 3x0ttebgTIApYi/mLjinjYEBiMePWcsmY3x+ieW7b8VsSoaEnoFIjpHrTUFyZZijn/mc r+CBanyIgtc6My7gKrQgrxagryUgfduL8Epo8UmjOid4ejdiF2IklX9AQCeN3wSPzDEH kNDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=9nv+ROqsgpSocQntAdImunfdo8vWVUVmNSrLKc0p9n4=; b=vnZufgGjCRHO5eqLYz8+pFy7DeQcoR9Mg5RVyXK+dRYlXiz02RCVf2tu5ykQmgEqJk hR2KC+us//jH1sP4KwxY0n26NmZBaJSHIlTKajGCsS8CTLSHo/0nQ7jdthUGDAuFZD6S oxEIQhDIZEd4LNf9DkYa0po2nM879f6Zkv0sNH6OVI5frUAd3yVmR9O6bHciqh9t+U0H hzxpWwc1Ch2oMuedlaqAqLVMWc4Jc4d+6KcAsHLroE5Wcs/3kPt7ic/1bz7u4FH/m3E6 CLpK1y2ONchoHQaHOm8tvihOJLBh3vwYO+DW4t4/Can5rlau8go9shN4OFekb1Phj3zt IiCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=jOgMUeCr; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f191si44615538pgc.570.2019.04.15.12.05.55; Mon, 15 Apr 2019 12:06:11 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=jOgMUeCr; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729575AbfDOTEi (ORCPT + 99 others); Mon, 15 Apr 2019 15:04:38 -0400 Received: from mail.kernel.org ([198.145.29.99]:37612 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729519AbfDOTEg (ORCPT ); Mon, 15 Apr 2019 15:04:36 -0400 Received: from localhost (83-86-89-107.cable.dynamic.v4.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2C5D4218A1; Mon, 15 Apr 2019 19:04:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1555355074; bh=UKLIMG4WBw3HBVghBcftB39EWuojo1+j/6tovH03OeE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jOgMUeCrmxzwe+wCOM+aLCVFRFPK/Lsx8XBFUN07QsQQJMYF6ulKlnxtZXxWC23nx 9BZaY1Eof7YcEBobnpoulPrJmiW9zlPoWjt/0Y5MuocBNdq1cu5h+0jhZG4Ks6kQqQ F2izIowblT9B7Xd0OhOsxFYWTHaGkTH8jSCYJBvU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Tom Lendacky , "Peter Zijlstra (Intel)" , Alexander Shishkin , Arnaldo Carvalho de Melo , Arnaldo Carvalho de Melo , Borislav Petkov , Jiri Olsa , Linus Torvalds , Namhyung Kim , Stephane Eranian , Thomas Gleixner , Vince Weaver , Ingo Molnar Subject: [PATCH 4.14 63/69] x86/perf/amd: Resolve race condition when disabling PMC Date: Mon, 15 Apr 2019 20:59:21 +0200 Message-Id: <20190415183735.786951968@linuxfoundation.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190415183726.036654568@linuxfoundation.org> References: <20190415183726.036654568@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Lendacky, Thomas commit 914123fa39042e651d79eaf86bbf63a1b938dddf upstream. On AMD processors, the detection of an overflowed counter in the NMI handler relies on the current value of the counter. So, for example, to check for overflow on a 48 bit counter, bit 47 is checked to see if it is 1 (not overflowed) or 0 (overflowed). There is currently a race condition present when disabling and then updating the PMC. Increased NMI latency in newer AMD processors makes this race condition more pronounced. If the counter value has overflowed, it is possible to update the PMC value before the NMI handler can run. The updated PMC value is not an overflowed value, so when the perf NMI handler does run, it will not find an overflowed counter. This may appear as an unknown NMI resulting in either a panic or a series of messages, depending on how the kernel is configured. To eliminate this race condition, the PMC value must be checked after disabling the counter. Add an AMD function, amd_pmu_disable_all(), that will wait for the NMI handler to reset any active and overflowed counter after calling x86_pmu_disable_all(). Signed-off-by: Tom Lendacky Signed-off-by: Peter Zijlstra (Intel) Cc: # 4.14.x- Cc: Alexander Shishkin Cc: Arnaldo Carvalho de Melo Cc: Arnaldo Carvalho de Melo Cc: Borislav Petkov Cc: Jiri Olsa Cc: Linus Torvalds Cc: Namhyung Kim Cc: Peter Zijlstra Cc: Stephane Eranian Cc: Thomas Gleixner Cc: Vince Weaver Link: https://lkml.kernel.org/r/Message-ID: Signed-off-by: Ingo Molnar Signed-off-by: Greg Kroah-Hartman --- arch/x86/events/amd/core.c | 65 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 62 insertions(+), 3 deletions(-) --- a/arch/x86/events/amd/core.c +++ b/arch/x86/events/amd/core.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include "../perf_event.h" @@ -429,6 +430,64 @@ static void amd_pmu_cpu_dead(int cpu) } } +/* + * When a PMC counter overflows, an NMI is used to process the event and + * reset the counter. NMI latency can result in the counter being updated + * before the NMI can run, which can result in what appear to be spurious + * NMIs. This function is intended to wait for the NMI to run and reset + * the counter to avoid possible unhandled NMI messages. + */ +#define OVERFLOW_WAIT_COUNT 50 + +static void amd_pmu_wait_on_overflow(int idx) +{ + unsigned int i; + u64 counter; + + /* + * Wait for the counter to be reset if it has overflowed. This loop + * should exit very, very quickly, but just in case, don't wait + * forever... + */ + for (i = 0; i < OVERFLOW_WAIT_COUNT; i++) { + rdmsrl(x86_pmu_event_addr(idx), counter); + if (counter & (1ULL << (x86_pmu.cntval_bits - 1))) + break; + + /* Might be in IRQ context, so can't sleep */ + udelay(1); + } +} + +static void amd_pmu_disable_all(void) +{ + struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); + int idx; + + x86_pmu_disable_all(); + + /* + * This shouldn't be called from NMI context, but add a safeguard here + * to return, since if we're in NMI context we can't wait for an NMI + * to reset an overflowed counter value. + */ + if (in_nmi()) + return; + + /* + * Check each counter for overflow and wait for it to be reset by the + * NMI if it has overflowed. This relies on the fact that all active + * counters are always enabled when this function is caled and + * ARCH_PERFMON_EVENTSEL_INT is always set. + */ + for (idx = 0; idx < x86_pmu.num_counters; idx++) { + if (!test_bit(idx, cpuc->active_mask)) + continue; + + amd_pmu_wait_on_overflow(idx); + } +} + static struct event_constraint * amd_get_event_constraints(struct cpu_hw_events *cpuc, int idx, struct perf_event *event) @@ -622,7 +681,7 @@ static ssize_t amd_event_sysfs_show(char static __initconst const struct x86_pmu amd_pmu = { .name = "AMD", .handle_irq = x86_pmu_handle_irq, - .disable_all = x86_pmu_disable_all, + .disable_all = amd_pmu_disable_all, .enable_all = x86_pmu_enable_all, .enable = x86_pmu_enable_event, .disable = x86_pmu_disable_event, @@ -728,7 +787,7 @@ void amd_pmu_enable_virt(void) cpuc->perf_ctr_virt_mask = 0; /* Reload all events */ - x86_pmu_disable_all(); + amd_pmu_disable_all(); x86_pmu_enable_all(0); } EXPORT_SYMBOL_GPL(amd_pmu_enable_virt); @@ -746,7 +805,7 @@ void amd_pmu_disable_virt(void) cpuc->perf_ctr_virt_mask = AMD64_EVENTSEL_HOSTONLY; /* Reload all events */ - x86_pmu_disable_all(); + amd_pmu_disable_all(); x86_pmu_enable_all(0); } EXPORT_SYMBOL_GPL(amd_pmu_disable_virt);