Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1031341AbbD1Omp (ORCPT ); Tue, 28 Apr 2015 10:42:45 -0400 Received: from mx1.redhat.com ([209.132.183.28]:56201 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S965965AbbD1Oml (ORCPT ); Tue, 28 Apr 2015 10:42:41 -0400 Date: Tue, 28 Apr 2015 10:42:07 -0400 From: Don Zickus To: Jiri Kosina Cc: Borislav Petkov , linux-edac , Borislav Petkov , "Rafael J. Wysocki" , Len Brown , Tony Luck , Tomasz Nowicki , "Chen, Gong" , Wolfram Sang , Lv Zheng , Naoya Horiguchi , linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, Huang Ying Subject: Re: [RFC PATCH 5/5] GHES: Make NMI handler have a single reader Message-ID: <20150428144207.GB98296@redhat.com> References: <1427448178-20689-1-git-send-email-bp@alien8.de> <1427448178-20689-6-git-send-email-bp@alien8.de> <20150428143009.GA98296@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150428143009.GA98296@redhat.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3439 Lines: 84 On Tue, Apr 28, 2015 at 10:30:09AM -0400, Don Zickus wrote: > On Wed, Apr 01, 2015 at 09:45:53AM +0200, Jiri Kosina wrote: > > On Fri, 27 Mar 2015, Borislav Petkov wrote: > > > > > From: Jiri Kosina > > > > > > Since GHES sources are global, we theoretically need only a single CPU > > > reading them per NMI instead of a thundering herd of CPUs waiting on a > > > spinlock in NMI context for no reason at all. > > > > I originally wasn't 100% sure whether GHES sources are global (i.e. if it > > really doesn't matter which CPU is reading the registers), but looking at > > the code more it actually seems that this is really the right thing to do. > > > > Rationale: ghes_ioremap_pfn_nmi() always ioremaps() (exclusively) the page > > with the registers, performs apei_read() (which is ghes-source specific, > > but not CPU-specific) and unmaps the page again. > > > > There is nothing that would make this CPU-specific. Adding Huang Ying (the > > original author of the code) to confirm this. Huang? > > Hi, > > I believe the answer to this question is no, they are not global but > instead external. All external NMIs are routed to one cpu, normally cpu0. > This spinlock was made global to handle the 'someday' case of hotplugging > the bsp cpu (cpu0). > > The other external NMIs (IO_CHECK and SERR) suffer from the same spinlock > problem. I tried using an irq_workqueue to work around quirks there and > PeterZ wasn't very fond of it (though he couldn't think of a better way to > solve it). > > This patch seems interesting but you might still run into the thundering > herd problem with the global spinlock in > arch/x86/kernel/nmi.c::default_do_nmi(). That functions grabs a global > spinlock before processing the external NMI list (which GHES is a part of). Grr, I mispoke. I sent a patchset a year ago to split out internal and external NMIs to simplify the problem. So I wrote the above paragraph thinking the GHES NMI handler was wrapped with the external NMI spinlock, when in fact it isn't. However, with perf running with lots of events, it is possible to start 'swallowing' NMIs which requires passing through the spinlock I just mentioned. This might cause random delays in your measurements and is still worth modifying. Cheers, Don > > > So I am assuming this patch solves the 'thundering herd' problem by > minimizing all the useless writes the spinlock would do for each cpu that > noticed it had no work to do? > > In that case, I am in favor of this solution and would like to apply a > similar solution to arch/x86/kernel/nmi.c, to see if that helps there too. > > Cheers, > Don > > > > > > > Do that. > > > > I think this should indeed be pushed forward. It fixes horrible spinlock > > contention on systems which are under NMI storm (such as when perf is > > active) unrelated to GHES. > > > > Thanks, > > > > -- > > Jiri Kosina > > SUSE Labs > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/