Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756235AbbGPVMN (ORCPT ); Thu, 16 Jul 2015 17:12:13 -0400 Received: from mail-ob0-f193.google.com ([209.85.214.193]:36571 "EHLO mail-ob0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755632AbbGPVML (ORCPT ); Thu, 16 Jul 2015 17:12:11 -0400 MIME-Version: 1.0 Reply-To: eranian@gmail.com In-Reply-To: References: <20150703131336.GI19282@twins.programming.kicks-ass.net> <20150703190420.GS3644@twins.programming.kicks-ass.net>

<20150715123546.GK2859@worktop.programming.kicks-ass.net> <20150716071536.GY19282@twins.programming.kicks-ass.net> Date: Thu, 16 Jul 2015 23:12:10 +0200 Message-ID: Subject: Re: perf: fuzzer triggered warning in intel_pmu_drain_pebs_nhm() From: Stephane Eranian To: Stephane Eranian Cc: Vince Weaver , LKML , Ingo Molnar , Arnaldo Carvalho de Melo , kan.liang@intel.com Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 5107 Lines: 115 hi, The assertion I am seeing on HSW now is: [ 114.652263] WARNING: CPU: 6 PID: 3538 at arch/x86/kernel/cpu/perf_event.c:1209 x86_pmu_start+0xaa/0x100() [ 114.652264] Modules linked in: snd_hda_codec_hdmi i915 bnep rfcomm bluetooth drm_kms_helper snd_hda_codec_realtek snd_hda_codec_generic drm snd_hda_intel snd_hda_codec snd_hda_core intel_rapl iosf_mbi snd_hwdep x86_pkg_temp_thermal snd_pcm intel_powerclamp snd_seq_midi snd_seq_midi_event coretemp snd_rawmidi snd_seq kvm_intel kvm snd_seq_device snd_timer snd crct10dif_pclmul crc32_pclmul soundcore mei_me mei ghash_clmulni_intel aesni_intel mxm_wmi aes_x86_64 lrw i2c_algo_bit gf128mul lpc_ich glue_helper shpchp serio_raw ablk_helper cryptd tpm_infineon soc_button_array wmi video mac_hid intel_smartconnect nls_iso8859_1 parport_pc ppdev lp parport uas usb_storage psmouse r8169 ahci libahci mii [ 114.652287] CPU: 6 PID: 3538 Comm: perf_fuzzer Not tainted 4.2.0-rc2+ #3 [ 114.652287] Hardware name: MSI MS-7816/Z87-G43 (MS-7816), BIOS V1.0 04/02/2013 [ 114.652288] ffffffff81a8e078 ffff880232e33da8 ffffffff8178881a 0000000000000007 [ 114.652289] 0000000000000000 ffff880232e33de8 ffffffff81073a4a ffff88021f844900 [ 114.652291] ffff88023f38bbc0 ffff8800a85d8800 0000000000000000 ffff88023f31a8f8 [ 114.652292] Call Trace: [ 114.652295] [] dump_stack+0x45/0x57 [ 114.652299] [] warn_slowpath_common+0x8a/0xc0 [ 114.652300] [] warn_slowpath_null+0x1a/0x20 [ 114.652302] [] x86_pmu_start+0xaa/0x100 [ 114.652304] [] perf_ioctl+0x3b9/0x400 [ 114.652306] [] do_vfs_ioctl+0x2ba/0x490 [ 114.652307] [] ? f_modown+0x4f/0xa0 [ 114.652308] [] ? f_setown+0x45/0x50 [ 114.652309] [] SyS_ioctl+0x79/0x90 [ 114.652310] [] tracesys_phase2+0x88/0x8d Which corresponds to: static void x86_pmu_start(struct perf_event *event, int flags) { struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); int idx = event->hw.idx; if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED))) return; ^^^^^ this one The kernel is trying to start an event which was not stopped. As for NHM, the irqstuck loop is there. After more instrumentation, it seems to hit only when sampling on fixed counter 3. On Thu, Jul 16, 2015 at 9:30 AM, Stephane Eranian wrote: > On Thu, Jul 16, 2015 at 12:15 AM, Peter Zijlstra wrote: >> On Thu, Jul 16, 2015 at 08:02:03AM +0200, Stephane Eranian wrote: >>> Been running it for a couple of hours, so far so good. I will let it >>> run all night. >> >> Thanks! >> > Well, it died on NHM in the same function despite your patch. Need to > look at the exact warning.\ > So more work is needed. But then I also saw the irq loop stuck message > before that. > > >>> > --- >>> > arch/x86/kernel/cpu/perf_event_intel_ds.c | 29 +++++++++++++---------------- >>> > 1 file changed, 13 insertions(+), 16 deletions(-) >>> > >>> > diff --git a/arch/x86/kernel/cpu/perf_event_intel_ds.c b/arch/x86/kernel/cpu/perf_event_intel_ds.c >>> > index 71fc40238843..68d0ced1d229 100644 >>> > --- a/arch/x86/kernel/cpu/perf_event_intel_ds.c >>> > +++ b/arch/x86/kernel/cpu/perf_event_intel_ds.c >>> > @@ -1142,6 +1142,7 @@ static void intel_pmu_drain_pebs_nhm(struct pt_regs *iregs) >>> > >>> > for (at = base; at < top; at += x86_pmu.pebs_record_size) { >>> > struct pebs_record_nhm *p = at; >>> > + u64 pebs_status; >>> > >>> > /* PEBS v3 has accurate status bits */ >>> > if (x86_pmu.intel_cap.pebs_format >= 3) { >>> > @@ -1152,12 +1153,14 @@ static void intel_pmu_drain_pebs_nhm(struct pt_regs *iregs) >>> > continue; >>> > } >>> > >>> > - bit = find_first_bit((unsigned long *)&p->status, >>> > + pebs_status = p->status & cpuc->pebs_enabled; >>> > + pebs_status &= (1ULL << x86_pmu.max_pebs_events) - 1; >>> > + >>> > + bit = find_first_bit((unsigned long *)&pebs_status, >>> > x86_pmu.max_pebs_events); >>> > if (bit >= x86_pmu.max_pebs_events) >>> > continue; >> >> Maybe we should WARN in this case? A PEBS entry without any PEBS bits >> set in the status field would be 'weird', right? >> >> Maybe something like: >> >> if (WARN(bit >= x86_pmu.max_pebs_events, >> "PEBS record without PEBS event! status=%Lx pebs_enabled=%Lx active_mask=%Lx", >> p->status, cpuc->pebs_enabled, cpuc->active_mask)) >> continue; >> >> If that triggers we at least get more info. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/