Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1423356AbbD2PGB (ORCPT ); Wed, 29 Apr 2015 11:06:01 -0400 Received: from userp1040.oracle.com ([156.151.31.81]:18584 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1423179AbbD2PF7 (ORCPT ); Wed, 29 Apr 2015 11:05:59 -0400 From: Ethan Zhao To: tony.luck@intel.com, bp@alien8.de, tglx@linutronix.de Cc: mingo@redhat.com, hpa@zytor.com, x86@kernel.org, linux-edac@vger.kernel.org, linux-kernel@vger.kernel.org, ethan.kernel@gmail.com, tim.uglow@oracle.com, Ethan Zhao Subject: [PATCH] x86/mce: fix mce_restart() race with CPU hotplug operation Date: Thu, 30 Apr 2015 00:04:53 +0900 Message-Id: <1430319893-23399-1-git-send-email-ethan.zhao@oracle.com> X-Mailer: git-send-email 1.8.3.1 X-Source-IP: userv0021.oracle.com [156.151.31.71] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4847 Lines: 123 while testing CPU hotplug and MCE with following two scripts, script 1: for i in {1..30}; do while :; do ((a=$RANDOM%160)); echo 0 >> /sys/devices/system/cpu/cpu${i}/online; echo 1 >> /sys/devices/system/cpu/cpu${i}/online; done & done script 2: while :; do for i in $(ls /sys/devices/system/machinecheck/machinecheck*/check_interval); do echo 1 >> $i; done; done We got panic call trace as: ------------[ cut here ]------------ kernel BUG at kernel/timer.c:929! invalid opcode: 0000 [#1] SMP Modules linked in: fuse tun coretemp acpi_cpufreq mperf freq_table intel_powerclampsmpboot: CPU 29 is now offline kvm_intel kvm crc32c_intel ghash_clmulni_intel aesni_intel xts aes_x86_64 lrw gf128mul ablk_helper cryptd iTCO_wdt iTCO_vendor_support ses microcode pcspkr enclosure i2c_i801 i2c_core lpc_ich i7core_edac mfd_core edac_core shpchp ext3 mbcache jbd sd_mod crc_t10dif ixgbe ptp igb pps_core ahci libahci dca megaraid_sas hwmon ipv6 autofs4 CPU 101 Pid: 0, comm: swapper/101 Tainted: G W 3.8.13 #2 Oracle Corporation Sun Fire X4800 M2 / RIP: 0010:[] [] add_timer_on+0xe2/0xf0 RSP: 0000:ffff88303f843de8 EFLAGS: 00010282 RAX: 0000000000000000 RBX: ffff88303f84c920 RCX: 000000011eb1d709 RDX: ffff88303f840000 RSI: 0000000000000065 RDI: ffff88303f84c920 RBP: ffff88303f843e18 R08: 000000011eb1d03a R09: ffff88303f843d68 R10: ffff88303f843d6c R11: 0000000000000006 R12: 00000000000007d0 R13: ffff883029710000 R14: 0000000000000065 R15: 0000000000000066 FS: 0000000000000000(0000) GS:ffff88303f840000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f7e1b7b7000 CR3: 0000000001886000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/101 (pid: 0, threadinfo ffff881028f94000, task ffff881028f92500) Stack: 0000000001000c18 ffff88303f84c920 00000000000007d0 0000000000000100 ffffffff81030bb0 0000000000000066 ffff88303f843e38 ffffffff81030c44 ffff88303f88c920 ffff88303f88c920 ffff88303f843e70 ffffffff8106acbb Call Trace: [] ? mce_cpu_restart+0x40/0x40 [] mce_timer_fn+0x94/0x130 [] call_timer_fn+0x3b/0x110 [] ? mce_cpu_restart+0x40/0x40 [] run_timer_softirq+0x1cd/0x2b0 [] __do_softirq+0xd8/0x210 [] ? intel_pstate_timer_func+0x3a0/0x3a0 [] call_softirq+0x1c/0x30 [] do_softirq+0x65/0xa0 [] irq_exit+0xa5/0xb0 [] smp_apic_timer_interrupt+0x6e/0x9c [] apic_timer_interrupt+0x6d/0x80 [] ? cpuidle_wrap_enter+0x41/0x80 [] cpuidle_enter_tk+0x10/0x20 [] cpuidle_idle_call+0xb7/0x1e0 [] cpu_idle+0xe5/0x140 [] start_secondary+0x24e/0x250 Code: 90 00 4d 85 ff 74 22 49 8b 0f 0f 1f 80 00 00 00 00 49 8b 7f 08 49 83 c7 10 4c 89 e2 48 89 de ff d1 49 8b 0f 48 85 c9 75 e8 eb 97 <0f> 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 RIP [] add_timer_on+0xe2/0xf0 RSP This panic was caused by race condition between mce_restart() and CPU hotplug operation. we should protect the mce_restart() operation with get_online_cpus(); put_online_cpus(); functions pair, just as other subsystem that does iteration on cpu_online_mask. This bug will affect stable branch 4.0, 3.8, 3.19 (didn't check others). and this patch has been verified on stable 4.0 branch. Reported-by: Tim Uglow Signed-off-by: Ethan Zhao --- arch/x86/kernel/cpu/mcheck/mce.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c index 3c036cb..fcc2794 100644 --- a/arch/x86/kernel/cpu/mcheck/mce.c +++ b/arch/x86/kernel/cpu/mcheck/mce.c @@ -1338,8 +1338,10 @@ static void mce_timer_delete_all(void) { int cpu; + get_online_cpus(); for_each_online_cpu(cpu) del_timer_sync(&per_cpu(mce_timer, cpu)); + put_online_cpus(); } static void mce_do_trigger(struct work_struct *work) @@ -2085,7 +2087,9 @@ static void mce_cpu_restart(void *data) static void mce_restart(void) { mce_timer_delete_all(); + get_online_cpus(); on_each_cpu(mce_cpu_restart, NULL, 1); + put_online_cpus(); } /* Toggle features for corrected errors */ -- 1.8.3.1 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/