Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754314AbaJIDae (ORCPT ); Wed, 8 Oct 2014 23:30:34 -0400 Received: from mail-wi0-f179.google.com ([209.85.212.179]:59950 "EHLO mail-wi0-f179.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752719AbaJIDaY (ORCPT ); Wed, 8 Oct 2014 23:30:24 -0400 Date: Thu, 9 Oct 2014 05:30:20 +0200 From: Frederic Weisbecker To: Dave Jones , Thomas Gleixner , Peter Zijlstra , LKML Subject: Re: hrtimer deadlock caused by nohz_full Message-ID: <20141009033017.GA24907@lerouge> References: <20140925141425.GA21702@redhat.com> <20140925234657.GB14870@lerouge> <20141002182409.GA29934@redhat.com> <20141007153024.GA14119@lerouge> <20141007173730.GA12147@redhat.com> <20141009020240.GA14090@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141009020240.GA14090@redhat.com> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Oct 08, 2014 at 10:02:40PM -0400, Dave Jones wrote: > On Tue, Oct 07, 2014 at 07:42:47PM +0200, Thomas Gleixner wrote: > > On Tue, 7 Oct 2014, Dave Jones wrote: > > > > > On Tue, Oct 07, 2014 at 05:30:27PM +0200, Frederic Weisbecker wrote: > > > > > > > > > Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3" > > > > > > > > > > > > I was about to make the pull request, the branch is acked by peterz. > > > > > > Would you like to pull it? It's all merge window material. > > > > > > > > > > > > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git > > > > > > nohz/fixes-v3 > > > > > > > > > > I'm now hitting this with such regularity that it's preventing me from > > > > > tracking down other bugs. Can we get this merged soon please ? > > > > > > > > I asked Thomas to pull but he probably missed it. So I did a formal pull request > > > > to Ingo a few days later but he told me that the pull request was made too late > > > > as the merge window was too close. I told him it fixes annoying bugs but that was on IRC > > > > so this probably got lost. > > > > Aarg. Lemme get this done. > > So, time for sad news. > This still doesn't solve the problem for me. > > Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2 > CPU: 2 PID: 8193 Comm: kworker/u8:1 Not tainted 3.17.0+ #67 > Workqueue: netns cleanup_net > 0000000000000000 000000002a473fb8 ffff880244c06c00 ffffffff9f81ea56 > ffffffff9fc63e00 ffff880244c06c80 ffffffff9f819003 0000000000000010 > ffff880244c06c90 ffff880244c06c30 000000002a473fb8 0000000000000000 > Call Trace: > [] dump_stack+0x4e/0x7a > [] panic+0xd7/0x20a > [] ? restart_watchdog_hrtimer+0x50/0x50 > [] watchdog_overflow_callback+0x118/0x120 > [] __perf_event_overflow+0xac/0x350 > [] ? x86_perf_event_set_period+0xde/0x150 > [] perf_event_overflow+0x14/0x20 > [] intel_pmu_handle_irq+0x206/0x410 > [] perf_event_nmi_handler+0x2b/0x50 > [] nmi_handle+0xd2/0x3b0 > [] ? nmi_handle+0x5/0x3b0 > [] ? get_lock_stats+0x19/0x60 > [] default_do_nmi+0x72/0x1c0 > [] ? hrtimer_try_to_cancel+0x58/0x1f0 > [] do_nmi+0xb8/0xf0 > [] end_repeat_nmi+0x1e/0x2e > [] ? hrtimer_try_to_cancel+0x58/0x1f0 > [] ? get_lock_stats+0x19/0x60 > [] ? get_lock_stats+0x19/0x60 > [] ? get_lock_stats+0x19/0x60 > <> [] lock_release_holdtime.part.29+0x50/0x160 > [] lock_release+0x235/0x300 > [] _raw_spin_unlock_irqrestore+0x24/0x70 > [] hrtimer_try_to_cancel+0x58/0x1f0 > [] hrtimer_cancel+0x1a/0x30 > [] tick_nohz_restart+0x17/0x90 > [] __tick_nohz_full_check+0xc8/0xe0 > [] nohz_full_kick_work_func+0xe/0x10 > [] irq_work_run_list+0x4f/0x70 > [] irq_work_tick+0x2b/0x50 > [] update_process_times+0x5b/0x70 > [] tick_sched_handle.isra.21+0x25/0x60 > [] tick_sched_timer+0x41/0x60 > [] __run_hrtimer+0x81/0x480 > [] ? tick_sched_do_timer+0x90/0x90 > [] hrtimer_interrupt+0x107/0x260 > [] local_apic_timer_interrupt+0x34/0x60 > [] smp_apic_timer_interrupt+0x3f/0x60 > [] apic_timer_interrupt+0x6f/0x80 > [] ? cache_clean+0x66/0x2f0 [sunrpc] > [] cache_flush+0x37/0x70 [sunrpc] > [] cache_purge+0x4f/0x80 [sunrpc] > [] svcauth_unix_purge+0xca/0x1c0 [sunrpc] > [] ? svcauth_unix_purge+0x5/0x1c0 [sunrpc] > [] nfsd_export_shutdown+0x112/0x250 [nfsd] > [] ? nfsd_export_shutdown+0x5/0x250 [nfsd] > [] nfsd_exit_net+0x1a/0x20 [nfsd] > [] ops_exit_list.isra.1+0x39/0x60 > [] cleanup_net+0x100/0x1e0 > [] process_one_work+0x1e0/0x6b0 > [] ? process_one_work+0x177/0x6b0 > [] worker_thread+0x11b/0x490 > [] ? process_one_work+0x6b0/0x6b0 > [] kthread+0xf6/0x110 > [] ? kthread_create_on_node+0x250/0x250 > [] ret_from_fork+0x7c/0xb0 > [] ? kthread_create_on_node+0x250/0x250 > > Here's what I have on top of Linus' current tree.. > http://paste.fedoraproject.org/140505/20075141/raw/ > > Do I miss some other essential patch ? If you have used the diff in that URL then you've missed: x86: Tell irq work about self IPI support Without that the whole patchset has no effect. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/