Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757992Ab1D1SfN (ORCPT ); Thu, 28 Apr 2011 14:35:13 -0400 Received: from 1wt.eu ([62.212.114.60]:35483 "EHLO 1wt.eu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755873Ab1D1SfL (ORCPT ); Thu, 28 Apr 2011 14:35:11 -0400 Date: Thu, 28 Apr 2011 20:34:34 +0200 From: Willy Tarreau To: Nikola Ciprich Cc: linux-kernel mlist , linux-stable mlist , =?iso-8859-1?Q?Herv=E9?= Commowick Subject: Re: [stable] 2.6.32.21 - uptime related crashes? Message-ID: <20110428183434.GG30645@1wt.eu> References: <20110428082625.GA23293@pcnci.linuxbox.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20110428082625.GA23293@pcnci.linuxbox.cz> User-Agent: Mutt/1.4.2.3i Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 5213 Lines: 100 Hello Nikola, On Thu, Apr 28, 2011 at 10:26:25AM +0200, Nikola Ciprich wrote: > Hello everybody, > > I'm trying to solve strange issue, today, my fourth machine running 2.6.32.21 just crashed. What makes the cases similar, apart fromn same kernel version is that all boxes had very similar uptimes: 214, 216, 216, and 224 days. This might just be a coincidence, but I think this might be important. Interestingly, one of our customers just had two machines who crashed yesterday after 212 days and 212+20h respectively. They were running debian's 2.6.32-bpo.5-amd64 which is based on 2.6.32.23 AIUI. The crash looks very similar to the following bug which we have updated : https://bugzilla.kernel.org/show_bug.cgi?id=16991 (bugzilla doesn't appear to respond as I'm posting this mail). The top of your ouput is missing. In our case as in the reports on the bug above, there was a divide by zero error. Did you happen to spot this one too, or do you just not know ? I observe "divide_error+0x15/0x20" in one of your reports, so it's possible that it matches the same pattern at least for one trace. Just in case, it would be nice to feed the bugzilla entry above. > Unfortunately I only have backtraces of two crashes (and those are trimmed, sorry), and they do not look as similar as I'd like, but still maybe there is something in common: > > [] pollwake+0x57/0x60 > [] ? default_wake_function+0x0/0x10 > [] __wake_up_common+0x5a/0x90 > [] __wake_up+0x43/0x70 > [] process_masterspan+0x643/0x670 [dahdi] > [] coretimer_func+0x135/0x1d0 [dahdi] > [] run_timer_softirq+0x15d/0x320 > [] ? coretimer_func+0x0/0x1d0 [dahdi] > [] __do_softirq+0xcc/0x220 > [] call_softirq+0x1c/0x30 > [] do_softirq+0x4a/0x80 > [] irq_exit+0x87/0x90 > [] do_IRQ+0x77/0xf0 > [] ret_from_intr+0x0/Oxa > [] ? acpi_idle_enter_bm+0x273/0x2a1 [processor] > [] ? acpi_idle_enter_bm+0x269/0x2a1 [processor] > [] ? cpuidle_idle_call+0xa5/0x150 > [] ? cpu_idle+0x4f/0x90 > [] ? rest_init+0x75/0x80 > [] ? start_kernel+0x2ef/0x390 > [] ? x86_64_start_reservations+0x81/0xc0 > [] ? x86_64_start_kernel+0xd6/0x100 > > this box (actually two of the crashed ones) is using dahdi_dummy module to generate timing for asterisk SW pbx, so maybe it's related to it. > > > [] handle_IRQ_event+0x63/0x1c0 > [] handle_edge_irq+0xce/0x160 > [] handle_irq+0x1f/0x30 > [] do_IRQ+0x6e/0xf0 > [] ret_from_intr+0x0/Oxa > [] ? _spin_un1ock_irq+0xf/0x40 > [] ? _spin_un1ock_irq+0x9/0x40 > [] ? exit_signals+0x8a/0x130 > [] ? do_exit+0x7e/0x7d0 > [] ? oops_end+0xa7/0xb0 > [] ? die+0x56/0x90 > [] ? do_trap+0x130/0x150 > [] ? do_divide_error+0x8a/0xa0 > [] ? find_busiest_group+0x3d7/0xa00 > [] ? cpuacct_charge+0x6b/0x90 > [] ? divide_error+0x15/0x20 > [] ? find_busiest_group+0x3d7/0xa00 > [] ? find_busiest_group+0x1af/0xa00 > [] ? thread_return+0x4ce/0x7bb > [] ? do_nanosleep+0x75/0x30 > [] ? hrtimer_nanosleep+0x9e/0x120 > [] ? hrtimer_wakeup+0x0/0x30 > [] ? sys_nanosleep+0x6f/0x80 > > another two don't use it. only similarity I see here is that it seems to be IRQ handling related, but both issues don't have anything in common. > Does anybody have an idea on where should I look? Of course I should update all those boxes to (at least) latest 2.6.32.x, and I'll do it for sure, but still I'd first like to know where the problem was, and if it has been fixed, or how to fix it... > I'd be gratefull for any help... There were quite a bunch of scheduler updates recently. We may be lucky and hope for the bug to have vanished with the changes, but we may as well see the same crash in 7 months :-/ My coworker Herv? (CC'd) who worked on the issue suggests that we might have something which goes wrong past a certain uptime (eg: 212 days), which needs a special event to be triggered (I/O, process exiting, etc...). I think this makes quite some sense. Could you check your CONFIG_HZ so that we could convert those uptimes to jiffies ? Maybe this will ring a bell in someone's head :-/ Best regards, Willy -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/