Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754618Ab3JJGwf (ORCPT ); Thu, 10 Oct 2013 02:52:35 -0400 Received: from mail-pb0-f47.google.com ([209.85.160.47]:40059 "EHLO mail-pb0-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752487Ab3JJGwe convert rfc822-to-8bit (ORCPT ); Thu, 10 Oct 2013 02:52:34 -0400 MIME-Version: 1.0 In-Reply-To: <20131010024613.GA10719@localhost> References: <524E57BA.805@nod.at> <52517109.90605@gmx.de> <5251C334.3010604@gmx.de> <5251CF94.5040101@gmx.de> <525591AD.4060401@gmx.de> <5255A3E6.6020100@nod.at> <20131009214733.GB25608@quack.suse.cz> <20131010024613.GA10719@localhost> Date: Thu, 10 Oct 2013 08:52:33 +0200 X-Google-Sender-Auth: 9Gj7oma21fsi8VwHqdwrTXo_JRg Message-ID: Subject: Re: [uml-devel] BUG: soft lockup for a user mode linux image From: Geert Uytterhoeven To: Fengguang Wu Cc: Jan Kara , Richard Weinberger , =?UTF-8?Q?Toralf_F=C3=B6rster?= , UML devel , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , "akpm@linux-foundation.org" , hannes@cmpxchg.org, darrick.wong@oracle.com, Michal Hocko , Gu Zheng , Benjamin LaHaise Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3899 Lines: 86 On Thu, Oct 10, 2013 at 4:46 AM, Fengguang Wu wrote: > On Wed, Oct 09, 2013 at 11:47:33PM +0200, Jan Kara wrote: >> On Wed 09-10-13 20:43:50, Richard Weinberger wrote: >> > Am 09.10.2013 19:26, schrieb Toralf Förster: >> > > On 10/08/2013 10:07 PM, Geert Uytterhoeven wrote: >> > >> On Sun, Oct 6, 2013 at 11:01 PM, Toralf Förster wrote: >> > >>>> Hmm, now pages_dirtied is zero, according to the backtrace, but the BUG_ON() >> > >>>> asserts its strict positive?!? >> > >>>> >> > >>>> Can you please try the following instead of the BUG_ON(): >> > >>>> >> > >>>> if (pause < 0) { >> > >>>> printk("pages_dirtied = %lu\n", pages_dirtied); >> > >>>> printk("task_ratelimit = %lu\n", task_ratelimit); >> > >>>> printk("pause = %ld\n", pause); >> > >>> I tried it in different ways already - I'm completely unsuccessful in getting any printk output. >> > >>> As soon as the issue happens I do have a >> > >>> >> > >>> BUG: soft lockup - CPU#0 stuck for 22s! [trinity-child0:1521] >> > >>> >> > >>> at stderr of the UML and then no further input is accepted. With uml_mconsole I'm however able >> > >>> to run very basic commands like a crash dump, sysrq ond so on. >> > >> >> > >> You may get an idea of the magnitude of pages_dirtied by using a chain of >> > >> BUG_ON()s, like: >> > >> >> > >> BUG_ON(pages_dirtied > 2000000000); >> > >> BUG_ON(pages_dirtied > 1000000000); >> > >> BUG_ON(pages_dirtied > 100000000); >> > >> BUG_ON(pages_dirtied > 10000000); >> > >> BUG_ON(pages_dirtied > 1000000); >> > >> >> > >> Probably 1 million is already too much for normal operation? >> > >> >> > > period = HZ * pages_dirtied / task_ratelimit; >> > > BUG_ON(pages_dirtied > 2000000000); >> > > BUG_ON(pages_dirtied > 1000000000); <-------------- this is line 1467 >> > >> > Summary for mm people: >> > >> > Toralf runs trinty on UML/i386. >> > After some time pages_dirtied becomes very large. >> > More than 1000000000 pages in this case. >> Huh, this is really strange. pages_dirtied is passed into >> balance_dirty_pages() from current->nr_dirtied. So I wonder how a value >> over 10^9 can get there. > > I noticed aio_setup_ring() in the call trace and find it recently > added a SetPageDirty() call in a loop by commit 36bc08cc01 ("fs/aio: > Add support to aio ring pages migration"). So added CC to its authors. > >> After all that is over 4TB so I somewhat doubt the >> task was ever able to dirty that much during its lifetime (but correct me >> if I'm wrong here, with UML and memory backed disks it is not totally >> impossible)... I went through the logic of handling ->nr_dirtied but >> I didn't find any obvious problem there. Hum, maybe one thing - what >> 'task_ratelimit' values do you see in balance_dirty_pages? If that one was >> huge, we could possibly accumulate huge current->nr_dirtied. >> >> > Thus, period = HZ * pages_dirtied / task_ratelimit overflows >> > and period/pause becomes extremely large. period/pause are signed long, so they become negative instead of extremely large when overflowing. >> > It looks like io_schedule_timeout() get's called with a very large timeout. >> > I don't know why "if (unlikely(pause > max_pause)) {" does not help. Because pause is now negative. Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/