Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932111AbWAJK77 (ORCPT ); Tue, 10 Jan 2006 05:59:59 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S932187AbWAJK77 (ORCPT ); Tue, 10 Jan 2006 05:59:59 -0500 Received: from tornado.reub.net ([202.89.145.182]:37596 "EHLO tornado.reub.net") by vger.kernel.org with ESMTP id S932111AbWAJK76 (ORCPT ); Tue, 10 Jan 2006 05:59:58 -0500 Message-ID: <43C3936C.7020502@reub.net> Date: Tue, 10 Jan 2006 23:58:52 +1300 From: Reuben Farrelly User-Agent: Thunderbird 1.6a1 (Windows/20060109) MIME-Version: 1.0 To: Andrew Morton CC: linux-kernel@vger.kernel.org Subject: Re: 2.6.15-mm2 References: <20060107052221.61d0b600.akpm@osdl.org> <43BFD8C1.9030404@reub.net> <20060107133103.530eb889.akpm@osdl.org> <43C38932.7070302@reub.net> <20060110023052.0d60a8ea.akpm@osdl.org> In-Reply-To: <20060110023052.0d60a8ea.akpm@osdl.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 43686 Lines: 924 On 10/01/2006 11:30 p.m., Andrew Morton wrote: > Reuben Farrelly wrote: >> Is there any other information I can gather to help narrow this one down? > > sysrq-t, please. Here you go: SysRq : Show State sibling task PC pid father child younger older init S 00000000 0 1 0 2 (NOTLB) c1921ec4 00000001 c0373f84 00000000 c0373f80 2d510cb0 00000033 000200d0 00000044 00000002 00000009 c1920b78 c1920a50 c1920540 c180c060 2d514cf2 00000033 00003a6e 00000001 c1921000 c0161800 00000282 c01234f1 c1921ed8 Call Trace: [] link_path_walk+0x65/0xcb [] lock_timer_base+0x15/0x2f [] __mod_timer+0x83/0x9e [] schedule_timeout+0x49/0xac [] add_wait_queue+0xf/0x30 [] process_timeout+0x0/0x5 [] do_select+0x2db/0x2f5 [] __pollwait+0x0/0x97 [] sys_select+0x1e8/0x39c [] sysenter_past_esp+0x54/0x75 migration/0 S 00000082 0 2 1 3 (L-TLB) c1927fb0 00000001 00000000 00000082 00000002 dffc3a1f 00000009 00000000 00000000 00000082 00000001 c1920158 c1920030 c036ea20 c1804060 e0360311 00000009 00000c60 00000000 c1927000 f7904f70 f7904f68 f7904f6c 00000296 Call Trace: [] complete+0x3a/0x4a [] migration_thread+0x7a/0x104 [] migration_thread+0x0/0x104 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb ksoftirqd/0 S C1804060 0 3 1 4 2 (L-TLB) c1929fb0 c1920a50 c036e880 c1804060 c1921eac e6799862 00000006 f7a47d80 00000000 00000082 0000000a c1928b78 c1928a50 c036ea20 c1804060 e694e651 00000006 0000063d 00000000 c1929000 00000000 00193641 f7ca1030 0000000e Call Trace: [] ksoftirqd+0xba/0xbc [] ksoftirqd+0x0/0xbc [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb watchdog/0 S C01157B3 0 4 1 5 3 (L-TLB) c192af88 00195ca5 c192af30 c01157b3 c1804060 2823ba88 00000034 00000000 00000002 c1928540 00000001 c1928668 c1928540 c036ea20 c1804060 2823c527 00000034 00000528 00000000 c192a000 00000000 00200282 c01234f1 c192af9c Call Trace: [] activate_task+0x9d/0xe7 [] lock_timer_base+0x15/0x2f [] __mod_timer+0x83/0x9e [] watchdog+0x0/0x62 [] schedule_timeout+0x49/0xac [] process_timeout+0x0/0x5 [] msleep_interruptible+0x31/0x3f [] watchdog+0x3e/0x62 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb migration/1 S 00000082 0 5 1 6 4 (L-TLB) c192bfb0 00000000 00000001 00000082 00000002 c0f4a8b8 00000009 f7bd4380 00000000 00000082 00000001 c1928158 c1928030 f7a86540 c180c060 c0f791ee 00000009 00000da7 00000001 c192b000 00000000 f7377f68 f743c540 00000753 Call Trace: [] migration_thread+0x7a/0x104 [] migration_thread+0x0/0x104 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb ksoftirqd/1 S C180C060 0 6 1 7 5 (L-TLB) c1931fb0 c1928030 c036e880 c180c060 c192bfb0 7bed7983 00000003 00000000 00000001 00000000 0000000a c1930b78 c1930a50 f7cd2030 c180c060 7c177ddd 00000003 00000c84 00000001 c1931000 c1804060 001a0cad 00000018 00000018 Call Trace: [] set_user_nice+0xd0/0x10a [] ksoftirqd+0xba/0xbc [] ksoftirqd+0x0/0xbc [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb watchdog/1 S C01157B3 0 7 1 8 6 (L-TLB) c1932f88 001a29ef c1932f30 c01157b3 c1804060 282397ff 00000034 00000000 00000002 c1930540 00000009 c1930668 c1930540 c1920540 c180c060 2823a171 00000034 000004b0 00000001 c1932000 00000000 00000282 c01234f1 c1932f9c Call Trace: [] activate_task+0x9d/0xe7 [] lock_timer_base+0x15/0x2f [] __mod_timer+0x83/0x9e [] watchdog+0x0/0x62 [] schedule_timeout+0x49/0xac [] process_timeout+0x0/0x5 [] msleep_interruptible+0x31/0x3f [] watchdog+0x3e/0x62 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb events/0 S 00000000 0 8 1 9 7 (L-TLB) c1935f40 00000000 c193be00 00000000 00000000 1057a540 00000034 00000003 c042e970 c042e964 0000000a c1930158 c1930030 c036ea20 c1804060 1057b3eb 00000034 0000081f 00000000 c1935000 00000000 00000001 c1935f40 c0117b55 Call Trace: [] __wake_up+0x32/0x43 [] worker_thread+0x14b/0x247 [] flush_to_ldisc+0x0/0x110 [] default_wake_function+0x0/0xc [] worker_thread+0x0/0x247 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb events/1 S C1B3C000 0 9 1 10 8 (L-TLB) c1951f40 c18ff800 c18fd440 c1b3c000 c1036760 e45128f1 00000033 00000282 c18fd440 c0373880 0000000a c1950b78 c1950a50 c1920540 c180c060 e4516b9a 00000033 00003c6e 00000001 c1951000 00000000 00000001 c1951f40 c0117b55 Call Trace: [] __wake_up+0x32/0x43 [] worker_thread+0x14b/0x247 [] cache_reap+0x0/0x17a [] default_wake_function+0x0/0xc [] worker_thread+0x0/0x247 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb khelper S 00800711 0 10 1 11 9 (L-TLB) c1937f40 00000000 00000000 00800711 c1937f2c bbef926b 00000007 00000000 00000000 00000000 0000000a c1936b78 c1936a50 f7cb7030 c1804060 bc293651 00000007 000035af 00000000 c1937000 00000000 00000001 c1920540 c0117b55 Call Trace: [] __wake_up+0x32/0x43 [] worker_thread+0x14b/0x247 [] __call_usermodehelper+0x0/0x51 [] default_wake_function+0x0/0xc [] worker_thread+0x0/0x247 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb kthread S C1B97030 0 11 1 14 162 10 (L-TLB) c1960f40 f7cb7540 00800711 c1b97030 00000002 9f7eae71 00000009 f79a3880 00000000 f7904f34 00000009 c1950668 c1950540 f743c540 c1804060 9fa720c2 00000009 0000070f 00000000 c1960000 00000000 000008ca f7cb7540 c0117b55 Call Trace: [] __wake_up+0x32/0x43 [] worker_thread+0x14b/0x247 [] keventd_create_kthread+0x0/0x46 [] default_wake_function+0x0/0xc [] worker_thread+0x0/0x247 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb kblockd/0 S F7C7DFA8 0 14 11 15 (L-TLB) c1969f40 00000019 c180c060 f7c7dfa8 f7dc872c 7125ee13 0000000a c012dc60 00000000 f7c7dfa8 0000000a c1967668 c1967540 c036ea20 c1804060 712612c5 0000000a 000014ca 00000000 c1969000 00000000 00000001 c1969f40 c0117b55 Call Trace: [] autoremove_wake_function+0x15/0x37 [] __wake_up+0x32/0x43 [] worker_thread+0x14b/0x247 [] blk_unplug_work+0x0/0x6 [] default_wake_function+0x0/0xc [] worker_thread+0x0/0x247 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb kblockd/1 S F7C7DFA8 0 15 11 16 14 (L-TLB) c196af40 00000000 00000000 f7c7dfa8 f7dc872c b24106b9 00000021 c012dc60 00000000 f7c7dfa8 0000000a c1967158 c1967030 c1920540 c180c060 b2411da4 00000021 00000c8e 00000001 c196a000 00000000 00000001 c196af40 c0117b55 Call Trace: [] autoremove_wake_function+0x15/0x37 [] __wake_up+0x32/0x43 [] worker_thread+0x14b/0x247 [] blk_unplug_work+0x0/0x6 [] default_wake_function+0x0/0xc [] worker_thread+0x0/0x247 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb kacpid S C0116ECF 0 16 11 108 15 (L-TLB) c19cbf40 00000001 c19cbf18 c0116ecf 00000000 01dac4a8 00000000 00000000 c18049c0 00000000 00000009 c1936668 c1936540 c1920540 c180c060 01db2fa0 00000000 000007e3 00000001 c19cb000 c1936540 c19cbf2c c0126d68 c1940254 Call Trace: [] find_busiest_group+0x1ec/0x351 [] do_sigaction+0x1a1/0x1f2 [] worker_thread+0x0/0x247 [] worker_thread+0x14b/0x247 [] schedule+0x31f/0xd03 [] default_wake_function+0x0/0xc [] worker_thread+0x0/0x247 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb khubd S 00000004 0 108 11 160 16 (L-TLB) c1b95f84 00000001 f7d3d420 00000004 000003e8 3624fba8 00000002 c1b95f72 f7d3ea14 c0269480 0000000a c1b92668 c1b92540 c036ea20 c1804060 3642f8ca 00000002 00000a25 00000000 c1b95000 00000000 00000002 c036ea20 f7d3ea00 Call Trace: [] hub_port_status+0x15/0x7e [] hub_thread+0x0/0x112 [] hub_thread+0xe4/0x112 [] autoremove_wake_function+0x0/0x37 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb pdflush S F7C0BF34 0 160 11 161 108 (L-TLB) f7c0bf94 c0117305 00000002 f7c0bf34 c1950540 927ec3a2 00000000 c1920540 c1920540 c036e880 0000000a f7c0ab78 f7c0aa50 c1920540 c180c060 928dfdbd 00000000 000008c6 00000001 f7c0b000 00000000 00000286 c01188d7 00000286 Call Trace: [] load_balance_newidle+0x42/0xf2 [] set_cpus_allowed+0x73/0x114 [] pdflush+0x0/0x2d [] __pdflush+0x7c/0x195 [] pdflush+0x28/0x2d [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb pdflush D F7C0CC9C 0 161 11 163 160 (L-TLB) f7c0ccc0 0140ef60 00000001 f7c0cc9c f7c79fa8 70e90228 0000000a f7c0cc94 c012dc60 00000000 0000000a f7c0a668 f7c0a540 c036ea20 c1804060 70f43bb2 0000000a 000b28d7 00000000 f7c0c000 f7c0ccb8 c0117b55 00000000 00000000 Call Trace: [] autoremove_wake_function+0x15/0x37 [] __wake_up+0x32/0x43 [] md_write_start+0xc7/0x159 [] autoremove_wake_function+0x0/0x37 [] make_request+0x58/0x458 [] __alloc_pages+0x57/0x2d2 [] generic_make_request+0xb7/0x12a [] cache_init_objs+0x43/0x85 [] submit_bio+0x46/0xd0 [] bio_alloc_bioset+0xd9/0x195 [] submit_bh+0xc0/0x10d [] ll_rw_block+0x74/0xc4 [] flush_commit_list+0x1d7/0x50e [] do_journal_end+0x80a/0x971 [] journal_end_sync+0x62/0x73 [] reiserfs_sync_fs+0x4f/0x5c [] sync_supers+0xa8/0xeb [] wb_kupdate+0x3a/0x148 [] pdflush+0x0/0x2d [] __pdflush+0xb3/0x195 [] pdflush+0x28/0x2d [] wb_kupdate+0x0/0x148 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb kswapd0 S F7C0A1E3 0 162 1 368 11 (L-TLB) f7c0df90 f7c0a1da 0000000a f7c0a1e3 f7c0df9c 92eeead4 00000000 00000000 0000000a f7c0a030 00000008 f7c0a158 f7c0a030 c036ea20 c1804060 92efc785 00000000 00001a3c 00000000 f7c0d000 c0372060 c036e338 c0124213 f7c0d000 Call Trace: [] free_uid+0x11/0x48 [] daemonize+0x1cb/0x22f [] kswapd+0x117/0x12e [] ret_from_fork+0x6/0x14 [] kswapd+0x0/0x12e [] autoremove_wake_function+0x0/0x37 [] kswapd+0x0/0x12e [] kernel_thread_helper+0x5/0xb aio/0 S C0116ECF 0 163 11 164 161 (L-TLB) c1b90f40 00000000 c1b90f18 c0116ecf 00000000 92eeead4 00000000 00000001 c180c9c0 00000000 00000009 c1b8e668 c1b8e540 c036ea20 c1804060 92effd03 00000000 00000a08 00000000 c1b90000 c1b8e540 c1b90f2c c0126d68 c1ba0d54 Call Trace: [] find_busiest_group+0x1ec/0x351 [] do_sigaction+0x1a1/0x1f2 [] worker_thread+0x0/0x247 [] worker_thread+0x14b/0x247 [] schedule+0x31f/0xd03 [] default_wake_function+0x0/0xc [] worker_thread+0x0/0x247 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb aio/1 S 00000000 0 164 11 252 163 (L-TLB) c1b54f40 00000001 c180c9c0 00000000 c04123bc 92f00d5b 00000000 00000000 00000000 00000001 00000009 c1b51158 c1b51030 c1920a50 c180c060 92f0389f 00000000 0000072f 00000001 c1b54000 00000000 c1b54f2c c036ea20 c1b7a4d4 Call Trace: [] worker_thread+0x0/0x247 [] worker_thread+0x14b/0x247 [] schedule+0x31f/0xd03 [] default_wake_function+0x0/0xc [] worker_thread+0x0/0x247 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb kseriod S C0312ABC 0 252 11 277 164 (L-TLB) f7cd4f80 00000001 f7cfc868 c0312abc 22222222 53e41492 00000002 00008080 00000000 f7ceb114 0000000a f7cd2668 f7cd2540 c036ea20 c1804060 53ef5a4a 00000002 00029830 00000000 f7cd4000 c0383060 f7d3d9c0 f7d3d9c0 c18ff100 Call Trace: [] __mutex_unlock_slowpath+0x6d/0x202 [] serio_thread+0x0/0x115 [] serio_thread+0xbe/0x115 [] autoremove_wake_function+0x0/0x37 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb ata/0 S C0116ECF 0 277 11 278 252 (L-TLB) f7cd3f40 00000001 f7cd3f18 c0116ecf 00000000 b6716cff 00000000 00000000 00000000 00000000 00000009 f7cd2b78 f7cd2a50 c1920a50 c1804060 b672017c 00000000 00000c77 00000000 f7cd3000 00000000 f7cd3f2c c1950540 f7ced454 Call Trace: [] find_busiest_group+0x1ec/0x351 [] worker_thread+0x0/0x247 [] worker_thread+0x14b/0x247 [] schedule+0x31f/0xd03 [] default_wake_function+0x0/0xc [] worker_thread+0x0/0x247 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb ata/1 S C0116ECF 0 278 11 280 277 (L-TLB) c1b96f40 00000000 c1b96f18 c0116ecf 00000000 b6702aea 00000000 00000001 c180c9c0 00000000 0000000a c1b92158 c1b92030 c1920540 c180c060 b6723300 00000000 000008fd 00000001 c1b96000 c1b92030 c1b96f2c c0126d68 c1bc1354 Call Trace: [] find_busiest_group+0x1ec/0x351 [] do_sigaction+0x1a1/0x1f2 [] worker_thread+0x0/0x247 [] worker_thread+0x14b/0x247 [] schedule+0x31f/0xd03 [] default_wake_function+0x0/0xc [] worker_thread+0x0/0x247 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb scsi_eh_0 S C031118F 0 280 11 281 278 (L-TLB) c1b93fb8 c1804060 c03cdfe0 c031118f c1b93fc8 56a412ec 00000001 00000082 00000002 56734bc4 00000009 c1b92b78 c1b92a50 c036ea20 c1804060 56a4b87d 00000001 000005eb 00000000 c1b93000 00000001 00000bb2 00000001 c1b93000 Call Trace: [] schedule+0x31f/0xd03 [] scsi_error_handler+0x0/0x9c [] scsi_error_handler+0x43/0x9c [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb scsi_eh_1 S C031118F 0 281 11 282 280 (L-TLB) f7ccefb8 c1804060 c03cdfe0 c031118f f7ccefc8 572a2f76 00000001 00000082 00000002 572a2f76 0000000a f7ccd158 f7ccd030 c036ea20 c1804060 575584e4 00000001 00000486 00000000 f7cce000 00000001 00000c04 00000000 f7cce000 Call Trace: [] schedule+0x31f/0xd03 [] scsi_error_handler+0x0/0x9c [] scsi_error_handler+0x43/0x9c [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb scsi_eh_2 S C031118F 0 282 11 283 281 (L-TLB) f7cd0fb8 c1804060 c03cdfe0 c031118f f7cd0fc8 57e0ef48 00000001 00000082 00000002 57e0ef48 0000000a f7ccdb78 f7ccda50 c036ea20 c1804060 58062a15 00000001 0000049a 00000000 f7cd0000 00000001 00000b06 00000000 f7cd0000 Call Trace: [] schedule+0x31f/0xd03 [] scsi_error_handler+0x0/0x9c [] scsi_error_handler+0x43/0x9c [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb scsi_eh_3 S C031118F 0 283 11 374 282 (L-TLB) f7cccfb8 c1804060 c03cdfe0 c031118f f7cccfc8 58b6b7bf 00000001 00000082 00000002 58b6b7bf 00000009 f7ccd668 f7ccd540 c036ea20 c1804060 58b6e355 00000001 00000532 00000000 f7ccc000 00000001 00000ade 00000000 f7ccc000 Call Trace: [] schedule+0x31f/0xd03 [] scsi_error_handler+0x0/0x9c [] scsi_error_handler+0x43/0x9c [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb kirqd S 00000000 0 368 1 445 162 (L-TLB) f7cb8f98 00000000 00000002 00000000 c1804060 16c46026 00000034 00000000 00000000 c19cc1b8 0000000a f7cc1b78 f7cc1a50 c1920540 c180c060 16c47299 00000034 00000cd4 00000001 f7cb8000 c1804060 00000282 c01234f1 f7cb8fac Call Trace: [] lock_timer_base+0x15/0x2f [] __mod_timer+0x83/0x9e [] schedule_timeout+0x49/0xac [] process_timeout+0x0/0x5 [] balanced_irq+0x7f/0xb5 [] balanced_irq+0x0/0xb5 [] kernel_thread_helper+0x5/0xb md5_raid1 S 00000008 0 374 11 378 283 (L-TLB) f7cbaf48 f7cba000 c0297e08 00000008 00000001 2fef2d6a 00000033 f7cdc02c c01da41a f7d83600 0000000a f7cc1668 f7cc1540 c1920540 c180c060 2ff29043 00000033 0133f6d5 00000001 f7cba000 00000000 00000282 c01234f1 f7cbaf5c Call Trace: [] md_check_recovery+0x18/0x435 [] generic_unplug_device+0x15/0x21 [] lock_timer_base+0x15/0x2f [] __mod_timer+0x83/0x9e [] schedule_timeout+0x49/0xac [] process_timeout+0x0/0x5 [] md_thread+0x118/0x15a [] autoremove_wake_function+0x0/0x37 [] md_thread+0x0/0x15a [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb md4_raid1 S 00000000 0 378 11 382 374 (L-TLB) f7c7ef48 f7c7e000 c0297e08 00000000 00000000 4d06524d 00000034 00000000 00000002 f7d83680 0000000a c1bc2b78 c1bc2a50 c1920540 c180c060 4d3dface 00000034 00748f57 00000001 f7c7e000 00000000 00000282 c01234f1 f7c7ef5c Call Trace: [] md_check_recovery+0x18/0x435 [] lock_timer_base+0x15/0x2f [] __mod_timer+0x83/0x9e [] schedule_timeout+0x49/0xac [] process_timeout+0x0/0x5 [] md_thread+0x118/0x15a [] autoremove_wake_function+0x0/0x37 [] md_thread+0x0/0x15a [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb md3_raid1 S 00000008 0 382 11 386 378 (L-TLB) f7ca6f48 f7ca6000 c0297e08 00000008 00000001 452bef0e 00000034 00000000 00000002 f7d83700 0000000a c1b97668 c1b97540 c1920540 c180c060 45446c75 00000034 0018781a 00000001 f7ca6000 00000000 00000282 c01234f1 f7ca6f5c Call Trace: [] md_check_recovery+0x18/0x435 [] lock_timer_base+0x15/0x2f [] __mod_timer+0x83/0x9e [] schedule_timeout+0x49/0xac [] process_timeout+0x0/0x5 [] md_thread+0x118/0x15a [] autoremove_wake_function+0x0/0x37 [] md_thread+0x0/0x15a [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb md2_raid1 D 00011210 0 386 11 390 382 (L-TLB) f7c7de84 00000010 c18ec340 00011210 00000001 7826cfa1 00000014 c1000000 00011210 00000000 0000000a c1b8e158 c1b8e030 c036ea20 c1804060 782777ad 00000014 00006310 00000000 f7c7d000 c197de8c f76b1b00 c01591d3 f7cdc744 Call Trace: [] bio_clone+0x9c/0xae [] md_super_wait+0xdf/0xf2 [] autoremove_wake_function+0x0/0x37 [] bitmap_unplug+0x1c7/0x1ce [] blk_remove_plug+0x26/0x5d [] raid1d+0x7f/0x594 [] lock_timer_base+0x15/0x2f [] del_timer_sync+0xa/0x10 [] schedule_timeout+0x50/0xac [] md_thread+0x41/0x15a [] autoremove_wake_function+0x0/0x37 [] md_thread+0x0/0x15a [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb md1_raid1 S C1804060 0 390 11 393 386 (L-TLB) c1b81f48 c1b81000 c0297e08 c1804060 0000000a 2e81c970 00000033 00000000 00000002 f7d83800 0000000a f7ca1b78 f7ca1a50 c036ea20 c1804060 2ead36a6 00000033 011f143c 00000000 c1b81000 00000000 00200282 c01234f1 c1b81f5c Call Trace: [] md_check_recovery+0x18/0x435 [] lock_timer_base+0x15/0x2f [] __mod_timer+0x83/0x9e [] schedule_timeout+0x49/0xac [] process_timeout+0x0/0x5 [] md_thread+0x118/0x15a [] autoremove_wake_function+0x0/0x37 [] md_thread+0x0/0x15a [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb md0_raid1 D 02E9B0CB 0 393 11 394 390 (L-TLB) f7c79e2c c04123bc 00000000 02e9b0cb 00000c80 70e8e00b 0000000a 00000246 f76666c0 f7d07660 0000000a c1b51668 c1b51540 c1920540 c180c060 70f4d67b 0000000a 00009eaf 00000001 f7c79000 f7cdc744 c1000000 00011210 00000000 Call Trace: [] md_super_wait+0xdf/0xf2 [] autoremove_wake_function+0x0/0x37 [] write_sb_page+0x42/0x71 [] write_page+0xc2/0x134 [] md_update_sb+0x7f/0x15b [] md_check_recovery+0x17f/0x435 [] raid1d+0x21/0x594 [] lock_timer_base+0x15/0x2f [] del_timer_sync+0xa/0x10 [] schedule_timeout+0x50/0xac [] md_thread+0x41/0x15a [] autoremove_wake_function+0x0/0x37 [] md_thread+0x0/0x15a [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb reiserfs/0 D F7DBBC98 0 394 11 395 393 (L-TLB) f7c74e9c f7bcce2c 00000000 f7dbbc98 00000000 7718f28d 0000000a ffffffff 00000000 00000000 0000000a f7c78668 f7c78540 c036ea20 c1804060 77190c89 0000000a 00000b06 00000000 f7c74000 c16f1540 00000001 00000003 f73849a4 Call Trace: [] __down+0xb7/0x11f [] schedule+0x31f/0xd03 [] default_wake_function+0x0/0xc [] __down_failed+0x7/0xc [] .text.lock.journal+0x8/0xfe [] flush_async_commits+0x73/0x75 [] worker_thread+0x1bd/0x247 [] flush_async_commits+0x0/0x75 [] default_wake_function+0x0/0xc [] worker_thread+0x0/0x247 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb reiserfs/1 D 01406F60 0 395 11 394 (L-TLB) c1b85db4 00000000 ffffffff 01406f60 00000082 7826ac88 00000014 f7dc872c 00000000 c1b85d90 0000000a f7c78b78 f7c78a50 c1920540 c180c060 78271282 00000014 00005b76 00000001 c1b85000 00000000 00000001 c1b85db4 c0117b55 Call Trace: [] __wake_up+0x32/0x43 [] io_schedule+0x26/0x30 [] sync_buffer+0x30/0x33 [] __wait_on_bit+0x42/0x5e [] sync_buffer+0x0/0x33 [] out_of_line_wait_on_bit+0x75/0x7d [] sync_buffer+0x0/0x33 [] bio_alloc_bioset+0xd9/0x195 [] wake_bit_function+0x0/0x3c [] __wait_on_buffer+0x24/0x29 [] write_ordered_buffers+0x1cf/0x20e [] write_ordered_chunk+0x0/0x52 [] load_balance_newidle+0x42/0xf2 [] do_writepages+0x23/0x37 [] schedule+0x31f/0xd03 [] flush_commit_list+0x472/0x50e [] flush_async_commits+0x73/0x75 [] worker_thread+0x1bd/0x247 [] flush_async_commits+0x0/0x75 [] default_wake_function+0x0/0xc [] worker_thread+0x0/0x247 [] kthread+0x93/0x97 [] kthread+0x0/0x97 [] kernel_thread_helper+0x5/0xb udevd S C0373F84 0 445 1 1297 368 (NOTLB) f7962ec4 3e8ac045 00000001 c0373f84 00000000 a499fb27 00000009 c013cd7f 000200d0 00000044 00000007 f7cb7668 f7cb7540 c036ea20 c1804060 a4baa159 00000009 00013a40 00000000 f7962000 c18e7500 c016cd2d f795d005 c0373f80 Call Trace: [] get_page_from_freelist+0x6d/0xae [] mntput_no_expire+0x13/0x5a [] schedule_timeout+0x6e/0xac [] add_wait_queue+0xf/0x30 [] inotify_poll+0x29/0x55 [] do_select+0x2db/0x2f5 [] __pollwait+0x0/0x97 [] sys_select+0x1e8/0x39c [] default_wake_function+0x0/0xc [] sysenter_past_esp+0x54/0x75 kjournald S 00000000 0 1297 1 1357 445 (L-TLB) f7ca8f74 c011bea4 f7ca8f80 00000000 00000000 fe56316f 00000003 f7a82458 00000000 00000246 00000003 c1b83668 c1b83540 c036ea20 c1804060 fe57c6c1 00000003 00002c60 00000000 f7ca8000 00000000 00000001 f7ca8f74 c0117b55 Call Trace: [] vprintk+0x275/0x2a1 [] __wake_up+0x32/0x43 [] kjournald+0x228/0x241 [] ret_from_fork+0x6/0x14 [] commit_timeout+0x0/0x5 [] autoremove_wake_function+0x0/0x37 [] kjournald+0x0/0x241 [] kernel_thread_helper+0x5/0xb rc S 00000003 0 1357 1 1794 1662 1297 (NOTLB) f7b5ef28 00000000 00000000 00000003 00000000 c092ae32 00000009 c0373fa8 f7c7ca50 000200d2 00000006 f7c7cb78 f7c7ca50 c036ea20 c1804060 c0c99f7b 00000009 00014d64 00000000 f7b5e000 0000000e c014003d c17d9a20 c17cb940 Call Trace: [] __pagevec_lru_add_active+0xa8/0xb3 [] do_wp_page+0x251/0x2a3 [] do_wait+0x332/0x3d0 [] schedule+0x31f/0xd03 [] default_wake_function+0x0/0xc [] sys_wait4+0x31/0x35 [] sys_waitpid+0x27/0x2b [] sysenter_past_esp+0x54/0x75 syslogd D F7CDC02C 0 1662 1 1665 1357 (NOTLB) f71eaeb4 c01d848f 00000010 f7cdc02c f7c7dfa8 7162b6e9 0000000a f7bd4380 00000001 00000000 00000009 c1b3db78 c1b3da50 c1b8e030 c180c060 718720da 0000000a 0000309b 00000001 f71ea000 00000000 c0117b55 f743c540 00000000 Call Trace: [] elv_set_request+0x1e/0x33 [] __wake_up+0x32/0x43 [] io_schedule+0x26/0x30 [] sync_page+0x31/0x3c [] __wait_on_bit+0x42/0x5e [] sync_page+0x0/0x3c [] wait_on_page_bit+0x75/0x7d [] find_get_pages_tag+0x31/0x6f [] wake_bit_function+0x0/0x3c [] wait_on_page_writeback_range+0x9e/0x104 [] scsi_issue_flush_fn+0x37/0x39 [] raid1_issue_flush+0x61/0xd0 [] raid1_issue_flush+0x0/0xd0 [] __mutex_unlock_slowpath+0x6d/0x202 [] do_fsync+0x78/0xbb [] sysenter_past_esp+0x54/0x75 klogd S C013CE17 0 1665 1 1676 1662 (NOTLB) f797fd90 000000d0 00000000 c013ce17 00000044 6f018104 0000000a f7bd4380 00000000 000200d0 0000000a f7cbd668 f7cbd540 c1b3da50 c180c060 6f372d3f 0000000a 00006b70 00000001 f797f000 00000000 f6cbd000 f743c540 000004b0 Call Trace: [] __alloc_pages+0x57/0x2d2 [