Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754409Ab0F1Gnb (ORCPT ); Mon, 28 Jun 2010 02:43:31 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:48124 "EHLO rcsinet10.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753909Ab0F1Gn3 (ORCPT ); Mon, 28 Jun 2010 02:43:29 -0400 Message-ID: <4C284419.3000905@oracle.com> Date: Mon, 28 Jun 2010 14:41:29 +0800 From: Tao Ma User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Jeff Moyer CC: axboe@kernel.dk, vgoyal@redhat.com, linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org, Joel Becker , Sunil Mushran , "ocfs2-devel@oss.oracle.com" Subject: Re: [PATCH 0/3 v5][RFC] ext3/4: enhance fsync performance when using CFQ References: <1277242502-9047-1-git-send-email-jmoyer@redhat.com> <4C21D442.8080703@oracle.com> <4C22F316.3080009@oracle.com> In-Reply-To: Content-Type: multipart/mixed; boundary="------------070603080802050506080407" X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090203.4C284480.0191:SCFMA4539814,ss=1,fgs=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 40798 Lines: 1324 This is a multi-part message in MIME format. --------------070603080802050506080407 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Jeff, On 06/27/2010 09:48 PM, Jeff Moyer wrote: > Tao Ma writes: > >> Hi Jeff, >> >> On 06/23/2010 05:30 PM, Tao Ma wrote: >>> Hi Jeff, >>> >>> On 06/23/2010 05:34 AM, Jeff Moyer wrote: >>>> Hi, >>>> >>>> Running iozone with the fsync flag, or fs_mark, the performance of CFQ is >>>> far worse than that of deadline for enterprise class storage when dealing >>>> with file sizes of 8MB or less. I used the following command line as a >>>> representative test case: >>>> >>>> fs_mark -S 1 -D 10000 -N 100000 -d /mnt/test/fs_mark -s 65536 -t 1 -w >>>> 4096 -F >>>> >>>> When run using the deadline I/O scheduler, an average of the first 5 >>>> numbers >>>> will give you 448.4 files / second. CFQ will yield only 106.7. With >>>> this patch series applied (and the two patches I sent yesterday), CFQ now >>>> achieves 462.5 files / second. >>> which 2 patches? Could you paste the link or the subject? Just want to >>> make my test env like yours. ;) >>> As Joel mentioned in another mail, ocfs2 also use jbd/jbd2, so I'd like >>> to give it a try and give you some feedback about the test. >> I am sorry to say that the patch make jbd2 locked up when I tested >> fs_mark using ocfs2. >> I have attached the log from my netconsole server. After I reverted >> the patch [3/3], the box works again. > > I can't reproduce this, unfortunately. Also, when building with the > .config you sent me, the disassembly doesn't line up with the stack > trace you posted. > > I'm not sure why yielding the queue would cause a deadlock. The only > explanation I can come up with is that I/O is not being issued. I'm > assuming that no other I/O will be completed to the file system in > question. Is that right? Could you send along the output from sysrq-t? yes, I just mounted it and begin the test, so there should be no outstanding I/O. So do you need me to setup another disk for test? I have attached the sysrq output in sysrq.log. please check. btw, I also met with a NULL pointer deference in cfq_yield. I have attached the null.log also. This seems to be related to the previous deadlock and happens when I try to remount the same volume after reboot and ocfs2 try to do some recovery. Regards, Tao --------------070603080802050506080407 Content-Type: text/x-log; name="sysrq.log" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="sysrq.log" SysRq : Show State task PC stack pid father init R running task 0 1 0 0x00000000 ffff88013badb510 0000000000000082 ffffffff8271b020 0000000001e0ef88 0000000000000286 ffffffff82050937 0000000000000286 ffffffff820507e7 0000000000000286 ffff88013badb510 00000000004c4b3f ffff88013baddad8 Call Trace: [] ? __hrtimer_start_range_ns+0x12f/0x142 [] ? hrtimer_try_to_cancel+0x90/0x9b [] ? schedule_hrtimeout_range_clock+0xc4/0xeb [] ? hrtimer_wakeup+0x0/0x22 [] ? poll_schedule_timeout+0x43/0x60 [] ? do_select+0x46c/0x4ed [] ? __pollwait+0x0/0xd9 [] ? pollwake+0x0/0x54 [] ? __wake_up+0x30/0x44 [] ? journal_stop+0x20f/0x222 [jbd] [] ? number+0x121/0x223 [] ? __ext3_journal_stop+0x1f/0x3d [ext3] [] ? ext3_writeback_write_end+0x94/0xb7 [ext3] [] ? vsnprintf+0x3e4/0x421 [] ? __d_lookup+0xb7/0xf6 [] ? do_lookup+0x89/0x1e2 [] ? dput+0x27/0x116 [] ? link_path_walk+0x90a/0x930 [] ? core_sys_select+0x1e1/0x2aa [] ? user_path_at+0x52/0x79 [] ? cp_new_stat+0xea/0x101 [] ? timespec_add_safe+0x37/0x66 [] ? sys_select+0x9a/0xc3 [] ? system_call_fastpath+0x16/0x1b kthreadd S 0000000000000000 0 2 0 0x00000000 ffff88013badae20 0000000000000046 ffff88013747b550 00000000ffffffff ffffffff82003610 0000000000000010 0000000000000202 0000000000000000 0000000000000018 ffff88013badae20 ffff880128015ac8 0000000000000000 Call Trace: [] ? kernel_thread_helper+0x0/0x10 [] ? kthreadd+0x72/0xeb [] ? kernel_thread_helper+0x4/0x10 [] ? kthreadd+0x0/0xeb [] ? kernel_thread_helper+0x0/0x10 ksoftirqd/0 S 0000000000000000 0 3 2 0x00000000 ffff88013bada730 0000000000000046 ffff88013bae8e60 0000000000013640 ffff88013bada730 0000000000000000 ffffffff8272bbe0 ffffffff822fb8de ffff88013badb510 0000000000000000 ffffffff82791c00 ffff88013baddd88 Call Trace: [] ? schedule+0x616/0x6c4 [] ? run_ksoftirqd+0x0/0x116 [] ? run_ksoftirqd+0x47/0x116 [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 migration/0 S 0000000000000000 0 4 2 0x00000000 ffff88013bada040 0000000000000046 ffff88013764cea0 000000008205d592 000000033badb510 ffff88013abbfe68 ffffffff820651c6 ffff88013abbfe70 0000000000000282 ffff880001e0f000 ffffffff820651c6 ffff88013abbfed8 Call Trace: [] ? stop_machine_cpu_stop+0x0/0x8f [] ? stop_machine_cpu_stop+0x0/0x8f [] ? cpu_stopper_thread+0x119/0x13f [] ? finish_task_switch+0x33/0x80 [] ? schedule+0x616/0x6c4 [] ? cpu_stopper_thread+0x0/0x13f [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 watchdog/0 R running task 0 5 2 0x00000000 ffff88013bae9550 0000000000000046 ffffffff8271b020 0000000000000086 01ff88013bae9550 ffffffff82051a4d 0000000000000286 0000000000000286 000000000000f070 ffff88013bae9550 0000000000000000 ffff88013baddd98 Call Trace: [] ? sched_clock_local+0x9/0x6c [] ? watchdog+0x0/0x9e [] ? watchdog+0x4f/0x9e [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 events/0 R running task 0 6 2 0x00000000 ffff88013bae8e60 0000000000000046 ffff88013b032ab0 0000000082042340 ffff88013baf1e90 ffff880001e16318 0000000000000001 ffffffff8204dff0 0000000000000246 ffff88013baf1ef8 ffffffff82747420 ffff880001e16300 Call Trace: [] ? prepare_to_wait+0x35/0x6d [] ? console_callback+0x0/0xf5 [] ? worker_thread+0x93/0x1e4 [] ? autoremove_wake_function+0x0/0x2e [] ? worker_thread+0x0/0x1e4 [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 khelper S ffffffff82049cae 0 7 2 0x00000000 ffff88013bae8770 0000000000000046 ffff88013747b550 0000000000000000 ffff88013baf3e90 ffff880001e16398 0000000000000001 ffffffff8204dff0 ffff880001e16380 ffff88013baf3ef8 ffff88012f9872c0 ffff880001e16380 Call Trace: [] ? prepare_to_wait+0x35/0x6d [] ? __call_usermodehelper+0x0/0x6f [] ? worker_thread+0x93/0x1e4 [] ? autoremove_wake_function+0x0/0x2e [] ? worker_thread+0x0/0x1e4 [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 async/mgr S 0000000000000000 0 45 2 0x00000000 ffff88013bb62970 0000000000000046 ffff8801374be9b0 000000008202e0e1 ffffffff8272bbe0 000000008202d2b5 0000000000000003 0000000000013640 0000000000000282 0000000000000001 0000000000000001 ffff88013badddb8 Call Trace: [] ? async_manager_thread+0x0/0xe8 [] ? async_manager_thread+0xc2/0xe8 [] ? default_wake_function+0x0/0x9 [] ? async_manager_thread+0x0/0xe8 [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 sync_supers R running task 0 153 2 0x00000000 ffff88013b048040 0000000000000046 ffffffff8271b020 0000000000000000 ffffffff8272bbe0 ffffffff822fb8de ffff88013badb510 0000000082027cc9 000000033b093610 ffff88013b095ef8 0000000000000000 ffff88013baddda8 Call Trace: [] ? schedule+0x616/0x6c4 [] ? bdi_sync_supers+0x0/0x54 [] ? bdi_sync_supers+0x3f/0x54 [] ? bdi_sync_supers+0x0/0x54 [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 bdi-default R running task 0 155 2 0x00000000 ffff88013b048e20 0000000000000046 ffffffff8271b020 00000000ffffffff ffff88013b08ddf0 00000000fffda363 0000000000000286 ffffffff820420c2 0000000000000286 ffffffff82894dc0 ffffffff82894dc0 00000000fffdb6eb Call Trace: [] ? try_to_del_timer_sync+0xa4/0xaf [] ? schedule_timeout+0x1ba/0x1e0 [] ? process_timeout+0x0/0x5 [] ? bdi_forker_task+0x160/0x295 [] ? schedule+0x616/0x6c4 [] ? bdi_forker_task+0x0/0x295 [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 kblockd/0 S ffffffff8216160d 0 156 2 0x00000000 ffff88013b049510 0000000000000046 ffff88013747b550 000000003a1caa20 ffff88013b081e90 ffff880001e16498 0000000000000001 ffffffff8204dff0 ffff88013a1caa20 ffff88013b081ef8 ffff88013a2a5b48 ffff880001e16480 Call Trace: [] ? prepare_to_wait+0x35/0x6d [] ? cfq_kick_queue+0x0/0x3a [] ? worker_thread+0x93/0x1e4 [] ? autoremove_wake_function+0x0/0x2e [] ? worker_thread+0x0/0x1e4 [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 kacpid S ffffffff82198b60 0 158 2 0x00000000 ffff88013b03caf0 0000000000000046 ffff88013b03d1e0 000000008202cfe7 ffff88013bbcfe90 ffff880001e16518 0000000000000001 ffffffff8204dff0 ffffffff8272bbe0 ffff88013bbcfef8 ffff88013bbb6620 ffff880001e16500 Call Trace: [] ? prepare_to_wait+0x35/0x6d [] ? bind_to_cpu0+0x0/0x22 [] ? worker_thread+0x93/0x1e4 [] ? autoremove_wake_function+0x0/0x2e [] ? worker_thread+0x0/0x1e4 [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 kacpi_notify S ffffffff82198b60 0 159 2 0x00000000 ffff88013b03d1e0 0000000000000046 ffff88013b03d8d0 000000008202cfe7 ffff88013b021e90 ffff880001e16598 0000000000000001 ffffffff8204dff0 ffffffff8272bbe0 ffff88013b021ef8 ffff88013bbb6620 ffff880001e16580 Call Trace: [] ? prepare_to_wait+0x35/0x6d [] ? bind_to_cpu0+0x0/0x22 [] ? worker_thread+0x93/0x1e4 [] ? autoremove_wake_function+0x0/0x2e [] ? worker_thread+0x0/0x1e4 [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 kacpi_hotplug S ffffffff82198b60 0 160 2 0x00000000 ffff88013b03d8d0 0000000000000046 ffff88013badb510 000000008202cfe7 ffff88013b023e90 ffff880001e16618 0000000000000001 ffffffff8204dff0 ffffffff8272bbe0 ffff88013b023ef8 ffff88013bbb6620 ffff880001e16600 Call Trace: [] ? prepare_to_wait+0x35/0x6d [] ? bind_to_cpu0+0x0/0x22 [] ? worker_thread+0x93/0x1e4 [] ? autoremove_wake_function+0x0/0x2e [] ? worker_thread+0x0/0x1e4 [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 khubd S 0000000000000001 0 245 2 0x00000000 ffff88013b052770 0000000000000046 ffff88013bb63060 0000000000000004 ffff88013bbd9e70 ffffffff82752610 0000000000000001 ffffffff8204dff0 ffff88013a107230 ffffffff82752600 0000000000000000 ffff88013a107200 Call Trace: [] ? prepare_to_wait+0x35/0x6d [] ? hub_thread+0xcee/0xddc [] ? finish_task_switch+0x33/0x80 [] ? autoremove_wake_function+0x0/0x2e [] ? hub_thread+0x0/0xddc [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 kseriod S ffff88013b0ab5c0 0 248 2 0x00000000 ffff88013b088100 0000000000000046 ffff88013bb63060 00000000821f3499 ffff88013b0b1e90 ffffffff82753c80 0000000000000001 ffffffff8204dff0 Call Trace: [] ? serio_thread+0x0/0x2c5 0000000000000000 ffffffff820420c2 ffffffff82894dc0 [] ? process_timeout+0x0/0x5 [] ? kernel_thread_helper+0x4/0x10 kswapd0 S ffffffff8204dff0 [] ? prepare_to_wait+0x35/0x6d [] ? kswapd+0x0/0x591 [] ? kthread+0x79/0x81 0 305 2 0x00000000 [] ? worker_thread+0x0/0x1e4 [] ? kernel_thread_helper+0x4/0x10 0000000000000001 Call Trace: [] ? kernel_thread_helper+0x4/0x10 0000000000000000 [] ? prepare_to_wait+0x35/0x6d [] ? kthread+0x79/0x81 [] ? worker_thread+0x0/0x1e4 ffff880001e1ab98 [] ? worker_thread+0x93/0x1e4 0000000000000000 [] ? worker_thread+0x93/0x1e4 [] ? autoremove_wake_function+0x0/0x2e ffff88013bb4cfe0 Call Trace: [] ? kernel_thread_helper+0x4/0x10 ffff88013a2a5010 [] ? scsi_error_handler+0x0/0x576 [scsi_mod] [] ? kthread+0x79/0x81 0000000000000293 ffff88013a342800 [] ? scsi_error_handler+0x0/0x576 [scsi_mod] ffff88013b053550 ffff88013a342000 [] ? scsi_error_handler+0x0/0x576 [scsi_mod] [] ? kernel_thread_helper+0x0/0x10 000000003a383e18 00000000fffdb6c4 [] ? bdi_start_fn+0x0/0xc2 [] ? kernel_thread_helper+0x4/0x10 0000000000000046 0000000000000000 [] ? schedule_timeout+0x1ba/0x1e0 [] ? wake_up_bit+0x11/0x22 [] ? kernel_thread_helper+0x0/0x10 0000000000000400 ffffffff82894dc0 [] ? bdi_start_fn+0x0/0xc2 [] ? kthread+0x79/0x81 ffff88013b0887f0 0000000000000046 Call Trace: [] ? wake_up_bit+0x11/0x22 [] ? kernel_thread_helper+0x4/0x10 0 555 2 0x00000000 0000000000000286 [] ? process_timeout+0x0/0x5 [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x0/0x10 ffff88013baf9590 [] ? bdi_start_fn+0x0/0xc2 [] ? kernel_thread_helper+0x4/0x10 flush-1:6 R 0000000000000000 [] ? schedule_timeout+0x1ba/0x1e0 [] ? wake_up_bit+0x11/0x22 [] ? kernel_thread_helper+0x0/0x10 0000000000000046 00000000fffdb6cf [] ? bdi_start_fn+0x0/0xc2 [] ? bdi_start_fn+0x63/0xc2 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 flush-1:8 R running task 000000003a3cfe18 [] ? schedule_timeout+0x1ba/0x1e0 [] ? bdi_start_fn+0x63/0xc2 flush-1:9 R 0000000000000000 [] ? schedule_timeout+0x1ba/0x1e0 [] ? bdi_writeback_task+0x79/0x104 [] ? kernel_thread_helper+0x4/0x10 000000003a3d5e18 00000000fffdb6d6 [] ? schedule_timeout+0x1ba/0x1e0 [] ? bdi_start_fn+0x63/0xc2 [] ? kernel_thread_helper+0x0/0x10 0000000000000000 00000000fffdb6d6 [] ? bdi_start_fn+0x0/0xc2 [] ? kernel_thread_helper+0x4/0x10 0000000000000046 0000000000000000 Call Trace: [] ? wake_up_bit+0x11/0x22 [] ? kernel_thread_helper+0x0/0x10 00000000fffda355 [] ? bdi_start_fn+0x63/0xc2 running task [] ? bdi_writeback_task+0x79/0x104 running task [] ? schedule_timeout+0x1ba/0x1e0 [] ? kernel_thread_helper+0x0/0x10 ffff88013a3ee590 [] ? kernel_thread_helper+0x0/0x10 ffff88013b01c340 [] ? kernel_thread_helper+0x0/0x10 0000000000000000 [] ? pollwake+0x0/0x54 [] ? cap_file_permission+0x0/0x3 [] ? do_page_fault+0x324/0x346 0 638 2 0x00000000 Call Trace: [] ? schedule_timeout+0x1ba/0x1e0 [] ? process_timeout+0x0/0x5 [] ? kthread+0x79/0x81 ffff88013b027160 ffff88013abe37c0 [] ? worker_thread+0x0/0x1e4 [] ? kernel_thread_helper+0x4/0x10 0000000000000046 Call Trace: [] ? worker_thread+0x0/0x1e4 [] ? kernel_thread_helper+0x0/0x10 ffff88013747ae60 ffff880001e1b380 [] ? worker_thread+0x93/0x1e4 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 ffffffff8204dff0 [] ? prepare_to_wait+0x35/0x6d [] ? kthread+0x79/0x81 [] ? kthread+0x0/0x81 Call Trace: [] ? kjournald+0x0/0x1f4 [jbd] 0000000000000000 [] ? autoremove_wake_function+0x0/0x2e [] ? kthread+0x0/0x81 [] ? process_req+0x0/0x12d [ib_addr] [] ? kernel_thread_helper+0x4/0x10 0000000000000046 0000000000000001 Call Trace: [] ? worker_thread+0x0/0x1e4 0000000000000000 [] ? worker_thread+0x93/0x1e4 [] ? kernel_thread_helper+0x0/0x10 [] ? prepare_to_wait+0x35/0x6d [] ? kthread+0x79/0x81 0 2512 2 0x00000000 Call Trace: [] ? autoremove_wake_function+0x0/0x2e [] ? kernel_thread_helper+0x0/0x10 0000000000000286 Call Trace: [] ? hrtimer_try_to_cancel+0x90/0x9b [] ? system_call_fastpath+0x16/0x1b ffffffff82050937 Call Trace: [] ? __hrtimer_start_range_ns+0x12f/0x142 [] ? hrtimer_try_to_cancel+0x90/0x9b [] ? schedule_hrtimeout_range_clock+0xc4/0xeb [] ? hrtimer_wakeup+0x0/0x22 [] ? poll_schedule_timeout+0x43/0x60 [] ? pollwake+0x0/0x54 [] ? snprintf+0x44/0x4c [] ? proc_flush_task+0x130/0x25a [] ? system_call_after_swapgs+0x17/0x65 [] ? hrtimer_try_to_cancel+0x90/0x9b [] ? do_select+0x46c/0x4ed [] ? __pollwait+0x0/0xd9 [] ? sock_aio_write+0xf5/0x10d [] ? system_call_fastpath+0x16/0x1b ffffffff8289a010 [] ? futex_wait_queue_me+0xb6/0xd1 [] ? __generic_file_aio_write+0x338/0x3a2 [] ? vfs_statfs+0x5b/0x76 0000000000000086 [] ? hrtimer_try_to_cancel+0x90/0x9b [] ? pollwake+0x0/0x54 [] ? do_sync_read+0xab/0xeb ffff88013764cea0 Call Trace: [] ? bit_waitqueue+0x10/0xa0 [] ? elv_insert+0x10e/0x1c0 [] ? cfq_close_cooperator+0xd3/0x163 [] ? schedule+0x616/0x6c4 0 3158 1 0x00000080 [] ? kmsg_read+0x44/0x4e [] ? sys_read+0x45/0x6e ffff88013bb035d0 [] ? ip_cork_release+0x2e/0x3b [] ? do_sys_poll+0x2ac/0x37e [] ? sock_sendmsg+0xc0/0xdb [] ? sys_poll+0x49/0xa9 [] ? schedule_hrtimeout_range_clock+0x3c/0xeb [] ? poll_schedule_timeout+0x43/0x60 [] ? pollwake+0x0/0x54 [] ? unix_dgram_sendmsg+0x3eb/0x473 [] ? core_sys_select+0x1e1/0x2aa [] ? sys_sendto+0xf3/0x127 [] ? sys_select+0x9a/0xc3 [] ? system_call_fastpath+0x16/0x1b rpciod/0 S 0000000000000000 ffff88012f42be90 Call Trace: [] ? worker_thread+0x0/0x1e4 [] ? kernel_thread_helper+0x4/0x10 ffffffff8271b020 ffffffff82894dc0 [] ? process_timeout+0x0/0x5 [] ? default_wake_function+0x0/0x9 ffff88013764cea0 [] ? check_preempt_wakeup+0xed/0x14b [] ? pollwake+0x0/0x54 [] ? pollwake+0x0/0x54 [] ? system_call_fastpath+0x16/0x1b 0000000000000286 [] ? schedule_hrtimeout_range_clock+0x3c/0xeb [] ? skb_queue_tail+0x17/0x3e [] ? sock_ioctl+0x207/0x215 [] ? schedule_timeout+0x21/0x1e0 [] ? filemap_fault+0xb6/0x310 0 3402 2 0x00000080 [] ? kthread+0x79/0x81 ffff88012faafe98 [] ? rfcomm_run+0x190/0x1198 [rfcomm] [] ? kernel_thread_helper+0x4/0x10 [] ? poll_schedule_timeout+0x43/0x60 [] ? ktime_get_ts+0x68/0xb1 [] ? core_sys_select+0x1e1/0x2aa 0 3472 1 0x00000080 [] ? hrtimer_wakeup+0x0/0x22 0 3476 1 0x00000080 [] ? do_sys_poll+0x2ac/0x37e [] ? pollwake+0x0/0x54 [] ? pollwake+0x0/0x54 [] ? unix_wait_for_peer+0x9e/0xaa [] ? autoremove_wake_function+0x0/0x2e [] ? skb_queue_tail+0x17/0x3e [] ? unix_dgram_sendmsg+0x3eb/0x473 [] ? handle_mm_fault+0x3bb/0x7b3 [] ? system_call_fastpath+0x16/0x1b 00007f92897f2010 Call Trace: [] ? __dequeue_signal+0x15/0x11b [] ? fput+0x1b0/0x1e1 ffff88013baf8ea0 ffff88012fadbb88 [] ? futex_wait+0xf6/0x230 [] ? do_futex+0xa4/0xb1c [] ? handle_mm_fault+0x754/0x7b3 0 3504 1 0x00000080 Call Trace: [] ? hrtimer_wakeup+0x0/0x22 [] ? flush_tlb_page+0x4f/0x6e [] ? sys_futex+0x11b/0x139 000000003abe30c0 [] ? finish_task_switch+0x4e/0x80 [] ? __pollwait+0x0/0xd9 [] ? get_page_from_freelist+0x425/0x577 [] ? __do_fault+0x3a4/0x3e5 [] ? do_page_fault+0x324/0x346 ffff88013baf8ea0 0000000000000000 [] ? delayacct_end+0x74/0x7f [] ? poll_schedule_timeout+0x43/0x60 [] ? do_futex+0xca/0xb1c [] ? __dentry_open+0x18d/0x278 [] ? finish_task_switch+0x33/0x80 [] ? system_call_fastpath+0x16/0x1b 000280da00000010 Call Trace: [] ? __pollwait+0x0/0xd9 [] ? find_get_page+0x18/0x78 [] ? handle_mm_fault+0x3bb/0x7b3 0 3544 1 0x00000080 [] ? _raw_spin_lock_bh+0x9/0x1f [] ? inet_csk_accept+0xb6/0x205 [] ? audit_syscall_entry+0x1bd/0x1e8 00000000004c4b3f [] ? schedule_hrtimeout_range_clock+0xc4/0xeb [] ? __pollwait+0x0/0xd9 [] ? __alloc_pages_nodemask+0x104/0x568 [] ? core_sys_select+0x1e1/0x2aa [] ? system_call_fastpath+0x16/0x1b ffff88013a08a8f0 Call Trace: [] ? schedule_hrtimeout_range_clock+0x3c/0xeb [] ? ext3_get_blocks_handle+0x9b/0x8df [ext3] [] ? find_get_page+0x18/0x78 [] ? poll_schedule_timeout+0x43/0x60 [] ? __pollwait+0x0/0xd9 [] ? ext3_writeback_write_end+0x94/0xb7 [ext3] [] ? core_sys_select+0x1e1/0x2aa [] ? flush_tlb_mm+0x51/0x70 0 3615 1 0x00000080 [] ? poll_schedule_timeout+0x43/0x60 [] ? ext3_writeback_write_end+0x94/0xb7 [ext3] [] ? system_call_fastpath+0x16/0x1b ffff88012fb99ad8 [] ? schedule_hrtimeout_range_clock+0x3c/0xeb [] ? skb_queue_tail+0x17/0x3e 0 3657 1 0x00000080 Call Trace: [] ? __pollwait+0x0/0xd9 [] ? seq_printf+0x67/0x8f ffff88013a08b6d0 Call Trace: 0000000000000286 [] ? get_page_from_freelist+0x425/0x577 [] ? __pollwait+0x0/0xd9 [] ? autoremove_wake_function+0x0/0x2e [] ? find_get_page+0x18/0x78 [] ? do_page_fault+0x324/0x346 ffffffff82050937 [] ? do_nanosleep+0x73/0xa7 [] ? system_call_fastpath+0x16/0x1b ffffffff82050937 Call Trace: [] ? poll_schedule_timeout+0x43/0x60 [] ? wake_up_bit+0x11/0x22 [] ? ext3_find_entry+0x4eb/0x586 [ext3] [] ? core_sys_select+0x1e1/0x2aa atd S ffffffff822ffed3 [] ? __hrtimer_start_range_ns+0x12f/0x142 [] ? do_page_fault+0x324/0x346 [] ? do_nanosleep+0x73/0xa7 [] ? hrtimer_nanosleep+0x86/0xf1 [] ? hrtimer_wakeup+0x0/0x22 [] ? sys_nanosleep+0x56/0x6c 0000000001e0ef88 [] ? __hrtimer_start_range_ns+0x12f/0x142 [] ? do_sys_poll+0x2ac/0x37e [] ? do_futex+0xca/0xb1c [] ? handle_mm_fault+0x3bb/0x7b3 [] ? sys_poll+0x49/0xa9 0000000001e0ef88 [] ? schedule_hrtimeout_range_clock+0xc4/0xeb [] ? do_sys_poll+0x2ac/0x37e [] ? pollwake+0x0/0x54 [] ? system_call_fastpath+0x16/0x1b 0000000000000082 Call Trace: [] ? sock_aio_read+0x101/0x119 [] ? __strncpy_from_user+0x28/0x41 0 3818 1 0x00000080 0000000000000003 [] ? schedule_hrtimeout_range_clock+0x3c/0xeb [] ? poll_schedule_timeout+0x43/0x60 [] ? pollwake+0x0/0x54 [] ? pollwake+0x0/0x54 [] ? cpumask_any_but+0x27/0x33 [] ? system_call_fastpath+0x16/0x1b 0000000000000001 0000000000000003 [] ? schedule_hrtimeout_range_clock+0x3c/0xeb [] ? __pollwait+0x0/0xd9 [] ? cpumask_any_but+0x27/0x33 [] ? handle_mm_fault+0x754/0x7b3 [] ? do_readv_writev+0x182/0x197 [] ? sys_poll+0x49/0xa9 Call Trace: [] ? evdev_read+0xc7/0x200 hald-addon-ke S [] ? prepare_to_wait+0x35/0x6d [] ? system_call_fastpath+0x16/0x1b ffff88013b032ab0 ffff88013a5fa000 [] ? autoremove_wake_function+0x0/0x2e running task [] ? hrtimer_wakeup+0x0/0x22 0 3890 1 0x00000080 [] ? hrtimer_try_to_cancel+0x90/0x9b [] ? schedule_hrtimeout_range_clock+0xc4/0xeb [] ? hrtimer_wakeup+0x0/0x22 [] ? poll_schedule_timeout+0x43/0x60 [] ? do_sys_poll+0x2ac/0x37e [] ? zone_statistics+0x3a/0x5b [] ? handle_mm_fault+0x3bb/0x7b3 [] ? sys_poll+0x49/0xa9 Call Trace: [] ? futex_wake+0xf2/0x104 [] ? mntput_no_expire+0x1c/0x97 [] ? system_call_fastpath+0x16/0x1b 0000000000000286 [] ? __pollwait+0x0/0xd9 [] ? __do_fault+0x3a4/0x3e5 [] ? sys_poll+0x49/0xa9 [] ? __pollwait+0x0/0xd9 [] ? find_get_page+0x18/0x78 [] ? sys_select+0x9a/0xc3 [] ? prepare_to_wait_exclusive+0x35/0x6d [] ? unix_accept+0x54/0x101 [] ? hrtimer_wakeup+0x0/0x22 [] ? pollwake+0x0/0x54 [] ? filemap_fault+0xb6/0x310 [] ? timespec_add_safe+0x37/0x66 ffff88013a475060 ffff88012f6c7eb8 [] ? autoremove_wake_function+0x0/0x2e [] ? sys_fcntl+0x2f4/0x48d Call Trace: [] ? __put_user_4+0x1d/0x30 saslauthd S ffffffff8204dff0 [] ? prepare_to_wait+0x35/0x6d [] ? system_call_fastpath+0x16/0x1b [] ? prepare_to_wait+0x35/0x6d [] ? fcntl_setlk+0x1a6/0x25f [] ? autoremove_wake_function+0x0/0x2e [] ? __put_user_4+0x1d/0x30 [] ? sys_fcntl+0x2f4/0x48d [] ? system_call_fastpath+0x16/0x1b [] ? __hrtimer_start_range_ns+0x12f/0x142 [] ? sys_nanosleep+0x56/0x6c ffff88013a08a200 ffff88013bb58240 [] ? do_page_fault+0x324/0x346 [] ? system_call_fastpath+0x16/0x1b 0000001300000000 [] ? __do_fault+0x3a4/0x3e5 [] ? n_tty_read+0x3e9/0x6b2 [] ? tty_read+0x6f/0xbb 0 4071 1 0x00000080 ffff8801374da800 [] ? schedule_timeout+0x21/0x1e0 [] ? tty_ldisc_ref_wait+0x10/0x91 [] ? system_call_fastpath+0x16/0x1b 00000000821db81b ffff88012f758800 [] ? flush_work+0xe/0x7e [] ? tty_read+0x6f/0xbb [] ? sys_read+0x45/0x6e [] ? try_to_del_timer_sync+0xa4/0xaf [] ? tty_ldisc_ref_wait+0x10/0x91 [] ? system_call_fastpath+0x16/0x1b 0000001300000000 Call Trace: [] ? flush_work+0xe/0x7e [] ? tty_ldisc_ref_wait+0x10/0x91 0 4085 4069 0x00000080 [] ? session_of_pgrp+0xe/0x37 [] ? do_wait+0x1ba/0x20e 0 4134 2 0x00000080 [] ? ocfs2_truncate_log_worker+0x0/0x140 [ocfs2] [] ? autoremove_wake_function+0x0/0x2e [] ? kernel_thread_helper+0x0/0x10 o2quot/0 S Call Trace: [] ? worker_thread+0x0/0x1e4 jbd2/sda11-15 R running task [] ? autoremove_wake_function+0x0/0x2e ffff88013bb6f0a0 0000000000000000 [] ? sys_wait4+0x90/0xab ffffffffa00bac05 [] ? __cond_resched+0x1d/0x26 [] ? do_page_fault+0x1b3/0x346 [] ? update_curr+0x77/0xe4 [] ? scsi_done+0x0/0x41 [scsi_mod] [] ? enqueue_task_fair+0x213/0x23b [] ? jbd2_journal_start+0x90/0xbf [jbd2] [] ? filemap_write_and_wait_range+0x3b/0x4a .sysctl_sched_features : 15471 .nr_uninterruptible : 0 .cpu_load[1] : 55559 .sched_goidle : 11896 .max_vruntime : 19789.211500 .nr_spread_over : 9 .rt_throttled : 0 ---------------------------------------------------------------------------------------------------------- 19786.211500 0.102040 152126.157713 19786.211500 0.184139 152631.345425 19786.211500 0.048046 147560.772980 19786.211500 0.051120 147561.204512 19786.211500 0.047465 147561.197120 last message repeated 3 times 19786.211500 0.036472 147561.895980 iscsid 2520 19786.211500 494 110 19786.211500 0.536960 117158.277565 19786.211500 0.015188 120124.852286 python 3549 19786.211500 24 120 sendmail 3657 19786.211500 27 120 19786.211500 18.205067 106358.280463 19786.211500 1.547554 105330.629658 19786.211500 1.231707 105331.170248 jbd2/sda11-15 4137 89173.901514 2 120 --------------070603080802050506080407 Content-Type: text/x-log; name="null.log" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="null.log" BUG: unable to handle kernel NULL pointer dereference at (null) IP: [] cfq_yield+0x5f/0x135 PGD 1342ff067 PUD 1342ee067 PMD 0 Oops: 0002 [#1] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:04:02.0/irq CPU 0 Modules linked in: ocfs2 jbd2 ocfs2_nodemanager configfs ocfs2_stackglue netconsole autofs4 hidp rfcomm l2cap crc16 bluetooth rfkill sunrpc ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_mirror dm_region_hash dm_log dm_multipath dm_mod video output sbs sbshc battery acpi_memhotplug ac lp sg dcdbas sr_mod cdrom option usb_wwan usbserial serio_raw parport_pc parport rtc_cmos rtc_core rtc_lib snd_hda_codec_analog tpm_tis tpm tpm_bios button snd_hda_intel snd_hda_codec snd_seq_dummy tg3 snd_seq_oss snd_seq_midi_event snd_seq e1000 snd_seq_device libphy snd_pcm_oss snd_mixer_oss i2c_i801 i2c_core snd_pcm snd_timer snd soundcore snd_page_alloc shpchp pcspkr ata_piix libata sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd [last unloaded: microcode] Pid: 4130, comm: ocfs2_wq Not tainted 2.6.35-rc3+ #5 0MM599/OptiPlex 745 RIP: 0010:[] [] cfq_yield+0x5f/0x135 RSP: 0018:ffff880123061c60 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88012c2b5ea8 RCX: ffff88012c3a30d0 RDX: ffff8801255953d8 RSI: 0000000000000000 RDI: ffff88013a2a3800 RBP: ffff88012c3ba770 R08: ffff88012c3a30b8 R09: ffff880001e136a0 R10: ffff88012c3bb598 R11: ffff88012c3a30c8 R12: ffff88013a2a3800 R13: ffff88013a1d0a20 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff880001e00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000000 CR3: 00000001342da000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process ocfs2_wq (pid: 4130, threadinfo ffff880123060000, task ffff88013ab38240) Stack: ffff88012c3a3000 ffff88012c3a3024 ffff88013414e580 ffff8801255953d8 0000000000000003 ffffffffa030e260 ffff88012c3a3000 0000000000000003 ffff8801255953d8 0000000000000003 0000000000000282 ffffffffa030db11 Call Trace: [] ? jbd2_log_wait_commit+0x3c/0x10e [jbd2] [] ? __jbd2_log_start_commit+0x2c/0x33 [jbd2] [] ? jbd2_journal_stop+0x1f7/0x21f [jbd2] [] ? jbd2_journal_start+0x90/0xbf [jbd2] [] ? ocfs2_commit_trans+0x23/0xb1 [ocfs2] [] ? ocfs2_complete_local_alloc_recovery+0x2fa/0x3a1 [ocfs2] [] ? update_curr+0x77/0xe4 [] ? ocfs2_complete_recovery+0x0/0xab2 [ocfs2] [] ? ocfs2_complete_recovery+0x206/0xab2 [ocfs2] [] ? schedule+0x616/0x6c4 [] ? prepare_to_wait+0x35/0x6d [] ? ocfs2_complete_recovery+0x0/0xab2 [ocfs2] [] ? worker_thread+0x14a/0x1e4 [] ? autoremove_wake_function+0x0/0x2e [] ? worker_thread+0x0/0x1e4 [] ? kthread+0x79/0x81 [] ? kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0x81 [] ? kernel_thread_helper+0x0/0x10 Code: 0f 84 f2 00 last message repeated 2 times 48 8d bd a0 05 00 00 e8 ab ba 19 00 48 8b b5 00 06 00 00 4c 89 e7 e8 69 eb ff ff 49 89 c6 48 8b 85 00 06 00 00 e> 48 ff 00 fe 85 a0 05 00 00 4d 85 f6 0f 84 a6 00 last message repeated 2 times 49 8b RIP [] cfq_yield+0x5f/0x135 RSP CR2: 0000000000000000 ---[ end trace 374ddf0f57161b27 ]--- --------------070603080802050506080407-- -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/