Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752084AbXA2WCl (ORCPT ); Mon, 29 Jan 2007 17:02:41 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752085AbXA2WCl (ORCPT ); Mon, 29 Jan 2007 17:02:41 -0500 Received: from smtp02.lnh.mail.rcn.net ([207.172.157.102]:22297 "EHLO smtp02.lnh.mail.rcn.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752084AbXA2WCi convert rfc822-to-8bit (ORCPT ); Mon, 29 Jan 2007 17:02:38 -0500 X-IronPort-AV: i="4.13,253,1167627600"; d="scan'208"; a="399654202:sNHT2339119924" From: "Matthew Kirk" To: "'Andrew Morton'" Cc: References: <001a01c74136$0028ecf0$6600a8c0@charm> <20070126031654.a41fe374.akpm@osdl.org> Subject: RE: fsync occasionally very slow Date: Mon, 29 Jan 2007 17:02:14 -0500 Message-ID: <002701c743f1$2271c850$6600a8c0@charm> MIME-Version: 1.0 Content-Type: text/plain; charset="windows-1250" Content-Transfer-Encoding: 8BIT X-Mailer: Microsoft Office Outlook 11 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 In-Reply-To: <20070126031654.a41fe374.akpm@osdl.org> Thread-Index: AcdBO4JmoSoJJ6VJQD2VrzXdKeDUzQCs9gBA X-Junkmail-Status: score=10/50, host=mr08.lnh.mail.rcn.net X-Junkmail-SD-Raw: score=unknown, refid=str=0001.0A090205.45BE6EEA.001F,ss=1,fgs=0, ip=207.172.4.11, so=2006-05-09 23:27:51, dmn=5.2.125/2006-10-10 Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 32336 Lines: 826 Regarding the long fsyncs, here's a trace... I upgraded to a more recent kernel - 2.6.18.6 - and ran it on a workstation. This particular box has In this case the elevator is CFQ. This sample came from a stall that lasted about 2.5 minutes(!) - the longest one I've seen yet. The box is a bit more memory constrained than the original system but exhibits similar behavior. It doesn't page. Also, there is no raid card - simply striped PATA drives. Thanks! Matt 2 5407 5405 (NOTLB) e2cf9eb4 00000086 00000001 00000286 c0141c54 c1467ec0 00000000 c18084e0 c04ea4e0 e06c7680 00000000 00003c4f 6cd9179d 0000001b c040e3e0 e06c7570 7fffffff 08a4b310 c04ff5d0 00000001 c02d06ba c012e965 f7fdb3c0 08a4b000 Call Trace: [] buffered_rmqueue+0x13a/0x173 [] schedule_timeout+0x13/0x8e [] get_futex_key+0x3d/0xd3 [] add_wait_queue+0x12/0x30 [] futex_wait+0x18c/0x1f2 [] __handle_mm_fault+0x185/0x28c [] default_wake_function+0x0/0x12 [] do_futex+0x35/0x95 [] sys_futex+0xfb/0x10a [] schedule_tail+0x2e/0x81 [] sysenter_past_esp+0x56/0x79 PolicyScrubbe S 2C3D55DE 0 5407 5392 5408 5406 (NOTLB) e1ea5f2c 00000086 c012e032 2c3d55de 000001de 00000003 f6c9b570 c18084e0 c04ea4e0 e06c7140 00000000 00000bbe 2c3d55de 000001de f6c9b570 e06c7030 e1ea5f68 00000000 00000001 af637014 c02d0aec e1ea5f68 09bdcead 000001dd Call Trace: [] enqueue_hrtimer+0x58/0x7f [] do_nanosleep+0x40/0x6d [] hrtimer_nanosleep+0x4c/0xe9 [] sys_rt_sigaction+0x76/0x8d [] hrtimer_wakeup+0x0/0x1c [] sys_nanosleep+0x44/0x4f [] sysenter_past_esp+0x56/0x79 PolicyScrubbe S 00000000 0 5408 5392 5407 (NOTLB) de739b50 00000086 e4d15030 00000000 00000213 00000000 e4d15030 c18084e0 c04ea4e0 e1ee3bc0 00000000 00000764 299782a2 000001c2 e4d15030 e1ee3ab0 de739b60 00099160 00001000 00000000 c02d0716 de739b60 00099160 c04fdcc0 Call Trace: [] schedule_timeout+0x6f/0x8e [] process_timeout+0x0/0x9 [] do_select+0x2fa/0x34b [] __pollwait+0x0/0x4c [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] __wake_up_common+0x2f/0x4b [] __wake_up+0x27/0x3b [] sock_def_readable+0x33/0x61 [] tcp_rcv_established+0x382/0x5da [] tcp_v4_do_rcv+0x1f/0xb5 [] tcp_v4_rcv+0x529/0x86f [] activate_task+0x55/0xa2 [] __activate_task+0x1e/0x30 [] resched_task+0x41/0x63 [] try_to_wake_up+0x2ab/0x2b6 [] __alloc_skb+0x49/0xf0 [] __wake_up_common+0x2f/0x4b [] _spin_unlock_irqrestore+0xf/0x23 [] skb_dequeue+0x3b/0x41 [] unix_stream_recvmsg+0xd1/0x350 [] unix_stream_recvmsg+0x2ca/0x350 [] copy_from_user+0x3a/0x5f [] core_sys_select+0x285/0x365 [] schedule+0x60b/0x625 [] schedule_timeout+0x13/0x8e [] get_futex_key+0x3d/0xd3 [] unqueue_me+0x84/0x8b [] futex_wait+0x1a8/0x1f2 [] find_extend_vma+0x17/0x58 [] get_futex_key+0x3d/0xd3 [] copy_from_user+0x3a/0x5f [] sys_select+0x9d/0x161 [] sys_gettimeofday+0x22/0x55 [] sysenter_past_esp+0x56/0x79 gnome-termina S 1FC984CA 0 5410 1 5411 5413 5396 (NOTLB) df785bd0 00000082 c01e93d8 1fc984ca 000001df 00000002 e4509570 c18084e0 c04ea4e0 e1ee3140 00000000 0000022f 1fca0963 000001df e4509570 e1ee3030 df785be0 0006ab15 e45b1980 00000000 c02d0716 df785be0 0006ab15 c04fd238 Call Trace: [] tty_ldisc_try+0x33/0x38 [] schedule_timeout+0x6f/0x8e [] process_timeout+0x0/0x9 [] do_poll+0x15b/0x19a [] do_sys_poll+0x175/0x21b [] __alloc_skb+0x49/0xf0 [] activate_task+0x55/0xa2 [] __activate_task+0x1e/0x30 [] _spin_lock+0xd/0x59 [] task_rq_lock+0x2c/0x52 [] _spin_unlock_irqrestore+0xf/0x23 [] try_to_wake_up+0x2ab/0x2b6 [] __pollwait+0x0/0x4c [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] do_sync_write+0xbe/0xf1 [] unix_ioctl+0xa4/0xb2 [] sock_ioctl+0x1a6/0x1c3 [] do_ioctl+0x28/0x65 [] do_gettimeofday+0x2e/0x117 [] copy_to_user+0x38/0x42 [] sys_poll+0x43/0x47 [] sysenter_past_esp+0x56/0x79 gnome-termina S BAC48363 0 5413 1 5443 5410 (NOTLB) f1491ec4 00000082 00000000 bac48363 0000001b 00000003 e6672570 c18084e0 c04ea4e0 dd45c680 00000000 00002a5d bac48363 0000001b e6672570 dd45c570 e6f3b400 b792f3e0 f1491eec e6f3b400 c016415d 00000000 dd45c570 c012bc09 Call Trace: [] pipe_wait+0x6d/0x8e [] autoremove_wake_function+0x0/0x3a [] autoremove_wake_function+0x0/0x3a [] pipe_readv+0x209/0x275 [] pipe_read+0x25/0x29 [] vfs_read+0x87/0x11d [] sys_read+0x3b/0x63 [] sysenter_past_esp+0x56/0x79 gnome-pty-hel S C019B2D0 0 5411 5410 5412 (NOTLB) df7bfd4c 00000086 00000b80 c019b2d0 f75453c4 00000000 e1ee3030 c18084e0 c04ea4e0 e1ee3680 00000000 00004eb4 b9572a54 0000001b e1ee3030 e1ee3570 7fffffff dfaf3a80 df7bfd94 7fffffff c02d06ba 00000180 00000a00 7fffffff Call Trace: [] __ext3_journal_stop+0x19/0x37 [] schedule_timeout+0x13/0x8e [] prepare_to_wait+0x12/0x4c [] unix_stream_data_wait+0xb8/0xf0 [] autoremove_wake_function+0x0/0x3a [] ext3_dirty_inode+0x5d/0x63 [] autoremove_wake_function+0x0/0x3a [] unix_stream_recvmsg+0x13c/0x350 [] do_sock_read+0x83/0x8b [] sock_aio_read+0x56/0x64 [] do_sync_read+0xbe/0xf1 [] ktime_get_ts+0x4b/0x51 [] autoremove_wake_function+0x0/0x3a [] mntput_no_expire+0x14/0x6c [] __fput+0x150/0x157 [] vfs_read+0x92/0x11d [] sys_read+0x3b/0x63 [] sysenter_past_esp+0x56/0x79 bash S 57E802AC 0 5412 5410 6741 5411 (NOTLB) e08aff30 00000086 fffb8000 57e802ac 00000159 00000002 e4502ab0 c18084e0 c04ea4e0 dd45cbc0 00000000 000050a2 57f07a0a 00000159 e4502ab0 dd45cab0 fffffe00 dd45cab0 dd45cb68 00000000 c011db3a 00000001 00000000 dd45cab0 Call Trace: [] do_wait+0x287/0x31e [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] copy_to_user+0x38/0x42 [] sys_wait4+0x28/0x2c [] sys_waitpid+0x13/0x1a [] sysenter_past_esp+0x56/0x79 ntpd S F99DA445 0 5675 1 5464 (NOTLB) ddbbfb58 00000082 c040e3e0 f99da445 000001de 00000003 f7e1f570 c18084e0 c04ea4e0 ddff3bc0 00000000 00000c3b f99da445 000001de f7e1f570 ddff3ab0 7fffffff 00000000 00000100 00000000 c02d06ba ddbbfbd0 e46749c0 00000000 Call Trace: [] schedule_timeout+0x13/0x8e [] sock_poll+0x13/0x17 [] do_select+0x2fa/0x34b [] __pollwait+0x0/0x4c [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] local_bh_enable+0x7b/0x91 [] dev_queue_xmit+0x230/0x237 [] ip_output+0x1c6/0x1fc [] ip_push_pending_frames+0x36f/0x3fa [] skb_copy_and_csum_datagram+0x8b/0x31c [] skb_copy_and_csum_datagram_iovec+0x65/0xb7 [] kfree_skbmem+0x6c/0x70 [] udp_recvmsg+0x190/0x1d9 [] sock_common_recvmsg+0x36/0x4b [] sock_recvmsg+0xda/0xf5 [] skb_checksum+0x45/0x25d [] sk_reset_timer+0x12/0x1e [] tcp_send_delayed_ack+0xb3/0xbb [] copy_from_user+0x3a/0x5f [] core_sys_select+0x285/0x365 [] convert_fxsr_to_user+0x100/0x157 [] __copy_to_user_ll+0x55/0x5f [] save_i387_fxsave+0xd7/0xe9 [] setup_sigcontext+0xd9/0x11e [] setup_frame+0xcc/0x1b7 [] handle_signal+0x9a/0x10e [] _spin_unlock_irq+0xe/0x22 [] handle_signal+0x108/0x10e [] do_signal+0x7a/0xfb [] convert_fxsr_from_user+0x18/0xdc [] sys_select+0x9d/0x161 [] restore_sigcontext+0x107/0x184 [] sys_sigreturn+0xb1/0xd7 [] sysenter_past_esp+0x56/0x79 in.telnetd S 2397C627 0 6556 4054 6568 6829 (NOTLB) c9613b58 00000086 c041c9a0 2397c627 000001df 00000002 e4502570 c18084e0 c04ea4e0 f43aa140 00000000 00000a63 2398171b 000001df e4502570 f43aa030 7fffffff 00000000 00000010 00000000 c02d06ba 00000104 c02d16a0 c01eb51f Call Trace: [] schedule_timeout+0x13/0x8e [] _spin_unlock_irqrestore+0xf/0x23 [] tty_poll+0x4a/0x54 [] do_select+0x2fa/0x34b [] __pollwait+0x0/0x4c [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] e100_xmit_prepare+0x0/0x90 [] __qdisc_run+0x34/0x1a3 [] local_bh_enable+0x7b/0x91 [] dev_queue_xmit+0x230/0x237 [] ip_output+0x1c6/0x1fc [] ip_queue_xmit+0x3be/0x40c [] ip_local_deliver+0x218/0x235 [] activate_task+0x55/0xa2 [] __activate_task+0x1e/0x30 [] try_to_wake_up+0x262/0x2b6 [] _spin_unlock_irqrestore+0xf/0x23 [] activate_task+0x55/0xa2 [] tcp_v4_send_check+0x83/0xc4 [] lock_timer_base+0x19/0x33 [] _spin_unlock_irqrestore+0xf/0x23 [] __mod_timer+0x8f/0x99 [] sk_reset_timer+0x12/0x1e [] tcp_write_xmit+0x1bc/0x212 [] tcp_write_xmit+0x1e8/0x212 [] __tcp_push_pending_frames+0x23/0x80 [] local_bh_disable+0xa/0xb [] local_bh_enable_ip+0x40/0x56 [] _spin_unlock_bh+0x15/0x17 [] tcp_sendmsg+0x954/0xa02 [] tcp_sendmsg+0x954/0xa02 [] copy_from_user+0x3a/0x5f [] core_sys_select+0x285/0x365 [] sock_aio_write+0x56/0x64 [] do_sync_write+0xbe/0xf1 [] read_chan+0x4b8/0x4c5 [] autoremove_wake_function+0x0/0x3a [] current_fs_time+0x44/0x4e [] sys_select+0x9d/0x161 [] sys_write+0x3b/0x63 [] sysenter_past_esp+0x56/0x79 login S C04EA4E0 0 6568 6556 6609 (NOTLB) c9b47f44 00000082 00000012 c04ea4e0 c10ca50c 3fa32065 00000000 c18084e0 c04ea4e0 df258140 00000000 00001291 11f54f26 0000013a c040e3e0 df258030 fffffe00 df258030 df2580e8 00000000 c011db3a 00000001 00000000 df258030 Call Trace: [] do_wait+0x287/0x31e [] default_wake_function+0x0/0x12 [] do_page_fault+0x225/0x565 [] default_wake_function+0x0/0x12 [] sys_wait4+0x28/0x2c [] sysenter_past_esp+0x56/0x79 bash R running 0 6609 6568 (NOTLB) DConsole S C0113925 0 6741 5412 6749 (NOTLB) c7619eac 00000086 f43aaab0 c0113925 e1ee3030 f43aaab0 c7619ebc c18084e0 c04ea4e0 f43aabc0 00000000 000003f8 7b69c894 0000015c c040e3e0 f43aaab0 7fffffff e3493180 c7619f0c e0147800 c02d06ba f7b6c900 f7af9c80 e45b1980 Call Trace: [] resched_task+0x41/0x63 [] schedule_timeout+0x13/0x8e [] add_wait_queue+0x12/0x30 [] read_chan+0x2af/0x4c5 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] tty_ldisc_deref+0x52/0x61 [] tty_read+0x67/0x95 [] vfs_read+0x87/0x11d [] sys_read+0x3b/0x63 [] sysenter_past_esp+0x56/0x79 DConsole S 006E106D 0 6749 5412 6741 (NOTLB) dbd11f2c 00000086 c012e032 006e106d 000001da 00000003 e6672570 c18084e0 c04ea4e0 f5cf0680 00000000 00001007 006e106d 000001da e6672570 f5cf0570 dbd11f68 00000000 00000001 00000000 c02d0aec dbd11f68 aa2b65d0 000001e5 Call Trace: [] enqueue_hrtimer+0x58/0x7f [] do_nanosleep+0x40/0x6d [] hrtimer_nanosleep+0x4c/0xe9 [] sys_stat64+0x20/0x27 [] hrtimer_wakeup+0x0/0x1c [] sys_nanosleep+0x44/0x4f [] sysenter_past_esp+0x56/0x79 CompressVersi D F7406880 0 6824 6823 (NOTLB) c9d8dea8 00000082 deed1ab0 f7406880 00000000 00000001 deee4030 c18084e0 c04ea4e0 c985e680 00000000 00000127 0517fc12 000001df deee4030 c985e570 f7406800 c9d8ded4 f7406860 00025918 c01a847b 00000000 00000000 c985e570 Call Trace: [] log_wait_commit+0xc3/0xfe [] autoremove_wake_function+0x0/0x3a [] schedule_timeout+0x77/0x8e [] autoremove_wake_function+0x0/0x3a [] process_timeout+0x0/0x9 [] journal_stop+0x199/0x1c9 [] journal_force_commit+0x20/0x26 [] ext3_force_commit+0x22/0x24 [] ext3_sync_file+0x82/0xb4 [] filemap_fdatawrite+0x16/0x1a [] do_fsync+0x41/0x65 [] __do_fsync+0x20/0x30 [] sys_fsync+0xb/0xe [] sysenter_past_esp+0x56/0x79 in.telnetd S C041C998 0 6829 4054 6830 6902 6556 (NOTLB) ce2dbb58 00000082 c041c9a0 c041c998 00000000 00000000 c2395030 c18084e0 c04ea4e0 f7e76bc0 00000000 00001a1c f158733f 00000180 c2395030 f7e76ab0 7fffffff 00000000 00000010 00000000 c02d06ba 00000104 c02d16a0 c01eb51f Call Trace: [] schedule_timeout+0x13/0x8e [] _spin_unlock_irqrestore+0xf/0x23 [] tty_poll+0x4a/0x54 [] do_select+0x2fa/0x34b [] __pollwait+0x0/0x4c [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] e100_xmit_prepare+0x0/0x90 [] __qdisc_run+0x34/0x1a3 [] local_bh_enable+0x7b/0x91 [] dev_queue_xmit+0x230/0x237 [] ip_output+0x1c6/0x1fc [] ip_queue_xmit+0x3be/0x40c [] e100_enable_irq+0x13/0x31 [] _spin_unlock_irqrestore+0xf/0x23 [] e100_poll+0x91/0xab [] net_rx_action+0x97/0x17f [] n_tty_receive_buf+0x83d/0x899 [] irq_exit+0x36/0x41 [] tcp_cwnd_restart+0x1e/0xb4 [] tcp_v4_send_check+0x83/0xc4 [] lock_timer_base+0x19/0x33 [] _spin_unlock_irqrestore+0xf/0x23 [] __mod_timer+0x8f/0x99 [] sk_reset_timer+0x12/0x1e [] tcp_write_xmit+0x1bc/0x212 [] tcp_write_xmit+0x1e8/0x212 [] __tcp_push_pending_frames+0x23/0x80 [] local_bh_disable+0xa/0xb [] local_bh_enable_ip+0x40/0x56 [] _spin_unlock_bh+0x15/0x17 [] tcp_sendmsg+0x954/0xa02 [] tcp_sendmsg+0x954/0xa02 [] opost+0x197/0x1a0 [] copy_from_user+0x3a/0x5f [] core_sys_select+0x285/0x365 [] sock_aio_write+0x56/0x64 [] __wake_up+0x27/0x3b [] do_sync_write+0xbe/0xf1 [] read_chan+0x4b8/0x4c5 [] autoremove_wake_function+0x0/0x3a [] current_fs_time+0x44/0x4e [] sys_select+0x9d/0x161 [] sys_write+0x3b/0x63 [] sysenter_past_esp+0x56/0x79 login S 00000004 0 6830 6829 6836 (NOTLB) c9eb1f44 00000082 00000004 00000004 c9eb1f54 00000000 deee4030 c18084e0 c04ea4e0 dfca3140 00000000 000001b1 650f3c84 00000170 deee4030 dfca3030 fffffe00 dfca3030 dfca30e8 00000000 c011db3a 00000001 00000000 dfca3030 Call Trace: [] do_wait+0x287/0x31e [] default_wake_function+0x0/0x12 [] do_page_fault+0x225/0x565 [] default_wake_function+0x0/0x12 [] sys_wait4+0x28/0x2c [] sysenter_past_esp+0x56/0x79 CompressVersi S 0000001D 0 6832 6831 (NOTLB) c4a91b50 00000086 f3140a24 0000001d c029c65c 00000000 f7fbb030 c18084e0 c04ea4e0 c985e140 00000000 000012b7 55fdf817 000001cc f7fbb030 c985e030 c4a91b60 003d4a14 00000200 00000000 c02d0716 c4a91b60 003d4a14 c04fdda8 Call Trace: [] tcp_rcv_established+0x5ab/0x5da [] schedule_timeout+0x6f/0x8e [] process_timeout+0x0/0x9 [] do_select+0x2fa/0x34b [] __pollwait+0x0/0x4c [] default_wake_function+0x0/0x12 [] schedule+0x60b/0x625 [] preempt_schedule+0x41/0x53 [] dev_queue_xmit+0x230/0x237 [] ip_output+0x1c6/0x1fc [] ip_queue_xmit+0x3be/0x40c [] ext3_getblk+0xdf/0x22a [] __find_get_block+0x45/0x4c [] __getblk+0x1d/0x35 [] tcp_cwnd_restart+0x1e/0xb4 [] lock_timer_base+0x19/0x33 [] _spin_unlock_irqrestore+0xf/0x23 [] __mod_timer+0x8f/0x99 [] tcp_init_tso_segs+0x2f/0x40 [] tcp_write_xmit+0x59/0x212 [] local_bh_disable+0xa/0xb [] local_bh_enable_ip+0x40/0x56 [] _spin_unlock_bh+0x15/0x17 [] tcp_recvmsg+0x65c/0x693 [] tcp_recvmsg+0x65c/0x693 [] copy_from_user+0x3a/0x5f [] core_sys_select+0x285/0x365 [] sock_aio_read+0x56/0x64 [] _atomic_dec_and_lock+0x2b/0x4c [] do_sync_read+0xbe/0xf1 [] cp_new_stat64+0xef/0xff [] autoremove_wake_function+0x0/0x3a [] copy_from_user+0x3a/0x5f [] sys_select+0x9d/0x161 [] sys_read+0x3b/0x63 [] sysenter_past_esp+0x56/0x79 bash S 07757686 0 6836 6830 6895 (NOTLB) cd503f30 00000086 00000000 07757686 00000172 00000002 e4502ab0 c18084e0 c04ea4e0 c20bb680 00000000 00003a4f 077c36df 00000172 e4502ab0 c20bb570 fffffe00 c20bb570 c20bb628 00000000 c011db3a 00000001 00000000 c20bb570 Call Trace: [] do_wait+0x287/0x31e [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] copy_to_user+0x38/0x42 [] sys_wait4+0x28/0x2c [] sys_waitpid+0x13/0x1a [] sysenter_past_esp+0x56/0x79 DConsole S 00000000 0 6895 6836 6896 (NOTLB) cf219ec4 00000086 e4509030 00000000 00000ae5 00000001 f7b0a570 c18084e0 c04ea4e0 ebd84140 00000000 000001d2 1007788c 000001df f7b0a570 ebd84030 e6f3bc00 e6f3bc00 cf219eec e6f3bc00 c016415d 00000000 ebd84030 c012bc09 Call Trace: [] pipe_wait+0x6d/0x8e [] autoremove_wake_function+0x0/0x3a [] __alloc_skb+0x49/0xf0 [] autoremove_wake_function+0x0/0x3a [] pipe_readv+0x209/0x275 [] e100_tx_clean+0xa4/0xcb [] pipe_read+0x25/0x29 [] vfs_read+0x87/0x11d [] sys_read+0x3b/0x63 [] sysenter_past_esp+0x56/0x79 DConsole S D3ECBF68 0 6896 6836 6895 (NOTLB) d3ecbf2c 00000086 c012e032 d3ecbf68 c180905c c1809050 c1809050 c18084e0 c04ea4e0 d38fabc0 00000000 000005e2 211ec3cc 000001d4 c040e3e0 d38faab0 d3ecbf68 00000000 00000001 00000000 c02d0aec d3ecbf68 c804672a 000001df Call Trace: [] enqueue_hrtimer+0x58/0x7f [] do_nanosleep+0x40/0x6d [] hrtimer_nanosleep+0x4c/0xe9 [] sys_stat64+0x20/0x27 [] hrtimer_wakeup+0x0/0x1c [] sys_nanosleep+0x44/0x4f [] sysenter_past_esp+0x56/0x79 in.telnetd S C041C998 0 6902 4054 6903 6829 (NOTLB) e5a49b58 00000082 c041c9a0 c041c998 00000000 00000001 e5a49b54 c18084e0 c04ea4e0 ebd84bc0 00000000 00000912 7f3133ce 000001de c040e3e0 ebd84ab0 7fffffff 00000000 00000010 00000000 c02d06ba 00000104 c02d16a0 c01eb51f Call Trace: [] schedule_timeout+0x13/0x8e [] _spin_unlock_irqrestore+0xf/0x23 [] tty_poll+0x4a/0x54 [] do_select+0x2fa/0x34b [] __pollwait+0x0/0x4c [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] default_wake_function+0x0/0x12 [] _spin_unlock_irqrestore+0xf/0x23 [] e100_exec_cb+0xf6/0xfe [] e100_xmit_frame+0x60/0xc8 [] e100_xmit_prepare+0x0/0x90 [] __qdisc_run+0x34/0x1a3 [] local_bh_enable+0x7b/0x91 [] dev_queue_xmit+0x230/0x237 [] ip_output+0x1c6/0x1fc [] ip_queue_xmit+0x3be/0x40c [] get_page_from_freelist+0x70/0x88 [] __mod_zone_page_state+0x10/0x58 [] mod_zone_page_state+0x15/0x1c [] kmem_getpages+0x67/0x8e [] cache_grow+0xe2/0x135 [] tcp_cwnd_restart+0x1e/0xb4 [] tcp_v4_send_check+0x83/0xc4 [] tcp_init_tso_segs+0x2f/0x40 [] tcp_write_xmit+0x59/0x212 [] __tcp_push_pending_frames+0x23/0x80 [] local_bh_disable+0xa/0xb [] local_bh_enable_ip+0x40/0x56 [] _spin_unlock_bh+0x15/0x17 [] tcp_sendmsg+0x954/0xa02 [] tcp_sendmsg+0x954/0xa02 [] copy_from_user+0x3a/0x5f [] core_sys_select+0x285/0x365 [] sock_aio_write+0x56/0x64 [] schedule+0x60b/0x625 [] do_sync_write+0xbe/0xf1 [] read_chan+0x4b8/0x4c5 [] autoremove_wake_function+0x0/0x3a [] current_fs_time+0x44/0x4e [] sys_select+0x9d/0x161 [] sys_write+0x3b/0x63 [] sysenter_past_esp+0x56/0x79 login S C04EA4E0 0 6903 6902 6908 (NOTLB) da65bf44 00000082 00000012 c04ea4e0 c115db6c 353b5065 00000000 c18084e0 c04ea4e0 f7e76680 00000000 00000656 8ba1b5c4 00000176 c040e3e0 f7e76570 fffffe00 f7e76570 f7e76628 00000000 c011db3a 00000001 00000000 f7e76570 Call Trace: [] do_wait+0x287/0x31e [