Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758509AbXKDMwV (ORCPT ); Sun, 4 Nov 2007 07:52:21 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1754031AbXKDMwI (ORCPT ); Sun, 4 Nov 2007 07:52:08 -0500 Received: from lucidpixels.com ([75.144.35.66]:44468 "EHLO lucidpixels.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754026AbXKDMwB (ORCPT ); Sun, 4 Nov 2007 07:52:01 -0500 Date: Sun, 4 Nov 2007 07:52:00 -0500 (EST) From: Justin Piszcz X-X-Sender: jpiszcz@p34.internal.lan To: Michael Tokarev cc: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, xfs@oss.sgi.com Subject: Re: 2.6.23.1: mdadm/raid5 hung/d-state In-Reply-To: <472DBF8C.2060508@msgid.tls.msk.ru> Message-ID: References: <472DBF8C.2060508@msgid.tls.msk.ru> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 63605 Lines: 1011 On Sun, 4 Nov 2007, Michael Tokarev wrote: > Justin Piszcz wrote: >> # ps auxww | grep D >> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND >> root 273 0.0 0.0 0 0 ? D Oct21 14:40 [pdflush] >> root 274 0.0 0.0 0 0 ? D Oct21 13:00 [pdflush] >> >> After several days/weeks, this is the second time this has happened, >> while doing regular file I/O (decompressing a file), everything on the >> device went into D-state. > > The next time you come across something like that, do a SysRq-T dump and > post that. It shows a stack trace of all processes - and in particular, > where exactly each task is stuck. > > /mjt > Yes I got it before I rebooted, ran that and then dmesg > file. Here it is: [1172609.665902] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172609.668768] ffffffff80747dc0 ffff81015c3aa918 ffff810091c899b4 ffff810091c899a8 [1172609.668871] Call Trace: [1172609.674472] [] schedule_timeout+0x5f/0xd0 [1172609.677362] [] process_timeout+0x0/0x10 [1172609.680243] [] do_select+0x468/0x560 [1172609.683105] [] __pollwait+0x0/0x130 [1172609.685969] [] default_wake_function+0x0/0x10 [1172609.688851] [] default_wake_function+0x0/0x10 [1172609.691712] [] default_wake_function+0x0/0x10 [1172609.694534] [] default_wake_function+0x0/0x10 [1172609.697324] [] skb_copy_datagram_iovec+0x1a1/0x260 [1172609.700103] [] _spin_lock_bh+0x9/0x20 [1172609.702856] [] release_sock+0x13/0xb0 [1172609.705598] [] tcp_recvmsg+0x370/0x940 [1172609.708303] [] sock_common_recvmsg+0x30/0x50 [1172609.710999] [] sock_aio_read+0x11b/0x130 [1172609.713694] [] core_sys_select+0x209/0x300 [1172609.716397] [] autoremove_wake_function+0x0/0x30 [1172609.719112] [] default_wake_function+0x0/0x10 [1172609.721824] [] current_fs_time+0x1e/0x30 [1172609.724525] [] tty_ldisc_deref+0x52/0x80 [1172609.727215] [] sys_select+0xd1/0x1c0 [1172609.729880] [] system_call+0x7e/0x83 [1172609.732517] [1172609.735115] bash S 0000000000000000 0 30959 30958 [1172609.737742] ffff810091c8be88 0000000000000086 0000000000000000 ffff8101ea172e20 [1172609.740404] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172609.743087] ffffffff80747dc0 ffff81015c3ab028 ffff810091c8be54 ffff810091c8be48 [1172609.743190] Call Trace: [1172609.748404] [] do_wait+0x599/0xc90 [1172609.751071] [] __wake_up+0x43/0x70 [1172609.753714] [] default_wake_function+0x0/0x10 [1172609.756345] [] system_call+0x7e/0x83 [1172609.758967] [1172609.761522] sr S 0000000000000000 0 30966 30959 [1172609.764123] ffff810122d7de88 0000000000000082 0000000000000000 ffff8101eab3ee20 [1172609.766769] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172609.769442] ffffffff80747dc0 ffff8101ea173028 ffff810122d7de54 ffff810122d7de48 [1172609.769545] Call Trace: [1172609.774734] [] do_wait+0x599/0xc90 [1172609.777369] [] default_wake_function+0x0/0x10 [1172609.779999] [] system_call+0x7e/0x83 [1172609.782616] [1172609.785168] screen S 0000000000000000 0 30972 30966 [1172609.787768] ffff810144597f68 0000000000000086 ffff810144597f30 00000000ffffffff [1172609.790416] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172609.793085] ffffffff80747dc0 ffff8101eab3f028 ffff810144597f34 ffff810144597f28 [1172609.793188] Call Trace: [1172609.798381] [] alarm_setitimer+0x35/0x70 [1172609.801049] [] sys_pause+0x19/0x30 [1172609.803705] [] system_call+0x7e/0x83 [1172609.806361] [1172609.808980] sshd S 0000000000000000 0 30973 7582 [1172609.811659] ffff810084003bf8 0000000000000082 0000000000000000 ffffffff80508e74 [1172609.814376] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172609.817104] ffffffff80747dc0 ffff8101ea172208 ffff810084003bc4 ffff810084003bb8 [1172609.817207] Call Trace: [1172609.822530] [] skb_queue_tail+0x24/0x60 [1172609.825292] [] schedule_timeout+0x95/0xd0 [1172609.828060] [] prepare_to_wait+0x23/0x80 [1172609.830820] [] unix_stream_recvmsg+0x386/0x550 [1172609.833587] [] autoremove_wake_function+0x0/0x30 [1172609.836344] [] link_path_walk+0x80/0xf0 [1172609.839074] [] sock_aio_read+0x11b/0x130 [1172609.841794] [] get_unused_fd_flags+0x79/0x120 [1172609.844488] [] do_sync_read+0xd9/0x120 [1172609.847161] [] autoremove_wake_function+0x0/0x30 [1172609.849848] [] __dentry_open+0x11f/0x1b0 [1172609.852541] [] do_filp_open+0x3a/0x50 [1172609.855235] [] vfs_read+0x157/0x160 [1172609.857922] [] sys_read+0x53/0x90 [1172609.860620] [] system_call+0x7e/0x83 [1172609.863343] [1172609.866063] sshd S 0000000000000000 0 30975 30973 [1172609.868838] ffff810175c219e8 0000000000000086 ffff810175c219b0 0000000000000002 [1172609.871649] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172609.874490] ffffffff80747dc0 ffff81021b27d738 ffff810175c219b4 ffff810175c219a8 [1172609.874594] Call Trace: [1172609.880153] [] schedule_timeout+0x5f/0xd0 [1172609.883020] [] process_timeout+0x0/0x10 [1172609.885890] [] do_select+0x468/0x560 [1172609.888742] [] __pollwait+0x0/0x130 [1172609.891581] [] default_wake_function+0x0/0x10 [1172609.894430] [] default_wake_function+0x0/0x10 [1172609.897258] [] default_wake_function+0x0/0x10 [1172609.900060] [] default_wake_function+0x0/0x10 [1172609.902841] [] add_partial+0x19/0x60 [1172609.905606] [] __slab_free+0x15d/0x310 [1172609.908363] [] _spin_lock_bh+0x9/0x20 [1172609.911093] [] release_sock+0x13/0xb0 [1172609.913795] [] tcp_recvmsg+0x370/0x940 [1172609.916486] [] sock_common_recvmsg+0x30/0x50 [1172609.919151] [] sock_aio_read+0x11b/0x130 [1172609.921799] [] core_sys_select+0x209/0x300 [1172609.924455] [] autoremove_wake_function+0x0/0x30 [1172609.927122] [] default_wake_function+0x0/0x10 [1172609.929786] [] current_fs_time+0x1e/0x30 [1172609.932438] [] tty_ldisc_deref+0x52/0x80 [1172609.935083] [] sys_select+0xd1/0x1c0 [1172609.937702] [] system_call+0x7e/0x83 [1172609.940292] [1172609.942843] bash S 0000000000000000 0 30976 30975 [1172609.945423] ffff8101bf371e88 0000000000000082 0000000000000000 ffff81021e322710 [1172609.948037] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172609.950671] ffffffff80747dc0 ffff8101882bf738 ffff8101bf371e54 ffff8101bf371e48 [1172609.950774] Call Trace: [1172609.955888] [] do_wait+0x599/0xc90 [1172609.958505] [] __wake_up+0x43/0x70 [1172609.961098] [] vfs_ioctl+0x220/0x2c0 [1172609.963662] [] default_wake_function+0x0/0x10 [1172609.966234] [] sys_ioctl+0x49/0x80 [1172609.968766] [] system_call+0x7e/0x83 [1172609.971279] [1172609.973759] screen S 0000000000000000 0 30991 30976 [1172609.976308] ffff8101a8329f68 0000000000000086 0000000000000000 00000000ffffffff [1172609.978892] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172609.981501] ffffffff80747dc0 ffff81021e322918 ffff8101a8329f34 ffff8101a8329f28 [1172609.981605] Call Trace: [1172609.986634] [] alarm_setitimer+0x35/0x70 [1172609.989220] [] sys_pause+0x19/0x30 [1172609.991766] [] system_call+0x7e/0x83 [1172609.994292] [1172609.996787] screen D ffff8100a18ff800 0 30992 30991 [1172609.999344] ffff8101a854dd28 0000000000000086 ffff81022854ddb7 ffff8101a854dcd8 [1172610.001953] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.004574] ffffffff80747dc0 ffff810170233028 ffffffff80656bcb ffffffff8021f8bc [1172610.004677] Call Trace: [1172610.009752] [] task_rq_lock+0x4c/0x90 [1172610.012366] [] try_to_wake_up+0x68/0x3b0 [1172610.014981] [] wait_for_completion+0x7d/0xc0 [1172610.017594] [] default_wake_function+0x0/0x10 [1172610.020208] [] flush_cpu_workqueue+0x6a/0x90 [1172610.022828] [] wq_barrier_func+0x0/0x10 [1172610.025447] [] flush_workqueue+0x33/0x50 [1172610.028076] [] release_dev+0x44f/0x750 [1172610.030710] [] mntput_no_expire+0x27/0xb0 [1172610.033339] [] tty_release+0x11/0x20 [1172610.035958] [] __fput+0xb1/0x1a0 [1172610.038547] [] filp_close+0x54/0x90 [1172610.041106] [] sys_close+0x96/0x100 [1172610.043652] [] system_call+0x7e/0x83 [1172610.046160] [1172610.048618] bash ? 0000000000000000 0 30993 30992 [1172610.051135] ffff8101aa2a3ee8 0000000000000046 ffff8101aa2a3eb0 0000000000000011 [1172610.053708] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.056312] ffffffff80747dc0 ffff810170233738 ffff8101aa2a3eb4 ffff8101aa2a3ea8 [1172610.056415] Call Trace: [1172610.061510] [] do_exit+0x5be/0x8a0 [1172610.064172] [] do_group_exit+0x2c/0x80 [1172610.066859] [] system_call+0x7e/0x83 [1172610.069537] [1172610.072190] sshd S 0000000000000000 0 7001 7582 [1172610.074908] ffff8100792b1bf8 0000000000000082 0000000000000000 ffff8101e9c51b80 [1172610.077679] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.080477] ffffffff80747dc0 ffff8102234ff738 ffff8100792b1bc4 ffff8100792b1bb8 [1172610.080580] Call Trace: [1172610.086042] [] schedule_timeout+0x95/0xd0 [1172610.088861] [] prepare_to_wait+0x23/0x80 [1172610.091673] [] unix_stream_recvmsg+0x386/0x550 [1172610.094492] [] autoremove_wake_function+0x0/0x30 [1172610.097318] [] link_path_walk+0x80/0xf0 [1172610.100148] [] sock_aio_read+0x11b/0x130 [1172610.102976] [] get_unused_fd_flags+0x79/0x120 [1172610.105822] [] do_sync_read+0xd9/0x120 [1172610.108651] [] autoremove_wake_function+0x0/0x30 [1172610.111495] [] __dentry_open+0x11f/0x1b0 [1172610.114319] [] do_filp_open+0x3a/0x50 [1172610.117118] [] vfs_read+0x157/0x160 [1172610.119902] [] sys_read+0x53/0x90 [1172610.122638] [] system_call+0x7e/0x83 [1172610.125360] [1172610.128056] sshd S 0000000000000000 0 7003 7001 [1172610.130818] ffff8100675a39e8 0000000000000082 ffff8100675a39b0 0000000000000002 [1172610.133623] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.136446] ffffffff80747dc0 ffff810225459028 ffff8100675a39b4 ffff8100675a39a8 [1172610.136549] Call Trace: [1172610.142064] [] schedule_timeout+0x5f/0xd0 [1172610.144899] [] process_timeout+0x0/0x10 [1172610.147716] [] do_select+0x468/0x560 [1172610.150495] [] __pollwait+0x0/0x130 [1172610.153260] [] default_wake_function+0x0/0x10 [1172610.156005] [] default_wake_function+0x0/0x10 [1172610.158707] [] default_wake_function+0x0/0x10 [1172610.161378] [] default_wake_function+0x0/0x10 [1172610.164026] [] skb_copy_datagram_iovec+0x1a1/0x260 [1172610.166675] [] _spin_lock_bh+0x9/0x20 [1172610.169315] [] release_sock+0x13/0xb0 [1172610.171917] [] tcp_recvmsg+0x370/0x940 [1172610.174494] [] sock_common_recvmsg+0x30/0x50 [1172610.177085] [] sock_aio_read+0x11b/0x130 [1172610.179638] [] core_sys_select+0x209/0x300 [1172610.182178] [] autoremove_wake_function+0x0/0x30 [1172610.184734] [] default_wake_function+0x0/0x10 [1172610.187290] [] current_fs_time+0x1e/0x30 [1172610.189837] [] tty_ldisc_deref+0x52/0x80 [1172610.192370] [] sys_select+0xd1/0x1c0 [1172610.194900] [] system_call+0x7e/0x83 [1172610.197426] [1172610.199919] bash S 000000000000000e 0 7004 7003 [1172610.202470] ffff8100cc263e88 0000000000000082 80000000804ca065 ffff81022367f530 [1172610.205071] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.207699] ffffffff80747dc0 ffff8102234fe918 ffff8100cc263e38 ffff810035a16348 [1172610.207802] Call Trace: [1172610.212949] [] do_page_fault+0x202/0x890 [1172610.215618] [] do_wait+0x599/0xc90 [1172610.218263] [] __wake_up+0x43/0x70 [1172610.220900] [] vfs_ioctl+0x220/0x2c0 [1172610.223509] [] default_wake_function+0x0/0x10 [1172610.226109] [] sys_ioctl+0x49/0x80 [1172610.228693] [] system_call+0x7e/0x83 [1172610.231240] [1172610.233746] aur S 0000000000000000 0 7014 7004 [1172610.236319] ffff810098071e88 0000000000000086 ffff810098071e50 ffffffff80232c93 [1172610.238941] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.241566] ffffffff80747dc0 ffff81022367f738 ffff810098071e54 ffff810098071e48 [1172610.241669] Call Trace: [1172610.246766] [] get_signal_to_deliver+0x73/0x470 [1172610.249380] [] do_wait+0x599/0xc90 [1172610.251983] [] default_wake_function+0x0/0x10 [1172610.254563] [] system_call+0x7e/0x83 [1172610.257122] [1172610.259648] aur S 0000000000000004 0 7066 7014 [1172610.262226] ffff810085231e88 0000000000000086 ffff8101ea314ce8 ffffffff80232c93 [1172610.264844] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.267471] ffffffff80747dc0 ffff8101ea314918 ffffffff802302ce ffffffff8020b3d6 [1172610.267574] Call Trace: [1172610.272674] [] get_signal_to_deliver+0x73/0x470 [1172610.275315] [] recalc_sigpending+0xe/0x30 [1172610.277948] [] do_notify_resume+0x536/0x7a0 [1172610.280577] [] do_wait+0x599/0xc90 [1172610.283199] [] default_wake_function+0x0/0x10 [1172610.285840] [] system_call+0x7e/0x83 [1172610.288491] [1172610.291116] unrar D ffff8100aa785c80 0 7135 7066 [1172610.293792] ffff8101ecf4ddb8 0000000000000086 ffff8101ecf4dd80 0000000000000000 [1172610.296525] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.299256] ffffffff80747dc0 ffff81021e53f028 ffff8101ecf4dd84 ffff8101ecf4dd78 [1172610.299359] Call Trace: [1172610.304629] [] vn_iowait+0x75/0xa0 [1172610.307301] [] autoremove_wake_function+0x0/0x30 [1172610.309979] [] xfs_trans_alloc+0x9c/0xb0 [1172610.312653] [] xfs_itruncate_start+0x35/0xe0 [1172610.315340] [] xfs_free_eofblocks+0x17a/0x280 [1172610.318032] [] xfs_release+0x134/0x1e0 [1172610.320711] [] xfs_file_release+0x1a/0x30 [1172610.323417] [] __fput+0xb1/0x1a0 [1172610.326144] [] filp_close+0x54/0x90 [1172610.328895] [] sys_close+0x96/0x100 [1172610.331631] [] system_call+0x7e/0x83 [1172610.334353] [1172610.337050] sshd D 0000000000000000 0 7187 7582 [1172610.339811] ffff81002b62fd28 0000000000000086 ffff81002b62fcf0 ffff81002b62fcd8 [1172610.342618] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.345448] ffffffff80747dc0 ffff8101ccda3028 ffff81002b62fcf4 ffff81002b62fce8 [1172610.345551] Call Trace: [1172610.351072] [] wait_for_completion+0x7d/0xc0 [1172610.353915] [] default_wake_function+0x0/0x10 [1172610.356765] [] flush_cpu_workqueue+0x6a/0x90 [1172610.359622] [] wq_barrier_func+0x0/0x10 [1172610.362477] [] flush_workqueue+0x33/0x50 [1172610.365337] [] release_dev+0x44f/0x750 [1172610.368184] [] sys_fchmodat+0x6a/0x120 [1172610.371026] [] tty_release+0x11/0x20 [1172610.373843] [] __fput+0xb1/0x1a0 [1172610.376628] [] filp_close+0x54/0x90 [1172610.379402] [] sys_close+0x96/0x100 [1172610.382135] [] system_call+0x7e/0x83 [1172610.384846] [1172610.387529] sshd ? 0000000000000000 0 7218 7187 [1172610.390280] ffff81013bd7bee8 0000000000000046 ffff81013bd7beb0 0000000000000011 [1172610.393084] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.395907] ffffffff80747dc0 ffff8101bf6ae918 ffff81013bd7beb4 ffff81013bd7bea8 [1172610.396010] Call Trace: [1172610.401528] [] __cond_resched+0x1c/0x50 [1172610.404362] [] do_exit+0x5be/0x8a0 [1172610.407192] [] do_group_exit+0x2c/0x80 [1172610.409993] [] system_call+0x7e/0x83 [1172610.412776] [1172610.415520] sshd S 0000000000000000 0 7236 7582 [1172610.418293] ffff8101e4a89bf8 0000000000000082 ffff8101e4a89bc0 ffff81013bf542c0 [1172610.421090] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.423898] ffffffff80747dc0 ffff810168684208 ffff8101e4a89bc4 ffff8101e4a89bb8 [1172610.424001] Call Trace: [1172610.429474] [] schedule_timeout+0x95/0xd0 [1172610.432284] [] prepare_to_wait+0x23/0x80 [1172610.435096] [] unix_stream_recvmsg+0x386/0x550 [1172610.437896] [] autoremove_wake_function+0x0/0x30 [1172610.440690] [] link_path_walk+0x80/0xf0 [1172610.443487] [] sock_aio_read+0x11b/0x130 [1172610.446249] [] get_unused_fd_flags+0x79/0x120 [1172610.448997] [] do_sync_read+0xd9/0x120 [1172610.451737] [] autoremove_wake_function+0x0/0x30 [1172610.454491] [] __dentry_open+0x11f/0x1b0 [1172610.457244] [] do_filp_open+0x3a/0x50 [1172610.459989] [] vfs_read+0x157/0x160 [1172610.462724] [] sys_read+0x53/0x90 [1172610.465430] [] system_call+0x7e/0x83 [1172610.468131] [1172610.470765] sshd S 0000000000000000 0 7238 7236 [1172610.473440] ffff810046e1f9e8 0000000000000082 ffff810046e1f9b0 0000000000000002 [1172610.476161] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.478873] ffffffff80747dc0 ffff810168685028 ffff810046e1f9b4 ffff810046e1f9a8 [1172610.478975] Call Trace: [1172610.484236] [] schedule_timeout+0x5f/0xd0 [1172610.486940] [] process_timeout+0x0/0x10 [1172610.489645] [] do_select+0x468/0x560 [1172610.492340] [] __pollwait+0x0/0x130 [1172610.495030] [] default_wake_function+0x0/0x10 [1172610.497738] [] default_wake_function+0x0/0x10 [1172610.500417] [] default_wake_function+0x0/0x10 [1172610.503076] [] default_wake_function+0x0/0x10 [1172610.505711] [] skb_copy_datagram_iovec+0x1a1/0x260 [1172610.508366] [] _spin_lock_bh+0x9/0x20 [1172610.511004] [] release_sock+0x13/0xb0 [1172610.513638] [] tcp_recvmsg+0x370/0x940 [1172610.516245] [] sock_common_recvmsg+0x30/0x50 [1172610.518841] [] sock_aio_read+0x11b/0x130 [1172610.521423] [] core_sys_select+0x209/0x300 [1172610.523974] [] autoremove_wake_function+0x0/0x30 [1172610.526518] [] default_wake_function+0x0/0x10 [1172610.529058] [] current_fs_time+0x1e/0x30 [1172610.531592] [] tty_ldisc_deref+0x52/0x80 [1172610.534118] [] sys_select+0xd1/0x1c0 [1172610.536645] [] system_call+0x7e/0x83 [1172610.539162] [1172610.541651] bash S 000000000000000e 0 7239 7238 [1172610.544203] ffff8100aae5fe88 0000000000000082 80000001bab2c065 ffff810145b6ae20 [1172610.546809] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.549446] ffffffff80747dc0 ffff810168685738 ffff8100aae5fe38 ffff810065785f18 [1172610.549550] Call Trace: [1172610.554709] [] do_page_fault+0x202/0x890 [1172610.557368] [] update_curr+0x109/0x120 [1172610.560022] [] do_wait+0x599/0xc90 [1172610.562647] [] __sched_text_start+0x166/0x23d [1172610.565267] [] __wake_up+0x43/0x70 [1172610.567871] [] vfs_ioctl+0x220/0x2c0 [1172610.570435] [] default_wake_function+0x0/0x10 [1172610.572998] [] sys_ioctl+0x49/0x80 [1172610.575555] [] system_call+0x7e/0x83 [1172610.578118] [1172610.580652] sshd S 0000000000000000 0 7248 7582 [1172610.583235] ffff8101120d5bf8 0000000000000082 ffff8101120d5bc0 ffff81001e998dc0 [1172610.585865] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.588489] ffffffff80747dc0 ffff810130906208 ffff8101120d5bc4 ffff8101120d5bb8 [1172610.588592] Call Trace: [1172610.593666] [] schedule_timeout+0x95/0xd0 [1172610.596253] [] prepare_to_wait+0x23/0x80 [1172610.598824] [] unix_stream_recvmsg+0x386/0x550 [1172610.601405] [] autoremove_wake_function+0x0/0x30 [1172610.603992] [] link_path_walk+0x80/0xf0 [1172610.606571] [] sock_aio_read+0x11b/0x130 [1172610.609138] [] get_unused_fd_flags+0x79/0x120 [1172610.611720] [] do_sync_read+0xd9/0x120 [1172610.614293] [] autoremove_wake_function+0x0/0x30 [1172610.616883] [] __dentry_open+0x11f/0x1b0 [1172610.619463] [] do_filp_open+0x3a/0x50 [1172610.622029] [] vfs_read+0x157/0x160 [1172610.624594] [] sys_read+0x53/0x90 [1172610.627144] [] system_call+0x7e/0x83 [1172610.629703] [1172610.632237] sshd S 0000000000000000 0 7250 7248 [1172610.634822] ffff810126f3d9e8 0000000000000086 ffff810126f3d9b0 0000000000000002 [1172610.637453] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.640086] ffffffff80747dc0 ffff810130907028 ffff810126f3d9b4 ffff810126f3d9a8 [1172610.640190] Call Trace: [1172610.645268] [] schedule_timeout+0x5f/0xd0 [1172610.647857] [] process_timeout+0x0/0x10 [1172610.650429] [] do_select+0x468/0x560 [1172610.652990] [] __pollwait+0x0/0x130 [1172610.655552] [] default_wake_function+0x0/0x10 [1172610.658131] [] default_wake_function+0x0/0x10 [1172610.660680] [] default_wake_function+0x0/0x10 [1172610.663230] [] default_wake_function+0x0/0x10 [1172610.665779] [] skb_copy_datagram_iovec+0x1a1/0x260 [1172610.668368] [] _spin_lock_bh+0x9/0x20 [1172610.670946] [] release_sock+0x13/0xb0 [1172610.673510] [] tcp_recvmsg+0x370/0x940 [1172610.676075] [] sock_common_recvmsg+0x30/0x50 [1172610.678653] [] sock_aio_read+0x11b/0x130 [1172610.681222] [] core_sys_select+0x209/0x300 [1172610.683798] [] autoremove_wake_function+0x0/0x30 [1172610.686386] [] default_wake_function+0x0/0x10 [1172610.688970] [] current_fs_time+0x1e/0x30 [1172610.691546] [] tty_ldisc_deref+0x52/0x80 [1172610.694114] [] sys_select+0xd1/0x1c0 [1172610.696683] [] system_call+0x7e/0x83 [1172610.699244] [1172610.701782] bash S 000000000000000e 0 7251 7250 [1172610.704370] ffff810121e8de88 0000000000000086 800000008e47c065 ffff8101afbec710 [1172610.707005] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.709641] ffffffff80747dc0 ffff810130907738 ffff810121e8de38 ffff8101e65ef9d8 [1172610.709744] Call Trace: [1172610.714827] [] do_page_fault+0x202/0x890 [1172610.717419] [] do_wait+0x599/0xc90 [1172610.719979] [] __wake_up+0x43/0x70 [1172610.722535] [] vfs_ioctl+0x220/0x2c0 [1172610.725088] [] default_wake_function+0x0/0x10 [1172610.727650] [] sys_ioctl+0x49/0x80 [1172610.730203] [] system_call+0x7e/0x83 [1172610.732759] [1172610.735278] su S 0000000000000000 0 7269 7251 [1172610.737850] ffff8101a5007e88 0000000000000086 ffff8101a5007e50 ffff8100219c0e20 [1172610.740475] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.743107] ffffffff80747dc0 ffff8101afbec918 ffff8101a5007e54 ffff8101a5007e48 [1172610.743210] Call Trace: [1172610.748316] [] do_wait+0x599/0xc90 [1172610.750913] [] default_wake_function+0x0/0x10 [1172610.753518] [] system_call+0x7e/0x83 [1172610.756084] [1172610.758600] bash S 0000000000000000 0 7270 7269 [1172610.761175] ffff81014bc9be88 0000000000000086 ffff81014bc9be50 ffff810139e7c000 [1172610.763792] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.766419] ffffffff80747dc0 ffff8100219c1028 ffff81014bc9be54 ffff81014bc9be48 [1172610.766521] Call Trace: [1172610.771636] [] do_wait+0x599/0xc90 [1172610.774264] [] __wake_up+0x43/0x70 [1172610.776885] [] vfs_ioctl+0x220/0x2c0 [1172610.779492] [] default_wake_function+0x0/0x10 [1172610.782107] [] sys_ioctl+0x49/0x80 [1172610.784719] [] system_call+0x7e/0x83 [1172610.787329] [1172610.789920] sshd S 0000000000000000 0 7278 7582 [1172610.792579] ffff810194cf5bf8 0000000000000086 ffff810194cf5bc0 ffff810010755600 [1172610.795276] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.797987] ffffffff80747dc0 ffff81002d667738 ffff810194cf5bc4 ffff810194cf5bb8 [1172610.798090] Call Trace: [1172610.803311] [] schedule_timeout+0x95/0xd0 [1172610.805992] [] prepare_to_wait+0x23/0x80 [1172610.808641] [] unix_stream_recvmsg+0x386/0x550 [1172610.811284] [] autoremove_wake_function+0x0/0x30 [1172610.813937] [] link_path_walk+0x80/0xf0 [1172610.816593] [] sock_aio_read+0x11b/0x130 [1172610.819250] [] get_unused_fd_flags+0x79/0x120 [1172610.821914] [] do_sync_read+0xd9/0x120 [1172610.824602] [] autoremove_wake_function+0x0/0x30 [1172610.827337] [] __dentry_open+0x11f/0x1b0 [1172610.830101] [] do_filp_open+0x3a/0x50 [1172610.832855] [] vfs_read+0x157/0x160 [1172610.835593] [] sys_read+0x53/0x90 [1172610.838321] [] system_call+0x7e/0x83 [1172610.841049] [1172610.843744] sshd S 0000000000000000 0 7280 7278 [1172610.846501] ffff81013acb39e8 0000000000000082 ffff81013acb39b0 0000000000000002 [1172610.849305] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.852125] ffffffff80747dc0 ffff81011060e918 ffff81013acb39b4 ffff81013acb39a8 [1172610.852228] Call Trace: [1172610.857719] [] schedule_timeout+0x5f/0xd0 [1172610.860553] [] process_timeout+0x0/0x10 [1172610.863393] [] do_select+0x468/0x560 [1172610.866228] [] __pollwait+0x0/0x130 [1172610.869046] [] default_wake_function+0x0/0x10 [1172610.871874] [] default_wake_function+0x0/0x10 [1172610.874652] [] default_wake_function+0x0/0x10 [1172610.877377] [] default_wake_function+0x0/0x10 [1172610.880068] [] add_partial+0x19/0x60 [1172610.882722] [] __slab_free+0x15d/0x310 [1172610.885354] [] _spin_lock_bh+0x9/0x20 [1172610.887984] [] release_sock+0x13/0xb0 [1172610.890616] [] tcp_recvmsg+0x370/0x940 [1172610.893251] [] sock_common_recvmsg+0x30/0x50 [1172610.895887] [] sock_aio_read+0x11b/0x130 [1172610.898512] [] core_sys_select+0x209/0x300 [1172610.901144] [] autoremove_wake_function+0x0/0x30 [1172610.903780] [] default_wake_function+0x0/0x10 [1172610.906421] [] current_fs_time+0x1e/0x30 [1172610.909040] [] tty_ldisc_deref+0x52/0x80 [1172610.911632] [] sys_select+0xd1/0x1c0 [1172610.914215] [] system_call+0x7e/0x83 [1172610.916754] [1172610.919253] bash S 000000000000000e 0 7281 7280 [1172610.921808] ffff8101919e3e88 0000000000000082 80000001542be065 ffff8100867c7530 [1172610.924409] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.927021] ffffffff80747dc0 ffff81011060f028 ffff8101919e3e38 ffff8101aae8a930 [1172610.927124] Call Trace: [1172610.932186] [] do_page_fault+0x202/0x890 [1172610.934771] [] do_wait+0x599/0xc90 [1172610.937337] [] __wake_up+0x43/0x70 [1172610.939863] [] default_wake_function+0x0/0x10 [1172610.942391] [] system_call+0x7e/0x83 [1172610.944923] [1172610.947429] su S 0000000000000000 0 7288 7281 [1172610.949987] ffff81004e873e88 0000000000000086 ffff81004e873e50 ffff81011060f530 [1172610.952588] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.955214] ffffffff80747dc0 ffff8100867c7738 ffff81004e873e54 ffff81004e873e48 [1172610.955317] Call Trace: [1172610.960412] [] do_wait+0x599/0xc90 [1172610.963007] [] default_wake_function+0x0/0x10 [1172610.965602] [] system_call+0x7e/0x83 [1172610.968186] [1172610.970703] bash S 0000000000000000 0 7289 7288 [1172610.973262] ffff810043dbfdb8 0000000000000082 ffff810043dbfd80 0000000000000fee [1172610.975867] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172610.978487] ffffffff80747dc0 ffff81011060f738 ffff810043dbfd84 ffff810043dbfd78 [1172610.978590] Call Trace: [1172610.983666] [] schedule_timeout+0x95/0xd0 [1172610.986293] [] add_wait_queue+0x1c/0x60 [1172610.988916] [] read_chan+0x228/0x6f0 [1172610.991531] [] default_wake_function+0x0/0x10 [1172610.994156] [] tty_read+0xb0/0x100 [1172610.996766] [] vfs_read+0xc5/0x160 [1172610.999359] [] sys_read+0x53/0x90 [1172611.001936] [] system_call+0x7e/0x83 [1172611.004527] [1172611.007092] strace S 0000000000000000 0 7319 7270 [1172611.009707] ffff8101534a9e88 0000000000000086 ffff8101534a9e50 0000000000000092 [1172611.012368] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172611.015032] ffffffff80747dc0 ffff810139e7c208 ffff8101534a9e54 ffff8101534a9e48 [1172611.015135] Call Trace: [1172611.020281] [] __group_send_sig_info+0x75/0xa0 [1172611.022914] [] do_wait+0x599/0xc90 [1172611.025526] [] kill_pid_info+0x51/0x90 [1172611.028133] [] default_wake_function+0x0/0x10 [1172611.030760] [] system_call+0x7e/0x83 [1172611.033389] [1172611.035983] rm D 0000000000000000 0 7463 7239 [1172611.038664] ffff8101254a3b08 0000000000000086 ffff8101254a3ad0 ffffffff80592c6c [1172611.041422] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172611.044223] ffffffff80747dc0 ffff810145b6b028 ffff8101254a3ad4 ffff8101254a3ac8 [1172611.044326] Call Trace: [1172611.049792] [] __down+0x10c/0x11f [1172611.052604] [] __down+0xa7/0x11f [1172611.055405] [] default_wake_function+0x0/0x10 [1172611.058227] [] __down_failed+0x35/0x3a [1172611.061044] [] xfs_buf_lock+0x3e/0x40 [1172611.063866] [] xfs_getsb+0x15/0x40 [1172611.066676] [] xfs_trans_getsb+0x5a/0xb0 [1172611.069478] [] xfs_trans_apply_sb_deltas+0xf/0x370 [1172611.072281] [] _xfs_trans_commit+0x9e/0x3c0 [1172611.075085] [] __up_read+0x21/0xb0 [1172611.077884] [] xfs_free_extent+0xe2/0x110 [1172611.080690] [] kmem_zone_alloc+0x5c/0xd0 [1172611.083499] [] kmem_zone_alloc+0x5c/0xd0 [1172611.086267] [] kmem_zone_zalloc+0x32/0x50 [1172611.089024] [] xfs_itruncate_finish+0xdb/0x320 [1172611.091768] [] xfs_inactive+0x3f1/0x520 [1172611.094486] [] xfs_fs_clear_inode+0xa9/0x100 [1172611.097203] [] clear_inode+0x58/0xf0 [1172611.099883] [] generic_delete_inode+0xe9/0xf0 [1172611.102557] [] do_unlinkat+0x14a/0x1c0 [1172611.105235] [] error_exit+0x0/0x84 [1172611.107916] [] system_call+0x7e/0x83 [1172611.110592] [1172611.113232] pickup S 0000000000000000 0 7573 30580 [1172611.115922] ffff81021d34be58 0000000000000086 ffff81021d34be20 0000000000000000 [1172611.118661] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172611.121398] ffffffff80747dc0 ffff8101964af028 ffff81021d34be24 ffff81021d34be18 [1172611.121501] Call Trace: [1172611.126801] [] schedule_timeout+0x5f/0xd0 [1172611.129501] [] process_timeout+0x0/0x10 [1172611.132189] [] sys_epoll_wait+0x1bd/0x4e0 [1172611.134877] [] default_wake_function+0x0/0x10 [1172611.137570] [] system_call+0x7e/0x83 [1172611.140247] [1172611.142890] bash D 0000000000000000 0 8896 1 [1172611.145570] ffff8101cdf07ac8 0000000000000046 ffff8101cdf07a90 ffff810226a79800 [1172611.148276] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172611.150995] ffffffff80747dc0 ffff810114417738 ffff8101cdf07a94 ffff8101cdf07a88 [1172611.151098] Call Trace: [1172611.156317] [] wait_for_completion+0x7d/0xc0 [1172611.159006] [] default_wake_function+0x0/0x10 [1172611.161706] [] flush_cpu_workqueue+0x6a/0x90 [1172611.164401] [] wq_barrier_func+0x0/0x10 [1172611.167086] [] flush_workqueue+0x33/0x50 [1172611.169785] [] release_dev+0x44f/0x750 [1172611.172499] [] __sched_text_start+0x166/0x23d [1172611.175232] [] tty_release+0x11/0x20 [1172611.177948] [] __fput+0xb1/0x1a0 [1172611.180652] [] filp_close+0x54/0x90 [1172611.183331] [] put_files_struct+0xb1/0xd0 [1172611.185991] [] do_exit+0x1a9/0x8a0 [1172611.188636] [] __dequeue_signal+0x165/0x1f0 [1172611.191258] [] do_group_exit+0x2c/0x80 [1172611.193857] [] get_signal_to_deliver+0x2c7/0x470 [1172611.196464] [] do_notify_resume+0xc5/0x7a0 [1172611.199077] [] send_signal+0x62/0x1f0 [1172611.201678] [] __group_send_sig_info+0x75/0xa0 [1172611.204289] [] group_send_sig_info+0x6e/0x90 [1172611.206890] [] sys_rt_sigreturn+0x324/0x3d0 [1172611.209498] [] sys_rt_sigaction+0x8e/0xc0 [1172611.212068] [] int_signal+0x12/0x17 [1172611.214618] [1172611.217129] su ? 0000000000000000 0 8903 8896 [1172611.219666] ffff8101e685dee8 0000000000000046 ffff8101e685deb0 0000000000000011 [1172611.222241] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172611.224859] ffffffff80747dc0 ffff8101158e9028 ffff8101e685deb4 ffff8101e685dea8 [1172611.224962] Call Trace: [1172611.230051] [] do_exit+0x5be/0x8a0 [1172611.232666] [] do_group_exit+0x2c/0x80 [1172611.235284] [] system_call+0x7e/0x83 [1172611.237904] [1172611.240493] bash D ffff8101bfb7e600 0 8977 1 [1172611.243132] ffff810106e37ac8 0000000000000046 ffff810106e37c08 ffff810226a79800 [1172611.245831] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80 [1172611.248548] ffffffff80747dc0 ffff81018e79d738 0000000000000000 0000000000000000 [1172611.248652] Call Trace: [1172611.253996] [] wait_for_completion+0x7d/0xc0 [1172611.256787] [] default_wake_function+0x0/0x10 [1172611.259581] [] flush_cpu_workqueue+0x6a/0x90 [1172611.262374] [] wq_barrier_func+0x0/0x10 [1172611.265167] [] flush_workqueue+0x33/0x50 [1172611.267952] [] release_dev+0x44f/0x750 [1172611.270733] [] tty_release+0x11/0x20 [1172611.273502] [] __fput+0xb1/0x1a0 [1172611.276259] [] filp_close+0x54/0x90 [1172611.279016] [] put_files_struct+0xb1/0xd0 [1172611.281764] [] do_exit+0x1a9/0x8a0 [1172611.284504] [] __dequeue_signal+0x165/0x1f0 [1172611.287232] [] do_group_exit+0x2c/0x80 [1172611.289939] [] get_signal_to_deliver+0x2c7/0x470 [1172611.292654] [] do_notify_resume+0xc5/0x7a0 [1172611.295338] [