Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752252Ab2KRP3Z (ORCPT ); Sun, 18 Nov 2012 10:29:25 -0500 Received: from mail-ee0-f46.google.com ([74.125.83.46]:58384 "EHLO mail-ee0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752039Ab2KRP3X (ORCPT ); Sun, 18 Nov 2012 10:29:23 -0500 MIME-Version: 1.0 In-Reply-To: References: <20121029222613.GU29378@dastard>

Date: Sun, 18 Nov 2012 16:29:22 +0100 Message-ID: Subject: Re: Hang in XFS reclaim on 3.7.0-rc3 From: Torsten Kaiser To: Dave Chinner Cc: xfs@oss.sgi.com, Linux Kernel Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 45085 Lines: 891 On Sun, Nov 18, 2012 at 11:24 AM, Torsten Kaiser wrote: > On Tue, Oct 30, 2012 at 9:37 PM, Torsten Kaiser > wrote: >> I will keep LOCKDEP enabled on that system, and if there really is >> another splat, I will report back here. But I rather doubt that this >> will be needed. > > After the patch, I did not see this problem again, but today I found > another LOCKDEP report that also looks XFS related. > I found it twice in the logs, and as both were slightly different, I > will attach both versions. > Nov 6 21:57:09 thoregon kernel: [ 9941.104353] 3.7.0-rc4 #1 Not tainted > Nov 6 21:57:09 thoregon kernel: [ 9941.104355] inconsistent > {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > Nov 6 21:57:09 thoregon kernel: [ 9941.104430] CPU0 > Nov 6 21:57:09 thoregon kernel: [ 9941.104431] ---- > Nov 6 21:57:09 thoregon kernel: [ 9941.104432] lock(&(&ip->i_lock)->mr_lock); > Nov 6 21:57:09 thoregon kernel: [ 9941.104433] > Nov 6 21:57:09 thoregon kernel: [ 9941.104434] > lock(&(&ip->i_lock)->mr_lock); > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] *** DEADLOCK *** Sorry! Copied the wrong report. Your fix only landed in -rc5, so my vanilla -rc4 did (also) report the old problem again. And I copy&pasted that report instead of the second appearance of the new problem. Here is the correct second report of the sb_internal vs ip->i_lock->mr_lock problem: [110926.972477] [110926.972482] ========================================================= [110926.972484] [ INFO: possible irq lock inversion dependency detected ] [110926.972486] 3.7.0-rc4 #1 Not tainted [110926.972487] --------------------------------------------------------- [110926.972489] kswapd0/725 just changed the state of lock: [110926.972490] (sb_internal){.+.+.?}, at: [] xfs_trans_alloc+0x28/0x50 [110926.972499] but this lock took another, RECLAIM_FS-unsafe lock in the past: [110926.972500] (&(&ip->i_lock)->mr_lock/1){+.+.+.} [110926.972500] [110926.972500] and interrupts could create inverse lock ordering between them. [110926.972500] [110926.972503] [110926.972503] other info that might help us debug this: [110926.972504] Possible interrupt unsafe locking scenario: [110926.972504] [110926.972505] CPU0 CPU1 [110926.972506] ---- ---- [110926.972507] lock(&(&ip->i_lock)->mr_lock/1); [110926.972509] local_irq_disable(); [110926.972509] lock(sb_internal); [110926.972511] lock(&(&ip->i_lock)->mr_lock/1); [110926.972512] [110926.972513] lock(sb_internal); [110926.972514] [110926.972514] *** DEADLOCK *** [110926.972514] [110926.972516] 2 locks held by kswapd0/725: [110926.972517] #0: (shrinker_rwsem){++++..}, at: [] shrink_slab+0x32/0x1f0 [110926.972522] #1: (&type->s_umount_key#20){++++.+}, at: [] grab_super_passive+0x3e/0x90 [110926.972527] [110926.972527] the shortest dependencies between 2nd lock and 1st lock: [110926.972533] -> (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 58117 { [110926.972536] HARDIRQ-ON-W at: [110926.972537] [] __lock_acquire+0x631/0x1c00 [110926.972540] [] lock_acquire+0x55/0x70 [110926.972542] [] down_write_nested+0x4a/0x70 [110926.972545] [] xfs_ilock+0x84/0xb0 [110926.972548] [] xfs_create+0x1d4/0x5a0 [110926.972550] [] xfs_vn_mknod+0x8a/0x1b0 [110926.972552] [] xfs_vn_create+0xe/0x10 [110926.972554] [] vfs_create+0x72/0xc0 [110926.972556] [] do_last.isra.69+0x80e/0xc80 [110926.972558] [] path_openat.isra.70+0xab/0x490 [110926.972560] [] do_filp_open+0x3d/0xa0 [110926.972562] [] do_sys_open+0xf9/0x1e0 [110926.972565] [] sys_open+0x1c/0x20 [110926.972567] [] system_call_fastpath+0x16/0x1b [110926.972570] SOFTIRQ-ON-W at: [110926.972571] [] __lock_acquire+0x667/0x1c00 [110926.972573] [] lock_acquire+0x55/0x70 [110926.972574] [] down_write_nested+0x4a/0x70 [110926.972576] [] xfs_ilock+0x84/0xb0 [110926.972578] [] xfs_create+0x1d4/0x5a0 [110926.972580] [] xfs_vn_mknod+0x8a/0x1b0 [110926.972581] [] xfs_vn_create+0xe/0x10 [110926.972583] [] vfs_create+0x72/0xc0 [110926.972585] [] do_last.isra.69+0x80e/0xc80 [110926.972587] [] path_openat.isra.70+0xab/0x490 [110926.972589] [] do_filp_open+0x3d/0xa0 [110926.972591] [] do_sys_open+0xf9/0x1e0 [110926.972593] [] sys_open+0x1c/0x20 [110926.972595] [] system_call_fastpath+0x16/0x1b [110926.972597] RECLAIM_FS-ON-W at: [110926.972598] [] mark_held_locks+0x7e/0x130 [110926.972600] [] lockdep_trace_alloc+0x63/0xc0 [110926.972601] [] kmem_cache_alloc+0x35/0xe0 [110926.972603] [] vm_map_ram+0x271/0x770 [110926.972606] [] _xfs_buf_map_pages+0x46/0xe0 [110926.972609] [] xfs_buf_get_map+0x8a/0x130 [110926.972610] [] xfs_trans_get_buf_map+0xa9/0xd0 [110926.972613] [] xfs_ialloc_inode_init+0xcd/0x1d0 [110926.972616] [] xfs_ialloc_ag_alloc+0x1be/0x560 [110926.972618] [] xfs_dialloc+0x185/0x2a0 [110926.972619] [] xfs_ialloc+0x58/0x650 [110926.972621] [] xfs_dir_ialloc+0x65/0x270 [110926.972623] [] xfs_create+0x3ac/0x5a0 [110926.972624] [] xfs_vn_mknod+0x8a/0x1b0 [110926.972626] [] xfs_vn_create+0xe/0x10 [110926.972628] [] vfs_create+0x72/0xc0 [110926.972630] [] do_last.isra.69+0x80e/0xc80 [110926.972632] [] path_openat.isra.70+0xab/0x490 [110926.972634] [] do_filp_open+0x3d/0xa0 [110926.972636] [] do_sys_open+0xf9/0x1e0 [110926.972638] [] sys_open+0x1c/0x20 [110926.972640] [] system_call_fastpath+0x16/0x1b [110926.972642] INITIAL USE at: [110926.972642] [] __lock_acquire+0x2e9/0x1c00 [110926.972644] [] lock_acquire+0x55/0x70 [110926.972646] [] down_write_nested+0x4a/0x70 [110926.972648] [] xfs_ilock+0x84/0xb0 [110926.972650] [] xfs_create+0x1d4/0x5a0 [110926.972651] [] xfs_vn_mknod+0x8a/0x1b0 [110926.972653] [] xfs_vn_create+0xe/0x10 [110926.972655] [] vfs_create+0x72/0xc0 [110926.972657] [] do_last.isra.69+0x80e/0xc80 [110926.972659] [] path_openat.isra.70+0xab/0x490 [110926.972661] [] do_filp_open+0x3d/0xa0 [110926.972663] [] do_sys_open+0xf9/0x1e0 [110926.972664] [] sys_open+0x1c/0x20 [110926.972666] [] system_call_fastpath+0x16/0x1b [110926.972668] } [110926.972669] ... key at: [] __key.41355+0x1/0x8 [110926.972672] ... acquired at: [110926.972672] [] lock_acquire+0x55/0x70 [110926.972674] [] down_write_nested+0x4a/0x70 [110926.972676] [] xfs_ilock+0x84/0xb0 [110926.972678] [] xfs_create+0x1d4/0x5a0 [110926.972679] [] xfs_vn_mknod+0x8a/0x1b0 [110926.972681] [] xfs_vn_create+0xe/0x10 [110926.972683] [] vfs_create+0x72/0xc0 [110926.972684] [] do_last.isra.69+0x80e/0xc80 [110926.972686] [] path_openat.isra.70+0xab/0x490 [110926.972688] [] do_filp_open+0x3d/0xa0 [110926.972690] [] do_sys_open+0xf9/0x1e0 [110926.972692] [] sys_open+0x1c/0x20 [110926.972694] [] system_call_fastpath+0x16/0x1b [110926.972696] [110926.972696] -> (sb_internal){.+.+.?} ops: 1710064 { [110926.972699] HARDIRQ-ON-R at: [110926.972700] [] __lock_acquire+0x50a/0x1c00 [110926.972702] [] lock_acquire+0x55/0x70 [110926.972704] [] __sb_start_write+0xab/0x190 [110926.972705] [] xfs_trans_alloc+0x28/0x50 [110926.972707] [] xfs_create+0x187/0x5a0 [110926.972709] [] xfs_vn_mknod+0x8a/0x1b0 [110926.972711] [] xfs_vn_create+0xe/0x10 [110926.972712] [] vfs_create+0x72/0xc0 [110926.972714] [] do_last.isra.69+0x80e/0xc80 [110926.972716] [] path_openat.isra.70+0xab/0x490 [110926.972718] [] do_filp_open+0x3d/0xa0 [110926.972720] [] do_sys_open+0xf9/0x1e0 [110926.972722] [] sys_open+0x1c/0x20 [110926.972724] [] system_call_fastpath+0x16/0x1b [110926.972726] SOFTIRQ-ON-R at: [110926.972727] [] __lock_acquire+0x667/0x1c00 [110926.972728] [] lock_acquire+0x55/0x70 [110926.972730] [] __sb_start_write+0xab/0x190 [110926.972732] [] xfs_trans_alloc+0x28/0x50 [110926.972734] [] xfs_create+0x187/0x5a0 [110926.972735] [] xfs_vn_mknod+0x8a/0x1b0 [110926.972737] [] xfs_vn_create+0xe/0x10 [110926.972739] [] vfs_create+0x72/0xc0 [110926.972741] [] do_last.isra.69+0x80e/0xc80 [110926.972743] [] path_openat.isra.70+0xab/0x490 [110926.972745] [] do_filp_open+0x3d/0xa0 [110926.972747] [] do_sys_open+0xf9/0x1e0 [110926.972749] [] sys_open+0x1c/0x20 [110926.972750] [] system_call_fastpath+0x16/0x1b [110926.972752] IN-RECLAIM_FS-R at: [110926.972753] [] __lock_acquire+0x57f/0x1c00 [110926.972755] [] lock_acquire+0x55/0x70 [110926.972757] [] __sb_start_write+0xab/0x190 [110926.972758] [] xfs_trans_alloc+0x28/0x50 [110926.972760] [] xfs_free_eofblocks+0x104/0x250 [110926.972762] [] xfs_inactive+0xa9/0x480 [110926.972763] [] xfs_fs_evict_inode+0x70/0x80 [110926.972765] [] evict+0xaf/0x1b0 [110926.972768] [] dispose_list+0x39/0x50 [110926.972770] [] prune_icache_sb+0x183/0x340 [110926.972772] [] prune_super+0xf3/0x1a0 [110926.972773] [] shrink_slab+0x11e/0x1f0 [110926.972775] [] kswapd+0x690/0xa10 [110926.972777] [] kthread+0xd6/0xe0 [110926.972779] [] ret_from_fork+0x7c/0xb0 [110926.972781] RECLAIM_FS-ON-R at: [110926.972782] [] mark_held_locks+0x7e/0x130 [110926.972784] [] lockdep_trace_alloc+0x63/0xc0 [110926.972785] [] kmem_cache_alloc+0x35/0xe0 [110926.972787] [] kmem_zone_alloc+0x5f/0xe0 [110926.972789] [] kmem_zone_zalloc+0x18/0x50 [110926.972790] [] _xfs_trans_alloc+0x32/0x90 [110926.972792] [] xfs_trans_alloc+0x38/0x50 [110926.972794] [] xfs_create+0x187/0x5a0 [110926.972796] [] xfs_vn_mknod+0x8a/0x1b0 [110926.972797] [] xfs_vn_create+0xe/0x10 [110926.972799] [] vfs_create+0x72/0xc0 [110926.972801] [] do_last.isra.69+0x80e/0xc80 [110926.972803] [] path_openat.isra.70+0xab/0x490 [110926.972805] [] do_filp_open+0x3d/0xa0 [110926.972807] [] do_sys_open+0xf9/0x1e0 [110926.972809] [] sys_open+0x1c/0x20 [110926.972811] [] system_call_fastpath+0x16/0x1b [110926.972813] INITIAL USE at: [110926.972814] [] __lock_acquire+0x2e9/0x1c00 [110926.972815] [] lock_acquire+0x55/0x70 [110926.972817] [] __sb_start_write+0xab/0x190 [110926.972819] [] xfs_trans_alloc+0x28/0x50 [110926.972820] [] xfs_create+0x187/0x5a0 [110926.972822] [] xfs_vn_mknod+0x8a/0x1b0 [110926.972824] [] xfs_vn_create+0xe/0x10 [110926.972826] [] vfs_create+0x72/0xc0 [110926.972827] [] do_last.isra.69+0x80e/0xc80 [110926.972829] [] path_openat.isra.70+0xab/0x490 [110926.972831] [] do_filp_open+0x3d/0xa0 [110926.972833] [] do_sys_open+0xf9/0x1e0 [110926.972835] [] sys_open+0x1c/0x20 [110926.972837] [] system_call_fastpath+0x16/0x1b [110926.972839] } [110926.972840] ... key at: [] xfs_fs_type+0x60/0x80 [110926.972842] ... acquired at: [110926.972843] [] check_usage_forwards+0x10b/0x140 [110926.972845] [] mark_lock+0x190/0x2f0 [110926.972846] [] __lock_acquire+0x57f/0x1c00 [110926.972848] [] lock_acquire+0x55/0x70 [110926.972850] [] __sb_start_write+0xab/0x190 [110926.972851] [] xfs_trans_alloc+0x28/0x50 [110926.972853] [] xfs_free_eofblocks+0x104/0x250 [110926.972855] [] xfs_inactive+0xa9/0x480 [110926.972856] [] xfs_fs_evict_inode+0x70/0x80 [110926.972858] [] evict+0xaf/0x1b0 [110926.972860] [] dispose_list+0x39/0x50 [110926.972861] [] prune_icache_sb+0x183/0x340 [110926.972863] [] prune_super+0xf3/0x1a0 [110926.972865] [] shrink_slab+0x11e/0x1f0 [110926.972866] [] kswapd+0x690/0xa10 [110926.972868] [] kthread+0xd6/0xe0 [110926.972870] [] ret_from_fork+0x7c/0xb0 [110926.972871] [110926.972872] [110926.972872] stack backtrace: [110926.972874] Pid: 725, comm: kswapd0 Not tainted 3.7.0-rc4 #1 [110926.972875] Call Trace: [110926.972878] [] print_irq_inversion_bug.part.37+0x1e8/0x1f0 [110926.972880] [] check_usage_forwards+0x10b/0x140 [110926.972883] [] mark_lock+0x190/0x2f0 [110926.972885] [] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 [110926.972887] [] __lock_acquire+0x57f/0x1c00 [110926.972889] [] ? xfs_iext_bno_to_ext+0x84/0x160 [110926.972892] [] ? xfs_bmbt_get_all+0x13/0x20 [110926.972895] [] ? xfs_bmap_search_multi_extents+0xa4/0x110 [110926.972897] [] lock_acquire+0x55/0x70 [110926.972899] [] ? xfs_trans_alloc+0x28/0x50 [110926.972901] [] __sb_start_write+0xab/0x190 [110926.972903] [] ? xfs_trans_alloc+0x28/0x50 [110926.972905] [] ? xfs_trans_alloc+0x28/0x50 [110926.972907] [] xfs_trans_alloc+0x28/0x50 [110926.972908] [] xfs_free_eofblocks+0x104/0x250 [110926.972910] [] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972912] [] xfs_inactive+0xa9/0x480 [110926.972914] [] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972916] [] xfs_fs_evict_inode+0x70/0x80 [110926.972918] [] evict+0xaf/0x1b0 [110926.972920] [] dispose_list+0x39/0x50 [110926.972922] [] prune_icache_sb+0x183/0x340 [110926.972924] [] prune_super+0xf3/0x1a0 [110926.972926] [] shrink_slab+0x11e/0x1f0 [110926.972928] [] kswapd+0x690/0xa10 [110926.972930] [] ? __init_waitqueue_head+0x60/0x60 [110926.972932] [] ? shrink_lruvec+0x540/0x540 [110926.972934] [] kthread+0xd6/0xe0 [110926.972936] [] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972938] [] ? flush_kthread_worker+0xe0/0xe0 [110926.972940] [] ret_from_fork+0x7c/0xb0 [110926.972942] [] ? flush_kthread_worker+0xe0/0xe0 > Nov 17 14:07:38 thoregon kernel: [66571.610863] > Nov 17 14:07:38 thoregon kernel: [66571.610869] > ========================================================= > Nov 17 14:07:38 thoregon kernel: [66571.610870] [ INFO: possible irq > lock inversion dependency detected ] > Nov 17 14:07:38 thoregon kernel: [66571.610873] 3.7.0-rc5 #1 Not tainted > Nov 17 14:07:38 thoregon kernel: [66571.610874] > --------------------------------------------------------- > Nov 17 14:07:38 thoregon kernel: [66571.610875] cc1/21330 just changed > the state of lock: > Nov 17 14:07:38 thoregon kernel: [66571.610877] > (sb_internal){.+.+.?}, at: [] > xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:38 thoregon kernel: [66571.610885] but this lock took > another, RECLAIM_FS-unsafe lock in the past: > Nov 17 14:07:38 thoregon kernel: [66571.610886] > (&(&ip->i_lock)->mr_lock/1){+.+.+.} > Nov 17 14:07:38 thoregon kernel: [66571.610886] > Nov 17 14:07:39 thoregon kernel: [66571.610886] and interrupts could > create inverse lock ordering between them. > Nov 17 14:07:39 thoregon kernel: [66571.610886] > Nov 17 14:07:39 thoregon kernel: [66571.610890] > Nov 17 14:07:39 thoregon kernel: [66571.610890] other info that might > help us debug this: > Nov 17 14:07:39 thoregon kernel: [66571.610891] Possible interrupt > unsafe locking scenario: > Nov 17 14:07:39 thoregon kernel: [66571.610891] > Nov 17 14:07:39 thoregon kernel: [66571.610892] CPU0 > CPU1 > Nov 17 14:07:39 thoregon kernel: [66571.610893] ---- > ---- > Nov 17 14:07:39 thoregon kernel: [66571.610894] > lock(&(&ip->i_lock)->mr_lock/1); > Nov 17 14:07:39 thoregon kernel: [66571.610896] > local_irq_disable(); > Nov 17 14:07:39 thoregon kernel: [66571.610897] > lock(sb_internal); > Nov 17 14:07:39 thoregon kernel: [66571.610898] > lock(&(&ip->i_lock)->mr_lock/1); > Nov 17 14:07:39 thoregon kernel: [66571.610900] > Nov 17 14:07:39 thoregon kernel: [66571.610901] lock(sb_internal); > Nov 17 14:07:39 thoregon kernel: [66571.610902] > Nov 17 14:07:39 thoregon kernel: [66571.610902] *** DEADLOCK *** > Nov 17 14:07:39 thoregon kernel: [66571.610902] > Nov 17 14:07:39 thoregon kernel: [66571.610904] 3 locks held by cc1/21330: > Nov 17 14:07:39 thoregon kernel: [66571.610905] #0: > (&mm->mmap_sem){++++++}, at: [] > __do_page_fault+0xfb/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.610910] #1: > (shrinker_rwsem){++++..}, at: [] > shrink_slab+0x32/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.610915] #2: > (&type->s_umount_key#20){++++.+}, at: [] > grab_super_passive+0x3e/0x90 > Nov 17 14:07:39 thoregon kernel: [66571.610921] > Nov 17 14:07:39 thoregon kernel: [66571.610921] the shortest > dependencies between 2nd lock and 1st lock: > Nov 17 14:07:39 thoregon kernel: [66571.610927] -> > (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 169649 { > Nov 17 14:07:39 thoregon kernel: [66571.610931] HARDIRQ-ON-W at: > Nov 17 14:07:39 thoregon kernel: [66571.610932] > [] __lock_acquire+0x631/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.610935] > [] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.610937] > [] down_write_nested+0x4a/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.610941] > [] xfs_ilock+0x84/0xb0 > Nov 17 14:07:39 thoregon kernel: [66571.610944] > [] xfs_create+0x1d4/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.610946] > [] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.610948] > [] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.610950] > [] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.610953] > [] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.610955] > [] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.610957] > [] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.610959] > [] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.610962] > [] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.610964] > [] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.610967] SOFTIRQ-ON-W at: > Nov 17 14:07:39 thoregon kernel: [66571.610968] > [] __lock_acquire+0x667/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.610970] > [] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.610972] > [] down_write_nested+0x4a/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.610974] > [] xfs_ilock+0x84/0xb0 > Nov 17 14:07:39 thoregon kernel: [66571.610976] > [] xfs_create+0x1d4/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.610977] > [] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.610979] > [] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.610981] > [] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.610983] > [] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.610985] > [] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.610987] > [] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.610989] > [] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.610991] > [] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.610993] > [] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.610995] RECLAIM_FS-ON-W at: > Nov 17 14:07:39 thoregon kernel: [66571.610996] > [] mark_held_locks+0x7e/0x130 > Nov 17 14:07:39 thoregon kernel: [66571.610998] > [] lockdep_trace_alloc+0x63/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.610999] > [] kmem_cache_alloc+0x35/0xe0 > Nov 17 14:07:39 thoregon kernel: [66571.611002] > [] vm_map_ram+0x271/0x770 > Nov 17 14:07:39 thoregon kernel: [66571.611004] > [] _xfs_buf_map_pages+0x46/0xe0 > Nov 17 14:07:39 thoregon kernel: [66571.611008] > [] xfs_buf_get_map+0x8a/0x130 > Nov 17 14:07:39 thoregon kernel: [66571.611009] > [] xfs_trans_get_buf_map+0xa9/0xd0 > Nov 17 14:07:39 thoregon kernel: [66571.611011] > [] xfs_ialloc_inode_init+0xcd/0x1d0 > Nov 17 14:07:39 thoregon kernel: [66571.611015] > [] xfs_ialloc_ag_alloc+0x18f/0x500 > Nov 17 14:07:39 thoregon kernel: [66571.611017] > [] xfs_dialloc+0x185/0x2a0 > Nov 17 14:07:39 thoregon kernel: [66571.611019] > [] xfs_ialloc+0x58/0x650 > Nov 17 14:07:39 thoregon kernel: [66571.611021] > [] xfs_dir_ialloc+0x65/0x270 > Nov 17 14:07:39 thoregon kernel: [66571.611023] > [] xfs_create+0x3ac/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611024] > [] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611026] > [] xfs_vn_mkdir+0x11/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611028] > [] vfs_mkdir+0x7f/0xd0 > Nov 17 14:07:39 thoregon kernel: [66571.611030] > [] sys_mkdirat+0x43/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611032] > [] sys_mkdir+0x14/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611034] > [] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611036] INITIAL USE at: > Nov 17 14:07:39 thoregon kernel: [66571.611037] > [] __lock_acquire+0x2e9/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611038] > [] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611040] > [] down_write_nested+0x4a/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611042] > [] xfs_ilock+0x84/0xb0 > Nov 17 14:07:39 thoregon kernel: [66571.611044] > [] xfs_create+0x1d4/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611046] > [] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611047] > [] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611049] > [] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611051] > [] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611053] > [] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611055] > [] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611057] > [] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611059] > [] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611061] > [] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611063] } > Nov 17 14:07:39 thoregon kernel: [66571.611064] ... key at: > [] __key.41357+0x1/0x8 > Nov 17 14:07:39 thoregon kernel: [66571.611066] ... acquired at: > Nov 17 14:07:39 thoregon kernel: [66571.611067] > [] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611069] > [] down_write_nested+0x4a/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611071] > [] xfs_ilock+0x84/0xb0 > Nov 17 14:07:39 thoregon kernel: [66571.611073] > [] xfs_create+0x1d4/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611074] > [] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611076] > [] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611078] > [] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611080] > [] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611082] > [] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611084] > [] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611086] > [] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611088] > [] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611090] > [] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611091] > Nov 17 14:07:39 thoregon kernel: [66571.611092] -> > (sb_internal){.+.+.?} ops: 1341531 { > Nov 17 14:07:39 thoregon kernel: [66571.611095] HARDIRQ-ON-R at: > Nov 17 14:07:39 thoregon kernel: [66571.611096] > [] __lock_acquire+0x50a/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611098] > [] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611100] > [] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611102] > [] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611104] > [] xfs_create+0x187/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611105] > [] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611107] > [] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611109] > [] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611111] > [] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611113] > [] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611115] > [] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611117] > [] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611119] > [] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611121] > [] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611123] SOFTIRQ-ON-R at: > Nov 17 14:07:39 thoregon kernel: [66571.611124] > [] __lock_acquire+0x667/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611126] > [] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611128] > [] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611130] > [] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611132] > [] xfs_create+0x187/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611133] > [] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611135] > [] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611137] > [] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611139] > [] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611141] > [] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611143] > [] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611145] > [] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611147] > [] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611149] > [] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611151] IN-RECLAIM_FS-R at: > Nov 17 14:07:39 thoregon kernel: [66571.611152] > [] __lock_acquire+0x57f/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611154] > [] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611156] > [] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611158] > [] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611159] > [] xfs_free_eofblocks+0x104/0x250 > Nov 17 14:07:39 thoregon kernel: [66571.611161] > [] xfs_inactive+0xa9/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611163] > [] xfs_fs_evict_inode+0x70/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611165] > [] evict+0xaf/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611168] > [] dispose_list+0x39/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611170] > [] prune_icache_sb+0x183/0x340 > Nov 17 14:07:39 thoregon kernel: [66571.611172] > [] prune_super+0xf3/0x1a0 > Nov 17 14:07:39 thoregon kernel: [66571.611173] > [] shrink_slab+0x11e/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.611175] > [] try_to_free_pages+0x21f/0x4e0 > Nov 17 14:07:39 thoregon kernel: [66571.611177] > [] __alloc_pages_nodemask+0x506/0x800 > Nov 17 14:07:39 thoregon kernel: [66571.611179] > [] handle_pte_fault+0x5ae/0x7a0 > Nov 17 14:07:39 thoregon kernel: [66571.611182] > [] handle_mm_fault+0x1f9/0x2a0 > Nov 17 14:07:39 thoregon kernel: [66571.611184] > [] __do_page_fault+0x16c/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611186] > [] do_page_fault+0x9/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611188] > [] page_fault+0x1f/0x30 > Nov 17 14:07:39 thoregon kernel: [66571.611190] RECLAIM_FS-ON-R at: > Nov 17 14:07:39 thoregon kernel: [66571.611191] > [] mark_held_locks+0x7e/0x130 > Nov 17 14:07:39 thoregon kernel: [66571.611193] > [] lockdep_trace_alloc+0x63/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611195] > [] kmem_cache_alloc+0x35/0xe0 > Nov 17 14:07:39 thoregon kernel: [66571.611197] > [] kmem_zone_alloc+0x5f/0xe0 > Nov 17 14:07:39 thoregon kernel: [66571.611198] > [] kmem_zone_zalloc+0x18/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611200] > [] _xfs_trans_alloc+0x32/0x90 > Nov 17 14:07:39 thoregon kernel: [66571.611202] > [] xfs_trans_alloc+0x38/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611204] > [] xfs_create+0x187/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611205] > [] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611207] > [] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611209] > [] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611211] > [] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611213] > [] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611215] > [] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611217] > [] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611219] > [] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611221] > [] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611223] INITIAL USE at: > Nov 17 14:07:39 thoregon kernel: [66571.611224] > [] __lock_acquire+0x2e9/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611225] > [] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611227] > [] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611229] > [] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611231] > [] xfs_create+0x187/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611232] > [] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611234] > [] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611236] > [] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611238] > [] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611240] > [] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611242] > [] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611244] > [] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611246] > [] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611248] > [] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611250] } > Nov 17 14:07:39 thoregon kernel: [66571.611251] ... key at: > [] xfs_fs_type+0x60/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611254] ... acquired at: > Nov 17 14:07:39 thoregon kernel: [66571.611254] > [] check_usage_forwards+0x10b/0x140 > Nov 17 14:07:39 thoregon kernel: [66571.611256] > [] mark_lock+0x190/0x2f0 > Nov 17 14:07:39 thoregon kernel: [66571.611258] > [] __lock_acquire+0x57f/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611260] > [] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611261] > [] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611263] > [] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611265] > [] xfs_free_eofblocks+0x104/0x250 > Nov 17 14:07:39 thoregon kernel: [66571.611266] > [] xfs_inactive+0xa9/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611268] > [] xfs_fs_evict_inode+0x70/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611270] > [] evict+0xaf/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611271] > [] dispose_list+0x39/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611273] > [] prune_icache_sb+0x183/0x340 > Nov 17 14:07:39 thoregon kernel: [66571.611275] > [] prune_super+0xf3/0x1a0 > Nov 17 14:07:39 thoregon kernel: [66571.611277] > [] shrink_slab+0x11e/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.611278] > [] try_to_free_pages+0x21f/0x4e0 > Nov 17 14:07:39 thoregon kernel: [66571.611280] > [] __alloc_pages_nodemask+0x506/0x800 > Nov 17 14:07:39 thoregon kernel: [66571.611282] > [] handle_pte_fault+0x5ae/0x7a0 > Nov 17 14:07:39 thoregon kernel: [66571.611284] > [] handle_mm_fault+0x1f9/0x2a0 > Nov 17 14:07:39 thoregon kernel: [66571.611286] > [] __do_page_fault+0x16c/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611287] > [] do_page_fault+0x9/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611289] > [