Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753737Ab0GaRJw (ORCPT ); Sat, 31 Jul 2010 13:09:52 -0400 Received: from mga01.intel.com ([192.55.52.88]:6126 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752546Ab0GaRJt (ORCPT ); Sat, 31 Jul 2010 13:09:49 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.55,295,1278313200"; d="scan'208";a="591641940" Date: Sun, 1 Aug 2010 00:13:58 +0800 From: Wu Fengguang To: Andrew Morton , stable@kernel.org Cc: Rik van Riel , KOSAKI Motohiro , Mel Gorman , Christoph Hellwig , "linux-kernel@vger.kernel.org" , "linux-fsdevel@vger.kernel.org" , "linux-mm@kvack.org" , Dave Chinner , Chris Mason , Nick Piggin , Johannes Weiner , KAMEZAWA Hiroyuki , Andrea Arcangeli , Minchan Kim , Andreas Mohr , Bill Davidsen , Ben Gamari Subject: Re: [PATCH] vmscan: raise the bar to PAGEOUT_IO_SYNC stalls Message-ID: <20100731161358.GA5147@localhost> References: <20100728071705.GA22964@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100728071705.GA22964@localhost> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 73823 Lines: 1169 On Wed, Jul 28, 2010 at 03:17:05PM +0800, Wu Fengguang wrote: > Fix "system goes unresponsive under memory pressure and lots of > dirty/writeback pages" bug. > > http://lkml.org/lkml/2010/4/4/86 > > In the above thread, Andreas Mohr described that > > Invoking any command locked up for minutes (note that I'm > talking about attempted additional I/O to the _other_, > _unaffected_ main system HDD - such as loading some shell > binaries -, NOT the external SSD18M!!). > > This happens when the two conditions are both meet: > - under memory pressure > - writing heavily to a slow device > > OOM also happens in Andreas' system. The OOM trace shows that 3 > processes are stuck in wait_on_page_writeback() in the direct reclaim > path. One in do_fork() and the other two in unix_stream_sendmsg(). They > are blocked on this condition: > > (sc->order && priority < DEF_PRIORITY - 2) > > which was introduced in commit 78dc583d (vmscan: low order lumpy reclaim > also should use PAGEOUT_IO_SYNC) one year ago. That condition may be too > permissive. In Andreas' case, 512MB/1024 = 512KB. If the direct reclaim > for the order-1 fork() allocation runs into a range of 512KB > hard-to-reclaim LRU pages, it will be stalled. > > It's a severe problem in three ways. FYI I did some memory stress test and find there are much more order-1 (and higher) users than fork(). This means lots of running applications may stall on direct reclaim. Basically all of these slab caches will do high order allocations: $ grep -v '1 :' /proc/slabinfo slabinfo - version: 2.1 # name : tunables : slabdata nf_conntrack_expect 0 0 352 23 2 : tunables 0 0 0 : slabdata 0 0 0 nf_conntrack_ffffffff826e7340 57 102 472 17 2 : tunables 0 0 0 : slabdata 6 6 0 ip6_dst_cache 52 72 448 18 2 : tunables 0 0 0 : slabdata 4 4 0 ndisc_cache 32 32 512 16 2 : tunables 0 0 0 : slabdata 2 2 0 RAWv6 20 20 1600 20 8 : tunables 0 0 0 : slabdata 1 1 0 UDPLITEv6 0 0 1600 20 8 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 40 40 1600 20 8 : tunables 0 0 0 : slabdata 2 2 0 TCPv6 24 24 2688 12 8 : tunables 0 0 0 : slabdata 2 2 0 btrfs_transaction_cache 13 13 592 13 2 : tunables 0 0 0 : slabdata 1 1 0 btrfs_inode_cache 29 42 2256 14 8 : tunables 0 0 0 : slabdata 3 3 0 fat_inode_cache 0 0 1376 23 8 : tunables 0 0 0 : slabdata 0 0 0 reiser_inode_cache 276868 278300 1592 20 8 : tunables 0 0 0 : slabdata 13915 13915 0 kvm_vcpu 0 0 10608 3 8 : tunables 0 0 0 : slabdata 0 0 0 mqueue_inode_cache 19 19 1664 19 8 : tunables 0 0 0 : slabdata 1 1 0 fuse_request 0 0 760 21 4 : tunables 0 0 0 : slabdata 0 0 0 fuse_inode 0 0 1472 22 8 : tunables 0 0 0 : slabdata 0 0 0 nfs_write_data 38 38 832 19 4 : tunables 0 0 0 : slabdata 2 2 0 nfs_read_data 42 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 nfs_inode_cache 0 0 1648 19 8 : tunables 0 0 0 : slabdata 0 0 0 hugetlbfs_inode_cache 12 12 1312 12 4 : tunables 0 0 0 : slabdata 1 1 0 ext4_inode_cache 68415 68748 1912 17 8 : tunables 0 0 0 : slabdata 4044 4044 0 ext3_inode_cache 232870 237481 1664 19 8 : tunables 0 0 0 : slabdata 12499 12499 0 kioctx 0 0 704 23 4 : tunables 0 0 0 : slabdata 0 0 0 rpc_buffers 15 15 2176 15 8 : tunables 0 0 0 : slabdata 1 1 0 rpc_tasks 21 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 rpc_inode_cache 20 20 1600 20 8 : tunables 0 0 0 : slabdata 1 1 0 UNIX 205 240 1600 20 8 : tunables 0 0 0 : slabdata 12 12 0 UDP-Lite 0 0 1472 22 8 : tunables 0 0 0 : slabdata 0 0 0 xfrm_dst_cache 0 0 448 18 2 : tunables 0 0 0 : slabdata 0 0 0 ip_dst_cache 75 90 448 18 2 : tunables 0 0 0 : slabdata 5 5 0 arp_cache 32 32 512 16 2 : tunables 0 0 0 : slabdata 2 2 0 RAW 46 46 1408 23 8 : tunables 0 0 0 : slabdata 2 2 0 UDP 49 66 1472 22 8 : tunables 0 0 0 : slabdata 3 3 0 TCP 37 60 2560 12 8 : tunables 0 0 0 : slabdata 5 5 0 sgpool-128 16 21 4224 7 8 : tunables 0 0 0 : slabdata 3 3 0 sgpool-64 30 30 2176 15 8 : tunables 0 0 0 : slabdata 2 2 0 sgpool-32 28 28 1152 14 4 : tunables 0 0 0 : slabdata 2 2 0 sgpool-16 24 24 640 12 2 : tunables 0 0 0 : slabdata 2 2 0 sgpool-8 44 63 384 21 2 : tunables 0 0 0 : slabdata 3 3 0 blkdev_queue 30 30 2984 10 8 : tunables 0 0 0 : slabdata 3 3 0 blkdev_requests 48 60 408 20 2 : tunables 0 0 0 : slabdata 3 3 0 biovec-256 7 7 4224 7 8 : tunables 0 0 0 : slabdata 1 1 0 biovec-128 30 30 2176 15 8 : tunables 0 0 0 : slabdata 2 2 0 biovec-64 28 28 1152 14 4 : tunables 0 0 0 : slabdata 2 2 0 biovec-16 42 42 384 21 2 : tunables 0 0 0 : slabdata 2 2 0 bip-256 7 7 4288 7 8 : tunables 0 0 0 : slabdata 1 1 0 bip-128 0 0 2240 14 8 : tunables 0 0 0 : slabdata 0 0 0 bip-64 0 0 1216 13 4 : tunables 0 0 0 : slabdata 0 0 0 bip-16 0 0 448 18 2 : tunables 0 0 0 : slabdata 0 0 0 sock_inode_cache 283 322 1408 23 8 : tunables 0 0 0 : slabdata 14 14 0 skbuff_fclone_cache 28 28 576 14 2 : tunables 0 0 0 : slabdata 2 2 0 shmem_inode_cache 2632 2646 1560 21 8 : tunables 0 0 0 : slabdata 126 126 0 taskstats 40 40 400 20 2 : tunables 0 0 0 : slabdata 2 2 0 proc_inode_cache 848 888 1288 12 4 : tunables 0 0 0 : slabdata 74 74 0 bdev_cache 54 54 1728 18 8 : tunables 0 0 0 : slabdata 3 3 0 filp 3704 3948 384 21 2 : tunables 0 0 0 : slabdata 188 188 0 inode_cache 4546 4836 1240 13 4 : tunables 0 0 0 : slabdata 372 372 0 names_cache 14 14 4224 7 8 : tunables 0 0 0 : slabdata 2 2 0 radix_tree_node 37173 37817 624 13 2 : tunables 0 0 0 : slabdata 2909 2909 0 mm_struct 125 144 1280 12 4 : tunables 0 0 0 : slabdata 12 12 0 files_cache 129 144 896 18 4 : tunables 0 0 0 : slabdata 8 8 0 signal_cache 227 247 1216 13 4 : tunables 0 0 0 : slabdata 19 19 0 sighand_cache 227 238 2304 14 8 : tunables 0 0 0 : slabdata 17 17 0 task_xstate 210 228 640 12 2 : tunables 0 0 0 : slabdata 19 19 0 task_struct 310 318 9104 3 8 : tunables 0 0 0 : slabdata 106 106 0 idr_layer_cache 336 338 616 13 2 : tunables 0 0 0 : slabdata 26 26 0 kmalloc-8192 41 42 8264 3 8 : tunables 0 0 0 : slabdata 14 14 0 kmalloc-4096 546 560 4168 7 8 : tunables 0 0 0 : slabdata 80 80 0 kmalloc-2048 451 495 2120 15 8 : tunables 0 0 0 : slabdata 33 33 0 kmalloc-1024 928 986 1096 29 8 : tunables 0 0 0 : slabdata 34 34 0 kmalloc-512 5011 5012 584 14 2 : tunables 0 0 0 : slabdata 358 358 0 In particular the radix_tree_node slab cache turns up a lot as in the below trace. It happens when the process tries to swap in a page (this happens a lot!) and need to allocate a new radix tree node for the swapper_space. Imagine the user want to switch to an application which involves many swap-in reads, and many of them will stall in wait_on_page_writeback() for seconds waiting for some unrelated dirty page to be synced! [ 8603.802898] usemem D 00000001001f68a1 2224 3920 1 0x00000004 [ 8603.803028] ffff8800aac3b5c8 0000000000000006 00000000ffffffff 00000000001d55c0 [ 8603.803246] ffff8800b4d1a350 00000000001d55c0 ffff8800aac3bfd8 ffff8800aac3bfd8 [ 8603.803464] 00000000001d55c0 ffff8800b4d1a6b8 00000000001d55c0 00000000001d55c0 [ 8603.803681] Call Trace: [ 8603.803747] [] ? sync_page+0x0/0xb0 [ 8603.803820] [] io_schedule+0x96/0x110 [ 8603.803896] [] sync_page+0x6f/0xb0 [ 8603.803966] [] __wait_on_bit+0x8d/0xe0 [ 8603.804037] [] wait_on_page_bit+0x8f/0xa0 [ 8603.804108] [] ? wake_bit_function+0x0/0x70 [ 8603.804179] [] shrink_page_list+0x241/0xcf0 [ 8603.804250] [] ? finish_wait+0x7f/0xb0 [ 8603.804322] [] ? autoremove_wake_function+0x0/0x60 [ 8603.804393] [] shrink_inactive_list+0x27c/0x4b0 [ 8603.804465] [] ? mark_held_locks+0x90/0xd0 [ 8603.804536] [] shrink_zone+0x499/0x610 [ 8603.804608] [] do_try_to_free_pages+0x13b/0x540 [ 8603.804679] [] try_to_free_pages+0x99/0x180 [ 8603.804751] [] __alloc_pages_nodemask+0x6f0/0xb50 [ 8603.804824] [] alloc_pages_current+0xcb/0x160 [ 8603.804895] [] new_slab+0x274/0x3b0 [ 8603.804965] [] __slab_alloc+0x4bf/0x840 [ 8603.805036] [] ? radix_tree_preload+0x44/0xd0 [ 8603.805108] [] kmem_cache_alloc+0x22b/0x240 [ 8603.805179] [] ? radix_tree_preload+0x44/0xd0 [ 8603.805250] [] radix_tree_preload+0x44/0xd0 [ 8603.805322] [] read_swap_cache_async+0x83/0x1f0 [ 8603.805393] [] ? valid_swaphandles+0x1cf/0x240 [ 8603.805465] [] swapin_readahead+0xab/0xf0 [ 8603.805536] [] ? find_get_page+0x0/0x1c0 [ 8603.805606] [] handle_mm_fault+0xabd/0xe70 [ 8603.805677] [] ? do_page_fault+0x144/0x770 [ 8603.805749] [] do_page_fault+0x1d9/0x770 [ 8603.805820] [] ? error_sti+0x5/0x6 [ 8603.805889] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 8603.805962] [] page_fault+0x25/0x30 Thanks, Fengguang The more sysrq-t output after `shutdown`. The system is so stressed that it cannot be shutdown at all.. [ 8603.783334] cp D 00000001001fb070 1832 3528 3513 0x00000004 [ 8603.783464] ffff88009f505a08 0000000000000006 0000000000000000 00000000001d55c0 [ 8603.783681] ffff8800b9a3a350 00000000001d55c0 ffff88009f505fd8 ffff88009f505fd8 [ 8603.783898] 00000000001d55c0 ffff8800b9a3a6b8 00000000001d55c0 00000000001d55c0 [ 8603.784117] Call Trace: [ 8603.784183] [] ? inode_wait+0x0/0x20 [ 8603.784253] [] inode_wait+0x15/0x20 [ 8603.784323] [] __wait_on_bit+0x8d/0xe0 [ 8603.784395] [] inode_wait_for_writeback+0xb9/0xf0 [ 8603.784467] [] ? wake_bit_function+0x0/0x70 [ 8603.784538] [] ? check_for_new_grace_period+0x1ac/0x1e0 [ 8603.784612] [] writeback_single_inode+0x8f/0x3e0 [ 8603.784684] [] sync_inode+0x41/0x70 [ 8603.784753] [] nfs_wb_all+0x4a/0x60 [ 8603.784824] [] nfs_do_fsync+0x38/0x90 [ 8603.784894] [] nfs_file_flush+0x8a/0xd0 [ 8603.784965] [] filp_close+0x54/0xd0 [ 8603.785036] [] put_files_struct+0x111/0x210 [ 8603.785106] [] ? put_files_struct+0x40/0x210 [ 8603.785178] [] exit_files+0x6e/0x90 [ 8603.785248] [] do_exit+0x1e6/0xc40 [ 8603.785319] [] ? trace_hardirqs_off+0x1b/0x30 [ 8603.785390] [] ? _raw_spin_unlock_irq+0x4c/0x70 [ 8603.785461] [] do_group_exit+0x6d/0x130 [ 8603.785533] [] get_signal_to_deliver+0x3de/0x6a0 [ 8603.785606] [] do_signal+0x83/0xae0 [ 8603.785676] [] ? fsnotify+0x35e/0x3d0 [ 8603.785747] [] ? might_fault+0xd6/0xf0 [ 8603.785818] [] do_notify_resume+0x8c/0xc0 [ 8603.785889] [] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 8603.785961] [] int_signal+0x12/0x17 [ 8603.786030] flush-0:23 D 00000001001f6fae 2440 3529 2 0x00000000 [ 8603.786160] ffff88003deeba50 0000000000000006 ffff880000000000 00000000001d55c0 [ 8603.786379] ffff8800b9a3c6a0 00000000001d55c0 ffff88003deebfd8 ffff88003deebfd8 [ 8603.786597] 00000000001d55c0 ffff8800b9a3ca08 00000000001d55c0 00000000001d55c0 [ 8603.786814] Call Trace: [ 8603.786881] [] ? sync_page+0x0/0xb0 [ 8603.786951] [] io_schedule+0x96/0x110 [ 8603.787020] [] sync_page+0x6f/0xb0 [ 8603.787090] [] __wait_on_bit+0x8d/0xe0 [ 8603.787160] [] ? find_get_pages_tag+0x16d/0x280 [ 8603.787231] [] ? find_get_pages_tag+0x0/0x280 [ 8603.787302] [] wait_on_page_bit+0x8f/0xa0 [ 8603.787373] [] ? wake_bit_function+0x0/0x70 [ 8603.787444] [] ? pagevec_lookup_tag+0x2c/0x40 [ 8603.787515] [] filemap_fdatawait_range+0x1de/0x250 [ 8603.787587] [] filemap_fdatawait+0x31/0x40 [ 8603.787658] [] writeback_single_inode+0x2df/0x3e0 [ 8603.787730] [] generic_writeback_sb_inodes+0x106/0x230 [ 8603.787802] [] do_writeback_sb_inodes+0x3f/0x50 [ 8603.787874] [] wb_writeback+0x172/0x510 [ 8603.787944] [] ? trace_hardirqs_off+0x1b/0x30 [ 8603.788015] [] wb_do_writeback+0xce/0x290 [ 8603.788086] [] bdi_writeback_thread+0x1aa/0x3e0 [ 8603.788157] [] ? bdi_writeback_thread+0x0/0x3e0 [ 8603.788229] [] kthread+0xcd/0xe0 [ 8603.788299] [] kernel_thread_helper+0x4/0x10 [ 8603.788370] [] ? restore_args+0x0/0x30 [ 8603.788440] [] ? kthread+0x0/0xe0 [ 8603.788510] [] ? kernel_thread_helper+0x0/0x10 [ 8603.788580] mem-stress-te S 000000010015b965 4056 3605 3202 0x00000000 [ 8603.788710] ffff880067ccde78 0000000000000002 0000000000000000 00000000001d55c0 [ 8603.788928] ffff8800b9b446a0 00000000001d55c0 ffff880067ccdfd8 ffff880067ccdfd8 [ 8603.789146] 00000000001d55c0 ffff8800b9b44a08 00000000001d55c0 00000000001d55c0 [ 8603.789364] Call Trace: [ 8603.789429] [] do_wait+0x245/0x340 [ 8603.789500] [] sys_wait4+0x86/0x150 [ 8603.789570] [] ? child_wait_callback+0x0/0xb0 [ 8603.789641] [] system_call_fastpath+0x16/0x1b [ 8603.789712] cp D 00000001001fb07b 1904 3612 3605 0x00000004 [ 8603.789843] ffff880008c41a08 0000000000000006 0000000000000000 00000000001d55c0 [ 8603.790061] ffff8800b4d40000 00000000001d55c0 ffff880008c41fd8 ffff880008c41fd8 [ 8603.790280] 00000000001d55c0 ffff8800b4d40368 00000000001d55c0 00000000001d55c0 [ 8603.790498] Call Trace: [ 8603.790565] [] ? inode_wait+0x0/0x20 [ 8603.790636] [] inode_wait+0x15/0x20 [ 8603.790706] [] __wait_on_bit+0x8d/0xe0 [ 8603.790777] [] inode_wait_for_writeback+0xb9/0xf0 [ 8603.790849] [] ? wake_bit_function+0x0/0x70 [ 8603.790921] [] ? check_for_new_grace_period+0x1ac/0x1e0 [ 8603.790994] [] writeback_single_inode+0x8f/0x3e0 [ 8603.791065] [] sync_inode+0x41/0x70 [ 8603.791135] [] nfs_wb_all+0x4a/0x60 [ 8603.791206] [] nfs_do_fsync+0x38/0x90 [ 8603.791276] [] nfs_file_flush+0x8a/0xd0 [ 8603.791346] [] filp_close+0x54/0xd0 [ 8603.791988] [] put_files_struct+0x111/0x210 [ 8603.792059] [] ? put_files_struct+0x40/0x210 [ 8603.792130] [] exit_files+0x6e/0x90 [ 8603.792200] [] do_exit+0x1e6/0xc40 [ 8603.792269] [] ? trace_hardirqs_off+0x1b/0x30 [ 8603.792340] [] ? _raw_spin_unlock_irq+0x4c/0x70 [ 8603.792412] [] do_group_exit+0x6d/0x130 [ 8603.792482] [] get_signal_to_deliver+0x3de/0x6a0 [ 8603.792555] [] do_signal+0x83/0xae0 [ 8603.792625] [] ? fsnotify+0x35e/0x3d0 [ 8603.792695] [] do_notify_resume+0x8c/0xc0 [ 8603.792766] [] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 8603.792837] [] int_signal+0x12/0x17 [ 8603.792907] su S 000000010012dd7d 4136 3857 3417 0x00000000 [ 8603.793037] ffff880099a71e78 0000000000000002 0000000000000000 00000000001d55c0 [ 8603.793254] ffff880073db2350 00000000001d55c0 ffff880099a71fd8 ffff880099a71fd8 [ 8603.793473] 00000000001d55c0 ffff880073db26b8 00000000001d55c0 00000000001d55c0 [ 8603.793690] Call Trace: [ 8603.793755] [] do_wait+0x245/0x340 [ 8603.793826] [] sys_wait4+0x86/0x150 [ 8603.793895] [] ? child_wait_callback+0x0/0xb0 [ 8603.793967] [] system_call_fastpath+0x16/0x1b [ 8603.794037] zsh S 0000000100196400 3760 3860 3857 0x00000000 [ 8603.794166] ffff880016f37f38 0000000000000006 0000000000000000 00000000001d55c0 [ 8603.794383] ffff8800a6180000 00000000001d55c0 ffff880016f37fd8 ffff880016f37fd8 [ 8603.794601] 00000000001d55c0 ffff8800a6180368 00000000001d55c0 00000000001d55c0 [ 8603.794820] Call Trace: [ 8603.794886] [] sys_rt_sigsuspend+0xfd/0x150 [ 8603.794957] [] system_call_fastpath+0x16/0x1b [ 8603.795027] mem-stress-te S 00000001001fa536 4056 3907 3202 0x00000000 [ 8603.795157] ffff8800489b3e78 0000000000000006 0000000000000000 00000000001d55c0 [ 8603.795376] ffff880073db0000 00000000001d55c0 ffff8800489b3fd8 ffff8800489b3fd8 [ 8603.795593] 00000000001d55c0 ffff880073db0368 00000000001d55c0 00000000001d55c0 [ 8603.795811] Call Trace: [ 8603.795876] [] do_wait+0x245/0x340 [ 8603.795946] [] sys_wait4+0x86/0x150 [ 8603.796017] [] ? child_wait_callback+0x0/0xb0 [ 8603.796089] [] system_call_fastpath+0x16/0x1b [ 8603.796159] cp D 00000001001b3f13 2152 3917 3907 0x00000004 [ 8603.796289] ffff880033ae7338 0000000000000006 ffff880000000000 00000000001d55c0 [ 8603.796507] ffff880073db46a0 00000000001d55c0 ffff880033ae7fd8 ffff880033ae7fd8 [ 8603.796725] 00000000001d55c0 ffff880073db4a08 00000000001d55c0 00000000001d55c0 [ 8603.796943] Call Trace: [ 8603.797010] [] ? sync_page+0x0/0xb0 [ 8603.797079] [] io_schedule+0x96/0x110 [ 8603.797150] [] sync_page+0x6f/0xb0 [ 8603.797220] [] __wait_on_bit+0x8d/0xe0 [ 8603.797291] [] wait_on_page_bit+0x8f/0xa0 [ 8603.797362] [] ? wake_bit_function+0x0/0x70 [ 8603.797433] [] shrink_page_list+0x241/0xcf0 [ 8603.797505] [] ? finish_wait+0x7f/0xb0 [ 8603.797576] [] ? autoremove_wake_function+0x0/0x60 [ 8603.797647] [] shrink_inactive_list+0x27c/0x4b0 [ 8603.797719] [] shrink_zone+0x499/0x610 [ 8603.797790] [] do_try_to_free_pages+0x13b/0x540 [ 8603.797862] [] try_to_free_pages+0x99/0x180 [ 8603.797933] [] ? __lock_acquire+0x618/0x2a20 [ 8603.798004] [] __alloc_pages_nodemask+0x6f0/0xb50 [ 8603.798076] [] ? local_clock+0x9d/0xb0 [ 8603.798147] [] alloc_pages_current+0xcb/0x160 [ 8603.798218] [] new_slab+0x274/0x3b0 [ 8603.798288] [] __slab_alloc+0x4bf/0x840 [ 8603.798360] [] ? radix_tree_preload+0x44/0xd0 [ 8603.798431] [] kmem_cache_alloc+0x22b/0x240 [ 8603.798502] [] ? radix_tree_preload+0x44/0xd0 [ 8603.798573] [] radix_tree_preload+0x44/0xd0 [ 8603.798644] [] add_to_page_cache_locked+0x8a/0x210 [ 8603.798716] [] add_to_page_cache_lru+0x49/0xc0 [ 8603.798787] [] mpage_readpages+0xd5/0x190 [ 8603.798859] [] ? ext4_get_block+0x0/0x30 [ 8603.798929] [] ? ext4_get_block+0x0/0x30 [ 8603.799000] [] ext4_readpages+0x24/0x30 [ 8603.799071] [] __do_page_cache_readahead+0x1e7/0x300 [ 8603.799142] [] ? __do_page_cache_readahead+0xcb/0x300 [ 8603.799215] [] ? trace_hardirqs_off_caller+0x2d/0x1f0 [ 8603.799287] [] ra_submit+0x28/0x40 [ 8603.799357] [] ondemand_readahead+0xd7/0x3b0 [ 8603.799429] [] page_cache_async_readahead+0xbf/0x140 [ 8603.799501] [] ? find_get_page+0x113/0x1c0 [ 8603.799572] [] ? find_get_page+0x0/0x1c0 [ 8603.799643] [] generic_file_aio_read+0x67b/0xa50 [ 8603.799715] [] ? trace_hardirqs_off+0x1b/0x30 [ 8603.799787] [] ? local_clock+0x9d/0xb0 [ 8603.799858] [] do_sync_read+0xf0/0x140 [ 8603.799929] [] ? fsnotify+0x35e/0x3d0 [ 8603.799999] [] vfs_read+0xc1/0x240 [ 8603.800069] [] sys_read+0x66/0xb0 [ 8603.800139] [] system_call_fastpath+0x16/0x1b [ 8603.800209] cp D 00000001001fb07e 2184 3919 3907 0x00000004 [ 8603.800339] ffff88006b77fa08 0000000000000002 0000000000000000 00000000001d55c0 [ 8603.800557] ffff8800b988c6a0 00000000001d55c0 ffff88006b77ffd8 ffff88006b77ffd8 [ 8603.800776] 00000000001d55c0 ffff8800b988ca08 00000000001d55c0 00000000001d55c0 [ 8603.800993] Call Trace: [ 8603.801059] [] ? inode_wait+0x0/0x20 [ 8603.801129] [] inode_wait+0x15/0x20 [ 8603.801199] [] __wait_on_bit+0x8d/0xe0 [ 8603.801270] [] inode_wait_for_writeback+0xb9/0xf0 [ 8603.801342] [] ? wake_bit_function+0x0/0x70 [ 8603.801413] [] ? check_for_new_grace_period+0x1ac/0x1e0 [ 8603.801486] [] writeback_single_inode+0x8f/0x3e0 [ 8603.801557] [] sync_inode+0x41/0x70 [ 8603.801627] [] nfs_wb_all+0x4a/0x60 [ 8603.801697] [] nfs_do_fsync+0x38/0x90 [ 8603.801767] [] nfs_file_flush+0x8a/0xd0 [ 8603.801837] [] filp_close+0x54/0xd0 [ 8603.801907] [] put_files_struct+0x111/0x210 [ 8603.801978] [] ? put_files_struct+0x40/0x210 [ 8603.802048] [] exit_files+0x6e/0x90 [ 8603.802118] [] do_exit+0x1e6/0xc40 [ 8603.802188] [] ? trace_hardirqs_off+0x1b/0x30 [ 8603.802259] [] ? _raw_spin_unlock_irq+0x4c/0x70 [ 8603.802330] [] do_group_exit+0x6d/0x130 [ 8603.802401] [] get_signal_to_deliver+0x3de/0x6a0 [ 8603.802473] [] do_signal+0x83/0xae0 [ 8603.802543] [] ? fsnotify+0x35e/0x3d0 [ 8603.802614] [] ? might_fault+0xd6/0xf0 [ 8603.802685] [] do_notify_resume+0x8c/0xc0 [ 8603.802756] [] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 8603.802828] [] int_signal+0x12/0x17 [ 8603.802898] usemem D 00000001001f68a1 2224 3920 1 0x00000004 [ 8603.803028] ffff8800aac3b5c8 0000000000000006 00000000ffffffff 00000000001d55c0 [ 8603.803246] ffff8800b4d1a350 00000000001d55c0 ffff8800aac3bfd8 ffff8800aac3bfd8 [ 8603.803464] 00000000001d55c0 ffff8800b4d1a6b8 00000000001d55c0 00000000001d55c0 [ 8603.803681] Call Trace: [ 8603.803747] [] ? sync_page+0x0/0xb0 [ 8603.803820] [] io_schedule+0x96/0x110 [ 8603.803896] [] sync_page+0x6f/0xb0 [ 8603.803966] [] __wait_on_bit+0x8d/0xe0 [ 8603.804037] [] wait_on_page_bit+0x8f/0xa0 [ 8603.804108] [] ? wake_bit_function+0x0/0x70 [ 8603.804179] [] shrink_page_list+0x241/0xcf0 [ 8603.804250] [] ? finish_wait+0x7f/0xb0 [ 8603.804322] [] ? autoremove_wake_function+0x0/0x60 [ 8603.804393] [] shrink_inactive_list+0x27c/0x4b0 [ 8603.804465] [] ? mark_held_locks+0x90/0xd0 [ 8603.804536] [] shrink_zone+0x499/0x610 [ 8603.804608] [] do_try_to_free_pages+0x13b/0x540 [ 8603.804679] [] try_to_free_pages+0x99/0x180 [ 8603.804751] [] __alloc_pages_nodemask+0x6f0/0xb50 [ 8603.804824] [] alloc_pages_current+0xcb/0x160 [ 8603.804895] [] new_slab+0x274/0x3b0 [ 8603.804965] [] __slab_alloc+0x4bf/0x840 [ 8603.805036] [] ? radix_tree_preload+0x44/0xd0 [ 8603.805108] [] kmem_cache_alloc+0x22b/0x240 [ 8603.805179] [] ? radix_tree_preload+0x44/0xd0 [ 8603.805250] [] radix_tree_preload+0x44/0xd0 [ 8603.805322] [] read_swap_cache_async+0x83/0x1f0 [ 8603.805393] [] ? valid_swaphandles+0x1cf/0x240 [ 8603.805465] [] swapin_readahead+0xab/0xf0 [ 8603.805536] [] ? find_get_page+0x0/0x1c0 [ 8603.805606] [] handle_mm_fault+0xabd/0xe70 [ 8603.805677] [] ? do_page_fault+0x144/0x770 [ 8603.805749] [] do_page_fault+0x1d9/0x770 [ 8603.805820] [] ? error_sti+0x5/0x6 [ 8603.805889] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 8603.805962] [] page_fault+0x25/0x30 [ 8603.806032] usemem D 00000001001f66b7 2560 3921 1 0x00000004 [ 8603.806163] ffff88006c98b5c8 0000000000000006 0000000000000000 00000000001d55c0 [ 8603.806383] ffff880008d5c6a0 00000000001d55c0 ffff88006c98bfd8 ffff88006c98bfd8 [ 8603.806602] 00000000001d55c0 ffff880008d5ca08 00000000001d55c0 00000000001d55c0 [ 8603.807389] Call Trace: [ 8603.807455] [] ? sync_page+0x0/0xb0 [ 8603.807526] [] io_schedule+0x96/0x110 [ 8603.807596] [] sync_page+0x6f/0xb0 [ 8603.807665] [] __wait_on_bit+0x8d/0xe0 [ 8603.807736] [] wait_on_page_bit+0x8f/0xa0 [ 8603.807807] [] ? wake_bit_function+0x0/0x70 [ 8603.807878] [] shrink_page_list+0x241/0xcf0 [ 8603.807949] [] ? finish_wait+0x7f/0xb0 [ 8603.808019] [] ? autoremove_wake_function+0x0/0x60 [ 8603.808092] [] shrink_inactive_list+0x27c/0x4b0 [ 8603.808163] [] ? mark_held_locks+0x90/0xd0 [ 8603.808234] [] shrink_zone+0x499/0x610 [ 8603.808305] [] do_try_to_free_pages+0x13b/0x540 [ 8603.808377] [] try_to_free_pages+0x99/0x180 [ 8603.808448] [] __alloc_pages_nodemask+0x6f0/0xb50 [ 8603.808520] [] alloc_pages_current+0xcb/0x160 [ 8603.808591] [] new_slab+0x274/0x3b0 [ 8603.808662] [] __slab_alloc+0x4bf/0x840 [ 8603.808733] [] ? radix_tree_preload+0x44/0xd0 [ 8603.808803] [] kmem_cache_alloc+0x22b/0x240 [ 8603.808875] [] ? radix_tree_preload+0x44/0xd0 [ 8603.808946] [] radix_tree_preload+0x44/0xd0 [ 8603.809017] [] read_swap_cache_async+0x83/0x1f0 [ 8603.809089] [] ? valid_swaphandles+0x1cf/0x240 [ 8603.809161] [] swapin_readahead+0xab/0xf0 [ 8603.809231] [] ? find_get_page+0x0/0x1c0 [ 8603.809302] [] handle_mm_fault+0xabd/0xe70 [ 8603.809372] [] ? do_page_fault+0x144/0x770 [ 8603.809444] [] do_page_fault+0x1d9/0x770 [ 8603.809514] [] ? error_sti+0x5/0x6 [ 8603.809584] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 8603.809656] [] page_fault+0x25/0x30 [ 8603.809725] usemem D 00000001001f6ec4 2616 3922 1 0x00000004 [ 8603.809855] ffff8800954ff5c8 0000000000000006 00000000ffffffff 00000000001d55c0 [ 8603.810073] ffff880003540000 00000000001d55c0 ffff8800954fffd8 ffff8800954fffd8 [ 8603.810290] 00000000001d55c0 ffff880003540368 00000000001d55c0 00000000001d55c0 [ 8603.810507] Call Trace: [ 8603.810573] [] ? sync_page+0x0/0xb0 [ 8603.810643] [] io_schedule+0x96/0x110 [ 8603.810713] [] sync_page+0x6f/0xb0 [ 8603.810783] [] __wait_on_bit+0x8d/0xe0 [ 8603.810854] [] wait_on_page_bit+0x8f/0xa0 [ 8603.810924] [] ? wake_bit_function+0x0/0x70 [ 8603.810996] [] shrink_page_list+0x241/0xcf0 [ 8603.811066] [] ? delayacct_end+0xa8/0xd0 [ 8603.811137] [] ? __delayacct_blkio_end+0x64/0x70 [ 8603.811208] [] ? io_schedule_timeout+0xe0/0x130 [ 8603.811280] [] ? autoremove_wake_function+0x0/0x60 [ 8603.811352] [] shrink_inactive_list+0x27c/0x4b0 [ 8603.811424] [] ? mark_held_locks+0x90/0xd0 [ 8603.811495] [] shrink_zone+0x499/0x610 [ 8603.811566] [] do_try_to_free_pages+0x13b/0x540 [ 8603.811637] [] try_to_free_pages+0x99/0x180 [ 8603.811709] [] __alloc_pages_nodemask+0x6f0/0xb50 [ 8603.811781] [] alloc_pages_current+0xcb/0x160 [ 8603.811853] [] new_slab+0x274/0x3b0 [ 8603.811924] [] __slab_alloc+0x4bf/0x840 [ 8603.811994] [] ? radix_tree_preload+0x44/0xd0 [ 8603.812065] [] kmem_cache_alloc+0x22b/0x240 [ 8603.812137] [] ? radix_tree_preload+0x44/0xd0 [ 8603.812208] [] radix_tree_preload+0x44/0xd0 [ 8603.812281] [] read_swap_cache_async+0x83/0x1f0 [ 8603.812355] [] ? valid_swaphandles+0x1cf/0x240 [ 8603.812429] [] swapin_readahead+0xab/0xf0 [ 8603.812499] [] ? find_get_page+0x0/0x1c0 [ 8603.812570] [] handle_mm_fault+0xabd/0xe70 [ 8603.812641] [] ? do_page_fault+0x144/0x770 [ 8603.812712] [] do_page_fault+0x1d9/0x770 [ 8603.812783] [] ? error_sti+0x5/0x6 [ 8603.812853] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 8603.812925] [] page_fault+0x25/0x30 [ 8603.812995] usemem D 00000001001f6604 2616 3923 1 0x00000004 [ 8603.813125] ffff8800229d75c8 0000000000000006 0000000000000000 00000000001d55c0 [ 8603.813343] ffff880003542350 00000000001d55c0 ffff8800229d7fd8 ffff8800229d7fd8 [ 8603.813561] 00000000001d55c0 ffff8800035426b8 00000000001d55c0 00000000001d55c0 [ 8603.813779] Call Trace: [ 8603.813845] [] ? sync_page+0x0/0xb0 [ 8603.813915] [] io_schedule+0x96/0x110 [ 8603.813985] [] sync_page+0x6f/0xb0 [ 8603.814055] [] __wait_on_bit+0x8d/0xe0 [ 8603.814125] [] wait_on_page_bit+0x8f/0xa0 [ 8603.814197] [] ? wake_bit_function+0x0/0x70 [ 8603.814268] [] shrink_page_list+0x241/0xcf0 [ 8603.814340] [] ? finish_wait+0x7f/0xb0 [ 8603.814410] [] ? autoremove_wake_function+0x0/0x60 [ 8603.814482] [] shrink_inactive_list+0x27c/0x4b0 [ 8603.814554] [] ? mark_held_locks+0x90/0xd0 [ 8603.814625] [] shrink_zone+0x499/0x610 [ 8603.814695] [] do_try_to_free_pages+0x13b/0x540 [ 8603.814766] [] try_to_free_pages+0x99/0x180 [ 8603.814838] [] __alloc_pages_nodemask+0x6f0/0xb50 [ 8603.814910] [] alloc_pages_current+0xcb/0x160 [ 8603.814980] [] new_slab+0x274/0x3b0 [ 8603.815050] [] __slab_alloc+0x4bf/0x840 [ 8603.815121] [] ? radix_tree_preload+0x44/0xd0 [ 8603.815192] [] kmem_cache_alloc+0x22b/0x240 [ 8603.815263] [] ? radix_tree_preload+0x44/0xd0 [ 8603.815334] [] radix_tree_preload+0x44/0xd0 [ 8603.815405] [] read_swap_cache_async+0x83/0x1f0 [ 8603.815477] [] ? valid_swaphandles+0x1cf/0x240 [ 8603.815548] [] swapin_readahead+0xab/0xf0 [ 8603.815619] [] ? find_get_page+0x0/0x1c0 [ 8603.815689] [] handle_mm_fault+0xabd/0xe70 [ 8603.815760] [] ? do_page_fault+0x144/0x770 [ 8603.815831] [] do_page_fault+0x1d9/0x770 [ 8603.815902] [] ? error_sti+0x5/0x6 [ 8603.815971] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 8603.816043] [] page_fault+0x25/0x30 [ 8603.816112] usemem D 00000001001f663a 2616 3924 1 0x00000004 [ 8603.816242] ffff88000341d5c8 0000000000000002 00000000ffffffff 00000000001d55c0 [ 8603.816459] ffff8800035446a0 00000000001d55c0 ffff88000341dfd8 ffff88000341dfd8 [ 8603.816677] 00000000001d55c0 ffff880003544a08 00000000001d55c0 00000000001d55c0 [ 8603.816895] Call Trace: [ 8603.816961] [] ? sync_page+0x0/0xb0 [ 8603.817032] [] io_schedule+0x96/0x110 [ 8603.817101] [] sync_page+0x6f/0xb0 [ 8603.817171] [] __wait_on_bit+0x8d/0xe0 [ 8603.817242] [] wait_on_page_bit+0x8f/0xa0 [ 8603.817313] [] ? wake_bit_function+0x0/0x70 [ 8603.817384] [] shrink_page_list+0x241/0xcf0 [ 8603.817455] [] ? finish_wait+0x7f/0xb0 [ 8603.817526] [] ? autoremove_wake_function+0x0/0x60 [ 8603.817598] [] shrink_inactive_list+0x27c/0x4b0 [ 8603.817669] [] ? mark_held_locks+0x90/0xd0 [ 8603.817740] [] shrink_zone+0x499/0x610 [ 8603.817811] [] do_try_to_free_pages+0x13b/0x540 [ 8603.817883] [] try_to_free_pages+0x99/0x180 [ 8603.817954] [] __alloc_pages_nodemask+0x6f0/0xb50 [ 8603.818026] [] alloc_pages_current+0xcb/0x160 [ 8603.818097] [] new_slab+0x274/0x3b0 [ 8603.818167] [] __slab_alloc+0x4bf/0x840 [ 8603.818238] [] ? radix_tree_preload+0x44/0xd0 [ 8603.818309] [] kmem_cache_alloc+0x22b/0x240 [ 8603.818381] [] ? radix_tree_preload+0x44/0xd0 [ 8603.818452] [] radix_tree_preload+0x44/0xd0 [ 8603.818523] [] read_swap_cache_async+0x83/0x1f0 [ 8603.818595] [] ? valid_swaphandles+0x1cf/0x240 [ 8603.818666] [] swapin_readahead+0xab/0xf0 [ 8603.818737] [] ? find_get_page+0x0/0x1c0 [ 8603.818807] [] handle_mm_fault+0xabd/0xe70 [ 8603.818878] [] ? do_page_fault+0x144/0x770 [ 8603.818950] [] do_page_fault+0x1d9/0x770 [ 8603.819020] [] ? error_sti+0x5/0x6 [ 8603.819090] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 8603.819162] [] page_fault+0x25/0x30 [ 8603.819231] usemem D 00000001001eefd8 2144 3925 1 0x00000004 [ 8603.819362] ffff880015edd5c8 0000000000000006 0000000000000000 00000000001d55c0 [ 8603.819581] ffff8800034e0000 00000000001d55c0 ffff880015eddfd8 ffff880015eddfd8 [ 8603.819800] 00000000001d55c0 ffff8800034e0368 00000000001d55c0 00000000001d55c0 [ 8603.820018] Call Trace: [ 8603.820084] [] ? sync_page+0x0/0xb0 [ 8603.820155] [] io_schedule+0x96/0x110 [ 8603.820225] [] sync_page+0x6f/0xb0 [ 8603.820295] [] __wait_on_bit+0x8d/0xe0 [ 8603.820366] [] wait_on_page_bit+0x8f/0xa0 [ 8603.820437] [] ? wake_bit_function+0x0/0x70 [ 8603.820508] [] shrink_page_list+0x241/0xcf0 [ 8603.820580] [] ? finish_wait+0x7f/0xb0 [ 8603.820650] [] ? autoremove_wake_function+0x0/0x60 [ 8603.820723] [] shrink_inactive_list+0x27c/0x4b0 [ 8603.820794] [] ? mark_held_locks+0x90/0xd0 [ 8603.820866] [] shrink_zone+0x499/0x610 [ 8603.820936] [