Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755809Ab3H1WpI (ORCPT ); Wed, 28 Aug 2013 18:45:08 -0400 Received: from mail-pd0-f180.google.com ([209.85.192.180]:59782 "EHLO mail-pd0-f180.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752364Ab3H1WpF (ORCPT ); Wed, 28 Aug 2013 18:45:05 -0400 MIME-Version: 1.0 In-Reply-To: <521E5A69.30408@profihost.ag> References: <52132243.8050803@profihost.ag> <521323D1.1070503@profihost.ag> <20130821234715.GA4630@moria.home.lan> <5215A130.9080508@profihost.ag> <20130822054336.GA14935@kmo-pixel> <5215A8A8.90900@profihost.ag> <20130822060215.GB14935@kmo-pixel> <5215BEA4.6040101@profihost.ag> <521BAACB.4070003@profihost.ag> <20130828060521.GC8032@kmo-pixel>

<521E4D87.8020705@profihost.ag> <521E5A69.30408@profihost.ag> Date: Wed, 28 Aug 2013 15:45:03 -0700 Message-ID: Subject: Re: bcache: Fix a writeback performance regression From: kernel neophyte To: Stefan Priebe Cc: Kent Overstreet , Jens Axboe , "linux-kernel@vger.kernel.org" , "linux-bcache@vger.kernel.org" Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 70622 Lines: 1354 On Wed, Aug 28, 2013 at 1:15 PM, Stefan Priebe wrote: > sorry but that's completely wrong. > > please use branch bcache-for-3.10 > > http://evilpiepirate.org/git/linux-bcache.git/log/?h=bcache-for-3.10 nope :-( doesn't help I still see deadlocks: [ 437.123184] bcache: bch_cached_dev_attach() Caching sda3 as bcache0 on set a9ad97be-0918-427a-8d2a-db6e0a2c0e7a [ 1442.355992] INFO: task kworker/0:0:4 blocked for more than 120 seconds. [ 1442.356003] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1442.356009] kworker/0:0 D 0000000000000000 0 4 2 0x00000000 [ 1442.356023] Workqueue: bch_allocator bch_allocator_thread [ 1442.356027] ffff882fa6a85c98 0000000000000046 ffff882fa6a85cb8 ffffffff8107fd3d [ 1442.356033] ffff882fa6a64560 ffff882fa6a85fd8 ffff882fa6a85fd8 ffff882fa6a85fd8 [ 1442.356037] ffff882fa6a62e40 ffff882fa6a64560 ffff882fa5f85c80 ffff88211ce60db0 [ 1442.356042] Call Trace: [ 1442.356053] [] ? dequeue_task_fair+0x2cd/0x530 [ 1442.356063] [] schedule+0x29/0x70 [ 1442.356068] [] schedule_preempt_disabled+0xe/0x10 [ 1442.356073] [] __mutex_lock_slowpath+0x112/0x1b0 [ 1442.356078] [] mutex_lock+0x2a/0x50 [ 1442.356082] [] bch_allocator_thread+0x112/0xf30 [ 1442.356089] [] ? add_wait_queue+0x60/0x60 [ 1442.356096] [] process_one_work+0x174/0x490 [ 1442.356101] [] worker_thread+0x11b/0x370 [ 1442.356106] [] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1442.356110] [] kthread+0xc0/0xd0 [ 1442.356115] [] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356122] [] ret_from_fork+0x7c/0xb0 [ 1442.356126] [] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356132] INFO: task kworker/1:1:47 blocked for more than 120 seconds. [ 1442.356136] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1442.356140] kworker/1:1 D ffffffff8180c0c0 0 47 2 0x00000000 [ 1442.356150] Workqueue: bcache bch_journal [ 1442.356153] ffff882fa542d4f8 0000000000000046 ffff882fa542d518 0000000800000000 [ 1442.356157] ffff882fa5f85c80 ffff882fa542dfd8 ffff882fa542dfd8 ffff882fa542dfd8 [ 1442.356162] ffff882fa6439720 ffff882fa5f85c80 ffff882fa5f85c80 ffff88211ce60db0 [ 1442.356167] Call Trace: [ 1442.356172] [] schedule+0x29/0x70 [ 1442.356177] [] schedule_preempt_disabled+0xe/0x10 [ 1442.356182] [] __mutex_lock_slowpath+0x112/0x1b0 [ 1442.356187] [] mutex_lock+0x2a/0x50 [ 1442.356191] [] bch_mca_shrink+0x1c5/0x310 [ 1442.356198] [] ? prune_super+0x162/0x1b0 [ 1442.356206] [] shrink_slab+0x154/0x300 [ 1442.356213] [] ? resched_task+0x68/0x70 [ 1442.356218] [] ? check_preempt_curr+0x75/0xa0 [ 1442.356225] [] ? fragmentation_index+0x19/0x70 [ 1442.356231] [] do_try_to_free_pages+0x20f/0x4b0 [ 1442.356236] [] try_to_free_pages+0xe4/0x1a0 [ 1442.356245] [] __alloc_pages_nodemask+0x60c/0x990 [ 1442.356249] [] ? resched_task+0x68/0x70 [ 1442.356257] [] alloc_pages_current+0xba/0x170 [ 1442.356263] [] __get_free_pages+0xe/0x40 [ 1442.356268] [] mca_data_alloc+0x73/0x1d0 [ 1442.356272] [] mca_bucket_alloc+0x14a/0x1f0 [ 1442.356276] [] mca_alloc+0x320/0x470 [ 1442.356281] [] bch_btree_node_alloc+0x9b/0x200 [ 1442.356285] [] ? enqueue_entity+0x1d5/0xa90 [ 1442.356290] [] btree_split+0x119/0x770 [ 1442.356298] [] ? native_smp_send_reschedule+0x48/0x60 [ 1442.356303] [] ? __bch_bset_search+0x3fc/0x4a0 [ 1442.356307] [] ? enqueue_entity+0x1d5/0xa90 [ 1442.356312] [] bch_btree_insert_recurse+0x3eb/0x500 [ 1442.356316] [] bch_btree_insert_recurse+0x388/0x500 [ 1442.356321] [] bch_btree_insert+0x156/0x330 [ 1442.356327] [] bch_btree_insert_async+0x1a/0xe0 [ 1442.356332] [] bch_journal+0x45/0x260 [ 1442.356337] [] process_one_work+0x174/0x490 [ 1442.356342] [] worker_thread+0x11b/0x370 [ 1442.356347] [] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1442.356350] [] kthread+0xc0/0xd0 [ 1442.356355] [] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356360] [] ret_from_fork+0x7c/0xb0 [ 1442.356364] [] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356373] INFO: task kworker/u64:7:349 blocked for more than 120 seconds. [ 1442.356377] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1442.356382] kworker/u64:7 D ffffffff8180bde0 0 349 2 0x00000000 [ 1442.356390] Workqueue: bcache_writeback write_dirty_finish [ 1442.356392] ffff882f88e8ba90 0000000000000046 ffff882f88e8ba48 ffffffff81138789 [ 1442.356397] ffff882f88c8dc80 ffff882f88e8bfd8 ffff882f88e8bfd8 ffff882f88e8bfd8 [ 1442.356401] ffffffff81c10440 ffff882f88c8dc80 ffff882f88e8bbd3 ffff882f88c8dc80 [ 1442.356406] Call Trace: [ 1442.356410] [] ? zone_statistics+0x99/0xc0 [ 1442.356415] [] schedule+0x29/0x70 [ 1442.356421] [] rwsem_down_read_failed+0x9d/0xe5 [ 1442.356429] [] call_rwsem_down_read_failed+0x14/0x30 [ 1442.356434] [] ? down_read+0x24/0x2b [ 1442.356438] [] bch_btree_insert+0x110/0x330 [ 1442.356444] [] write_dirty_finish+0x27a/0x2c0 [ 1442.356450] [] process_one_work+0x174/0x490 [ 1442.356454] [] worker_thread+0x11b/0x370 [ 1442.356459] [] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1442.356463] [] kthread+0xc0/0xd0 [ 1442.356467] [] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356472] [] ret_from_fork+0x7c/0xb0 [ 1442.356476] [] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356485] INFO: task kworker/0:2:1967 blocked for more than 120 seconds. [ 1442.356489] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1442.356493] kworker/0:2 D 0000000000000000 0 1967 2 0x00000000 [ 1442.356500] Workqueue: bcache bch_journal [ 1442.356502] ffff882f890b7bf0 0000000000000046 0000000100000000 ffff882fbf273140 [ 1442.356507] ffff882f88d8ae40 ffff882f890b7fd8 ffff882f890b7fd8 ffff882f890b7fd8 [ 1442.356511] ffff882f924d4560 ffff882f88d8ae40 ffffffff81516b08 ffff882f88d8ae40 [ 1442.356516] Call Trace: [ 1442.356521] [] ? journal_write_endio+0x38/0x40 [ 1442.356526] [] schedule+0x29/0x70 [ 1442.356530] [] rwsem_down_read_failed+0x9d/0xe5 [ 1442.356536] [] ? internal_add_timer+0x20/0x50 [ 1442.356541] [] call_rwsem_down_read_failed+0x14/0x30 [ 1442.356546] [] ? down_read+0x24/0x2b [ 1442.356550] [] bch_btree_insert+0x110/0x330 [ 1442.356556] [] bch_btree_insert_async+0x1a/0xe0 [ 1442.356561] [] bch_journal+0x45/0x260 [ 1442.356566] [] process_one_work+0x174/0x490 [ 1442.356570] [] worker_thread+0x11b/0x370 [ 1442.356575] [] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1442.356579] [] kthread+0xc0/0xd0 [ 1442.356583] [] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356588] [] ret_from_fork+0x7c/0xb0 [ 1442.356592] [] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356595] INFO: task kworker/u64:0:1980 blocked for more than 120 seconds. [ 1442.356600] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1442.356604] kworker/u64:0 D ffffffff8180bde0 0 1980 2 0x00000000 [ 1442.356610] Workqueue: bch_btree_io btree_write_work [ 1442.356612] ffff8822b4447cc8 0000000000000046 ffff8822b4447ca8 ffffffff81514c30 [ 1442.356616] ffff882f924d4560 ffff8822b4447fd8 ffff8822b4447fd8 ffff8822b4447fd8 [ 1442.356621] ffff882fa6ab0000 ffff882f924d4560 0000000000000000 ffff882f924d4560 [ 1442.356625] Call Trace: [ 1442.356630] [] ? __btree_sort+0x160/0x250 [ 1442.356635] [] schedule+0x29/0x70 [ 1442.356640] [] rwsem_down_write_failed+0xf5/0x1a0 [ 1442.356645] [] call_rwsem_down_write_failed+0x13/0x20 [ 1442.356649] [] ? down_write+0x31/0x40 [ 1442.356654] [] btree_write_work+0x2f/0x60 [ 1442.356658] [] process_one_work+0x174/0x490 [ 1442.356663] [] worker_thread+0x11b/0x370 [ 1442.356668] [] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1442.356671] [] kthread+0xc0/0xd0 [ 1442.356676] [] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356680] [] ret_from_fork+0x7c/0xb0 [ 1442.356684] [] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356688] INFO: task iozone:1993 blocked for more than 120 seconds. [ 1442.356692] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1442.356696] iozone D 0000000000000000 0 1993 1470 0x00000000 [ 1442.356700] ffff881a3c981978 0000000000000082 ffff881a3c981938 ffffffff812fdc7a [ 1442.356705] ffff882f88d89720 ffff881a3c981fd8 ffff881a3c981fd8 ffff881a3c981fd8 [ 1442.356710] ffff882fa5f85c80 ffff882f88d89720 ffff881a3c981978 ffff882fbf2339f8 [ 1442.356714] Call Trace: [ 1442.356721] [] ? generic_make_request+0xca/0x100 [ 1442.356726] [] schedule+0x29/0x70 [ 1442.356731] [] io_schedule+0x8f/0xd0 [ 1442.356740] [] do_blockdev_direct_IO+0x1a7c/0x1fb0 [ 1442.356751] [] ? ext2_get_blocks+0xa60/0xa60 [ext2] [ 1442.356757] [] __blockdev_direct_IO+0x55/0x60 [ 1442.356763] [] ? ext2_get_blocks+0xa60/0xa60 [ext2] [ 1442.356769] [] ext2_direct_IO+0x79/0xe0 [ext2] [ 1442.356775] [] ? ext2_get_blocks+0xa60/0xa60 [ext2] [ 1442.356781] [] ? current_fs_time+0x16/0x60 [ 1442.356787] [] generic_file_direct_write+0xc6/0x180 [ 1442.356793] [] __generic_file_aio_write+0x2dd/0x3b0 [ 1442.356798] [] generic_file_aio_write+0x69/0xd0 [ 1442.356803] [] do_sync_write+0x7a/0xb0 [ 1442.356808] [] vfs_write+0xce/0x1e0 [ 1442.356812] [] SyS_write+0x52/0xa0 [ 1442.356818] [] system_call_fastpath+0x16/0x1b [ 1562.355919] INFO: task kworker/0:0:4 blocked for more than 120 seconds. [ 1562.355927] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1562.355935] kworker/0:0 D 0000000000000000 0 4 2 0x00000000 [ 1562.355945] Workqueue: bch_allocator bch_allocator_thread [ 1562.355949] ffff882fa6a85c98 0000000000000046 ffff882fa6a85cb8 ffffffff8107fd3d [ 1562.355954] ffff882fa6a64560 ffff882fa6a85fd8 ffff882fa6a85fd8 ffff882fa6a85fd8 [ 1562.355959] ffff882fa6a62e40 ffff882fa6a64560 ffff882fa5f85c80 ffff88211ce60db0 [ 1562.355963] Call Trace: [ 1562.355971] [] ? dequeue_task_fair+0x2cd/0x530 [ 1562.355977] [] schedule+0x29/0x70 [ 1562.355982] [] schedule_preempt_disabled+0xe/0x10 [ 1562.355987] [] __mutex_lock_slowpath+0x112/0x1b0 [ 1562.355992] [] mutex_lock+0x2a/0x50 [ 1562.355996] [] bch_allocator_thread+0x112/0xf30 [ 1562.356001] [] ? add_wait_queue+0x60/0x60 [ 1562.356006] [] process_one_work+0x174/0x490 [ 1562.356011] [] worker_thread+0x11b/0x370 [ 1562.356016] [] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1562.356020] [] kthread+0xc0/0xd0 [ 1562.356025] [] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356030] [] ret_from_fork+0x7c/0xb0 [ 1562.356035] [] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356040] INFO: task kworker/1:1:47 blocked for more than 120 seconds. [ 1562.356045] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1562.356051] kworker/1:1 D ffffffff8180c0c0 0 47 2 0x00000000 [ 1562.356059] Workqueue: bcache bch_journal [ 1562.356061] ffff882fa542d4f8 0000000000000046 ffff882fa542d518 0000000800000000 [ 1562.356066] ffff882fa5f85c80 ffff882fa542dfd8 ffff882fa542dfd8 ffff882fa542dfd8 [ 1562.356070] ffff882fa6439720 ffff882fa5f85c80 ffff882fa5f85c80 ffff88211ce60db0 [ 1562.356075] Call Trace: [ 1562.356080] [] schedule+0x29/0x70 [ 1562.356085] [] schedule_preempt_disabled+0xe/0x10 [ 1562.356089] [] __mutex_lock_slowpath+0x112/0x1b0 [ 1562.356094] [] mutex_lock+0x2a/0x50 [ 1562.356098] [] bch_mca_shrink+0x1c5/0x310 [ 1562.356104] [] ? prune_super+0x162/0x1b0 [ 1562.356109] [] shrink_slab+0x154/0x300 [ 1562.356115] [] ? resched_task+0x68/0x70 [ 1562.356120] [] ? check_preempt_curr+0x75/0xa0 [ 1562.356125] [] ? fragmentation_index+0x19/0x70 [ 1562.356130] [] do_try_to_free_pages+0x20f/0x4b0 [ 1562.356135] [] try_to_free_pages+0xe4/0x1a0 [ 1562.356142] [] __alloc_pages_nodemask+0x60c/0x990 [ 1562.356147] [] ? resched_task+0x68/0x70 [ 1562.356153] [] alloc_pages_current+0xba/0x170 [ 1562.356159] [] __get_free_pages+0xe/0x40 [ 1562.356163] [] mca_data_alloc+0x73/0x1d0 [ 1562.356168] [] mca_bucket_alloc+0x14a/0x1f0 [ 1562.356172] [] mca_alloc+0x320/0x470 [ 1562.356177] [] bch_btree_node_alloc+0x9b/0x200 [ 1562.356181] [] ? enqueue_entity+0x1d5/0xa90 [ 1562.356186] [] btree_split+0x119/0x770 [ 1562.356191] [] ? native_smp_send_reschedule+0x48/0x60 [ 1562.356196] [] ? __bch_bset_search+0x3fc/0x4a0 [ 1562.356200] [] ? enqueue_entity+0x1d5/0xa90 [ 1562.356205] [] bch_btree_insert_recurse+0x3eb/0x500 [ 1562.356209] [] bch_btree_insert_recurse+0x388/0x500 [ 1562.356214] [] bch_btree_insert+0x156/0x330 [ 1562.356220] [] bch_btree_insert_async+0x1a/0xe0 [ 1562.356225] [] bch_journal+0x45/0x260 [ 1562.356229] [] process_one_work+0x174/0x490 [ 1562.356234] [] worker_thread+0x11b/0x370 [ 1562.356239] [] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1562.356243] [] kthread+0xc0/0xd0 [ 1562.356247] [] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356252] [] ret_from_fork+0x7c/0xb0 [ 1562.356256] [] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356262] INFO: task kworker/u64:7:349 blocked for more than 120 seconds. [ 1562.356267] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1562.356273] kworker/u64:7 D ffffffff8180bde0 0 349 2 0x00000000 [ 1562.356280] Workqueue: bcache_writeback write_dirty_finish [ 1562.356282] ffff882f88e8ba90 0000000000000046 ffff882f88e8ba48 ffffffff81138789 [ 1562.356286] ffff882f88c8dc80 ffff882f88e8bfd8 ffff882f88e8bfd8 ffff882f88e8bfd8 [ 1562.356291] ffffffff81c10440 ffff882f88c8dc80 ffff882f88e8bbd3 ffff882f88c8dc80 [ 1562.356295] Call Trace: [ 1562.356300] [] ? zone_statistics+0x99/0xc0 [ 1562.356305] [] schedule+0x29/0x70 [ 1562.356310] [] rwsem_down_read_failed+0x9d/0xe5 [ 1562.356316] [] call_rwsem_down_read_failed+0x14/0x30 [ 1562.356320] [] ? down_read+0x24/0x2b [ 1562.356325] [] bch_btree_insert+0x110/0x330 [ 1562.356330] [] write_dirty_finish+0x27a/0x2c0 [ 1562.356336] [] process_one_work+0x174/0x490 [ 1562.356341] [] worker_thread+0x11b/0x370 [ 1562.356346] [] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1562.356350] [] kthread+0xc0/0xd0 [ 1562.356354] [] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356359] [] ret_from_fork+0x7c/0xb0 [ 1562.356363] [] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356373] INFO: task kworker/0:2:1967 blocked for more than 120 seconds. [ 1562.356377] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1562.356383] kworker/0:2 D 0000000000000000 0 1967 2 0x00000000 [ 1562.356389] Workqueue: bcache bch_journal [ 1562.356391] ffff882f890b7bf0 0000000000000046 0000000100000000 ffff882fbf273140 [ 1562.356396] ffff882f88d8ae40 ffff882f890b7fd8 ffff882f890b7fd8 ffff882f890b7fd8 [ 1562.356400] ffff882f924d4560 ffff882f88d8ae40 ffffffff81516b08 ffff882f88d8ae40 [ 1562.356405] Call Trace: [ 1562.356410] [] ? journal_write_endio+0x38/0x40 [ 1562.356414] [] schedule+0x29/0x70 [ 1562.356419] [] rwsem_down_read_failed+0x9d/0xe5 [ 1562.356424] [] ? internal_add_timer+0x20/0x50 [ 1562.356429] [] call_rwsem_down_read_failed+0x14/0x30 [ 1562.356433] [] ? down_read+0x24/0x2b [ 1562.356438] [] bch_btree_insert+0x110/0x330 [ 1562.356444] [] bch_btree_insert_async+0x1a/0xe0 [ 1562.356449] [] bch_journal+0x45/0x260 [ 1562.356453] [] process_one_work+0x174/0x490 [ 1562.356458] [] worker_thread+0x11b/0x370 [ 1562.356463] [] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1562.356466] [] kthread+0xc0/0xd0 [ 1562.356471] [] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356476] [] ret_from_fork+0x7c/0xb0 [ 1562.356480] [] ? flush_kthread_worker+0xb0/0xb0 -Neo > Stefan > > Am 28.08.2013 22:12, schrieb kernel neophyte: > >> On Wed, Aug 28, 2013 at 12:20 PM, Stefan Priebe >> wrote: >>> >>> Am 28.08.2013 20:47, schrieb kernel neophyte: >>> >>>> On Wed, Aug 28, 2013 at 11:38 AM, Stefan Priebe - Profihost AG >>>> wrote: >>>>> >>>>> >>>>> I don't had one for a few days. Which kernel so you use? >>>> >>>> >>>> >>>> 3.10 kernel with all of kent's stable patches and perf patches.. >>> >>> >>> >>> Which exact 3.10 version? Which patches on top? >> >> >> 3.10.0-rc7 >> with following patches: >> bcache: Fix a dumb CPU spinning bug in writeback >> bcache: Fix a flush/fua performance bug >> bcache: Fix a writeback performance regression >> bcache: FUA fixes >> >> I have also tried bcache perf branch i.e. >> (http://evilpiepirate.org/git/linux-bcache.git/log/?h=bcache-perf) >> >>> >>> >>>> and all defaults except for: >>>> >>>> echo writeback > /sys/block/bcache0/bcache/cache_mode >>>> echo 0 > /sys/block/bcache0/bcache/sequential_cutoff >>>> echo 0 > >>>> >>>> /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_read_threshold_us >>>> echo 0 > >>>> >>>> /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_write_threshold_us >>>> >>>> -Neo >>>> >>>>> >>>>> Stefan >>>>> >>>>> This mail was sent with my iPhone. >>>>> >>>>> Am 28.08.2013 um 20:12 schrieb kernel neophyte >>>>> : >>>>> >>>>>> On Tue, Aug 27, 2013 at 11:05 PM, Kent Overstreet >>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> On Tue, Aug 27, 2013 at 08:37:54PM -0700, kernel neophyte wrote: >>>>>>>> >>>>>>>> >>>>>>>> Hi Kent, >>>>>>>> >>>>>>>> I am still seeing deadlock: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Seeing blk_throtl_bio (what asshole misspells words in their function >>>>>>> names, so when you go to grep for them you grep for the wrong thing?) >>>>>>> in >>>>>>> the backtrace - is that what it takes to hit it? >>>>>> >>>>>> >>>>>> >>>>>> I don't think so, >>>>>> I had another deadlock: >>>>>> >>>>>> [ 1179.193990] bio: create slab at 2 >>>>>> [ 1179.194491] bcache: run_cache_set() invalidating existing data >>>>>> [ 1179.212308] bcache: register_cache() registered cache device rxd1 >>>>>> [ 1203.846220] bcache: register_bdev() registered backing device sda3 >>>>>> [ 1227.780088] bcache: bch_cached_dev_attach() Caching sda3 as bcache0 >>>>>> on set a079dda3-27fd-4b1c-a616-d46110bfae6c >>>>>> [ 2162.497950] INFO: task kworker/0:0:4 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.497961] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.497967] kworker/0:0 D 0000000000000000 0 4 2 >>>>>> 0x00000000 >>>>>> [ 2162.497982] Workqueue: events write_dirty_finish >>>>>> [ 2162.497986] ffff882fa6ac9b30 0000000000000046 ffffffff81511b8c >>>>>> ffffffff815137c0 >>>>>> [ 2162.497992] ffff882fa6a84cb0 ffff882fa6ac9fd8 ffff882fa6ac9fd8 >>>>>> ffff882fa6ac9fd8 >>>>>> [ 2162.497996] ffff882f78139990 ffff882fa6a84cb0 ffffffff815117a1 >>>>>> ffff882fa6a84cb0 >>>>>> [ 2162.498001] Call Trace: >>>>>> [ 2162.498010] [] ? >>>>>> bch_btree_map_nodes_recurse+0x6c/0x170 >>>>>> [ 2162.498015] [] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498019] [] ? bch_btree_node_get+0x71/0x280 >>>>>> [ 2162.498028] [] schedule+0x29/0x70 >>>>>> [ 2162.498034] [] rwsem_down_read_failed+0x9d/0xe5 >>>>>> [ 2162.498043] [] >>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>> [ 2162.498048] [] ? down_read+0x24/0x2b >>>>>> [ 2162.498052] [] __bch_btree_map_nodes+0xe5/0x1c0 >>>>>> [ 2162.498057] [] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498066] [] ? update_curr+0x141/0x1f0 >>>>>> [ 2162.498072] [] ? dirty_io_destructor+0xe/0x10 >>>>>> [ 2162.498076] [] bch_btree_insert+0xb4/0x120 >>>>>> [ 2162.498082] [] write_dirty_finish+0x1e3/0x270 >>>>>> [ 2162.498087] [] ? pick_next_task_fair+0x6b/0x150 >>>>>> [ 2162.498093] [] ? dequeue_task+0x93/0xb0 >>>>>> [ 2162.498100] [] process_one_work+0x174/0x490 >>>>>> [ 2162.498105] [] worker_thread+0x11b/0x370 >>>>>> [ 2162.498110] [] ? >>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>> [ 2162.498115] [] kthread+0xc0/0xd0 >>>>>> [ 2162.498120] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498127] [] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.498131] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498145] INFO: task bcache_allocato:1766 blocked for more than >>>>>> 120 seconds. >>>>>> [ 2162.498149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498154] bcache_allocato D 0000000000000001 0 1766 2 >>>>>> 0x00000000 >>>>>> [ 2162.498158] ffff8822630f3d88 0000000000000046 ffff8822630f3da8 >>>>>> ffffffff810808ad >>>>>> [ 2162.498163] ffff882f891de640 ffff8822630f3fd8 ffff8822630f3fd8 >>>>>> ffff8822630f3fd8 >>>>>> [ 2162.498167] ffff882f81a91990 ffff882f891de640 ffff882f88b33320 >>>>>> ffff8801fad20d98 >>>>>> [ 2162.498172] Call Trace: >>>>>> [ 2162.498177] [] ? dequeue_task_fair+0x2cd/0x530 >>>>>> [ 2162.498182] [] schedule+0x29/0x70 >>>>>> [ 2162.498187] [] >>>>>> schedule_preempt_disabled+0xe/0x10 >>>>>> [ 2162.498192] [] __mutex_lock_slowpath+0x112/0x1b0 >>>>>> [ 2162.498197] [] mutex_lock+0x2a/0x50 >>>>>> [ 2162.498202] [] bch_allocator_thread+0x10f/0xe20 >>>>>> [ 2162.498207] [] ? bch_bucket_add_unused+0xe0/0xe0 >>>>>> [ 2162.498211] [] kthread+0xc0/0xd0 >>>>>> [ 2162.498215] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498220] [] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.498224] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498228] INFO: task bcache_writebac:1771 blocked for more than >>>>>> 120 seconds. >>>>>> [ 2162.498233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498237] bcache_writebac D 0000000000000000 0 1771 2 >>>>>> 0x00000000 >>>>>> [ 2162.498241] ffff88197d2adc58 0000000000000046 ffff88197d2add48 >>>>>> ffffffff811269e2 >>>>>> [ 2162.498245] ffff882f891d8000 ffff88197d2adfd8 ffff88197d2adfd8 >>>>>> ffff88197d2adfd8 >>>>>> [ 2162.498250] ffff882f81a91990 ffff882f891d8000 ffff88197d2adc68 >>>>>> ffff882f89550ba0 >>>>>> [ 2162.498254] Call Trace: >>>>>> [ 2162.498264] [] ? >>>>>> __alloc_pages_nodemask+0x152/0x9b0 >>>>>> [ 2162.498269] [] schedule+0x29/0x70 >>>>>> [ 2162.498274] [] schedule_timeout+0x1e5/0x250 >>>>>> [ 2162.498280] [] ? try_to_wake_up+0x24f/0x2b0 >>>>>> [ 2162.498285] [] __down_common+0x8e/0xe3 >>>>>> [ 2162.498293] [] ? alloc_pages_current+0x90/0x170 >>>>>> [ 2162.498298] [] __down+0x1d/0x1f >>>>>> [ 2162.498304] [] down+0x41/0x50 >>>>>> [ 2162.498310] [] bch_writeback_thread+0x35d/0x7f0 >>>>>> [ 2162.498315] [] ? idle_balance+0xeb/0x150 >>>>>> [ 2162.498321] [] ? write_dirty+0xc0/0xc0 >>>>>> [ 2162.498325] [] kthread+0xc0/0xd0 >>>>>> [ 2162.498329] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498334] [] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.498338] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498342] INFO: task kworker/0:2:1789 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.498346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498351] kworker/0:2 D ffffffff81813a60 0 1789 2 >>>>>> 0x00000000 >>>>>> [ 2162.498357] Workqueue: events write_dirty_finish >>>>>> [ 2162.498360] ffff88277062fb30 0000000000000046 ffffffff81511b8c >>>>>> ffffffff815137c0 >>>>>> [ 2162.498364] ffff882fa6786640 ffff88277062ffd8 ffff88277062ffd8 >>>>>> ffff88277062ffd8 >>>>>> [ 2162.498368] ffffffff81c10440 ffff882fa6786640 ffffffff815117a1 >>>>>> ffff882fa6786640 >>>>>> [ 2162.498373] Call Trace: >>>>>> [ 2162.498378] [] ? >>>>>> bch_btree_map_nodes_recurse+0x6c/0x170 >>>>>> [ 2162.498382] [] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498387] [] ? bch_btree_node_get+0x71/0x280 >>>>>> [ 2162.498392] [] schedule+0x29/0x70 >>>>>> [ 2162.498397] [] rwsem_down_read_failed+0x9d/0xe5 >>>>>> [ 2162.498402] [] >>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>> [ 2162.498407] [] ? down_read+0x24/0x2b >>>>>> [ 2162.498411] [] __bch_btree_map_nodes+0xe5/0x1c0 >>>>>> [ 2162.498416] [] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498420] [] ? update_curr+0x141/0x1f0 >>>>>> [ 2162.498425] [] ? dirty_io_destructor+0xe/0x10 >>>>>> [ 2162.498430] [] bch_btree_insert+0xb4/0x120 >>>>>> [ 2162.498435] [] write_dirty_finish+0x1e3/0x270 >>>>>> [ 2162.498439] [] ? pick_next_task_fair+0x6b/0x150 >>>>>> [ 2162.498444] [] ? dequeue_task+0x93/0xb0 >>>>>> [ 2162.498449] [] process_one_work+0x174/0x490 >>>>>> [ 2162.498454] [] worker_thread+0x11b/0x370 >>>>>> [ 2162.498459] [] ? >>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>> [ 2162.498463] [] kthread+0xc0/0xd0 >>>>>> [ 2162.498467] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498472] [] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.498476] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498480] INFO: task iozone:2052 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.498484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498488] iozone D 0000000000000001 0 2052 1944 >>>>>> 0x00000000 >>>>>> [ 2162.498492] ffff882f7e2d3978 0000000000000082 ffff882f7e2d3938 >>>>>> ffffffff81301d7a >>>>>> [ 2162.498497] ffff882f8a116640 ffff882f7e2d3fd8 ffff882f7e2d3fd8 >>>>>> ffff882f7e2d3fd8 >>>>>> [ 2162.498501] ffff882f88b33320 ffff882f8a116640 ffff882f7e2d3978 >>>>>> ffff882fbf2339f8 >>>>>> [ 2162.498505] Call Trace: >>>>>> [ 2162.498511] [] ? generic_make_request+0xca/0x100 >>>>>> [ 2162.498517] [] schedule+0x29/0x70 >>>>>> [ 2162.498522] [] io_schedule+0x8f/0xd0 >>>>>> [ 2162.498530] [] >>>>>> do_blockdev_direct_IO+0x1a7c/0x1fb0 >>>>>> [ 2162.498539] [] ? ext2_get_blocks+0xa60/0xa60 >>>>>> [ext2] >>>>>> [ 2162.498546] [] __blockdev_direct_IO+0x55/0x60 >>>>>> [ 2162.498552] [] ? ext2_get_blocks+0xa60/0xa60 >>>>>> [ext2] >>>>>> [ 2162.498556] [] ? update_curr+0x141/0x1f0 >>>>>> [ 2162.498562] [] ext2_direct_IO+0x79/0xe0 [ext2] >>>>>> [ 2162.498567] [] ? ext2_get_blocks+0xa60/0xa60 >>>>>> [ext2] >>>>>> [ 2162.498572] [] ? set_next_entity+0xa5/0xc0 >>>>>> [ 2162.498579] [] ? current_fs_time+0x16/0x60 >>>>>> [ 2162.498585] [] >>>>>> generic_file_direct_write+0xc6/0x180 >>>>>> [ 2162.498590] [] >>>>>> __generic_file_aio_write+0x2dd/0x3b0 >>>>>> [ 2162.498595] [] ? mutex_unlock+0x14/0x20 >>>>>> [ 2162.498600] [] generic_file_aio_write+0x69/0xd0 >>>>>> [ 2162.498607] [] do_sync_write+0x7a/0xb0 >>>>>> [ 2162.498612] [] vfs_write+0xce/0x1e0 >>>>>> [ 2162.498617] [] ? >>>>>> generic_file_llseek_size+0x110/0x110 >>>>>> [ 2162.498621] [] SyS_write+0x52/0xa0 >>>>>> [ 2162.498627] [] system_call_fastpath+0x16/0x1b >>>>>> [ 2162.498630] INFO: task kworker/0:1:2293 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.498635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498639] kworker/0:1 D ffffffff81813a60 0 2293 2 >>>>>> 0x00000000 >>>>>> [ 2162.498645] Workqueue: events write_dirty_finish >>>>>> [ 2162.498647] ffff882f8a05bb30 0000000000000046 ffffffff81511b8c >>>>>> ffffffff815137c0 >>>>>> [ 2162.498652] ffff882f8a163320 ffff882f8a05bfd8 ffff882f8a05bfd8 >>>>>> ffff882f8a05bfd8 >>>>>> [ 2162.498656] ffffffff81c10440 ffff882f8a163320 ffffffff815117a1 >>>>>> ffff882f8a163320 >>>>>> [ 2162.498661] Call Trace: >>>>>> [ 2162.498665] [] ? >>>>>> bch_btree_map_nodes_recurse+0x6c/0x170 >>>>>> [ 2162.498669] [] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498674] [] ? bch_btree_node_get+0x71/0x280 >>>>>> [ 2162.498679] [] schedule+0x29/0x70 >>>>>> [ 2162.498684] [] rwsem_down_read_failed+0x9d/0xe5 >>>>>> [ 2162.498689] [] >>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>> [ 2162.498693] [] ? down_read+0x24/0x2b >>>>>> [ 2162.498698] [] __bch_btree_map_nodes+0xe5/0x1c0 >>>>>> [ 2162.498702] [] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498707] [] ? update_curr+0x141/0x1f0 >>>>>> [ 2162.498712] [] ? dirty_io_destructor+0xe/0x10 >>>>>> [ 2162.498716] [] bch_btree_insert+0xb4/0x120 >>>>>> [ 2162.498721] [] write_dirty_finish+0x1e3/0x270 >>>>>> [ 2162.498726] [] ? pick_next_task_fair+0x6b/0x150 >>>>>> [ 2162.498731] [] ? dequeue_task+0x93/0xb0 >>>>>> [ 2162.498735] [] process_one_work+0x174/0x490 >>>>>> [ 2162.498740] [] worker_thread+0x11b/0x370 >>>>>> [ 2162.498745] [] ? >>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>> [ 2162.498749] [] kthread+0xc0/0xd0 >>>>>> [ 2162.498753] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498758] [] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.498762] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498766] INFO: task kworker/0:3:2534 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.498772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498777] kworker/0:3 D ffffffff81813a60 0 2534 2 >>>>>> 0x00000000 >>>>>> [ 2162.498783] Workqueue: events write_dirty_finish >>>>>> [ 2162.498785] ffff88237d0bdb30 0000000000000046 ffffffff81511b8c >>>>>> ffffffff815137c0 >>>>>> [ 2162.498790] ffff882fa66fe640 ffff88237d0bdfd8 ffff88237d0bdfd8 >>>>>> ffff88237d0bdfd8 >>>>>> [ 2162.498794] ffffffff81c10440 ffff882fa66fe640 ffffffff815117a1 >>>>>> ffff882fa66fe640 >>>>>> [ 2162.498799] Call Trace: >>>>>> [ 2162.498803] [] ? >>>>>> bch_btree_map_nodes_recurse+0x6c/0x170 >>>>>> [ 2162.498807] [] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498812] [] ? bch_btree_node_get+0x71/0x280 >>>>>> [ 2162.498817] [] schedule+0x29/0x70 >>>>>> [ 2162.498822] [] rwsem_down_read_failed+0x9d/0xe5 >>>>>> [ 2162.498826] [] >>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>> [ 2162.498831] [] ? down_read+0x24/0x2b >>>>>> [ 2162.498836] [] __bch_btree_map_nodes+0xe5/0x1c0 >>>>>> [ 2162.498840] [] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498845] [] ? update_curr+0x141/0x1f0 >>>>>> [ 2162.498849] [] ? dirty_io_destructor+0xe/0x10 >>>>>> [ 2162.498854] [] bch_btree_insert+0xb4/0x120 >>>>>> [ 2162.498859] [] write_dirty_finish+0x1e3/0x270 >>>>>> [ 2162.498864] [] ? pick_next_task_fair+0x6b/0x150 >>>>>> [ 2162.498868] [] ? dequeue_task+0x93/0xb0 >>>>>> [ 2162.498873] [] process_one_work+0x174/0x490 >>>>>> [ 2162.498878] [] worker_thread+0x11b/0x370 >>>>>> [ 2162.498882] [] ? >>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>> [ 2162.498886] [] kthread+0xc0/0xd0 >>>>>> [ 2162.498890] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498896] [] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.498900] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498903] INFO: task kworker/0:4:2537 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.498908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498914] kworker/0:4 D 0000000000000000 0 2537 2 >>>>>> 0x00000000 >>>>>> [ 2162.498920] Workqueue: events write_dirty_finish >>>>>> [ 2162.498922] ffff882f76d8fb30 0000000000000046 ffffffff81511b8c >>>>>> ffffffff815137c0 >>>>>> [ 2162.498926] ffff882f88b30000 ffff882f76d8ffd8 ffff882f76d8ffd8 >>>>>> ffff882f76d8ffd8 >>>>>> [ 2162.498931] ffff882f78beb320 ffff882f88b30000 ffffffff815117a1 >>>>>> ffff882f88b30000 >>>>>> [ 2162.498935] Call Trace: >>>>>> [ 2162.498940] [] ? >>>>>> bch_btree_map_nodes_recurse+0x6c/0x170 >>>>>> [ 2162.498944] [] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498948] [] ? bch_btree_node_get+0x71/0x280 >>>>>> [ 2162.498953] [] schedule+0x29/0x70 >>>>>> [ 2162.498958] [] rwsem_down_read_failed+0x9d/0xe5 >>>>>> [ 2162.498963] [] >>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>> [ 2162.498968] [] ? down_read+0x24/0x2b >>>>>> [ 2162.498972] [] __bch_btree_map_nodes+0xe5/0x1c0 >>>>>> [ 2162.498977] [] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498981] [] ? update_curr+0x141/0x1f0 >>>>>> [ 2162.498986] [] ? dirty_io_destructor+0xe/0x10 >>>>>> [ 2162.498990] [] bch_btree_insert+0xb4/0x120 >>>>>> [ 2162.498995] [] write_dirty_finish+0x1e3/0x270 >>>>>> [ 2162.499000] [] ? pick_next_task_fair+0x6b/0x150 >>>>>> [ 2162.499004] [] ? dequeue_task+0x93/0xb0 >>>>>> [ 2162.499009] [] process_one_work+0x174/0x490 >>>>>> [ 2162.499014] [] worker_thread+0x11b/0x370 >>>>>> [ 2162.499019] [] ? >>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>> [ 2162.499023] [] kthread+0xc0/0xd0 >>>>>> [ 2162.499027] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.499032] [] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.499036] [] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.499039] INFO: task kworker/1:1:2544 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.499044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.499050] kworker/1:1 D ffffffff81813d40 0 2544 2 >>>>>> 0x00000000 >>>>>> [ 2162.499057] Workqueue: bcache bch_data_insert_keys >>>>>> [ 2162.499059] ffff882fa677b3f8 0000000000000046 ffff882fa677b418 >>>>>> ffff882f8a709000 >>>>>> [ 2162.499063] ffff882f88b33320 ffff882fa677bfd8 ffff882fa677bfd8 >>>>>> ffff882fa677bfd8 >>>>>> [ 2162.499068] ffff882fa6471990 ffff882f88b33320 ffff882f88b33320 >>>>>> ffff8801fad20d98 >>>>>> [ 2162.499072] Call Trace: >>>>>> [ 2162.499077] [] schedule+0x29/0x70 >>>>>> [ 2162.499082] [] >>>>>> schedule_preempt_disabled+0xe/0x10 >>>>>> [ 2162.499087] [] __mutex_lock_slowpath+0x112/0x1b0 >>>>>> [ 2162.499092] [] mutex_lock+0x2a/0x50 >>>>>> [ 2162.499096] [] bch_mca_shrink+0x1b5/0x2f0 >>>>>> [ 2162.499102] [] ? prune_super+0x162/0x1b0 >>>>>> [ 2162.499109] [] shrink_slab+0x154/0x300 >>>>>> [ 2162.499113] [] ? resched_task+0x68/0x70 >>>>>> [ 2162.499118] [] ? check_preempt_curr+0x75/0xa0 >>>>>> [ 2162.499126] [] ? fragmentation_index+0x19/0x70 >>>>>> [ 2162.499131] [] do_try_to_free_pages+0x20f/0x4b0 >>>>>> [ 2162.499137] [] try_to_free_pages+0xe4/0x1a0 >>>>>> [ 2162.499143] [] >>>>>> __alloc_pages_nodemask+0x60c/0x9b0 >>>>>> [ 2162.499149] [] alloc_pages_current+0xba/0x170 >>>>>> [ 2162.499155] [] __get_free_pages+0xe/0x40 >>>>>> [ 2162.499159] [] mca_data_alloc+0x73/0x1d0 >>>>>> [ 2162.499163] [] mca_alloc+0x277/0x470 >>>>>> [ 2162.499168] [] bch_btree_node_alloc+0x8c/0x1c0 >>>>>> [ 2162.499173] [] btree_split+0x110/0x5c0 >>>>>> [ 2162.499179] [] ? >>>>>> scsi_pool_alloc_command+0x33/0x80 >>>>>> [ 2162.499184] [] ? bch_keylist_pop_front+0x47/0x50 >>>>>> [ 2162.499188] [] ? >>>>>> bch_btree_insert_keys+0x56/0x250 >>>>>> [ 2162.499193] [] ? update_group_power+0x13f/0x220 >>>>>> [ 2162.499198] [] bch_btree_insert_node+0xb2/0x2f0 >>>>>> [ 2162.499202] [] btree_insert_fn+0x28/0x50 >>>>>> [ 2162.499207] [] >>>>>> bch_btree_map_nodes_recurse+0x6c/0x170 >>>>>> [ 2162.499211] [] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.499216] [] ? down_write+0x16/0x40 >>>>>> [ 2162.499221] [] ? bch_btree_node_get+0x71/0x280 >>>>>> [ 2162.499225] [