Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753265AbbFYKPD (ORCPT ); Thu, 25 Jun 2015 06:15:03 -0400 Received: from mail.siteground.com ([67.19.240.234]:50602 "EHLO mail.siteground.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754056AbbFYKNd (ORCPT ); Thu, 25 Jun 2015 06:13:33 -0400 Message-ID: <558BD447.1010503@kyup.com> Date: Thu, 25 Jun 2015 13:13:27 +0300 From: Nikolay Borisov Organization: Kyup Ltd. User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: tytso@mit.edu CC: "linux-ext4"@vger.kernel.org, linux-kernel@vger.kernel.org, Michal Hocko , Marian Marinov Subject: Lockup in wait_transaction_locked under memory pressure Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - mail.siteground.com X-AntiAbuse: Original Domain - vger.kernel.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - kyup.com X-Get-Message-Sender-Via: mail.siteground.com: none X-Source: X-Source-Args: X-Source-Dir: Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 5164 Lines: 101 Hello, On a fairly busy server, running LXC I'm observing that sometimes the processes for a particular container lock up by going into D (uninterruptible sleep) state. Obtaining backtraces for those processes one thing which stands out is that they are all blocked in wait_transaction_locked (part of JBD2). This occurs sporadically when the particular container is under memory pressure and a process is selected by OOM for killing. I'm running kernel 4.0.0 and oom_kill_allocating_task is enabled. Here are backtraces from multiple such processes: alxc9 kernel: nginx D ffff8820a90b39a8 11496 9331 30627 0x00000004 alxc9 kernel: ffff8820a90b39a8 ffff881ff284f010 ffff88396c6d1e90 0000000000000282 alxc9 kernel: ffff8820a90b0010 ffff883ff12f3870 ffff883ff12f3800 0000000000027df1 alxc9 kernel: ffff880413c08000 ffff8820a90b39c8 ffffffff815ab76e ffff883ff12f3870 alxc9 kernel: Call Trace: alxc9 kernel: [] schedule+0x3e/0x90 alxc9 kernel: [] wait_transaction_locked+0x85/0xc0 alxc9 kernel: [] ? woken_wake_function+0x20/0x20 alxc9 kernel: [] ? __schedule+0x39a/0x870 alxc9 kernel: [] add_transaction_credits+0xf0/0x250 alxc9 kernel: [] ? schedule+0x3e/0x90 alxc9 kernel: [] ? schedule_timeout+0x165/0x1f0 alxc9 kernel: [] start_this_handle+0x184/0x420 alxc9 kernel: [] ? __delayacct_blkio_end+0x30/0x50 alxc9 kernel: [] ? kmem_cache_alloc+0xee/0x1c0 alxc9 kernel: [] jbd2__journal_start+0x100/0x200 alxc9 kernel: [] ? ext4_dirty_inode+0x3c/0x80 alxc9 kernel: [] __ext4_journal_start_sb+0x79/0x100 alxc9 kernel: [] ext4_dirty_inode+0x3c/0x80 alxc9 kernel: [] __mark_inode_dirty+0x173/0x400 alxc9 kernel: [] generic_update_time+0x85/0xd0 alxc9 kernel: [] ? filemap_map_pages+0x1ca/0x210 alxc9 kernel: [] file_update_time+0xb2/0x110 alxc9 kernel: [] __generic_file_write_iter+0x172/0x3a0 alxc9 kernel: [] ext4_file_write_iter+0x134/0x460 alxc9 kernel: [] ? update_rmtp+0x80/0x80 alxc9 kernel: [] new_sync_write+0x97/0xc0 alxc9 kernel: [] vfs_write+0xce/0x180 alxc9 kernel: [] SyS_write+0x5a/0xd0 alxc9 kernel: [] system_call_fastpath+0x12/0x17 alxc9 kernel: mysqld D ffff8821352638d8 11936 5176 30627 0x00000006 alxc9 kernel: ffff8821352638d8 ffff881ff2848000 ffff8812d3d28a30 0000000000000286 alxc9 kernel: ffff882135260010 ffff883ff12f3870 ffff883ff12f3800 0000000000027df1 alxc9 kernel: ffff880413c08000 ffff8821352638f8 ffffffff815ab76e ffff883ff12f3870 alxc9 kernel: Call Trace: alxc9 kernel: [] schedule+0x3e/0x90 alxc9 kernel: [] wait_transaction_locked+0x85/0xc0 alxc9 kernel: [] ? woken_wake_function+0x20/0x20 alxc9 kernel: [] add_transaction_credits+0xf0/0x250 alxc9 kernel: [] start_this_handle+0x184/0x420 alxc9 kernel: [] ? kmem_cache_alloc+0xee/0x1c0 alxc9 kernel: [] jbd2__journal_start+0x100/0x200 alxc9 kernel: [] ? ext4_evict_inode+0x190/0x490 alxc9 kernel: [] __ext4_journal_start_sb+0x79/0x100 alxc9 kernel: [] ext4_evict_inode+0x190/0x490 alxc9 kernel: [] evict+0xb8/0x1a0 alxc9 kernel: [] iput_final+0xf6/0x190 alxc9 kernel: [] iput+0xa0/0xe0 alxc9 kernel: [] dentry_iput+0xa8/0xf0 alxc9 kernel: [] __dentry_kill+0x85/0x130 alxc9 kernel: [] dput+0x1bc/0x220 alxc9 kernel: [] __fput+0x144/0x200 alxc9 kernel: [] ____fput+0xe/0x10 alxc9 kernel: [] task_work_run+0xd5/0x120 alxc9 kernel: [] do_exit+0x1b9/0x560 alxc9 kernel: [] ? hrtimer_cancel+0x22/0x30 alxc9 kernel: [] do_group_exit+0x56/0x100 alxc9 kernel: [] get_signal+0x237/0x530 alxc9 kernel: [] do_signal+0x25/0x130 alxc9 kernel: [] ? rcu_eqs_exit+0x79/0xb0 alxc9 kernel: [] ? rcu_user_exit+0x13/0x20 alxc9 kernel: [] do_notify_resume+0x78/0xb0 alxc9 kernel: [] int_signal+0x12/0x17 My hypotheses is that the OOM is killing a process while its performing a write to a transaction without it having a chance to complete it which leaves all other processes waiting to be woken up, which naturally never happens. I wonder whether such a failure scenario is even possible? If not then what could possibly force a transaction to stall for hours? Regards, Nikolay -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/