From: Nikolay Borisov <kernel@kyup.com>
Subject: Re: Lockup in wait_transaction_locked under memory pressure
Date: Thu, 25 Jun 2015 16:54:23 +0300
Message-ID: <558C080F.7040104@kyup.com>
References: <558BD447.1010503@kyup.com> <558BD507.9070002@kyup.com> <20150625112116.GC17237@dhcp22.suse.cz> <558BE96E.7080101@kyup.com> <20150625115025.GD17237@dhcp22.suse.cz> <558C023B.1040204@kyup.com> <20150625134558.GF17237@dhcp22.suse.cz>
Mime-Version: 1.0
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
Cc: linux-ext4@vger.kernel.org, Marian Marinov <mm@1h.com>
To: Michal Hocko <mhocko@suse.cz>
Return-path: <linux-ext4-owner@vger.kernel.org>
Received: from mail.siteground.com ([67.19.240.234]:53083 "EHLO
	mail.siteground.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1751636AbbFYNyZ (ORCPT
	<rfc822;linux-ext4@vger.kernel.org>); Thu, 25 Jun 2015 09:54:25 -0400
In-Reply-To: <20150625134558.GF17237@dhcp22.suse.cz>
Sender: linux-ext4-owner@vger.kernel.org
List-ID: <linux-ext4.vger.kernel.org>



On 06/25/2015 04:45 PM, Michal Hocko wrote:
> On Thu 25-06-15 16:29:31, Nikolay Borisov wrote:
>> I couldn't find any particular OOM which stands out, here how a typical 
>> one looks like: 
>>
>> alxc9 kernel: Memory cgroup out of memory (oom_kill_allocating_task): Kill process 9703 (postmaster) score 0 or sacrifice child
>> alxc9 kernel: Killed process 9703 (postmaster) total-vm:205800kB, anon-rss:1128kB, file-rss:0kB
>> alxc9 kernel: php invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
>> alxc9 kernel: php cpuset=cXXXX mems_allowed=0-1
>> alxc9 kernel: CPU: 12 PID: 1000 Comm: php Not tainted 4.0.0-clouder9+ #31
>> alxc9 kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.2 01/16/2015
>> alxc9 kernel: ffff8805d8440400 ffff88208d863c78 ffffffff815aaca3 ffff8820b947c750
>> alxc9 kernel: ffff8820b947c750 ffff88208d863cc8 ffffffff81123b2e ffff882000000000
>> alxc9 kernel: ffffffff000000d0 ffff8805d8440400 ffff8820b947c750 ffff8820b947cee0
>> alxc9 kernel: Call Trace:
>> alxc9 kernel: [<ffffffff815aaca3>] dump_stack+0x48/0x5d
>> alxc9 kernel: [<ffffffff81123b2e>] dump_header+0x8e/0xe0
>> alxc9 kernel: [<ffffffff81123fa7>] oom_kill_process+0x1d7/0x3c0
>> alxc9 kernel: [<ffffffff810d85a1>] ? cpuset_mems_allowed_intersects+0x21/0x30
>> alxc9 kernel: [<ffffffff8118c2bd>] mem_cgroup_out_of_memory+0x2bd/0x370
>> alxc9 kernel: [<ffffffff81189b37>] ? mem_cgroup_iter+0x177/0x390
>> alxc9 kernel: [<ffffffff8118c5d7>] mem_cgroup_oom_synchronize+0x267/0x290
>> alxc9 kernel: [<ffffffff811874f0>] ? mem_cgroup_wait_acct_move+0x140/0x140
>> alxc9 kernel: [<ffffffff81124504>] pagefault_out_of_memory+0x24/0xe0
>> alxc9 kernel: [<ffffffff81041927>] mm_fault_error+0x47/0x160
>> alxc9 kernel: [<ffffffff81041db0>] __do_page_fault+0x340/0x3c0
>> alxc9 kernel: [<ffffffff81041e6c>] do_page_fault+0x3c/0x90
>> alxc9 kernel: [<ffffffff815b1758>] page_fault+0x28/0x30
>> alxc9 kernel: Task in /lxc/cXXXX killed as a result of limit of /lxc/cXXXX
>> alxc9 kernel: memory: usage 2097152kB, limit 2097152kB, failcnt 7832302
>> alxc9 kernel: memory+swap: usage 2097152kB, limit 2621440kB, failcnt 0
>> alxc9 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
>> alxc9 kernel: Memory cgroup stats for /lxc/cXXXX: cache:22708KB rss:2074444KB rss_huge:0KB 
>> mapped_file:19960KB writeback:4KB swap:0KB inactive_anon:20364KB active_anon:2074896KB 
>> inactive_file:1236KB active_file:464KB unevictable:0KB
>>
>> The backtrace for other processes is exactly the same. 
> 
> OK, so this is not the global OOM killer. That wasn't clear from your
> previous description. It makes a difference because it means that the
> system is still healthy globaly and allocation requests will not loop
> for ever in the allocator. Memcg charging path will not get blocked
> until the OOM resolves and return ENOMEM when not called from the page
> fault path.

Yes, overall, the machine is healthy, only the processes for a
particular container would all go into uninterruptible sleep.

> 
> memcg oom killer ignores oom_kill_allocating_task so the victim might be
> different from the current task. That means the victim might get stuck
> behind a lock held by somebody else. If the ext4 journaling code depends
> on memcg charges and retry endlessly then the waiters would get stuck as
> well.

I've patched the cgroup OOM so that it takes into account the
oom_kill_allocating_task.

> 
> I can see some calls to find_or_create_page from fs/ext4/mballoc.c but
> AFAIU they are handling ENOMEM and lead to transaction abort - but I am
> not familiar with this code enough so somebody familiar with ext4 should
> double check that.
> 
> This all suggests that your lockup is caused by something else than OOM
> most probably.
>