Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754711Ab3HEU4S (ORCPT ); Mon, 5 Aug 2013 16:56:18 -0400 Received: from zene.cmpxchg.org ([85.214.230.12]:51952 "EHLO zene.cmpxchg.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753013Ab3HEU4Q (ORCPT ); Mon, 5 Aug 2013 16:56:16 -0400 Date: Mon, 5 Aug 2013 16:56:04 -0400 From: Johannes Weiner To: Michal Hocko Cc: Andrew Morton , David Rientjes , KAMEZAWA Hiroyuki , azurIt , KOSAKI Motohiro , linux-mm@kvack.org, cgroups@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [patch 7/7] mm: memcg: do not trap chargers with full callstack on OOM Message-ID: <20130805205604.GC715@cmpxchg.org> References: <1375549200-19110-1-git-send-email-hannes@cmpxchg.org> <1375549200-19110-8-git-send-email-hannes@cmpxchg.org> <20130805095429.GJ10146@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130805095429.GJ10146@dhcp22.suse.cz> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4918 Lines: 94 On Mon, Aug 05, 2013 at 11:54:29AM +0200, Michal Hocko wrote: > On Sat 03-08-13 13:00:00, Johannes Weiner wrote: > > The memcg OOM handling is incredibly fragile and can deadlock. When a > > task fails to charge memory, it invokes the OOM killer and loops right > > there in the charge code until it succeeds. Comparably, any other > > task that enters the charge path at this point will go to a waitqueue > > right then and there and sleep until the OOM situation is resolved. > > The problem is that these tasks may hold filesystem locks and the > > mmap_sem; locks that the selected OOM victim may need to exit. > > > > For example, in one reported case, the task invoking the OOM killer > > was about to charge a page cache page during a write(), which holds > > the i_mutex. The OOM killer selected a task that was just entering > > truncate() and trying to acquire the i_mutex: > > > > OOM invoking task: > > [] mem_cgroup_handle_oom+0x241/0x3b0 > > [] T.1146+0x5ab/0x5c0 > > [] mem_cgroup_cache_charge+0xbe/0xe0 > > [] add_to_page_cache_locked+0x4c/0x140 > > [] add_to_page_cache_lru+0x22/0x50 > > [] grab_cache_page_write_begin+0x8b/0xe0 > > [] ext3_write_begin+0x88/0x270 > > [] generic_file_buffered_write+0x116/0x290 > > [] __generic_file_aio_write+0x27c/0x480 > > [] generic_file_aio_write+0x76/0xf0 # takes ->i_mutex > > [] do_sync_write+0xea/0x130 > > [] vfs_write+0xf3/0x1f0 > > [] sys_write+0x51/0x90 > > [] system_call_fastpath+0x18/0x1d > > [] 0xffffffffffffffff > > > > OOM kill victim: > > [] do_truncate+0x58/0xa0 # takes i_mutex > > [] do_last+0x250/0xa30 > > [] path_openat+0xd7/0x440 > > [] do_filp_open+0x49/0xa0 > > [] do_sys_open+0x106/0x240 > > [] sys_open+0x20/0x30 > > [] system_call_fastpath+0x18/0x1d > > [] 0xffffffffffffffff > > > > The OOM handling task will retry the charge indefinitely while the OOM > > killed task is not releasing any resources. > > > > A similar scenario can happen when the kernel OOM killer for a memcg > > is disabled and a userspace task is in charge of resolving OOM > > situations. In this case, ALL tasks that enter the OOM path will be > > made to sleep on the OOM waitqueue and wait for userspace to free > > resources or increase the group's limit. But a userspace OOM handler > > is prone to deadlock itself on the locks held by the waiting tasks. > > For example one of the sleeping tasks may be stuck in a brk() call > > with the mmap_sem held for writing but the userspace handler, in order > > to pick an optimal victim, may need to read files from /proc/, > > which tries to acquire the same mmap_sem for reading and deadlocks. > > > > This patch changes the way tasks behave after detecting a memcg OOM > > and makes sure nobody loops or sleeps with locks held: > > > > 1. When OOMing in a user fault, invoke the OOM killer and restart the > > fault instead of looping on the charge attempt. This way, the OOM > > victim can not get stuck on locks the looping task may hold. > > > > 2. When OOMing in a user fault but somebody else is handling it > > (either the kernel OOM killer or a userspace handler), don't go to > > sleep in the charge context. Instead, remember the OOMing memcg in > > the task struct and then fully unwind the page fault stack with > > -ENOMEM. pagefault_out_of_memory() will then call back into the > > memcg code to check if the -ENOMEM came from the memcg, and then > > either put the task to sleep on the memcg's OOM waitqueue or just > > restart the fault. The OOM victim can no longer get stuck on any > > lock a sleeping task may hold. > > > > Reported-by: Reported-by: azurIt > > Debugged-by: Michal Hocko > > Signed-off-by: Johannes Weiner > > I was thinking whether we should add task_in_memcg_oom into return to > the userspace path just in case but this should be OK for now and new > users of mem_cgroup_enable_oom will be fought against hard. Absolutely, I would have liked it to be at the lowest possible point in the stack as well, but this seemed like a good trade off. And I expect the sites enabling and disabling memcg OOM killing to be fairly static. > Acked-by: Michal Hocko Thanks! -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/