Received: by 2002:a25:b794:0:0:0:0:0 with SMTP id n20csp4219864ybh; Tue, 6 Aug 2019 08:09:43 -0700 (PDT) X-Google-Smtp-Source: APXvYqx35uXKMtPpihOgODuohVRNd5bIbXS2x6McMuXIACiM5G7r9+HpjeT7U2rr3s4KO4j276Hu X-Received: by 2002:a17:90a:24ac:: with SMTP id i41mr3669781pje.124.1565104183625; Tue, 06 Aug 2019 08:09:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565104183; cv=none; d=google.com; s=arc-20160816; b=TYVh1rV7XvBTII9pZZ1s23chHUMS5CUODkYMuAWCZ32h61TL5UokJMqrLHib3znaq3 txsvNJ9CLFTQLof6sqxXH/fFDlMQ0VXaZN3HI2MU15ft2b/asi+6tjzehTWGz5L9JNcG u48+oXTxkAm/brgTJYrwJ2tgqd/dnTS7JBvsUu+JzLRTfH6swUjrxrOHHfwyiwJbBQ7n Rbtxto31kPUXwaIzEcDjXeTX23BYBVaVQzPsShjSesdenKIWnhQQg0S2V4j2dMIWfFVt A/Yava+qMJ9yW/zhvVAebqVqww+uU3//TDsEP0orb4xOLzOwItJPxSqcW+XMtDm3XshT Ts1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date; bh=y7BrABq3gCM5NwrWP3u6YNUhVFfVU59M/ZD27Xiu4XY=; b=Z0SM3HXnywy41JJu14nIZakb/UmvJnZCD17fhAHEKwIwSmdgSbntB5tqgzQeA5YMS/ Wk3bGQ2UWFxRS0UWgcppgGvDcTj79ycVfTk6pZHSne672XgFBAir7URHBB+wArw7BIAS kDbQenePrS8ski40xgSf1xuxAq+VY9L8YEfFX02wssaSqSQzgD6wCLpeVfeLhVp2UD+s VwQZ5SWbWTfPNU1jFV6OVQppvwy7JaWzT0hhrBoLMI0wLkxdNZfhL7H7dQCrA3cRc5xq Ub46PeuVSsVmleFYVKT/8cY9pwLYpQB5Wq+Q5YgkpkdAVvCX3SaiUF53NOm7TKY6jvE/ 4yNA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y71si44147815pgd.271.2019.08.06.08.09.27; Tue, 06 Aug 2019 08:09:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732990AbfHFOgM (ORCPT + 99 others); Tue, 6 Aug 2019 10:36:12 -0400 Received: from mx2.suse.de ([195.135.220.15]:33814 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1731783AbfHFOgM (ORCPT ); Tue, 6 Aug 2019 10:36:12 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 7E64CAE1B; Tue, 6 Aug 2019 14:36:10 +0000 (UTC) Date: Tue, 6 Aug 2019 16:36:08 +0200 From: Michal Hocko To: Johannes Weiner Cc: Vlastimil Babka , Suren Baghdasaryan , "Artem S. Tashkinov" , LKML , linux-mm Subject: Re: Let's talk about the elephant in the room - the Linux kernel's inability to gracefully handle low memory pressure Message-ID: <20190806143608.GE11812@dhcp22.suse.cz> References: <20190805193148.GB4128@cmpxchg.org> <398f31f3-0353-da0c-fc54-643687bb4774@suse.cz> <20190806142728.GA12107@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190806142728.GA12107@cmpxchg.org> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue 06-08-19 10:27:28, Johannes Weiner wrote: > On Tue, Aug 06, 2019 at 11:36:48AM +0200, Vlastimil Babka wrote: > > On 8/6/19 3:08 AM, Suren Baghdasaryan wrote: > > >> @@ -1280,3 +1285,50 @@ static int __init psi_proc_init(void) > > >> return 0; > > >> } > > >> module_init(psi_proc_init); > > >> + > > >> +#define OOM_PRESSURE_LEVEL 80 > > >> +#define OOM_PRESSURE_PERIOD (10 * NSEC_PER_SEC) > > > > > > 80% of the last 10 seconds spent in full stall would definitely be a > > > problem. If the system was already low on memory (which it probably > > > is, or we would not be reclaiming so hard and registering such a big > > > stall) then oom-killer would probably kill something before 8 seconds > > > are passed. > > > > If oom killer can act faster, than great! On small embedded systems you probably > > don't enable PSI anyway? > > > > > If my line of thinking is correct, then do we really > > > benefit from such additional protection mechanism? I might be wrong > > > here because my experience is limited to embedded systems with > > > relatively small amounts of memory. > > > > Well, Artem in his original mail describes a minutes long stall. Things are > > really different on a fast desktop/laptop with SSD. I have experienced this as > > well, ending up performing manual OOM by alt-sysrq-f (then I put more RAM than > > 8GB in the laptop). IMHO the default limit should be set so that the user > > doesn't do that manual OOM (or hard reboot) before the mechanism kicks in. 10 > > seconds should be fine. > > That's exactly what I have experienced in the past, and this was also > the consistent story in the bug reports we have had. > > I suspect it requires a certain combination of RAM size, CPU speed, > and IO capacity: the OOM killer kicks in when reclaim fails, which > happens when all scanned LRU pages were locked and under IO. So IO > needs to be slow enough, or RAM small enough, that the CPU can scan > all LRU pages while they are temporarily unreclaimable (page lock). > > It may well be that on phones the RAM is small enough relative to CPU > size. > > But on desktops/servers, we frequently see that there is a wider > window of memory consumption in which reclaim efficiency doesn't drop > low enough for the OOM killer to kick in. In the time it takes the CPU > to scan through RAM, enough pages will have *just* finished reading > for reclaim to free them again and continue to make "progress". > > We do know that the OOM killer might not kick in for at least 20-25 > minutes while the system is entirely unresponsive. People usually > don't wait this long before forcibly rebooting. In a managed fleet, > ssh heartbeat tests eventually fail and force a reboot. > > I'm not sure 10s is the perfect value here, but I do think the kernel > should try to get out of such a state, where interacting with the > system is impossible, within a reasonable amount of time. > > It could be a little too short for non-interactive number-crunching > systems... Would it be possible to have a module with tunning knobs as parameters and hook into the PSI infrastructure? People can play with the setting to their need, we wouldn't really have think about the user visible API for the tuning and this could be easily adopted as an opt-in mechanism without a risk of regressions. I would really love to see a simple threshing watchdog like the one you have proposed earlier. It is self contained and easy to play with if the parameters are not hardcoded. -- Michal Hocko SUSE Labs