Received: by 2002:ac0:950c:0:0:0:0:0 with SMTP id f12csp2278518imc; Tue, 12 Mar 2019 10:26:58 -0700 (PDT) X-Google-Smtp-Source: APXvYqxV/JLvapXwqut84to4N7yhNUO5dm0YT28kUjBu2jnQ6rM5uc1DDMAC+kwwgB6HSMTf42Ur X-Received: by 2002:a17:902:7202:: with SMTP id ba2mr40110268plb.147.1552411618450; Tue, 12 Mar 2019 10:26:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552411618; cv=none; d=google.com; s=arc-20160816; b=CE89tvY7btbhiyZC/0yi+kb5EcA9hKBg0OiaWR43SXDIttErMZ+3TvBjasyN34iZe1 czy7gGfT7XZKPvhOEQqQD4fXahWEzaCStMCwEa7T427u5mvYIlBlYpfB4AyC+KKj9SCx 4d1dytMNrml9lZXkRB+54FxRlQqZWZNj7jUrRAkfj7GJXj43INDkl0+k0EPRdNkFx4cw vArJOFbppfeTjeQHz9ZPSMvaY1DEDkP7QZrtBBoIQQB9kzJuAl/Iy/pXUBJ9ULFFQCpE oFNw+IIkE+7PX93s+Kdw3eNSVTC1bQeORooUNucop82VLrwUg+YDr0VVgAUrt9GIPaR8 fr6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=2NlXEi3emhG/WYSsl8nOZrJjFK+P8wjzHIuDi4pDMGw=; b=QSRWV2PVP78dOd52Xl9sVBCpGPODtGVcxDElDW2tVNpvmaMXW85Fda96Ok/DJeXLsq UVh/WYQ5owQSpe7ETUrJtu5Ge2yMos3bNWBlRBdPILCl5qHdDxKeL41Ek3iKCfRv2GaT 9DLgG6vBMlEs7UHQR+c7FCZzdZHhjIQCZnFRTy8FY8LpS7Ldtu00tVoV5Oq+r/dBt0DB Nl3hD7yd7UhHwP8rGVyNk8LAO+GMo+jH2T0WGR52JKPstV5kJuqdQTG5ZbpAiIl+fXIw /NNIreK0wQxU065g1emOb8vRQNDDQFcJqpyjDfm0I+BtPFu/ji0eCpoty1xU9k/AuUuz Io+A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="S/lWy1OY"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a9si8836302pla.226.2019.03.12.10.26.42; Tue, 12 Mar 2019 10:26:58 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="S/lWy1OY"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729173AbfCLRYK (ORCPT + 99 others); Tue, 12 Mar 2019 13:24:10 -0400 Received: from mail-it1-f196.google.com ([209.85.166.196]:54462 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729807AbfCLRR4 (ORCPT ); Tue, 12 Mar 2019 13:17:56 -0400 Received: by mail-it1-f196.google.com with SMTP id w18so5662751itj.4 for ; Tue, 12 Mar 2019 10:17:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=2NlXEi3emhG/WYSsl8nOZrJjFK+P8wjzHIuDi4pDMGw=; b=S/lWy1OYNMcZAszvY/+S7u7iFYwB2tvImYNHp+h2BZ1K/Qpw9siYUo1kfqyOtBr11n 7uI5B0cXPo9fJM7fc0psjlMxFUFi8RcfZ8Qug4Ztk0U72111mMpX7GGa3vAXDs50wj2F 7ocGYkrxrH2/yvN9k0oRg8t4WNvS2cdseKRn6WjnEHA1FxvsxMfnqVBzopnTOJ5GjVkh +bk2LxlWFMW7/obEuady1Ys9ctDnwQ2tzfZhHl5n2ukImhNSycJ4NAFqpVeLxbTFq5bt bB3YmQqQx0lcG0pRG3V4LGfBPhyisqifZ7xft7jCv5i1kKLv9kwhnCa8h1enJfI3JYjG Vj5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=2NlXEi3emhG/WYSsl8nOZrJjFK+P8wjzHIuDi4pDMGw=; b=sfBosrvQGdHXp9yFlV48gHuHBpKUWZp11sgOFxISe4+SGr0FcTOFN2z1MfyPDPuGEx 0NPiQxxGTij24Jf05leERgvXEM+i/gsV+JRhLQGS3g84bFwBk6PiFXzRZYpFpbARZtgY Run3ewX7ohuqHb8HwTCYdHdKNey7/ZZdwbVZbde3RTkOXLGA3AipkE/Ylm3c/j1XUd+0 tl/Cq7CN4mkogXFP003shZdZ5LYWfHIzQMpoi00Asz2/+OFzD7yuG6jeEjvYiqumbbyN Dx6yh/I4oRwkudrlKMdRj/fbUpfDoSFyoX7UvXzczajBuvGZSC2gwievfll6cmMvynL+ qvIw== X-Gm-Message-State: APjAAAWXc+QxS+1C0V3HX8fYYhjQre83pe8A5NKLo2N8x768DjXhp1wd 1Cb+LiVXxbEolJG4NyDLtweprchisTJllu9TFosIag== X-Received: by 2002:a24:3a8b:: with SMTP id m133mr2651650itm.26.1552411074982; Tue, 12 Mar 2019 10:17:54 -0700 (PDT) MIME-Version: 1.0 References: <20190310203403.27915-1-sultan@kerneltoast.com> <20190311174320.GC5721@dhcp22.suse.cz> <20190311175800.GA5522@sultan-box.localdomain> <20190311204626.GA3119@sultan-box.localdomain> <20190312080532.GE5721@dhcp22.suse.cz> <20190312163741.GA2762@sultan-box.localdomain> In-Reply-To: <20190312163741.GA2762@sultan-box.localdomain> From: Tim Murray Date: Tue, 12 Mar 2019 10:17:43 -0700 Message-ID: Subject: Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android To: Sultan Alsawaf Cc: Michal Hocko , Suren Baghdasaryan , Greg Kroah-Hartman , =?UTF-8?B?QXJ2ZSBIasO4bm5ldsOlZw==?= , Todd Kjos , Martijn Coenen , Joel Fernandes , Christian Brauner , Ingo Molnar , Peter Zijlstra , LKML , devel@driverdev.osuosl.org, linux-mm Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Mar 12, 2019 at 9:37 AM Sultan Alsawaf wrote: > > On Tue, Mar 12, 2019 at 09:05:32AM +0100, Michal Hocko wrote: > > The only way to control the OOM behavior pro-actively is to throttle > > allocation speed. We have memcg high limit for that purpose. Along with > > PSI, I can imagine a reasonably working user space early oom > > notifications and reasonable acting upon that. > > The issue with pro-active memory management that prompted me to create this was > poor memory utilization. All of the alternative means of reclaiming pages in the > page allocator's slow path turn out to be very useful for maximizing memory > utilization, which is something that we would have to forgo by relying on a > purely pro-active solution. I have not had a chance to look at PSI yet, but > unless a PSI-enabled solution allows allocations to reach the same point as when > the OOM killer is invoked (which is contradictory to what it sets out to do), > then it cannot take advantage of all of the alternative memory-reclaim means > employed in the slowpath, and will result in killing a process before it is > _really_ necessary. There are two essential parts of a lowmemorykiller implementation: when to kill and how to kill. There are a million possible approaches to decide when to kill an unimportant process. They usually trade off between the same two failure modes depending on the workload. If you kill too aggressively, a transient spike that could be imperceptibly absorbed by evicting some file pages or moving some pages to ZRAM will result in killing processes, which then get started up later and have a performance/battery cost. If you don't kill aggressively enough, you will encounter a workload that thrashes the page cache, constantly evicting and reloading file pages and moving things in and out of ZRAM, which makes the system unusable when a process should have been killed instead. As far as I've seen, any methodology that uses single points in time to decide when to kill without completely biasing toward one or the other is susceptible to both. The minfree approach used by lowmemorykiller/lmkd certainly is; it is both too aggressive for some workloads and not aggressive enough for other workloads. My guess is that simple LMK won't kill on transient spikes but will be extremely susceptible to page cache thrashing. This is not an improvement; page cache thrashing manifests as the entire system running very slowly. What you actually want from lowmemorykiller/lmkd on Android is to only kill once it becomes clear that the system will continue to try to reclaim memory to the extent that it could impact what the user actually cares about. That means tracking how much time is spent in reclaim/paging operations and the like, and that's exactly what PSI does. lmkd has had support for using PSI as a replacement for vmpressure for use as a wakeup trigger (to check current memory levels against the minfree thresholds) since early February. It works fine; unsurprisingly it's better than vmpressure at avoiding false wakeups. Longer term, there's a lot of work to be done in lmkd to turn PSI into a kill trigger and remove minfree entirely. It's tricky (mainly because of the "when to kill another process" problem discussed later), but I believe it's feasible. How to kill is similarly messy. The latency of reclaiming memory post SIGKILL can be severe (usually tens of milliseconds, occasionally >100ms). The latency we see on Android usually isn't because those threads are blocked in uninterruptible sleep, it's because times of memory pressure are also usually times of significant CPU contention and these are overwhelmingly CFS threads, some of which may be assigned a very low priority. lmkd now sets priorities and resets cpusets upon killing a process, and we have seen improved reclaim latency because of this. oom reaper might be a good approach to avoid this latency (I think some in-kernel lowmemorykiller implementations rely on it), but we can't use it from userspace. Something for future consideration. A non-obvious consequence of both of these concerns is that when to kill a second process is a distinct and more difficult problem than when to kill the first. A second process should be killed if reclaim from the first process has finished and there has been insufficient memory reclaimed to avoid perceptible impact. Identifying whether memory pressure continues at the same level can probably be handled through multiple PSI monitors with different thresholds and window lengths, but this is an area of future work. Knowing whether a SIGKILL'd process has finished reclaiming is as far as I know not possible without something like procfds. That's where the 100ms timeout in lmkd comes in. lowmemorykiller and lmkd both attempt to wait up to 100ms for reclaim to finish by checking for the continued existence of the thread that received the SIGKILL, but this really means that they wait up to 100ms for the _thread_ to finish, which doesn't tell you anything about the memory used by that process. If those threads terminate early and lowmemorykiller/lmkd get a signal to kill again, then there may be two processes competing for CPU time to reclaim memory. That doesn't reclaim any faster and may be an unnecessary kill. So, in summary, the impactful LMK improvements seem like - get lmkd and PSI to the point that lmkd can use PSI signals as a kill trigger and remove all static memory thresholds from lmkd completely. I think this is mostly on the lmkd side, but there may be some PSI or PSI monitor changes that would help - give userspace some path to start reclaiming memory without waiting for every thread in a process to be scheduled--could be oom reaper, could be something else - offer a way to wait for process termination so lmkd can tell when reclaim has finished and know when killing another process is appropriate