Received: by 2002:a25:8b12:0:0:0:0:0 with SMTP id i18csp157621ybl; Wed, 21 Aug 2019 16:42:50 -0700 (PDT) X-Google-Smtp-Source: APXvYqzkI/qTlY2fVp7uTMmJCNVqSi15+o63mYPz69mQmQk5PudZ+vA48A0g/EoOFK6K8ODvhhre X-Received: by 2002:a63:31cc:: with SMTP id x195mr30885838pgx.147.1566430970443; Wed, 21 Aug 2019 16:42:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566430970; cv=none; d=google.com; s=arc-20160816; b=lkwwKL6BscbngzHtVjT6VUOD7pTw/LkG2Ep6djJaDRaN0E5r2V/2/Mc3eYL6g1iizl X6teJ7ykZNn4VFeQC15foRHLjY3PFoPQx8YL4oQ7PdqljJ1mW2s9YQRttP4Nxgm1ftVP X2hjjxv1OV6/Ei7llqrNfuMQISlTdzgAPlsXtUawcXx3m2gZ4+AuN/GzVHYaVvbsaNIR dBPNN2xD1ZdSUmETYarp3Li65MuN479XFEtm2Md+mTaqk64YBvx1XHpCnxrZ+zl5QZ/c 7e4c8pMJbiXUjdqDdwUcVoi3ed9D5WrmOo8QN59o+lCOpYpmBAUWI99mh7mhdPbDGjG4 a2/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=lh39CY7OKDO4zGxrF3Tom1rSCBIenfklbHsSPnOIW5A=; b=PRn7IGusOjnWpHWL1BF0qTsUfNTeScbejfN6bRhnX6tIJmbYnw94vpgSThEsr90lCo pU5bJUtZVW+Adw+G4cZ/cXbaij5BgQd7kpo3Fb46SNicjqkdvOZ96TBr2UtfF86PP/wG zgqm+fupy1OkK77HZXzkoOqqsI2NVYeZpVBxFgaUt8qC0d4+YT39d8tdYa76jzXwBt1x c8eCvOByMbtp8h3lDKBU1V4UI/zWpKIoPeLZHy/TNPAdeY5d6yWnn2MjQgm5VxqVRCOK PzlQlKUPBSfQjmpDIBKahVbtFcmE036CwFQ6lSC5D6XrT9dWTtv6tMbDWKO6tNNvtzjj GJww== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=mDh1flxQ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a15si15160572pgk.523.2019.08.21.16.42.35; Wed, 21 Aug 2019 16:42:50 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=mDh1flxQ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728708AbfHUVnH (ORCPT + 99 others); Wed, 21 Aug 2019 17:43:07 -0400 Received: from mail-yw1-f68.google.com ([209.85.161.68]:44759 "EHLO mail-yw1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727923AbfHUVnH (ORCPT ); Wed, 21 Aug 2019 17:43:07 -0400 Received: by mail-yw1-f68.google.com with SMTP id l79so1524189ywe.11 for ; Wed, 21 Aug 2019 14:43:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=lh39CY7OKDO4zGxrF3Tom1rSCBIenfklbHsSPnOIW5A=; b=mDh1flxQMt84OrIH1YADm/mAFBaDJX+S+ZjFk5nrTwD/xeAnqlUKzJQ1rRRxdoSCAT M9WWHHiFGBlbXKN1m/Nf3GC7Cd/+lH4Ebaf9r7gTcid2OB7D66Ufnz8zmRPThAsqOvrU dT0uif0mW22hFLKMxHl28mLX1/nlXfHcu4QwZI7Q+whEIGi9eg58GjKTFCutCa4T8Paf sJ1MVANY0FpVae45yFZHQBZGNgwu7DUt85P7wpuekKTLikDL5y7UkOq4nI/qVkWOvnnh YBp2N+fd0Ll5rJAzyP+FHrJmddQV3ZANKqLpkZcLdv8KYP1yltxQum2SKaOg9Cu4To8A 6pVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=lh39CY7OKDO4zGxrF3Tom1rSCBIenfklbHsSPnOIW5A=; b=mzFps4bD6OazWk592QI0FogeDmpnXLuKwP7/oiVHIeVWVFp0I+4QM1tlcFY0gRmIg0 2ifk9DIcfzf6tliyoXU12V/Z7W8+owroQ1HtIFKj3RrMfqekDMHh6EbRX4HfEUTZKej1 2n6GD5NYNVKdHT3cCOz0K/f10kWzDJzrdXPGLOg2iAUYg0oI1/ywACEYtxwKt9b+XxqY +Tze9BUB1bCGyIjGsIibz6YsXnBk26HiX3K5tiRlehUaRpP0M0kQ68p1EQm27kr0wfg5 w4AElL2EG+SM3dR9d0WDr5Fmu/WcdSiKzxmiIpvRdtAwpuGBp5Zz0bB/PZGbMaQaOcjC v5DQ== X-Gm-Message-State: APjAAAU2baZfbvsmkVLLNxeEbjigDaruaYrID1WqwBuTbCaO2UTw2gCr muxw5YhqS14QA+nRv3PU2XZKdYeC1vvMH4SpxNQ= X-Received: by 2002:a81:608a:: with SMTP id u132mr3521701ywb.474.1566423786210; Wed, 21 Aug 2019 14:43:06 -0700 (PDT) MIME-Version: 1.0 References: <20190820064620.5119-1-drake@endlessm.com> In-Reply-To: <20190820064620.5119-1-drake@endlessm.com> From: James Courtier-Dutton Date: Wed, 21 Aug 2019 22:42:29 +0100 Message-ID: Subject: Re: Let's talk about the elephant in the room - the Linux kernel's inability to gracefully handle low memory pressure To: Daniel Drake Cc: "Artem S. Tashkinov" , LKML Mailing List , linux@endlessm.com, hadess@hadess.net, Johannes Weiner Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 20 Aug 2019 at 07:47, Daniel Drake wrote: > > Hi, > > And if there is a meaningful way to make the kernel behave better, that would > obviously be of huge value too. > > Thanks > Daniel Hi, Is there a way for an application to be told that there is a memory pressure situation? For example, say I do a "make -j32" and half way through the compile it hits a memory pressure situation. If make could have been told about it. It could give up on some of the parallel compiles, and instead proceed as if the user have typed "make -j4". It could then re-try the failed compile parts, that failed due to memory pressure. I know all applications won't be this clever, but providing a kernel API so that an application could do something about it, if the programmer of that application has thought about it. It could be similar with say, a hadoop application. If the hadoop process detects memory pressure, if could back off, and process the data more slowly and not try to do so much at the same time. The kernel could also detect which processes are contributing most to the memory pressure (trying to do mallocs) and give them less processor time, and instead ask all processes to release some memory, and those processes that understood the kernel API for that notification could actually do something about it. I have also found, for the desktop, one of the biggest pressures on the system is disk pressure. Accessing the disk causes the UI to be less responsive. For example, if I am in vim, and trying to type letters on the keyboard, whether some other application is using the disk or not should have no impact on my letter writing. Has anyone got any ideas with regards to what we can do about that? Kind Regards James