Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Roman Gushchin <guro@fb.com>
To:     Johannes Weiner <hannes@cmpxchg.org>
CC:     Andrew Morton <akpm@linux-foundation.org>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Kernel Team <Kernel-team@fb.com>
Subject: Re: [PATCH] mm: fix false-positive OVERCOMMIT_GUESS failures
Thread-Topic: [PATCH] mm: fix false-positive OVERCOMMIT_GUESS failures
Thread-Index: AQHU8WPzV7whvXpmeUeLhsl6IRCndKY486cA
Date:   Fri, 12 Apr 2019 20:06:34 +0000
Message-ID: <20190412200629.GA24377@tower.DHCP.thefacebook.com>
References: <20190412191418.26333-1-hannes@cmpxchg.org>
In-Reply-To: <20190412191418.26333-1-hannes@cmpxchg.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-originating-ip: [2620:10d:c090:200::1:2586]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 618b90be-36fd-4858-de96-08d6bf825c63
x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600139)(711020)(4605104)(2017052603328)(7193020);SRVR:BYAPR15MB3464;
x-ms-traffictypediagnostic: BYAPR15MB3464:
x-microsoft-antispam-prvs: <BYAPR15MB34644CDEFE076D0722E31AC2BE280@BYAPR15MB3464.namprd15.prod.outlook.com>
x-forefront-prvs: 0005B05917
x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(376002)(346002)(396003)(366004)(39860400002)(136003)(199004)(189003)(76176011)(476003)(11346002)(33656002)(81156014)(81166006)(486006)(478600001)(446003)(68736007)(229853002)(8936002)(6436002)(102836004)(6512007)(5660300002)(1076003)(6116002)(54906003)(316002)(4326008)(14454004)(256004)(7736002)(71190400001)(2906002)(71200400001)(6506007)(386003)(305945005)(9686003)(6916009)(86362001)(25786009)(6486002)(14444005)(106356001)(105586002)(6246003)(99286004)(186003)(97736004)(53936002)(52116002)(8676002)(46003);DIR:OUT;SFP:1102;SCL:1;SRVR:BYAPR15MB3464;H:BYAPR15MB2631.namprd15.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1;
received-spf: None (protection.outlook.com: fb.com does not designate
 permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: a2FLjq9PZzm9RXIQb4/s5oW76KDP6WrSHcN+R/HqE1rdkyXjmhAOBLvBaAOfW/iStb+8udrALGRjiqLWleAdAJcFE2X15zykVg2CQmQ/STrbbPXEFMBJxtanOre2uRID6En4FmqFQUAriQwrca8ahwGOohONKxxhCoopzU7xosBceSRLn1j66biTVRdULoHQZCoCdmR8C4LyTVGF9YaLySnMINLcbxMHgEZUzd8Ko3Lk3/Tr03r3BhSa8HeY3PryaopYaWno5Ori5BglrEmcAgGORdhHw/To9CK6Lw7lNVJJb+0JQkpTYB+IhvnjO4FSOYwfIg5DrwpbAwyZvXzi5iGCcDK16TJ6FvAbKH6c0Ax/cMnwaiviKw9SaOTvYwv/KHMkt5YZbkf4V8wzoBuF3ZGTZBrrjLEGNx4qCEHDzqo=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <07E8811B2E537E488E0FA5B519045F3F@namprd15.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 618b90be-36fd-4858-de96-08d6bf825c63
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Apr 2019 20:06:34.1190
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR15MB3464
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-12_11:,,
 signatures=0
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Apr 12, 2019 at 03:14:18PM -0400, Johannes Weiner wrote:
> With the default overcommit=3D=3Dguess we occasionally run into mmap
> rejections despite plenty of memory that would get dropped under
> pressure but just isn't accounted reclaimable. One example of this is
> dying cgroups pinned by some page cache. A previous case was auxiliary
> path name memory associated with dentries; we have since annotated
> those allocations to avoid overcommit failures (see d79f7aa496fc ("mm:
> treat indirectly reclaimable memory as free in overcommit logic")).
>=20
> But trying to classify all allocated memory reliably as reclaimable
> and unreclaimable is a bit of a fool's errand. There could be a myriad
> of dependencies that constantly change with kernel versions.
>=20
> It becomes even more questionable of an effort when considering how
> this estimate of available memory is used: it's not compared to the
> system-wide allocated virtual memory in any way. It's not even
> compared to the allocating process's address space. It's compared to
> the single allocation request at hand!
>=20
> So we have an elaborate left-hand side of the equation that tries to
> assess the exact breathing room the system has available down to a
> page - and then compare it to an isolated allocation request with no
> additional context. We could fail an allocation of N bytes, but for
> two allocations of N/2 bytes we'd do this elaborate dance twice in a
> row and then still let N bytes of virtual memory through. This doesn't
> make a whole lot of sense.
>=20
> Let's take a step back and look at the actual goal of the
> heuristic. From the documentation:
>=20
>    Heuristic overcommit handling. Obvious overcommits of address
>    space are refused. Used for a typical system. It ensures a
>    seriously wild allocation fails while allowing overcommit to
>    reduce swap usage.  root is allowed to allocate slightly more
>    memory in this mode. This is the default.
>=20
> If all we want to do is catch clearly bogus allocation requests
> irrespective of the general virtual memory situation, the physical
> memory counter-part doesn't need to be that complicated, either.
>=20
> When in GUESS mode, catch wild allocations by comparing their request
> size to total amount of ram and swap in the system.
>=20
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>

My 2c here: any kinds of percpu counters and percpu data is accounted
as unreclaimable and can alter the calculation significantly.

This is a special problem on hosts, which were idle for some time.
Without any memory pressure, kernel caches do occupy most of the memory,
so than a following attempt to start a workload fails.

With a big pleasure:
Acked-by: Roman Gushchin <guro@fb.com>

Thanks!