Received: by 2002:a05:6a10:1d13:0:0:0:0 with SMTP id pp19csp2908852pxb; Tue, 24 Aug 2021 10:21:41 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxRlGR3Rc9Nt+BTQiVKL1m1BXkN6ztV7+fh8CM+/EnWWuOhctAMY+k6Nw2dRTAjoObVeguQ X-Received: by 2002:a05:6638:3781:: with SMTP id w1mr26334352jal.37.1629825701608; Tue, 24 Aug 2021 10:21:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1629825701; cv=none; d=google.com; s=arc-20160816; b=au7uM+tywiJ4ZC0Fmb6k5s6MiT7W6Ek32Lml0TRSHe+fB/QV1dFjFZMfixA4GRf2oh aYR2/l6KbxAvR7mahTY4AcUAdkczmdsUGSxz6Uz3QHjGpZUbOvpwdwwkhDrLnFTRYGkj 1Si8T4mSo6XmysLO2aqqrfosNuLaLiAZXE3YHAI1XlWK8c09Hy7zGFNZjoMQq6WtHYNo A9sObfRUiMcPfGO/M2JLfqWzpN4ZRQ0h8iowffy7Uq156d4G8ZDkfOQciAA399aZA5cw DwS0xDSB89P2Maz9MOwwnwwYG+WowX0bJSbgItznZorHxeynex4IgtUstTVKkDkpa4jr 8u/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=lwSaNfx/kS/GwBQpwrNPQjffc53OLF6RjS+2Cus8U6M=; b=ldB4rVgmwd0uXfXSvgZgnZnSjpcofzrYGSk+W39iv+sOnujzorblczM/QNf3hzBf7+ En6AUDeGMnNbr96kahkVmKogxLUEGSzxEyW5dkX1/rbEa5HjRcisJ+Dc/CIrnuQKhvnW qk5VyCsXKmhd1bTSohyd607RmlYFa81F7RW5MnV1rAZhDUX2SKTfkjNytGARlTUkZ4kt /Lw5y8Xaj4owLw8Y+jhcNVc5pdaIz7WNllmpr8XelmdFMnYyfLE1ffUAbE1fuwYd7fKe 3r87B2Unh5MFvOJ+gYvllU94MeaZVkIDseqebNyJu/ye96/RW3s51t8dgfEcEQNfjo3b bE9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=sChyFNgW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id q4si20423253jaj.45.2021.08.24.10.21.29; Tue, 24 Aug 2021 10:21:41 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=sChyFNgW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240162AbhHXRV2 (ORCPT + 99 others); Tue, 24 Aug 2021 13:21:28 -0400 Received: from mail.kernel.org ([198.145.29.99]:56448 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239802AbhHXRRv (ORCPT ); Tue, 24 Aug 2021 13:17:51 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id C7A7661AD2; Tue, 24 Aug 2021 17:02:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1629824524; bh=p+qhYVm330lRQxgt4fFTwQHyQfqhWgnJpPkiaYXOtQo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sChyFNgWMi2o/LHBiD7IqI95VVgML+lUCutvXor+LVUt7yZ/DnY52Tq9t/E3qQSjm gTqMhmE8pb4WMVbhton55V/R+h6j4Cn859iFAjKI1OV3z2Zu/YzWJGnQjovTg0UPX7 bsNWGNrWsV9l9Trsh4ovqjIOQ1Tg/6h9XwFR4oisc2S7T+KOP2FmJWOCEMn8S7Tdj0 zr08zD94Ru55ZL7iNf8iDt1Zf0euugVxksOiYSjsYefFLLWV+pVhbCKLWezk33IDOG sQGgsc991aDXtfJc2ZNO5XA1hCg7zQqUe4wYraZ50xfrZwoQGQ1whyWYKTSq7yD6Zr zjjTxeuJxnqyg== From: Sasha Levin To: linux-kernel@vger.kernel.org, stable@vger.kernel.org Cc: Yafang Shao , Chris Down , Andrew Morton , Michal Hocko , Johannes Weiner , Roman Gushchin , Linus Torvalds , Sasha Levin Subject: [PATCH 5.4 57/61] mm, memcg: avoid stale protection values when cgroup is above protection Date: Tue, 24 Aug 2021 13:01:02 -0400 Message-Id: <20210824170106.710221-58-sashal@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210824170106.710221-1-sashal@kernel.org> References: <20210824170106.710221-1-sashal@kernel.org> MIME-Version: 1.0 X-KernelTest-Patch: http://kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.4.143-rc1.gz X-KernelTest-Tree: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git X-KernelTest-Branch: linux-5.4.y X-KernelTest-Patches: git://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git X-KernelTest-Version: 5.4.143-rc1 X-KernelTest-Deadline: 2021-08-26T17:01+00:00 X-stable: review X-Patchwork-Hint: Ignore Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Yafang Shao [ Upstream commit 22f7496f0b901249f23c5251eb8a10aae126b909 ] Patch series "mm, memcg: memory.{low,min} reclaim fix & cleanup", v4. This series contains a fix for a edge case in my earlier protection calculation patches, and a patch to make the area overall a little more robust to hopefully help avoid this in future. This patch (of 2): A cgroup can have both memory protection and a memory limit to isolate it from its siblings in both directions - for example, to prevent it from being shrunk below 2G under high pressure from outside, but also from growing beyond 4G under low pressure. Commit 9783aa9917f8 ("mm, memcg: proportional memory.{low,min} reclaim") implemented proportional scan pressure so that multiple siblings in excess of their protection settings don't get reclaimed equally but instead in accordance to their unprotected portion. During limit reclaim, this proportionality shouldn't apply of course: there is no competition, all pressure is from within the cgroup and should be applied as such. Reclaim should operate at full efficiency. However, mem_cgroup_protected() never expected anybody to look at the effective protection values when it indicated that the cgroup is above its protection. As a result, a query during limit reclaim may return stale protection values that were calculated by a previous reclaim cycle in which the cgroup did have siblings. When this happens, reclaim is unnecessarily hesitant and potentially slow to meet the desired limit. In theory this could lead to premature OOM kills, although it's not obvious this has occurred in practice. Workaround the problem by special casing reclaim roots in mem_cgroup_protection. These memcgs are never participating in the reclaim protection because the reclaim is internal. We have to ignore effective protection values for reclaim roots because mem_cgroup_protected might be called from racing reclaim contexts with different roots. Calculation is relying on root -> leaf tree traversal therefore top-down reclaim protection invariants should hold. The only exception is the reclaim root which should have effective protection set to 0 but that would be problematic for the following setup: Let's have global and A's reclaim in parallel: | A (low=2G, usage = 3G, max = 3G, children_low_usage = 1.5G) |\ | C (low = 1G, usage = 2.5G) B (low = 1G, usage = 0.5G) for A reclaim we have B.elow = B.low C.elow = C.low For the global reclaim A.elow = A.low B.elow = min(B.usage, B.low) because children_low_usage <= A.elow C.elow = min(C.usage, C.low) With the effective values resetting we have A reclaim A.elow = 0 B.elow = B.low C.elow = C.low and global reclaim could see the above and then B.elow = C.elow = 0 because children_low_usage > A.elow Which means that protected memcgs would get reclaimed. In future we would like to make mem_cgroup_protected more robust against racing reclaim contexts but that is likely more complex solution than this simple workaround. [hannes@cmpxchg.org - large part of the changelog] [mhocko@suse.com - workaround explanation] [chris@chrisdown.name - retitle] Fixes: 9783aa9917f8 ("mm, memcg: proportional memory.{low,min} reclaim") Signed-off-by: Yafang Shao Signed-off-by: Chris Down Signed-off-by: Andrew Morton Acked-by: Michal Hocko Acked-by: Johannes Weiner Acked-by: Chris Down Acked-by: Roman Gushchin Link: http://lkml.kernel.org/r/cover.1594638158.git.chris@chrisdown.name Link: http://lkml.kernel.org/r/044fb8ecffd001c7905d27c0c2ad998069fdc396.1594638158.git.chris@chrisdown.name Signed-off-by: Linus Torvalds Signed-off-by: Sasha Levin --- include/linux/memcontrol.h | 42 ++++++++++++++++++++++++++++++++++++-- mm/memcontrol.c | 8 ++++++++ mm/vmscan.c | 3 ++- 3 files changed, 50 insertions(+), 3 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index fb5b2a41bd45..059f55841cc8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -356,12 +356,49 @@ static inline bool mem_cgroup_disabled(void) return !cgroup_subsys_enabled(memory_cgrp_subsys); } -static inline unsigned long mem_cgroup_protection(struct mem_cgroup *memcg, +static inline unsigned long mem_cgroup_protection(struct mem_cgroup *root, + struct mem_cgroup *memcg, bool in_low_reclaim) { if (mem_cgroup_disabled()) return 0; + /* + * There is no reclaim protection applied to a targeted reclaim. + * We are special casing this specific case here because + * mem_cgroup_protected calculation is not robust enough to keep + * the protection invariant for calculated effective values for + * parallel reclaimers with different reclaim target. This is + * especially a problem for tail memcgs (as they have pages on LRU) + * which would want to have effective values 0 for targeted reclaim + * but a different value for external reclaim. + * + * Example + * Let's have global and A's reclaim in parallel: + * | + * A (low=2G, usage = 3G, max = 3G, children_low_usage = 1.5G) + * |\ + * | C (low = 1G, usage = 2.5G) + * B (low = 1G, usage = 0.5G) + * + * For the global reclaim + * A.elow = A.low + * B.elow = min(B.usage, B.low) because children_low_usage <= A.elow + * C.elow = min(C.usage, C.low) + * + * With the effective values resetting we have A reclaim + * A.elow = 0 + * B.elow = B.low + * C.elow = C.low + * + * If the global reclaim races with A's reclaim then + * B.elow = C.elow = 0 because children_low_usage > A.elow) + * is possible and reclaiming B would be violating the protection. + * + */ + if (root == memcg) + return 0; + if (in_low_reclaim) return READ_ONCE(memcg->memory.emin); @@ -847,7 +884,8 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, { } -static inline unsigned long mem_cgroup_protection(struct mem_cgroup *memcg, +static inline unsigned long mem_cgroup_protection(struct mem_cgroup *root, + struct mem_cgroup *memcg, bool in_low_reclaim) { return 0; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2701497edda5..6d7fe3589e4a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6446,6 +6446,14 @@ enum mem_cgroup_protection mem_cgroup_protected(struct mem_cgroup *root, if (!root) root = root_mem_cgroup; + + /* + * Effective values of the reclaim targets are ignored so they + * can be stale. Have a look at mem_cgroup_protection for more + * details. + * TODO: calculation should be more robust so that we do not need + * that special casing. + */ if (memcg == root) return MEMCG_PROT_NONE; diff --git a/mm/vmscan.c b/mm/vmscan.c index 10feb872d9a4..dc44da27673d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2462,7 +2462,8 @@ out: unsigned long protection; lruvec_size = lruvec_lru_size(lruvec, lru, sc->reclaim_idx); - protection = mem_cgroup_protection(memcg, + protection = mem_cgroup_protection(sc->target_mem_cgroup, + memcg, sc->memcg_low_reclaim); if (protection) { -- 2.30.2