Received: by 2002:a05:6a10:1a4d:0:0:0:0 with SMTP id nk13csp5576238pxb; Mon, 14 Feb 2022 02:25:11 -0800 (PST) X-Google-Smtp-Source: ABdhPJwFpbCsLnYgGPorIG1lv4uQy+/uJrxAVNFQ9zEYiJmjSC8jN+tRlNLNh/4AaToVOr7zV5/f X-Received: by 2002:a05:6402:4392:: with SMTP id o18mr14790790edc.338.1644834311441; Mon, 14 Feb 2022 02:25:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1644834311; cv=none; d=google.com; s=arc-20160816; b=dUWgyqZzE6wlgT1x8/+Z7mJVjyBWvDw7A9mJynyhlL1m1HFEqjN/MSxmWuAhzdVTd0 U+q/RYWSZK1V1B6unJkw8nfYU/Bl8mZMK9t4y6T7/GFV6CY5+yjuaiymnIayjnkYc+z9 P7spCzbt/dLBG7neV/PEG8Q+eNHs06+kyuV2u6lyi0cQB6gj9YPX1KoZlR8z/Eh3RN/N 3sY9jRVls573eutptOyrytfh0f/X3X/86OTk//mybcst0sSEATC2ipCMiLS6aBKSA8e9 kgHLGBxlmcsbDyRnRrefV/0ba38o0Kzb+XPchrg3o2Jx/dZldKYTDchKeKzcpfMPTHoh 4mlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:user-agent:in-reply-to:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :dkim-signature; bh=OZFYLN+S6IkbZKtWQGnLOGDHtfELHim7rgW7KJNl2oA=; b=bBKkL9dFirCFuwG4WCOZ6Wc0ySL/KeeiDb6VGnJwuRDGV+dfBq4TENRPUE86iuRYlP xFWYdTY0nQCy+DIL/HSOR+VXZEtXa7WFniFtpp8KOrzm7HNR7dGbPYVQgSmrNhMi2gQe iPWwzQsTg2XIpfzyxcfyZQYoiWoRVdCilQFWP/qH1uWDgAOgy0YAke4Q2wvLaHtoOLcz YhWZ4ah27li58S76gBgCenI6FtDpeT6jTbaar6a/IHmtFaawCvDPzoK5Vsfuon1rwZAS kerA/h860lNoszKbA2uc6JZjMlWg2SkhTEg/bWN6xNgJuBeQxRmlM8MWplVOWRSSCKM9 bytw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chrisdown.name header.s=google header.b=GgzQiLfz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chrisdown.name Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qw25si12211951ejc.717.2022.02.14.02.24.48; Mon, 14 Feb 2022 02:25:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@chrisdown.name header.s=google header.b=GgzQiLfz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chrisdown.name Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344320AbiBKMNw (ORCPT + 93 others); Fri, 11 Feb 2022 07:13:52 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:55920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236946AbiBKMNu (ORCPT ); Fri, 11 Feb 2022 07:13:50 -0500 Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DBAE1E9 for ; Fri, 11 Feb 2022 04:13:48 -0800 (PST) Received: by mail-ej1-x62f.google.com with SMTP id fj5so19598138ejc.4 for ; Fri, 11 Feb 2022 04:13:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chrisdown.name; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=OZFYLN+S6IkbZKtWQGnLOGDHtfELHim7rgW7KJNl2oA=; b=GgzQiLfzSmdOrEWpzLCjSq3QC0hcb9rDCloigLalwdcFW/g9GKJ3a3Z0+28hLH8wbK xp8IGtFCVCVQSyA9ZhpEaqrHjiShC/ZCizU4JmHjIzZ1F66JmvkQELHdJPnjFBAkJI3Q hXYU9/IjNZR9rTRX5wDuSaFBFFg905KXUmqWg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=OZFYLN+S6IkbZKtWQGnLOGDHtfELHim7rgW7KJNl2oA=; b=XkP5eWwTuDzjLXUpe5Crx0ZyZoHUCYaMQ95waOiIhosINJRckptjYpCDZotWVv07yP dLlEBwFDTB4/Q2UaeE8fohUINQdDGN5duInxlx4BBQmZgDOcn9/wYTaLsjkGJURzDaGQ 27hJlV3i6vaLzjVExArje9x3O8rXx9eJb089ffiVsKQOY2uruR6Jn1xncEvniRX/NQ4L PvH+BXebwetKbBvOYqLCo8sC67XltJrs2Sq/AX79Fzl7Qxp9+1H9720Pq5PDyGSUPrYH nRKFKl0YV2R5ZqcyXSLDOB5EsumqnRVZn3JlETPg/TUGFNCU7uMMZkhQF92QFJC71PjJ ik4g== X-Gm-Message-State: AOAM531NoI3xf4ANMD0k9LL4LT3b7R+4cJ8QJYayYICmZcj7vyVLrN+M rJKwww4uxkcqlrmT7pF/0cqdKQ== X-Received: by 2002:a17:906:3b42:: with SMTP id h2mr1147665ejf.647.1644581626601; Fri, 11 Feb 2022 04:13:46 -0800 (PST) Received: from localhost ([2620:10d:c093:400::5:bc0d]) by smtp.gmail.com with ESMTPSA id i22sm5828777ejx.128.2022.02.11.04.13.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Feb 2022 04:13:45 -0800 (PST) Date: Fri, 11 Feb 2022 12:13:45 +0000 From: Chris Down To: Shakeel Butt Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2 4/4] memcg: synchronously enforce memory.high for large overcharges Message-ID: References: <20220211064917.2028469-1-shakeelb@google.com> <20220211064917.2028469-5-shakeelb@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20220211064917.2028469-5-shakeelb@google.com> User-Agent: Mutt/2.1.5 (31b18ae9) (2021-12-30) X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Shakeel Butt writes: >The high limit is used to throttle the workload without invoking the >oom-killer. Recently we tried to use the high limit to right size our >internal workloads. More specifically dynamically adjusting the limits >of the workload without letting the workload get oom-killed. However due >to the limitation of the implementation of high limit enforcement, we >observed the mechanism fails for some real workloads. > >The high limit is enforced on return-to-userspace i.e. the kernel let >the usage goes over the limit and when the execution returns to >userspace, the high reclaim is triggered and the process can get >throttled as well. However this mechanism fails for workloads which do >large allocations in a single kernel entry e.g. applications that >mlock() a large chunk of memory in a single syscall. Such applications >bypass the high limit and can trigger the oom-killer. > >To make high limit enforcement more robust, this patch makes the limit >enforcement synchronous only if the accumulated overcharge becomes >larger than MEMCG_CHARGE_BATCH. So, most of the allocations would still >be throttled on the return-to-userspace path but only the extreme >allocations which accumulates large amount of overcharge without >returning to the userspace will be throttled synchronously. The value >MEMCG_CHARGE_BATCH is a bit arbitrary but most of other places in the >memcg codebase uses this constant therefore for now uses the same one. Note that mem_cgroup_handle_over_high() has its own allocator throttling grace period, where it bails out if the penalty to apply is less than 10ms. The reclaim will still happen, though. So throttling might not happen even for roughly MEMCG_CHARGE_BATCH-sized allocations, depending on the overall size of the cgroup and its protection. >Signed-off-by: Shakeel Butt >--- >Changes since v1: >- Based on Roman's comment simply the sync enforcement and only target > the extreme cases. > > mm/memcontrol.c | 5 +++++ > 1 file changed, 5 insertions(+) > >diff --git a/mm/memcontrol.c b/mm/memcontrol.c >index 292b0b99a2c7..0da4be4798e7 100644 >--- a/mm/memcontrol.c >+++ b/mm/memcontrol.c >@@ -2703,6 +2703,11 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, > } > } while ((memcg = parent_mem_cgroup(memcg))); > >+ if (current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH && >+ !(current->flags & PF_MEMALLOC) && >+ gfpflags_allow_blocking(gfp_mask)) { >+ mem_cgroup_handle_over_high(); Thanks, I was going to comment on v1 that I prefer to keep the implementation of mem_cgroup_handle_over_high if possible since we know that the mechanism has been safe in production over the past few years. One question I have is about throttling. It looks like this new mem_cgroup_handle_over_high callsite may mean that throttling is invoked more than once on a misbehaving workload that's failing to reclaim since the throttling could be invoked both here and in return to userspace, right? That might not be a problem, but we should think about the implications of that, especially in relation to MEMCG_MAX_HIGH_DELAY_JIFFIES. Maybe we should record if throttling happened previously and avoid doing it again for this entry into kernelspace? Not certain that's the right answer, but we should think about what the new semantics should be. >+ } > return 0; > } > >-- >2.35.1.265.g69c8d7142f-goog >