Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp5069481yba; Wed, 10 Apr 2019 10:38:49 -0700 (PDT) X-Google-Smtp-Source: APXvYqy59gUVrnd/cHOdByonfw+FfDvhX2Ddx8J8AOHu/AfOijvyRfF+Np8a32pQ7yhYU+uzYnFR X-Received: by 2002:a17:902:b210:: with SMTP id t16mr40204811plr.84.1554917929390; Wed, 10 Apr 2019 10:38:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554917929; cv=none; d=google.com; s=arc-20160816; b=qIFBXNXZFuFLVAI8EiWSy/9p2vMh51ZUspWJEgUKpaRZyx2oFMhDn3n1bn2LnKXbyG 1Gt+g+9qstCJv2ScKYMOtBAsAMl8VmPMIrj6sfIe5Pqj+KIqVjqXuXslmno1qzr3R/Pu QlJk+Hz0auF8O4Mv1L+h9WzC16JrrRGuJKzMyTn9KNT4RFA+9uFN6EE9cA1y9nE+Oc6p Rs2XeZ3JWUapzBEP006sFsQ14pnWbf57W5gjoOaDCHntgPi0eSJrCh+O4oLUoCELUj4i dnmeL3j9QCEddYtyU1vBEKKG1lZ8rKfMhHux1aXTKNp1RdgMJhdlHDUh7tigCSGeLgAW +MAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=t8ABffEdZGV5cihLZzSHe2fhMN6ofhtObW3edkrWfCs=; b=MWrKeN20RXyu+X7feIrqRZv4M1reCgjTeqKQnst3u9flXIzQPXY2DtDBcysjv8wSG1 dILkrVpANgrcgSlmmylg4RJdiCRqkntrVLNNiDCxgcvnfmmtSf9G4GjKxIBUAJv9QQF1 VY5i2RfG0RFBsR01b4P2s53TekfsQ/9Zu/onzSKy5GD6Y5tLJTyOPwTzSxMiftyrDN0z xQWbliLcIjQP2f3z4s4YJGi/P99hL/4718NrMAbMCKJgbXj6FTZMUsTCRqwrx+DxcTvb rAnOEUIX5lBTzAhLz0FCyLk4Tist7i5r5ALDCkQOgOUkyKb+rx9DSITcjLreuZVpHXbr I7tg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=F0XoHP8u; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id d3si20009211pgc.472.2019.04.10.10.38.33; Wed, 10 Apr 2019 10:38:49 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=F0XoHP8u; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732229AbfDJOQe (ORCPT + 99 others); Wed, 10 Apr 2019 10:16:34 -0400 Received: from mail-yb1-f196.google.com ([209.85.219.196]:42908 "EHLO mail-yb1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729037AbfDJOQe (ORCPT ); Wed, 10 Apr 2019 10:16:34 -0400 Received: by mail-yb1-f196.google.com with SMTP id l124so863817ybf.9 for ; Wed, 10 Apr 2019 07:16:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=t8ABffEdZGV5cihLZzSHe2fhMN6ofhtObW3edkrWfCs=; b=F0XoHP8usrEVkRqi/39Q5G7p1Znhdzs54MQuTynfelohW8v07Lvor7o0OFxpPdsaxI 0QomlGzN85H47lfms4oFdvxrzaB7VgL/67pa4iaTJenTZZ7fJU5k5tXJNKis7AVlAO/i +n4ET2R6ywH9/I2tkRLxZ8HS0g0ca+rZu3+9jyHleEfyyX0oCOZ2jWojccTgcoQBP490 vhHm+qQcT3Fd7qVjbo1Mr2C4GSH7wDeEZV4QxjBnOuqOaX+vHLKx7jBVgJDscyVBeae6 WnenZJ1MR9RdSJ+PV8iLsKrPoF/bDO2KDmFPmhKPnnVXKhtUtoaK9Hq5u+0xy4ZUeaYY Gcxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=t8ABffEdZGV5cihLZzSHe2fhMN6ofhtObW3edkrWfCs=; b=V8/NO9YoRvpUz9K5sw7bwos1OV52YDNlZBSH8DW3e9klt2yzo3NiW7R21N4d3DQJR/ zrWFHSOmChEfnrdZ1ORJDm/X03qto5jR6XWB3gwLU2K4DAKE3wezVVOEBT/Tr8675ceK ShBgodoucnbKt6MLOxKizLbhOiV5SeLrB+0eawuaLt3Sk3ST92fm9nmtgA5yYSTtzNPD q2Xx4HBn0s0UWiJbKKwalr21CR1Y8d0kJ25FgMXoWc8pwNBSGo69uxHhrVecnZrwoiXe HsypUcSRnvN9kaLNr5HT7xrP60AGohFmW12tkKZdv5RT5s6/Yh8dG6dt2hbYFXxqNC+n Meag== X-Gm-Message-State: APjAAAVSqgivViJeZGKjhCcsagQMRYlWizaz3K7C9ptJmuX0PvQPLMdG SM727n4dVb7u4y3c+4AwyeZ5SwJtyrszHbczijNbZA== X-Received: by 2002:a25:1a8b:: with SMTP id a133mr21702120yba.428.1554905792918; Wed, 10 Apr 2019 07:16:32 -0700 (PDT) MIME-Version: 1.0 References: <20190410101947.8603-1-juha-matti.tilli@foreca.com> In-Reply-To: <20190410101947.8603-1-juha-matti.tilli@foreca.com> From: Eric Dumazet Date: Wed, 10 Apr 2019 07:16:21 -0700 Message-ID: Subject: Re: [PATCH] net: add big honking pfmemalloc OOM warning To: Juha-Matti Tilli Cc: LKML , netdev , Rafael Aquini , Murphy Zhou , Yongcheng Yang , Jianhong Yin Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Apr 10, 2019 at 3:20 AM Juha-Matti Tilli wrote: > > A system administrator is not notified (except via an obscure SNMP > counter that most sysadmins don't know to look for) if packets are > dropped due to out-of-memory condition when SKBs use pfmemalloc > reserves. This can for example lead to NFS connections hanging on > high-volume systems. > > Implement a ratelimited big honking out of memory warning that directs > the sysadmin to bump up vm.min_free_kbytes in case this problem happens. > Our experience shows that with default vm.min_free_kbytes (90112, or > about 90 megabytes), NFS connections hang approximately once per day, > whereas with 901120 (default multiplied by 10) NFS connections never > hang. > If NFS sessions hang, then there is a bug to eventually root cause and fix. Just telling the user : Increase the limit is the same thing than admitting : Our limit system or TCP or NFS stacks are broken and unable to recover, so lets disable the limit system and work around a more serious bug. Maybe the bug is in a NIC driver, please share more details before adding yet another noisy signal in syslog SNMP counters are per netns, and more useful in the modern computing era, where a host is shared by many different containers. > > Signed-off-by: Juha-Matti Tilli > Cc: Eric Dumazet > Cc: Rafael Aquini > Cc: Murphy Zhou > Cc: Yongcheng Yang > Cc: Jianhong Yin > --- > net/core/filter.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/net/core/filter.c b/net/core/filter.c > index fc92ebc4e200..7d8ef239b9af 100644 > --- a/net/core/filter.c > +++ b/net/core/filter.c > @@ -99,6 +99,8 @@ int sk_filter_trim_cap(struct sock *sk, struct sk_buff *skb, unsigned int cap) > * helping free memory > */ > if (skb_pfmemalloc(skb) && !sock_flag(sk, SOCK_MEMALLOC)) { > + printk_ratelimited(KERN_WARNING > + "dropped packet due to out-of-memory condition, please bump up vm.min_free_kbytes\n"); > NET_INC_STATS(sock_net(sk), LINUX_MIB_PFMEMALLOCDROP); > return -ENOMEM; > } > -- > 2.17.1 >