Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "King, Colin" <colin.king@intel.com>
To:     "Eric W. Biederman" <ebiederm@xmission.com>,
        Luis Chamberlain <mcgrof@kernel.org>
CC:     Shakeel Butt <shakeelb@google.com>,
        Colin Ian King <colin.king@canonical.com>,
        NeilBrown <neilb@suse.de>, Vasily Averin <vvs@virtuozzo.com>,
        Vlastimil Babka <vbabka@suse.cz>,
        "Hocko, Michal" <mhocko@suse.com>,
        Roman Gushchin <roman.gushchin@linux.dev>,
        Linux MM <linux-mm@kvack.org>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "David S. Miller" <davem@davemloft.net>,
        Jakub Kicinski <kuba@kernel.org>, Tejun Heo <tj@kernel.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Eric Dumazet <edumazet@google.com>,
        Kees Cook <keescook@chromium.org>,
        Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>,
        David Ahern <dsahern@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "kernel@openvz.org" <kernel@openvz.org>
Subject: RE: [PATCH RFC] net: memcg accounting for veth devices
Thread-Topic: [PATCH RFC] net: memcg accounting for veth devices
Thread-Index: AQHYLbQc4Rbuq/ZYIUySV5ooMqdS86yrDNdjgAEJZiA=
Date:   Wed, 2 Mar 2022 13:30:16 +0000
Message-ID: <BY5PR11MB4102FBFC3E9FAA8371787C358D039@BY5PR11MB4102.namprd11.prod.outlook.com>
References: <a5e09e93-106d-0527-5b1e-48dbf3b48b4e@virtuozzo.com>
        <YhzeCkXEvga7+o/A@bombadil.infradead.org>
        <20220301180917.tkibx7zpcz2faoxy@google.com>
        <Yh5lyr8dJXmEoFG6@bombadil.infradead.org>
 <87wnhdwg75.fsf@email.froward.int.ebiederm.org>
In-Reply-To: <87wnhdwg75.fsf@email.froward.int.ebiederm.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?O45oI228W7jBei2TyfyyaUTCwfZQE1Bzl52BJ/FPWgN3UPmq0Xj/8Eyi4Cc7?=
 =?us-ascii?Q?DnFGWknXDvN4roK76BDy0x8oo1F3dgH7tRRzlI9nPJ1/uYtUJZPbcA5zNAQN?=
 =?us-ascii?Q?E5TCupzcWG2IfRtwOgOBGcvQfFLtTB8uAZNNLovZCfh1l2umnbTOSsZ+HO6L?=
 =?us-ascii?Q?Z+HBRQtJJdkBia8VE7YUvKRrr1gCRBb3GdZblQ8wiovFs8wvFPbBNkbSyQmC?=
 =?us-ascii?Q?3+M3Qj7pwNfl1hfClOQYYfR8f5KT5ePolqon8lhTp1A9fV5zb73sqkOOXKVG?=
 =?us-ascii?Q?nOaNssLXL4AxpBMziKbrN2IVgy997Q8f3MY3xfDJis/Gn2JxeBXbHffvv535?=
 =?us-ascii?Q?zT8sjNsN0SuIEwaoL2ZGtYMyO4exT2qPSWUwwtUhi0iBLzDreY5GL/wV/UQC?=
 =?us-ascii?Q?zdnsSFCZJ+d4s4YecrvGK+GL4z2OpBkk9ZvfOMhlBHNcrckb2cvDaT9dqweU?=
 =?us-ascii?Q?CTit8Ku9hin0RBx3px5qFFAA3AIuLFOCDzv3b/w/qXxtX7v6aulLXpnpSVNn?=
 =?us-ascii?Q?Gkebz+0e8GXytallZ9vdfj20a1XTmhbF07dcKCcWLaxoBCmXUWN1rW3bSvba?=
 =?us-ascii?Q?GtyhqwgTkp4Ov9r7u8H9AH4f4ebSEqRIn4GXkBCxo0hD7CyO4lCoc0oxFPKU?=
 =?us-ascii?Q?oHDQuv81wx2/a5x0riJ0aj1bwlUTc5IBeIk1W2kT/DLvn+DugZaGKHS7dcG0?=
 =?us-ascii?Q?Zlo6Wb6UUhy/+c0W7zqJjHpxYHTjEZAvP2qf+XFdMB/uFhoK4gzQsptw9wv8?=
 =?us-ascii?Q?R8JQqOPyJ77uWBWbwfpgdEC7E9Na3T3JYlySw0Dv14qvslnlaeJxx/QDL1dU?=
 =?us-ascii?Q?VwJWJhmXbCK3sm4pe7it1Xo6ZsfeAJWNqQc23WsCNtttO/ngSQX+u/MpUG7w?=
 =?us-ascii?Q?SMtGuf5uHu27VSUDNTh6bSw5tzsCE3lC7FNO2Ma9fzBmLBGgMe/gir6r2RWL?=
 =?us-ascii?Q?mCj/Z4bdbe8nl9f2L1qwwhzw3G8hX/AKO4EZbsbQJTr3cH92JrUhBtUTB5+l?=
 =?us-ascii?Q?MPEPqRPJYivO9fHftaboZVUrMp0tub8B10fDe2MG2WfTfZSjhJbxTiCAra3d?=
 =?us-ascii?Q?l4yKSaBORFJZ4/fRc1sfM1I69wn1satgtgrTFTud2oIB02C2ThUsHhTkgzB+?=
 =?us-ascii?Q?dwmA3c6BDo6J34Y1gtWLCJNN9J65ZZLwtxTKvuC8JZnmIvsl8PQ39kx7RNz0?=
 =?us-ascii?Q?6rhUeqGoMYDs7QptDIuQzfrUfncE7yq+/EZJnsXu7JbCRyKSYsaDUTXpZ5u/?=
 =?us-ascii?Q?2BQW1RYfGBWuxYld5Jnlg277b1qcqh1CYdz21nX97mj2npfseeSlnCKGTrJ3?=
 =?us-ascii?Q?XHXKT+Lada8Ydjk9M2ZKyASORvlZTmxP75TRLtclIWKtKqfqz02hKlcS+P3V?=
 =?us-ascii?Q?xl8Z5bBudbTKjLYfQK6hIB8ABR8BWvonnzwjGm1TgcQvRRcEo2r/MsC9QLC0?=
 =?us-ascii?Q?tjfmy463nxTO1u86n2DEintpjinQd/YqiKsx4l+Izjr3/QuJb7QQGrunRL7c?=
 =?us-ascii?Q?0ozAl6rpS34G7z0S3UvHd+0WSmkj1XV3Z/SOWDKNLBH9vZw5PxY+W1JdhfQb?=
 =?us-ascii?Q?XikDPnBVglnVQ9tZLSL2D1WU2gCP9AxahIhB4w1SWLFfRY5OhXfTS0vHud0e?=
 =?us-ascii?Q?aID+Cujx8pacStuB/vDVC0Y=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BY5PR11MB4102.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 76f7a2e9-5219-42ff-d522-08d9fc50ca26
X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Mar 2022 13:30:16.7667
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: bQ5sZbnDAusq1eismbRNsjhGiabe/dV1xWPSwgI+EH8kPndyKK9JfpalWA9c4+Lm30Jd/lQ/EdB4wXbeN1xP3g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR11MB3348
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


Just to note that stress-ng does attempt to set the maximum ulimits across =
all ulimits before it invokes a stressor to try and stress the system as mu=
ch as possible. It also changes the per-stressor oom adjust setting to make=
 stressors less OOMable, one can disable this with --no-oom-adjust and one =
can force stressors from being restarted on an OOM with the --oomable optio=
n.


-----Original Message-----
From: Eric W. Biederman <ebiederm@xmission.com>=20
Sent: 01 March 2022 20:50
To: Luis Chamberlain <mcgrof@kernel.org>
Cc: Shakeel Butt <shakeelb@google.com>; Colin Ian King <colin.king@canonica=
l.com>; NeilBrown <neilb@suse.de>; Vasily Averin <vvs@virtuozzo.com>; Vlast=
imil Babka <vbabka@suse.cz>; Hocko, Michal <mhocko@suse.com>; Roman Gushchi=
n <roman.gushchin@linux.dev>; Linux MM <linux-mm@kvack.org>; netdev@vger.ke=
rnel.org; David S. Miller <davem@davemloft.net>; Jakub Kicinski <kuba@kerne=
l.org>; Tejun Heo <tj@kernel.org>; Greg Kroah-Hartman <gregkh@linuxfoundati=
on.org>; Eric Dumazet <edumazet@google.com>; Kees Cook <keescook@chromium.o=
rg>; Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>; David Ahern <dsahern@kern=
el.org>; linux-kernel@vger.kernel.org; kernel@openvz.org
Subject: Re: [PATCH RFC] net: memcg accounting for veth devices

Luis Chamberlain <mcgrof@kernel.org> writes:

> On Tue, Mar 01, 2022 at 10:09:17AM -0800, Shakeel Butt wrote:
>> On Mon, Feb 28, 2022 at 06:36:58AM -0800, Luis Chamberlain wrote:
>> > On Mon, Feb 28, 2022 at 10:17:16AM +0300, Vasily Averin wrote:
>> > > Following one-liner running inside memcg-limited container=20
>> > > consumes huge number of host memory and can trigger global OOM.
>> > >
>> > > for i in `seq 1 xxx` ; do ip l a v$i type veth peer name vp$i ;=20
>> > > done
>> > >
>> > > Patch accounts most part of these allocations and can protect host.
>> > > ---[cut]---
>> > > It is not polished, and perhaps should be splitted.
>> > > obviously it affects other kind of netdevices too.
>> > > Unfortunately I'm not sure that I will have enough time to handle=20
>> > > it
>> > properly
>> > > and decided to publish current patch version as is.
>> > > OpenVz workaround it by using per-container limit for number of=20
>> > > available netdevices, but upstream does not have any kind of=20
>> > > per-container configuration.
>> > > ------
>>=20
>> > Should this just be a new ucount limit on kernel/ucount.c and have=20
>> > veth use something like inc_ucount(current_user_ns(),=20
>> > current_euid(), UCOUNT_VETH)?
>>=20
>> > This might be abusing ucounts though, not sure, Eric?
>>=20
>>=20
>> For admins of systems running multiple workloads, there is no easy=20
>> way to set such limits for each workload.
>
> That's why defaults would exist. Today's ulimits IMHO are insane and=20
> some are arbitrarily large.

My perspective is that we have two basic kinds of limits.

Limits to catch programs that go out of control hopefully before they bring=
 down the entire system.  This is the purpose I see of rlimits and ucounts.=
  Such limits should be set by default so large that no one has to care unl=
ess their program is broken.

Limits to contain programs and keep them from having a negative impact on o=
ther programs.  Generally this is the role I see the cgroups playing.  This=
 limits must be much more tightly managed.

The problem with veth that was reported was that the memory cgroup limits f=
ails to contain veth's allocations and veth manages to affect process outsi=
de the memory cgroup where the veth ``lives''.  The effect is an OOM but th=
e problem is that it is affecting processes out of the memory control group=
.

Part of the reason for the recent ucount work is so that ordinary users can=
 create user namespaces and root in that user namespace won't be able to ex=
ceed the limits that were set when the user namespace was created by creati=
ng additional users.

Part of the reason for my ucount work is my frustration that cgroups would =
up something completely different than what was originally proposed and sol=
ve a rather problem set.  Originally the proposal was that cgroups would be=
 the user interface for the bean-counter patches.
(Roughly counts like the ucounts are now).  Except for maybe the pid contro=
ller you mention below cgroups look nothing like that today.
So I went and I solved the original problem because it was still not solved=
.

The network stack should already have packet limits to prevent a global OOM=
 so I am a bit curious why those limits aren't preventing a global OOM in f=
or the veth device.


I am not saying that the patch is correct (although from 10,000 feet the pa=
tch sounds like it is solving the reported problem).  I am answering the qu=
estion of how I understand limits to work.

Luis does this explanation of how limits work help?


>> From admin's perspective it is preferred to have minimal knobs to set=20
>> and if these objects are charged to memcg then the memcg limits would=20
>> limit them. There was similar situation for inotify instances where=20
>> fs sysctl inotify/max_user_instances already limits the inotify=20
>> instances but we memcg charged them to not worry about setting such=20
>> limits. See ac7b79fd190b ("inotify, memcg: account inotify instances=20
>> to kmemcg").
>
> Yes but we want sensible defaults out of the box. What those should be=20
> IMHO might be work which needs to be figured out well.
>
> IMHO today's ulimits are a bit over the top today. This is off=20
> slightly off topic but for instance play with:
>
> git clone https://github.com/ColinIanKing/stress-ng
> cd stress-ng
> make -j 8
> echo 0 > /proc/sys/vm/oom_dump_tasks                                     =
      =20
> i=3D1; while true; do echo "RUNNING TEST $i"; ./stress-ng --unshare 8192=
=20
> --unshare-ops 10000;  sleep 1; let i=3D$i+1; done
>
> If you see:
>
> [  217.798124] cgroup: fork rejected by pids controller in=20
> /user.slice/user-1000.slice/session-1.scope
>                                                                          =
      =20
> Edit /usr/lib/systemd/system/user-.slice.d/10-defaults.conf to be:
>
> [Slice]                                                                  =
      =20
> TasksMax=3DMAX_TASKS|infinity
>
> Even though we have max_threads set to 61343, things ulimits have a=20
> different limit set, and what this means is the above can end up=20
> easily creating over 1048576 (17 times max_threads) threads all=20
> eagerly doing nothing to just exit, essentially allowing a sort of fork b=
omb on exit.
> Your system may or not fall to its knees.


What max_threads are you talking about here?  The global max_threads expose=
d in /proc/sys/kernel/threads-max?  I don't see how you can get around that=
.  Especially since the count is not decremented until the process is reape=
d.

Or is this the pids controller having a low limit and /proc/sys/kernel/thre=
ads-max having a higher limit?

I really have not looked at this pids controller.

So I am not certain I understand your example here but I hope I have answer=
ed your question.

Eric