Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Sender: Tejun Heo <htejun@gmail.com>
Date:   Mon, 31 Oct 2022 15:45:59 -1000
From:   Tejun Heo <tj@kernel.org>
To:     Josh Don <joshdon@google.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        linux-kernel@vger.kernel.org,
        Joel Fernandes <joel@joelfernandes.org>
Subject: Re: [PATCH v2] sched: async unthrottling for cfs bandwidth
Message-ID: <Y2B6V1PPuCcTXGp6@slm.duckdns.org>
References: <20221026224449.214839-1-joshdon@google.com>
 <Y1/HzzA1FIawYM11@hirez.programming.kicks-ass.net>
 <CABk29Nu=XcjwRxnGBtKHfknxnDPpspghou06+W0fufnkGF6NkA@mail.gmail.com>
 <Y2BDFNpkSawKnE9S@slm.duckdns.org>
 <CABk29Nta-RJpTcybgOk9u4DH=1mwQFZsOxFuQ-UpCcTwzjzAuA@mail.gmail.com>
 <Y2Bf+CeQ8x2jKQ3S@slm.duckdns.org>
 <CABk29Nvqv-T1JuAq2cf9=AwRu=y1+YOR4xS2qnVo6+XpWd2UNQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CABk29Nvqv-T1JuAq2cf9=AwRu=y1+YOR4xS2qnVo6+XpWd2UNQ@mail.gmail.com>
Precedence: bulk

On Mon, Oct 31, 2022 at 06:01:19PM -0700, Josh Don wrote:
> > Yeah, especially with narrow cpuset (or task cpu affinity) configurations,
> > it can get pretty bad. Outside that tho, at least I haven't seen a lot of
> > problematic cases as long as the low priority one isn't tightly entangled
> > with high priority tasks, mostly because 1. if the resource the low pri one
> > is holding affects large part of the system, the problem is self-solving as
> > the system quickly runs out of other things to do 2. if the resource isn't
> > affecting large part of the system, their blast radius is usually reasonably
> > confined to things tightly coupled with it. I'm sure there are exceptions
> > and we definitely wanna improve the situation where it makes sense.
> 
> cgroup_mutex and kernfs rwsem beg to differ :) These are shared with
> control plane threads, so it is pretty easy to starve those out even
> while the system has plenty of work to do.

Hahaha yeah, good point. We definitely wanna improve them. There were some
efforts to improve kernfs locking granularity earlier this year. It was
promising but didn't get to the finish line. cgroup_mutex, w/ cgroup2 and
especially with the optimizations around CLONE_INTO_CGROUP, we avoid that in
most hot paths and hopefully that should help quite a bit. If it continues
to be a problem, we definitely wanna further improve it.

Just to better understand the situation, can you give some more details on
the scenarios where cgroup_mutex was in the middle of a shitshow?

Thanks.

-- 
tejun