Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From:   Joel Fernandes <joel@joelfernandes.org>
Mime-Version: 1.0 (1.0)
Subject: Re: [RFC PATCH 00/11] Reviving the Proxy Execution Series
Date:   Mon, 17 Oct 2022 08:27:48 -0400
Message-Id: <4D33BE61-16B9-4E70-9781-FB8F3C791FCA@joelfernandes.org>
References: <5ea4949e-3e8b-2ec0-bdcf-93e5744caee1@bytedance.com>
Cc:     Connor O'Brien <connoro@google.com>, linux-kernel@vger.kernel.org,
        kernel-team@android.com, John Stultz <jstultz@google.com>,
        Joel Fernandes <joelaf@google.com>,
        Qais Yousef <qais.yousef@arm.com>,
        Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        Will Deacon <will@kernel.org>,
        Waiman Long <longman@redhat.com>,
        Boqun Feng <boqun.feng@gmail.com>,
        "Paul E . McKenney" <paulmck@kernel.org>
In-Reply-To: <5ea4949e-3e8b-2ec0-bdcf-93e5744caee1@bytedance.com>
To:     Chengming Zhou <zhouchengming@bytedance.com>
Precedence: bulk


> On Oct 17, 2022, at 12:26 AM, Chengming Zhou <zhouchengming@bytedance.com>=
 wrote:
>=20
> =EF=BB=BFOn 2022/10/17 11:56, Joel Fernandes wrote:
>>=20
>>=20
>>>> On Oct 16, 2022, at 11:25 PM, Chengming Zhou <zhouchengming@bytedance.c=
om> wrote:
>>>=20
>>> =EF=BB=BFHello,
>>>=20
>>>> On 2022/10/4 05:44, Connor O'Brien wrote:
>>>> Proxy execution is an approach to implementing priority inheritance
>>>> based on distinguishing between a task's scheduler context (information=

>>>> required in order to make scheduling decisions about when the task gets=

>>>> to run, such as its scheduler class and priority) and its execution
>>>> context (information required to actually run the task, such as CPU
>>>> affinity). With proxy execution enabled, a task p1 that blocks on a
>>>> mutex remains on the runqueue, but its "blocked" status and the mutex o=
n
>>>> which it blocks are recorded. If p1 is selected to run while still
>>>> blocked, the lock owner p2 can run "on its behalf", inheriting p1's
>>>> scheduler context. Execution context is not inherited, meaning that
>>>> e.g. the CPUs where p2 can run are still determined by its own affinity=

>>>> and not p1's.
>>>=20
>>> This is cool. We have a problem (others should have encountered it too) t=
hat
>>> priority inversion happened when the rwsem writer is waiting for many re=
aders
>>> which held lock but are throttled by CFS bandwidth control. (In our use c=
ase,
>>> the rwsem is the mm_struct->mmap_sem)
>>>=20
>>> So I'm curious if this work can also solve this problem? If we don't deq=
ueue
>>> the rwsem writer when it blocked on the rwsem, then CFS scheduler pick i=
t to
>>> run, we can use blocked chain to find the readers to run?
>>=20
>> That seems a lot harder and unsupported by current patch set AFAICS (my e=
xposure to this work is about a week so take it with a grain of salt). You c=
ould have multiple readers so how would you choose which reader to proxy for=
 (round robin?).  Also, you no longer have a chain but a tree of chains, wit=
h the leaves being each reader - so you have to track that somehow, then kee=
p migrating the blocked tasks in the chain to each readers CPU. Possibly mig=
rating a lot more than in the case of a single chain. Also it=E2=80=99s not c=
lear if it will be beneficial as proxying for one reader does not mean you=E2=
=80=99re improving the situation if it is another reader that is in need of t=
he boost.
>>=20
>=20
> Thanks for your reply, it's indeed more complex than I think, and proxying=
 for just one reader
> is also less efficient.
>=20
> But this rwsem priority inversion problem hurts us so much that we are afr=
aid to use
> CFS bandwidth control now. Imaging when 10 readers held mmap_sem then thro=
ttled for 20ms,
> the writer will have to wait for at least 200ms, which become worse if the=
 writer held other lock.

I hear you. But on the other hand with so many readers the writer is bound t=
o starve anyway. Rwsem is unfair to the writer by definition. But yes, I agr=
ee PE (if made to) can help here. I suggest also look into the per-VMA locks=
 and maple tree work that Suren et all are doing, to improve the situation.

Thanks.

>=20
> Thanks.
>=20