Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <CAERHkrtvLKxrpvfw04urAuougsYOWnNw4-H1vUDFx27Dvy0=Ww@mail.gmail.com>
 <20190725143003.GA992@aaronlu> <20190726152101.GA27884@sinkpad>
 <7dc86e3c-aa3f-905f-3745-01181a3b0dac@linux.intel.com> <20190802153715.GA18075@sinkpad>
 <f4778816-69e5-146c-2a30-ec42e7f1677f@linux.intel.com> <20190806032418.GA54717@aaronlu>
 <CAERHkrtJ3f1ggfG7Qo-KnznGo66p0Y3E0sAfb3ki6U=ADT6__g@mail.gmail.com>
 <54fa27ff-69a7-b2ac-6152-6915f78a57f9@linux.alibaba.com> <CANaguZDPdUp3Nb7hYjEiTpJTMVrKJyw2JDKP5EEphMjV-PAYpA@mail.gmail.com>
 <20190806141617.GR2332@hirez.programming.kicks-ass.net>
In-Reply-To: <20190806141617.GR2332@hirez.programming.kicks-ass.net>
From:   Vineeth Remanan Pillai <vpillai@digitalocean.com>
Date:   Tue, 6 Aug 2019 11:53:29 -0400
Message-ID: <CANaguZC3v-x7O0RV3G7vk6p4BdW1E4uxBqf9NA=MOipeBWt+fw@mail.gmail.com>
Subject: Re: [RFC PATCH v3 00/16] Core scheduling v3
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     Aaron Lu <aaron.lu@linux.alibaba.com>,
        Aubrey Li <aubrey.intel@gmail.com>,
        Tim Chen <tim.c.chen@linux.intel.com>,
        Julien Desfossez <jdesfossez@digitalocean.com>,
        "Li, Aubrey" <aubrey.li@linux.intel.com>,
        Subhra Mazumdar <subhra.mazumdar@oracle.com>,
        Nishanth Aravamudan <naravamudan@digitalocean.com>,
        Ingo Molnar <mingo@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Paul Turner <pjt@google.com>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Linux List Kernel Mailing <linux-kernel@vger.kernel.org>,
        =?UTF-8?B?RnLDqWTDqXJpYyBXZWlzYmVja2Vy?= <fweisbec@gmail.com>,
        Kees Cook <keescook@chromium.org>,
        Greg Kerr <kerrnel@google.com>, Phil Auld <pauld@redhat.com>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        Pawan Gupta <pawan.kumar.gupta@linux.intel.com>,
        Paolo Bonzini <pbonzini@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

>
> What accounting in particular is upset? Is it things like
> select_idle_sibling() that thinks the thread is idle and tries to place
> tasks there?
>
The major issue that we saw was, certain work load causes the idle cpu to never
wakeup and schedule again even when there are runnable threads in there. If
I remember correctly, this happened when the sibling had only one cpu intensive
task and did not enter the pick_next_task for a long time. There were other
situations as well which caused this prolonged idle state on the cpu.
One was when
pick_next_task was called on the sibling but it always won there
because vruntime
was not progressing on the idle cpu.

Having a coresched idle makes sure that the idle thread is not overloaded. Also
vruntime moves forward and tsk vruntime comparison across cpus works when
we normalize.

> It should be possible to change idle_cpu() to not report a forced-idle
> CPU as idle.
I agree. If we can identify all the places the idle thread is
considered special and
also account for the vruntime progress for force idle, this should be a better
approach compared to coresched idle thread per cpu.

>
> (also; it should be possible to optimize select_idle_sibling() for the
> core-sched case specifically)
>
We haven't seen this because, most of our micro test cases did not have more
threads than the cpus. Thanks for pointing this out, we shall cook some tests
to observe this behavior.

Thanks,
Vineeth