Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [RFC PATCH v3 00/16] Core scheduling v3
To:     Julien Desfossez <jdesfossez@digitalocean.com>,
        Aaron Lu <aaron.lu@linux.alibaba.com>
Cc:     Aubrey Li <aubrey.intel@gmail.com>,
        Subhra Mazumdar <subhra.mazumdar@oracle.com>,
        Vineeth Remanan Pillai <vpillai@digitalocean.com>,
        Nishanth Aravamudan <naravamudan@digitalocean.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Tim Chen <tim.c.chen@linux.intel.com>,
        Ingo Molnar <mingo@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Paul Turner <pjt@google.com>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Linux List Kernel Mailing <linux-kernel@vger.kernel.org>,
        =?UTF-8?B?RnLDqWTDqXJpYyBXZWlzYmVja2Vy?= <fweisbec@gmail.com>,
        Kees Cook <keescook@chromium.org>,
        Greg Kerr <kerrnel@google.com>, Phil Auld <pauld@redhat.com>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        Pawan Gupta <pawan.kumar.gupta@linux.intel.com>,
        Paolo Bonzini <pbonzini@redhat.com>
References: <20190531210816.GA24027@sinkpad> <20190606152637.GA5703@sinkpad>
 <20190612163345.GB26997@sinkpad>
 <635c01b0-d8f3-561b-5396-10c75ed03712@oracle.com>
 <20190613032246.GA17752@sinkpad>
 <CAERHkrsMFjjBpPZS7jDhzbob4PSmiPj83OfqEeiKgaDAU3ajOA@mail.gmail.com>
 <20190619183302.GA6775@sinkpad> <20190718100714.GA469@aaronlu>
 <CAERHkrtvLKxrpvfw04urAuougsYOWnNw4-H1vUDFx27Dvy0=Ww@mail.gmail.com>
 <20190725143003.GA992@aaronlu> <20190726152101.GA27884@sinkpad>
From:   "Li, Aubrey" <aubrey.li@linux.intel.com>
Message-ID: <7dc86e3c-aa3f-905f-3745-01181a3b0dac@linux.intel.com>
Date:   Wed, 31 Jul 2019 10:42:15 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.1.1
MIME-Version: 1.0
In-Reply-To: <20190726152101.GA27884@sinkpad>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 2019/7/26 23:21, Julien Desfossez wrote:
> On 25-Jul-2019 10:30:03 PM, Aaron Lu wrote:
>>
>> I tried a different approach based on vruntime with 3 patches following.
> [...]
> 
> We have experimented with this new patchset and indeed the fairness is
> now much better. Interactive tasks with v3 were complete starving when
> there were cpu-intensive tasks running, now they can run consistently.

Yeah, the fairness is much better now.

For two cgroups created, I limited the cpuset to be one core(two siblings)
of these two cgroups, I still run gemmbench and sysbench-mysql, and here
is the mysql result:

Latency:
.----------------------------------------------------------------------------------------------.
|NA/AVX	vanilla-SMT	[std% / sem%]	  cpu% |coresched-SMT	[std% / sem%]	  +/-	  cpu% |
|----------------------------------------------------------------------------------------------|
|  1/1	        6.7	[13.8%/ 1.4%]	  2.1% |          6.4	[14.6%/ 1.5%]	  4.0%	  2.0% |
|  2/2	        9.1	[ 5.0%/ 0.5%]	  4.0% |         11.4	[ 6.8%/ 0.7%]	-24.9%	  3.9% |
'----------------------------------------------------------------------------------------------'

Throughput:
.----------------------------------------------------------------------------------------------.
|NA/AVX	vanilla-SMT	[std% / sem%]	  cpu% |coresched-SMT	[std% / sem%]	  +/-	  cpu% |
|----------------------------------------------------------------------------------------------|
|  1/1	      310.2	[ 4.1%/ 0.4%]	  2.1% |        296.2	[ 5.0%/ 0.5%]	 -4.5%	  2.0% | 
|  2/2	      547.7	[ 3.6%/ 0.4%]	  4.0% |        368.3	[ 4.8%/ 0.5%]	-32.8%	  3.9% | 
'----------------------------------------------------------------------------------------------'

Note: 2/2 case means 4 threads run on one core, which is overloaded.(cpu% is overall system report)

Though the latency/throughput has regressions, but standard deviation is much better now.

> With my initial test of TPC-C running in large VMs with a lot of
> background noise VMs, the results are pretty similar to v3, I will run
> more thorough tests and report the results back here.

I see something similar. I guess task placement could be another problem.
We don't check cookie matching in load balance and task wakeup, so
- if tasks with different cookie happen to be dispatched onto different cores,
The result should be good
- if tasks with different cookie are unfortunately dispatched onto the same
core, the result should be bad.

This problem is bypassed in my testing setup above, but may be one cause
of my other scenarios, need a while to sort out.

Thanks,
-Aubrey