Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Fri, 6 Nov 2020 12:54:27 -0500
From:   Joel Fernandes <joel@joelfernandes.org>
To:     "Li, Aubrey" <aubrey.li@linux.intel.com>
Cc:     "Ning, Hongyu" <hongyu.ning@linux.intel.com>,
        Nishanth Aravamudan <naravamudan@digitalocean.com>,
        Julien Desfossez <jdesfossez@digitalocean.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Tim Chen <tim.c.chen@linux.intel.com>,
        Vineeth Pillai <viremana@linux.microsoft.com>,
        Aaron Lu <aaron.lwe@gmail.com>,
        Aubrey Li <aubrey.intel@gmail.com>, tglx@linutronix.de,
        linux-kernel@vger.kernel.org, mingo@kernel.org,
        torvalds@linux-foundation.org, fweisbec@gmail.com,
        keescook@chromium.org, kerrnel@google.com,
        Phil Auld <pauld@redhat.com>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        Pawan Gupta <pawan.kumar.gupta@linux.intel.com>,
        Paolo Bonzini <pbonzini@redhat.com>, vineeth@bitbyteword.org,
        Chen Yu <yu.c.chen@intel.com>,
        Christian Brauner <christian.brauner@ubuntu.com>,
        Agata Gruza <agata.gruza@intel.com>,
        Antonio Gomez Iglesias <antonio.gomez.iglesias@intel.com>,
        graf@amazon.com, konrad.wilk@oracle.com, dfaggioli@suse.com,
        pjt@google.com, rostedt@goodmis.org, derkling@google.com,
        benbjiang@tencent.com,
        Alexandre Chartre <alexandre.chartre@oracle.com>,
        James.Bottomley@hansenpartnership.com, OWeisse@umich.edu,
        Dhaval Giani <dhaval.giani@oracle.com>,
        Junaid Shahid <junaids@google.com>, jsbarnes@google.com,
        chris.hyser@oracle.com, Tim Chen <tim.c.chen@intel.com>
Subject: Re: [PATCH v8 -tip 00/26] Core scheduling
Message-ID: <20201106175427.GB2845264@google.com>
References: <20201020014336.2076526-1-joel@joelfernandes.org>
 <f7fc588b-12cf-95a8-6142-e4d112fb1689@linux.intel.com>
 <bf2ee997-1f53-0eef-40ad-1e98274da587@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <bf2ee997-1f53-0eef-40ad-1e98274da587@linux.intel.com>
Precedence: bulk

On Fri, Nov 06, 2020 at 10:58:58AM +0800, Li, Aubrey wrote:

> > 
> > 	-- workload D, new added syscall workload, performance drop in cs_on:
> > 	+----------------------+------+-------------------------------+
> > 	|                      | **   | will-it-scale  * 192          |
> > 	|                      |      | (pipe based context_switch)   |
> > 	+======================+======+===============================+
> > 	| cgroup               | **   | cg_will-it-scale              |
> > 	+----------------------+------+-------------------------------+
> > 	| record_item          | **   | threads_avg                   |
> > 	+----------------------+------+-------------------------------+
> > 	| coresched_normalized | **   | 0.2                           |
> > 	+----------------------+------+-------------------------------+
> > 	| default_normalized   | **   | 1                             |
> > 	+----------------------+------+-------------------------------+
> > 	| smtoff_normalized    | **   | 0.89                          |
> > 	+----------------------+------+-------------------------------+
> 
> will-it-scale may be a very extreme case. The story here is,
> - On one sibling reader/writer gets blocked and tries to schedule another reader/writer in.
> - The other sibling tries to wake up reader/writer.
> 
> Both CPUs are acquiring rq->__lock,
> 
> So when coresched off, they are two different locks, lock stat(1 second delta) below:
> 
> class name    con-bounces    contentions   waittime-min   waittime-max waittime-total   waittime-avg    acq-bounces   acquisitions   holdtime-min   holdtime-max holdtime-total   holdtime-avg
> &rq->__lock:          210            210           0.10           3.04         180.87           0.86            797       79165021           0.03          20.69    60650198.34           0.77
> 
> But when coresched on, they are actually one same lock, lock stat(1 second delta) below:
> 
> class name    con-bounces    contentions   waittime-min   waittime-max waittime-total   waittime-avg    acq-bounces   acquisitions   holdtime-min   holdtime-max holdtime-total   holdtime-avg
> &rq->__lock:      6479459        6484857           0.05         216.46    60829776.85           9.38        8346319       15399739           0.03          95.56    81119515.38           5.27
> 
> This nature of core scheduling may degrade the performance of similar workloads with frequent context switching.

When core sched is off, is SMT off as well? From the above table, it seems to
be. So even for core sched off, there will be a single lock per physical CPU
core (assuming SMT is also off) right? Or did I miss something?

thanks,

 - Joel