Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <a70eaa87-2afa-a5c7-a463-7199744cefa4@bytedance.com>
Date:   Thu, 13 Jul 2023 11:43:50 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.13.0
Subject: Re: Re: [PATCH v2 5/7] sched: Implement shared runqueue in CFS
Content-Language: en-US
To:     David Vernet <void@manifault.com>
Cc:     mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com,
        vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
        rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de,
        bristot@redhat.com, vschneid@redhat.com, gautham.shenoy@amd.com,
        kprateek.nayak@amd.com, aaron.lu@intel.com, clm@meta.com,
        tj@kernel.org, roman.gushchin@linux.dev, kernel-team@meta.com,
        linux-kernel@vger.kernel.org
References: <20230710200342.358255-1-void@manifault.com>
 <20230710200342.358255-6-void@manifault.com>
 <93260dd9-818a-7f98-e030-635e0dc8cad8@bytedance.com>
 <20230712221657.GF12207@maniforge>
From:   Abel Wu <wuyun.abel@bytedance.com>
In-Reply-To: <20230712221657.GF12207@maniforge>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 7/13/23 6:16 AM, David Vernet wrote:
> On Wed, Jul 12, 2023 at 06:47:26PM +0800, Abel Wu wrote:
>>> + *
>>> + * HOW
>>> + * ===
>>> + *
>>> + * An shared_runq is comprised of a list, and a spinlock for synchronization.
>>> + * Given that the critical section for a shared_runq is typically a fast list
>>> + * operation, and that the shared_runq is localized to a single LLC, the
>>> + * spinlock will typically only be contended on workloads that do little else
>>> + * other than hammer the runqueue.
>>
>> Would there be scalability issues on large LLCs?
> 
> See the next patch in the series [0] where we shard the per-LLC shared
> runqueues to avoid contention.
> 
> [0]: https://lore.kernel.org/lkml/20230710200342.358255-7-void@manifault.com/

Sorry, I should have read the cover letter more carefully. By sharding,
the LLC is partitioned into several zones, hence contention is relieved.
But sharding itself might be tricky. Making the SMT siblings not cross
shards, as suggested by Peter, is generally a good thing. But I wonder
if there is any workload might benefit from other sharding form.

>>
>>> +
>>> +	task_rq_unlock(src_rq, p, &src_rf);
>>> +
>>> +	raw_spin_rq_lock(rq);
>>> +	rq_repin_lock(rq, rf);
>>
>> By making it looks more ugly, we can save some cycles..
>>
>> 	if (src_rq != rq) {
>> 		task_rq_unlock(src_rq, p, &src_rf);
>> 	} else {
>> 		rq_unpin_lock(src_rq, src_rf);
>> 		raw_spin_unlock_irqrestore(&p->pi_lock, src_rf.flags);
>> 		rq_repin_lock(rq, rf);
>> 	}

I forgot the repin part when src_rq != rq, but I'm sure you already got
my point :)

Cheers,
	Abel