Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <5b20ac43-43b4-cabd-fa40-67f8be5fcef0@bytedance.com>
Date:   Thu, 4 May 2023 16:04:23 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.8.0
Subject: Re: [External] Re: [PATCH v4] sched/core: Adapt WARN_DOUBLE_CLOCK
 machinery for core-sched
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     mingo@redhat.com, mingo@kernel.org, juri.lelli@redhat.com,
        vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
        rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de,
        bristot@redhat.com, vschneid@redhat.com,
        mgorman@techsingularity.net, linux-kernel@vger.kernel.org,
        Phil Auld <pauld@redhat.com>
References: <20230406064415.17110-1-jiahao.os@bytedance.com>
 <20230504072336.GG1734100@hirez.programming.kicks-ass.net>
From:   Hao Jia <jiahao.os@bytedance.com>
In-Reply-To: <20230504072336.GG1734100@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 2023/5/4 Peter Zijlstra wrote:
> On Thu, Apr 06, 2023 at 02:44:15PM +0800, Hao Jia wrote:
>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>> index 0d18c3969f90..c6e2c79152ef 100644
>> --- a/kernel/sched/core.c
>> +++ b/kernel/sched/core.c
>> @@ -429,11 +429,32 @@ void sched_core_put(void)
>>   		schedule_work(&_work);
>>   }
>>   
>> +/*
>> + * Now, we have obtained a core-wide rq->lock, then we need to clear
>> + * RQCF_UPDATED of rq->clock_update_flags of the sibiling CPU
>> + * on this core to avoid the WARN_DOUBLE_CLOCK warning.
>> + */
>> +static inline void sched_core_clear_rqcf_updated(struct rq *rq)
>> +{
>> +#ifdef CONFIG_SCHED_DEBUG
>> +	const struct cpumask *smt_mask;
>> +	int i;
>> +
>> +	if (rq->core_enabled) {
>> +		smt_mask = cpu_smt_mask(rq->cpu);
>> +		for_each_cpu(i, smt_mask) {
>> +			if (rq->cpu != i)
>> +				cpu_rq(i)->clock_update_flags &= (RQCF_REQ_SKIP|RQCF_ACT_SKIP);
>> +		}
>> +	}
>> +#endif
>> +}
>>   #else /* !CONFIG_SCHED_CORE */
>>   
>>   static inline void sched_core_enqueue(struct rq *rq, struct task_struct *p) { }
>>   static inline void
>>   sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags) { }
>> +static inline void sched_core_clear_rqcf_updated(struct rq *rq) { }
>>   
>>   #endif /* CONFIG_SCHED_CORE */
>>   
>> @@ -548,6 +569,7 @@ void raw_spin_rq_lock_nested(struct rq *rq, int subclass)
>>   		if (likely(lock == __rq_lockp(rq))) {
>>   			/* preempt_count *MUST* be > 1 */
>>   			preempt_enable_no_resched();
>> +			sched_core_clear_rqcf_updated(rq);
>>   			return;
>>   		}
>>   		raw_spin_unlock(lock);
> 
> This still looks absolutely wrong. The whole RQCF thing is a pin action.

Do you think it is better for us to extend rq_pin_lock() to clean RQCF 
updated than to do it in raw_spin_rq_lock_nested()?

Before doing this, we need to solve the situation where rq_pin_lock() 
and raw_spin_rq_lock() are used separately.

Any suggestion will be very helpful for me.

Thanks,
Hao