Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <39cb5352-4b8d-7ca8-4414-6fbb27bab806@joelfernandes.org>
Date:   Tue, 4 Oct 2022 12:22:08 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.3.0
Subject: Re: [PATCH v7 02/11] rcu: Make call_rcu() lazy to save power
Content-Language: en-US
To:     Uladzislau Rezki <urezki@gmail.com>,
        "Paul E. McKenney" <paulmck@kernel.org>
Cc:     rcu@vger.kernel.org, linux-kernel@vger.kernel.org,
        rushikesh.s.kadam@intel.com, neeraj.iitr10@gmail.com,
        frederic@kernel.org, rostedt@goodmis.org, youssefesmat@google.com,
        surenb@google.com
References: <20221004024157.2470238-1-joel@joelfernandes.org>
 <20221004024157.2470238-3-joel@joelfernandes.org> <Yzwb8vgYhAvDJzK+@pc636>
 <20221004133004.GD4196@paulmck-ThinkPad-P17-Gen-1> <YzxI1frUjDnB6MOP@pc636>
From:   Joel Fernandes <joel@joelfernandes.org>
In-Reply-To: <YzxI1frUjDnB6MOP@pc636>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 10/4/2022 10:53 AM, Uladzislau Rezki wrote:
> On Tue, Oct 04, 2022 at 06:30:04AM -0700, Paul E. McKenney wrote:
>> On Tue, Oct 04, 2022 at 01:41:38PM +0200, Uladzislau Rezki wrote:
>>>>  		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
>>>>  		rcu_nocb_lock_irqsave(rdp, flags);
>>>>  		lockdep_assert_held(&rdp->nocb_lock);
>>>>  		bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
>>>> -		if (bypass_ncbs &&
>>>> +		lazy_ncbs = READ_ONCE(rdp->lazy_len);
>>>> +
>>>> +		if (bypass_ncbs && (lazy_ncbs == bypass_ncbs) &&
>>>> +		    (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush) ||
>>>> +		     bypass_ncbs > 2 * qhimark)) {
>>> Do you know why we want double "qhimark" threshold? It is not only this
>>> place, there are several. I am asking because it is not expected by the
>>> user.
>>
>> OK, I will bite...  What does the user expect?  Or, perhaps a better
>> question, how is this choice causing the user problems?
>>
> Yesterday when i was checking the lazy-v6 on Android i noticed the following:
> 
> <snip>
> ...
>      rcuop/4-48      [006] d..1   184.780328: rcu_batch_start: rcu_preempt CBs=15572 bl=121
>      rcuop/6-62      [000] d..1   184.796939: rcu_batch_start: rcu_preempt CBs=21503 bl=167
>      rcuop/6-62      [003] d..1   184.800706: rcu_batch_start: rcu_preempt CBs=24677 bl=192
>      rcuop/6-62      [005] d..1   184.803773: rcu_batch_start: rcu_preempt CBs=27117 bl=211
>      rcuop/6-62      [005] d..1   184.805732: rcu_batch_start: rcu_preempt CBs=22391 bl=174
>      rcuop/6-62      [005] d..1   184.809083: rcu_batch_start: rcu_preempt CBs=12554 bl=98
>      rcuop/6-62      [005] d..1   184.824228: rcu_batch_start: rcu_preempt CBs=16177 bl=126
>      rcuop/4-48      [006] d..1   184.836193: rcu_batch_start: rcu_preempt CBs=24129 bl=188
>      rcuop/4-48      [006] d..1   184.844147: rcu_batch_start: rcu_preempt CBs=25854 bl=201
>      rcuop/4-48      [006] d..1   184.847257: rcu_batch_start: rcu_preempt CBs=21328 bl=166
>      rcuop/4-48      [006] d..1   184.852128: rcu_batch_start: rcu_preempt CBs=21710 bl=169

This looks normal to me and working as intended, due to the first flush after
hitting the limit, future callbacks will not be considered lazy until the
existing ones are drained out of the queue. It is possible the additional
callbacks showed up before the whole queue was drained. Then add any scheduling
delays to the rcuop thread etc etc.

thanks,

 - Joel


> ...
> <snip>
> 
> On my device the "qhimark" is set to:
> 
> <snip>
> XQ-CT54:/sys/module/rcutree/parameters # cat qhimark
> 10000
> XQ-CT54:/sys/module/rcutree/parameters #
> <snip>
> 
> so i expect that once we pass 10 000 callbacks threshold the flush 
> should occur. This parameter gives us an opportunity to control a
> memory that should be reclaimed sooner or later.
> 
> --
> Uladzislau Rezki