Message-ID: <56DDDACF.6000702@hpe.com>
Date: Mon, 7 Mar 2016 14:47:27 -0500
From: Waiman Long <waiman.long@hpe.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20130109 Thunderbird/10.0.12
MIME-Version: 1.0
To: Christoph Lameter <cl@linux.com>
CC: Tejun Heo <tj@kernel.org>, Dave Chinner <dchinner@redhat.com>,
        <xfs@oss.sgi.com>, <linux-kernel@vger.kernel.org>,
        Ingo Molnar <mingo@redhat.com>, Peter Zijlstra <peterz@infradead.org>,
        Scott J Norton <scott.norton@hp.com>,
        Douglas Hatch <doug.hatch@hp.com>
Subject: Re: [RFC PATCH 1/2] percpu_counter: Allow falling back to global
 counter on large system
References: <1457146299-1601-1-git-send-email-Waiman.Long@hpe.com> <1457146299-1601-2-git-send-email-Waiman.Long@hpe.com> <alpine.DEB.2.20.1603071220470.1453@east.gentwo.org>
In-Reply-To: <alpine.DEB.2.20.1603071220470.1453@east.gentwo.org>
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit
SpamDiagnosticOutput: 1:23
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Mar 2016 19:47:37.6048 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: TU4PR84MB0319
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 1989
Lines: 44

On 03/07/2016 01:24 PM, Christoph Lameter wrote:
> On Fri, 4 Mar 2016, Waiman Long wrote:
>
>> This patch provides a mechanism to selectively degenerate per-cpu
>> counters to global counters at per-cpu counter initialization time. The
>> following new API is added:
>>
>>    percpu_counter_set_limit(struct percpu_counter *fbc,
>>                             u32 percpu_limit)
>>
>> The function should be called after percpu_counter_set(). It will
>> compare the total limit (nr_cpu * percpu_limit) against the current
>> counter value.  If the limit is not smaller, it will disable per-cpu
>> counter and use only the global counter instead. At run time, when
>> the counter value grows past the total limit, per-cpu counter will
>> be enabled again.
> Hmmm... That is requiring manual setting of a limit. Would it not be
> possible to completely automatize the switch over? F.e. one could
> keep a cpumask of processors that use the per cpu counters.

The limit is usually the batch size used or a multiple of it.

> Then in the fastpath if the current cpu is a member increment the per cpu
> counter. If not do the spinlock thing. If there is contention add the
> cpu to the cpumask and use the  per cpu counters. Thus automatically
> scaling for the processors on which frequent increments are operating.

That is an interesting idea. I will do some prototyping and see how it 
goes. One of the downside that I see is the increase in the size of the 
percpu_counter structure.

> Then regularly (once per minute or so) degenerate the counter by folding
> the per cpu diffs into the global count and zapping the cpumask.

Actually, I think we need 2 cpumasks - one for deciding to use global or 
percpu count and another one for which percpu counts are used as it is 
not safe to change a per-cpu count other than your own one.

> If the cpumask is empty you can use the global count. Otherwise you just
> need to add up the counters of the cpus set in the cpumask.

Cheers,
Longman