Date: Mon, 7 Mar 2016 12:24:31 -0600 (CST)
From: Christoph Lameter <cl@linux.com>
To: Waiman Long <Waiman.Long@hpe.com>
cc: Tejun Heo <tj@kernel.org>, Dave Chinner <dchinner@redhat.com>,
        xfs@oss.sgi.com, linux-kernel@vger.kernel.org,
        Ingo Molnar <mingo@redhat.com>, Peter Zijlstra <peterz@infradead.org>,
        Scott J Norton <scott.norton@hp.com>,
        Douglas Hatch <doug.hatch@hp.com>
Subject: Re: [RFC PATCH 1/2] percpu_counter: Allow falling back to global
 counter on large system
In-Reply-To: <1457146299-1601-2-git-send-email-Waiman.Long@hpe.com>
Message-ID: <alpine.DEB.2.20.1603071220470.1453@east.gentwo.org>
References: <1457146299-1601-1-git-send-email-Waiman.Long@hpe.com> <1457146299-1601-2-git-send-email-Waiman.Long@hpe.com>
Content-Type: text/plain; charset=US-ASCII
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 1441
Lines: 30

On Fri, 4 Mar 2016, Waiman Long wrote:

> This patch provides a mechanism to selectively degenerate per-cpu
> counters to global counters at per-cpu counter initialization time. The
> following new API is added:
>
>   percpu_counter_set_limit(struct percpu_counter *fbc,
>                            u32 percpu_limit)
>
> The function should be called after percpu_counter_set(). It will
> compare the total limit (nr_cpu * percpu_limit) against the current
> counter value.  If the limit is not smaller, it will disable per-cpu
> counter and use only the global counter instead. At run time, when
> the counter value grows past the total limit, per-cpu counter will
> be enabled again.

Hmmm... That is requiring manual setting of a limit. Would it not be
possible to completely automatize the switch over? F.e. one could
keep a cpumask of processors that use the per cpu counters.

Then in the fastpath if the current cpu is a member increment the per cpu
counter. If not do the spinlock thing. If there is contention add the
cpu to the cpumask and use the  per cpu counters. Thus automatically
scaling for the processors on which frequent increments are operating.

Then regularly (once per minute or so) degenerate the counter by folding
the per cpu diffs into the global count and zapping the cpumask.

If the cpumask is empty you can use the global count. Otherwise you just
need to add up the counters of the cpus set in the cpumask.