Received: by 2002:a25:b794:0:0:0:0:0 with SMTP id n20csp6643736ybh; Thu, 8 Aug 2019 03:29:58 -0700 (PDT) X-Google-Smtp-Source: APXvYqyWqdpuG7GAbEJcMlTtbNm803d9x0HqdyKJtpWEnj0XaTkldZt3VJzmtYXMTp4RGFG/wO8p X-Received: by 2002:a62:2f04:: with SMTP id v4mr14367414pfv.14.1565260198533; Thu, 08 Aug 2019 03:29:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565260198; cv=none; d=google.com; s=arc-20160816; b=KNPDUL2tDxgfio/7RbBFF8BZ6q/G10wZhPEUZb4Z+LPIAtW0vf7l/mxNRd5a3TFb8Y OHjahgvZ+o6nh2RlqCfS0WfPlTKDGLkmIA0LdAF/LR8Ig3SHK6SIQVmx66/Ql7MryBi4 07LHjxxYFcKLreRpS96TTQv+4IURG37fvRMb6bFoPqwJ8WLocHYRiICjsBB58S9qIPDf MR02RhF0w8NzT/AtqNMCVu80XND5PpJ0DK1AEtjrdNmcOMLmOQStf8see4AcDmgWMtBZ MfHB+sim8pd9UVXLkFZ834Am+g+gEMgIc1LHQ+5AAO7z1n1O1a//zj2R5PAqx/V2eqEK efrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date; bh=HfPaP1lSOGqIRJ4mcGBdZIQ1rIaU4ZZuiBqm1E4JkiQ=; b=K9nuCrlEZ5tH8CohFsO2WDftdqWeYu5OiPh/CVV1+heKPjiEr6OsAzXPiFcxsQp74X wrigWCKiHRLaiFQv+C0k9vcuQm73UoOw7MiBMOjPlR+C+40AQHijqWSXctEx9UFEKIyC SkRpWtfrRIDA6cg9AxRefzykqX72WnIVhkLioh9wkBDjQ4zsdCvKK8Yow5RbEkvo9lPe 5iseSHuN5EgxAwYXGONXNKjmZeOh8+wLe2/owqctk16mRujvq6w+ogvuBFwrFl/CW9zM 8PcS72N4zmJGNwizH6B4NpOt6SXfWrwG/sdqrJYeY5mR7Wzqd3I6WHzIv3cm1SLTHxAD ks7Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y84si59217702pfc.33.2019.08.08.03.29.42; Thu, 08 Aug 2019 03:29:58 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389984AbfHHK1j (ORCPT + 99 others); Thu, 8 Aug 2019 06:27:39 -0400 Received: from lgeamrelo11.lge.com ([156.147.23.51]:41808 "EHLO lgeamrelo11.lge.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389947AbfHHK1j (ORCPT ); Thu, 8 Aug 2019 06:27:39 -0400 Received: from unknown (HELO lgeamrelo04.lge.com) (156.147.1.127) by 156.147.23.51 with ESMTP; 8 Aug 2019 19:27:36 +0900 X-Original-SENDERIP: 156.147.1.127 X-Original-MAILFROM: byungchul.park@lge.com Received: from unknown (HELO X58A-UD3R) (10.177.222.33) by 156.147.1.127 with ESMTP; 8 Aug 2019 19:27:36 +0900 X-Original-SENDERIP: 10.177.222.33 X-Original-MAILFROM: byungchul.park@lge.com Date: Thu, 8 Aug 2019 19:26:10 +0900 From: Byungchul Park To: Joel Fernandes Cc: "Paul E. McKenney" , linux-kernel@vger.kernel.org, Rao Shoaib , max.byungchul.park@gmail.com, kernel-team@android.com, kernel-team@lge.com, Davidlohr Bueso , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , rcu@vger.kernel.org, Steven Rostedt Subject: Re: [PATCH RFC v1 1/2] rcu/tree: Add basic support for kfree_rcu batching Message-ID: <20190808102610.GA7227@X58A-UD3R> References: <20190806212041.118146-1-joel@joelfernandes.org> <20190806235631.GU28441@linux.ibm.com> <20190807094504.GB169551@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190807094504.GB169551@google.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Aug 07, 2019 at 05:45:04AM -0400, Joel Fernandes wrote: > On Tue, Aug 06, 2019 at 04:56:31PM -0700, Paul E. McKenney wrote: [snip] > > On Tue, Aug 06, 2019 at 05:20:40PM -0400, Joel Fernandes (Google) wrote: > > Of course, I am hoping that a later patch uses an array of pointers built > > at kfree_rcu() time, similar to Rao's patch (with or without kfree_bulk) > > in order to reduce per-object cache-miss overhead. This would make it > > easier for callback invocation to keep up with multi-CPU kfree_rcu() > > floods. > > I think Byungchul tried an experiment with array of pointers and wasn't > immediately able to see a benefit. Perhaps his patch needs a bit more polish > or another test-case needed to show benefit due to cache-misses, and the perf > tool could be used to show if cache misses were reduced. For this initial > pass, we decided to keep it without the array optimization. I'm still seeing no improvement with kfree_bulk(). I've been thinking I could see improvement with kfree_bulk() because: 1. As you guys said, the number of cache misses will be reduced. 2. We can save (N - 1) irq-disable instructions while N kfrees. 3. As Joel said, saving/restoring CPU status that kfree() does inside is not required. But even with the following patch applied, the result was same as just batching test. We might need to get kmalloc objects from random addresses to maximize the result when using kfree_bulk() and this is even closer to real practical world too. And the second and third reasons doesn't seem to work as much as I expected. Do you have any idea? Or what do you think about it? Thanks, Byungchul -----8<----- diff --git a/kernel/rcu/rcuperf.c b/kernel/rcu/rcuperf.c index 988e1ae..6f2ab06 100644 --- a/kernel/rcu/rcuperf.c +++ b/kernel/rcu/rcuperf.c @@ -651,10 +651,10 @@ struct kfree_obj { return -ENOMEM; } - for (i = 0; i < kfree_alloc_num; i++) { - if (!kfree_no_batch) { - kfree_rcu(alloc_ptrs[i], rh); - } else { + if (!kfree_no_batch) { + kfree_bulk(kfree_alloc_num, (void **)alloc_ptrs); + } else { + for (i = 0; i < kfree_alloc_num; i++) { rcu_callback_t cb; cb = (rcu_callback_t)(unsigned long)offsetof(struct kfree_obj, rh);