Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Wed, 24 Jan 2018 21:53:56 -0800
From:   "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To:     lianglihao@huawei.com
Cc:     guohanjun@huawei.com, heng.z@huawei.com, hb.chen@huawei.com,
        lihao.liang@gmail.com, linux-kernel@vger.kernel.org
Subject: Re: [PATCH RFC 00/16] A new RCU implementation based on a fast
 consensus protocol
Reply-To: paulmck@linux.vnet.ibm.com
References: <1516694381-20333-1-git-send-email-lianglihao@huawei.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1516694381-20333-1-git-send-email-lianglihao@huawei.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Message-Id: <20180125055356.GT3741@linux.vnet.ibm.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Jan 23, 2018 at 03:59:25PM +0800, lianglihao@huawei.com wrote:
> From: Lihao Liang <lianglihao@huawei.com>
> 
> Dear Paul,
> 
> This patch set implements a preemptive version of RCU (PRCU) based on the following paper:
> 
> Fast Consensus Using Bounded Staleness for Scalable Read-mostly Synchronization.
> Haibo Chen, Heng Zhang, Ran Liu, Binyu Zang, and Haibing Guan.
> IEEE Transactions on Parallel and Distributed Systems (TPDS), 2016.
> https://dl.acm.org/citation.cfm?id=3024114.3024143
> 
> We have also added preliminary callback-handling support.  Thus, the current version
> provides APIs prcu_read_lock(), prcu_read_unlock(), synchronize_prcu(), call_prcu(),
> and prcu_barrier().
> 
> This is an experimental patch, so it would be good to have some feedback.
> 
> Known shortcoming is that the grace-period version is incremented in synchronize_prcu().
> If call_prcu() or prcu_barrier() is called but there is no synchronized_prcu() invoked,
> callbacks cannot be invoked.  Later version should address this issue, e.g. adding a
> grace-period expedition mechanism.  Others include to use a a hierarchical structure,
> taking into account the NUMA topology, to send IPI in synchronize_prcu().
> 
> We have tested the implementation using rcutorture on both an x86 and ARM64 machine.
> PRCU passed 1h and 3h tests on all the newly added config files except PRCU07 reported BUG 
> in a 1h run.
> 
> [ 1593.604201] ---[ end trace b3bae911bec86152 ]---
> [ 1594.629450] prcu-torture:torture_onoff task: offlining 14
> [ 1594.755553] smpboot: CPU 14 is now offline
> [ 1594.757732] prcu-torture:torture_onoff task: offlined 14
> [ 1597.765149] prcu-torture:torture_onoff task: onlining 11
> [ 1597.766795] smpboot: Booting Node 0 Processor 11 APIC 0xb
> [ 1597.804102] prcu-torture:torture_onoff task: onlined 11
> [ 1599.365098] prcu-torture: rtc: ffffffffb0277b90 ver: 66358 tfle: 0 rta: 66358 rtaf: 0 
> rtf: 66349 rtmbe: 0 rtbe: 1 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 2233418 
> onoff: 191/191:199/199 34,199:59,5102 10403:0 (HZ=1000) barrier: 188/189:1 cbflood: 225
> [ 1599.367946] prcu-torture: !!!
> [ 1599.367966] ------------[ cut here ]------------

The "rtbe: 1" indicates that your implementation of prcu_barrier()
failed to wait for all preceding call_prcu() callbacks to be invoked.

Does the immediately following "Reader Pipe:" list have any but the
first two numbers non-zero?

> We have also compared PRCU with TREE RCU using rcuperf with gp_exp set to true, that is
> synchronize_rcu_expedited was tested.
> 
> The rcuperf results are as follows (average grace-period duration in ms of ten 10min runs):
> 
> 16*Intel Xeon CPU@2.4GHz, 16GB memory, Ubuntu Linux 3.13.0-47-generic
> 
> CPUs      2       4       8      12      15       16
> PRCU   0.14    1.07    4.15    8.02   10.79    15.16 
> TREE  49.30  104.75  277.55  390.82  620.82  1381.54
> 
> 64*Cortex-A72 CPU@2.4GHz, 130GB memory, Ubuntu Linux 4.10.0-21.23-generic
> 
> CPUs       2       4        8      16      32       48       63        64
> PRCU    0.23   19.69    38.28   63.21   95.41   167.18   252.01   1841.44
> TREE  416.73  901.89  1060.86  743.00  920.66  1325.21  1646.20  23806.27

Well, at the very least, this is a bug report on either expedited RCU
grace-period latency or on rcuperf's measurements, and thank you for that.
I will look into this.  In the meantime, could you please let me know
exactly how you invoked rcuperf?

I have a few comments on some of your patches based on a quick scan
through them.

							Thanx, Paul

> Best wishes,
> Lihao.
> 
> 
> Lihao Liang (15):
>   rcutorture: Add PRCU rcu_torture_ops
>   rcutorture: Add PRCU test config files
>   rcuperf: Add PRCU rcu_perf_ops
>   rcuperf: Add PRCU test config files
>   rcuperf: Set gp_exp to true for tests to run
>   prcu: Implement call_prcu() API
>   prcu: Implement PRCU callback processing
>   prcu: Implement prcu_barrier() API
>   rcutorture: Test call_prcu() and prcu_barrier()
>   rcutorture: Add basic ARM64 support to run scripts
>   prcu: Add PRCU Kconfig parameter
>   prcu: Comment source code
>   rcuperf: Add config files with various CONFIG_NR_CPUS
>   rcutorture: Add scripts to run experiments
>   Add GPLv2 license
> 
> Heng Zhang (1):
>   prcu: Add PRCU implementation
> 
>  include/linux/interrupt.h                          |   3 +
>  include/linux/prcu.h                               | 122 +++++
>  include/linux/rcupdate.h                           |   1 +
>  init/Kconfig                                       |   7 +
>  init/main.c                                        |   2 +
>  kernel/rcu/Makefile                                |   1 +
>  kernel/rcu/prcu.c                                  | 497 +++++++++++++++++++++
>  kernel/rcu/rcuperf.c                               |  33 +-
>  kernel/rcu/rcutorture.c                            |  40 +-
>  kernel/rcu/tree.c                                  |   1 +
>  kernel/sched/core.c                                |   2 +
>  kernel/time/timer.c                                |   2 +
>  kvm.sh                                             | 452 +++++++++++++++++++
>  run-rcuperf.sh                                     |  26 ++
>  .../testing/selftests/rcutorture/bin/functions.sh  |  17 +-
>  .../selftests/rcutorture/configs/rcu/CFLIST        |   5 +
>  .../selftests/rcutorture/configs/rcu/PRCU02        |  27 ++
>  .../selftests/rcutorture/configs/rcu/PRCU02.boot   |   1 +
>  .../selftests/rcutorture/configs/rcu/PRCU03        |  23 +
>  .../selftests/rcutorture/configs/rcu/PRCU03.boot   |   2 +
>  .../selftests/rcutorture/configs/rcu/PRCU06        |  26 ++
>  .../selftests/rcutorture/configs/rcu/PRCU06.boot   |   5 +
>  .../selftests/rcutorture/configs/rcu/PRCU07        |  25 ++
>  .../selftests/rcutorture/configs/rcu/PRCU07.boot   |   2 +
>  .../selftests/rcutorture/configs/rcu/PRCU09        |  19 +
>  .../selftests/rcutorture/configs/rcu/PRCU09.boot   |   1 +
>  .../selftests/rcutorture/configs/rcuperf/CFLIST    |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU      |  20 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU-12   |  21 +
>  .../rcutorture/configs/rcuperf/PRCU-12.boot        |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU-14   |  21 +
>  .../rcutorture/configs/rcuperf/PRCU-14.boot        |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU-15   |  21 +
>  .../rcutorture/configs/rcuperf/PRCU-15.boot        |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU-16   |  21 +
>  .../rcutorture/configs/rcuperf/PRCU-16.boot        |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU-2    |  21 +
>  .../rcutorture/configs/rcuperf/PRCU-2.boot         |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU-32   |  21 +
>  .../rcutorture/configs/rcuperf/PRCU-32.boot        |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU-4    |  21 +
>  .../rcutorture/configs/rcuperf/PRCU-4.boot         |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU-48   |  21 +
>  .../rcutorture/configs/rcuperf/PRCU-48.boot        |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU-56   |  21 +
>  .../rcutorture/configs/rcuperf/PRCU-56.boot        |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU-60   |  21 +
>  .../rcutorture/configs/rcuperf/PRCU-60.boot        |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU-62   |  21 +
>  .../rcutorture/configs/rcuperf/PRCU-62.boot        |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU-64   |  21 +
>  .../rcutorture/configs/rcuperf/PRCU-64.boot        |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU-8    |  21 +
>  .../rcutorture/configs/rcuperf/PRCU-8.boot         |   1 +
>  .../selftests/rcutorture/configs/rcuperf/PRCU.boot |   1 +
>  .../selftests/rcutorture/configs/rcuperf/TREE-12   |  21 +
>  .../selftests/rcutorture/configs/rcuperf/TREE-14   |  21 +
>  .../selftests/rcutorture/configs/rcuperf/TREE-15   |  21 +
>  .../selftests/rcutorture/configs/rcuperf/TREE-16   |  21 +
>  .../selftests/rcutorture/configs/rcuperf/TREE-2    |  21 +
>  .../selftests/rcutorture/configs/rcuperf/TREE-32   |  21 +
>  .../selftests/rcutorture/configs/rcuperf/TREE-4    |  21 +
>  .../selftests/rcutorture/configs/rcuperf/TREE-48   |  21 +
>  .../selftests/rcutorture/configs/rcuperf/TREE-56   |  21 +
>  .../selftests/rcutorture/configs/rcuperf/TREE-60   |  21 +
>  .../selftests/rcutorture/configs/rcuperf/TREE-62   |  21 +
>  .../selftests/rcutorture/configs/rcuperf/TREE-64   |  21 +
>  .../selftests/rcutorture/configs/rcuperf/TREE-8    |  21 +
>  68 files changed, 1918 insertions(+), 5 deletions(-)
>  create mode 100644 include/linux/prcu.h
>  create mode 100644 kernel/rcu/prcu.c
>  create mode 100755 kvm.sh
>  create mode 100755 run-rcuperf.sh
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/PRCU02
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/PRCU02.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/PRCU03
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/PRCU03.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/PRCU06
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/PRCU06.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/PRCU07
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/PRCU07.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/PRCU09
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/PRCU09.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-12
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-12.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-14
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-14.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-15
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-15.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-16
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-16.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-2
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-2.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-32
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-32.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-4
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-4.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-48
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-48.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-56
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-56.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-60
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-60.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-62
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-62.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-64
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-64.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-8
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU-8.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/PRCU.boot
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/TREE-12
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/TREE-14
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/TREE-15
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/TREE-16
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/TREE-2
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/TREE-32
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/TREE-4
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/TREE-48
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/TREE-56
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/TREE-60
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/TREE-62
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/TREE-64
>  create mode 100644 tools/testing/selftests/rcutorture/configs/rcuperf/TREE-8
> 
> -- 
> 2.14.1.729.g59c0ea183
>