Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Zhang, Qiang1" <qiang1.zhang@intel.com>
To:     Frederic Weisbecker <frederic@kernel.org>,
        "paulmck@kernel.org" <paulmck@kernel.org>
CC:     "joel@joelfernandes.org" <joel@joelfernandes.org>,
        "rcu@vger.kernel.org" <rcu@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH] srcu: Add detection of boot CPU srcu_data
 structure's->srcu_cblist
Thread-Topic: [PATCH] srcu: Add detection of boot CPU srcu_data
 structure's->srcu_cblist
Thread-Index: AQHY+nXe3fxP1SzD7UGw8RnvZujcXK5DKoyAgADb12A=
Date:   Fri, 18 Nov 2022 04:53:22 +0000
Message-ID: <PH0PR11MB58807CB6626A08CADD822409DA099@PH0PR11MB5880.namprd11.prod.outlook.com>
References: <20221117112050.3942407-1-qiang1.zhang@intel.com>
 <20221117142025.GE839309@lothringen>
In-Reply-To: <20221117142025.GE839309@lothringen>
Accept-Language: en-US
Content-Language: en-US
dlp-product: dlpe-windows
dlp-version: 11.6.500.17
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?MlOesrkuqKCPP5nMOIxB51W0+E5hfLyQs6Iw007DA/C+TmkWqeS5EGPXxAM9?=
 =?us-ascii?Q?T6v+3eSTOXiBVKczdGNBrWtgl42njZaIxp8MUq6NuBHX2Dpy813A0E4ndDIP?=
 =?us-ascii?Q?c67IHaHZGHYnP73I4+5wrPsAjtwqKhPmaA/ZZgxlBa4Aw46iNHWfLZJqK5Pt?=
 =?us-ascii?Q?GprjS2N/ZAch6ev653mj4RzI8RSwDD5PGgXLfxnDfsTl+8sFXeb7yYJ9K/xA?=
 =?us-ascii?Q?6Y/P+hhVd3C2S72qpwH9s1vT2TNlYCYNte6CMHawdqEfpR94cJNU4CHWxlqc?=
 =?us-ascii?Q?i7eGWPfjr4z/Px14xXyQDR8L0JQ3A1b3NJYkLk4yI5CUWjjngYofiiIY8I+5?=
 =?us-ascii?Q?s8pAqYZ0o4c7Hl8emR+Mf9S2GbrM6KlcZ63b2J7uLIfYLhS17WHFokQo5mRM?=
 =?us-ascii?Q?p6uAsqOaEXK2QqBl/eM1+w42I6nbXdmyleDxxmVRMsrekdt4AVp2vuQHLq2k?=
 =?us-ascii?Q?UPn13/TnuC8mQb/8e7Phquw7AtUs6ZQVNXpCTlJFI6jbRxKpI69gOk07zUM3?=
 =?us-ascii?Q?cXxQHN/qc5Vrr9F6cRNDFVTfXMHpOrL1E3Wr9HM22X1zlWcTYefLNBDVbFD8?=
 =?us-ascii?Q?nKlUYC+34QQkfeSVMA/aL82STeNMaj5/hUJZ+nx9g/13yq4+UYhhPd0y1YTY?=
 =?us-ascii?Q?7pmSKAKKiBrtNvUUDBsBZYgiLieCyOrmz0OVGN86lclGIJOVhSyuduodAnce?=
 =?us-ascii?Q?goOZj2qm2xcvZBxeAzX6yUOlchv8cIvC/TvOYgQparLHNqH3AFJkTvBnVcRZ?=
 =?us-ascii?Q?v+iwlxXIezTQIl2yG5rerG6wd24RYMjhVVORGrNIr6IbYd/bx5iopFRWbrsV?=
 =?us-ascii?Q?rojZSiovqaX0u0/KY6dn/18vN5aYZltRgbR7M4QAaG+LpDM/5TiiuwgBGHIM?=
 =?us-ascii?Q?Z26eCLerjbRrVCg4iF90PgszUj6oLEWC1ENFthAsIu7tsyurRMpYP44U8Xq9?=
 =?us-ascii?Q?tKj1I4J38RMvKFIyiQYGcEuTRBqI+oYHOnywEDjRSV5CgRmwrE6hO5OjXptZ?=
 =?us-ascii?Q?DW4+x249MKkZRp5TY4MaBY787YWHDSZ+vxbQFbzGJ4n8JMPB0QnRo+WhBUfx?=
 =?us-ascii?Q?h36QkCaT0Y8os+PE28QK5BU+oX06y/Yx80IBAP5/ljzS6OkmbuQ6oAWAbZ5p?=
 =?us-ascii?Q?n8XRWLqYXfZKoF4FcGYj+SZLSsCAhp9szIefxGenXuuxpFjlPxAktPsuwg66?=
 =?us-ascii?Q?oEnxbFfUSn77SQ7MBFdZ+kHqjpPGcxxDHnnI9WRPkUDjuSU/K2+4Lj5pDhjJ?=
 =?us-ascii?Q?y4EZn1aX3VVxmoMJQJyoGkdfxBEPyFRnZsT1qgbU+VIrsfoIhtNjx1Mk+ZjN?=
 =?us-ascii?Q?KQhdXVr16AASNnxPwO4kvNEcrBlSyQbs/rrDZ0J273rKL6D8w2TULaTNnY5M?=
 =?us-ascii?Q?447wPhmvwqa2Ye06sJqdgxfwcvmWJApRJsQfHzrPzPaOajrKTapL4I1Y0sMA?=
 =?us-ascii?Q?Ks6qriVLEZuRw+Ldy2YmrVKWf/Zgh2gs7UnXcfL1aR7+fpNaqAj89+Qn6EUN?=
 =?us-ascii?Q?b+5RiPc6vCPl7Z3AOjIPXH1uPaSsCLedBJtU0zymdFVKWSoa3pqAqxQNWp6A?=
 =?us-ascii?Q?rjpRfsdfh3fpxJydIrVhvSg0ySJj5IR/ejC8/4Ui?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5880.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 996c02cb-45fb-45e5-e591-08dac920d221
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Nov 2022 04:53:22.7594
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: MZzv69M7J8cV9ULLqohIqUhCoi5OU4YW16dW4xkKL8pa0PVDCpGhd+yIOKMUjQvt5UJWm3Boyuw5zTr1i9+t2Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR11MB5986
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Nov 17, 2022 at 07:20:50PM +0800, Zqiang wrote:
> Before SRCU_SIZE_WAIT_CALL srcu_size_state is reached, the srcu
> callback only insert to boot-CPU srcu_data structure's->srcu_cblist
> and other CPU srcu_data structure's->srcu_cblist is always empty. so
> before SRCU_SIZE_WAIT_CALL is reached, need to correctly check boot-CPU
> pend cbs in srcu_might_be_idle().
>=20
> Signed-off-by: Zqiang <qiang1.zhang@intel.com>
> ---
>  kernel/rcu/srcutree.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>=20
> diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> index 6af031200580..6d9af9901765 100644
> --- a/kernel/rcu/srcutree.c
> +++ b/kernel/rcu/srcutree.c
> @@ -1098,8 +1098,11 @@ static bool srcu_might_be_idle(struct srcu_struct =
*ssp)
>  	unsigned long tlast;
> =20
>  	check_init_srcu_struct(ssp);
> -	/* If the local srcu_data structure has callbacks, not idle.  */
> -	sdp =3D raw_cpu_ptr(ssp->sda);
> +	/* If the boot CPU or local srcu_data structure has callbacks, not idle=
.  */
> +	if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_CALL)
> +		sdp =3D per_cpu_ptr(ssp->sda, get_boot_cpu_id());
> +	else
> +		sdp =3D raw_cpu_ptr(ssp->sda);
>
>While at it if someone is interested in documenting/commenting on the mean=
ing of
>all these SRCU_SIZE_* things, it would be much appreciated!

In the initial stage(SRCU_SIZE_SMALL), there is no srcu_node structures, on=
ly initialized
per-CPU srcu_data structures, at this time, we only use boot-cpu srcu_data =
structures's ->srcu_cblist
to store srcu callbacks. =20
if the contention of srcu_struct and srcu_data structure's->lock become bus=
y,
transition to SRCU_SIZE_ALLOC.  allocated memory for srcu_node structure at=
 end of the SRCU
grace period.  =20
if allocated success,  transition to SRCU_SIZE_WAIT_BARRIER,  in this state=
, invoke
srcu_barrier() will iterate over all possible CPUs, but we still only use b=
oot-CPU srcu_cblist to store callbacks.
the task calling call_srcu() may have access to a new srcu_node structure o=
r may not,=20
because call_srcu() is protected by SRCU, we may not transition to SRCU_SIZ=
E_WAIT_CALL quickly,
need to wait in this state for a SRCU grace period to end.
After transition to SRCU_SIZE_WAIT_CALL, we enqueue srcu callbacks to own s=
rcu_data structures's ->srcu_cblist.

Does my description make my commit more detailed?

Thanks
Zqiang


>
>Thanks.
>
>  	spin_lock_irqsave_rcu_node(sdp, flags);
>  	if (rcu_segcblist_pend_cbs(&sdp->srcu_cblist)) {
>  		spin_unlock_irqrestore_rcu_node(sdp, flags);
> --=20
> 2.25.1
>=20