Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <80166ee7-4ee3-bb2d-c715-a8180dddee31@intel.com>
Date:   Fri, 24 Mar 2023 20:10:31 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.0
Subject: Re: [PATCH RFC rcu 03/19] srcu: Begin offloading srcu_struct fields
 to srcu_update
To:     "Paul E. McKenney" <paulmck@kernel.org>, <rcu@vger.kernel.org>
CC:     <linux-kernel@vger.kernel.org>, <kernel-team@meta.com>,
        <rostedt@goodmis.org>, <hch@lst.de>,
        =?UTF-8?B?TWljaGHFgiBNaXJvc8WCYXc=?= <mirq-linux@rere.qmqm.pl>,
        Dmitry Osipenko <dmitry.osipenko@collabora.com>
References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop>
 <20230324001938.3443499-3-paulmck@kernel.org>
Content-Language: en-US
From:   "Wysocki, Rafael J" <rafael.j.wysocki@intel.com>
In-Reply-To: <20230324001938.3443499-3-paulmck@kernel.org>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RWRDV01QNjIzRHBya00yYnVZMkF4ejBqOGRyZVRLaHFKakpEU1lOUGVDcHNq?=
 =?utf-8?B?UTJhTHNEeTlGTGZ0My9TRWZaWW9ldlpWUmhVNUdKb2NVSk1pVnRrMiszZFQ5?=
 =?utf-8?B?T3FsL09RcnJnWndFQmlwVTUyMGFpcitUbVg1SW9SNzBqcnVOWGtQVVZXR3kz?=
 =?utf-8?B?S2VFWEJEWmV5WlM1NjVNbFltcUdqUGhIWWFRczJIaWt5S050V2owNDA2ZXY5?=
 =?utf-8?B?SE92OHdua3VmblJwWGVQV2NxbjlZMWtnUzVtc2Z0cmlqRHdLSXFLWks5cUhV?=
 =?utf-8?B?VmI1Sy80Wk5RSVVjaEtnZm1JTWQ2b3JORGMrQlYzdERoRFlqUHd0Z0EzZklt?=
 =?utf-8?B?L1RSaHhBY28wSlNaYXd2T2Z2bkhNS2dYeGVUN2t5Y3d2VXhpRnhtZGFJNXJp?=
 =?utf-8?B?MEs4Qk1JNlI0UEUxb0F0YzhRelhxcjJNbFdId0txR0pVWEZKWGpjK3F5cTBu?=
 =?utf-8?B?WSsyZ1R4bktYWFYwSVNaR0N2SFFYNVFXbmJtRGJyUnBQZ0hNaXE3MWpvT3FJ?=
 =?utf-8?B?VTFyajRQajBPV0VxK0xGdEgyNGVIMSsySVZzbFFudGdTcXVwZnZjeWFIaTN0?=
 =?utf-8?B?OFZWNU5pN1R6Yi9IWTlxZWF1dkNJMHZ0R0FOdVFrL3c5Q2VlaVJQVmFZeTlt?=
 =?utf-8?B?S2lEaVJVUSs0QXJhbGxGdmVsQWFFMDNBTlAyZVpHVXBjT3FNOElXVjNUWjlx?=
 =?utf-8?B?VytraDFmR3hhRVc3RWpWSjdZbUNvRW1sMXdPSzdocEk3aU1xTy93SzBmZU43?=
 =?utf-8?B?Qk9DNFVteGdmK01Ia0FrbG10eXI1MmJNbG9LRlJ0WlM5cSs5aXVGU0pVeVR5?=
 =?utf-8?B?TDZMZkIzSElXQ0Y5TGNzNENpWUJ3SnJNNTVPc3hjZGtlMU4ya05GU0V5UkJ4?=
 =?utf-8?B?bGdNR2RTb1pHSlluV2xOSGJ4Z2IwQ3lzQk9ORHk3aVBIdEpNWUNoNFRaY0tT?=
 =?utf-8?B?RjlibHo4dEJwSjBBZkVlU05CNXowaTRqQk9hZUF0SEkrRXMxaVJpYlZVOEFm?=
 =?utf-8?B?RWdnWW1ZdWRsMGE4aVRuQWZtY0g4T1JnMHVHUjZZMmhaU1VJT1Q3cnkwSzBw?=
 =?utf-8?B?MnI4eXcyVnY4NEN5cUtNZEtCdDRYMC83M0dFenZuSDlnbjVpcTlVeHI3L1p5?=
 =?utf-8?B?emNURE5VcHVGUXRWWmIyNWYyNmtvSThxL1hLT2M4KzQ2blJ1a3VwZDdrYk1q?=
 =?utf-8?B?bkhrd0Z2c3hYK3JtemlCK0thTUtRb0VWRkJiQkh6SWw0dmhFN3psRCt6dlJY?=
 =?utf-8?B?b2x0TUkrMUhsaEQ3aXlvT1cyYUpha0xaWXJJYmhkOU1MQzJCSEd6cnN3eHds?=
 =?utf-8?B?Yit3V0xXbmRqaUV0TXp1bGF4UVVLd2xhaWZRTGxVUGNzSW5ScHFkNUV6eTVO?=
 =?utf-8?B?anE4anA1UjR6NEQxRlVOcGZCbzFDbHBmc2loY2M2VzFZNFNDN3lhTXJLYmt6?=
 =?utf-8?B?MUZhM0tzS2x0ZlN0S3hpMHVrZWFKMWxNbmUrbWZGRjkxMC96cm5QRFM2d3JR?=
 =?utf-8?B?OE92WUxITC82eHBxS1ZMWlBkcEtyZXNtOE83WmprWGl0MXhjODB6OGVPZWhp?=
 =?utf-8?B?L2hZNWVxUnBXWEdJb0pQY1JMbDcyTUpvZWJpSklKNHFPbFY0ZEtIdy9iMWhU?=
 =?utf-8?B?TUJjY21MYmZxWWphL3BCQjg4d01VQjhOa3ljUEhSYXR4cmpKQy9wM2dNcEo4?=
 =?utf-8?B?M1d0bzFLdFJWT2I5VUozOFdrVTZRYjhHY05JUjdFUEd0cktpY3dlcWxvWFRB?=
 =?utf-8?B?RnUwczBWczBzZHdHSURpU2hYSGgvWEozcm9lb1FQZk9INmpra0ExM2tMcTBm?=
 =?utf-8?B?YlI4eWRMeHBrQkhsUnRWZlFid0pPTk9GYm1xZFl6WitBdm1QczJYa1BQbnNW?=
 =?utf-8?B?OURLZWpTN1lzbXRNNHd3cCs5ZEUzOWNUK3lmT3dTTTlwbXFvalVKL0NUOURy?=
 =?utf-8?B?OVVjaDJHL3dJTERzRE9iYkdBQ0VESjB1NHl3WGdneGF2Y1VSQlZTYzA1eVEx?=
 =?utf-8?B?NXI0ZUNVSlpFWFR2dTQ3N0ZrYTBCaUNvbXdjN3ROdkRNejNuYUhBUzZUTVZN?=
 =?utf-8?B?T0xENnlSLzE0SGJrVVJUQUtoTWhFVzBYeGc2Y25TdmgySmpUKzdMWktiUWFk?=
 =?utf-8?B?VzMwS2NaekV6aDVsWmtmZ0pPWHI5L1NpdUk2MUxGaFMzckZzN3NhUWN4amZh?=
 =?utf-8?B?UUE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a15325a6-fdb1-4a1c-ff9d-08db2c9b74ac
X-MS-Exchange-CrossTenant-AuthSource: MW5PR11MB5810.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2023 19:10:39.5698
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ScHSoJKbqayzGlx/ff49P2jOB1z0ypQZ06bzKh47lmb6Ud67F2xCqgW6WifO/0AjKcrOH96O5Hlbg74zI/HB6bQkI5+7+INjnrWZ7QO8Bqw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR11MB7676
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_FILL_THIS_FORM_SHORT autolearn=unavailable
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 3/24/2023 1:19 AM, Paul E. McKenney wrote:
> The current srcu_struct structure is on the order of 200 bytes in size
> (depending on architecture and .config), which is much better than the
> old-style 26K bytes, but still all too inconvenient when one is trying
> to achieve good cache locality on a fastpath involving SRCU readers.
>
> However, only a few fields in srcu_struct are used by SRCU readers.
> The remaining fields could be offloaded to a new srcu_update
> structure, thus shrinking the srcu_struct structure down to a few
> tens of bytes.  This commit begins this noble quest, a quest that is
> complicated by open-coded initialization of the srcu_struct within the
> srcu_notifier_head structure.  This complication is addressed by updating
> the srcu_notifier_head structure's open coding, given that there does
> not appear to be a straightforward way of abstracting that initialization.
>
> This commit moves only the ->node pointer to srcu_update.  Later commits
> will move additional fields.
>
> [ paulmck: Fold in qiang1.zhang@intel.com's memory-leak fix. ]
>
> Link: https://lore.kernel.org/all/20230320055751.4120251-1-qiang1.zhang@intel.com/
> Suggested-by: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
> Cc: "Michał Mirosław" <mirq-linux@rere.qmqm.pl>
> Cc: Dmitry Osipenko <dmitry.osipenko@collabora.com>

Fine with me.

Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>


> ---
>   include/linux/notifier.h |  5 ++++-
>   include/linux/srcutiny.h |  6 +++---
>   include/linux/srcutree.h | 27 ++++++++++++++++++---------
>   kernel/rcu/rcu.h         |  6 ++++--
>   kernel/rcu/srcutree.c    | 28 +++++++++++++++++++---------
>   5 files changed, 48 insertions(+), 24 deletions(-)
>
> diff --git a/include/linux/notifier.h b/include/linux/notifier.h
> index aef88c2d1173..2aba75145144 100644
> --- a/include/linux/notifier.h
> +++ b/include/linux/notifier.h
> @@ -73,6 +73,9 @@ struct raw_notifier_head {
>   
>   struct srcu_notifier_head {
>   	struct mutex mutex;
> +#ifdef CONFIG_TREE_SRCU
> +	struct srcu_usage srcuu;
> +#endif
>   	struct srcu_struct srcu;
>   	struct notifier_block __rcu *head;
>   };
> @@ -107,7 +110,7 @@ extern void srcu_init_notifier_head(struct srcu_notifier_head *nh);
>   	{							\
>   		.mutex = __MUTEX_INITIALIZER(name.mutex),	\
>   		.head = NULL,					\
> -		.srcu = __SRCU_STRUCT_INIT(name.srcu, pcpu),	\
> +		.srcu = __SRCU_STRUCT_INIT(name.srcu, name.srcuu, pcpu), \
>   	}
>   
>   #define ATOMIC_NOTIFIER_HEAD(name)				\
> diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h
> index 5aa5e0faf6a1..ebd72491af99 100644
> --- a/include/linux/srcutiny.h
> +++ b/include/linux/srcutiny.h
> @@ -31,7 +31,7 @@ struct srcu_struct {
>   
>   void srcu_drive_gp(struct work_struct *wp);
>   
> -#define __SRCU_STRUCT_INIT(name, __ignored)				\
> +#define __SRCU_STRUCT_INIT(name, __ignored, ___ignored)			\
>   {									\
>   	.srcu_wq = __SWAIT_QUEUE_HEAD_INITIALIZER(name.srcu_wq),	\
>   	.srcu_cb_tail = &name.srcu_cb_head,				\
> @@ -44,9 +44,9 @@ void srcu_drive_gp(struct work_struct *wp);
>    * Tree SRCU, which needs some per-CPU data.
>    */
>   #define DEFINE_SRCU(name) \
> -	struct srcu_struct name = __SRCU_STRUCT_INIT(name, name)
> +	struct srcu_struct name = __SRCU_STRUCT_INIT(name, name, name)
>   #define DEFINE_STATIC_SRCU(name) \
> -	static struct srcu_struct name = __SRCU_STRUCT_INIT(name, name)
> +	static struct srcu_struct name = __SRCU_STRUCT_INIT(name, name, name)
>   
>   void synchronize_srcu(struct srcu_struct *ssp);
>   
> diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h
> index 428480152375..2689e64024bb 100644
> --- a/include/linux/srcutree.h
> +++ b/include/linux/srcutree.h
> @@ -57,11 +57,17 @@ struct srcu_node {
>   	int grphi;				/* Biggest CPU for node. */
>   };
>   
> +/*
> + * Per-SRCU-domain structure, update-side data linked from srcu_struct.
> + */
> +struct srcu_usage {
> +	struct srcu_node *node;			/* Combining tree. */
> +};
> +
>   /*
>    * Per-SRCU-domain structure, similar in function to rcu_state.
>    */
>   struct srcu_struct {
> -	struct srcu_node *node;			/* Combining tree. */
>   	struct srcu_node *level[RCU_NUM_LVLS + 1];
>   						/* First node at each level. */
>   	int srcu_size_state;			/* Small-to-big transition state. */
> @@ -90,6 +96,7 @@ struct srcu_struct {
>   	unsigned long reschedule_count;
>   	struct delayed_work work;
>   	struct lockdep_map dep_map;
> +	struct srcu_usage *srcu_sup;		/* Update-side data. */
>   };
>   
>   // Values for size state variable (->srcu_size_state).  Once the state
> @@ -121,24 +128,24 @@ struct srcu_struct {
>   #define SRCU_STATE_SCAN1	1
>   #define SRCU_STATE_SCAN2	2
>   
> -#define __SRCU_STRUCT_INIT_COMMON(name)								\
> +#define __SRCU_STRUCT_INIT_COMMON(name, usage_name)						\
>   	.lock = __SPIN_LOCK_UNLOCKED(name.lock),						\
>   	.srcu_gp_seq_needed = -1UL,								\
>   	.work = __DELAYED_WORK_INITIALIZER(name.work, NULL, 0),					\
> +	.srcu_sup = &usage_name,								\
>   	__SRCU_DEP_MAP_INIT(name)
>   
> -#define __SRCU_STRUCT_INIT_MODULE(name)								\
> +#define __SRCU_STRUCT_INIT_MODULE(name, usage_name)						\
>   {												\
> -	__SRCU_STRUCT_INIT_COMMON(name)								\
> +	__SRCU_STRUCT_INIT_COMMON(name, usage_name)						\
>   }
>   
> -#define __SRCU_STRUCT_INIT(name, pcpu_name)							\
> +#define __SRCU_STRUCT_INIT(name, usage_name, pcpu_name)						\
>   {												\
>   	.sda = &pcpu_name,									\
> -	__SRCU_STRUCT_INIT_COMMON(name)								\
> +	__SRCU_STRUCT_INIT_COMMON(name, usage_name)						\
>   }
>   
> -
>   /*
>    * Define and initialize a srcu struct at build time.
>    * Do -not- call init_srcu_struct() nor cleanup_srcu_struct() on it.
> @@ -160,15 +167,17 @@ struct srcu_struct {
>    */
>   #ifdef MODULE
>   # define __DEFINE_SRCU(name, is_static)								\
> -	is_static struct srcu_struct name = __SRCU_STRUCT_INIT_MODULE(name);			\
> +	static struct srcu_usage name##_srcu_usage;						\
> +	is_static struct srcu_struct name = __SRCU_STRUCT_INIT_MODULE(name, name##_srcu_usage);	\
>   	extern struct srcu_struct * const __srcu_struct_##name;					\
>   	struct srcu_struct * const __srcu_struct_##name						\
>   		__section("___srcu_struct_ptrs") = &name
>   #else
>   # define __DEFINE_SRCU(name, is_static)								\
>   	static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data);				\
> +	static struct srcu_usage name##_srcu_usage;						\
>   	is_static struct srcu_struct name =							\
> -		__SRCU_STRUCT_INIT(name, name##_srcu_data)
> +		__SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data)
>   #endif
>   #define DEFINE_SRCU(name)		__DEFINE_SRCU(name, /* not static */)
>   #define DEFINE_STATIC_SRCU(name)	__DEFINE_SRCU(name, static)
> diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h
> index a3adcf9a9919..4a1b9622598b 100644
> --- a/kernel/rcu/rcu.h
> +++ b/kernel/rcu/rcu.h
> @@ -378,11 +378,13 @@ extern void rcu_init_geometry(void);
>    * specified state structure (for SRCU) or the only rcu_state structure
>    * (for RCU).
>    */
> -#define srcu_for_each_node_breadth_first(sp, rnp) \
> +#define _rcu_for_each_node_breadth_first(sp, rnp) \
>   	for ((rnp) = &(sp)->node[0]; \
>   	     (rnp) < &(sp)->node[rcu_num_nodes]; (rnp)++)
>   #define rcu_for_each_node_breadth_first(rnp) \
> -	srcu_for_each_node_breadth_first(&rcu_state, rnp)
> +	_rcu_for_each_node_breadth_first(&rcu_state, rnp)
> +#define srcu_for_each_node_breadth_first(ssp, rnp) \
> +	_rcu_for_each_node_breadth_first(ssp->srcu_sup, rnp)
>   
>   /*
>    * Scan the leaves of the rcu_node hierarchy for the rcu_state structure.
> diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> index 7a6d9452a5d0..ad1d5ca42a99 100644
> --- a/kernel/rcu/srcutree.c
> +++ b/kernel/rcu/srcutree.c
> @@ -173,12 +173,12 @@ static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flags)
>   
>   	/* Initialize geometry if it has not already been initialized. */
>   	rcu_init_geometry();
> -	ssp->node = kcalloc(rcu_num_nodes, sizeof(*ssp->node), gfp_flags);
> -	if (!ssp->node)
> +	ssp->srcu_sup->node = kcalloc(rcu_num_nodes, sizeof(*ssp->srcu_sup->node), gfp_flags);
> +	if (!ssp->srcu_sup->node)
>   		return false;
>   
>   	/* Work out the overall tree geometry. */
> -	ssp->level[0] = &ssp->node[0];
> +	ssp->level[0] = &ssp->srcu_sup->node[0];
>   	for (i = 1; i < rcu_num_lvls; i++)
>   		ssp->level[i] = ssp->level[i - 1] + num_rcu_lvl[i - 1];
>   	rcu_init_levelspread(levelspread, num_rcu_lvl);
> @@ -195,7 +195,7 @@ static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flags)
>   		snp->srcu_gp_seq_needed_exp = SRCU_SNP_INIT_SEQ;
>   		snp->grplo = -1;
>   		snp->grphi = -1;
> -		if (snp == &ssp->node[0]) {
> +		if (snp == &ssp->srcu_sup->node[0]) {
>   			/* Root node, special case. */
>   			snp->srcu_parent = NULL;
>   			continue;
> @@ -236,8 +236,12 @@ static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flags)
>    */
>   static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static)
>   {
> +	if (!is_static)
> +		ssp->srcu_sup = kzalloc(sizeof(*ssp->srcu_sup), GFP_KERNEL);
> +	if (!ssp->srcu_sup)
> +		return -ENOMEM;
>   	ssp->srcu_size_state = SRCU_SIZE_SMALL;
> -	ssp->node = NULL;
> +	ssp->srcu_sup->node = NULL;
>   	mutex_init(&ssp->srcu_cb_mutex);
>   	mutex_init(&ssp->srcu_gp_mutex);
>   	ssp->srcu_idx = 0;
> @@ -249,8 +253,11 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static)
>   	ssp->sda_is_static = is_static;
>   	if (!is_static)
>   		ssp->sda = alloc_percpu(struct srcu_data);
> -	if (!ssp->sda)
> +	if (!ssp->sda) {
> +		if (!is_static)
> +			kfree(ssp->srcu_sup);
>   		return -ENOMEM;
> +	}
>   	init_srcu_struct_data(ssp);
>   	ssp->srcu_gp_seq_needed_exp = 0;
>   	ssp->srcu_last_gp_end = ktime_get_mono_fast_ns();
> @@ -259,6 +266,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static)
>   			if (!ssp->sda_is_static) {
>   				free_percpu(ssp->sda);
>   				ssp->sda = NULL;
> +				kfree(ssp->srcu_sup);
>   				return -ENOMEM;
>   			}
>   		} else {
> @@ -656,13 +664,15 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)
>   			rcu_seq_current(&ssp->srcu_gp_seq), ssp->srcu_gp_seq_needed);
>   		return; /* Caller forgot to stop doing call_srcu()? */
>   	}
> +	kfree(ssp->srcu_sup->node);
> +	ssp->srcu_sup->node = NULL;
> +	ssp->srcu_size_state = SRCU_SIZE_SMALL;
>   	if (!ssp->sda_is_static) {
>   		free_percpu(ssp->sda);
>   		ssp->sda = NULL;
> +		kfree(ssp->srcu_sup);
> +		ssp->srcu_sup = NULL;
>   	}
> -	kfree(ssp->node);
> -	ssp->node = NULL;
> -	ssp->srcu_size_state = SRCU_SIZE_SMALL;
>   }
>   EXPORT_SYMBOL_GPL(cleanup_srcu_struct);
>