Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Zhang, Qiang1" <qiang1.zhang@intel.com>
To:     "paulmck@kernel.org" <paulmck@kernel.org>
CC:     "frederic@kernel.org" <frederic@kernel.org>,
        "rcu@vger.kernel.org" <rcu@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "urezki@gmail.com" <urezki@gmail.com>
Subject: RE: [PATCH v2] rcu: Dump all rcuc kthreads status for CPUs that not
 report quiescent state
Thread-Topic: [PATCH v2] rcu: Dump all rcuc kthreads status for CPUs that not
 report quiescent state
Thread-Index: AQHYU68ghF0IexEC8UWUHs//592Qaaz8E2SAgAJYy3A=
Date:   Sun, 24 Apr 2022 03:19:15 +0000
Message-ID: <PH0PR11MB5880E088CB975674DB791B3BDAF99@PH0PR11MB5880.namprd11.prod.outlook.com>
References: <20220419053426.2820350-1-qiang1.zhang@intel.com>
 <20220422152647.GY4285@paulmck-ThinkPad-P17-Gen-1>
In-Reply-To: <20220422152647.GY4285@paulmck-ThinkPad-P17-Gen-1>
Accept-Language: en-US
Content-Language: en-US
dlp-product: dlpe-windows
dlp-version: 11.6.401.20
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?hqQmQtxUh3VQUq/aVyVnMlOtRmw9KyG5hnPif3IqMIjuKyNGvHdjg/kSjdwt?=
 =?us-ascii?Q?/ui/Wjol87izCEdMrYZk3iebiycW+8/kxzQH6EYr+DXuBxZBPNpf3V6y1Pv+?=
 =?us-ascii?Q?R9YVx7n9oNOGD0QHKflJcf5OzIJAC9HAdvU1CbvDKN9+9q/DeyM8V8OHCLwm?=
 =?us-ascii?Q?vij60D1kZ4W0/cKUrElbuRUDNESv2opJMeTs2+3Y3sM40/XNy5NGftcCJqxR?=
 =?us-ascii?Q?bxhZ5JRO4DP6J298zkMgkgyZwsQcXlZ6hID7sXhA8gqLe8mIKnfDYwrjQufT?=
 =?us-ascii?Q?46jWYqHiobAS4JZDD8oQGIILHFyrhvqjNSY5nIGMo+qdv4JZZZIpajim95UZ?=
 =?us-ascii?Q?ZppHg9GB2eja/dMkZjyylcF3UhtFlt7ZSd62A21b6jVcdMAdCD+IBtFxMFQk?=
 =?us-ascii?Q?zrbCBOZVpgmo5OXyuEbWxHfCyLr0MjDxw/QR9hFL3iBOuBiJX2dNqk0uDZCD?=
 =?us-ascii?Q?QY7JcUBwu9FM7qAod+XBZ+TN+AM9MQ+ddoPTM97xVcXTWx51nCqvY3pWZui1?=
 =?us-ascii?Q?q1gH2tD5t/fASn3uRJ+jqB8+EVCAz3xxVq364e/2tY4/BdwQqIQscA3FQS5B?=
 =?us-ascii?Q?CDqwzAFX8N51wvn2YePDyT0zXWCNxyDrauLGJD02syuPUCvbHNJ/e9gQrw4E?=
 =?us-ascii?Q?Vfj5MqH5vPAKV3FvixNG7gxHZaTI3ka3GyF5QplqwXDaUDnHnc5DuOvWuyEO?=
 =?us-ascii?Q?U84rGBljPFizQoJO0r0sRnMywtYCNJMEe/OP1ziTtHTYy1lUnRiM6y/uS4bn?=
 =?us-ascii?Q?yOphWoCnzOcNuN1FeCp4a1/Szkc6jW4NvuBFv4ajQDvq1nVrcbzULSu6CHTt?=
 =?us-ascii?Q?SzHEIPWWxoV4tVXyHdMgH/9XlslPDsK8Guj/H+9KmmPyS7tRvqZz88y+DT04?=
 =?us-ascii?Q?r9rjtaftMmvfLbcFvtzFRXAkKaVp4Xz5yD7PpNUsyD00p4pfwETryjsRx0o3?=
 =?us-ascii?Q?LfM1yclhzZU2yjLDQajz4vJ4i+101sRiS3XMtzLld2S5UvCS1JFrC/5g2G0v?=
 =?us-ascii?Q?SbwCNNS1+YSRjJFPbJAiHFUddqL9bGO40pJxNCz3/uURdV5nTOrADvvFTGDB?=
 =?us-ascii?Q?NCT6XzhmSCOk7ikFkp/Q/EwaI+zY2dEndNNhutzHat5vQULWBfqqWb9Nb89i?=
 =?us-ascii?Q?4bocAll8BtXZAUGqfKUJAxROZYvzyOm7UnNW+nVFXSypIuEkzf0Uwjr6fh9X?=
 =?us-ascii?Q?iIz6Yvz19MwlwM4M0WWc4iZdfLrQ/68HuFv2N3abACBXOGiXxD3IjkAIIHOw?=
 =?us-ascii?Q?O82WM1d5ULtlWDtSY+a2dtosZxLzEy7V/wOT4TLCXYjX4mNZInuqWBcTHF4K?=
 =?us-ascii?Q?F/ALKV7JNSvUSO3Nkt9TRoPhp01TCBb5pv2a3MfPC+phKSY2EHW4+7px6afI?=
 =?us-ascii?Q?U1WbuL29lEotmmPpbCy/q6N4IX57Ykg/wVgke+/ajsBGTuFrXRzoc/S9jpWJ?=
 =?us-ascii?Q?WBGgnUSb/i+7fGlfrmhS22DXsK1YU5kc0vjcXTITGVwyopuO8ktYajQPc0aL?=
 =?us-ascii?Q?YNombQG9pvF7j4fgnxZflIozLIw+rI+ObYZ3ckDPFUXp8ZBTHJJQjmCVhQ2y?=
 =?us-ascii?Q?uJ4pUCUpIcjJTpSWLLk1qTrhguVZ+ppbIK/9XhsicpnRn/iggc+xPTo+WMov?=
 =?us-ascii?Q?AdC840NCFGwlo56j96KRKLZFgHhFXTqkBk8lOWbuxE4pVW+1telTYaR4jEu/?=
 =?us-ascii?Q?tcITha8s+YDRzUVOhilnFFQaQ5jTyp5UWGGqsgSlnaq3htix60qbsgd5790x?=
 =?us-ascii?Q?W2WRnOWPXg=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5880.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 643acac6-8ecd-4e7b-5fb4-08da25a1363c
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Apr 2022 03:19:15.5766
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: d1u8VlSP82WXYJ506dmtwp1X7V6/PxdC49pZJn57YUOlOMKBFvd1ofOwSdVWF0zdkfu5HjqmQi4gM/GRE+jOsw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR11MB6173
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On Tue, Apr 19, 2022 at 01:34:26PM +0800, Zqiang wrote:
> If the rcutree.use_softirq is configured, when RCU Stall event
> happened, dump status of all rcuc kthreads who due to starvation
> prevented grace period ends on CPUs that not report quiescent
> state.

>Please accept my apologies for the delay, and please let me try
>again.  ;-)
>
>Your earlier patch added at most one line and one stack backtrace to
>the RCU CPU stall warning text, which is OK.  Sort of, anyway.  I was
>relying on the fact that the people who have (rightly) complained about
>RCU CPU stall-warning verbosity never run with !use_softirq.  But it is
>only a matter of time.  Yes, we could argue that they should use faster
>console serial lines, faster management-console hardware, faster networks,
>faster mass storage, and so on, but I would expect them to in turn ask
>us if we were volunteering to pay for all that.
>
>In contrast, this patch can add one line per stalled CPU on top of the
>existing output.  Which is better than your earlier patch, which could
>add a line plus a stack trace per stalled CPU.  But that can still be
>a lot of added output, and that added output can cause problems.
>
>So, could you please merge this rcuc-stalled information into the
>existing per-CPU line printed by print_cpu_stall_info()?  Right now,
>each such line looks something like this:
>
>rcu: 0-....: (4 ticks this GP) idle=3D1e6/1/0x4000000000000002 softirq=3D1=
2470/12470 fqs=3D2
>
>One approach would be to add the number of jiffies that the rcuc
>task was stalled to this line, maybe something like this:
>
>rcu: 0-....: (4 ticks this GP) idle=3D1e6/1/0x4000000000000002 softirq=3D1=
2470/12470 fqs=3D2 rcuc=3D15384
>
>Of course, this "rcuc=3D" string should only ut only if the stall lasted
>for longer than (say) one eighth of the stall timeout.
>
>Any "(false positive?)" needs to remain at the end of the line:
>
>rcu: 0-....: (4 ticks this GP) idle=3D1e6/1/0x4000000000000002 softirq=3D1=
2470/12470 fqs=3D2 rcuc=3D15384 (false positive?)
>
>Thoughts?

Thanks suggestion, I will resend v3.

>
>							Thanx, Paul
>
> Signed-off-by: Zqiang <qiang1.zhang@intel.com>
> ---
>  v1->v2:
>  rework rcuc_kthread_dump function
>=20
>  kernel/rcu/tree_stall.h | 32 ++++++++++++++++++++++++++------
>  1 file changed, 26 insertions(+), 6 deletions(-)
>=20
> diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
> index d7956c03edbd..fcf0b2e1a71c 100644
> --- a/kernel/rcu/tree_stall.h
> +++ b/kernel/rcu/tree_stall.h
> @@ -465,11 +465,13 @@ static void print_cpu_stall_info(int cpu)
>  	       falsepositive ? " (false positive?)" : "");
>  }
> =20
> -static void rcuc_kthread_dump(struct rcu_data *rdp)
> +static void __rcuc_kthread_dump(int cpu)
>  {
> -	int cpu;
> -	unsigned long j;
> +	struct rcu_data *rdp;
>  	struct task_struct *rcuc;
> +	unsigned long j;
> +
> +	rdp =3D per_cpu_ptr(&rcu_data, cpu);
> =20
>  	rcuc =3D rdp->rcu_cpu_kthread_task;
>  	if (!rcuc)
> @@ -488,6 +490,21 @@ static void rcuc_kthread_dump(struct rcu_data *rdp)
>  		dump_cpu_task(cpu);
>  }
> =20
> +static void rcuc_kthread_dump(void)
> +{
> +	int cpu;
> +	struct rcu_node *rnp;
> +	unsigned long flags;
> +
> +	rcu_for_each_leaf_node(rnp) {
> +		raw_spin_lock_irqsave_rcu_node(rnp, flags);
> +		for_each_leaf_node_possible_cpu(rnp, cpu)
> +			if (rnp->qsmask & leaf_node_cpu_bit(rnp, cpu))
> +				__rcuc_kthread_dump(cpu);
> +		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> +	}
> +}
> +
>  /* Complain about starvation of grace-period kthread.  */
>  static void rcu_check_gp_kthread_starvation(void)
>  {
> @@ -597,6 +614,9 @@ static void print_other_cpu_stall(unsigned long gp_se=
q, unsigned long gps)
>  	if (ndetected) {
>  		rcu_dump_cpu_stacks();
> =20
> +		if (!use_softirq)
> +			rcuc_kthread_dump();
> +
>  		/* Complain about tasks blocking the grace period. */
>  		rcu_for_each_leaf_node(rnp)
>  			rcu_print_detail_task_stall_rnp(rnp);
> @@ -659,11 +679,11 @@ static void print_cpu_stall(unsigned long gps)
>  	rcu_check_gp_kthread_expired_fqs_timer();
>  	rcu_check_gp_kthread_starvation();
> =20
> -	if (!use_softirq)
> -		rcuc_kthread_dump(rdp);
> -
>  	rcu_dump_cpu_stacks();
> =20
> +	if (!use_softirq)
> +		rcuc_kthread_dump();
> +
>  	raw_spin_lock_irqsave_rcu_node(rnp, flags);
>  	/* Rewrite if needed in case of slow consoles. */
>  	if (ULONG_CMP_GE(jiffies, READ_ONCE(rcu_state.jiffies_stall)))
> --=20
> 2.25.1
>=20