Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Zhang, Qiang1" <qiang1.zhang@intel.com>
To:     Peter Zijlstra <peterz@infradead.org>
CC:     "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH] irq_work: Add event-tracing points for irq_work
Thread-Topic: [PATCH] irq_work: Add event-tracing points for irq_work
Thread-Index: AQHYT8pATG0NrCtQb021NYs63/f6NKzvJWsAgAEN6hA=
Date:   Fri, 15 Apr 2022 02:56:03 +0000
Message-ID: <PH0PR11MB5880057FA69287500580FEA3DAEE9@PH0PR11MB5880.namprd11.prod.outlook.com>
References: <20220414063829.2472251-1-qiang1.zhang@intel.com>
 <YlfqFJLau+7PlPJ3@hirez.programming.kicks-ass.net>
In-Reply-To: <YlfqFJLau+7PlPJ3@hirez.programming.kicks-ass.net>
Accept-Language: en-US
Content-Language: en-US
dlp-product: dlpe-windows
dlp-version: 11.6.401.20
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?jUVpDzjeKZkwo1AYeyBd3kJFz4BHB1YzkdabDdwh9GfPh7TWH8aqrhPfREVm?=
 =?us-ascii?Q?CEOn6FkyTw0p3jRoxueS37rjk9hfT4yT+VhUurr9+3dSi2YBDoWkRi8xtCtA?=
 =?us-ascii?Q?DAL3F8jXSC/oID6M57L+wV12GXN7gEHcuygVRIBLZgjbAlxx3mLnZpONGLPs?=
 =?us-ascii?Q?16QrLVgVcElNdVlhndHgI2Vq65Mh/ez8Bf2Rg1j5jZnaV9xqKlgbqHiANcpQ?=
 =?us-ascii?Q?fbgkMCvSSfNmnE5B5yjV6rVCaLehpqpQGuyGtI+3SC/UWFRQJR0H6okSur2l?=
 =?us-ascii?Q?rh5oRTkX5/esvxzoPtEFPf0RmyryikSZ8cgORX6YwA9kO/myew9SPXnjqrVW?=
 =?us-ascii?Q?EdXJgAymCrXktvdOWdPby1JCW53znH8C5ZeSkxmRe9EvxMJv1SW9F5OMwg16?=
 =?us-ascii?Q?Dvtngf7iXcWX2DU0Tbeajh7opn5+dEJ3eU13+DGRDjRDFGvM4Up6x1yRP8bY?=
 =?us-ascii?Q?BZRENX+TEcluSl39R3KMf6SP5vvDlSl68LtTB+RVddeK5YISveGFIQhEAA1W?=
 =?us-ascii?Q?UB2bEVT7SGn2owLyTPIfibLz6gqgF/nld7P33Im3X2qG3lqgigFdLbCqcvDg?=
 =?us-ascii?Q?lBpKJjNnuABcY87+bO3Nid27Ek7mFa+Fz5l6GJe9cMayXwkrLmhdwCdX6YEl?=
 =?us-ascii?Q?lxQyCjQpQolOlZM2V2OtY1XCXyfSevNIFjxIsFs5QvN7kjWvZTxkmECLFiy1?=
 =?us-ascii?Q?fGsYyAny1i25U4YCusxopl52vn5IHVoYFArqPho86JLxIitSGlTlSJO+75IM?=
 =?us-ascii?Q?Sk0A6PFond5fyveI0umxds4m0oDvgnCXqLVtiLvmUXQjBojqNhCCiSRlh/FF?=
 =?us-ascii?Q?6owoTV57aWRdJDC8hBudNiEsnDSgd8tmoyMKO3DjXooehlbOMCRWMMacxYEo?=
 =?us-ascii?Q?JGs+9UnKiOKbxZSlC1BzkXgbmo56STF2Dz+v/PtcHqzaVau8wNyrsWKw52E9?=
 =?us-ascii?Q?OXylbQixM4hcoig9Z7TQg6f46rneSeimntQwYTgcZuoGKW2HA/FVMkv5fKgC?=
 =?us-ascii?Q?VJ+kQUmSTljVy42PhQQeFndKWjZVb6uq2Or6htTxXpNH7sq44Rl9jz5LlPLO?=
 =?us-ascii?Q?nseWlMaH4MPDRcJtd3NnDOoUg/fstuzPRZDaN17TUmaXuQInpEEXctnSaCeP?=
 =?us-ascii?Q?xaL+HCm93nN6MV18ozePVtu3JG8AbiqelYK/7rpMwaNx5a9UvhVVFgoidG6J?=
 =?us-ascii?Q?G7SYYnkrGg4FtvjyWgulFj7eb5Lg/M8atMNBeoHvuUZU2jm43x5Nu7p2mDA3?=
 =?us-ascii?Q?l8KNVo4dY7MrPoQBJROjp6ZqtEtJCs52U1uafHV2Id9g6tVkDXQMFxwNNBIL?=
 =?us-ascii?Q?wUsoLySaljScqqzs3vUszh9gkDhOvcpvsWuxS7AwguOqYRlzgrUuIcDqSP/o?=
 =?us-ascii?Q?WzLWnDiKrh49L8aiPLTVH47WwbdXV0wjSToNaifR1EjDSuw+QG4xUrVht2Ze?=
 =?us-ascii?Q?CpbWefb7eToymRJpfV+862UIxPJD4P8DOSSvLEatKKFq49eTidfS4zGafa1B?=
 =?us-ascii?Q?1mQSq9MWYEpW9MABpsiwz+u8DJjxjVE3yEDc7s8Mec/GFYNLXoMtU96q7xJN?=
 =?us-ascii?Q?1s4CNucnK+3zfFnA4WFmlVdwpXQPXoUzxfeLwmOcIGlz3BOyRoiobXFai8oN?=
 =?us-ascii?Q?jx/fn5rxdrthXp8kua6XujZW2uKHWqWRTpO4zGh/LGn/Veg4kM8VVo1VcUOK?=
 =?us-ascii?Q?5jugYDiopAJyR4mFsNzIJPBx3mvy91yhC1vKSEQ6gZ8dW2tGGyCma+b1BYwg?=
 =?us-ascii?Q?WaV5X/SC6g=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5880.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2a0a3421-d924-4fda-eab9-08da1e8b7aa4
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Apr 2022 02:56:03.2311
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: LjSNx5G9MbDO4XZfCmEvMQNM1FDNjdoBiyvtcNE3Hc3DEWswLWuj233LNDwL5Tm1NbYHJ0mkqOWOnRWYH07R3A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR11MB4134
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Apr 14, 2022 at 02:38:29PM +0800, Zqiang wrote:
> Add irq_work_queue_on tracepoints allow tracing when and how a=20
> irq-work is queued, irq_work_execute_start/end tracepoints allow know=20
> when a irq-work is executed and the executed time.

Yeah, but why?=09

Tracing stacktrace help find where a irq-work is queued, and view whether a=
 irq-work
be executed  in irq context or task context,  if in task context, is it bec=
ause there are lots of
lazy irq-works executed in irq_work rt kthreads,  cause other fair task can=
 not get CPU,=20
for example, there is a problem in RCU which I find in testing
Enable RCU_STRICT_GRACE_PERIOD in preempt-rt, the system booting occur hang=
,=20
through add irq-work tracepoints find a lot of irq-work (rcu_preempt_deferr=
ed_qs_handler)
be executed in rt-fifo irq_work kthreads,  init kthreads  unable to obtain =
CPUs, cause hang.

=20
> diff --git a/kernel/irq_work.c b/kernel/irq_work.c index=20
> 7afa40fe5cc4..edad992556d0 100644
> --- a/kernel/irq_work.c
> +++ b/kernel/irq_work.c
> @@ -22,6 +22,9 @@
>  #include <asm/processor.h>
>  #include <linux/kasan.h>
> =20
> +#define CREATE_TRACE_POINTS
> +#include <trace/events/irq_work.h>
> +
>  static DEFINE_PER_CPU(struct llist_head, raised_list);  static=20
> DEFINE_PER_CPU(struct llist_head, lazy_list);  static=20
> DEFINE_PER_CPU(struct task_struct *, irq_workd); @@ -81,7 +84,9 @@=20
> static void __irq_work_queue_local(struct irq_work *work)
>  	bool rt_lazy_work =3D false;
>  	bool lazy_work =3D false;
>  	int work_flags;
> +	int cpu =3D smp_processor_id();
> =20
> +	trace_irq_work_queue_on(cpu, work);

That's not very nice, you made this function more expensive for no reason. =
And you violated coding style while doing it :/

You mean the trace_irq_work_queue_on() tracepoints should not be added in t=
his place ? or
replace this tracepoint with  trace_irq_work_queue_on(raw_smp_processor_id(=
), work) ?

Thanks,
Zqiang

>  	work_flags =3D atomic_read(&work->node.a_flags);
>  	if (work_flags & IRQ_WORK_LAZY)
>  		lazy_work =3D true;
> @@ -143,7 +148,7 @@ bool irq_work_queue_on(struct irq_work *work, int cpu=
)
>  	if (cpu !=3D smp_processor_id()) {
>  		/* Arch remote IPI send/receive backend aren't NMI safe */
>  		WARN_ON_ONCE(in_nmi());
> -
> +		trace_irq_work_queue_on(cpu, work);
>  		/*
>  		 * On PREEMPT_RT the items which are not marked as
>  		 * IRQ_WORK_HARD_IRQ are added to the lazy list and a HARD work @@=20
> -208,7 +213,9 @@ void irq_work_single(void *arg)
>  	smp_mb();
> =20
>  	lockdep_irq_work_enter(flags);
> +	trace_irq_work_execute_start(work);
>  	work->func(work);
> +	trace_irq_work_execute_end(work);
>  	lockdep_irq_work_exit(flags);
> =20
>  	/*