Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4;
Date:   Sun, 3 Dec 2023 19:45:30 +0800
From:   Pengfei Xu <pengfei.xu@intel.com>
To:     Steven Rostedt <rostedt@goodmis.org>
CC:     Yu Kuai <yukuai1@huaweicloud.com>,
        Edward Adam Davis <eadavis@qq.com>,
        "syzbot+ed812ed461471ab17a0c@syzkaller.appspotmail.com" 
        <syzbot+ed812ed461471ab17a0c@syzkaller.appspotmail.com>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        "axboe@kernel.dk" <axboe@kernel.dk>,
        "dvyukov@google.com" <dvyukov@google.com>,
        "linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-trace-kernel@vger.kernel.org" 
        <linux-trace-kernel@vger.kernel.org>,
        "mhiramat@kernel.org" <mhiramat@kernel.org>,
        "syzkaller-bugs@googlegroups.com" <syzkaller-bugs@googlegroups.com>,
        "yukuai (C)" <yukuai3@huawei.com>, <andreyknvl@google.com>,
        <balbi@kernel.org>, <heng.su@intel.com>
Subject: Re: [PATCH next] trace/blktrace: fix task hung in blk_trace_ioctl
Message-ID: <ZWxqWjwPft3sG12Y@xpf.sh.intel.com>
References: <00000000000047eb7e060b652d9a@google.com>
 <tencent_6537E04AAC74F976B567603CEB377A96FA09@qq.com>
 <5116cbb4-2c85-2459-5499-56c95bb42d16@huaweicloud.com>
 <20231202170743.7557e7b5@rorschach.local.home>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20231202170743.7557e7b5@rorschach.local.home>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OE80QXZpR2lPanJlT3NLRHZBV3dJYjRwM0Y5b2NPN2N5ZDQ1VnlscXVkVDRE?=
 =?utf-8?B?L0Z3U1d4bU1EWmlWSnFLdlUwSVl4cnNLQmM2WHRYVS8yUkNJUmdqb1BlVm5X?=
 =?utf-8?B?Nm0rdDJwNlA4UXVzb0E0MVJVR2d3UFZyTy95OEpCUitvZUQ0YWRUcnYrUFpk?=
 =?utf-8?B?RDJ4QVhRcC9GS3crYW1SU0REUWhUdkVTT2dWTUFpWmR0QjB1YVpwVURCK2pz?=
 =?utf-8?B?bFFvVFZRQ3lvc1VNVndwRFVrYTNJdkpEbTNFZ0V1ZURJNTV4K0lXTjYwV2Zh?=
 =?utf-8?B?QlE3Uzg5OEYvaFNucjR4ZkdFQVNWa2JueU16Q1kxTExEbHNhWkExOFA5L0FF?=
 =?utf-8?B?SGdHRWF5ZWhEbDllR3NCbTN0MzBOQ29iMUlTSzJUY2JLOHFGOTUwZlVrNmN0?=
 =?utf-8?B?UVZ5QnlxWHR6ZnFvZW93N3E5ck11WHdUQUg2QStNS0tYWHBoUkVnaDJlRWcv?=
 =?utf-8?B?em1OSjlRbU9ISnZwWDdHYUJEdWR2WForV25rOFVBRzFJK2NDZDR4T20wTXZC?=
 =?utf-8?B?SzVaSkp6c1lVU2g1RmZ0UFVNeFcyaHNjS2x6OVY0cHZOTVRwclZtK3hRQlJX?=
 =?utf-8?B?TFlJNEVYQndKa2ZUS3FVU0N6L1ZNNzJ3dVpZOXA2QUc1dm1heWpCS2FXSVhF?=
 =?utf-8?B?ZWxUeVBzWUJzSmV5QS9LUzZqOUJiMGNnMkJseDZnRERzQkRrTGhRNUE4RFFx?=
 =?utf-8?B?VlVnOFFrZXFPYWpoWXduQVdjczlFYjVjRkRaOG1kbzdLNnRhOENkQXJ0M0Zu?=
 =?utf-8?B?OGxLcHhabVB6MTUyN1JySGdiN0hZWXpLYjIyVStZbXRGSlhWQWg5WVhWVjYy?=
 =?utf-8?B?aHVJWGIyMkFUUXZqQzltQTRETm9STFhIcmpCTEZ3V0UybWVMeUdXL2p6VjZI?=
 =?utf-8?B?VlBHVHpwYzJmbTRBdkNDVDFLQjltc2J6c2JQdlVLcURQaENEekwvaElxL0tK?=
 =?utf-8?B?WDZOL1E4WVp3WVZKZEwyY2k5TGZOandjSVdySFZxWFFlNnBmcVM4MUIrQWtQ?=
 =?utf-8?B?MjZyZXIwRHkwemwxdXRPaXJ5M0Z2UmpwSmYyb1NuemYrVXgxK29pVG5ISmZF?=
 =?utf-8?B?V3ZSSFV3MGVudDhBRmNJbnF0Zlo5cnU2TU9QSmh2SVNkV3RoUURBYjJhRkY2?=
 =?utf-8?B?V1B0Ty96b2JRUWNhMW5yMi9oNnFueW92K3YzRURFSm1wY0dIUjJud0Q0Y0dU?=
 =?utf-8?B?YW5yYXJyN1JSc2h4RGdxRiswMlNDbEF5ZndxdjFUK2tSYkVkQWJQSGxLeXdI?=
 =?utf-8?B?WTFabFlHUEwwNVZmNXIvN0JqaFJZayt2T1JJTmF3UlZna2licFFaUDVrN3Nu?=
 =?utf-8?B?OXZZS1FqNXRsQ0hKNGdacHZHYWdQSXdHb2hHbTM1cHVVWkZXZXFjV2kvTXRv?=
 =?utf-8?B?L3lBYU5hOG5kNDh0Z25oVXlmRnRwNU1qbzlrVUI5SFpSckszUzRQSXpoR0Vh?=
 =?utf-8?B?REEyNm9OdzczZllqWW9VSlJLL1hSVE45T0ZSRy9MVHRrekFjN0RjWmw2SnRF?=
 =?utf-8?B?c0IyWTY1bWhyaWtiNXRlVlZFb1NuVjVaZEZjQVo0bHhyWGIvRGtGVE80TUlL?=
 =?utf-8?B?WDNsbFlQMTdnekM2WUJoYkRFeEJpL3kraDljdkJCeTV5Smx3SDFEcFVFbGhR?=
 =?utf-8?B?eWtMc1hJY09rNlgyOU13dlZNRmJxNDVtdldLdENWenpxRVJERjdSUTRyQWNs?=
 =?utf-8?B?b2xiZXdaRkF0TWg3UmZJb29PWEdtOHBKQTBSeTkzdXd5U1F2Q2ZKZHdrZlpU?=
 =?utf-8?B?ckRMUHNIMUxVWnFpRzhWc0dnMjZPNGdBaktXWGhIZDR0RDc4cXN5T1ljZi93?=
 =?utf-8?B?d2VCMnRlbG0vaFpsb1NOR05kV2RRMExDTkQ1TUtWRUVibUZEYU03Z2ZFT0Rk?=
 =?utf-8?B?WFRQTEJieDM4UlpOK0tnTGwzZjY0MXZzSU53OEFBamdJVFNHNldQWHFLRVZl?=
 =?utf-8?B?S0lUeEtqTmhmWnRXYktISnVPcWhyZnJjdis1WkFkM04zWVNtdWxFbGVnNEI0?=
 =?utf-8?B?ZmtoMGNGRjdzbW1xUFc1Q3ROelpNNUtkZzc0bG5keDhuR0NaWnJPd3h0eGM0?=
 =?utf-8?B?TzFaRS9Ud3ZLa0djMXB0cWZ6VnlYNjllTi94T3phRFBMOWczTXB1TjU5cUpm?=
 =?utf-8?Q?T/Wpu2x8ku9entqi3rStACbLT?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d4baadd5-704d-4334-ebc4-08dbf3f62bbd
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB4839.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Dec 2023 11:51:22.6258
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: kppsK8hNX+aN7v8TxIzCakECyEkw9f77ut+h+Sos/dPeECirm4I9oPIDbajp2iKtkCb9z4mdwCHAPO/ARsvJTA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB6784
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Sun, 03 Dec 2023 03:51:42 -0800 (PST)

Hi,

On 2023-12-03 at 06:07:43 +0800, Steven Rostedt wrote:
> On Sat, 2 Dec 2023 17:19:25 +0800
> Yu Kuai <yukuai1@huaweicloud.com> wrote:
> 
> > Hi,
> > 
> > 在 2023/12/02 17:01, Edward Adam Davis 写道:
> > > The reproducer involves running test programs on multiple processors separately,
> > > in order to enter blkdev_ioctl() and ultimately reach blk_trace_ioctl() through
> > > two different paths, triggering an AA deadlock.
> > > 
> > > 	CPU0						CPU1
> > > 	---						---
> > > 	mutex_lock(&q->debugfs_mutex)			mutex_lock(&q->debugfs_mutex)
> > > 	mutex_lock(&q->debugfs_mutex)			mutex_lock(&q->debugfs_mutex)
> > > 
> > > 
> > > The first path:
> > > blkdev_ioctl()->
> > > 	blk_trace_ioctl()->
> > > 		mutex_lock(&q->debugfs_mutex)
> > > 
> > > The second path:
> > > blkdev_ioctl()->				
> > > 	blkdev_common_ioctl()->
> > > 		blk_trace_ioctl()->
> > > 			mutex_lock(&q->debugfs_mutex)  
> > I still don't understand how this AA deadlock is triggered, does the
> > 'debugfs_mutex' already held before calling blk_trace_ioctl()?
> 
> Right, I don't see where the mutex is taken twice. You don't need two
> paths for an AA lock, you only need one.
> 
> > 
> > > 
> > > The solution I have proposed is to exit blk_trace_ioctl() to avoid AA locks if
> > > a task has already obtained debugfs_mutex.
> > > 
> > > Fixes: 0d345996e4cb ("x86/kernel: increase kcov coverage under arch/x86/kernel folder")
> 
> How does it fix the above? I don't see how the above is even related to this.

I bisected this issue and the following fix information is more accurate:
"
Fixes: f2c2e717642c ("usb: gadget: add raw-gadget interface")
"

All the bisected info is in link: https://github.com/xupengfe/syzkaller_logs/tree/main/231203_140738_blk_trace_ioctl

Acked-by: Pengfei Xu <pengfei.xu@intel.com>

Thanks!

> 
> -- Steve
> 
> > > Reported-and-tested-by: syzbot+ed812ed461471ab17a0c@syzkaller.appspotmail.com
> > > Signed-off-by: Edward Adam Davis <eadavis@qq.com>
> > > ---
> > >   kernel/trace/blktrace.c | 3 ++-
> > >   1 file changed, 2 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c