Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4;
Message-ID: <b3a5e717-90ab-42f8-8fe1-4a24fff404c3@oracle.com>
Date:   Fri, 29 Sep 2023 11:03:47 -0500
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] scsi: qla4xxx: fix potential deadlock on frwd/back lock
To:     Chengfeng Ye <dg573847474@gmail.com>, njavali@marvell.com,
        mrangankar@marvell.com, GR-QLogic-Storage-Upstream@marvell.com,
        jejb@linux.ibm.com, martin.petersen@oracle.com, lduncan@suse.com,
        cleech@redhat.com
Cc:     linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org
References: <20230928145401.32918-1-dg573847474@gmail.com>
Content-Language: en-US
From:   Mike Christie <michael.christie@oracle.com>
In-Reply-To: <20230928145401.32918-1-dg573847474@gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?LytoaEpmUTFSRmxhdXZndUE4UGVIelpGUGEyRjh3YVdua2l4Mi9mZHJyZDhM?=
 =?utf-8?B?LzJwS3NtOGRhVUF3WGVnWWNWdjd2Ykh6blZpeXQvUndaMWxmckQwaDB6T1U5?=
 =?utf-8?B?WHpHOXBQT0VmMHFGZE8wM25aVUFjU2tZNGJHcnRLOERBN01PelhET0JPQzdi?=
 =?utf-8?B?dzZlREVHcVNFQS92WmFNaDM5a0w2YUY4ek9SV2FDZS9aM1g1UG5yMDhZdHV5?=
 =?utf-8?B?V0VHTDlVeTNxVGdPU2pvRS9kaEx1bWIra1JPSy9Jd3lwS2hMWTNGSVgwMUhS?=
 =?utf-8?B?eUNvM1lsZVd0blZsYytxRGQ1dmk1eDhCQ1RRQzlwOERUek5SclhUSTViWDIy?=
 =?utf-8?B?ZzdtQTFkS1NoUm1sR2pMbmpFcUZ1MnBYZGk3M2czeitkd1RPcU9BUGJadlRM?=
 =?utf-8?B?M1B4N3B3NEZzSng0Sm4wZzhaYXNyeWovV08wQlFqTlRFZElJYlh0dGJpUitK?=
 =?utf-8?B?M0VvZ1dXbVhWUTIyVldMZGNzYU0rSEJQaGRPc2VZQWlrdFh3eHhheHZTSWpz?=
 =?utf-8?B?RnBhNWlxNERyV3o1VXRSU3NvNDBpVzd6QkRKTEU4cnJkRlZIaEhyV3EyTm9H?=
 =?utf-8?B?bGJrOGlCMmZWQkJOT2RJMXdGYThoSjJ2Q3F5end6Z2NmS0tmdW1CNUhxWi9l?=
 =?utf-8?B?dHhHSUduOEY4R0t4emgvcDg4QzhCL0FON2VTL0RaUVBRV0ppV1VVaWNkK0Mx?=
 =?utf-8?B?RG1kTVBpME12MTJPQStTbGxhd3dYNUU2UzhtUXJnNlhTa1hUZUYvaUNnc1FX?=
 =?utf-8?B?cFdNSkd0b3dpMjkvU3BhWG1jL0hrWXk4OEI3bEZrWDZLM0dBZkE2UWRLYS8z?=
 =?utf-8?B?enVjY3lkb0dGZFkxOHJTbTljVzhmWWs1LzFlQTZFRzBsdTk1cGVYU0hORTZo?=
 =?utf-8?B?VEZKK2ZzZEh5L2hEbXFqWFYzc2FucHJqTnpnWW9rWEJRREpYR0psdTJ0QzRR?=
 =?utf-8?B?SEZzUHdISXg0MWs0LzNjU0huK0xIOEt2UElYb0NPVE9sUEJHSER2Uk15UmJZ?=
 =?utf-8?B?SmZ4Q3ZuRnFpTUVCYVJoR1N2V3hCYnFTV0p0VlRhZDJuTkxxWWtad2hIcjZD?=
 =?utf-8?B?WkJEQW1tcmFTRFNibXFacVRyRUlKdjVnRU9TcTdkd3E4Z1dBREdDd3B0VGZU?=
 =?utf-8?B?Yy9kaGlDblIyYlNWWTlvQ1pPV1BGdFJudWROKzI3Q2QrZHgzSmxHU2pFMmNy?=
 =?utf-8?B?YzhWeHZFMkEwK1RFVUpjVVQrajUwa1QxbWxlR2plenYyRFY5K294T0RTcHU1?=
 =?utf-8?B?M3dybWg1Nlg1cjNxMEw4RzlISnM4TFNzbWh5MUdLcmlGOTNEeldWTVoyWTg3?=
 =?utf-8?B?bXFWVGd2L1B4eW9FejVvNDFESytkV2Zob1dDOUZCUkErSUpXSGNoVUhkNHdM?=
 =?utf-8?B?cXNmZjlqR1NXdnM3KzFYQ0xSMW1TbFdDdlVxd1NUZm56VzNCSmdDVHBqZEVw?=
 =?utf-8?B?NXhlSlErMWpOU3RyOTJzL1JiaTlaYmhPeTVCb0dzNEh3V3ZtS013T1F1KzI3?=
 =?utf-8?B?SXA1cmpLcVVQUTlqZVMyWXQ4RUlka1BiUVRhR0tjRnRVc0tScEwzaGYwc1NW?=
 =?utf-8?B?OFEvRVNYd05jUVpjMlkvQ0tvMytGSm5GYVFNZ29vQXRMeVF2RXRFVUZwVnpP?=
 =?utf-8?B?dVoybjVhKzNVd1F0MWUyakRETkJUWTh1QkVyWlI1YWlvSXFsQ0R3bUM5ZHZR?=
 =?utf-8?B?SVplVDQ5eWpiQmtoL1FRc1dnYUd1RFcveWJYaXJOeW5SajY0S1lJMDRtUEh3?=
 =?utf-8?B?Vkh2WEtTdjg5ditWWnhJYktlZzJwN2Z1dEFrelZ3UUVoUzJ4VlRSdzhGRHVh?=
 =?utf-8?B?NkZyRlo5Ky8rdFFOTGVTZjRzVDdNdFRLV1lva0RlaklWOExhN1dsdlA5dFl0?=
 =?utf-8?B?bDBWZ21Ma0tIU1BsSkg3Q05HU05jZjNLYkV3ck44MEV2cU5SbTlmQ0N2NnRU?=
 =?utf-8?B?dVhtNy8weThYL3kxaUg3dlhiN3A0a1huWE1mVEpHakVNdHlLV0VRa1Y1NnpK?=
 =?utf-8?B?OCtSc01jVFhXRHFtaXRodGd3UTdwTEx0THpnVkk1T0JxMU92YnNIaGpoNHRL?=
 =?utf-8?B?MXRNUURBL3l4dHhEK0xnUUVjNjlUTHpBdm4rTFhiK2tGZ0FJMjRyVjd5Tk1J?=
 =?utf-8?B?M2ZlT2Z4eXhXaFpuMDJrSlpRSjJkYVhtSWxMdENTVld6Q2tSTFJKYkFacWZV?=
 =?utf-8?B?dlE9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: XUtbcieJ0J6bX6jQi4LSrbjHv15SP6ibiaAeUosyb27VIiyWCPgMqOYbIePtEvp9KuiwJilEVBp0vVqjkeqCNjwbjCMx0MClOTqlL4JKo9g3TpXD3nsVRKfkEiQ9rx+crytpGUJoxkj7eOc519YIS4MLa6X9ntlO2eY4tE/YRl2wagvPV4UmCrcdyzLeDPN6kSVpxvu7Y6bl/3ug5I2x/X8N/Gx1ENXkZS4nN6trP24EhpSRlc8s3PTpXifDj+yVi+nZgK8bAmXVAKx/S3tCLDynFQWf+SIqNZQx2j1UGnf78GQ05dXCjqcaQaVYRo/ZL9J4tcENvcWQueT06Ia+BTz9RoWS5Y0llYD0jO54o8FYtoc+GgPU29BhCBM+9PXTNWCy/D3k9gKNKGMKKL8PEvoE/t++KHm5LHPvHdOUJ4xreOOjaBcCj87clyEbL5aIaRkjEdUhpoR3cpJ4vMk6YbUpoi4SZAkEJKPdjW6dSFXlESlzJAu6njMQslhbofeF8hzYenIwj91UZfeRlNK4dDOoCmVAiSuUtJkrdXrTaUw8CUGLQ9ymkHsaGBCV1lZh0RFNGV42ONpJEDjD7lL/Yw2lvK2/AGeV080g9xjZXcQ8R/Dg5HofKkF6k2VOgs83RqzZABFwMpKjZeT7sSRhC0wd5BLSGvvrlAsjb09TtSrW5Yf71TB1LjnyCyIDgitcAPZyYZe+IVHc1fX2v7C7YP8l11eoDQZ5wghiVnuZ7qSSWys+xNdyf5SneiNStlRVlOZx0Lu6u0u1RMtMAORcoBLSAwLceh0ufvMtcDNOFSyFL1+27Yl7rUUTpQj1k4robaFdyR/2uiiDC5631W6ppAyEPsFBPnZ7rSxs3F5ytynC8Im8eWfhnJZyBiDkB4FhNEe/OT4rYJ3WVWyyADhI2IaSZSll+AuS7w2kgCTbK00=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c9d88d0a-21a2-424e-fb72-08dbc105ab4b
X-MS-Exchange-CrossTenant-AuthSource: CY8PR10MB7243.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Sep 2023 16:03:49.7684
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zIYkYOwHEq1rHSTFA8uTXgY8gZNP3Yo1QCNCYF7Sb/Lv1RynWdSg59E+3k7X84S0Mjc2J7+9m+XRTrHddm51uCoLh8XRugi0+3PFHMVvcF4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB7840
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-09-29_14,2023-09-28_03,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0
 suspectscore=0 adultscore=0 phishscore=0 mlxscore=0 spamscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2309180000 definitions=main-2309290139
X-Proofpoint-ORIG-GUID: yOX9BMqCr392VBK7c51FwXMGJTDCa8Pv
X-Proofpoint-GUID: yOX9BMqCr392VBK7c51FwXMGJTDCa8Pv
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 29 Sep 2023 09:04:24 -0700 (PDT)

On 9/28/23 9:54 AM, Chengfeng Ye wrote:
> iscsi_complete_pdu() is called by several rx callback, under
> softirq context. Thus the callsite of it inside qla4xxx_task_work()
> should better disable bottom half, as work queue is executed under
> process context, it needs to protect race with softirq context
> locking to avoid deadlock.
> 
> <potential deadlock #1>
> qla4xxx_task_work()
> --> iscsi_complete_pdu()
> --> spin_lock(&conn->session->back_lock);
> <interrupt>
>    --> iscsi_tcp_data_recv_done()
>    --> iscsi_complete_pdu()
>    --> spin_lock(&conn->session->back_lock) (deadlock)
> 

This will not happen, because qla4xxx does not use the libiscsi_tcp
code. If you had a iscsi session with qla4xxx and another
iscsi session with iscsi_tcp, then you could get an interrupt like
this, but the "conn->session->back_lock" that's taken above would
be for the iscsi_tcp session.

So you would need to make sure that the above can't happen for
iscsi_tcp an iscsi_tcp session. Same with the below analysis.


> <potential deadlock #2>
> qla4xxx_task_work()
> --> iscsi_complete_pdu()
> --> __iscsi_complete_pdu()
> --> spin_lock(&conn->session->frwd_lock)
> <interrupt>
>    --> iscsi_tcp_data_recv_done()
>    --> iscsi_complete_pdu()
>    --> __iscsi_complete_pdu()
>    --> spin_lock(&conn->session->frwd_lock) (deadlock)
> 
> This flaw was found by an experimental static analysis tool I am
> developing for irq-related deadlock.
> 
> To avoid the problem, disable bh inside qla4xxx_task_work() before
> calling iscsi_complete_pdu().
> 
> Signed-off-by: Chengfeng Ye <dg573847474@gmail.com>
> ---
>  drivers/scsi/qla4xxx/ql4_os.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/scsi/qla4xxx/ql4_os.c b/drivers/scsi/qla4xxx/ql4_os.c
> index 675332e49a7b..c60781148e6c 100644
> --- a/drivers/scsi/qla4xxx/ql4_os.c
> +++ b/drivers/scsi/qla4xxx/ql4_os.c
> @@ -3382,7 +3382,9 @@ static void qla4xxx_task_work(struct work_struct *wdata)
>  		hdr->itt = itt;
>  		data = task_data->resp_buffer + hdr_len;
>  		data_len = task_data->resp_len - hdr_len;
> +		local_bh_disable();
>  		iscsi_complete_pdu(conn, hdr, data, data_len);
> +		local_bh_enable();
>  		break;
>  	default:
>  		ql4_printk(KERN_ERR, ha, "Passthru failed status = 0x%x\n",