Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Gulam Mohamed <gulam.mohamed@oracle.com>
To:     Ming Lei <ming.lei@redhat.com>
CC:     "hch@infradead.org" <hch@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
        Junxiao Bi <junxiao.bi@oracle.com>,
        Martin Petersen <martin.petersen@oracle.com>,
        "axboe@kernel.dk" <axboe@kernel.dk>
Subject: RE: Race condition in Kernel
Thread-Topic: Race condition in Kernel
Thread-Index: Adcgp2qLb8M/LezAQRGiUPxNQ9d23QAcTiOAAX6XFUA=
Date:   Thu, 1 Apr 2021 16:27:37 +0000
Message-ID: <CO1PR10MB4563673A20326DF5170CF60F987B9@CO1PR10MB4563.namprd10.prod.outlook.com>
References: <CO1PR10MB4563A6404AD789EEF93F995798639@CO1PR10MB4563.namprd10.prod.outlook.com>
 <YFvrZUzmdLpj7E4F@T590>
In-Reply-To: <YFvrZUzmdLpj7E4F@T590>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 9a200f42-db8e-439a-6f02-08d8f52b1013
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Apr 2021 16:27:37.4326
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 6vaI8kAVLo/6sfR50KoZpglwH6DrqearaT8dE8skIJo5zwZDzGXHdWcb+CawHDCsPr8B6++Exb7h6FEq2nl7DrZlCHS3FjQcX1Tc3nvo8uQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1001MB2125
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9941 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxscore=0 bulkscore=0
 suspectscore=0 phishscore=0 malwarescore=0 mlxlogscore=999 spamscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103310000
 definitions=main-2104010107
X-Proofpoint-GUID: AtqUYuruzaL2Est0Oy1-5AjXJw7mAlsW
X-Proofpoint-ORIG-GUID: AtqUYuruzaL2Est0Oy1-5AjXJw7mAlsW
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9941 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999 mlxscore=0
 lowpriorityscore=0 suspectscore=0 priorityscore=1501 phishscore=0
 clxscore=1015 impostorscore=0 malwarescore=0 bulkscore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103310000
 definitions=main-2104010107
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Ming,

      Thanks for taking a look into this. Can you please see my inline comm=
ents in below mail?

Regards,
Gulam Mohamed.

-----Original Message-----
From: Ming Lei <ming.lei@redhat.com>=20
Sent: Thursday, March 25, 2021 7:16 AM
To: Gulam Mohamed <gulam.mohamed@oracle.com>
Cc: hch@infradead.org; linux-kernel@vger.kernel.org; linux-block@vger.kerne=
l.org; Junxiao Bi <junxiao.bi@oracle.com>; Martin Petersen <martin.petersen=
@oracle.com>; axboe@kernel.dk
Subject: Re: Race condition in Kernel

On Wed, Mar 24, 2021 at 12:37:03PM +0000, Gulam Mohamed wrote:
> Hi All,
>=20
> We are facing a stale link (of the device) issue during the iscsi-logout =
process if we use parted command just before the iscsi logout. Here are the=
 details:
> 	 	=20
> As part of iscsi logout, the partitions and the disk will be removed. The=
 parted command, used to list the partitions, will open the disk in RW mode=
 which results in systemd-udevd re-reading the partitions. This will trigge=
r the rescan partitions which will also delete and re-add the partitions. S=
o, both iscsi logout processing and the parted (through systemd-udevd) will=
 be involved in add/delete of partitions. In our case, the following sequen=
ce of operations happened (the iscsi device is /dev/sdb with partition sdb1=
):
> =09
> 	1. sdb1 was removed by PARTED
> 	2. kworker, as part of iscsi logout, couldn't remove sdb1 as it was alre=
ady removed by PARTED
> 	3. sdb1 was added by parted

After kworker is started for logout, I guess all IOs are supposed to be fai=
led at that time, so just wondering why 'sdb1' is still added by parted(sys=
temd-udev)?=20
ioctl(BLKRRPART) needs to read partition table for adding back partitions, =
if IOs are failed by iscsi logout, I guess the issue can be avoided too?

[GULAM]: Yes, the ioctl(BLKRRPART) reads the partition table for adding bac=
k the partitions. I kept a printk in the code just after the partition tabl=
e is read. Noticed that the partition table was read before the iscsi-logou=
t kworker started the logout processing.
                   Following are the logs for your reference:

 Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: ORA:: Calling sysfs_delete_l=
ink() for dev: sdb3 command: systemd-udevd		<=3D=3D sdb3 Removed by PARTED=
=20
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: ORA:: rescan_partitions() Rea=
d Complete to the disk: sdb command: systemd-udevd   <=3D=3D Reading sdb co=
mpleted, before iscsi-logout worker started
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: ORA:: Calling sysfs_delete_li=
nk() for dev: 3:0:0:0 command: kworker/u16:3
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: sdb: sdb1 sdb2 sdb3
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: ORA:: device: 'sdb3': device_=
add command: systemd-udevd		<=3D=3D sdb3 Added by PARTED=20
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: ORA:: Calling sysfs_delete_li=
nk() for dev: 8:16 command: kworker/u16:3
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: ORA:: Calling sysfs_delete_li=
nk() for dev: sdb command: kworker/u16:3	<=3D=3D sdb Removed by iscsi=20
Apr  1 09:23:27 gms-iscsi-initiator-2 kernel: scsi 3:0:0:0: alua: Detached

--
Ming