Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Gulam Mohamed <gulam.mohamed@oracle.com>
To:     "hch@infradead.org" <hch@infradead.org>,
        "ming.lei@redhat.com" <ming.lei@redhat.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-block@vger.kernel.org" <linux-block@vger.kernel.org>
CC:     Junxiao Bi <junxiao.bi@oracle.com>,
        Martin Petersen <martin.petersen@oracle.com>,
        "axboe@kernel.dk" <axboe@kernel.dk>
Subject: Race condition in Kernel
Thread-Topic: Race condition in Kernel
Thread-Index: Adcgp2qLb8M/LezAQRGiUPxNQ9d23Q==
Date:   Wed, 24 Mar 2021 12:37:03 +0000
Message-ID: <CO1PR10MB4563A6404AD789EEF93F995798639@CO1PR10MB4563.namprd10.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 9799f0b1-c7c2-464c-f18e-08d8eec18728
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Mar 2021 12:37:03.5376
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 3Lk/Nf5npVWdyvInRozu84rmOw1YXzMwWd2CY17rLHlYzuv0fqa18OZSQK6wr5b6aBb5IodgfID4EuYCCdjzL/U5MHN08+IngMf4HfOnCrg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1001MB2270
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9932 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 phishscore=0
 mlxlogscore=835 suspectscore=0 spamscore=0 malwarescore=0 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000
 definitions=main-2103240096
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9932 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 lowpriorityscore=0 priorityscore=1501
 impostorscore=0 spamscore=0 mlxscore=0 suspectscore=0 mlxlogscore=999
 phishscore=0 bulkscore=0 adultscore=0 malwarescore=0 clxscore=1011
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000
 definitions=main-2103240096
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi All,

We are facing a stale link (of the device) issue during the iscsi-logout pr=
ocess if we use parted command just before the iscsi logout. Here are the d=
etails:
	 	=20
As part of iscsi logout, the partitions and the disk will be removed. The p=
arted command, used to list the partitions, will open the disk in RW mode w=
hich results in systemd-udevd re-reading the partitions. This will trigger =
the rescan partitions which will also delete and re-add the partitions. So,=
 both iscsi logout processing and the parted (through systemd-udevd) will b=
e involved in add/delete of partitions. In our case, the following sequence=
 of operations happened (the iscsi device is /dev/sdb with partition sdb1):
=09
	1. sdb1 was removed by PARTED
	2. kworker, as part of iscsi logout, couldn't remove sdb1 as it was alread=
y removed by PARTED
	3. sdb1 was added by parted
	4. sdb was NOW removed as part of iscsi logout (the last part of the devic=
e removal after remoing the partitions)

Since the symlink /sys/class/block/sdb1 points to /sys/class/devices/platfo=
rm/hostx/sessionx/targetx:x:x:x/x:x:x:x/block/sdb/sdb1 and since sdb is alr=
eady removed, the symlink /sys/class/block/sdb1 will be orphan and stale. S=
o, this stale link is a result of the race condition in kernel between the =
systemd-udevd and iscsi-logout processing as described above. We are able t=
o reproduce this even with latest upstream kernel.
=09
We have come across a patch from Ming Lei which was created for "avoid to d=
rop & re-add partitions if partitions aren't changed":
https://lore.kernel.org/linux-block/20210216084430.GA23694@lst.de/T/
=09
This patch could resolve our problem of stale link but it just seems to be =
a work-around and not the actual fix for the race. We were looking for help=
 to fix this race in kernel. Do you have any idea how to fix this race cond=
ition?
=09
Following is the script we are using to reproduce the issue:
=09
#!/bin/bash
 =20
dir=3D/sys/class/block
iter_count=3D0
while [ $iter_count -lt 10000000 ]; do
    iscsiadm -m node -T iqn.2016-01.com.example:target1 -p 100.100.242.162:=
3260 -l

    poll_loop=3D0
    while [ ! -e /sys/class/block/sdb1 ]; do
        ls  -i -l /sys/class/block/sd* > /dev/null
        let poll_loop+=3D1
        if [ $poll_loop -gt 1000000 ]; then
            ls  -i -l /sys/class/block/sd* --color
            exit 1
        fi
    done

    ls -i -l /sys/class/block/sd* --color
    mount /dev/sdb1 /mnt
    dd of=3D/mnt/sdb1 if=3D/dev/sdb2 bs=3D1M count=3D100 &
    pid_01=3D$!
    wait $pid_01
    umount -l /mnt &
    pid_02=3D$!
    wait $pid_02

    parted /dev/sdb -s print
    iscsiadm -m node -T iqn.2016-01.com.example:target1 -p 100.100.242.162:=
3260 -u &
    pid_1=3D$!

    iscsiadm -m node -T iqn.2016-01.com.example:target2 -p 100.100.242.162:=
3260 -l &
    pid_2=3D$!

    sleep 1
    ls -i -l /sys/class/block/sd* --color

    for i in `ls  $dir`; do
        if [ ! -e $dir/$i ]; then
            echo "broken link: $dir/$i"
            exit 1
        fi
    done

    parted /dev/sdb -s print
    iscsiadm -m node -T iqn.2016-01.com.example:target2 -p 100.100.242.162:=
3260 -u
    iter_count=3D`expr $iter_count + 1`
done


Regards,
Gulam Mohamed.