Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 18 Mar 2021 19:33:16 +0000
From:   Luis Chamberlain <mcgrof@kernel.org>
To:     linux-block@vger.kernel.org, jejb@linux.ibm.com,
        martin.petersen@oracle.com
Cc:     linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org,
        dgilbert@interlog.com, hare@suse.de, mcgrof@kernel.org
Subject: Re: blktests: block/009 next-20210304 failure rate average of 1/448
Message-ID: <20210318193316.GS13911@42.do-not-panic.com>
References: <20210316174645.GI4332@42.do-not-panic.com>
 <20210316184739.GJ4332@42.do-not-panic.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20210316184739.GJ4332@42.do-not-panic.com>
Precedence: bulk

On Tue, Mar 16, 2021 at 06:47:39PM +0000, Luis Chamberlain wrote:
> On Tue, Mar 16, 2021 at 05:46:45PM +0000, Luis Chamberlain wrote:
> > I've managed to reproduce blktests block/009 failures with kdevops [0]
> > on linux-next tag next-20210304 with a current failure rate average of
> > 1/448 (3 counted failures so far).
> 
> Confirmed on next-20210316 with current failure rate at 1/1008

Just in case this was a scsi_debug issue instead (I am covering that
prospect on another bug just for scsi_debug korg#212337 [0]) I tried
a userspace solution based on what I have observed I still can reproduce
this block/009 failure. The failure rate is much lower though, I have it
now at 1/1705 but alas it is still failing.

[0] https://bugzilla.kernel.org/show_bug.cgi?id=212337

The patch below demonstrates the exra settle work for scsi_debug
attempted, and with it, this is still failing. So either the settle
work needs *more* effort, or this is a real issue.

diff --git a/common/scsi_debug b/common/scsi_debug
index b48cdc9..ecdbcc6 100644
--- a/common/scsi_debug
+++ b/common/scsi_debug
@@ -8,13 +8,42 @@ _have_scsi_debug() {
 	_have_modules scsi_debug
 }
 
+# As per korg#212337 [0] we must do more work in userspace to settle
+# scsi_debug devices a bit more carefully.
+
+# [0] https://bugzilla.kernel.org/show_bug.cgi?id=212337
+_settle_scsi_debug_device() {
+	SCSI_DEBUG_MAX_WAIT=10
+	SCSI_DEBUG_COUNT_WAIT_LOOP=0
+	while true ; do
+		if [[ -b $1 ]]; then
+			SCSI_DEBUG_LSOF_COUNT=$(lsof $1 | wc -l)
+			if [[ $SCSI_DEBUG_LSOF_COUNT -ne 0 ]]; then
+				sleep 1;
+			else
+				break
+			fi
+		else
+			# Let device come up
+			sleep 1
+
+			let SCSI_DEBUG_COUNT_WAIT_LOOP=$SCSI_DEBUG_COUNT_WAIT_LOOP+1
+			if [[ $SCSI_DEBUG_COUNT_WAIT_LOOP -ge $SCSI_DEBUG_MAX_WAIT ]]; then
+				break
+			fi
+		fi
+	done
+}
+
 _init_scsi_debug() {
 	if ! modprobe -r scsi_debug || ! modprobe scsi_debug "$@"; then
 		return 1
 	fi
-
 	udevadm settle
 
+	# Allow dependencies to load
+	sleep 1
+
 	local host_sysfs host target_sysfs target
 	SCSI_DEBUG_HOSTS=()
 	SCSI_DEBUG_TARGETS=()
@@ -43,6 +72,10 @@ _init_scsi_debug() {
 		return 1
 	fi
 
+	for i in $SCSI_DEBUG_DEVICES ; do
+		_settle_scsi_debug_device /dev/$i
+	done
+
 	return 0
 }