Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\))
Subject: Re: error=Invalid slot
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <76d240e019d0ccc35bce05c1edb1ca104d7c18bf.camel@hammerspace.com>
Date:   Mon, 15 Apr 2019 12:11:11 -0400
Cc:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <32DC3CC2-700B-4F44-849C-F229BAA2CB57@oracle.com>
References: <2996CEEE-0A1A-4C15-AA8B-A97E917D7924@oracle.com>
 <76d240e019d0ccc35bce05c1edb1ca104d7c18bf.camel@hammerspace.com>
To:     Trond Myklebust <trondmy@hammerspace.com>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk


> On Apr 15, 2019, at 12:05 PM, Trond Myklebust =
<trondmy@hammerspace.com> wrote:
>=20
> Hi Chuck,
>=20
>=20
> On Mon, 2019-04-15 at 11:04 -0400, Chuck Lever wrote:
>> Just happened again. Any thoughts about where I should start looking?
>>=20
>> Mon Apr 15 11:01:40 EDT 2019
>> 4k100test: (g=3D0): rw=3Drandread, bs=3D(R) 4096B-4096B, (W) =
4096B-4096B,
>> (T) 4096B-4096B, ioengine=3Dlibaio, iodepth=3D1024
>> ...
>> fio-3.1
>> Starting 12 processes
>> 4k100test: Laying out IO file (1 file / 1024MiB)
>> fio: native_fallocate call failed: Operation not supported
>> 4k100test: Laying out IO file (1 file / 1024MiB)
>> fio: native_fallocate call failed: Operation not supported
>> 4k100test: Laying out IO file (1 file / 1024MiB)
>> fio: native_fallocate call failed: Operation not supported
>> 4k100test: Laying out IO file (1 file / 1024MiB)
>> fio: native_fallocate call failed: Operation not supported
>> 4k100test: Laying out IO file (1 file / 1024MiB)
>> fio: native_fallocate call failed: Operation not supported
>> 4k100test: Laying out IO file (1 file / 1024MiB)
>> fio: native_fallocate call failed: Operation not supported
>> 4k100test: Laying out IO file (1 file / 1024MiB)
>> fio: native_fallocate call failed: Operation not supported
>> 4k100test: Laying out IO file (1 file / 1024MiB)
>> fio: native_fallocate call failed: Operation not supported
>> 4k100test: Laying out IO file (1 file / 1024MiB)
>> fio: native_fallocate call failed: Operation not supported
>> 4k100test: Laying out IO file (1 file / 1024MiB)
>> fio: native_fallocate call failed: Operation not supported
>> 4k100test: Laying out IO file (1 file / 1024MiB)
>> fio: native_fallocate call failed: Operation not supported
>> 4k100test: Laying out IO file (1 file / 1024MiB)
>> fio: native_fallocate call failed: Operation not supported
>> fio: io_u error on file 4k100test.7.0: Invalid slot: read
>> offset=3D938229760, buflen=3D4096
>=20
> Does the following patch fix the race?
>=20
> 8<--------------------------------------
> =46rom 4c8759eafad9bb7ea2626a53296e30618aeefcc7 Mon Sep 17 00:00:00 =
2001
> From: Trond Myklebust <trond.myklebust@hammerspace.com>
> Date: Mon, 15 Apr 2019 11:54:13 -0400
> Subject: [PATCH] SUNRPC: Ignore queue transmission errors on =
successful
> transmission
>=20
> If a request transmission fails due to write space or slot =
unavailability
> errors, but the queued task then gets transmitted before it has time =
to
> process the error in call_transmit_status() or =
call_bc_transmit_status(),
> we need to suppress the transmission error code to prevent it from =
leaking
> out of the RPC layer.
>=20
> Reported-by: Chuck Lever <chuck.lever@oracle.com>
> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> ---
> net/sunrpc/clnt.c | 7 +++++--
> 1 file changed, 5 insertions(+), 2 deletions(-)
>=20
> diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
> index fa900bb44cd5..369a2648dafc 100644
> --- a/net/sunrpc/clnt.c
> +++ b/net/sunrpc/clnt.c
> @@ -2101,8 +2101,8 @@ call_transmit_status(struct rpc_task *task)
> 	 * test first.
> 	 */
> 	if (rpc_task_transmitted(task)) {
> -		if (task->tk_status =3D=3D 0)
> -			xprt_request_wait_receive(task);
> +		task->tk_status =3D 0;
> +		xprt_request_wait_receive(task);
> 		return;
> 	}
>=20
> @@ -2187,6 +2187,9 @@ call_bc_transmit_status(struct rpc_task *task)
> {
> 	struct rpc_rqst *req =3D task->tk_rqstp;
>=20
> +	if (rpc_task_transmitted(task))
> +		task->tk_status =3D 0;
> +
> 	dprint_status(task);
>=20
> 	switch (task->tk_status) {

I was about to try something like this.

I don't have a 100% reproducer. I will apply your patch and
wait for the problem to appear over the next few days.


--
Chuck Lever