Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Received-SPF: Pass (protection.outlook.com: domain of mit.edu designates
 18.9.28.11 as permitted sender) receiver=protection.outlook.com;
 client-ip=18.9.28.11; helo=outgoing.mit.edu;
Date:   Thu, 21 Feb 2019 13:52:38 -0500
From:   "Theodore Y. Ts'o" <tytso@mit.edu>
To:     Luis Chamberlain <mcgrof@kernel.org>
CC:     James Bottomley <James.Bottomley@HansenPartnership.com>,
        Sasha Levin <sashal@kernel.org>,
        Greg KH <gregkh@linuxfoundation.org>,
        Amir Goldstein <amir73il@gmail.com>,
        Steve French <smfrench@gmail.com>,
        <lsf-pc@lists.linux-foundation.org>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        linux-mm <linux-mm@kvack.org>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: [LSF/MM TOPIC] FS, MM, and stable trees
Message-ID: <20190221185238.GF10245@mit.edu>
Mail-Followup-To: "Theodore Y. Ts'o" <tytso@mit.edu>,
        Luis Chamberlain <mcgrof@kernel.org>,
        James Bottomley <James.Bottomley@HansenPartnership.com>,
        Sasha Levin <sashal@kernel.org>,
        Greg KH <gregkh@linuxfoundation.org>,
        Amir Goldstein <amir73il@gmail.com>,
        Steve French <smfrench@gmail.com>,
        lsf-pc@lists.linux-foundation.org,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        linux-mm <linux-mm@kvack.org>, LKML <linux-kernel@vger.kernel.org>
References: <20190213073707.GA2875@kroah.com>
 <CAOQ4uxgQGCSbhppBfhHQmDDXS3TGmgB4m=Vp3nyyWTFiyv6z6g@mail.gmail.com>
 <20190213091803.GA2308@kroah.com>
 <20190213192512.GH69686@sasha-vm>
 <20190213195232.GA10047@kroah.com>
 <1550088875.2871.21.camel@HansenPartnership.com>
 <20190215015020.GJ69686@sasha-vm>
 <1550198902.2802.12.camel@HansenPartnership.com>
 <20190216182835.GF23000@mit.edu>
 <20190221153415.GL11489@garbanzo.do-not-panic.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <20190221153415.GL11489@garbanzo.do-not-panic.com>
User-Agent: Mutt/1.10.1 (2018-07-13)
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Feb 2019 18:52:42.0686
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 377d23b0-da3a-453a-a18e-08d6982dc2c8
X-MS-Exchange-CrossTenant-Id: 64afd9ba-0ecf-4acf-bc36-935f6235ba8b
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=64afd9ba-0ecf-4acf-bc36-935f6235ba8b;Ip=[18.9.28.11];Helo=[outgoing.mit.edu]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR01MB3281
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Feb 21, 2019 at 07:34:15AM -0800, Luis Chamberlain wrote:
> On Sat, Feb 16, 2019 at 01:28:35PM -0500, Theodore Y. Ts'o wrote:
> > The block/*, loop/* and scsi/* tests in blktests do seem to be in
> > pretty good shape.  The nvme, nvmeof, and srp tests are *definitely*
> > not as mature.
>=20
> Can you say more about this later part. What would you like to see more
> of for nvme tests for instance?
>=20
> It sounds like a productive session would include tracking our:
>=20
>   a) sour spots
>   b) who's already working on these
>   c) gather volutneers for these sour spots

I complained on another LSF/MM topic thread, but there are a lot of
failures where it's not clear whether it's because I guessed
incorrectly about which version of nvme-cli I should be using (debian
stable and head of nvme-cli both are apparently wrong answers), or
kernel bugs or kernel misconfiguration issues on my side.

Current nvme/* failures that I'm still seeing are attached below.

	       		     	       - Ted

nvme/012 (run mkfs and data verification fio job on NVMeOF block device-bac=
ked ns) [failed]
    runtime  ...  100.265s
    something found in dmesg:
    [ 1857.188083] run blktests nvme/012 at 2019-02-12 01:11:33
    [ 1857.437322] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
    [ 1857.456187] nvmet: creating controller 1 for subsystem blktests-subs=
ystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:78dc695a-2c99-4841-968d-c2c=
16a49a02a.
    [ 1857.458162] nvme nvme0: ANA group 1: optimized.
    [ 1857.458257] nvme nvme0: creating 2 I/O queues.
    [ 1857.460893] nvme nvme0: new ctrl: "blktests-subsystem-1"
   =20
    [ 1857.720666] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D
    [ 1857.726308] WARNING: possible recursive locking detected
    [ 1857.731784] 5.0.0-rc3-xfstests-00014-g1236f7d60242 #843 Not tainted
    ...
    (See '/results/nodev/nvme/012.dmesg' for the entire message)
nvme/013 (run mkfs and data verification fio job on NVMeOF file-backed ns) =
[failed]
    runtime  ...  32.634s
    --- tests/nvme/013.out	2019-02-11 18:57:39.000000000 -0500
    +++ /results/nodev/nvme/013.out.bad	2019-02-12 01:13:46.708757206 -0500
    @@ -1,5 +1,9 @@
     Running nvme/013
     91fdba0d-f87b-4c25-b80f-db7be1418b9e
     uuid.91fdba0d-f87b-4c25-b80f-db7be1418b9e
    +fio: io_u error on file /mnt/blktests///verify.0.0: Input/output error=
: write offset=3D329326592, buflen=3D4096
    +fio: io_u error on file /mnt/blktests///verify.0.0: Input/output error=
: write offset=3D467435520, buflen=3D4096
    +fio exited with status 0
    +4;fio-3.2;verify;0;5;0;0;0;0;0;0;0.000000;0.000000;0;0;0.000000;0.0000=
00;1.000000%=3D0;5.000000%=3D0;10.000000%=3D0;20.000000%=3D0;30.000000%=3D0=
;40.000000%=3D0;50.000000%=3D0;60.000000%=3D0;70.000000%=3D0;80.000000%=3D0=
;90.000000%=3D0;95.000000%=3D0;99.000000%=3D0;99.500000%=3D0;99.900000%=3D0=
;99.950000%=3D0;99.990000%=3D0;0%=3D0;0%=3D0;0%=3D0;0;0;0.000000;0.000000;0=
;0;0.000000%;0.000000;0.000000;192672;6182;1546;31166;4;9044;63.332763;57.9=
79218;482;29948;10268.332290;1421.459893;1.000000%=3D4145;5.000000%=3D9109;=
10.000000%=3D9502;20.000000%=3D9764;30.000000%=3D10027;40.000000%=3D10289;5=
0.000000%=3D10420;60.000000%=3D10551;70.000000%=3D10682;80.000000%=3D10682;=
90.000000%=3D10944;95.000000%=3D11206;99.000000%=3D13172;99.500000%=3D16318=
;99.900000%=3D24510;99.950000%=3D27394;99.990000%=3D29229;0%=3D0;0%=3D0;0%=
=3D0;507;30005;10331.973087;1421.131712;6040;8232;100.000000%;6189.741935;2=
67.091495;0;0;0;0;0;0;0.000000;0.000000;0;0;0.000000;0.000000;1.000000%=3D0=
;5.000000%=3D0;10.000000%=3D0;20.000000%=3D0;30.000000%=3D0;40.000000%=3D0;=
50.000000%=3D0;60.000000%=3D0;70.000000%=3D0;80.000000%=3D0;90.000000%=3D0;=
95.000000%=3D0;99.000000%=3D0;99.500000%=3D0;99.900000%=3D0;99.950000%=3D0;=
99.990000%=3D0;0%=3D0;0%=3D0;0%=3D0;0;0;0.000000;0.000000;0;0;0.000000%;0.0=
00000;0.000000;3.991657%;6.484839%;91142;0;1024;0.1%;0.1%;0.1%;0.1%;100.0%;=
0.0%;0.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.20%;0.34%;0.25%=
;0.19%;27.86%;70.89%;0.23%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;nvme0n=
1;0;0;0;0;0;0;0;0.00%
    ...
    (Run 'diff -u tests/nvme/013.out /results/nodev/nvme/013.out.bad' to se=
e the entire diff)
nvme/015 (unit test for NVMe flush for file backed ns)       [failed]
    runtime  ...  8.914s
    --- tests/nvme/015.out	2019-02-11 18:57:39.000000000 -0500
    +++ /results/nodev/nvme/015.out.bad	2019-02-12 01:14:05.429328259 -0500
    @@ -1,6 +1,6 @@
     Running nvme/015
     91fdba0d-f87b-4c25-b80f-db7be1418b9e
     uuid.91fdba0d-f87b-4c25-b80f-db7be1418b9e
    -NVMe Flush: success
    +NVME IO command error:INTERNAL: The command was not completed successf=
ully due to an internal error(6006)
     NQN:blktests-subsystem-1 disconnected 1 controller(s)
     Test complete
nvme/016 (create/delete many NVMeOF block device-backed ns and test discove=
ry)
    runtime  ...
nvme/016 (create/delete many NVMeOF block device-backed ns and test discove=
ry) [failed]
    runtime  ...  23.576s
    --- tests/nvme/016.out	2019-02-11 18:57:39.000000000 -0500
    +++ /results/nodev/nvme/016.out.bad	2019-02-12 01:14:29.173378854 -0500
    @@ -1,11 +1,11 @@
     Running nvme/016
    =20
    -Discovery Log Number of Records 1, Generation counter 1
    +Discovery Log Number of Records 1, Generation counter 5
     =3D=3D=3D=3D=3DDiscovery Log Entry 0=3D=3D=3D=3D=3D=3D
     trtype:  loop
     adrfam:  pci
    ...
    (Run 'diff -u tests/nvme/016.out /results/nodev/nvme/016.out.bad' to se=
e the entire diff)
nvme/017 (create/delete many file-ns and test discovery)   =20
    runtime  ...
nvme/017 (create/delete many file-ns and test discovery)     [failed]
    runtime  ...  23.592s
    --- tests/nvme/017.out	2019-02-11 18:57:39.000000000 -0500
    +++ /results/nodev/nvme/017.out.bad	2019-02-12 01:14:52.880762691 -0500
    @@ -1,11 +1,11 @@
     Running nvme/017
    =20
    -Discovery Log Number of Records 1, Generation counter 1
    +Discovery Log Number of Records 1, Generation counter 2
     =3D=3D=3D=3D=3DDiscovery Log Entry 0=3D=3D=3D=3D=3D=3D
     trtype:  loop
     adrfam:  pci
    ...
    (Run 'diff -u tests/nvme/017.out /results/nodev/nvme/017.out.bad' to se=
e the entire diff)