Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Message-ID: <1550198902.2802.12.camel@HansenPartnership.com>
Subject: Re: [LSF/MM TOPIC] FS, MM, and stable trees
From:   James Bottomley <James.Bottomley@HansenPartnership.com>
To:     Sasha Levin <sashal@kernel.org>
Cc:     Greg KH <gregkh@linuxfoundation.org>,
        Amir Goldstein <amir73il@gmail.com>,
        Steve French <smfrench@gmail.com>,
        lsf-pc@lists.linux-foundation.org,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        linux-mm <linux-mm@kvack.org>,
        LKML <linux-kernel@vger.kernel.org>,
        "Luis R. Rodriguez" <mcgrof@kernel.org>
Date:   Thu, 14 Feb 2019 18:48:22 -0800
In-Reply-To: <20190215015020.GJ69686@sasha-vm>
References: <20190212170012.GF69686@sasha-vm>
         <CAH2r5mviqHxaXg5mtVe30s2OTiPW2ZYa9+wPajjzz3VOarAUfw@mail.gmail.com>
         <CAOQ4uxjMYWJPF8wFF_7J7yy7KCdGd8mZChfQc5GzNDcfqA7UAA@mail.gmail.com>
         <20190213073707.GA2875@kroah.com>
         <CAOQ4uxgQGCSbhppBfhHQmDDXS3TGmgB4m=Vp3nyyWTFiyv6z6g@mail.gmail.com>
         <20190213091803.GA2308@kroah.com> <20190213192512.GH69686@sasha-vm>
         <20190213195232.GA10047@kroah.com>
         <1550088875.2871.21.camel@HansenPartnership.com>
         <20190215015020.GJ69686@sasha-vm>
Content-Type: text/plain; charset="UTF-8"
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Thu, 2019-02-14 at 20:50 -0500, Sasha Levin wrote:
> On Wed, Feb 13, 2019 at 12:14:35PM -0800, James Bottomley wrote:
> > On Wed, 2019-02-13 at 20:52 +0100, Greg KH wrote:
> > > On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote:
> > > > On Wed, Feb 13, 2019 at 10:18:03AM +0100, Greg KH wrote:
> > > > > On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein
> > > > > wrote:
> > > > > > Best effort testing in timely manner is good, but a good
> > > > > > way to improve confidence in stable kernel releases is a
> > > > > > publicly available list of tests that the release went
> > > > > > through.
> > > > > 
> > > > > We have that, you aren't noticing them...
> > > > 
> > > > This is one of the biggest things I want to address: there is a
> > > > disconnect between the stable kernel testing story and the
> > > > tests the fs/ and mm/ folks expect to see here.
> > > > 
> > > > On one had, the stable kernel folks see these kernels go
> > > > through entire suites of testing by multiple individuals and
> > > > organizations, receiving way more coverage than any of Linus's
> > > > releases.
> > > > 
> > > > On the other hand, things like LTP and selftests tend to barely
> > > > scratch the surface of our mm/ and fs/ code, and the
> > > > maintainers of these subsystems do not see LTP-like suites as
> > > > something that adds significant value and ignore them. Instead,
> > > > they have a (convoluted) set of testing they do with different
> > > > tools and configurations that qualifies their code as being
> > > > "tested".
> > > > 
> > > > So really, it sounds like a low hanging fruit: we don't really
> > > > need to write much more testing code code nor do we have to
> > > > refactor existing test suites. We just need to make sure the
> > > > right tests are running on stable kernels. I really want to
> > > > clarify what each subsystem sees as "sufficient" (and have that
> > > > documented somewhere).
> > > 
> > > kernel.ci and 0-day and Linaro are starting to add the fs and mm
> > > tests to their test suites to address these issues (I think 0-day
> > > already has many of them).  So this is happening, but not quite
> > > obvious.  I know I keep asking Linaro about this :(
> > 
> > 0day has xfstests at least, but it's opt-in only (you have to
> > request that it be run on your trees).  When I did it for the SCSI
> > tree, I had to email Fenguangg directly, there wasn't any other way
> > of getting it.
> 
> It's very tricky to do even if someone would just run it.

It is?  It's a test suite, so you just run it and it exercises standard
and growing set of regression tests.

>  I worked with the xfs folks for quite a while to gather the various
> configs they want to use, and to establish the baseline for a few of
> the stable trees (some tests are know to fail, etc).

The only real config issue is per-fs non-standard tests (features
specific to a given filesystem).  I just want it to exercise the
storage underneath, so the SCSI tree is configured for the default set
on xfs.

> So just running xfstests "blindly" doesn't add much value beyond ltp
> I think.

Well, we differ on the value of running regression tests, then.  The
whole point of a test infrastructure is that it's simple to run 'make
check' in autoconf parlance.  xfstests does provide a useful baseline
set of regression tests.  However, since my goal is primarily to detect
problems in the storage path rather than the filesystem, the utility is
exercising that path, although I fully appreciate that filesystem
regression tests aren't going to catch every SCSI issue, they do
provide some level of assurance against bugs.

Hopefully we can switch over to blktests when it's ready, but in the
meantime xfstests is way better than nothing.

James