Date: Fri, 16 Jun 2017 21:29:52 +0200
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: "Luis R. Rodriguez" <mcgrof@kernel.org>
Cc: Sumit Semwal <sumit.semwal@linaro.org>,
        Brian Norris <computersforpeace@gmail.com>,
        Kees Cook <keescook@chromium.org>,
        Fengguang Wu <fengguang.wu@intel.com>, shuah@kernel.org,
        LKML <linux-kernel@vger.kernel.org>, stable@vger.kernel.org,
        linux-kselftest@vger.kernel.org, Martin Fuzzey <mfuzzey@parkeon.com>,
        Alan Cox <alan@linux.intel.com>
Subject: Re: LTS testing with latest kselftests - some failures
Message-ID: <20170616192952.GA22037@kroah.com>
References: <CAO_48GFNXyqTDN85y8FKLV3wj19JJ28sWXuX4Vex3LVzaxzGCw@mail.gmail.com>
 <20170616164651.GA21846@wotan.suse.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20170616164651.GA21846@wotan.suse.de>
User-Agent: Mutt/1.8.3 (2017-05-23)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 2126
Lines: 51

On Fri, Jun 16, 2017 at 06:46:51PM +0200, Luis R. Rodriguez wrote:
> Kees, please review 47e0bbb7fa98 below.
> Brian, please review be4a1326d12c below.
> 
> On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:
> > Hello Greg, Shuah,
> > 
> > While testing 4.4.y and 4.9.y LTS kernels with latest kselftest,
> 
> To be clear it seems like you are taking the latest upstream ksefltest and run
> it against older stable kernels. Furthermore you seem to only run the shell
> script tests but are using older kselftests drivers? Is this all correct?
> Otherwise it is unclear how you are running into the issues below.
> 
> Does 0-day so the same? I thought 0-day takes just the kselftest from each tree
> submitted. That *seemed* to me like the way it was designed. Shuah ?
> 
> What's the name of *this* testing effort BTW? Is this part of the overall
> kselftest ? Or is this something Linaro does for LTS kernels ? If there
> is a name to your effort can you document it here so that others are aware:

It's a "test LTS kernels to make sure Greg didn't break anything" type
of testing effort that Linaro is helping out with.

This could also be called, "it's about time someone did this..." :)

> > we found a couple more test failures due to test-kernel mismatch:
> > 
> > 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to
> > tests, and related updates to lib/test_firmware.c to improve the
> > tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable
> > 4.9 misses the second update.
> 
> <-- snip, skipped 2. and 3. -->
> 
> > For all the 3 listed above, we will try and update the tests to gracefully exit.
> 
> Hmm, this actually raises a good kselftest question:
> 
> I *though* kselftests were running tests on par with the kernels, so we would
> *not* take latest upstream kselftests to test against older kernels. Is this
> incorrect?

That is incorrect.  Your test should always degrade gracefully if the
feature is not present in the kernel under test.  If the test is for a
bug that was fixed, then that fix should also go to a stable kernel
release.

thanks,

greg k-h