Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Thu, 25 Jul 2019 11:39:46 -0500
From:   Dan Rue <dan.rue@linaro.org>
To:     Paolo Bonzini <pbonzini@redhat.com>
Cc:     Sean Christopherson <sean.j.christopherson@intel.com>,
        Naresh Kamboju <naresh.kamboju@linaro.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Anders Roxell <anders.roxell@linaro.org>,
        Ben Hutchings <ben.hutchings@codethink.co.uk>,
        wanpengli@tencent.com,
        Linus Torvalds <torvalds@linux-foundation.org>,
        patches@kernelci.org,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        lkft-triage@lists.linaro.org,
        linux- stable <stable@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Shuah Khan <shuah@kernel.org>,
        Guenter Roeck <linux@roeck-us.net>, jmattson@google.com
Subject: Re: [PATCH 5.2 000/413] 5.2.3-stable review
Message-ID: <20190725163946.xt2p3pvxwuabzojj@xps.therub.org>
Mail-Followup-To: Paolo Bonzini <pbonzini@redhat.com>,
        Sean Christopherson <sean.j.christopherson@intel.com>,
        Naresh Kamboju <naresh.kamboju@linaro.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Anders Roxell <anders.roxell@linaro.org>,
        Ben Hutchings <ben.hutchings@codethink.co.uk>,
        wanpengli@tencent.com,
        Linus Torvalds <torvalds@linux-foundation.org>,
        patches@kernelci.org,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        lkft-triage@lists.linaro.org,
        linux- stable <stable@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Shuah Khan <shuah@kernel.org>, Guenter Roeck <linux@roeck-us.net>,
        jmattson@google.com
References: <20190724191735.096702571@linuxfoundation.org>
 <CADYN=9+WLxhmqX3JNL_s-kWSN97G=8WhD=TF=uAuKecJnKcj_Q@mail.gmail.com>
 <20190725113437.GA27429@kroah.com>
 <230a5b34-d23e-8318-0b1f-d23ada7318e0@redhat.com>
 <CA+G9fYsWdmboyquZ=Bs3tkTwRFTzd1yuL0_EVpHOecNi4E_stA@mail.gmail.com>
 <20190725160939.GC18612@linux.intel.com>
 <33f1cfaa-525d-996a-4977-fda32dc368ee@redhat.com>
 <20190725162053.GD18612@linux.intel.com>
 <7bc207e0-0812-e41a-bfd5-e3fbfd43f242@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <7bc207e0-0812-e41a-bfd5-e3fbfd43f242@redhat.com>
User-Agent: NeoMutt/20180716
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Thu, Jul 25, 2019 at 06:30:10PM +0200, Paolo Bonzini wrote:
> On 25/07/19 18:20, Sean Christopherson wrote:
> > On Thu, Jul 25, 2019 at 06:10:37PM +0200, Paolo Bonzini wrote:
> >> On 25/07/19 18:09, Sean Christopherson wrote:
> >>>> This investigation confirms it is a new test code failure on stable-rc 5.2.3
> >>> No, it only confirms that kvm-unit-tests/master fails on 5.2.*.  To confirm
> >>> a new failure in 5.2.3 you would need to show a test that passes on 5.2.2
> >>> and fails on 5.2.3.
> >>
> >> I think he meant "a failure in new test code". :)
> > 
> > Ah, that does appear to be the case.  So just to be clear, we're good, right?
> 
> Yes.  I'm happy to gather ideas on how to avoid this (i.e. 1) if a
> submodule would be useful; 2) where to stick it).

Hi!

First, to be clear: from LKFT perspective there are no kernel
regressions here.

To your point Paolo - reporting 'fail' because of a missing kernel
feature is a generic problem we see across test suites, and causes tons
of pain and misery for CI people. As a general rule, I'd avoid
submodules, and even branches that track specific kernels. Rather, and I
don't know if it's possible in this case, but the best way to manage it
from both a test author and a test runner POV is to wrap the test in
kernel feature checks, kernel version checks, kernel config checks, etc.
Report 'skip' if the environment in which the test is running isn't
sufficient to run the test. Then, you only have to maintain one version
of the test suite, users can always use the latest, and critically: all
failures are actual failures.

Dan

> 
> Paolo

-- 
Linaro - Kernel Validation