Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Thu, 25 Jul 2019 15:19:33 -0500
From:   Dan Rue <dan.rue@linaro.org>
To:     Paolo Bonzini <pbonzini@redhat.com>
Cc:     Sean Christopherson <sean.j.christopherson@intel.com>,
        Naresh Kamboju <naresh.kamboju@linaro.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Anders Roxell <anders.roxell@linaro.org>,
        Ben Hutchings <ben.hutchings@codethink.co.uk>,
        wanpengli@tencent.com,
        Linus Torvalds <torvalds@linux-foundation.org>,
        patches@kernelci.org,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        lkft-triage@lists.linaro.org,
        linux- stable <stable@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Shuah Khan <shuah@kernel.org>,
        Guenter Roeck <linux@roeck-us.net>, jmattson@google.com
Subject: Re: [PATCH 5.2 000/413] 5.2.3-stable review
Message-ID: <20190725201933.aiqh6oj7bacdwact@xps.therub.org>
Mail-Followup-To: Paolo Bonzini <pbonzini@redhat.com>,
        Sean Christopherson <sean.j.christopherson@intel.com>,
        Naresh Kamboju <naresh.kamboju@linaro.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Anders Roxell <anders.roxell@linaro.org>,
        Ben Hutchings <ben.hutchings@codethink.co.uk>,
        wanpengli@tencent.com,
        Linus Torvalds <torvalds@linux-foundation.org>,
        patches@kernelci.org,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        lkft-triage@lists.linaro.org,
        linux- stable <stable@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Shuah Khan <shuah@kernel.org>, Guenter Roeck <linux@roeck-us.net>,
        jmattson@google.com
References: <CADYN=9+WLxhmqX3JNL_s-kWSN97G=8WhD=TF=uAuKecJnKcj_Q@mail.gmail.com>
 <20190725113437.GA27429@kroah.com>
 <230a5b34-d23e-8318-0b1f-d23ada7318e0@redhat.com>
 <CA+G9fYsWdmboyquZ=Bs3tkTwRFTzd1yuL0_EVpHOecNi4E_stA@mail.gmail.com>
 <20190725160939.GC18612@linux.intel.com>
 <33f1cfaa-525d-996a-4977-fda32dc368ee@redhat.com>
 <20190725162053.GD18612@linux.intel.com>
 <7bc207e0-0812-e41a-bfd5-e3fbfd43f242@redhat.com>
 <20190725163946.xt2p3pvxwuabzojj@xps.therub.org>
 <3e55414d-cb4f-8f3f-a359-e374b6298715@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <3e55414d-cb4f-8f3f-a359-e374b6298715@redhat.com>
User-Agent: NeoMutt/20180716
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Thu, Jul 25, 2019 at 07:06:19PM +0200, Paolo Bonzini wrote:
> On 25/07/19 18:39, Dan Rue wrote:
> > To your point Paolo - reporting 'fail' because of a missing kernel
> > feature is a generic problem we see across test suites, and causes tons
> > of pain and misery for CI people. As a general rule, I'd avoid
> > submodules, and even branches that track specific kernels. Rather, and I
> > don't know if it's possible in this case, but the best way to manage it
> > from both a test author and a test runner POV is to wrap the test in
> > kernel feature checks, kernel version checks, kernel config checks, etc.
> > Report 'skip' if the environment in which the test is running isn't
> > sufficient to run the test. Then, you only have to maintain one version
> > of the test suite, users can always use the latest, and critically: all
> > failures are actual failures.
> 
> Note that kvm-unit-tests are not really testing new kernel features;
> those are covered by tools/testing/selftests/kvm.  For some of these
> kvm-unit-tests there are some CPU features that we can check from the
> virtual machine, but those are easy to handle and they produce SKIP
> results just fine.
> 
> The problematic ones are tests that cover emulation accuracy.  These are
> effectively bugfixes, so the failures you see _are_ actual failures.  At
> the same time, the bugs are usually inoffensive(*), while the fixes are
> invasive and a bad source of cause conflicts in older Linux versions.
> This combines so that backporting to stable is not feasible.

In this case, a fail result seems correct then. The thing we're doing
that we need to fix is to run against a pinned version of kvm-unit-tests
and upgrade it independently so that we can identify such failures and
mark them as known issues.

> 
> Passing the host kernel version would be really ugly because 1) the
> tests can run on other hypervisor or emulators or even bare metal, and
> of course the host kernel version has no bearing if you're using
> userspace emulation 2) there are thousands of tests that would be
> littered with kernel version checks of little significance.
> 
> So this is why I suggested a submodule: using a submodule effectively
> ignores all tests that were added after a given Linus release, and thus
> all the failures for which backports are just not going to happen.
> However, if Sean's idea of creating a linux-M.N branch in
> kvm-unit-tests.git works for you, we can also do that as a stopgap
> measure to ease your testing.

I would still prefer to run the latest tests against all kernel versions
(but better control when we upgrade it). Like I said, we can handle
expected failures, and it would even help to validate backports for
fixes that do get backported. I'm afraid on your behalf that snapping
(and maintaining) branches per kernel branch is going to be a lot to
manage.

In any case, _thank you so much_ for jumping on this and helping us run
these tests. Is there anything else we can do to make this better for
you?

Dan

> 
> Thanks,
> 
> Paolo
> 
> (*) if they aren't, we *do* mark them for backport!

-- 
Linaro - Kernel Validation