Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [PATCH 5.2 000/413] 5.2.3-stable review
To:     Sean Christopherson <sean.j.christopherson@intel.com>,
        Naresh Kamboju <naresh.kamboju@linaro.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Anders Roxell <anders.roxell@linaro.org>,
        Ben Hutchings <ben.hutchings@codethink.co.uk>,
        wanpengli@tencent.com,
        Linus Torvalds <torvalds@linux-foundation.org>,
        patches@kernelci.org,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        lkft-triage@lists.linaro.org,
        linux- stable <stable@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Shuah Khan <shuah@kernel.org>,
        Guenter Roeck <linux@roeck-us.net>, jmattson@google.com
References: <20190724191735.096702571@linuxfoundation.org>
 <CADYN=9+WLxhmqX3JNL_s-kWSN97G=8WhD=TF=uAuKecJnKcj_Q@mail.gmail.com>
 <20190725113437.GA27429@kroah.com>
 <230a5b34-d23e-8318-0b1f-d23ada7318e0@redhat.com>
 <CA+G9fYsWdmboyquZ=Bs3tkTwRFTzd1yuL0_EVpHOecNi4E_stA@mail.gmail.com>
 <20190725160939.GC18612@linux.intel.com>
 <33f1cfaa-525d-996a-4977-fda32dc368ee@redhat.com>
 <20190725162053.GD18612@linux.intel.com>
 <7bc207e0-0812-e41a-bfd5-e3fbfd43f242@redhat.com>
 <20190725163946.xt2p3pvxwuabzojj@xps.therub.org>
From:   Paolo Bonzini <pbonzini@redhat.com>
Openpgp: preference=signencrypt
Message-ID: <3e55414d-cb4f-8f3f-a359-e374b6298715@redhat.com>
Date:   Thu, 25 Jul 2019 19:06:19 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.8.0
MIME-Version: 1.0
In-Reply-To: <20190725163946.xt2p3pvxwuabzojj@xps.therub.org>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 25/07/19 18:39, Dan Rue wrote:
> To your point Paolo - reporting 'fail' because of a missing kernel
> feature is a generic problem we see across test suites, and causes tons
> of pain and misery for CI people. As a general rule, I'd avoid
> submodules, and even branches that track specific kernels. Rather, and I
> don't know if it's possible in this case, but the best way to manage it
> from both a test author and a test runner POV is to wrap the test in
> kernel feature checks, kernel version checks, kernel config checks, etc.
> Report 'skip' if the environment in which the test is running isn't
> sufficient to run the test. Then, you only have to maintain one version
> of the test suite, users can always use the latest, and critically: all
> failures are actual failures.

Note that kvm-unit-tests are not really testing new kernel features;
those are covered by tools/testing/selftests/kvm.  For some of these
kvm-unit-tests there are some CPU features that we can check from the
virtual machine, but those are easy to handle and they produce SKIP
results just fine.

The problematic ones are tests that cover emulation accuracy.  These are
effectively bugfixes, so the failures you see _are_ actual failures.  At
the same time, the bugs are usually inoffensive(*), while the fixes are
invasive and a bad source of cause conflicts in older Linux versions.
This combines so that backporting to stable is not feasible.

Passing the host kernel version would be really ugly because 1) the
tests can run on other hypervisor or emulators or even bare metal, and
of course the host kernel version has no bearing if you're using
userspace emulation 2) there are thousands of tests that would be
littered with kernel version checks of little significance.

So this is why I suggested a submodule: using a submodule effectively
ignores all tests that were added after a given Linus release, and thus
all the failures for which backports are just not going to happen.
However, if Sean's idea of creating a linux-M.N branch in
kvm-unit-tests.git works for you, we can also do that as a stopgap
measure to ease your testing.

Thanks,

Paolo

(*) if they aren't, we *do* mark them for backport!