Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Tom Gall <tom.gall@linaro.org>
Content-Type: text/plain;
        charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\))
Subject: Towards 4.14 LTS
Message-Id: <5E1947BC-9D22-42EB-BA29-96EA15A9C817@linaro.org>
Date:   Thu, 16 Nov 2017 22:50:23 -0600
Cc:     shuahkh@osg.samsung.com, Guenter Roeck <linux@roeck-us.net>,
        ltp@lists.linux.it, linux-kselftest@vger.kernel.org
To:     linux-kernel@vger.kernel.org,
        linux- stable <stable@vger.kernel.org>,
        torvalds@linux-foundation.org,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

At Linaro we=E2=80=99ve been putting effort into regularly running =
kernel tests over=20
arm, arm64 and x86_64 targets. On those targets we=E2=80=99re running =
mainline, -next,=20
4.4, and 4.9 kernels and yes we are adding to this list as the hardware=20=

capacity grows.

For test buckets we=E2=80=99re using just LTP, kselftest and =
libhugetlbfs and
like kernels we will add to this list.=20

With the 4.14 cycle being a little =E2=80=98different=E2=80=99 in so =
much as the goal to=20
have it be an LTS kernel I think it=E2=80=99s important to take a look =
at some=20
4.14 test results.=20

Grab a beverage, this is a bit of a long post. But quick summery 4.14 as=20=

released looks just as good as 4.13, for the test buckets I named above.

I=E2=80=99ve enclosed our short form report. We break down the =
boards/arch combos for
each bucket pass/skip or potentially fails. Pretty straight forward. =
Skips
generally happen for a few reasons
1) crappy test cases
2) test isn=E2=80=99t appropriate (x86 specific tests so don=E2=80=99t =
run elsewhere)

With this, we have a decent baseline for 4.14 and other kernels going
forward.=20

Summary
------------------------------------------------------------------------

kernel: 4.14.0
git repo: =
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git branch: master
git commit: bebc6082da0a9f5d47a1ea2edc099bf671058bd4
git describe: v4.14
Test details: =
https://qa-reports.linaro.org/lkft/linux-mainline-oe/build/v4.14


No regressions (compared to build v4.14-rc8)

Boards, architectures and test suites:
-------------------------------------

hi6220-hikey - arm64
* boot - pass: 20
* kselftest - skip: 16, pass: 38
* libhugetlbfs - skip: 1, pass: 90
* ltp-cap_bounds-tests - pass: 2
* ltp-containers-tests - pass: 76
* ltp-fcntl-locktests-tests - pass: 2
* ltp-filecaps-tests - pass: 2
* ltp-fs-tests - pass: 60
* ltp-fs_bind-tests - pass: 2
* ltp-fs_perms_simple-tests - pass: 19
* ltp-fsx-tests - pass: 2
* ltp-hugetlb-tests - skip: 1, pass: 21
* ltp-io-tests - pass: 3
* ltp-ipc-tests - pass: 9
* ltp-math-tests - pass: 11
* ltp-nptl-tests - pass: 2
* ltp-pty-tests - pass: 4
* ltp-sched-tests - pass: 14
* ltp-securebits-tests - pass: 4
* ltp-syscalls-tests - skip: 122, pass: 983
* ltp-timers-tests - pass: 12

juno-r2 - arm64
* boot - pass: 20
* kselftest - skip: 15, pass: 38
* libhugetlbfs - skip: 1, pass: 90
* ltp-cap_bounds-tests - pass: 2
* ltp-containers-tests - pass: 76
* ltp-fcntl-locktests-tests - pass: 2
* ltp-filecaps-tests - pass: 2
* ltp-fs-tests - pass: 60
* ltp-fs_bind-tests - pass: 2
* ltp-fs_perms_simple-tests - pass: 19
* ltp-fsx-tests - pass: 2
* ltp-hugetlb-tests - pass: 22
* ltp-io-tests - pass: 3
* ltp-ipc-tests - pass: 9
* ltp-math-tests - pass: 11
* ltp-nptl-tests - pass: 2
* ltp-pty-tests - pass: 4
* ltp-sched-tests - pass: 10
* ltp-securebits-tests - pass: 4
* ltp-syscalls-tests - skip: 156, pass: 943
* ltp-timers-tests - pass: 12

x15 - arm
* boot - pass: 20
* kselftest - skip: 17, pass: 36
* libhugetlbfs - skip: 1, pass: 87
* ltp-cap_bounds-tests - pass: 2
* ltp-containers-tests - pass: 64
* ltp-fcntl-locktests-tests - pass: 2
* ltp-filecaps-tests - pass: 2
* ltp-fs-tests - pass: 60
* ltp-fs_bind-tests - pass: 2
* ltp-fs_perms_simple-tests - pass: 19
* ltp-fsx-tests - pass: 2
* ltp-hugetlb-tests - skip: 2, pass: 20
* ltp-io-tests - pass: 3
* ltp-ipc-tests - pass: 9
* ltp-math-tests - pass: 11
* ltp-nptl-tests - pass: 2
* ltp-pty-tests - pass: 4
* ltp-sched-tests - skip: 1, pass: 13
* ltp-securebits-tests - pass: 4
* ltp-syscalls-tests - skip: 66, pass: 1040
* ltp-timers-tests - pass: 12

dell-poweredge-r200 - x86_64
* boot - pass: 19
* kselftest - skip: 11, pass: 54
* libhugetlbfs - skip: 1, pass: 76
* ltp-cap_bounds-tests - pass: 1
* ltp-containers-tests - pass: 64
* ltp-fcntl-locktests-tests - pass: 2
* ltp-filecaps-tests - pass: 2
* ltp-fs-tests - skip: 1, pass: 61
* ltp-fs_bind-tests - pass: 1
* ltp-fs_perms_simple-tests - pass: 19
* ltp-fsx-tests - pass: 2
* ltp-hugetlb-tests - pass: 22
* ltp-io-tests - pass: 3
* ltp-ipc-tests - pass: 8
* ltp-math-tests - pass: 11
* ltp-nptl-tests - pass: 2
* ltp-pty-tests - pass: 4
* ltp-sched-tests - pass: 9
* ltp-securebits-tests - pass: 3
* ltp-syscalls-tests - skip: 163, pass: 962

Lots of green.


Let=E2=80=99s now talk about coverage, the pandora=E2=80=99s box of =
validation. It=E2=80=99s never
perfect. There=E2=80=99s a bazillion different build combos. Even tools =
can
make a difference. We=E2=80=99ve seen a case where the dhcp client from =
open embedded=20
didn=E2=80=99t trigger a network regression in one of the LTS RCs but =
Debian=E2=80=99s dhclient
did.

Of no surprise between what we and others have, it=E2=80=99s not perfect =
coverage,
and there are only so many build, boot and run cycles to execute the =
test=20
buckets with various combinations so we need to stay sensible as far as=20=

kernel configs go.=20

Does this kind of system actually FIND anything and is it useful for=20
watching for 4.14 regressions as fixes are introduced?

I would assert the answer is yes. We do have data for a couple of kernel
cycles but it=E2=80=99s also somewhat dirty as we have been in the =
process of=20
detecting and tossing out dodgy test cases.=20

Take 4.14-RC7, there was one failure that is no longer there.
ltp-syscalls-tests : perf_event_open02 (arm64)

As things are getting merged post 4.14 there are some failures
cropping up. Here=E2=80=99s an example:
=
https://qa-reports.linaro.org/lkft/linux-mainline-oe/tests/ltp-fs-tests/pr=
oc01

Note the Build column, the kernels are identified by their git describe.=20=

Don=E2=80=99t be alarmed if you see n/a in some columns, the queues are =
catching up
so data will be filling in.


So why didn=E2=80=99t we report these? As mentioned we=E2=80=99ve been =
tossing out dodgy
test cases to get to a clean baseline. We don=E2=80=99t need or want =
noise.=20

For LTS, I want the system when it detects a failure to enable a quick=20=

bisect involving the affected test bucket. Given the nature of kernel=20
bugs tho, there is that class of bug which only happens occasionally.

This brings up a conundrum when you have a system like this. A failure
turns up, it=E2=80=99s not consistently failing and a path forward =
isn=E2=80=99t=20
necessarily obvious. Remember for an LTS RC, there=E2=80=99s a defined =
window=20
to comment.

I=E2=80=99ve been flamed for reporting a LTS RC test failure which =
didn't include=20
a fix, just a =E2=80=98this fails, and we=E2=80=99re looking at it.=E2=80=99=
 I=E2=80=99ve been flamed=20
for not reporting a failure that had been detected but not raised to the=20=

list since it was still being debugged after the RC comment window had
closed.

My 1990s vintage asbestos underwear thankfully is functional.

There is probably a case to be made either way. It boils down to
either: =20

Red Pill) Be fully open reporting early and often
Blue Pill) Be closed and only pass up failures that include a patch to =
fix a bug.

Red Pill does expose drama yet it also creates an opportunity for others =
to
get involved.

Blue Pill protects the community from noise and the creation of =
frustration
that the system has cried wolf for perhaps a stupid test case.=20

Likewise from a maintainer or dev perspective, there=E2=80=99s a sea of =
data.=20
Time is precious, and who wants to waste it on some snipe hunt?

I=E2=80=99m personally in the Red Pill camp. I like being open.

Be it 0day, LKFT or whatever I think the responsibility is on us
running these projects to be open and give full guidance. Yes there=20
will be noise. Noise can suggest dodgy test cases or bugs that are
hard to trigger. Either way they warrant a look. Take Arnd Bergman=E2=80=99=
s=20
work to get rid of kernel warnings. Same concept in my opinion.

Dodgy test cases can easily be put onto skip lists. As we=E2=80=99ve =
been
running for a number of months now, data and ol fashioned code=20
review has been our guide to banish dodgy test cases to skip lists.
Going forward new test cases will pop up. Some of them will be dodgy.=20

There=E2=80=99s lots of room for collaboration in improving test cases.=20=


In summary I think for mainline, LTS kernels etc, we have a good=20
warning system to detect regressions as patches flow in. It will evolve=20=

and improve as is the nature of our open community. =46rom kernelci,=20
LKFT, 0day, etc, that=E2=80=99s a good set of automated systems to =
ferret out=20
problems introduced by patches.

Tom=

From 1585336857272350530@xxx Tue Nov 28 18:50:43 +0000 2017
X-GM-THRID: 1585254701857422353
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread