## TL;DR
This patchset adds a centralized executor to dispatch tests rather than
relying on late_initcall to schedule each test suite separately along
with a couple of new features that depend on it.
## What am I trying to do?
Conceptually, I am trying to provide a mechanism by which test suites
can be grouped together so that they can be reasoned about collectively.
The last two of three patches in this series add features which depend
on this:
PATCH 5/7 Prints out a test plan right before KUnit tests are run[1];
this is valuable because it makes it possible for a test
harness to detect whether the number of tests run matches the
number of tests expected to be run, ensuring that no tests
silently failed.
PATCH 6/7 Add a new kernel command-line option which allows the user to
specify that the kernel poweroff, halt, or reboot after
completing all KUnit tests; this is very handy for running
KUnit tests on UML or a VM so that the UML/VM process exits
cleanly immediately after running all tests without needing a
special initramfs.
In addition, by dispatching tests from a single location, we can
guarantee that all KUnit tests run after late_init is complete, which
was a concern during the initial KUnit patchset review (this has not
been a problem in practice, but resolving with certainty is nevertheless
desirable).
Other use cases for this exist, but the above features should provide an
idea of the value that this could provide.
Alan Maguire (1):
kunit: test: create a single centralized executor for all tests
Brendan Higgins (5):
vmlinux.lds.h: add linker section for KUnit test suites
arch: um: add linker section for KUnit test suites
init: main: add KUnit to kernel init
kunit: test: add test plan to KUnit TAP format
Documentation: Add kunit_shutdown to kernel-parameters.txt
David Gow (1):
kunit: Add 'kunit_shutdown' option
.../admin-guide/kernel-parameters.txt | 7 ++
arch/um/include/asm/common.lds.S | 4 +
include/asm-generic/vmlinux.lds.h | 8 ++
include/kunit/test.h | 82 ++++++++++++-------
init/main.c | 4 +
lib/kunit/Makefile | 3 +-
lib/kunit/executor.c | 71 ++++++++++++++++
lib/kunit/test.c | 11 ---
tools/testing/kunit/kunit_kernel.py | 2 +-
tools/testing/kunit/kunit_parser.py | 76 ++++++++++++++---
.../test_is_test_passed-all_passed.log | 1 +
.../test_data/test_is_test_passed-crash.log | 1 +
.../test_data/test_is_test_passed-failure.log | 1 +
13 files changed, 217 insertions(+), 54 deletions(-)
create mode 100644 lib/kunit/executor.c
--
2.25.0.341.g760bfbb309-goog
From: Alan Maguire <[email protected]>
Add a centralized executor to dispatch tests rather than relying on
late_initcall to schedule each test suite separately. Centralized
execution is for built-in tests only; modules will execute tests
when loaded.
Signed-off-by: Alan Maguire <[email protected]>
Co-developed-by: Iurii Zaikin <[email protected]>
Signed-off-by: Iurii Zaikin <[email protected]>
Co-developed-by: Brendan Higgins <[email protected]>
Signed-off-by: Brendan Higgins <[email protected]>
Reviewed-by: Stephen Boyd <[email protected]>
---
include/kunit/test.h | 73 +++++++++++++++++++++++++++-----------------
lib/kunit/Makefile | 3 +-
lib/kunit/executor.c | 36 ++++++++++++++++++++++
3 files changed, 83 insertions(+), 29 deletions(-)
create mode 100644 lib/kunit/executor.c
diff --git a/include/kunit/test.h b/include/kunit/test.h
index 2dfb550c6723a..8a02f93a6b505 100644
--- a/include/kunit/test.h
+++ b/include/kunit/test.h
@@ -197,46 +197,63 @@ void kunit_init_test(struct kunit *test, const char *name);
int kunit_run_tests(struct kunit_suite *suite);
-/**
- * kunit_test_suites() - used to register one or more &struct kunit_suite
- * with KUnit.
- *
- * @suites: a statically allocated list of &struct kunit_suite.
- *
- * Registers @suites with the test framework. See &struct kunit_suite for
- * more information.
- *
- * When builtin, KUnit tests are all run as late_initcalls; this means
- * that they cannot test anything where tests must run at a different init
- * phase. One significant restriction resulting from this is that KUnit
- * cannot reliably test anything that is initialize in the late_init phase;
- * another is that KUnit is useless to test things that need to be run in
- * an earlier init phase.
- *
- * An alternative is to build the tests as a module. Because modules
- * do not support multiple late_initcall()s, we need to initialize an
- * array of suites for a module.
- *
- * TODO([email protected]): Don't run all KUnit tests as
- * late_initcalls. I have some future work planned to dispatch all KUnit
- * tests from the same place, and at the very least to do so after
- * everything else is definitely initialized.
+/*
+ * If a test suite is built-in, module_init() gets translated into
+ * an initcall which we don't want as the idea is that for builtins
+ * the executor will manage execution. So ensure we do not define
+ * module_{init|exit} functions for the builtin case when registering
+ * suites via kunit_test_suites() below.
*/
-#define kunit_test_suites(...) \
- static struct kunit_suite *suites[] = { __VA_ARGS__, NULL}; \
- static int kunit_test_suites_init(void) \
+#ifdef MODULE
+#define kunit_test_suites_for_module(__suites) \
+ static int __init kunit_test_suites_init(void) \
{ \
+ struct kunit_suite *suites[] = (__suites); \
unsigned int i; \
+ \
for (i = 0; suites[i] != NULL; i++) \
kunit_run_tests(suites[i]); \
return 0; \
} \
- late_initcall(kunit_test_suites_init); \
+ module_init(kunit_test_suites_init); \
+ \
static void __exit kunit_test_suites_exit(void) \
{ \
return; \
} \
module_exit(kunit_test_suites_exit)
+#else
+#define kunit_test_suites_for_module(__suites)
+#endif /* MODULE */
+
+#define __kunit_test_suites(unique_array, unique_suites, ...) \
+ static struct kunit_suite *unique_array[] = { __VA_ARGS__, NULL }; \
+ kunit_test_suites_for_module(unique_array); \
+ static struct kunit_suite **unique_suites \
+ __used __aligned(8) __section(.kunit_test_suites) = unique_array
+
+/**
+ * kunit_test_suites() - used to register one or more &struct kunit_suite
+ * with KUnit.
+ *
+ * @suites: a statically allocated list of &struct kunit_suite.
+ *
+ * Registers @suites with the test framework. See &struct kunit_suite for
+ * more information.
+ *
+ * When builtin, KUnit tests are all run via executor; this is done
+ * by placing the array of struct kunit_suite * in the .kunit_test_suites
+ * ELF section.
+ *
+ * An alternative is to build the tests as a module. Because modules do not
+ * support multiple initcall()s, we need to initialize an array of suites for a
+ * module.
+ *
+ */
+#define kunit_test_suites(...) \
+ __kunit_test_suites(__UNIQUE_ID(array), \
+ __UNIQUE_ID(suites), \
+ __VA_ARGS__)
#define kunit_test_suite(suite) kunit_test_suites(&suite)
diff --git a/lib/kunit/Makefile b/lib/kunit/Makefile
index fab55649b69a5..c282f02ca066b 100644
--- a/lib/kunit/Makefile
+++ b/lib/kunit/Makefile
@@ -3,7 +3,8 @@ obj-$(CONFIG_KUNIT) += kunit.o
kunit-objs += test.o \
string-stream.o \
assert.o \
- try-catch.o
+ try-catch.o \
+ executor.o
obj-$(CONFIG_KUNIT_TEST) += kunit-test.o
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
new file mode 100644
index 0000000000000..6429927d598a5
--- /dev/null
+++ b/lib/kunit/executor.c
@@ -0,0 +1,36 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <kunit/test.h>
+
+/*
+ * These symbols point to the .kunit_test_suites section and are defined in
+ * include/asm-generic/vmlinux.lds.h, and consequently must be extern.
+ */
+extern struct kunit_suite * const * const __kunit_suites_start[];
+extern struct kunit_suite * const * const __kunit_suites_end[];
+
+#if IS_BUILTIN(CONFIG_KUNIT)
+
+static int kunit_run_all_tests(void)
+{
+ struct kunit_suite * const * const *suites, * const *subsuite;
+ bool has_test_failed = false;
+
+ for (suites = __kunit_suites_start;
+ suites < __kunit_suites_end;
+ suites++) {
+ for (subsuite = *suites; *subsuite != NULL; subsuite++) {
+ if (kunit_run_tests(*subsuite))
+ has_test_failed = true;
+ }
+ }
+
+ if (has_test_failed)
+ return -EFAULT;
+
+ return 0;
+}
+
+late_initcall(kunit_run_all_tests);
+
+#endif /* IS_BUILTIN(CONFIG_KUNIT) */
--
2.25.0.341.g760bfbb309-goog
TAP 14 allows an optional test plan to be emitted before the start of
the start of testing[1]; this is valuable because it makes it possible
for a test harness to detect whether the number of tests run matches the
number of tests expected to be run, ensuring that no tests silently
failed.
Link[1]: https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan
Signed-off-by: Brendan Higgins <[email protected]>
Reviewed-by: Stephen Boyd <[email protected]>
---
lib/kunit/executor.c | 17 +++++
lib/kunit/test.c | 11 ---
tools/testing/kunit/kunit_parser.py | 74 ++++++++++++++++---
.../test_is_test_passed-all_passed.log | 1 +
.../test_data/test_is_test_passed-crash.log | 1 +
.../test_data/test_is_test_passed-failure.log | 1 +
6 files changed, 82 insertions(+), 23 deletions(-)
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
index b75a46c560847..7fd16feff157e 100644
--- a/lib/kunit/executor.c
+++ b/lib/kunit/executor.c
@@ -11,11 +11,28 @@ extern struct kunit_suite * const * const __kunit_suites_end[];
#if IS_BUILTIN(CONFIG_KUNIT)
+static void kunit_print_tap_header(void)
+{
+ struct kunit_suite * const * const *suites, * const *subsuite;
+ int num_of_suites = 0;
+
+ for (suites = __kunit_suites_start;
+ suites < __kunit_suites_end;
+ suites++)
+ for (subsuite = *suites; *subsuite != NULL; subsuite++)
+ num_of_suites++;
+
+ pr_info("TAP version 14\n");
+ pr_info("1..%d\n", num_of_suites);
+}
+
int kunit_run_all_tests(void)
{
struct kunit_suite * const * const *suites, * const *subsuite;
bool has_test_failed = false;
+ kunit_print_tap_header();
+
for (suites = __kunit_suites_start;
suites < __kunit_suites_end;
suites++) {
diff --git a/lib/kunit/test.c b/lib/kunit/test.c
index 9242f932896c7..da56b94261b43 100644
--- a/lib/kunit/test.c
+++ b/lib/kunit/test.c
@@ -18,16 +18,6 @@ static void kunit_set_failure(struct kunit *test)
WRITE_ONCE(test->success, false);
}
-static void kunit_print_tap_version(void)
-{
- static bool kunit_has_printed_tap_version;
-
- if (!kunit_has_printed_tap_version) {
- pr_info("TAP version 14\n");
- kunit_has_printed_tap_version = true;
- }
-}
-
static size_t kunit_test_cases_len(struct kunit_case *test_cases)
{
struct kunit_case *test_case;
@@ -41,7 +31,6 @@ static size_t kunit_test_cases_len(struct kunit_case *test_cases)
static void kunit_print_subtest_start(struct kunit_suite *suite)
{
- kunit_print_tap_version();
pr_info("\t# Subtest: %s\n", suite->name);
pr_info("\t1..%zd\n", kunit_test_cases_len(suite->test_cases));
}
diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
index 4ffbae0f67325..78b3bdd03b1e4 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -45,6 +45,7 @@ class TestStatus(Enum):
FAILURE = auto()
TEST_CRASHED = auto()
NO_TESTS = auto()
+ FAILURE_TO_PARSE_TESTS = auto()
kunit_start_re = re.compile(r'^TAP version [0-9]+$')
kunit_end_re = re.compile('List of all partitions:')
@@ -106,7 +107,7 @@ OkNotOkResult = namedtuple('OkNotOkResult', ['is_ok','description', 'text'])
OK_NOT_OK_SUBTEST = re.compile(r'^\t(ok|not ok) [0-9]+ - (.*)$')
-OK_NOT_OK_MODULE = re.compile(r'^(ok|not ok) [0-9]+ - (.*)$')
+OK_NOT_OK_MODULE = re.compile(r'^(ok|not ok) ([0-9]+) - (.*)$')
def parse_ok_not_ok_test_case(lines: List[str],
test_case: TestCase,
@@ -196,7 +197,9 @@ def max_status(left: TestStatus, right: TestStatus) -> TestStatus:
else:
return TestStatus.SUCCESS
-def parse_ok_not_ok_test_suite(lines: List[str], test_suite: TestSuite) -> bool:
+def parse_ok_not_ok_test_suite(lines: List[str],
+ test_suite: TestSuite,
+ expected_suite_index: int) -> bool:
consume_non_diagnositic(lines)
if not lines:
test_suite.status = TestStatus.TEST_CRASHED
@@ -209,6 +212,12 @@ def parse_ok_not_ok_test_suite(lines: List[str], test_suite: TestSuite) -> bool:
test_suite.status = TestStatus.SUCCESS
else:
test_suite.status = TestStatus.FAILURE
+ suite_index = int(match.group(2))
+ if suite_index != expected_suite_index:
+ print_with_timestamp(
+ red('[ERROR] ') + 'expected_suite_index ' +
+ str(expected_suite_index) + ', but got ' +
+ str(suite_index))
return True
else:
return False
@@ -221,7 +230,7 @@ def bubble_up_test_case_errors(test_suite: TestSuite) -> TestStatus:
max_test_case_status = bubble_up_errors(lambda x: x.status, test_suite.cases)
return max_status(max_test_case_status, test_suite.status)
-def parse_test_suite(lines: List[str]) -> TestSuite:
+def parse_test_suite(lines: List[str], expected_suite_index: int) -> TestSuite:
if not lines:
return None
consume_non_diagnositic(lines)
@@ -240,7 +249,7 @@ def parse_test_suite(lines: List[str]) -> TestSuite:
test_suite.cases.append(test_case)
test_case = parse_test_case(lines, expected_test_case_num > 0)
expected_test_case_num -= 1
- if parse_ok_not_ok_test_suite(lines, test_suite):
+ if parse_ok_not_ok_test_suite(lines, test_suite, expected_suite_index):
test_suite.status = bubble_up_test_case_errors(test_suite)
return test_suite
elif not lines:
@@ -260,6 +269,17 @@ def parse_tap_header(lines: List[str]) -> bool:
else:
return False
+TEST_PLAN = re.compile(r'[0-9]+\.\.([0-9]+)')
+
+def parse_test_plan(lines: List[str]) -> int:
+ consume_non_diagnositic(lines)
+ match = TEST_PLAN.match(lines[0])
+ if match:
+ lines.pop(0)
+ return int(match.group(1))
+ else:
+ return None
+
def bubble_up_suite_errors(test_suite_list: List[TestSuite]) -> TestStatus:
return bubble_up_errors(lambda x: x.status, test_suite_list)
@@ -268,19 +288,34 @@ def parse_test_result(lines: List[str]) -> TestResult:
return TestResult(TestStatus.NO_TESTS, [], lines)
consume_non_diagnositic(lines)
if not parse_tap_header(lines):
- return None
+ return TestResult(TestStatus.NO_TESTS, [], lines)
+ expected_test_suite_num = parse_test_plan(lines)
+ if not expected_test_suite_num:
+ return TestResult(TestStatus.FAILURE_TO_PARSE_TESTS, [], lines)
test_suites = []
- test_suite = parse_test_suite(lines)
- while test_suite:
- test_suites.append(test_suite)
- test_suite = parse_test_suite(lines)
- return TestResult(bubble_up_suite_errors(test_suites), test_suites, lines)
+ for i in range(1, expected_test_suite_num + 1):
+ test_suite = parse_test_suite(lines, i)
+ if test_suite:
+ test_suites.append(test_suite)
+ else:
+ print_with_timestamp(
+ red('[ERROR] ') + ' expected ' +
+ str(expected_test_suite_num) +
+ ' test suites, but got ' + str(i - 2))
+ break
+ test_suite = parse_test_suite(lines, -1)
+ if test_suite:
+ print_with_timestamp(red('[ERROR] ') +
+ 'got unexpected test suite: ' + test_suite.name)
+ if test_suites:
+ return TestResult(bubble_up_suite_errors(test_suites), test_suites, lines)
+ else:
+ return TestResult(TestStatus.NO_TESTS, [], lines)
-def parse_run_tests(kernel_output) -> TestResult:
+def print_and_count_results(test_result: TestResult) -> None:
total_tests = 0
failed_tests = 0
crashed_tests = 0
- test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
for test_suite in test_result.suites:
if test_suite.status == TestStatus.SUCCESS:
print_suite_divider(green('[PASSED] ') + test_suite.name)
@@ -302,6 +337,21 @@ def parse_run_tests(kernel_output) -> TestResult:
print_with_timestamp(red('[FAILED] ') + test_case.name)
print_log(map(yellow, test_case.log))
print_with_timestamp('')
+ return total_tests, failed_tests, crashed_tests
+
+def parse_run_tests(kernel_output) -> TestResult:
+ total_tests = 0
+ failed_tests = 0
+ crashed_tests = 0
+ test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
+ if test_result.status == TestStatus.NO_TESTS:
+ print(red('[ERROR] ') + yellow('no tests run!'))
+ elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:
+ print(red('[ERROR] ') + yellow('could not parse test results!'))
+ else:
+ (total_tests,
+ failed_tests,
+ crashed_tests) = print_and_count_results(test_result)
print_with_timestamp(DIVIDER)
fmt = green if test_result.status == TestStatus.SUCCESS else red
print_with_timestamp(
diff --git a/tools/testing/kunit/test_data/test_is_test_passed-all_passed.log b/tools/testing/kunit/test_data/test_is_test_passed-all_passed.log
index 62ebc0288355c..bc0dc8fe35b76 100644
--- a/tools/testing/kunit/test_data/test_is_test_passed-all_passed.log
+++ b/tools/testing/kunit/test_data/test_is_test_passed-all_passed.log
@@ -1,4 +1,5 @@
TAP version 14
+1..2
# Subtest: sysctl_test
1..8
# sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed
diff --git a/tools/testing/kunit/test_data/test_is_test_passed-crash.log b/tools/testing/kunit/test_data/test_is_test_passed-crash.log
index 0b249870c8be4..4d97f6708c4a5 100644
--- a/tools/testing/kunit/test_data/test_is_test_passed-crash.log
+++ b/tools/testing/kunit/test_data/test_is_test_passed-crash.log
@@ -1,6 +1,7 @@
printk: console [tty0] enabled
printk: console [mc-1] enabled
TAP version 14
+1..2
# Subtest: sysctl_test
1..8
# sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed
diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure.log b/tools/testing/kunit/test_data/test_is_test_passed-failure.log
index 9e89d32d5667a..7a416497e3bec 100644
--- a/tools/testing/kunit/test_data/test_is_test_passed-failure.log
+++ b/tools/testing/kunit/test_data/test_is_test_passed-failure.log
@@ -1,4 +1,5 @@
TAP version 14
+1..2
# Subtest: sysctl_test
1..8
# sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed
--
2.25.0.341.g760bfbb309-goog
From: David Gow <[email protected]>
Add a new kernel command-line option, 'kunit_shutdown', which allows the
user to specify that the kernel poweroff, halt, or reboot after
completing all KUnit tests; this is very handy for running KUnit tests
on UML or a VM so that the UML/VM process exits cleanly immediately
after running all tests without needing a special initramfs.
Signed-off-by: David Gow <[email protected]>
Signed-off-by: Brendan Higgins <[email protected]>
Reviewed-by: Stephen Boyd <[email protected]>
---
lib/kunit/executor.c | 20 ++++++++++++++++++++
tools/testing/kunit/kunit_kernel.py | 2 +-
tools/testing/kunit/kunit_parser.py | 2 +-
3 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
index 7fd16feff157e..a93821116ccec 100644
--- a/lib/kunit/executor.c
+++ b/lib/kunit/executor.c
@@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
+#include <linux/reboot.h>
#include <kunit/test.h>
/*
@@ -11,6 +12,23 @@ extern struct kunit_suite * const * const __kunit_suites_end[];
#if IS_BUILTIN(CONFIG_KUNIT)
+static char *kunit_shutdown;
+core_param(kunit_shutdown, kunit_shutdown, charp, 0644);
+
+static void kunit_handle_shutdown(void)
+{
+ if (!kunit_shutdown)
+ return;
+
+ if (!strcmp(kunit_shutdown, "poweroff"))
+ kernel_power_off();
+ else if (!strcmp(kunit_shutdown, "halt"))
+ kernel_halt();
+ else if (!strcmp(kunit_shutdown, "reboot"))
+ kernel_restart(NULL);
+
+}
+
static void kunit_print_tap_header(void)
{
struct kunit_suite * const * const *suites, * const *subsuite;
@@ -42,6 +60,8 @@ int kunit_run_all_tests(void)
}
}
+ kunit_handle_shutdown();
+
if (has_test_failed)
return -EFAULT;
diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
index cc5d844ecca13..43314aa537d30 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -141,7 +141,7 @@ class LinuxSourceTree(object):
return True
def run_kernel(self, args=[], timeout=None, build_dir=''):
- args.extend(['mem=256M'])
+ args.extend(['mem=256M', 'kunit_shutdown=halt'])
process = self._ops.linux_bin(args, timeout, build_dir)
with open(os.path.join(build_dir, 'test.log'), 'w') as f:
for line in process.stdout:
diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
index 78b3bdd03b1e4..633811dd9bce8 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -48,7 +48,7 @@ class TestStatus(Enum):
FAILURE_TO_PARSE_TESTS = auto()
kunit_start_re = re.compile(r'^TAP version [0-9]+$')
-kunit_end_re = re.compile('List of all partitions:')
+kunit_end_re = re.compile(r'reboot: System halted')
def isolate_kunit_output(kernel_output):
started = False
--
2.25.0.341.g760bfbb309-goog
Quoting Brendan Higgins (2020-01-30 15:08:05)
> ## TL;DR
>
> This patchset adds a centralized executor to dispatch tests rather than
> relying on late_initcall to schedule each test suite separately along
> with a couple of new features that depend on it.
Is there any diff from v1 to v2? I don't know what changed, but I see
that my Reviewed-by tag has been put on everything, so I guess
everything I said was addressed or discussed in the previous round.
On Mon, Feb 3, 2020 at 11:19 PM Stephen Boyd <[email protected]> wrote:
>
> Quoting Brendan Higgins (2020-01-30 15:08:05)
> > ## TL;DR
> >
> > This patchset adds a centralized executor to dispatch tests rather than
> > relying on late_initcall to schedule each test suite separately along
> > with a couple of new features that depend on it.
>
> Is there any diff from v1 to v2? I don't know what changed, but I see
> that my Reviewed-by tag has been put on everything, so I guess
> everything I said was addressed or discussed in the previous round.
Oh yes, sorry about that. I have gotten a bit lazy in regard to
changing logs. I noticed that a lot of people don't seem to care. I'll
make a note that you do.
Changes since last revision:
- On patch 6/7, I flipped the include order and removed braces from the if
statements.
- On patch 7/7, I removed the periods from the short descriptions.
On 1/30/20 5:08 PM, Brendan Higgins wrote:
> ## TL;DR
>
> This patchset adds a centralized executor to dispatch tests rather than
> relying on late_initcall to schedule each test suite separately along
> with a couple of new features that depend on it.
And the "couple of new features" are .... ?
>
> ## What am I trying to do?
>
> Conceptually, I am trying to provide a mechanism by which test suites
> can be grouped together so that they can be reasoned about collectively.
> The last two of three patches in this series add features which depend
> on this:
>
> PATCH 5/7 Prints out a test plan right before KUnit tests are run[1];
> this is valuable because it makes it possible for a test
> harness to detect whether the number of tests run matches the
> number of tests expected to be run, ensuring that no tests
> silently failed.
>
> PATCH 6/7 Add a new kernel command-line option which allows the user to
> specify that the kernel poweroff, halt, or reboot after
> completing all KUnit tests; this is very handy for running
> KUnit tests on UML or a VM so that the UML/VM process exits
> cleanly immediately after running all tests without needing a
> special initramfs.
>
> In addition, by dispatching tests from a single location, we can
> guarantee that all KUnit tests run after late_init is complete, which
That the tests will run after late init (and are guaranteed to do such)
needs to be added to the documentation.
-Frank
> was a concern during the initial KUnit patchset review (this has not
> been a problem in practice, but resolving with certainty is nevertheless
> desirable).
>
> Other use cases for this exist, but the above features should provide an
> idea of the value that this could provide.
>
> Alan Maguire (1):
> kunit: test: create a single centralized executor for all tests
>
> Brendan Higgins (5):
> vmlinux.lds.h: add linker section for KUnit test suites
> arch: um: add linker section for KUnit test suites
> init: main: add KUnit to kernel init
> kunit: test: add test plan to KUnit TAP format
> Documentation: Add kunit_shutdown to kernel-parameters.txt
>
> David Gow (1):
> kunit: Add 'kunit_shutdown' option
>
> .../admin-guide/kernel-parameters.txt | 7 ++
> arch/um/include/asm/common.lds.S | 4 +
> include/asm-generic/vmlinux.lds.h | 8 ++
> include/kunit/test.h | 82 ++++++++++++-------
> init/main.c | 4 +
> lib/kunit/Makefile | 3 +-
> lib/kunit/executor.c | 71 ++++++++++++++++
> lib/kunit/test.c | 11 ---
> tools/testing/kunit/kunit_kernel.py | 2 +-
> tools/testing/kunit/kunit_parser.py | 76 ++++++++++++++---
> .../test_is_test_passed-all_passed.log | 1 +
> .../test_data/test_is_test_passed-crash.log | 1 +
> .../test_data/test_is_test_passed-failure.log | 1 +
> 13 files changed, 217 insertions(+), 54 deletions(-)
> create mode 100644 lib/kunit/executor.c
>
On Tue, Feb 4, 2020 at 1:18 PM Frank Rowand <[email protected]> wrote:
>
> On 1/30/20 5:08 PM, Brendan Higgins wrote:
> > ## TL;DR
> >
> > This patchset adds a centralized executor to dispatch tests rather than
> > relying on late_initcall to schedule each test suite separately along
> > with a couple of new features that depend on it.
>
> And the "couple of new features" are .... ?
Sorry, I thought I addressed that in the next section, specifically by
calling out: printing the test plan, and adding the new kernel command
line option. I will call these out here in any future cover letters.
> > ## What am I trying to do?
> >
> > Conceptually, I am trying to provide a mechanism by which test suites
> > can be grouped together so that they can be reasoned about collectively.
> > The last two of three patches in this series add features which depend
> > on this:
> >
> > PATCH 5/7 Prints out a test plan right before KUnit tests are run[1];
> > this is valuable because it makes it possible for a test
> > harness to detect whether the number of tests run matches the
> > number of tests expected to be run, ensuring that no tests
> > silently failed.
> >
> > PATCH 6/7 Add a new kernel command-line option which allows the user to
> > specify that the kernel poweroff, halt, or reboot after
> > completing all KUnit tests; this is very handy for running
> > KUnit tests on UML or a VM so that the UML/VM process exits
> > cleanly immediately after running all tests without needing a
> > special initramfs.
> >
>
> > In addition, by dispatching tests from a single location, we can
> > guarantee that all KUnit tests run after late_init is complete, which
>
> That the tests will run after late init (and are guaranteed to do such)
> needs to be added to the documentation.
Yeah, that's reasonable. I am not sure where I should put this in the
documentation, however. This seems kind of a technical detail, and all
the pages I have now are more of how-tos, I think.
Maybe I should send a patch which adds a page detailing how KUnit works?
I would like to get some other people's thoughts on this. Such a
technical guide wouldn't provide me a lot of value, at least not now,
so I want to make sure that something like that would be valuable to
others.
On 1/30/20 5:08 PM, Brendan Higgins wrote:
> From: Alan Maguire <[email protected]>
>
> Add a centralized executor to dispatch tests rather than relying on
> late_initcall to schedule each test suite separately. Centralized
> execution is for built-in tests only; modules will execute tests
> when loaded.
>
> Signed-off-by: Alan Maguire <[email protected]>
> Co-developed-by: Iurii Zaikin <[email protected]>
> Signed-off-by: Iurii Zaikin <[email protected]>
> Co-developed-by: Brendan Higgins <[email protected]>
> Signed-off-by: Brendan Higgins <[email protected]>
> Reviewed-by: Stephen Boyd <[email protected]>
> ---
> include/kunit/test.h | 73 +++++++++++++++++++++++++++-----------------
> lib/kunit/Makefile | 3 +-
> lib/kunit/executor.c | 36 ++++++++++++++++++++++
> 3 files changed, 83 insertions(+), 29 deletions(-)
> create mode 100644 lib/kunit/executor.c
>
> diff --git a/include/kunit/test.h b/include/kunit/test.h
> index 2dfb550c6723a..8a02f93a6b505 100644
> --- a/include/kunit/test.h
> +++ b/include/kunit/test.h
The following fragment does not match the test.h in Linux 5.5 or 5.4-rc1 (as one
possible earlier version). And we are not to Linux 5.5-rc1 yet. (Simple way
to check for the mis-match - 5.5 has kunit_test_suite() instead of
kunit_test_suites().)
I know that there is an alternate tree where some of the development occurs.
Can you please add a link in MAINTAINERS? And please note (at least in
patch 0) what tree the series is developed against?
-Frank
> @@ -197,46 +197,63 @@ void kunit_init_test(struct kunit *test, const char *name);
>
> int kunit_run_tests(struct kunit_suite *suite);
>
> -/**
> - * kunit_test_suites() - used to register one or more &struct kunit_suite
> - * with KUnit.
> - *
> - * @suites: a statically allocated list of &struct kunit_suite.
> - *
> - * Registers @suites with the test framework. See &struct kunit_suite for
> - * more information.
> - *
> - * When builtin, KUnit tests are all run as late_initcalls; this means
> - * that they cannot test anything where tests must run at a different init
> - * phase. One significant restriction resulting from this is that KUnit
> - * cannot reliably test anything that is initialize in the late_init phase;
> - * another is that KUnit is useless to test things that need to be run in
> - * an earlier init phase.
> - *
> - * An alternative is to build the tests as a module. Because modules
> - * do not support multiple late_initcall()s, we need to initialize an
> - * array of suites for a module.
> - *
> - * TODO([email protected]): Don't run all KUnit tests as
> - * late_initcalls. I have some future work planned to dispatch all KUnit
> - * tests from the same place, and at the very least to do so after
> - * everything else is definitely initialized.
> +/*
> + * If a test suite is built-in, module_init() gets translated into
> + * an initcall which we don't want as the idea is that for builtins
> + * the executor will manage execution. So ensure we do not define
> + * module_{init|exit} functions for the builtin case when registering
> + * suites via kunit_test_suites() below.
> */
> -#define kunit_test_suites(...) \
> - static struct kunit_suite *suites[] = { __VA_ARGS__, NULL}; \
> - static int kunit_test_suites_init(void) \
> +#ifdef MODULE
> +#define kunit_test_suites_for_module(__suites) \
> + static int __init kunit_test_suites_init(void) \
> { \
> + struct kunit_suite *suites[] = (__suites); \
> unsigned int i; \
> + \
> for (i = 0; suites[i] != NULL; i++) \
> kunit_run_tests(suites[i]); \
> return 0; \
> } \
> - late_initcall(kunit_test_suites_init); \
> + module_init(kunit_test_suites_init); \
> + \
> static void __exit kunit_test_suites_exit(void) \
> { \
> return; \
> } \
> module_exit(kunit_test_suites_exit)
> +#else
> +#define kunit_test_suites_for_module(__suites)
> +#endif /* MODULE */
> +
> +#define __kunit_test_suites(unique_array, unique_suites, ...) \
> + static struct kunit_suite *unique_array[] = { __VA_ARGS__, NULL }; \
> + kunit_test_suites_for_module(unique_array); \
> + static struct kunit_suite **unique_suites \
> + __used __aligned(8) __section(.kunit_test_suites) = unique_array
> +
> +/**
> + * kunit_test_suites() - used to register one or more &struct kunit_suite
> + * with KUnit.
> + *
> + * @suites: a statically allocated list of &struct kunit_suite.
> + *
> + * Registers @suites with the test framework. See &struct kunit_suite for
> + * more information.
> + *
> + * When builtin, KUnit tests are all run via executor; this is done
> + * by placing the array of struct kunit_suite * in the .kunit_test_suites
> + * ELF section.
> + *
> + * An alternative is to build the tests as a module. Because modules do not
> + * support multiple initcall()s, we need to initialize an array of suites for a
> + * module.
> + *
> + */
> +#define kunit_test_suites(...) \
> + __kunit_test_suites(__UNIQUE_ID(array), \
> + __UNIQUE_ID(suites), \
> + __VA_ARGS__)
>
> #define kunit_test_suite(suite) kunit_test_suites(&suite)
>
> diff --git a/lib/kunit/Makefile b/lib/kunit/Makefile
> index fab55649b69a5..c282f02ca066b 100644
> --- a/lib/kunit/Makefile
> +++ b/lib/kunit/Makefile
> @@ -3,7 +3,8 @@ obj-$(CONFIG_KUNIT) += kunit.o
> kunit-objs += test.o \
> string-stream.o \
> assert.o \
> - try-catch.o
> + try-catch.o \
> + executor.o
>
> obj-$(CONFIG_KUNIT_TEST) += kunit-test.o
>
> diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
> new file mode 100644
> index 0000000000000..6429927d598a5
> --- /dev/null
> +++ b/lib/kunit/executor.c
> @@ -0,0 +1,36 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +#include <kunit/test.h>
> +
> +/*
> + * These symbols point to the .kunit_test_suites section and are defined in
> + * include/asm-generic/vmlinux.lds.h, and consequently must be extern.
> + */
> +extern struct kunit_suite * const * const __kunit_suites_start[];
> +extern struct kunit_suite * const * const __kunit_suites_end[];
> +
> +#if IS_BUILTIN(CONFIG_KUNIT)
> +
> +static int kunit_run_all_tests(void)
> +{
> + struct kunit_suite * const * const *suites, * const *subsuite;
> + bool has_test_failed = false;
> +
> + for (suites = __kunit_suites_start;
> + suites < __kunit_suites_end;
> + suites++) {
> + for (subsuite = *suites; *subsuite != NULL; subsuite++) {
> + if (kunit_run_tests(*subsuite))
> + has_test_failed = true;
> + }
> + }
> +
> + if (has_test_failed)
> + return -EFAULT;
> +
> + return 0;
> +}
> +
> +late_initcall(kunit_run_all_tests);
> +
> +#endif /* IS_BUILTIN(CONFIG_KUNIT) */
>
On Tue, Feb 4, 2020 at 2:27 PM Frank Rowand <[email protected]> wrote:
>
> On 1/30/20 5:08 PM, Brendan Higgins wrote:
> > From: Alan Maguire <[email protected]>
> >
> > Add a centralized executor to dispatch tests rather than relying on
> > late_initcall to schedule each test suite separately. Centralized
> > execution is for built-in tests only; modules will execute tests
> > when loaded.
> >
> > Signed-off-by: Alan Maguire <[email protected]>
> > Co-developed-by: Iurii Zaikin <[email protected]>
> > Signed-off-by: Iurii Zaikin <[email protected]>
> > Co-developed-by: Brendan Higgins <[email protected]>
> > Signed-off-by: Brendan Higgins <[email protected]>
> > Reviewed-by: Stephen Boyd <[email protected]>
> > ---
> > include/kunit/test.h | 73 +++++++++++++++++++++++++++-----------------
> > lib/kunit/Makefile | 3 +-
> > lib/kunit/executor.c | 36 ++++++++++++++++++++++
> > 3 files changed, 83 insertions(+), 29 deletions(-)
> > create mode 100644 lib/kunit/executor.c
> >
> > diff --git a/include/kunit/test.h b/include/kunit/test.h
> > index 2dfb550c6723a..8a02f93a6b505 100644
> > --- a/include/kunit/test.h
> > +++ b/include/kunit/test.h
>
> The following fragment does not match the test.h in Linux 5.5 or 5.4-rc1 (as one
> possible earlier version). And we are not to Linux 5.5-rc1 yet. (Simple way
> to check for the mis-match - 5.5 has kunit_test_suite() instead of
> kunit_test_suites().)
>
> I know that there is an alternate tree where some of the development occurs.
> Can you please add a link in MAINTAINERS? And please note (at least in
> patch 0) what tree the series is developed against?
Yep, I was planning on sending an update to the MAINTAINERS with that
and some other updates soon.
In future revisions, I will put a link to the tree I developed
against. In the meantime, we send pull-requests from here:
https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=kunit
And that is what I used as my development base for this series.
On 1/30/20 5:08 PM, Brendan Higgins wrote:
> TAP 14 allows an optional test plan to be emitted before the start of
> the start of testing[1]; this is valuable because it makes it possible
> for a test harness to detect whether the number of tests run matches the
> number of tests expected to be run, ensuring that no tests silently
> failed.
>
> Link[1]: https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan
> Signed-off-by: Brendan Higgins <[email protected]>
> Reviewed-by: Stephen Boyd <[email protected]>
> ---
> lib/kunit/executor.c | 17 +++++
> lib/kunit/test.c | 11 ---
> tools/testing/kunit/kunit_parser.py | 74 ++++++++++++++++---
> .../test_is_test_passed-all_passed.log | 1 +
> .../test_data/test_is_test_passed-crash.log | 1 +
> .../test_data/test_is_test_passed-failure.log | 1 +
> 6 files changed, 82 insertions(+), 23 deletions(-)
>
> diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
> index b75a46c560847..7fd16feff157e 100644
> --- a/lib/kunit/executor.c
> +++ b/lib/kunit/executor.c
> @@ -11,11 +11,28 @@ extern struct kunit_suite * const * const __kunit_suites_end[];
>
> #if IS_BUILTIN(CONFIG_KUNIT)
>
> +static void kunit_print_tap_header(void)
> +{
> + struct kunit_suite * const * const *suites, * const *subsuite;
> + int num_of_suites = 0;
> +
> + for (suites = __kunit_suites_start;
> + suites < __kunit_suites_end;
> + suites++)
> + for (subsuite = *suites; *subsuite != NULL; subsuite++)
> + num_of_suites++;
> +
> + pr_info("TAP version 14\n");
> + pr_info("1..%d\n", num_of_suites);
> +}
> +
Subsuites are not in Linux 5.5, as far as I can tell, so it is hard to review
this.
But I think this observation will still be valid even though I am not up
to speed on subsuites.
num_of_suites is all test suites declared (as built in). In Linux 5.5,
each kunit_suite contains one or more test cases. Each test case leads
to either "ok" or "not ok". So instead of counting number of tests
suites, the number of test cases needs to be counted.
Ahhh, I think my lack of visibility of what tree this series is against
is about to lead me astray! Are the test cases in the subsuite
implemented as what TAP 14 calls "subtests"? If so, then the above
code that counts suites instead of test cases would be correct.
I'm spinning my wheels now, since I don't know what I am
reviewing against, so I'm going to stop reviewing, other
than maybe a quick scan.
-Frank
> int kunit_run_all_tests(void)
> {
> struct kunit_suite * const * const *suites, * const *subsuite;
> bool has_test_failed = false;
>
> + kunit_print_tap_header();
> +
> for (suites = __kunit_suites_start;
> suites < __kunit_suites_end;
> suites++) {
> diff --git a/lib/kunit/test.c b/lib/kunit/test.c
> index 9242f932896c7..da56b94261b43 100644
> --- a/lib/kunit/test.c
> +++ b/lib/kunit/test.c
> @@ -18,16 +18,6 @@ static void kunit_set_failure(struct kunit *test)
> WRITE_ONCE(test->success, false);
> }
>
> -static void kunit_print_tap_version(void)
> -{
> - static bool kunit_has_printed_tap_version;
> -
> - if (!kunit_has_printed_tap_version) {
> - pr_info("TAP version 14\n");
> - kunit_has_printed_tap_version = true;
> - }
> -}
> -
> static size_t kunit_test_cases_len(struct kunit_case *test_cases)
> {
> struct kunit_case *test_case;
> @@ -41,7 +31,6 @@ static size_t kunit_test_cases_len(struct kunit_case *test_cases)
>
> static void kunit_print_subtest_start(struct kunit_suite *suite)
> {
> - kunit_print_tap_version();
> pr_info("\t# Subtest: %s\n", suite->name);
> pr_info("\t1..%zd\n", kunit_test_cases_len(suite->test_cases));
> }
> diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
> index 4ffbae0f67325..78b3bdd03b1e4 100644
> --- a/tools/testing/kunit/kunit_parser.py
> +++ b/tools/testing/kunit/kunit_parser.py
> @@ -45,6 +45,7 @@ class TestStatus(Enum):
> FAILURE = auto()
> TEST_CRASHED = auto()
> NO_TESTS = auto()
> + FAILURE_TO_PARSE_TESTS = auto()
>
> kunit_start_re = re.compile(r'^TAP version [0-9]+$')
> kunit_end_re = re.compile('List of all partitions:')
> @@ -106,7 +107,7 @@ OkNotOkResult = namedtuple('OkNotOkResult', ['is_ok','description', 'text'])
>
> OK_NOT_OK_SUBTEST = re.compile(r'^\t(ok|not ok) [0-9]+ - (.*)$')
>
> -OK_NOT_OK_MODULE = re.compile(r'^(ok|not ok) [0-9]+ - (.*)$')
> +OK_NOT_OK_MODULE = re.compile(r'^(ok|not ok) ([0-9]+) - (.*)$')
>
> def parse_ok_not_ok_test_case(lines: List[str],
> test_case: TestCase,
> @@ -196,7 +197,9 @@ def max_status(left: TestStatus, right: TestStatus) -> TestStatus:
> else:
> return TestStatus.SUCCESS
>
> -def parse_ok_not_ok_test_suite(lines: List[str], test_suite: TestSuite) -> bool:
> +def parse_ok_not_ok_test_suite(lines: List[str],
> + test_suite: TestSuite,
> + expected_suite_index: int) -> bool:
> consume_non_diagnositic(lines)
> if not lines:
> test_suite.status = TestStatus.TEST_CRASHED
> @@ -209,6 +212,12 @@ def parse_ok_not_ok_test_suite(lines: List[str], test_suite: TestSuite) -> bool:
> test_suite.status = TestStatus.SUCCESS
> else:
> test_suite.status = TestStatus.FAILURE
> + suite_index = int(match.group(2))
> + if suite_index != expected_suite_index:
> + print_with_timestamp(
> + red('[ERROR] ') + 'expected_suite_index ' +
> + str(expected_suite_index) + ', but got ' +
> + str(suite_index))
> return True
> else:
> return False
> @@ -221,7 +230,7 @@ def bubble_up_test_case_errors(test_suite: TestSuite) -> TestStatus:
> max_test_case_status = bubble_up_errors(lambda x: x.status, test_suite.cases)
> return max_status(max_test_case_status, test_suite.status)
>
> -def parse_test_suite(lines: List[str]) -> TestSuite:
> +def parse_test_suite(lines: List[str], expected_suite_index: int) -> TestSuite:
> if not lines:
> return None
> consume_non_diagnositic(lines)
> @@ -240,7 +249,7 @@ def parse_test_suite(lines: List[str]) -> TestSuite:
> test_suite.cases.append(test_case)
> test_case = parse_test_case(lines, expected_test_case_num > 0)
> expected_test_case_num -= 1
> - if parse_ok_not_ok_test_suite(lines, test_suite):
> + if parse_ok_not_ok_test_suite(lines, test_suite, expected_suite_index):
> test_suite.status = bubble_up_test_case_errors(test_suite)
> return test_suite
> elif not lines:
> @@ -260,6 +269,17 @@ def parse_tap_header(lines: List[str]) -> bool:
> else:
> return False
>
> +TEST_PLAN = re.compile(r'[0-9]+\.\.([0-9]+)')
> +
> +def parse_test_plan(lines: List[str]) -> int:
> + consume_non_diagnositic(lines)
> + match = TEST_PLAN.match(lines[0])
> + if match:
> + lines.pop(0)
> + return int(match.group(1))
> + else:
> + return None
> +
> def bubble_up_suite_errors(test_suite_list: List[TestSuite]) -> TestStatus:
> return bubble_up_errors(lambda x: x.status, test_suite_list)
>
> @@ -268,19 +288,34 @@ def parse_test_result(lines: List[str]) -> TestResult:
> return TestResult(TestStatus.NO_TESTS, [], lines)
> consume_non_diagnositic(lines)
> if not parse_tap_header(lines):
> - return None
> + return TestResult(TestStatus.NO_TESTS, [], lines)
> + expected_test_suite_num = parse_test_plan(lines)
> + if not expected_test_suite_num:
> + return TestResult(TestStatus.FAILURE_TO_PARSE_TESTS, [], lines)
> test_suites = []
> - test_suite = parse_test_suite(lines)
> - while test_suite:
> - test_suites.append(test_suite)
> - test_suite = parse_test_suite(lines)
> - return TestResult(bubble_up_suite_errors(test_suites), test_suites, lines)
> + for i in range(1, expected_test_suite_num + 1):
> + test_suite = parse_test_suite(lines, i)
> + if test_suite:
> + test_suites.append(test_suite)
> + else:
> + print_with_timestamp(
> + red('[ERROR] ') + ' expected ' +
> + str(expected_test_suite_num) +
> + ' test suites, but got ' + str(i - 2))
> + break
> + test_suite = parse_test_suite(lines, -1)
> + if test_suite:
> + print_with_timestamp(red('[ERROR] ') +
> + 'got unexpected test suite: ' + test_suite.name)
> + if test_suites:
> + return TestResult(bubble_up_suite_errors(test_suites), test_suites, lines)
> + else:
> + return TestResult(TestStatus.NO_TESTS, [], lines)
>
> -def parse_run_tests(kernel_output) -> TestResult:
> +def print_and_count_results(test_result: TestResult) -> None:
> total_tests = 0
> failed_tests = 0
> crashed_tests = 0
> - test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
> for test_suite in test_result.suites:
> if test_suite.status == TestStatus.SUCCESS:
> print_suite_divider(green('[PASSED] ') + test_suite.name)
> @@ -302,6 +337,21 @@ def parse_run_tests(kernel_output) -> TestResult:
> print_with_timestamp(red('[FAILED] ') + test_case.name)
> print_log(map(yellow, test_case.log))
> print_with_timestamp('')
> + return total_tests, failed_tests, crashed_tests
> +
> +def parse_run_tests(kernel_output) -> TestResult:
> + total_tests = 0
> + failed_tests = 0
> + crashed_tests = 0
> + test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
> + if test_result.status == TestStatus.NO_TESTS:
> + print(red('[ERROR] ') + yellow('no tests run!'))
> + elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:
> + print(red('[ERROR] ') + yellow('could not parse test results!'))
> + else:
> + (total_tests,
> + failed_tests,
> + crashed_tests) = print_and_count_results(test_result)
> print_with_timestamp(DIVIDER)
> fmt = green if test_result.status == TestStatus.SUCCESS else red
> print_with_timestamp(
> diff --git a/tools/testing/kunit/test_data/test_is_test_passed-all_passed.log b/tools/testing/kunit/test_data/test_is_test_passed-all_passed.log
> index 62ebc0288355c..bc0dc8fe35b76 100644
> --- a/tools/testing/kunit/test_data/test_is_test_passed-all_passed.log
> +++ b/tools/testing/kunit/test_data/test_is_test_passed-all_passed.log
> @@ -1,4 +1,5 @@
> TAP version 14
> +1..2
> # Subtest: sysctl_test
> 1..8
> # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed
> diff --git a/tools/testing/kunit/test_data/test_is_test_passed-crash.log b/tools/testing/kunit/test_data/test_is_test_passed-crash.log
> index 0b249870c8be4..4d97f6708c4a5 100644
> --- a/tools/testing/kunit/test_data/test_is_test_passed-crash.log
> +++ b/tools/testing/kunit/test_data/test_is_test_passed-crash.log
> @@ -1,6 +1,7 @@
> printk: console [tty0] enabled
> printk: console [mc-1] enabled
> TAP version 14
> +1..2
> # Subtest: sysctl_test
> 1..8
> # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed
> diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure.log b/tools/testing/kunit/test_data/test_is_test_passed-failure.log
> index 9e89d32d5667a..7a416497e3bec 100644
> --- a/tools/testing/kunit/test_data/test_is_test_passed-failure.log
> +++ b/tools/testing/kunit/test_data/test_is_test_passed-failure.log
> @@ -1,4 +1,5 @@
> TAP version 14
> +1..2
> # Subtest: sysctl_test
> 1..8
> # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed
>
On 2/4/20 5:05 PM, Frank Rowand wrote:
> On 1/30/20 5:08 PM, Brendan Higgins wrote:
>> From: David Gow <[email protected]>
>>
>> Add a new kernel command-line option, 'kunit_shutdown', which allows the
>> user to specify that the kernel poweroff, halt, or reboot after
>> completing all KUnit tests; this is very handy for running KUnit tests
>> on UML or a VM so that the UML/VM process exits cleanly immediately
>> after running all tests without needing a special initramfs.
>
> kunit_shutdown needs to be added to Documentation/admin-guide/kernel-parameters.txt
Oops, patch 7/7 does exactly that.
-Frank
>
> -Frank
>
>>
>> Signed-off-by: David Gow <[email protected]>
>> Signed-off-by: Brendan Higgins <[email protected]>
>> Reviewed-by: Stephen Boyd <[email protected]>
>> ---
>> lib/kunit/executor.c | 20 ++++++++++++++++++++
>> tools/testing/kunit/kunit_kernel.py | 2 +-
>> tools/testing/kunit/kunit_parser.py | 2 +-
>> 3 files changed, 22 insertions(+), 2 deletions(-)
>>
>> diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
>> index 7fd16feff157e..a93821116ccec 100644
>> --- a/lib/kunit/executor.c
>> +++ b/lib/kunit/executor.c
>> @@ -1,5 +1,6 @@
>> // SPDX-License-Identifier: GPL-2.0
>>
>> +#include <linux/reboot.h>
>> #include <kunit/test.h>
>>
>> /*
>> @@ -11,6 +12,23 @@ extern struct kunit_suite * const * const __kunit_suites_end[];
>>
>> #if IS_BUILTIN(CONFIG_KUNIT)
>>
>> +static char *kunit_shutdown;
>> +core_param(kunit_shutdown, kunit_shutdown, charp, 0644);
>> +
>> +static void kunit_handle_shutdown(void)
>> +{
>> + if (!kunit_shutdown)
>> + return;
>> +
>> + if (!strcmp(kunit_shutdown, "poweroff"))
>> + kernel_power_off();
>> + else if (!strcmp(kunit_shutdown, "halt"))
>> + kernel_halt();
>> + else if (!strcmp(kunit_shutdown, "reboot"))
>> + kernel_restart(NULL);
>> +
>> +}
>> +
>> static void kunit_print_tap_header(void)
>> {
>> struct kunit_suite * const * const *suites, * const *subsuite;
>> @@ -42,6 +60,8 @@ int kunit_run_all_tests(void)
>> }
>> }
>>
>> + kunit_handle_shutdown();
>> +
>> if (has_test_failed)
>> return -EFAULT;
>>
>> diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
>> index cc5d844ecca13..43314aa537d30 100644
>> --- a/tools/testing/kunit/kunit_kernel.py
>> +++ b/tools/testing/kunit/kunit_kernel.py
>> @@ -141,7 +141,7 @@ class LinuxSourceTree(object):
>> return True
>>
>> def run_kernel(self, args=[], timeout=None, build_dir=''):
>> - args.extend(['mem=256M'])
>> + args.extend(['mem=256M', 'kunit_shutdown=halt'])
>> process = self._ops.linux_bin(args, timeout, build_dir)
>> with open(os.path.join(build_dir, 'test.log'), 'w') as f:
>> for line in process.stdout:
>> diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
>> index 78b3bdd03b1e4..633811dd9bce8 100644
>> --- a/tools/testing/kunit/kunit_parser.py
>> +++ b/tools/testing/kunit/kunit_parser.py
>> @@ -48,7 +48,7 @@ class TestStatus(Enum):
>> FAILURE_TO_PARSE_TESTS = auto()
>>
>> kunit_start_re = re.compile(r'^TAP version [0-9]+$')
>> -kunit_end_re = re.compile('List of all partitions:')
>> +kunit_end_re = re.compile(r'reboot: System halted')
>>
>> def isolate_kunit_output(kernel_output):
>> started = False
>>
>
>
On 1/30/20 5:08 PM, Brendan Higgins wrote:
> From: David Gow <[email protected]>
>
> Add a new kernel command-line option, 'kunit_shutdown', which allows the
> user to specify that the kernel poweroff, halt, or reboot after
> completing all KUnit tests; this is very handy for running KUnit tests
> on UML or a VM so that the UML/VM process exits cleanly immediately
> after running all tests without needing a special initramfs.
kunit_shutdown needs to be added to Documentation/admin-guide/kernel-parameters.txt
-Frank
>
> Signed-off-by: David Gow <[email protected]>
> Signed-off-by: Brendan Higgins <[email protected]>
> Reviewed-by: Stephen Boyd <[email protected]>
> ---
> lib/kunit/executor.c | 20 ++++++++++++++++++++
> tools/testing/kunit/kunit_kernel.py | 2 +-
> tools/testing/kunit/kunit_parser.py | 2 +-
> 3 files changed, 22 insertions(+), 2 deletions(-)
>
> diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
> index 7fd16feff157e..a93821116ccec 100644
> --- a/lib/kunit/executor.c
> +++ b/lib/kunit/executor.c
> @@ -1,5 +1,6 @@
> // SPDX-License-Identifier: GPL-2.0
>
> +#include <linux/reboot.h>
> #include <kunit/test.h>
>
> /*
> @@ -11,6 +12,23 @@ extern struct kunit_suite * const * const __kunit_suites_end[];
>
> #if IS_BUILTIN(CONFIG_KUNIT)
>
> +static char *kunit_shutdown;
> +core_param(kunit_shutdown, kunit_shutdown, charp, 0644);
> +
> +static void kunit_handle_shutdown(void)
> +{
> + if (!kunit_shutdown)
> + return;
> +
> + if (!strcmp(kunit_shutdown, "poweroff"))
> + kernel_power_off();
> + else if (!strcmp(kunit_shutdown, "halt"))
> + kernel_halt();
> + else if (!strcmp(kunit_shutdown, "reboot"))
> + kernel_restart(NULL);
> +
> +}
> +
> static void kunit_print_tap_header(void)
> {
> struct kunit_suite * const * const *suites, * const *subsuite;
> @@ -42,6 +60,8 @@ int kunit_run_all_tests(void)
> }
> }
>
> + kunit_handle_shutdown();
> +
> if (has_test_failed)
> return -EFAULT;
>
> diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
> index cc5d844ecca13..43314aa537d30 100644
> --- a/tools/testing/kunit/kunit_kernel.py
> +++ b/tools/testing/kunit/kunit_kernel.py
> @@ -141,7 +141,7 @@ class LinuxSourceTree(object):
> return True
>
> def run_kernel(self, args=[], timeout=None, build_dir=''):
> - args.extend(['mem=256M'])
> + args.extend(['mem=256M', 'kunit_shutdown=halt'])
> process = self._ops.linux_bin(args, timeout, build_dir)
> with open(os.path.join(build_dir, 'test.log'), 'w') as f:
> for line in process.stdout:
> diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
> index 78b3bdd03b1e4..633811dd9bce8 100644
> --- a/tools/testing/kunit/kunit_parser.py
> +++ b/tools/testing/kunit/kunit_parser.py
> @@ -48,7 +48,7 @@ class TestStatus(Enum):
> FAILURE_TO_PARSE_TESTS = auto()
>
> kunit_start_re = re.compile(r'^TAP version [0-9]+$')
> -kunit_end_re = re.compile('List of all partitions:')
> +kunit_end_re = re.compile(r'reboot: System halted')
>
> def isolate_kunit_output(kernel_output):
> started = False
>
On 2/4/20 1:35 PM, Brendan Higgins wrote:
> On Mon, Feb 3, 2020 at 11:19 PM Stephen Boyd <[email protected]> wrote:
>>
>> Quoting Brendan Higgins (2020-01-30 15:08:05)
>>> ## TL;DR
>>>
>>> This patchset adds a centralized executor to dispatch tests rather than
>>> relying on late_initcall to schedule each test suite separately along
>>> with a couple of new features that depend on it.
>>
>> Is there any diff from v1 to v2? I don't know what changed, but I see
>> that my Reviewed-by tag has been put on everything, so I guess
>> everything I said was addressed or discussed in the previous round.
>
> Oh yes, sorry about that. I have gotten a bit lazy in regard to
> changing logs. I noticed that a lot of people don't seem to care. I'll
> make a note that you do.
Please ignore those who don't care. Just always include a change log.
You may encounter bike shedding about where the log belongs (in patch 0,
in the modified patches, in both locations). The color of my bike shed
is simply that they exist somewhere, but my most favorite color is both
places.
>
> Changes since last revision:
> - On patch 6/7, I flipped the include order and removed braces from the if
> statements.
> - On patch 7/7, I removed the periods from the short descriptions.
>
On Tue, Feb 4, 2020 at 3:01 PM Frank Rowand <[email protected]> wrote:
>
> On 1/30/20 5:08 PM, Brendan Higgins wrote:
> > TAP 14 allows an optional test plan to be emitted before the start of
> > the start of testing[1]; this is valuable because it makes it possible
> > for a test harness to detect whether the number of tests run matches the
> > number of tests expected to be run, ensuring that no tests silently
> > failed.
> >
> > Link[1]: https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan
> > Signed-off-by: Brendan Higgins <[email protected]>
> > Reviewed-by: Stephen Boyd <[email protected]>
> > ---
> > lib/kunit/executor.c | 17 +++++
> > lib/kunit/test.c | 11 ---
> > tools/testing/kunit/kunit_parser.py | 74 ++++++++++++++++---
> > .../test_is_test_passed-all_passed.log | 1 +
> > .../test_data/test_is_test_passed-crash.log | 1 +
> > .../test_data/test_is_test_passed-failure.log | 1 +
> > 6 files changed, 82 insertions(+), 23 deletions(-)
> >
> > diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
> > index b75a46c560847..7fd16feff157e 100644
> > --- a/lib/kunit/executor.c
> > +++ b/lib/kunit/executor.c
> > @@ -11,11 +11,28 @@ extern struct kunit_suite * const * const __kunit_suites_end[];
> >
> > #if IS_BUILTIN(CONFIG_KUNIT)
> >
> > +static void kunit_print_tap_header(void)
> > +{
> > + struct kunit_suite * const * const *suites, * const *subsuite;
> > + int num_of_suites = 0;
> > +
> > + for (suites = __kunit_suites_start;
> > + suites < __kunit_suites_end;
> > + suites++)
> > + for (subsuite = *suites; *subsuite != NULL; subsuite++)
> > + num_of_suites++;
> > +
> > + pr_info("TAP version 14\n");
> > + pr_info("1..%d\n", num_of_suites);
> > +}
> > +
>
> Subsuites are not in Linux 5.5, as far as I can tell, so it is hard to review
> this.
>
> But I think this observation will still be valid even though I am not up
> to speed on subsuites.
"substitutes" are just an artifact from having to group together all
suites that occur in a module. It should be mostly immaterial to most
users, but we have to have all the suites that occur in a module
grouped together in order to support KUnit tests as loadable modules.
I just call them subsuites here because I didn't know what else to
call them. Still, the relationship between them should be mostly
ignored by users as you seem to recognize below.
Also, module support for KUnit was just recently accepted into torvalds/master:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=08a3ef8f6b0b1341c670caba35f782c9a452d488
You should be able to apply this change to torvalds/master now.
> num_of_suites is all test suites declared (as built in). In Linux 5.5,
> each kunit_suite contains one or more test cases. Each test case leads
> to either "ok" or "not ok". So instead of counting number of tests
> suites, the number of test cases needs to be counted.
Nope, but I think you understand correctly below...
> Ahhh, I think my lack of visibility of what tree this series is against
> is about to lead me astray! Are the test cases in the subsuite
> implemented as what TAP 14 calls "subtests"? If so, then the above
> code that counts suites instead of test cases would be correct.
Yep! You got it! Test cases are "subtests" in TAP 14. So there is a
ok/not ok line for each suite, and nested as subtests above that are
indented ok/not ok lines for each test case. KUnit already does this
today. We are just moving some of the printing logic out so we can
count the suites and print a test plan.
Technically, what we print today (before this change) is legal TAP 14.
The test plan is not required. However, the test plan is very helpful
as it makes it possible to know if some tests just never ran for some
reason.
> I'm spinning my wheels now, since I don't know what I am
> reviewing against, so I'm going to stop reviewing, other
> than maybe a quick scan.
I just applied this patch series against torvalds/master which I
pulled this morning, and it applied cleanly. I will specify in any
subsequent revisions that this is based on linux-kselftest/kunit since
that's where these patches will be applied, but they should apply
against any rc of 5.6 just fine once they are cut.
Cheers!