Implementation of support for parameterized testing in KUnit. This
approach requires the creation of a test case using the
KUNIT_CASE_PARAM() macro that accepts a generator function as input.
This generator function should return the next parameter given the
previous parameter in parameterized tests. It also provides a macro to
generate common-case generators based on arrays. Generators may also
optionally provide a human-readable description of parameters, which is
displayed where available.
Note, currently the result of each parameter run is displayed in
diagnostic lines, and only the overall test case output summarizes
TAP-compliant success or failure of all parameter runs. In future, when
supported by kunit-tool, these can be turned into subsubtest outputs.
Signed-off-by: Arpitha Raghunandan <[email protected]>
Co-developed-by: Marco Elver <[email protected]>
Signed-off-by: Marco Elver <[email protected]>
---
Changes v6->v7:
- Clarify commit message.
- Introduce ability to optionally generate descriptions for parameters;
if no description is provided, we'll still print 'param-N'.
- Change diagnostic line format to:
# <test-case-name>: <ok|not ok> N - [<param description>]
Changes v5->v6:
- Fix alignment to maintain consistency
Changes v4->v5:
- Update kernel-doc comments.
- Use const void* for generator return and prev value types.
- Add kernel-doc comment for KUNIT_ARRAY_PARAM.
- Rework parameterized test case execution strategy: each parameter is executed
as if it was its own test case, with its own test initialization and cleanup
(init and exit are called, etc.). However, we cannot add new test cases per TAP
protocol once we have already started execution. Instead, log the result of
each parameter run as a diagnostic comment.
Changes v3->v4:
- Rename kunit variables
- Rename generator function helper macro
- Add documentation for generator approach
- Display test case name in case of failure along with param index
Changes v2->v3:
- Modifictaion of generator macro and method
Changes v1->v2:
- Use of a generator method to access test case parameters
Changes v6->v7:
- Clarify commit message.
- Introduce ability to optionally generate descriptions for parameters;
if no description is provided, we'll still print 'param-N'.
- Change diagnostic line format to:
# <test-case-name>: <ok|not ok> N - [<param description>]
- Before execution of parameterized test case, count number of
parameters and display number of parameters. Currently also as a
diagnostic line, but this may be used in future to generate a subsubtest
plan. A requirement of this change is that generators must generate a
deterministic number of parameters.
Changes v5->v6:
- Fix alignment to maintain consistency
Changes v4->v5:
- Update kernel-doc comments.
- Use const void* for generator return and prev value types.
- Add kernel-doc comment for KUNIT_ARRAY_PARAM.
- Rework parameterized test case execution strategy: each parameter is executed
as if it was its own test case, with its own test initialization and cleanup
(init and exit are called, etc.). However, we cannot add new test cases per TAP
protocol once we have already started execution. Instead, log the result of
each parameter run as a diagnostic comment.
Changes v3->v4:
- Rename kunit variables
- Rename generator function helper macro
- Add documentation for generator approach
- Display test case name in case of failure along with param index
Changes v2->v3:
- Modifictaion of generator macro and method
Changes v1->v2:
- Use of a generator method to access test case parameters
include/kunit/test.h | 51 ++++++++++++++++++++++++++++++++++++++
lib/kunit/test.c | 59 ++++++++++++++++++++++++++++++++++----------
2 files changed, 97 insertions(+), 13 deletions(-)
diff --git a/include/kunit/test.h b/include/kunit/test.h
index db1b0ae666c4..cf5f33b1c890 100644
--- a/include/kunit/test.h
+++ b/include/kunit/test.h
@@ -94,6 +94,9 @@ struct kunit;
/* Size of log associated with test. */
#define KUNIT_LOG_SIZE 512
+/* Maximum size of parameter description string. */
+#define KUNIT_PARAM_DESC_SIZE 64
+
/*
* TAP specifies subtest stream indentation of 4 spaces, 8 spaces for a
* sub-subtest. See the "Subtests" section in
@@ -107,6 +110,7 @@ struct kunit;
*
* @run_case: the function representing the actual test case.
* @name: the name of the test case.
+ * @generate_params: the generator function for parameterized tests.
*
* A test case is a function with the signature,
* ``void (*)(struct kunit *)``
@@ -141,6 +145,7 @@ struct kunit;
struct kunit_case {
void (*run_case)(struct kunit *test);
const char *name;
+ const void* (*generate_params)(const void *prev, char *desc);
/* private: internal use only. */
bool success;
@@ -163,6 +168,27 @@ static inline char *kunit_status_to_string(bool status)
*/
#define KUNIT_CASE(test_name) { .run_case = test_name, .name = #test_name }
+/**
+ * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case
+ *
+ * @test_name: a reference to a test case function.
+ * @gen_params: a reference to a parameter generator function.
+ *
+ * The generator function::
+ *
+ * const void* gen_params(const void *prev, char *desc)
+ *
+ * is used to lazily generate a series of arbitrarily typed values that fit into
+ * a void*. The argument @prev is the previously returned value, which should be
+ * used to derive the next value; @prev is set to NULL on the initial generator
+ * call. When no more values are available, the generator must return NULL.
+ * Optionally write a string into @desc (size of KUNIT_PARAM_DESC_SIZE)
+ * describing the parameter.
+ */
+#define KUNIT_CASE_PARAM(test_name, gen_params) \
+ { .run_case = test_name, .name = #test_name, \
+ .generate_params = gen_params }
+
/**
* struct kunit_suite - describes a related collection of &struct kunit_case
*
@@ -208,6 +234,10 @@ struct kunit {
const char *name; /* Read only after initialization! */
char *log; /* Points at case log after initialization */
struct kunit_try_catch try_catch;
+ /* param_value is the current parameter value for a test case. */
+ const void *param_value;
+ /* param_index stores the index of the parameter in parameterized tests. */
+ int param_index;
/*
* success starts as true, and may only be set to false during a
* test case; thus, it is safe to update this across multiple
@@ -1742,4 +1772,25 @@ do { \
fmt, \
##__VA_ARGS__)
+/**
+ * KUNIT_ARRAY_PARAM() - Define test parameter generator from an array.
+ * @name: prefix for the test parameter generator function.
+ * @array: array of test parameters.
+ * @get_desc: function to convert param to description; NULL to use default
+ *
+ * Define function @name_gen_params which uses @array to generate parameters.
+ */
+#define KUNIT_ARRAY_PARAM(name, array, get_desc) \
+ static const void *name##_gen_params(const void *prev, char *desc) \
+ { \
+ typeof((array)[0]) * __next = prev ? ((typeof(__next)) prev) + 1 : (array); \
+ if (__next - (array) < ARRAY_SIZE((array))) { \
+ void (*__get_desc)(typeof(__next), char *) = get_desc; \
+ if (__get_desc) \
+ __get_desc(__next, desc); \
+ return __next; \
+ } \
+ return NULL; \
+ }
+
#endif /* _KUNIT_TEST_H */
diff --git a/lib/kunit/test.c b/lib/kunit/test.c
index 750704abe89a..ec9494e914ef 100644
--- a/lib/kunit/test.c
+++ b/lib/kunit/test.c
@@ -325,39 +325,72 @@ static void kunit_catch_run_case(void *data)
* occur in a test case and reports them as failures.
*/
static void kunit_run_case_catch_errors(struct kunit_suite *suite,
- struct kunit_case *test_case)
+ struct kunit_case *test_case,
+ struct kunit *test)
{
struct kunit_try_catch_context context;
struct kunit_try_catch *try_catch;
- struct kunit test;
- kunit_init_test(&test, test_case->name, test_case->log);
- try_catch = &test.try_catch;
+ kunit_init_test(test, test_case->name, test_case->log);
+ try_catch = &test->try_catch;
kunit_try_catch_init(try_catch,
- &test,
+ test,
kunit_try_run_case,
kunit_catch_run_case);
- context.test = &test;
+ context.test = test;
context.suite = suite;
context.test_case = test_case;
kunit_try_catch_run(try_catch, &context);
- test_case->success = test.success;
-
- kunit_print_ok_not_ok(&test, true, test_case->success,
- kunit_test_case_num(suite, test_case),
- test_case->name);
+ test_case->success = test->success;
}
int kunit_run_tests(struct kunit_suite *suite)
{
+ char param_desc[KUNIT_PARAM_DESC_SIZE];
struct kunit_case *test_case;
kunit_print_subtest_start(suite);
- kunit_suite_for_each_test_case(suite, test_case)
- kunit_run_case_catch_errors(suite, test_case);
+ kunit_suite_for_each_test_case(suite, test_case) {
+ struct kunit test = { .param_value = NULL, .param_index = 0 };
+ bool test_success = true;
+
+ if (test_case->generate_params) {
+ /* Get initial param. */
+ param_desc[0] = '\0';
+ test.param_value = test_case->generate_params(NULL, param_desc);
+ }
+
+ do {
+ kunit_run_case_catch_errors(suite, test_case, &test);
+ test_success &= test_case->success;
+
+ if (test_case->generate_params) {
+ if (param_desc[0] == '\0') {
+ snprintf(param_desc, sizeof(param_desc),
+ "param-%d", test.param_index);
+ }
+
+ kunit_log(KERN_INFO, &test,
+ KUNIT_SUBTEST_INDENT
+ "# %s: %s %d - %s",
+ test_case->name,
+ kunit_status_to_string(test.success),
+ test.param_index + 1, param_desc);
+
+ /* Get next param. */
+ param_desc[0] = '\0';
+ test.param_value = test_case->generate_params(test.param_value, param_desc);
+ test.param_index++;
+ }
+ } while (test.param_value);
+
+ kunit_print_ok_not_ok(&test, true, test_success,
+ kunit_test_case_num(suite, test_case),
+ test_case->name);
+ }
kunit_print_subtest_end(suite);
--
2.25.1
Modify fs/ext4/inode-test.c to use the parameterized testing
feature of KUnit.
Signed-off-by: Arpitha Raghunandan <[email protected]>
Signed-off-by: Marco Elver <[email protected]>
---
Changes v6->v7:
- Introduce timestamp_expectation_to_desc() to convert param to
description.
Changes v5->v6:
- No change to this patch of the patch series
Changes v4->v5:
- No change to this patch of the patch series
Changes v3->v4:
- Modification based on latest implementation of KUnit parameterized testing
Changes v2->v3:
- Marked hardcoded test data const
- Modification based on latest implementation of KUnit parameterized testing
Changes v1->v2:
- Modification based on latest implementation of KUnit parameterized testing
fs/ext4/inode-test.c | 320 ++++++++++++++++++++++---------------------
1 file changed, 164 insertions(+), 156 deletions(-)
diff --git a/fs/ext4/inode-test.c b/fs/ext4/inode-test.c
index d62d802c9c12..97390ab13681 100644
--- a/fs/ext4/inode-test.c
+++ b/fs/ext4/inode-test.c
@@ -80,6 +80,145 @@ struct timestamp_expectation {
bool lower_bound;
};
+static const struct timestamp_expectation test_data[] = {
+ {
+ .test_case_name = LOWER_BOUND_NEG_NO_EXTRA_BITS_CASE,
+ .msb_set = true,
+ .lower_bound = true,
+ .extra_bits = 0,
+ .expected = {.tv_sec = -0x80000000LL, .tv_nsec = 0L},
+ },
+
+ {
+ .test_case_name = UPPER_BOUND_NEG_NO_EXTRA_BITS_CASE,
+ .msb_set = true,
+ .lower_bound = false,
+ .extra_bits = 0,
+ .expected = {.tv_sec = -1LL, .tv_nsec = 0L},
+ },
+
+ {
+ .test_case_name = LOWER_BOUND_NONNEG_NO_EXTRA_BITS_CASE,
+ .msb_set = false,
+ .lower_bound = true,
+ .extra_bits = 0,
+ .expected = {0LL, 0L},
+ },
+
+ {
+ .test_case_name = UPPER_BOUND_NONNEG_NO_EXTRA_BITS_CASE,
+ .msb_set = false,
+ .lower_bound = false,
+ .extra_bits = 0,
+ .expected = {.tv_sec = 0x7fffffffLL, .tv_nsec = 0L},
+ },
+
+ {
+ .test_case_name = LOWER_BOUND_NEG_LO_1_CASE,
+ .msb_set = true,
+ .lower_bound = true,
+ .extra_bits = 1,
+ .expected = {.tv_sec = 0x80000000LL, .tv_nsec = 0L},
+ },
+
+ {
+ .test_case_name = UPPER_BOUND_NEG_LO_1_CASE,
+ .msb_set = true,
+ .lower_bound = false,
+ .extra_bits = 1,
+ .expected = {.tv_sec = 0xffffffffLL, .tv_nsec = 0L},
+ },
+
+ {
+ .test_case_name = LOWER_BOUND_NONNEG_LO_1_CASE,
+ .msb_set = false,
+ .lower_bound = true,
+ .extra_bits = 1,
+ .expected = {.tv_sec = 0x100000000LL, .tv_nsec = 0L},
+ },
+
+ {
+ .test_case_name = UPPER_BOUND_NONNEG_LO_1_CASE,
+ .msb_set = false,
+ .lower_bound = false,
+ .extra_bits = 1,
+ .expected = {.tv_sec = 0x17fffffffLL, .tv_nsec = 0L},
+ },
+
+ {
+ .test_case_name = LOWER_BOUND_NEG_HI_1_CASE,
+ .msb_set = true,
+ .lower_bound = true,
+ .extra_bits = 2,
+ .expected = {.tv_sec = 0x180000000LL, .tv_nsec = 0L},
+ },
+
+ {
+ .test_case_name = UPPER_BOUND_NEG_HI_1_CASE,
+ .msb_set = true,
+ .lower_bound = false,
+ .extra_bits = 2,
+ .expected = {.tv_sec = 0x1ffffffffLL, .tv_nsec = 0L},
+ },
+
+ {
+ .test_case_name = LOWER_BOUND_NONNEG_HI_1_CASE,
+ .msb_set = false,
+ .lower_bound = true,
+ .extra_bits = 2,
+ .expected = {.tv_sec = 0x200000000LL, .tv_nsec = 0L},
+ },
+
+ {
+ .test_case_name = UPPER_BOUND_NONNEG_HI_1_CASE,
+ .msb_set = false,
+ .lower_bound = false,
+ .extra_bits = 2,
+ .expected = {.tv_sec = 0x27fffffffLL, .tv_nsec = 0L},
+ },
+
+ {
+ .test_case_name = UPPER_BOUND_NONNEG_HI_1_NS_1_CASE,
+ .msb_set = false,
+ .lower_bound = false,
+ .extra_bits = 6,
+ .expected = {.tv_sec = 0x27fffffffLL, .tv_nsec = 1L},
+ },
+
+ {
+ .test_case_name = LOWER_BOUND_NONNEG_HI_1_NS_MAX_CASE,
+ .msb_set = false,
+ .lower_bound = true,
+ .extra_bits = 0xFFFFFFFF,
+ .expected = {.tv_sec = 0x300000000LL,
+ .tv_nsec = MAX_NANOSECONDS},
+ },
+
+ {
+ .test_case_name = LOWER_BOUND_NONNEG_EXTRA_BITS_1_CASE,
+ .msb_set = false,
+ .lower_bound = true,
+ .extra_bits = 3,
+ .expected = {.tv_sec = 0x300000000LL, .tv_nsec = 0L},
+ },
+
+ {
+ .test_case_name = UPPER_BOUND_NONNEG_EXTRA_BITS_1_CASE,
+ .msb_set = false,
+ .lower_bound = false,
+ .extra_bits = 3,
+ .expected = {.tv_sec = 0x37fffffffLL, .tv_nsec = 0L},
+ }
+};
+
+static void timestamp_expectation_to_desc(const struct timestamp_expectation *t,
+ char *desc)
+{
+ strcpy(desc, t->test_case_name);
+}
+
+KUNIT_ARRAY_PARAM(ext4_inode, test_data, timestamp_expectation_to_desc);
+
static time64_t get_32bit_time(const struct timestamp_expectation * const test)
{
if (test->msb_set) {
@@ -101,166 +240,35 @@ static time64_t get_32bit_time(const struct timestamp_expectation * const test)
*/
static void inode_test_xtimestamp_decoding(struct kunit *test)
{
- const struct timestamp_expectation test_data[] = {
- {
- .test_case_name = LOWER_BOUND_NEG_NO_EXTRA_BITS_CASE,
- .msb_set = true,
- .lower_bound = true,
- .extra_bits = 0,
- .expected = {.tv_sec = -0x80000000LL, .tv_nsec = 0L},
- },
-
- {
- .test_case_name = UPPER_BOUND_NEG_NO_EXTRA_BITS_CASE,
- .msb_set = true,
- .lower_bound = false,
- .extra_bits = 0,
- .expected = {.tv_sec = -1LL, .tv_nsec = 0L},
- },
-
- {
- .test_case_name = LOWER_BOUND_NONNEG_NO_EXTRA_BITS_CASE,
- .msb_set = false,
- .lower_bound = true,
- .extra_bits = 0,
- .expected = {0LL, 0L},
- },
-
- {
- .test_case_name = UPPER_BOUND_NONNEG_NO_EXTRA_BITS_CASE,
- .msb_set = false,
- .lower_bound = false,
- .extra_bits = 0,
- .expected = {.tv_sec = 0x7fffffffLL, .tv_nsec = 0L},
- },
-
- {
- .test_case_name = LOWER_BOUND_NEG_LO_1_CASE,
- .msb_set = true,
- .lower_bound = true,
- .extra_bits = 1,
- .expected = {.tv_sec = 0x80000000LL, .tv_nsec = 0L},
- },
-
- {
- .test_case_name = UPPER_BOUND_NEG_LO_1_CASE,
- .msb_set = true,
- .lower_bound = false,
- .extra_bits = 1,
- .expected = {.tv_sec = 0xffffffffLL, .tv_nsec = 0L},
- },
-
- {
- .test_case_name = LOWER_BOUND_NONNEG_LO_1_CASE,
- .msb_set = false,
- .lower_bound = true,
- .extra_bits = 1,
- .expected = {.tv_sec = 0x100000000LL, .tv_nsec = 0L},
- },
-
- {
- .test_case_name = UPPER_BOUND_NONNEG_LO_1_CASE,
- .msb_set = false,
- .lower_bound = false,
- .extra_bits = 1,
- .expected = {.tv_sec = 0x17fffffffLL, .tv_nsec = 0L},
- },
-
- {
- .test_case_name = LOWER_BOUND_NEG_HI_1_CASE,
- .msb_set = true,
- .lower_bound = true,
- .extra_bits = 2,
- .expected = {.tv_sec = 0x180000000LL, .tv_nsec = 0L},
- },
-
- {
- .test_case_name = UPPER_BOUND_NEG_HI_1_CASE,
- .msb_set = true,
- .lower_bound = false,
- .extra_bits = 2,
- .expected = {.tv_sec = 0x1ffffffffLL, .tv_nsec = 0L},
- },
-
- {
- .test_case_name = LOWER_BOUND_NONNEG_HI_1_CASE,
- .msb_set = false,
- .lower_bound = true,
- .extra_bits = 2,
- .expected = {.tv_sec = 0x200000000LL, .tv_nsec = 0L},
- },
-
- {
- .test_case_name = UPPER_BOUND_NONNEG_HI_1_CASE,
- .msb_set = false,
- .lower_bound = false,
- .extra_bits = 2,
- .expected = {.tv_sec = 0x27fffffffLL, .tv_nsec = 0L},
- },
-
- {
- .test_case_name = UPPER_BOUND_NONNEG_HI_1_NS_1_CASE,
- .msb_set = false,
- .lower_bound = false,
- .extra_bits = 6,
- .expected = {.tv_sec = 0x27fffffffLL, .tv_nsec = 1L},
- },
-
- {
- .test_case_name = LOWER_BOUND_NONNEG_HI_1_NS_MAX_CASE,
- .msb_set = false,
- .lower_bound = true,
- .extra_bits = 0xFFFFFFFF,
- .expected = {.tv_sec = 0x300000000LL,
- .tv_nsec = MAX_NANOSECONDS},
- },
-
- {
- .test_case_name = LOWER_BOUND_NONNEG_EXTRA_BITS_1_CASE,
- .msb_set = false,
- .lower_bound = true,
- .extra_bits = 3,
- .expected = {.tv_sec = 0x300000000LL, .tv_nsec = 0L},
- },
-
- {
- .test_case_name = UPPER_BOUND_NONNEG_EXTRA_BITS_1_CASE,
- .msb_set = false,
- .lower_bound = false,
- .extra_bits = 3,
- .expected = {.tv_sec = 0x37fffffffLL, .tv_nsec = 0L},
- }
- };
-
struct timespec64 timestamp;
- int i;
-
- for (i = 0; i < ARRAY_SIZE(test_data); ++i) {
- timestamp.tv_sec = get_32bit_time(&test_data[i]);
- ext4_decode_extra_time(×tamp,
- cpu_to_le32(test_data[i].extra_bits));
-
- KUNIT_EXPECT_EQ_MSG(test,
- test_data[i].expected.tv_sec,
- timestamp.tv_sec,
- CASE_NAME_FORMAT,
- test_data[i].test_case_name,
- test_data[i].msb_set,
- test_data[i].lower_bound,
- test_data[i].extra_bits);
- KUNIT_EXPECT_EQ_MSG(test,
- test_data[i].expected.tv_nsec,
- timestamp.tv_nsec,
- CASE_NAME_FORMAT,
- test_data[i].test_case_name,
- test_data[i].msb_set,
- test_data[i].lower_bound,
- test_data[i].extra_bits);
- }
+
+ struct timestamp_expectation *test_param =
+ (struct timestamp_expectation *)(test->param_value);
+
+ timestamp.tv_sec = get_32bit_time(test_param);
+ ext4_decode_extra_time(×tamp,
+ cpu_to_le32(test_param->extra_bits));
+
+ KUNIT_EXPECT_EQ_MSG(test,
+ test_param->expected.tv_sec,
+ timestamp.tv_sec,
+ CASE_NAME_FORMAT,
+ test_param->test_case_name,
+ test_param->msb_set,
+ test_param->lower_bound,
+ test_param->extra_bits);
+ KUNIT_EXPECT_EQ_MSG(test,
+ test_param->expected.tv_nsec,
+ timestamp.tv_nsec,
+ CASE_NAME_FORMAT,
+ test_param->test_case_name,
+ test_param->msb_set,
+ test_param->lower_bound,
+ test_param->extra_bits);
}
static struct kunit_case ext4_inode_test_cases[] = {
- KUNIT_CASE(inode_test_xtimestamp_decoding),
+ KUNIT_CASE_PARAM(inode_test_xtimestamp_decoding, ext4_inode_gen_params),
{}
};
--
2.25.1
On Sat, 14 Nov 2020 at 13:38, Arpitha Raghunandan <[email protected]> wrote:
> Implementation of support for parameterized testing in KUnit. This
> approach requires the creation of a test case using the
> KUNIT_CASE_PARAM() macro that accepts a generator function as input.
>
> This generator function should return the next parameter given the
> previous parameter in parameterized tests. It also provides a macro to
> generate common-case generators based on arrays. Generators may also
> optionally provide a human-readable description of parameters, which is
> displayed where available.
>
> Note, currently the result of each parameter run is displayed in
> diagnostic lines, and only the overall test case output summarizes
> TAP-compliant success or failure of all parameter runs. In future, when
> supported by kunit-tool, these can be turned into subsubtest outputs.
>
> Signed-off-by: Arpitha Raghunandan <[email protected]>
> Co-developed-by: Marco Elver <[email protected]>
> Signed-off-by: Marco Elver <[email protected]>
> ---
> Changes v6->v7:
> - Clarify commit message.
> - Introduce ability to optionally generate descriptions for parameters;
> if no description is provided, we'll still print 'param-N'.
> - Change diagnostic line format to:
> # <test-case-name>: <ok|not ok> N - [<param description>]
>
> Changes v5->v6:
> - Fix alignment to maintain consistency
>
> Changes v4->v5:
> - Update kernel-doc comments.
> - Use const void* for generator return and prev value types.
> - Add kernel-doc comment for KUNIT_ARRAY_PARAM.
> - Rework parameterized test case execution strategy: each parameter is executed
> as if it was its own test case, with its own test initialization and cleanup
> (init and exit are called, etc.). However, we cannot add new test cases per TAP
> protocol once we have already started execution. Instead, log the result of
> each parameter run as a diagnostic comment.
>
> Changes v3->v4:
> - Rename kunit variables
> - Rename generator function helper macro
> - Add documentation for generator approach
> - Display test case name in case of failure along with param index
>
> Changes v2->v3:
> - Modifictaion of generator macro and method
>
> Changes v1->v2:
> - Use of a generator method to access test case parameters
> Changes v6->v7:
> - Clarify commit message.
> - Introduce ability to optionally generate descriptions for parameters;
> if no description is provided, we'll still print 'param-N'.
> - Change diagnostic line format to:
> # <test-case-name>: <ok|not ok> N - [<param description>]
> - Before execution of parameterized test case, count number of
> parameters and display number of parameters. Currently also as a
> diagnostic line, but this may be used in future to generate a subsubtest
> plan. A requirement of this change is that generators must generate a
> deterministic number of parameters.
>
> Changes v5->v6:
> - Fix alignment to maintain consistency
>
> Changes v4->v5:
> - Update kernel-doc comments.
> - Use const void* for generator return and prev value types.
> - Add kernel-doc comment for KUNIT_ARRAY_PARAM.
> - Rework parameterized test case execution strategy: each parameter is executed
> as if it was its own test case, with its own test initialization and cleanup
> (init and exit are called, etc.). However, we cannot add new test cases per TAP
> protocol once we have already started execution. Instead, log the result of
> each parameter run as a diagnostic comment.
>
> Changes v3->v4:
> - Rename kunit variables
> - Rename generator function helper macro
> - Add documentation for generator approach
> - Display test case name in case of failure along with param index
>
> Changes v2->v3:
> - Modifictaion of generator macro and method
>
> Changes v1->v2:
> - Use of a generator method to access test case parameters
>
> include/kunit/test.h | 51 ++++++++++++++++++++++++++++++++++++++
> lib/kunit/test.c | 59 ++++++++++++++++++++++++++++++++++----------
> 2 files changed, 97 insertions(+), 13 deletions(-)
>
> diff --git a/include/kunit/test.h b/include/kunit/test.h
> index db1b0ae666c4..cf5f33b1c890 100644
> --- a/include/kunit/test.h
> +++ b/include/kunit/test.h
> @@ -94,6 +94,9 @@ struct kunit;
> /* Size of log associated with test. */
> #define KUNIT_LOG_SIZE 512
>
> +/* Maximum size of parameter description string. */
> +#define KUNIT_PARAM_DESC_SIZE 64
I think we need to make this larger, perhaps 128. I just noticed a few
of the inode-test strings are >64 chars (and it should probably also
use strncpy() to copy to description, which is my bad).
> /*
> * TAP specifies subtest stream indentation of 4 spaces, 8 spaces for a
> * sub-subtest. See the "Subtests" section in
> @@ -107,6 +110,7 @@ struct kunit;
[...]
> +/**
> + * KUNIT_ARRAY_PARAM() - Define test parameter generator from an array.
> + * @name: prefix for the test parameter generator function.
> + * @array: array of test parameters.
> + * @get_desc: function to convert param to description; NULL to use default
> + *
> + * Define function @name_gen_params which uses @array to generate parameters.
> + */
> +#define KUNIT_ARRAY_PARAM(name, array, get_desc) \
> + static const void *name##_gen_params(const void *prev, char *desc) \
> + { \
> + typeof((array)[0]) * __next = prev ? ((typeof(__next)) prev) + 1 : (array); \
Why did you reintroduce a space between * and __next? AFAIK, this
should follow the same style as the rest of the kernel, and it should
just be 'thetype *ptr'.
> + if (__next - (array) < ARRAY_SIZE((array))) { \
> + void (*__get_desc)(typeof(__next), char *) = get_desc; \
> + if (__get_desc) \
> + __get_desc(__next, desc); \
> + return __next; \
> + } \
> + return NULL; \
> + }
> +
Thanks,
-- Marco
Greeting,
FYI, we noticed the following commit (built with gcc-9):
commit: 4ffef91a3b1b35b64502da85fe682897e4a123cd ("[PATCH v7 2/2] fs: ext4: Modify inode-test.c to use KUnit parameterized testing feature")
url: https://github.com/0day-ci/linux/commits/Arpitha-Raghunandan/kunit-Support-for-Parameterized-Testing/20201114-204322
base: https://git.kernel.org/cgit/linux/kernel/git/tytso/ext4.git dev
in testcase: boot
on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
+---------------------------------------------------------------------------------------+------------+------------+
| | 0f0922dc1c | 4ffef91a3b |
+---------------------------------------------------------------------------------------+------------+------------+
| Kernel_panic-not_syncing:stack-protector:Kernel_stack_is_corrupted_in:kunit_run_tests | 0 | 16 |
+---------------------------------------------------------------------------------------+------------+------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>
[ 21.894954] # inode_test_xtimestamp_decoding: ok 14 - 2378-04-22 Lower bound of 32bit>= timestamp. Extra sec bits 1. Max ns
[ 21.896398] # inode_test_xtimestamp_decoding: ok 15 - 2378-04-22 Lower bound of 32bit >=0 timestamp. All extra sec bits on
[ 21.898125] # inode_test_xtimestamp_decoding: ok 16 - 2446-05-10 Upper bound of 32bit >=0 timestamp. All extra sec bits on
[ 21.899303] ok 1 - inode_test_xtimestamp_decoding
[ 21.900451] ok 2 - ext4_inode_test
[ 21.901545] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: kunit_run_tests+0x301/0x315
[ 21.905219] CPU: 0 PID: 1 Comm: swapper Not tainted 5.10.0-rc3-00098-g4ffef91a3b1b #7
[ 21.906045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
[ 21.906871] Call Trace:
[ 21.907143] dump_stack+0x20/0x22
[ 21.907541] panic+0x101/0x2d1
[ 21.907868] ? kunit_run_tests+0x301/0x315
[ 21.908281] __stack_chk_fail+0x14/0x20
[ 21.908646] kunit_run_tests+0x301/0x315
[ 21.909026] ? kunit_catch_run_case+0x40/0x40
[ 21.909476] ? kunit_cleanup+0x60/0x60
[ 21.909912] ? kunit_cleanup+0x41/0x60
[ 21.910316] __kunit_test_suites_init+0x27/0x50
[ 21.910797] kunit_run_all_tests.cold+0x37/0x40
[ 21.911283] kernel_init_freeable+0x8b/0xb3
[ 21.911736] ? rest_init+0x233/0x233
[ 21.912115] kernel_init+0x9/0xfd
[ 21.912464] ret_from_fork+0x22/0x30
[ 21.912893] Kernel Offset: disabled
Kboot worker: lkp-worker53
Elapsed time: 60
To reproduce:
# build kernel
cd linux
cp config-5.10.0-rc3-00098-g4ffef91a3b1b .config
make HOSTCC=gcc-9 CC=gcc-9 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email
Thanks,
Oliver Sang
On 15/11/20 2:28 pm, Marco Elver wrote:
> On Sat, 14 Nov 2020 at 13:38, Arpitha Raghunandan <[email protected]> wrote:
>> Implementation of support for parameterized testing in KUnit. This
>> approach requires the creation of a test case using the
>> KUNIT_CASE_PARAM() macro that accepts a generator function as input.
>>
>> This generator function should return the next parameter given the
>> previous parameter in parameterized tests. It also provides a macro to
>> generate common-case generators based on arrays. Generators may also
>> optionally provide a human-readable description of parameters, which is
>> displayed where available.
>>
>> Note, currently the result of each parameter run is displayed in
>> diagnostic lines, and only the overall test case output summarizes
>> TAP-compliant success or failure of all parameter runs. In future, when
>> supported by kunit-tool, these can be turned into subsubtest outputs.
>>
>> Signed-off-by: Arpitha Raghunandan <[email protected]>
>> Co-developed-by: Marco Elver <[email protected]>
>> Signed-off-by: Marco Elver <[email protected]>
>> ---
>> Changes v6->v7:
>> - Clarify commit message.
>> - Introduce ability to optionally generate descriptions for parameters;
>> if no description is provided, we'll still print 'param-N'.
>> - Change diagnostic line format to:
>> # <test-case-name>: <ok|not ok> N - [<param description>]
>>
>> Changes v5->v6:
>> - Fix alignment to maintain consistency
>>
>> Changes v4->v5:
>> - Update kernel-doc comments.
>> - Use const void* for generator return and prev value types.
>> - Add kernel-doc comment for KUNIT_ARRAY_PARAM.
>> - Rework parameterized test case execution strategy: each parameter is executed
>> as if it was its own test case, with its own test initialization and cleanup
>> (init and exit are called, etc.). However, we cannot add new test cases per TAP
>> protocol once we have already started execution. Instead, log the result of
>> each parameter run as a diagnostic comment.
>>
>> Changes v3->v4:
>> - Rename kunit variables
>> - Rename generator function helper macro
>> - Add documentation for generator approach
>> - Display test case name in case of failure along with param index
>>
>> Changes v2->v3:
>> - Modifictaion of generator macro and method
>>
>> Changes v1->v2:
>> - Use of a generator method to access test case parameters
>> Changes v6->v7:
>> - Clarify commit message.
>> - Introduce ability to optionally generate descriptions for parameters;
>> if no description is provided, we'll still print 'param-N'.
>> - Change diagnostic line format to:
>> # <test-case-name>: <ok|not ok> N - [<param description>]
>> - Before execution of parameterized test case, count number of
>> parameters and display number of parameters. Currently also as a
>> diagnostic line, but this may be used in future to generate a subsubtest
>> plan. A requirement of this change is that generators must generate a
>> deterministic number of parameters.
>>
>> Changes v5->v6:
>> - Fix alignment to maintain consistency
>>
>> Changes v4->v5:
>> - Update kernel-doc comments.
>> - Use const void* for generator return and prev value types.
>> - Add kernel-doc comment for KUNIT_ARRAY_PARAM.
>> - Rework parameterized test case execution strategy: each parameter is executed
>> as if it was its own test case, with its own test initialization and cleanup
>> (init and exit are called, etc.). However, we cannot add new test cases per TAP
>> protocol once we have already started execution. Instead, log the result of
>> each parameter run as a diagnostic comment.
>>
>> Changes v3->v4:
>> - Rename kunit variables
>> - Rename generator function helper macro
>> - Add documentation for generator approach
>> - Display test case name in case of failure along with param index
>>
>> Changes v2->v3:
>> - Modifictaion of generator macro and method
>>
>> Changes v1->v2:
>> - Use of a generator method to access test case parameters
>>
>> include/kunit/test.h | 51 ++++++++++++++++++++++++++++++++++++++
>> lib/kunit/test.c | 59 ++++++++++++++++++++++++++++++++++----------
>> 2 files changed, 97 insertions(+), 13 deletions(-)
>>
>> diff --git a/include/kunit/test.h b/include/kunit/test.h
>> index db1b0ae666c4..cf5f33b1c890 100644
>> --- a/include/kunit/test.h
>> +++ b/include/kunit/test.h
>> @@ -94,6 +94,9 @@ struct kunit;
>> /* Size of log associated with test. */
>> #define KUNIT_LOG_SIZE 512
>>
>> +/* Maximum size of parameter description string. */
>> +#define KUNIT_PARAM_DESC_SIZE 64
>
> I think we need to make this larger, perhaps 128. I just noticed a few
> of the inode-test strings are >64 chars (and it should probably also
> use strncpy() to copy to description, which is my bad).
>
Okay, I will make the description size larger and use strncpy().
>> /*
>> * TAP specifies subtest stream indentation of 4 spaces, 8 spaces for a
>> * sub-subtest. See the "Subtests" section in
>> @@ -107,6 +110,7 @@ struct kunit;
> [...]
>> +/**
>> + * KUNIT_ARRAY_PARAM() - Define test parameter generator from an array.
>> + * @name: prefix for the test parameter generator function.
>> + * @array: array of test parameters.
>> + * @get_desc: function to convert param to description; NULL to use default
>> + *
>> + * Define function @name_gen_params which uses @array to generate parameters.
>> + */
>> +#define KUNIT_ARRAY_PARAM(name, array, get_desc) \
>> + static const void *name##_gen_params(const void *prev, char *desc) \
>> + { \
>> + typeof((array)[0]) * __next = prev ? ((typeof(__next)) prev) + 1 : (array); \
>
> Why did you reintroduce a space between * and __next? AFAIK, this
> should follow the same style as the rest of the kernel, and it should
> just be 'thetype *ptr'.
>
I introduced this space because checkpatch.pl gave an error without the space:
ERROR: need consistent spacing around '*' (ctx:WxV)
#1786: FILE: ./include/kunit/test.h:1786:
+ typeof((array)[0]) *__next = prev ? ((typeof(__next)) prev) + 1 : (array); \
But, if this is a mistake as it doesn't recognize __next to be a pointer, I will remove the space.
>> + if (__next - (array) < ARRAY_SIZE((array))) { \
>> + void (*__get_desc)(typeof(__next), char *) = get_desc; \
>> + if (__get_desc) \
>> + __get_desc(__next, desc); \
>> + return __next; \
>> + } \
>> + return NULL; \
>> + }
>> +
>
> Thanks,
> -- Marco
>
Thanks!
On Sun, 15 Nov 2020 at 13:18, Arpitha Raghunandan <[email protected]> wrote:
>
> On 15/11/20 2:28 pm, Marco Elver wrote:
> > On Sat, 14 Nov 2020 at 13:38, Arpitha Raghunandan <[email protected]> wrote:
> >> Implementation of support for parameterized testing in KUnit. This
> >> approach requires the creation of a test case using the
> >> KUNIT_CASE_PARAM() macro that accepts a generator function as input.
> >>
> >> This generator function should return the next parameter given the
> >> previous parameter in parameterized tests. It also provides a macro to
> >> generate common-case generators based on arrays. Generators may also
> >> optionally provide a human-readable description of parameters, which is
> >> displayed where available.
> >>
> >> Note, currently the result of each parameter run is displayed in
> >> diagnostic lines, and only the overall test case output summarizes
> >> TAP-compliant success or failure of all parameter runs. In future, when
> >> supported by kunit-tool, these can be turned into subsubtest outputs.
> >>
> >> Signed-off-by: Arpitha Raghunandan <[email protected]>
> >> Co-developed-by: Marco Elver <[email protected]>
> >> Signed-off-by: Marco Elver <[email protected]>
> >> ---
> >> Changes v6->v7:
> >> - Clarify commit message.
> >> - Introduce ability to optionally generate descriptions for parameters;
> >> if no description is provided, we'll still print 'param-N'.
> >> - Change diagnostic line format to:
> >> # <test-case-name>: <ok|not ok> N - [<param description>]
> >>
> >> Changes v5->v6:
> >> - Fix alignment to maintain consistency
> >>
> >> Changes v4->v5:
> >> - Update kernel-doc comments.
> >> - Use const void* for generator return and prev value types.
> >> - Add kernel-doc comment for KUNIT_ARRAY_PARAM.
> >> - Rework parameterized test case execution strategy: each parameter is executed
> >> as if it was its own test case, with its own test initialization and cleanup
> >> (init and exit are called, etc.). However, we cannot add new test cases per TAP
> >> protocol once we have already started execution. Instead, log the result of
> >> each parameter run as a diagnostic comment.
> >>
> >> Changes v3->v4:
> >> - Rename kunit variables
> >> - Rename generator function helper macro
> >> - Add documentation for generator approach
> >> - Display test case name in case of failure along with param index
> >>
> >> Changes v2->v3:
> >> - Modifictaion of generator macro and method
> >>
> >> Changes v1->v2:
> >> - Use of a generator method to access test case parameters
> >> Changes v6->v7:
> >> - Clarify commit message.
> >> - Introduce ability to optionally generate descriptions for parameters;
> >> if no description is provided, we'll still print 'param-N'.
> >> - Change diagnostic line format to:
> >> # <test-case-name>: <ok|not ok> N - [<param description>]
> >> - Before execution of parameterized test case, count number of
> >> parameters and display number of parameters. Currently also as a
> >> diagnostic line, but this may be used in future to generate a subsubtest
> >> plan. A requirement of this change is that generators must generate a
> >> deterministic number of parameters.
> >>
> >> Changes v5->v6:
> >> - Fix alignment to maintain consistency
> >>
> >> Changes v4->v5:
> >> - Update kernel-doc comments.
> >> - Use const void* for generator return and prev value types.
> >> - Add kernel-doc comment for KUNIT_ARRAY_PARAM.
> >> - Rework parameterized test case execution strategy: each parameter is executed
> >> as if it was its own test case, with its own test initialization and cleanup
> >> (init and exit are called, etc.). However, we cannot add new test cases per TAP
> >> protocol once we have already started execution. Instead, log the result of
> >> each parameter run as a diagnostic comment.
> >>
> >> Changes v3->v4:
> >> - Rename kunit variables
> >> - Rename generator function helper macro
> >> - Add documentation for generator approach
> >> - Display test case name in case of failure along with param index
> >>
> >> Changes v2->v3:
> >> - Modifictaion of generator macro and method
> >>
> >> Changes v1->v2:
> >> - Use of a generator method to access test case parameters
> >>
> >> include/kunit/test.h | 51 ++++++++++++++++++++++++++++++++++++++
> >> lib/kunit/test.c | 59 ++++++++++++++++++++++++++++++++++----------
> >> 2 files changed, 97 insertions(+), 13 deletions(-)
> >>
> >> diff --git a/include/kunit/test.h b/include/kunit/test.h
> >> index db1b0ae666c4..cf5f33b1c890 100644
> >> --- a/include/kunit/test.h
> >> +++ b/include/kunit/test.h
> >> @@ -94,6 +94,9 @@ struct kunit;
> >> /* Size of log associated with test. */
> >> #define KUNIT_LOG_SIZE 512
> >>
> >> +/* Maximum size of parameter description string. */
> >> +#define KUNIT_PARAM_DESC_SIZE 64
> >
> > I think we need to make this larger, perhaps 128. I just noticed a few
> > of the inode-test strings are >64 chars (and it should probably also
> > use strncpy() to copy to description, which is my bad).
> >
>
> Okay, I will make the description size larger and use strncpy().
Thanks. There's also a report by the test robot now which noticed this.
> >> /*
> >> * TAP specifies subtest stream indentation of 4 spaces, 8 spaces for a
> >> * sub-subtest. See the "Subtests" section in
> >> @@ -107,6 +110,7 @@ struct kunit;
> > [...]
> >> +/**
> >> + * KUNIT_ARRAY_PARAM() - Define test parameter generator from an array.
> >> + * @name: prefix for the test parameter generator function.
> >> + * @array: array of test parameters.
> >> + * @get_desc: function to convert param to description; NULL to use default
> >> + *
> >> + * Define function @name_gen_params which uses @array to generate parameters.
> >> + */
> >> +#define KUNIT_ARRAY_PARAM(name, array, get_desc) \
> >> + static const void *name##_gen_params(const void *prev, char *desc) \
> >> + { \
> >> + typeof((array)[0]) * __next = prev ? ((typeof(__next)) prev) + 1 : (array); \
> >
> > Why did you reintroduce a space between * and __next? AFAIK, this
> > should follow the same style as the rest of the kernel, and it should
> > just be 'thetype *ptr'.
> >
>
> I introduced this space because checkpatch.pl gave an error without the space:
> ERROR: need consistent spacing around '*' (ctx:WxV)
> #1786: FILE: ./include/kunit/test.h:1786:
> + typeof((array)[0]) *__next = prev ? ((typeof(__next)) prev) + 1 : (array); \
>
> But, if this is a mistake as it doesn't recognize __next to be a pointer, I will remove the space.
I think checkpatch.pl thinks this is a multiplication. It's definitely
a false positive. Please do format it like a normal pointer.
Thanks,
-- Marco