2023-07-19 22:47:05

by Rae Moar

[permalink] [raw]
Subject: [PATCH v1 0/9] kunit: Add test attributes API

Hello everyone,

This patch series adds a test attributes framework to KUnit.

There has been interest in filtering out "slow" KUnit tests. Most notably,
a new config, CONFIG_MEMCPY_SLOW_KUNIT_TEST, has been added to exclude a
particularly slow memcpy test
(https://lore.kernel.org/all/[email protected]/).

This attributes framework can be used to save and access test associated
data, including whether a test is slow. These attributes are reportable
(via KTAP and command line output) and are also filterable.

This framework is designed to allow for the addition of other attributes in
the future. These attributes could include whether the test can be run
concurrently, test file path, etc.

To try out the framework I suggest running:
"./tools/testing/kunit/kunit.py run --filter speed!=slow"

This patch series was originally sent out as an RFC. Here is a link to the
RFC v2:
https://lore.kernel.org/all/[email protected]/

Thanks!
Rae

Rae Moar (9):
kunit: Add test attributes API structure
kunit: Add speed attribute
kunit: Add module attribute
kunit: Add ability to filter attributes
kunit: tool: Add command line interface to filter and report
attributes
kunit: memcpy: Mark tests as slow using test attributes
kunit: time: Mark test as slow using test attributes
kunit: add tests for filtering attributes
kunit: Add documentation of KUnit test attributes

.../dev-tools/kunit/running_tips.rst | 166 +++++++
include/kunit/attributes.h | 50 +++
include/kunit/test.h | 70 ++-
kernel/time/time_test.c | 2 +-
lib/Kconfig.debug | 3 +
lib/kunit/Makefile | 3 +-
lib/kunit/attributes.c | 421 ++++++++++++++++++
lib/kunit/executor.c | 115 ++++-
lib/kunit/executor_test.c | 128 +++++-
lib/kunit/kunit-example-test.c | 9 +
lib/kunit/test.c | 27 +-
lib/memcpy_kunit.c | 8 +-
tools/testing/kunit/kunit.py | 70 ++-
tools/testing/kunit/kunit_kernel.py | 8 +-
tools/testing/kunit/kunit_parser.py | 11 +-
tools/testing/kunit/kunit_tool_test.py | 39 +-
16 files changed, 1054 insertions(+), 76 deletions(-)
create mode 100644 include/kunit/attributes.h
create mode 100644 lib/kunit/attributes.c


base-commit: 64bd4641310c41a1ecf07c13c67bc0ed61045dfd
--
2.41.0.255.g8b1d071c50-goog



2023-07-19 22:48:15

by Rae Moar

[permalink] [raw]
Subject: [PATCH v1 7/9] kunit: time: Mark test as slow using test attributes

Mark the time KUnit test, time64_to_tm_test_date_range, as slow using test
attributes.

This test ran relatively much slower than most other KUnit tests.

By marking this test as slow, the test can now be filtered using the KUnit
test attribute filtering feature. Example: --filter "speed>slow". This will
run only the tests that have speeds faster than slow. The slow attribute
will also be outputted in KTAP.

Reviewed-by: David Gow <[email protected]>
Signed-off-by: Rae Moar <[email protected]>
---

Changes since RFC v2:
- No changes.
Changes since RFC v1:
- No changes.

kernel/time/time_test.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/time/time_test.c b/kernel/time/time_test.c
index 831e8e779ace..ca058c8af6ba 100644
--- a/kernel/time/time_test.c
+++ b/kernel/time/time_test.c
@@ -86,7 +86,7 @@ static void time64_to_tm_test_date_range(struct kunit *test)
}

static struct kunit_case time_test_cases[] = {
- KUNIT_CASE(time64_to_tm_test_date_range),
+ KUNIT_CASE_SLOW(time64_to_tm_test_date_range),
{}
};

--
2.41.0.255.g8b1d071c50-goog


2023-07-19 23:05:13

by Rae Moar

[permalink] [raw]
Subject: [PATCH v1 1/9] kunit: Add test attributes API structure

Add the basic structure of the test attribute API to KUnit, which can be
used to save and access test associated data.

Add attributes.c and attributes.h to hold associated structs and functions
for the API.

Create a struct that holds a variety of associated helper functions for
each test attribute. These helper functions will be used to get the
attribute value, convert the value to a string, and filter based on the
value. This struct is flexible by design to allow for attributes of
numerous types and contexts.

Add a method to print test attributes in the format of "# [<test_name if
not suite>.]<attribute_name>: <attribute_value>".

Example for a suite: "# speed: slow"

Example for a test case: "# test_case.speed: very_slow"

Use this method to report attributes in the KTAP output (KTAP spec:
https://docs.kernel.org/dev-tools/ktap.html) and _list_tests output when
kernel's new kunit.action=list_attr option is used. Note this is derivative
of the kunit.action=list option.

In test.h, add fields and associated helper functions to test cases and
suites to hold user-inputted test attributes.

Signed-off-by: Rae Moar <[email protected]>
---

Changes since RFC v2:
- No major changes

Changes since RFC v1:
- Add option to only include attributes in the _list_tests output
when this module param is set
- Add printing options for attributes to print always, print only for
suites, or print never.

include/kunit/attributes.h | 19 +++++++++
include/kunit/test.h | 33 ++++++++++++++++
lib/kunit/Makefile | 3 +-
lib/kunit/attributes.c | 80 ++++++++++++++++++++++++++++++++++++++
lib/kunit/executor.c | 21 ++++++++--
lib/kunit/test.c | 17 ++++----
6 files changed, 161 insertions(+), 12 deletions(-)
create mode 100644 include/kunit/attributes.h
create mode 100644 lib/kunit/attributes.c

diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h
new file mode 100644
index 000000000000..9fcd184cce36
--- /dev/null
+++ b/include/kunit/attributes.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * KUnit API to save and access test attributes
+ *
+ * Copyright (C) 2023, Google LLC.
+ * Author: Rae Moar <[email protected]>
+ */
+
+#ifndef _KUNIT_ATTRIBUTES_H
+#define _KUNIT_ATTRIBUTES_H
+
+/*
+ * Print all test attributes for a test case or suite.
+ * Output format for test cases: "# <test_name>.<attribute>: <value>"
+ * Output format for test suites: "# <attribute>: <value>"
+ */
+void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level);
+
+#endif /* _KUNIT_ATTRIBUTES_H */
diff --git a/include/kunit/test.h b/include/kunit/test.h
index 23120d50499e..1fc9155988e9 100644
--- a/include/kunit/test.h
+++ b/include/kunit/test.h
@@ -63,12 +63,16 @@ enum kunit_status {
KUNIT_SKIPPED,
};

+/* Holds attributes for each test case and suite */
+struct kunit_attributes {};
+
/**
* struct kunit_case - represents an individual test case.
*
* @run_case: the function representing the actual test case.
* @name: the name of the test case.
* @generate_params: the generator function for parameterized tests.
+ * @attr: the attributes associated with the test
*
* A test case is a function with the signature,
* ``void (*)(struct kunit *)``
@@ -104,6 +108,7 @@ struct kunit_case {
void (*run_case)(struct kunit *test);
const char *name;
const void* (*generate_params)(const void *prev, char *desc);
+ struct kunit_attributes attr;

/* private: internal use only. */
enum kunit_status status;
@@ -133,6 +138,18 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
*/
#define KUNIT_CASE(test_name) { .run_case = test_name, .name = #test_name }

+/**
+ * KUNIT_CASE_ATTR - A helper for creating a &struct kunit_case
+ * with attributes
+ *
+ * @test_name: a reference to a test case function.
+ * @attributes: a reference to a struct kunit_attributes object containing
+ * test attributes
+ */
+#define KUNIT_CASE_ATTR(test_name, attributes) \
+ { .run_case = test_name, .name = #test_name, \
+ .attr = attributes }
+
/**
* KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case
*
@@ -154,6 +171,20 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
{ .run_case = test_name, .name = #test_name, \
.generate_params = gen_params }

+/**
+ * KUNIT_CASE_PARAM_ATTR - A helper for creating a parameterized &struct
+ * kunit_case with attributes
+ *
+ * @test_name: a reference to a test case function.
+ * @gen_params: a reference to a parameter generator function.
+ * @attributes: a reference to a struct kunit_attributes object containing
+ * test attributes
+ */
+#define KUNIT_CASE_PARAM_ATTR(test_name, gen_params, attributes) \
+ { .run_case = test_name, .name = #test_name, \
+ .generate_params = gen_params, \
+ .attr = attributes }
+
/**
* struct kunit_suite - describes a related collection of &struct kunit_case
*
@@ -163,6 +194,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
* @init: called before every test case.
* @exit: called after every test case.
* @test_cases: a null terminated array of test cases.
+ * @attr: the attributes associated with the test suite
*
* A kunit_suite is a collection of related &struct kunit_case s, such that
* @init is called before every test case and @exit is called after every
@@ -182,6 +214,7 @@ struct kunit_suite {
int (*init)(struct kunit *test);
void (*exit)(struct kunit *test);
struct kunit_case *test_cases;
+ struct kunit_attributes attr;

/* private: internal use only */
char status_comment[KUNIT_STATUS_COMMENT_SIZE];
diff --git a/lib/kunit/Makefile b/lib/kunit/Makefile
index cb417f504996..46f75f23dfe4 100644
--- a/lib/kunit/Makefile
+++ b/lib/kunit/Makefile
@@ -6,7 +6,8 @@ kunit-objs += test.o \
string-stream.o \
assert.o \
try-catch.o \
- executor.o
+ executor.o \
+ attributes.o

ifeq ($(CONFIG_KUNIT_DEBUGFS),y)
kunit-objs += debugfs.o
diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
new file mode 100644
index 000000000000..9bda5a5f4030
--- /dev/null
+++ b/lib/kunit/attributes.c
@@ -0,0 +1,80 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * KUnit API to save and access test attributes
+ *
+ * Copyright (C) 2023, Google LLC.
+ * Author: Rae Moar <[email protected]>
+ */
+
+#include <kunit/test.h>
+#include <kunit/attributes.h>
+
+/* Options for printing attributes:
+ * PRINT_ALWAYS - attribute is printed for every test case and suite if set
+ * PRINT_SUITE - attribute is printed for every suite if set but not for test cases
+ * PRINT_NEVER - attribute is never printed
+ */
+enum print_ops {
+ PRINT_ALWAYS,
+ PRINT_SUITE,
+ PRINT_NEVER,
+};
+
+/**
+ * struct kunit_attr - represents a test attribute and holds flexible
+ * helper functions to interact with attribute.
+ *
+ * @name: name of test attribute, eg. speed
+ * @get_attr: function to return attribute value given a test
+ * @to_string: function to return string representation of given
+ * attribute value
+ * @filter: function to indicate whether a given attribute value passes a
+ * filter
+ */
+struct kunit_attr {
+ const char *name;
+ void *(*get_attr)(void *test_or_suite, bool is_test);
+ const char *(*to_string)(void *attr, bool *to_free);
+ int (*filter)(void *attr, const char *input, int *err);
+ void *attr_default;
+ enum print_ops print;
+};
+
+/* List of all Test Attributes */
+
+static struct kunit_attr kunit_attr_list[] = {};
+
+/* Helper Functions to Access Attributes */
+
+void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level)
+{
+ int i;
+ bool to_free;
+ void *attr;
+ const char *attr_name, *attr_str;
+ struct kunit_suite *suite = is_test ? NULL : test_or_suite;
+ struct kunit_case *test = is_test ? test_or_suite : NULL;
+
+ for (i = 0; i < ARRAY_SIZE(kunit_attr_list); i++) {
+ if (kunit_attr_list[i].print == PRINT_NEVER ||
+ (test && kunit_attr_list[i].print == PRINT_SUITE))
+ continue;
+ attr = kunit_attr_list[i].get_attr(test_or_suite, is_test);
+ if (attr) {
+ attr_name = kunit_attr_list[i].name;
+ attr_str = kunit_attr_list[i].to_string(attr, &to_free);
+ if (test) {
+ kunit_log(KERN_INFO, test, "%*s# %s.%s: %s",
+ KUNIT_INDENT_LEN * test_level, "", test->name,
+ attr_name, attr_str);
+ } else {
+ kunit_log(KERN_INFO, suite, "%*s# %s: %s",
+ KUNIT_INDENT_LEN * test_level, "", attr_name, attr_str);
+ }
+
+ /* Free to_string of attribute if needed */
+ if (to_free)
+ kfree(attr_str);
+ }
+ }
+}
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
index 74982b83707c..12e38a48a5cc 100644
--- a/lib/kunit/executor.c
+++ b/lib/kunit/executor.c
@@ -2,6 +2,7 @@

#include <linux/reboot.h>
#include <kunit/test.h>
+#include <kunit/attributes.h>
#include <linux/glob.h>
#include <linux/moduleparam.h>

@@ -24,7 +25,8 @@ module_param_named(action, action_param, charp, 0);
MODULE_PARM_DESC(action,
"Changes KUnit executor behavior, valid values are:\n"
"<none>: run the tests like normal\n"
- "'list' to list test names instead of running them.\n");
+ "'list' to list test names instead of running them.\n"
+ "'list_attr' to list test names and attributes instead of running them.\n");

/* glob_match() needs NULL terminated strings, so we need a copy of filter_glob_param. */
struct kunit_test_filter {
@@ -172,7 +174,7 @@ static void kunit_exec_run_tests(struct suite_set *suite_set)
__kunit_test_suites_init(suite_set->start, num_suites);
}

-static void kunit_exec_list_tests(struct suite_set *suite_set)
+static void kunit_exec_list_tests(struct suite_set *suite_set, bool include_attr)
{
struct kunit_suite * const *suites;
struct kunit_case *test_case;
@@ -180,10 +182,19 @@ static void kunit_exec_list_tests(struct suite_set *suite_set)
/* Hack: print a ktap header so kunit.py can find the start of KUnit output. */
pr_info("KTAP version 1\n");

- for (suites = suite_set->start; suites < suite_set->end; suites++)
+ for (suites = suite_set->start; suites < suite_set->end; suites++) {
+ /* Print suite name and suite attributes */
+ pr_info("%s\n", (*suites)->name);
+ if (include_attr)
+ kunit_print_attr((void *)(*suites), false, 0);
+
+ /* Print test case name and attributes in suite */
kunit_suite_for_each_test_case((*suites), test_case) {
pr_info("%s.%s\n", (*suites)->name, test_case->name);
+ if (include_attr)
+ kunit_print_attr((void *)test_case, true, 0);
}
+ }
}

int kunit_run_all_tests(void)
@@ -206,7 +217,9 @@ int kunit_run_all_tests(void)
if (!action_param)
kunit_exec_run_tests(&suite_set);
else if (strcmp(action_param, "list") == 0)
- kunit_exec_list_tests(&suite_set);
+ kunit_exec_list_tests(&suite_set, false);
+ else if (strcmp(action_param, "list_attr") == 0)
+ kunit_exec_list_tests(&suite_set, true);
else
pr_err("kunit executor: unknown action '%s'\n", action_param);

diff --git a/lib/kunit/test.c b/lib/kunit/test.c
index 84e4666555c9..9ee55139ecd1 100644
--- a/lib/kunit/test.c
+++ b/lib/kunit/test.c
@@ -9,6 +9,7 @@
#include <kunit/resource.h>
#include <kunit/test.h>
#include <kunit/test-bug.h>
+#include <kunit/attributes.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/moduleparam.h>
@@ -168,6 +169,13 @@ size_t kunit_suite_num_test_cases(struct kunit_suite *suite)
}
EXPORT_SYMBOL_GPL(kunit_suite_num_test_cases);

+/* Currently supported test levels */
+enum {
+ KUNIT_LEVEL_SUITE = 0,
+ KUNIT_LEVEL_CASE,
+ KUNIT_LEVEL_CASE_PARAM,
+};
+
static void kunit_print_suite_start(struct kunit_suite *suite)
{
/*
@@ -181,17 +189,11 @@ static void kunit_print_suite_start(struct kunit_suite *suite)
pr_info(KUNIT_SUBTEST_INDENT "KTAP version 1\n");
pr_info(KUNIT_SUBTEST_INDENT "# Subtest: %s\n",
suite->name);
+ kunit_print_attr((void *)suite, false, KUNIT_LEVEL_CASE);
pr_info(KUNIT_SUBTEST_INDENT "1..%zd\n",
kunit_suite_num_test_cases(suite));
}

-/* Currently supported test levels */
-enum {
- KUNIT_LEVEL_SUITE = 0,
- KUNIT_LEVEL_CASE,
- KUNIT_LEVEL_CASE_PARAM,
-};
-
static void kunit_print_ok_not_ok(struct kunit *test,
unsigned int test_level,
enum kunit_status status,
@@ -651,6 +653,7 @@ int kunit_run_tests(struct kunit_suite *suite)
}
}

+ kunit_print_attr((void *)test_case, true, KUNIT_LEVEL_CASE);

kunit_print_test_stats(&test, param_stats);

--
2.41.0.255.g8b1d071c50-goog


2023-07-19 23:23:26

by Rae Moar

[permalink] [raw]
Subject: [PATCH v1 4/9] kunit: Add ability to filter attributes

Add filtering of test attributes. Users can filter tests using the
module_param called "filter".

Filters are imputed in the format: <attribute_name><operation><value>

Example: kunit.filter="speed>slow"

Operations include: >, <, >=, <=, !=, and =. These operations will act the
same for attributes of the same type but may not between types.

Note multiple filters can be inputted by separating them with a comma.
Example: kunit.filter="speed=slow, module!=example"

Since both suites and test cases can have attributes, there may be
conflicts. The process of filtering follows these rules:
- Filtering always operates at a per-test level.
- If a test has an attribute set, then the test's value is filtered on.
- Otherwise, the value falls back to the suite's value.
- If neither are set, the attribute has a global "default" value, which
is used.

Filtered tests will not be run or show in output. The tests can instead be
skipped using the configurable option "kunit.filter_action=skip".

Note the default settings for running tests remains unfiltered.

Finally, add "filter" methods for the speed and module attributes to parse
and compare attribute values.

Note this filtering functionality will be added to kunit.py in the next
patch.

Signed-off-by: Rae Moar <[email protected]>
---

Changes since RFC v2:
- Change to output only one error before exiting.

Changes since RFC v1:
- Change method for inputting filters to allow for spaces in filtering
values
- Add option to skip filtered tests instead of not run or show them with
the --filter_skip flag

include/kunit/attributes.h | 31 +++++
lib/kunit/attributes.c | 271 +++++++++++++++++++++++++++++++++++++
lib/kunit/executor.c | 94 ++++++++++---
lib/kunit/executor_test.c | 12 +-
lib/kunit/test.c | 10 +-
5 files changed, 390 insertions(+), 28 deletions(-)

diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h
index 9fcd184cce36..bc76a0b786d2 100644
--- a/include/kunit/attributes.h
+++ b/include/kunit/attributes.h
@@ -9,6 +9,20 @@
#ifndef _KUNIT_ATTRIBUTES_H
#define _KUNIT_ATTRIBUTES_H

+/*
+ * struct kunit_attr_filter - representation of attributes filter with the
+ * attribute object and string input
+ */
+struct kunit_attr_filter {
+ struct kunit_attr *attr;
+ char *input;
+};
+
+/*
+ * Returns the name of the filter's attribute.
+ */
+const char *kunit_attr_filter_name(struct kunit_attr_filter filter);
+
/*
* Print all test attributes for a test case or suite.
* Output format for test cases: "# <test_name>.<attribute>: <value>"
@@ -16,4 +30,21 @@
*/
void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level);

+/*
+ * Returns the number of fitlers in input.
+ */
+int kunit_get_filter_count(char *input);
+
+/*
+ * Parse attributes filter input and return an objects containing the
+ * attribute object and the string input of the next filter.
+ */
+struct kunit_attr_filter kunit_next_attr_filter(char **filters, int *err);
+
+/*
+ * Returns a copy of the suite containing only tests that pass the filter.
+ */
+struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *const suite,
+ struct kunit_attr_filter filter, char *action, int *err);
+
#endif /* _KUNIT_ATTRIBUTES_H */
diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
index 43dcb5de8b8f..861b5ae3c1a1 100644
--- a/lib/kunit/attributes.c
+++ b/lib/kunit/attributes.c
@@ -67,6 +67,104 @@ static const char *attr_string_to_string(void *attr, bool *to_free)
return (char *) attr;
}

+/* Filter Methods */
+
+static const char op_list[] = "<>!=";
+
+/*
+ * Returns whether the inputted integer value matches the filter given
+ * by the operation string and inputted integer.
+ */
+static int int_filter(long val, const char *op, int input, int *err)
+{
+ if (!strncmp(op, "<=", 2))
+ return (val <= input);
+ else if (!strncmp(op, ">=", 2))
+ return (val >= input);
+ else if (!strncmp(op, "!=", 2))
+ return (val != input);
+ else if (!strncmp(op, ">", 1))
+ return (val > input);
+ else if (!strncmp(op, "<", 1))
+ return (val < input);
+ else if (!strncmp(op, "=", 1))
+ return (val == input);
+ *err = -EINVAL;
+ pr_err("kunit executor: invalid filter operation: %s\n", op);
+ return false;
+}
+
+/*
+ * Returns whether the inputted enum value "attr" matches the filter given
+ * by the input string. Note: the str_list includes the corresponding string
+ * list to the enum values.
+ */
+static int attr_enum_filter(void *attr, const char *input, int *err,
+ const char * const str_list[], int max)
+{
+ int i, j, input_int;
+ long test_val = (long)attr;
+ const char *input_val;
+
+ for (i = 0; input[i]; i++) {
+ if (!strchr(op_list, input[i])) {
+ input_val = input + i;
+ break;
+ }
+ }
+
+ if (!input_val) {
+ *err = -EINVAL;
+ pr_err("kunit executor: filter value not found: %s\n", input);
+ return false;
+ }
+
+ for (j = 0; j <= max; j++) {
+ if (!strcmp(input_val, str_list[j]))
+ input_int = j;
+ }
+
+ if (!input_int) {
+ *err = -EINVAL;
+ pr_err("kunit executor: invalid filter input: %s\n", input);
+ return false;
+ }
+
+ return int_filter(test_val, input, input_int, err);
+}
+
+static int attr_speed_filter(void *attr, const char *input, int *err)
+{
+ return attr_enum_filter(attr, input, err, speed_str_list, KUNIT_SPEED_MAX);
+}
+
+/*
+ * Returns whether the inputted string value (attr) matches the filter given
+ * by the input string.
+ */
+static int attr_string_filter(void *attr, const char *input, int *err)
+{
+ char *str = attr;
+
+ if (!strncmp(input, "<", 1)) {
+ *err = -EINVAL;
+ pr_err("kunit executor: invalid filter input: %s\n", input);
+ return false;
+ } else if (!strncmp(input, ">", 1)) {
+ *err = -EINVAL;
+ pr_err("kunit executor: invalid filter input: %s\n", input);
+ return false;
+ } else if (!strncmp(input, "!=", 2)) {
+ return (strcmp(input + 2, str) != 0);
+ } else if (!strncmp(input, "=", 1)) {
+ return (strcmp(input + 1, str) == 0);
+ }
+ *err = -EINVAL;
+ pr_err("kunit executor: invalid filter operation: %s\n", input);
+ return false;
+}
+
+
/* Get Attribute Methods */

static void *attr_speed_get(void *test_or_suite, bool is_test)
@@ -98,6 +196,7 @@ static const struct kunit_attr speed_attr = {
.name = "speed",
.get_attr = attr_speed_get,
.to_string = attr_speed_to_string,
+ .filter = attr_speed_filter,
.attr_default = (void *)KUNIT_SPEED_NORMAL,
.print = PRINT_ALWAYS,
};
@@ -106,6 +205,7 @@ static const struct kunit_attr module_attr = {
.name = "module",
.get_attr = attr_module_get,
.to_string = attr_string_to_string,
+ .filter = attr_string_filter,
.attr_default = (void *)"",
.print = PRINT_SUITE,
};
@@ -116,6 +216,11 @@ static struct kunit_attr kunit_attr_list[] = {speed_attr, module_attr};

/* Helper Functions to Access Attributes */

+const char *kunit_attr_filter_name(struct kunit_attr_filter filter)
+{
+ return filter.attr->name;
+}
+
void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level)
{
int i;
@@ -148,3 +253,169 @@ void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level
}
}
}
+
+/* Helper Functions to Filter Attributes */
+
+int kunit_get_filter_count(char *input)
+{
+ int i, comma_index, count = 0;
+
+ for (i = 0; input[i]; i++) {
+ if (input[i] == ',') {
+ if ((i - comma_index) > 1)
+ count++;
+ comma_index = i;
+ }
+ }
+ if ((i - comma_index) > 0)
+ count++;
+ return count;
+}
+
+struct kunit_attr_filter kunit_next_attr_filter(char **filters, int *err)
+{
+ struct kunit_attr_filter filter;
+ int i, j, comma_index, new_start_index;
+ int op_index = -1, attr_index = -1;
+ char op;
+ char *input = *filters;
+
+ /* Parse input until operation */
+ for (i = 0; input[i]; i++) {
+ if (op_index < 0 && strchr(op_list, input[i])) {
+ op_index = i;
+ } else if (!comma_index && input[i] == ',') {
+ comma_index = i;
+ } else if (comma_index && input[i] != ' ') {
+ new_start_index = i;
+ break;
+ }
+ }
+
+ if (op_index <= 0) {
+ *err = -EINVAL;
+ pr_err("kunit executor: filter operation not found: %s\n", input);
+ return filter;
+ }
+
+ /* Temporarily set operator to \0 character. */
+ op = input[op_index];
+ input[op_index] = '\0';
+
+ /* Find associated kunit_attr object */
+ for (j = 0; j < ARRAY_SIZE(kunit_attr_list); j++) {
+ if (!strcmp(input, kunit_attr_list[j].name)) {
+ attr_index = j;
+ break;
+ }
+ }
+
+ input[op_index] = op;
+
+ if (attr_index < 0) {
+ *err = -EINVAL;
+ pr_err("kunit executor: attribute not found: %s\n", input);
+ } else {
+ filter.attr = &kunit_attr_list[attr_index];
+ }
+
+ if (comma_index) {
+ input[comma_index] = '\0';
+ filter.input = input + op_index;
+ input = input + new_start_index;
+ } else {
+ filter.input = input + op_index;
+ input = NULL;
+ }
+
+ *filters = input;
+
+ return filter;
+}
+
+struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *const suite,
+ struct kunit_attr_filter filter, char *action, int *err)
+{
+ int n = 0;
+ struct kunit_case *filtered, *test_case;
+ struct kunit_suite *copy;
+ void *suite_val, *test_val;
+ bool suite_result, test_result, default_result, result;
+
+ /* Allocate memory for new copy of suite and list of test cases */
+ copy = kmemdup(suite, sizeof(*copy), GFP_KERNEL);
+ if (!copy)
+ return ERR_PTR(-ENOMEM);
+
+ kunit_suite_for_each_test_case(suite, test_case) { n++; }
+
+ filtered = kcalloc(n + 1, sizeof(*filtered), GFP_KERNEL);
+ if (!filtered) {
+ kfree(copy);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ n = 0;
+
+ /* Save filtering result on default value */
+ default_result = filter.attr->filter(filter.attr->attr_default, filter.input, err);
+ if (*err) {
+ kfree(copy);
+ kfree(filtered);
+ return NULL;
+ }
+
+ /* Save suite attribute value and filtering result on that value */
+ suite_val = filter.attr->get_attr((void *)suite, false);
+ suite_result = filter.attr->filter(suite_val, filter.input, err);
+ if (*err) {
+ kfree(copy);
+ kfree(filtered);
+ return NULL;
+ }
+
+ /* For each test case, save test case if passes filtering. */
+ kunit_suite_for_each_test_case(suite, test_case) {
+ test_val = filter.attr->get_attr((void *) test_case, true);
+ test_result = filter.attr->filter(filter.attr->get_attr(test_case, true),
+ filter.input, err);
+ if (*err) {
+ kfree(copy);
+ kfree(filtered);
+ return NULL;
+ }
+
+ /*
+ * If attribute value of test case is set, filter on that value.
+ * If not, filter on suite value if set. If not, filter on
+ * default value.
+ */
+ result = false;
+ if (test_val) {
+ if (test_result)
+ result = true;
+ } else if (suite_val) {
+ if (suite_result)
+ result = true;
+ } else if (default_result) {
+ result = true;
+ }
+
+ if (result) {
+ filtered[n++] = *test_case;
+ } else if (action && strcmp(action, "skip") == 0) {
+ test_case->status = KUNIT_SKIPPED;
+ filtered[n++] = *test_case;
+ }
+ }
+
+ if (n == 0) {
+ kfree(copy);
+ kfree(filtered);
+ return NULL;
+ }
+
+ copy->test_cases = filtered;
+
+ return copy;
+}
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
index 12e38a48a5cc..c286ae47435a 100644
--- a/lib/kunit/executor.c
+++ b/lib/kunit/executor.c
@@ -17,6 +17,9 @@ extern struct kunit_suite * const __kunit_suites_end[];

static char *filter_glob_param;
static char *action_param;
+static char *filter_param;
+static char *filter_action_param;
+

module_param_named(filter_glob, filter_glob_param, charp, 0);
MODULE_PARM_DESC(filter_glob,
@@ -27,15 +30,23 @@ MODULE_PARM_DESC(action,
"<none>: run the tests like normal\n"
"'list' to list test names instead of running them.\n"
"'list_attr' to list test names and attributes instead of running them.\n");
+module_param_named(filter, filter_param, charp, 0);
+MODULE_PARM_DESC(filter,
+ "Filter which KUnit test suites/tests run at boot-time using attributes, e.g. speed>slow");
+module_param_named(filter_action, filter_action_param, charp, 0);
+MODULE_PARM_DESC(filter_action,
+ "Changes behavior of filtered tests using attributes, valid values are:\n"
+ "<none>: do not run filtered tests as normal\n"
+ "'skip': skip all filtered tests instead so tests will appear in output\n");

/* glob_match() needs NULL terminated strings, so we need a copy of filter_glob_param. */
-struct kunit_test_filter {
+struct kunit_glob_filter {
char *suite_glob;
char *test_glob;
};

/* Split "suite_glob.test_glob" into two. Assumes filter_glob is not empty. */
-static void kunit_parse_filter_glob(struct kunit_test_filter *parsed,
+static void kunit_parse_glob_filter(struct kunit_glob_filter *parsed,
const char *filter_glob)
{
const int len = strlen(filter_glob);
@@ -57,7 +68,7 @@ static void kunit_parse_filter_glob(struct kunit_test_filter *parsed,

/* Create a copy of suite with only tests that match test_glob. */
static struct kunit_suite *
-kunit_filter_tests(const struct kunit_suite *const suite, const char *test_glob)
+kunit_filter_glob_tests(const struct kunit_suite *const suite, const char *test_glob)
{
int n = 0;
struct kunit_case *filtered, *test_case;
@@ -111,12 +122,15 @@ static void kunit_free_suite_set(struct suite_set suite_set)

static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
const char *filter_glob,
+ char *filters,
+ char *filter_action,
int *err)
{
- int i;
- struct kunit_suite **copy, *filtered_suite;
+ int i, j, k, filter_count;
+ struct kunit_suite **copy, *filtered_suite, *new_filtered_suite;
struct suite_set filtered;
- struct kunit_test_filter filter;
+ struct kunit_glob_filter parsed_glob;
+ struct kunit_attr_filter *parsed_filters;

const size_t max = suite_set->end - suite_set->start;

@@ -127,17 +141,52 @@ static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
return filtered;
}

- kunit_parse_filter_glob(&filter, filter_glob);
-
- for (i = 0; &suite_set->start[i] != suite_set->end; i++) {
- if (!glob_match(filter.suite_glob, suite_set->start[i]->name))
- continue;
+ if (filter_glob)
+ kunit_parse_glob_filter(&parsed_glob, filter_glob);

- filtered_suite = kunit_filter_tests(suite_set->start[i], filter.test_glob);
- if (IS_ERR(filtered_suite)) {
- *err = PTR_ERR(filtered_suite);
+ /* Parse attribute filters */
+ if (filters) {
+ filter_count = kunit_get_filter_count(filters);
+ parsed_filters = kcalloc(filter_count + 1, sizeof(*parsed_filters), GFP_KERNEL);
+ for (j = 0; j < filter_count; j++)
+ parsed_filters[j] = kunit_next_attr_filter(&filters, err);
+ if (*err)
return filtered;
+ }
+
+ for (i = 0; &suite_set->start[i] != suite_set->end; i++) {
+ filtered_suite = suite_set->start[i];
+ if (filter_glob) {
+ if (!glob_match(parsed_glob.suite_glob, filtered_suite->name))
+ continue;
+ filtered_suite = kunit_filter_glob_tests(filtered_suite,
+ parsed_glob.test_glob);
+ if (IS_ERR(filtered_suite)) {
+ *err = PTR_ERR(filtered_suite);
+ return filtered;
+ }
+ }
+ if (filter_count) {
+ for (k = 0; k < filter_count; k++) {
+ new_filtered_suite = kunit_filter_attr_tests(filtered_suite,
+ parsed_filters[k], filter_action, err);
+
+ /* Free previous copy of suite */
+ if (k > 0 || filter_glob)
+ kfree(filtered_suite);
+ filtered_suite = new_filtered_suite;
+
+ if (*err)
+ return filtered;
+ if (IS_ERR(filtered_suite)) {
+ *err = PTR_ERR(filtered_suite);
+ return filtered;
+ }
+ if (!filtered_suite)
+ break;
+ }
}
+
if (!filtered_suite)
continue;

@@ -145,8 +194,14 @@ static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
}
filtered.end = copy;

- kfree(filter.suite_glob);
- kfree(filter.test_glob);
+ if (filter_glob) {
+ kfree(parsed_glob.suite_glob);
+ kfree(parsed_glob.test_glob);
+ }
+
+ if (filter_count)
+ kfree(parsed_filters);
+
return filtered;
}

@@ -206,8 +261,9 @@ int kunit_run_all_tests(void)
goto out;
}

- if (filter_glob_param) {
- suite_set = kunit_filter_suites(&suite_set, filter_glob_param, &err);
+ if (filter_glob_param || filter_param) {
+ suite_set = kunit_filter_suites(&suite_set, filter_glob_param,
+ filter_param, filter_action_param, &err);
if (err) {
pr_err("kunit executor: error filtering suites: %d\n", err);
goto out;
@@ -223,7 +279,7 @@ int kunit_run_all_tests(void)
else
pr_err("kunit executor: unknown action '%s'\n", action_param);

- if (filter_glob_param) { /* a copy was made of each suite */
+ if (filter_glob_param || filter_param) { /* a copy was made of each suite */
kunit_free_suite_set(suite_set);
}

diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c
index ce6749af374d..d7ab069324b5 100644
--- a/lib/kunit/executor_test.c
+++ b/lib/kunit/executor_test.c
@@ -24,15 +24,15 @@ static struct kunit_case dummy_test_cases[] = {

static void parse_filter_test(struct kunit *test)
{
- struct kunit_test_filter filter = {NULL, NULL};
+ struct kunit_glob_filter filter = {NULL, NULL};

- kunit_parse_filter_glob(&filter, "suite");
+ kunit_parse_glob_filter(&filter, "suite");
KUNIT_EXPECT_STREQ(test, filter.suite_glob, "suite");
KUNIT_EXPECT_FALSE(test, filter.test_glob);
kfree(filter.suite_glob);
kfree(filter.test_glob);

- kunit_parse_filter_glob(&filter, "suite.test");
+ kunit_parse_glob_filter(&filter, "suite.test");
KUNIT_EXPECT_STREQ(test, filter.suite_glob, "suite");
KUNIT_EXPECT_STREQ(test, filter.test_glob, "test");
kfree(filter.suite_glob);
@@ -50,7 +50,7 @@ static void filter_suites_test(struct kunit *test)
subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases);

/* Want: suite1, suite2, NULL -> suite2, NULL */
- got = kunit_filter_suites(&suite_set, "suite2", &err);
+ got = kunit_filter_suites(&suite_set, "suite2", NULL, NULL, &err);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
KUNIT_ASSERT_EQ(test, err, 0);
kfree_at_end(test, got.start);
@@ -74,7 +74,7 @@ static void filter_suites_test_glob_test(struct kunit *test)
subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases);

/* Want: suite1, suite2, NULL -> suite2 (just test1), NULL */
- got = kunit_filter_suites(&suite_set, "suite2.test2", &err);
+ got = kunit_filter_suites(&suite_set, "suite2.test2", NULL, NULL, &err);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
KUNIT_ASSERT_EQ(test, err, 0);
kfree_at_end(test, got.start);
@@ -100,7 +100,7 @@ static void filter_suites_to_empty_test(struct kunit *test)
subsuite[0] = alloc_fake_suite(test, "suite1", dummy_test_cases);
subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases);

- got = kunit_filter_suites(&suite_set, "not_found", &err);
+ got = kunit_filter_suites(&suite_set, "not_found", NULL, NULL, &err);
KUNIT_ASSERT_EQ(test, err, 0);
kfree_at_end(test, got.start); /* just in case */

diff --git a/lib/kunit/test.c b/lib/kunit/test.c
index 9ee55139ecd1..cb9797fa6303 100644
--- a/lib/kunit/test.c
+++ b/lib/kunit/test.c
@@ -613,18 +613,22 @@ int kunit_run_tests(struct kunit_suite *suite)
kunit_suite_for_each_test_case(suite, test_case) {
struct kunit test = { .param_value = NULL, .param_index = 0 };
struct kunit_result_stats param_stats = { 0 };
- test_case->status = KUNIT_SKIPPED;

kunit_init_test(&test, test_case->name, test_case->log);
-
- if (!test_case->generate_params) {
+ if (test_case->status == KUNIT_SKIPPED) {
+ /* Test marked as skip */
+ test.status = KUNIT_SKIPPED;
+ kunit_update_stats(&param_stats, test.status);
+ } else if (!test_case->generate_params) {
/* Non-parameterised test. */
+ test_case->status = KUNIT_SKIPPED;
kunit_run_case_catch_errors(suite, test_case, &test);
kunit_update_stats(&param_stats, test.status);
} else {
/* Get initial param. */
param_desc[0] = '\0';
test.param_value = test_case->generate_params(NULL, param_desc);
+ test_case->status = KUNIT_SKIPPED;
kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT
"KTAP version 1\n");
kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT
--
2.41.0.255.g8b1d071c50-goog


2023-07-22 02:56:06

by kernel test robot

[permalink] [raw]
Subject: Re: [PATCH v1 4/9] kunit: Add ability to filter attributes

Hi Rae,

kernel test robot noticed the following build warnings:

[auto build test WARNING on 64bd4641310c41a1ecf07c13c67bc0ed61045dfd]

url: https://github.com/intel-lab-lkp/linux/commits/Rae-Moar/kunit-Add-test-attributes-API-structure/20230720-062623
base: 64bd4641310c41a1ecf07c13c67bc0ed61045dfd
patch link: https://lore.kernel.org/r/20230719222338.259684-5-rmoar%40google.com
patch subject: [PATCH v1 4/9] kunit: Add ability to filter attributes
config: x86_64-rhel-8.3-rust (https://download.01.org/0day-ci/archive/20230722/[email protected]/config)
compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)
reproduce: (https://download.01.org/0day-ci/archive/20230722/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> lib/kunit/attributes.c:109:14: warning: variable 'input_val' is used uninitialized whenever 'for' loop exits because its condition is false [-Wsometimes-uninitialized]
for (i = 0; input[i]; i++) {
^~~~~~~~
lib/kunit/attributes.c:116:7: note: uninitialized use occurs here
if (!input_val) {
^~~~~~~~~
lib/kunit/attributes.c:109:14: note: remove the condition if it is always true
for (i = 0; input[i]; i++) {
^~~~~~~~
lib/kunit/attributes.c:107:23: note: initialize the variable 'input_val' to silence this warning
const char *input_val;
^
= NULL
lib/kunit/attributes.c:215:47: error: initializer element is not a compile-time constant
static struct kunit_attr kunit_attr_list[] = {speed_attr, module_attr};
^~~~~~~~~~
1 warning and 1 error generated.


vim +109 lib/kunit/attributes.c

96
97 /*
98 * Returns whether the inputted enum value "attr" matches the filter given
99 * by the input string. Note: the str_list includes the corresponding string
100 * list to the enum values.
101 */
102 static int attr_enum_filter(void *attr, const char *input, int *err,
103 const char * const str_list[], int max)
104 {
105 int i, j, input_int;
106 long test_val = (long)attr;
107 const char *input_val;
108
> 109 for (i = 0; input[i]; i++) {
110 if (!strchr(op_list, input[i])) {
111 input_val = input + i;
112 break;
113 }
114 }
115
116 if (!input_val) {
117 *err = -EINVAL;
118 pr_err("kunit executor: filter value not found: %s\n", input);
119 return false;
120 }
121
122 for (j = 0; j <= max; j++) {
123 if (!strcmp(input_val, str_list[j]))
124 input_int = j;
125 }
126
127 if (!input_int) {
128 *err = -EINVAL;
129 pr_err("kunit executor: invalid filter input: %s\n", input);
130 return false;
131 }
132
133 return int_filter(test_val, input, input_int, err);
134 }
135

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki