Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230707210947.1208717-1-rmoar@google.com> <20230707210947.1208717-5-rmoar@google.com>
In-Reply-To: <20230707210947.1208717-5-rmoar@google.com>
From:   David Gow <davidgow@google.com>
Date:   Tue, 18 Jul 2023 15:39:10 +0800
Message-ID: <CABVgOS=X+C2g6_q-=3qdzp1YMPjazq8F2C5ZqvFhub+t3_sUdQ@mail.gmail.com>
Subject: Re: [RFC v2 4/9] kunit: Add ability to filter attributes
To:     Rae Moar <rmoar@google.com>
Cc:     shuah@kernel.org, dlatypov@google.com, brendan.higgins@linux.dev,
        linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com,
        linux-kernel@vger.kernel.org, keescook@chromium.org,
        linux-hardening@vger.kernel.org, jstultz@google.com,
        tglx@linutronix.de, sboyd@kernel.org
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256;
        boundary="0000000000007952c70600be0389"
Precedence: bulk

--0000000000007952c70600be0389
Content-Type: text/plain; charset="UTF-8"

On Sat, 8 Jul 2023 at 05:10, Rae Moar <rmoar@google.com> wrote:
>
> Add filtering of test attributes. Users can filter tests using the
> module_param called "filter".
>
> Filters are imputed in the format: <attribute_name><operation><value>
>
> Example: kunit.filter="speed>slow"
>
> Operations include: >, <, >=, <=, !=, and =. These operations will act the
> same for attributes of the same type but may not between types.
>
> Note multiple filters can be inputted by separating them with a comma.
> Example: kunit.filter="speed=slow, module!=example"
>
> Since both suites and test cases can have attributes, there may be
> conflicts. The process of filtering follows these rules:
> - Filtering always operates at a per-test level.
> - If a test has an attribute set, then the test's value is filtered on.
> - Otherwise, the value falls back to the suite's value.
> - If neither are set, the attribute has a global "default" value, which
>   is used.
>
> Filtered tests will not be run or show in output. The tests can instead be
> skipped using the configurable option "kunit.filter_action=skip".
>
> Note the default settings for running tests remains unfiltered.
>
> Finally, add "filter" methods for the speed and module attributes to parse
> and compare attribute values.
>
> Note this filtering functionality will be added to kunit.py in the next
> patch.
>
> Signed-off-by: Rae Moar <rmoar@google.com>
> ---
>
> Changes since v1:
> - Change method for inputting filters to allow for spaces in filtering
>   values
> - Add option to skip filtered tests instead of not run or show them with
>   the --filter_skip flag
>
>  include/kunit/attributes.h |  31 +++++
>  lib/kunit/attributes.c     | 256 +++++++++++++++++++++++++++++++++++++
>  lib/kunit/executor.c       |  94 +++++++++++---
>  lib/kunit/executor_test.c  |  12 +-
>  lib/kunit/test.c           |  10 +-
>  5 files changed, 375 insertions(+), 28 deletions(-)
>
> diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h
> index 9fcd184cce36..bc76a0b786d2 100644
> --- a/include/kunit/attributes.h
> +++ b/include/kunit/attributes.h
> @@ -9,6 +9,20 @@
>  #ifndef _KUNIT_ATTRIBUTES_H
>  #define _KUNIT_ATTRIBUTES_H
>
> +/*
> + * struct kunit_attr_filter - representation of attributes filter with the
> + * attribute object and string input
> + */
> +struct kunit_attr_filter {
> +       struct kunit_attr *attr;
> +       char *input;
> +};
> +
> +/*
> + * Returns the name of the filter's attribute.
> + */
> +const char *kunit_attr_filter_name(struct kunit_attr_filter filter);
> +
>  /*
>   * Print all test attributes for a test case or suite.
>   * Output format for test cases: "# <test_name>.<attribute>: <value>"
> @@ -16,4 +30,21 @@
>   */
>  void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level);
>
> +/*
> + * Returns the number of fitlers in input.
> + */
> +int kunit_get_filter_count(char *input);
> +
> +/*
> + * Parse attributes filter input and return an objects containing the
> + * attribute object and the string input of the next filter.
> + */
> +struct kunit_attr_filter kunit_next_attr_filter(char **filters, int *err);
> +
> +/*
> + * Returns a copy of the suite containing only tests that pass the filter.
> + */
> +struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *const suite,
> +               struct kunit_attr_filter filter, char *action, int *err);
> +
>  #endif /* _KUNIT_ATTRIBUTES_H */
> diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
> index 43dcb5de8b8f..91cbcacafba9 100644
> --- a/lib/kunit/attributes.c
> +++ b/lib/kunit/attributes.c
> @@ -67,6 +67,104 @@ static const char *attr_string_to_string(void *attr, bool *to_free)
>         return (char *) attr;
>  }
>
> +/* Filter Methods */
> +
> +static const char op_list[] = "<>!=";
> +
> +/*
> + * Returns whether the inputted integer value matches the filter given
> + * by the operation string and inputted integer.
> + */
> +static int int_filter(long val, const char *op, int input, int *err)
> +{
> +       if (!strncmp(op, "<=", 2))
> +               return (val <= input);
> +       else if (!strncmp(op, ">=", 2))
> +               return (val >= input);
> +       else if (!strncmp(op, "!=", 2))
> +               return (val != input);
> +       else if (!strncmp(op, ">", 1))
> +               return (val > input);
> +       else if (!strncmp(op, "<", 1))
> +               return (val < input);
> +       else if (!strncmp(op, "=", 1))
> +               return (val == input);
> +       *err = -EINVAL;
> +       pr_err("kunit executor: invalid filter operation: %s\n", op);

More a nitpick for the kunit.py patch, but I'd love to have this shown
to the user as an error when run under kunit.py. It's very annoying to
miss this and only get a "no tests run" error (or worse, unfiltered
results) back.


> +       return false;
> +}
> +
> +/*
> + * Returns whether the inputted enum value "attr" matches the filter given
> + * by the input string. Note: the str_list includes the corresponding string
> + * list to the enum values.
> + */
> +static int attr_enum_filter(void *attr, const char *input, int *err,
> +               const char * const str_list[], int max)
> +{
> +       int i, j, input_int;
> +       long test_val = (long)attr;
> +       const char *input_val;
> +
> +       for (i = 0; input[i]; i++) {
> +               if (!strchr(op_list, input[i])) {
> +                       input_val = input + i;
> +                       break;
> +               }
> +       }
> +
> +       if (!input_val) {
> +               *err = -EINVAL;
> +               pr_err("kunit executor: filter value not found: %s\n", input);
> +               return false;
> +       }
> +
> +       for (j = 0; j <= max; j++) {
> +               if (!strcmp(input_val, str_list[j]))
> +                       input_int = j;
> +       }
> +
> +       if (!input_int) {
> +               *err = -EINVAL;
> +               pr_err("kunit executor: invalid filter input: %s\n", input);
> +               return false;
> +       }
> +
> +       return int_filter(test_val, input, input_int, err);
> +}
> +
> +static int attr_speed_filter(void *attr, const char *input, int *err)
> +{
> +       return attr_enum_filter(attr, input, err, speed_str_list, KUNIT_SPEED_MAX);
> +}
> +
> +/*
> + * Returns whether the inputted string value (attr) matches the filter given
> + * by the input string.
> + */
> +static int attr_string_filter(void *attr, const char *input, int *err)
> +{
> +       char *str = attr;
> +
> +       if (!strncmp(input, "<", 1)) {
> +               *err = -EINVAL;
> +               pr_err("kunit executor: invalid filter input: %s\n", input);
> +               return false;
> +       } else if (!strncmp(input, ">", 1)) {
> +               *err = -EINVAL;
> +               pr_err("kunit executor: invalid filter input: %s\n", input);
> +               return false;
> +       } else if (!strncmp(input, "!=", 2)) {
> +               return (strcmp(input + 2, str) != 0);
> +       } else if (!strncmp(input, "=", 1)) {
> +               return (strcmp(input + 1, str) == 0);
> +       }
> +       *err = -EINVAL;
> +       pr_err("kunit executor: invalid filter operation: %s\n", input);
> +       return false;
> +}
> +
> +
>  /* Get Attribute Methods */
>
>  static void *attr_speed_get(void *test_or_suite, bool is_test)
> @@ -98,6 +196,7 @@ static const struct kunit_attr speed_attr = {
>         .name = "speed",
>         .get_attr = attr_speed_get,
>         .to_string = attr_speed_to_string,
> +       .filter = attr_speed_filter,
>         .attr_default = (void *)KUNIT_SPEED_NORMAL,
>         .print = PRINT_ALWAYS,
>  };
> @@ -106,6 +205,7 @@ static const struct kunit_attr module_attr = {
>         .name = "module",
>         .get_attr = attr_module_get,
>         .to_string = attr_string_to_string,
> +       .filter = attr_string_filter,
>         .attr_default = (void *)"",
>         .print = PRINT_SUITE,
>  };
> @@ -116,6 +216,11 @@ static struct kunit_attr kunit_attr_list[] = {speed_attr, module_attr};
>
>  /* Helper Functions to Access Attributes */
>
> +const char *kunit_attr_filter_name(struct kunit_attr_filter filter)
> +{
> +       return filter.attr->name;
> +}
> +
>  void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level)
>  {
>         int i;
> @@ -148,3 +253,154 @@ void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level
>                 }
>         }
>  }
> +
> +/* Helper Functions to Filter Attributes */
> +
> +int kunit_get_filter_count(char *input)
> +{
> +       int i, comma_index, count = 0;
> +
> +       for (i = 0; input[i]; i++) {
> +               if (input[i] == ',') {
> +                       if ((i - comma_index) > 1)
> +                               count++;
> +                       comma_index = i;
> +               }
> +       }
> +       if ((i - comma_index) > 1)
> +               count++;
> +       return count;
> +}
> +
> +struct kunit_attr_filter kunit_next_attr_filter(char **filters, int *err)
> +{
> +       struct kunit_attr_filter filter;
> +       int i, j, comma_index, new_start_index;
> +       int op_index = -1, attr_index = -1;
> +       char op;
> +       char *input = *filters;
> +
> +       /* Parse input until operation */
> +       for (i = 0; input[i]; i++) {
> +               if (op_index < 0 && strchr(op_list, input[i])) {
> +                       op_index = i;
> +               } else if (!comma_index && input[i] == ',') {
> +                       comma_index = i;
> +               } else if (comma_index && input[i] != ' ') {
> +                       new_start_index = i;
> +                       break;
> +               }
> +       }
> +
> +       if (op_index <= 0) {
> +               *err = -EINVAL;
> +               pr_err("kunit executor: filter operation not found: %s\n", input);
> +               return filter;
> +       }
> +
> +       /* Temporarily set operator to \0 character. */
> +       op = input[op_index];
> +       input[op_index] = '\0';
> +
> +       /* Find associated kunit_attr object */
> +       for (j = 0; j < ARRAY_SIZE(kunit_attr_list); j++) {
> +               if (!strcmp(input, kunit_attr_list[j].name)) {
> +                       attr_index = j;
> +                       break;
> +               }
> +       }
> +
> +       input[op_index] = op;
> +
> +       if (attr_index < 0) {
> +               *err = -EINVAL;
> +               pr_err("kunit executor: attribute not found: %s\n", input);
> +       } else {
> +               filter.attr = &kunit_attr_list[attr_index];
> +       }
> +
> +       if (comma_index) {
> +               input[comma_index] = '\0';
> +               filter.input = input + op_index;
> +               input = input + new_start_index;
> +       } else {
> +               filter.input = input + op_index;
> +               input = NULL;
> +       }
> +
> +       *filters = input;
> +
> +       return filter;
> +}
> +
> +struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *const suite,
> +               struct kunit_attr_filter filter, char *action, int *err)
> +{
> +       int n = 0;
> +       struct kunit_case *filtered, *test_case;
> +       struct kunit_suite *copy;
> +       void *suite_val, *test_val;
> +       bool suite_result, test_result, default_result, result;
> +
> +       /* Allocate memory for new copy of suite and list of test cases */
> +       copy = kmemdup(suite, sizeof(*copy), GFP_KERNEL);
> +       if (!copy)
> +               return ERR_PTR(-ENOMEM);
> +
> +       kunit_suite_for_each_test_case(suite, test_case) { n++; }
> +
> +       filtered = kcalloc(n + 1, sizeof(*filtered), GFP_KERNEL);
> +       if (!filtered) {
> +               kfree(copy);
> +               return ERR_PTR(-ENOMEM);
> +       }
> +
> +       n = 0;
> +
> +       /* Save filtering result on default value */
> +       default_result = filter.attr->filter(filter.attr->attr_default, filter.input, err);
> +
> +       /* Save suite attribute value and filtering result on that value */
> +       suite_val = filter.attr->get_attr((void *)suite, false);
> +       suite_result = filter.attr->filter(suite_val, filter.input, err);
> +
> +       /* For each test case, save test case if passes filtering. */
> +       kunit_suite_for_each_test_case(suite, test_case) {
> +               test_val = filter.attr->get_attr((void *) test_case, true);
> +               test_result = filter.attr->filter(filter.attr->get_attr(test_case, true),
> +                               filter.input, err);
> +
> +               /*
> +                * If attribute value of test case is set, filter on that value.
> +                * If not, filter on suite value if set. If not, filter on
> +                * default value.
> +                */
> +               result = false;
> +               if (test_val) {
> +                       if (test_result)
> +                               result = true;
> +               } else if (suite_val) {
> +                       if (suite_result)
> +                               result = true;
> +               } else if (default_result) {
> +                       result = true;
> +               }
> +
> +               if (result) {
> +                       filtered[n++] = *test_case;
> +               } else if (action && strcmp(action, "skip") == 0) {
> +                       test_case->status = KUNIT_SKIPPED;
> +                       filtered[n++] = *test_case;
> +               }
> +       }
> +
> +       if (n == 0) {
> +               kfree(copy);
> +               kfree(filtered);
> +               return NULL;
> +       }
> +
> +       copy->test_cases = filtered;
> +
> +       return copy;
> +}
> diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
> index 12e38a48a5cc..c286ae47435a 100644
> --- a/lib/kunit/executor.c
> +++ b/lib/kunit/executor.c
> @@ -17,6 +17,9 @@ extern struct kunit_suite * const __kunit_suites_end[];
>
>  static char *filter_glob_param;
>  static char *action_param;
> +static char *filter_param;
> +static char *filter_action_param;
> +
>
>  module_param_named(filter_glob, filter_glob_param, charp, 0);
>  MODULE_PARM_DESC(filter_glob,
> @@ -27,15 +30,23 @@ MODULE_PARM_DESC(action,
>                  "<none>: run the tests like normal\n"
>                  "'list' to list test names instead of running them.\n"
>                  "'list_attr' to list test names and attributes instead of running them.\n");
> +module_param_named(filter, filter_param, charp, 0);
> +MODULE_PARM_DESC(filter,
> +               "Filter which KUnit test suites/tests run at boot-time using attributes, e.g. speed>slow");
> +module_param_named(filter_action, filter_action_param, charp, 0);
> +MODULE_PARM_DESC(filter_action,
> +               "Changes behavior of filtered tests using attributes, valid values are:\n"
> +               "<none>: do not run filtered tests as normal\n"
> +               "'skip': skip all filtered tests instead so tests will appear in output\n");
>
>  /* glob_match() needs NULL terminated strings, so we need a copy of filter_glob_param. */
> -struct kunit_test_filter {
> +struct kunit_glob_filter {
>         char *suite_glob;
>         char *test_glob;
>  };
>
>  /* Split "suite_glob.test_glob" into two. Assumes filter_glob is not empty. */
> -static void kunit_parse_filter_glob(struct kunit_test_filter *parsed,
> +static void kunit_parse_glob_filter(struct kunit_glob_filter *parsed,
>                                     const char *filter_glob)
>  {
>         const int len = strlen(filter_glob);
> @@ -57,7 +68,7 @@ static void kunit_parse_filter_glob(struct kunit_test_filter *parsed,
>
>  /* Create a copy of suite with only tests that match test_glob. */
>  static struct kunit_suite *
> -kunit_filter_tests(const struct kunit_suite *const suite, const char *test_glob)
> +kunit_filter_glob_tests(const struct kunit_suite *const suite, const char *test_glob)
>  {
>         int n = 0;
>         struct kunit_case *filtered, *test_case;
> @@ -111,12 +122,15 @@ static void kunit_free_suite_set(struct suite_set suite_set)
>
>  static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
>                                             const char *filter_glob,
> +                                               char *filters,
> +                                               char *filter_action,
>                                             int *err)
>  {
> -       int i;
> -       struct kunit_suite **copy, *filtered_suite;
> +       int i, j, k, filter_count;
> +       struct kunit_suite **copy, *filtered_suite, *new_filtered_suite;
>         struct suite_set filtered;
> -       struct kunit_test_filter filter;
> +       struct kunit_glob_filter parsed_glob;
> +       struct kunit_attr_filter *parsed_filters;
>
>         const size_t max = suite_set->end - suite_set->start;
>
> @@ -127,17 +141,52 @@ static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
>                 return filtered;
>         }
>
> -       kunit_parse_filter_glob(&filter, filter_glob);
> -
> -       for (i = 0; &suite_set->start[i] != suite_set->end; i++) {
> -               if (!glob_match(filter.suite_glob, suite_set->start[i]->name))
> -                       continue;
> +       if (filter_glob)
> +               kunit_parse_glob_filter(&parsed_glob, filter_glob);
>
> -               filtered_suite = kunit_filter_tests(suite_set->start[i], filter.test_glob);
> -               if (IS_ERR(filtered_suite)) {
> -                       *err = PTR_ERR(filtered_suite);
> +       /* Parse attribute filters */
> +       if (filters) {
> +               filter_count = kunit_get_filter_count(filters);
> +               parsed_filters = kcalloc(filter_count + 1, sizeof(*parsed_filters), GFP_KERNEL);
> +               for (j = 0; j < filter_count; j++)
> +                       parsed_filters[j] = kunit_next_attr_filter(&filters, err);
> +               if (*err)
>                         return filtered;
> +       }
> +
> +       for (i = 0; &suite_set->start[i] != suite_set->end; i++) {
> +               filtered_suite = suite_set->start[i];
> +               if (filter_glob) {
> +                       if (!glob_match(parsed_glob.suite_glob, filtered_suite->name))
> +                               continue;
> +                       filtered_suite = kunit_filter_glob_tests(filtered_suite,
> +                                       parsed_glob.test_glob);
> +                       if (IS_ERR(filtered_suite)) {
> +                               *err = PTR_ERR(filtered_suite);
> +                               return filtered;
> +                       }
> +               }
> +               if (filter_count) {
> +                       for (k = 0; k < filter_count; k++) {
> +                               new_filtered_suite = kunit_filter_attr_tests(filtered_suite,
> +                                               parsed_filters[k], filter_action, err);
> +
> +                               /* Free previous copy of suite */
> +                               if (k > 0 || filter_glob)
> +                                       kfree(filtered_suite);
> +                               filtered_suite = new_filtered_suite;
> +
> +                               if (*err)
> +                                       return filtered;
> +                               if (IS_ERR(filtered_suite)) {
> +                                       *err = PTR_ERR(filtered_suite);
> +                                       return filtered;
> +                               }
> +                               if (!filtered_suite)
> +                                       break;
> +                       }
>                 }
> +
>                 if (!filtered_suite)
>                         continue;
>
> @@ -145,8 +194,14 @@ static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
>         }
>         filtered.end = copy;
>
> -       kfree(filter.suite_glob);
> -       kfree(filter.test_glob);
> +       if (filter_glob) {
> +               kfree(parsed_glob.suite_glob);
> +               kfree(parsed_glob.test_glob);
> +       }
> +
> +       if (filter_count)
> +               kfree(parsed_filters);
> +
>         return filtered;
>  }
>
> @@ -206,8 +261,9 @@ int kunit_run_all_tests(void)
>                 goto out;
>         }
>
> -       if (filter_glob_param) {
> -               suite_set = kunit_filter_suites(&suite_set, filter_glob_param, &err);
> +       if (filter_glob_param || filter_param) {
> +               suite_set = kunit_filter_suites(&suite_set, filter_glob_param,
> +                               filter_param, filter_action_param, &err);
>                 if (err) {
>                         pr_err("kunit executor: error filtering suites: %d\n", err);
>                         goto out;
> @@ -223,7 +279,7 @@ int kunit_run_all_tests(void)
>         else
>                 pr_err("kunit executor: unknown action '%s'\n", action_param);
>
> -       if (filter_glob_param) { /* a copy was made of each suite */
> +       if (filter_glob_param || filter_param) { /* a copy was made of each suite */
>                 kunit_free_suite_set(suite_set);
>         }
>
> diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c
> index ce6749af374d..d7ab069324b5 100644
> --- a/lib/kunit/executor_test.c
> +++ b/lib/kunit/executor_test.c
> @@ -24,15 +24,15 @@ static struct kunit_case dummy_test_cases[] = {
>
>  static void parse_filter_test(struct kunit *test)
>  {
> -       struct kunit_test_filter filter = {NULL, NULL};
> +       struct kunit_glob_filter filter = {NULL, NULL};
>
> -       kunit_parse_filter_glob(&filter, "suite");
> +       kunit_parse_glob_filter(&filter, "suite");
>         KUNIT_EXPECT_STREQ(test, filter.suite_glob, "suite");
>         KUNIT_EXPECT_FALSE(test, filter.test_glob);
>         kfree(filter.suite_glob);
>         kfree(filter.test_glob);
>
> -       kunit_parse_filter_glob(&filter, "suite.test");
> +       kunit_parse_glob_filter(&filter, "suite.test");
>         KUNIT_EXPECT_STREQ(test, filter.suite_glob, "suite");
>         KUNIT_EXPECT_STREQ(test, filter.test_glob, "test");
>         kfree(filter.suite_glob);
> @@ -50,7 +50,7 @@ static void filter_suites_test(struct kunit *test)
>         subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases);
>
>         /* Want: suite1, suite2, NULL -> suite2, NULL */
> -       got = kunit_filter_suites(&suite_set, "suite2", &err);
> +       got = kunit_filter_suites(&suite_set, "suite2", NULL, NULL, &err);
>         KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
>         KUNIT_ASSERT_EQ(test, err, 0);
>         kfree_at_end(test, got.start);
> @@ -74,7 +74,7 @@ static void filter_suites_test_glob_test(struct kunit *test)
>         subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases);
>
>         /* Want: suite1, suite2, NULL -> suite2 (just test1), NULL */
> -       got = kunit_filter_suites(&suite_set, "suite2.test2", &err);
> +       got = kunit_filter_suites(&suite_set, "suite2.test2", NULL, NULL, &err);
>         KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
>         KUNIT_ASSERT_EQ(test, err, 0);
>         kfree_at_end(test, got.start);
> @@ -100,7 +100,7 @@ static void filter_suites_to_empty_test(struct kunit *test)
>         subsuite[0] = alloc_fake_suite(test, "suite1", dummy_test_cases);
>         subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases);
>
> -       got = kunit_filter_suites(&suite_set, "not_found", &err);
> +       got = kunit_filter_suites(&suite_set, "not_found", NULL, NULL, &err);
>         KUNIT_ASSERT_EQ(test, err, 0);
>         kfree_at_end(test, got.start); /* just in case */
>
> diff --git a/lib/kunit/test.c b/lib/kunit/test.c
> index 9ee55139ecd1..cb9797fa6303 100644
> --- a/lib/kunit/test.c
> +++ b/lib/kunit/test.c
> @@ -613,18 +613,22 @@ int kunit_run_tests(struct kunit_suite *suite)
>         kunit_suite_for_each_test_case(suite, test_case) {
>                 struct kunit test = { .param_value = NULL, .param_index = 0 };
>                 struct kunit_result_stats param_stats = { 0 };
> -               test_case->status = KUNIT_SKIPPED;
>
>                 kunit_init_test(&test, test_case->name, test_case->log);
> -
> -               if (!test_case->generate_params) {
> +               if (test_case->status == KUNIT_SKIPPED) {
> +                       /* Test marked as skip */
> +                       test.status = KUNIT_SKIPPED;
> +                       kunit_update_stats(&param_stats, test.status);
> +               } else if (!test_case->generate_params) {
>                         /* Non-parameterised test. */
> +                       test_case->status = KUNIT_SKIPPED;
>                         kunit_run_case_catch_errors(suite, test_case, &test);
>                         kunit_update_stats(&param_stats, test.status);
>                 } else {
>                         /* Get initial param. */
>                         param_desc[0] = '\0';
>                         test.param_value = test_case->generate_params(NULL, param_desc);
> +                       test_case->status = KUNIT_SKIPPED;
>                         kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT
>                                   "KTAP version 1\n");
>                         kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT
> --
> 2.41.0.255.g8b1d071c50-goog
>

--0000000000007952c70600be0389
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIIPnwYJKoZIhvcNAQcCoIIPkDCCD4wCAQExDzANBglghkgBZQMEAgEFADALBgkqhkiG9w0BBwGg
ggz5MIIEtjCCA56gAwIBAgIQeAMYYHb81ngUVR0WyMTzqzANBgkqhkiG9w0BAQsFADBMMSAwHgYD
VQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMzETMBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UE
AxMKR2xvYmFsU2lnbjAeFw0yMDA3MjgwMDAwMDBaFw0yOTAzMTgwMDAwMDBaMFQxCzAJBgNVBAYT
AkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSowKAYDVQQDEyFHbG9iYWxTaWduIEF0bGFz
IFIzIFNNSU1FIENBIDIwMjAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvLe9xPU9W
dpiHLAvX7kFnaFZPuJLey7LYaMO8P/xSngB9IN73mVc7YiLov12Fekdtn5kL8PjmDBEvTYmWsuQS
6VBo3vdlqqXZ0M9eMkjcKqijrmDRleudEoPDzTumwQ18VB/3I+vbN039HIaRQ5x+NHGiPHVfk6Rx
c6KAbYceyeqqfuJEcq23vhTdium/Bf5hHqYUhuJwnBQ+dAUcFndUKMJrth6lHeoifkbw2bv81zxJ
I9cvIy516+oUekqiSFGfzAqByv41OrgLV4fLGCDH3yRh1tj7EtV3l2TngqtrDLUs5R+sWIItPa/4
AJXB1Q3nGNl2tNjVpcSn0uJ7aFPbAgMBAAGjggGKMIIBhjAOBgNVHQ8BAf8EBAMCAYYwHQYDVR0l
BBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFHzM
CmjXouseLHIb0c1dlW+N+/JjMB8GA1UdIwQYMBaAFI/wS3+oLkUkrk1Q+mOai97i3Ru8MHsGCCsG
AQUFBwEBBG8wbTAuBggrBgEFBQcwAYYiaHR0cDovL29jc3AyLmdsb2JhbHNpZ24uY29tL3Jvb3Ry
MzA7BggrBgEFBQcwAoYvaHR0cDovL3NlY3VyZS5nbG9iYWxzaWduLmNvbS9jYWNlcnQvcm9vdC1y
My5jcnQwNgYDVR0fBC8wLTAroCmgJ4YlaHR0cDovL2NybC5nbG9iYWxzaWduLmNvbS9yb290LXIz
LmNybDBMBgNVHSAERTBDMEEGCSsGAQQBoDIBKDA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5n
bG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5LzANBgkqhkiG9w0BAQsFAAOCAQEANyYcO+9JZYyqQt41
TMwvFWAw3vLoLOQIfIn48/yea/ekOcParTb0mbhsvVSZ6sGn+txYAZb33wIb1f4wK4xQ7+RUYBfI
TuTPL7olF9hDpojC2F6Eu8nuEf1XD9qNI8zFd4kfjg4rb+AME0L81WaCL/WhP2kDCnRU4jm6TryB
CHhZqtxkIvXGPGHjwJJazJBnX5NayIce4fGuUEJ7HkuCthVZ3Rws0UyHSAXesT/0tXATND4mNr1X
El6adiSQy619ybVERnRi5aDe1PTwE+qNiotEEaeujz1a/+yYaaTY+k+qJcVxi7tbyQ0hi0UB3myM
A/z2HmGEwO8hx7hDjKmKbDCCA18wggJHoAMCAQICCwQAAAAAASFYUwiiMA0GCSqGSIb3DQEBCwUA
MEwxIDAeBgNVBAsTF0dsb2JhbFNpZ24gUm9vdCBDQSAtIFIzMRMwEQYDVQQKEwpHbG9iYWxTaWdu
MRMwEQYDVQQDEwpHbG9iYWxTaWduMB4XDTA5MDMxODEwMDAwMFoXDTI5MDMxODEwMDAwMFowTDEg
MB4GA1UECxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjMxEzARBgNVBAoTCkdsb2JhbFNpZ24xEzAR
BgNVBAMTCkdsb2JhbFNpZ24wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDMJXaQeQZ4
Ihb1wIO2hMoonv0FdhHFrYhy/EYCQ8eyip0EXyTLLkvhYIJG4VKrDIFHcGzdZNHr9SyjD4I9DCuu
l9e2FIYQebs7E4B3jAjhSdJqYi8fXvqWaN+JJ5U4nwbXPsnLJlkNc96wyOkmDoMVxu9bi9IEYMpJ
pij2aTv2y8gokeWdimFXN6x0FNx04Druci8unPvQu7/1PQDhBjPogiuuU6Y6FnOM3UEOIDrAtKeh
6bJPkC4yYOlXy7kEkmho5TgmYHWyn3f/kRTvriBJ/K1AFUjRAjFhGV64l++td7dkmnq/X8ET75ti
+w1s4FRpFqkD2m7pg5NxdsZphYIXAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8E
BTADAQH/MB0GA1UdDgQWBBSP8Et/qC5FJK5NUPpjmove4t0bvDANBgkqhkiG9w0BAQsFAAOCAQEA
S0DbwFCq/sgM7/eWVEVJu5YACUGssxOGhigHM8pr5nS5ugAtrqQK0/Xx8Q+Kv3NnSoPHRHt44K9u
bG8DKY4zOUXDjuS5V2yq/BKW7FPGLeQkbLmUY/vcU2hnVj6DuM81IcPJaP7O2sJTqsyQiunwXUaM
ld16WCgaLx3ezQA3QY/tRG3XUyiXfvNnBB4V14qWtNPeTCekTBtzc3b0F5nCH3oO4y0IrQocLP88
q1UOD5F+NuvDV0m+4S4tfGCLw0FREyOdzvcya5QBqJnnLDMfOjsl0oZAzjsshnjJYS8Uuu7bVW/f
hO4FCU29KNhyztNiUGUe65KXgzHZs7XKR1g/XzCCBNgwggPAoAMCAQICEAEDPnEOWzT2vYIrJhGq
c1swDQYJKoZIhvcNAQELBQAwVDELMAkGA1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYt
c2ExKjAoBgNVBAMTIUdsb2JhbFNpZ24gQXRsYXMgUjMgU01JTUUgQ0EgMjAyMDAeFw0yMzA1MTIx
NjMzMjlaFw0yMzExMDgxNjMzMjlaMCQxIjAgBgkqhkiG9w0BCQEWE2RhdmlkZ293QGdvb2dsZS5j
b20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCfIQuFV9ECjSKrnHc+/gEoEHeMu29G
hkC9x5KA7Tgm7ZISSdxxP+b9Q23vqKKYcaXlXzxDUweAEa7KrhRdZMpcF1p14/qI6AG7rBn8otbO
t6QSE9nwXQRL5ITEHtPRcQzLU5H9Yyq4b9MmEZAq+ByKX1t6FrXw461kqV8I/oCueKmD0p6mU/4k
xzQWik4ZqST0MXkJiZenSKDDN+U1qGgHKC3HAzsIlWpNh/WsWcD4RRcEtwfW1h9DwRfGFp78OFQg
65qXbeub4G7ELSIdjGygCzVG+g1jo6we5uqPep3iRCzn92KROEVxP5lG9FlwQ2YWMt+dNiGrJdKy
Kw4TK7CrAgMBAAGjggHUMIIB0DAeBgNVHREEFzAVgRNkYXZpZGdvd0Bnb29nbGUuY29tMA4GA1Ud
DwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDBAYIKwYBBQUHAwIwHQYDVR0OBBYEFG/UTu3x
9IGQSBx2i4m+hGXJpET+MEwGA1UdIARFMEMwQQYJKwYBBAGgMgEoMDQwMgYIKwYBBQUHAgEWJmh0
dHBzOi8vd3d3Lmdsb2JhbHNpZ24uY29tL3JlcG9zaXRvcnkvMAwGA1UdEwEB/wQCMAAwgZoGCCsG
AQUFBwEBBIGNMIGKMD4GCCsGAQUFBzABhjJodHRwOi8vb2NzcC5nbG9iYWxzaWduLmNvbS9jYS9n
c2F0bGFzcjNzbWltZWNhMjAyMDBIBggrBgEFBQcwAoY8aHR0cDovL3NlY3VyZS5nbG9iYWxzaWdu
LmNvbS9jYWNlcnQvZ3NhdGxhc3Izc21pbWVjYTIwMjAuY3J0MB8GA1UdIwQYMBaAFHzMCmjXouse
LHIb0c1dlW+N+/JjMEYGA1UdHwQ/MD0wO6A5oDeGNWh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20v
Y2EvZ3NhdGxhc3Izc21pbWVjYTIwMjAuY3JsMA0GCSqGSIb3DQEBCwUAA4IBAQCRI3Z4cAidgFcv
Usqdz765x6KMZSfg/WtFrYg8ewsP2NpCxVM2+EhPyyEQ0k0DhtzdtGoI/Ug+jdFDyCKB9P2+EPLh
iMjMnFILp7Zs4r18ECHlvZuDZfH9m0BchXIxu5jLIuQyKUWrCRDZZEDNr510ZhhVfYSFPA8ms1nk
jyzYFOHYQyv5IfML/3IBFKlON5OZa+V8EZYULYcNkp03DdWglafj7SXZ1/XgAbVYrC381UvrsYN8
jndVvoa1GWwe+NVlIIK7Q3uAjV3qLEDQpaNPg1rr0oAn6YmvTccjVMqj2YNwN+RHhKNzgRGxY5ct
FaN+8fXZhRhpv3bVbAWuPZXoMYICajCCAmYCAQEwaDBUMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQ
R2xvYmFsU2lnbiBudi1zYTEqMCgGA1UEAxMhR2xvYmFsU2lnbiBBdGxhcyBSMyBTTUlNRSBDQSAy
MDIwAhABAz5xDls09r2CKyYRqnNbMA0GCWCGSAFlAwQCAQUAoIHUMC8GCSqGSIb3DQEJBDEiBCBp
K07R+bpnWRC275Tz2GehCRocXYIGZsFC/FZg1NpdXDAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcB
MBwGCSqGSIb3DQEJBTEPFw0yMzA3MTgwNzM5MjJaMGkGCSqGSIb3DQEJDzFcMFowCwYJYIZIAWUD
BAEqMAsGCWCGSAFlAwQBFjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwCwYJKoZIhvcNAQEKMAsG
CSqGSIb3DQEBBzALBglghkgBZQMEAgEwDQYJKoZIhvcNAQEBBQAEggEAGvjplSUvI1sr+0hZu8Yb
XrbCuVv13VzXv9nsb0Luo6r/PY87SgkvOfG8uxlIhqvioI2iOqmPW0DmmGEUC7sZT2Q8AbX2UG81
K4qvG8zaqHXbXG8FMgTMaiGUw3YQtpRlaVbps80aQpr2BiYl0ETt11quLJSZBQ4OPEPJb2gWCSkL
6JgKG1T/M7uNxfQP8UdoqFR7xKhXSQKLJDFDbTUS8Or4rOmgWttQR/GCMar+Hw5d6Jb/171UZmww
kz2SBM58gLNMd8HhXs9ym73kTKZOFJhRM81d/zBMKawHFlQN3n2E6NvdXzFFCqgT2VAh+brmAk8a
3jGa4OS3xm8bGPgqQA==
--0000000000007952c70600be0389--