2020-05-01 08:36:40

by Anders Roxell

[permalink] [raw]
Subject: [PATCH] kunit: Kconfig: enable a KUNIT_RUN_ALL fragment

Make it easier to enable all KUnit fragments. This is needed for kernel
test-systems, so its easy to get all KUnit tests enabled and if new gets
added they will be enabled as well. Fragments that has to be builtin
will be missed if CONFIG_KUNIT_RUN_ALL is set as a module.

Adding 'if !KUNIT_RUN_ALL' so individual test can be turned of if
someone wants that even though KUNIT_RUN_ALL is enabled.

Signed-off-by: Anders Roxell <[email protected]>
---
drivers/base/Kconfig | 3 ++-
drivers/base/test/Kconfig | 3 ++-
fs/ext4/Kconfig | 3 ++-
lib/Kconfig.debug | 6 ++++--
lib/Kconfig.kcsan | 3 ++-
lib/kunit/Kconfig | 15 ++++++++++++---
security/apparmor/Kconfig | 3 ++-
7 files changed, 26 insertions(+), 10 deletions(-)

diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
index 5f0bc74d2409..c48e6e4ef367 100644
--- a/drivers/base/Kconfig
+++ b/drivers/base/Kconfig
@@ -149,8 +149,9 @@ config DEBUG_TEST_DRIVER_REMOVE
test this functionality.

config PM_QOS_KUNIT_TEST
- bool "KUnit Test for PM QoS features"
+ bool "KUnit Test for PM QoS features" if !KUNIT_RUN_ALL
depends on KUNIT=y
+ default KUNIT_RUN_ALL

config HMEM_REPORTING
bool
diff --git a/drivers/base/test/Kconfig b/drivers/base/test/Kconfig
index 305c7751184a..0d662d689f6b 100644
--- a/drivers/base/test/Kconfig
+++ b/drivers/base/test/Kconfig
@@ -9,5 +9,6 @@ config TEST_ASYNC_DRIVER_PROBE

If unsure say N.
config KUNIT_DRIVER_PE_TEST
- bool "KUnit Tests for property entry API"
+ bool "KUnit Tests for property entry API" if !KUNIT_RUN_ALL
depends on KUNIT=y
+ default KUNIT_RUN_ALL
diff --git a/fs/ext4/Kconfig b/fs/ext4/Kconfig
index 2a592e38cdfe..76785143259d 100644
--- a/fs/ext4/Kconfig
+++ b/fs/ext4/Kconfig
@@ -103,9 +103,10 @@ config EXT4_DEBUG
echo 1 > /sys/module/ext4/parameters/mballoc_debug

config EXT4_KUNIT_TESTS
- tristate "KUnit tests for ext4"
+ tristate "KUnit tests for ext4" if !KUNIT_RUN_ALL
select EXT4_FS
depends on KUNIT
+ default KUNIT_RUN_ALL
help
This builds the ext4 KUnit tests.

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 8e4aded46281..993e0c5549bc 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2123,8 +2123,9 @@ config TEST_SYSCTL
If unsure, say N.

config SYSCTL_KUNIT_TEST
- tristate "KUnit test for sysctl"
+ tristate "KUnit test for sysctl" if !KUNIT_RUN_ALL
depends on KUNIT
+ default KUNIT_RUN_ALL
help
This builds the proc sysctl unit test, which runs on boot.
Tests the API contract and implementation correctness of sysctl.
@@ -2134,8 +2135,9 @@ config SYSCTL_KUNIT_TEST
If unsure, say N.

config LIST_KUNIT_TEST
- tristate "KUnit Test for Kernel Linked-list structures"
+ tristate "KUnit Test for Kernel Linked-list structures" if !KUNIT_RUN_ALL
depends on KUNIT
+ default KUNIT_RUN_ALL
help
This builds the linked list KUnit test suite.
It tests that the API and basic functionality of the list_head type
diff --git a/lib/Kconfig.kcsan b/lib/Kconfig.kcsan
index ea28245c6c1d..91398300a1bc 100644
--- a/lib/Kconfig.kcsan
+++ b/lib/Kconfig.kcsan
@@ -46,8 +46,9 @@ config KCSAN_SELFTEST
works as intended.

config KCSAN_TEST
- tristate "KCSAN test for integrated runtime behaviour"
+ tristate "KCSAN test for integrated runtime behaviour" if !KUNIT_RUN_ALL
depends on TRACEPOINTS && KUNIT
+ default KUNIT_RUN_ALL
select TORTURE_TEST
help
KCSAN test focusing on behaviour of the integrated runtime. Tests
diff --git a/lib/kunit/Kconfig b/lib/kunit/Kconfig
index 95d12e3d6d95..d6a912779816 100644
--- a/lib/kunit/Kconfig
+++ b/lib/kunit/Kconfig
@@ -15,7 +15,8 @@ menuconfig KUNIT
if KUNIT

config KUNIT_DEBUGFS
- bool "KUnit - Enable /sys/kernel/debug/kunit debugfs representation"
+ bool "KUnit - Enable /sys/kernel/debug/kunit debugfs representation" if !KUNIT_RUN_ALL
+ default KUNIT_RUN_ALL
help
Enable debugfs representation for kunit. Currently this consists
of /sys/kernel/debug/kunit/<test_suite>/results files for each
@@ -23,7 +24,8 @@ config KUNIT_DEBUGFS
run that occurred.

config KUNIT_TEST
- tristate "KUnit test for KUnit"
+ tristate "KUnit test for KUnit" if !KUNIT_RUN_ALL
+ default KUNIT_RUN_ALL
help
Enables the unit tests for the KUnit test framework. These tests test
the KUnit test framework itself; the tests are both written using
@@ -32,7 +34,8 @@ config KUNIT_TEST
expected.

config KUNIT_EXAMPLE_TEST
- tristate "Example test for KUnit"
+ tristate "Example test for KUnit" if !KUNIT_RUN_ALL
+ default KUNIT_RUN_ALL
help
Enables an example unit test that illustrates some of the basic
features of KUnit. This test only exists to help new users understand
@@ -41,4 +44,10 @@ config KUNIT_EXAMPLE_TEST
is intended for curious hackers who would like to understand how to
use KUnit for kernel development.

+config KUNIT_RUN_ALL
+ tristate "KUnit run all test"
+ help
+ Enables all KUnit tests. If they can be enabled.
+ That depends on if KUnit is enabled as a module or builtin.
+
endif # KUNIT
diff --git a/security/apparmor/Kconfig b/security/apparmor/Kconfig
index 0fe336860773..c4648426ea5d 100644
--- a/security/apparmor/Kconfig
+++ b/security/apparmor/Kconfig
@@ -70,8 +70,9 @@ config SECURITY_APPARMOR_DEBUG_MESSAGES
the kernel message buffer.

config SECURITY_APPARMOR_KUNIT_TEST
- bool "Build KUnit tests for policy_unpack.c"
+ bool "Build KUnit tests for policy_unpack.c" if !KUNIT_RUN_ALL
depends on KUNIT=y && SECURITY_APPARMOR
+ default KUNIT_RUN_ALL
help
This builds the AppArmor KUnit tests.

--
2.20.1


2020-05-01 09:58:33

by Marco Elver

[permalink] [raw]
Subject: Re: [PATCH] kunit: Kconfig: enable a KUNIT_RUN_ALL fragment

On Fri, 1 May 2020 at 10:35, Anders Roxell <[email protected]> wrote:
>
> Make it easier to enable all KUnit fragments. This is needed for kernel
> test-systems, so its easy to get all KUnit tests enabled and if new gets
> added they will be enabled as well. Fragments that has to be builtin
> will be missed if CONFIG_KUNIT_RUN_ALL is set as a module.
>
> Adding 'if !KUNIT_RUN_ALL' so individual test can be turned of if
> someone wants that even though KUNIT_RUN_ALL is enabled.
>
> Signed-off-by: Anders Roxell <[email protected]>
> ---
> drivers/base/Kconfig | 3 ++-
> drivers/base/test/Kconfig | 3 ++-
> fs/ext4/Kconfig | 3 ++-
> lib/Kconfig.debug | 6 ++++--
> lib/Kconfig.kcsan | 3 ++-
> lib/kunit/Kconfig | 15 ++++++++++++---
> security/apparmor/Kconfig | 3 ++-
> 7 files changed, 26 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
> index 5f0bc74d2409..c48e6e4ef367 100644
> --- a/drivers/base/Kconfig
> +++ b/drivers/base/Kconfig
> @@ -149,8 +149,9 @@ config DEBUG_TEST_DRIVER_REMOVE
> test this functionality.
>
> config PM_QOS_KUNIT_TEST
> - bool "KUnit Test for PM QoS features"
> + bool "KUnit Test for PM QoS features" if !KUNIT_RUN_ALL
> depends on KUNIT=y
> + default KUNIT_RUN_ALL
>
> config HMEM_REPORTING
> bool
> diff --git a/drivers/base/test/Kconfig b/drivers/base/test/Kconfig
> index 305c7751184a..0d662d689f6b 100644
> --- a/drivers/base/test/Kconfig
> +++ b/drivers/base/test/Kconfig
> @@ -9,5 +9,6 @@ config TEST_ASYNC_DRIVER_PROBE
>
> If unsure say N.
> config KUNIT_DRIVER_PE_TEST
> - bool "KUnit Tests for property entry API"
> + bool "KUnit Tests for property entry API" if !KUNIT_RUN_ALL
> depends on KUNIT=y
> + default KUNIT_RUN_ALL
> diff --git a/fs/ext4/Kconfig b/fs/ext4/Kconfig
> index 2a592e38cdfe..76785143259d 100644
> --- a/fs/ext4/Kconfig
> +++ b/fs/ext4/Kconfig
> @@ -103,9 +103,10 @@ config EXT4_DEBUG
> echo 1 > /sys/module/ext4/parameters/mballoc_debug
>
> config EXT4_KUNIT_TESTS
> - tristate "KUnit tests for ext4"
> + tristate "KUnit tests for ext4" if !KUNIT_RUN_ALL
> select EXT4_FS
> depends on KUNIT
> + default KUNIT_RUN_ALL
> help
> This builds the ext4 KUnit tests.
>
> diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> index 8e4aded46281..993e0c5549bc 100644
> --- a/lib/Kconfig.debug
> +++ b/lib/Kconfig.debug
> @@ -2123,8 +2123,9 @@ config TEST_SYSCTL
> If unsure, say N.
>
> config SYSCTL_KUNIT_TEST
> - tristate "KUnit test for sysctl"
> + tristate "KUnit test for sysctl" if !KUNIT_RUN_ALL
> depends on KUNIT
> + default KUNIT_RUN_ALL
> help
> This builds the proc sysctl unit test, which runs on boot.
> Tests the API contract and implementation correctness of sysctl.
> @@ -2134,8 +2135,9 @@ config SYSCTL_KUNIT_TEST
> If unsure, say N.
>
> config LIST_KUNIT_TEST
> - tristate "KUnit Test for Kernel Linked-list structures"
> + tristate "KUnit Test for Kernel Linked-list structures" if !KUNIT_RUN_ALL
> depends on KUNIT
> + default KUNIT_RUN_ALL
> help
> This builds the linked list KUnit test suite.
> It tests that the API and basic functionality of the list_head type
> diff --git a/lib/Kconfig.kcsan b/lib/Kconfig.kcsan
> index ea28245c6c1d..91398300a1bc 100644
> --- a/lib/Kconfig.kcsan
> +++ b/lib/Kconfig.kcsan
> @@ -46,8 +46,9 @@ config KCSAN_SELFTEST
> works as intended.
>
> config KCSAN_TEST
> - tristate "KCSAN test for integrated runtime behaviour"
> + tristate "KCSAN test for integrated runtime behaviour" if !KUNIT_RUN_ALL
> depends on TRACEPOINTS && KUNIT
> + default KUNIT_RUN_ALL
> select TORTURE_TEST
> help
> KCSAN test focusing on behaviour of the integrated runtime. Tests

I think if you want this patch to be picked up you need to split it,
with one patch for each test that is not yet in mainline or the tree
that should pick this patch up.

The KCSAN test is in the -rcu tree, but I don't expect it to be merged
before 5.9. Most likely, we would only be able to pick up the patch
that would make the chance to the KCSAN Kconfig entry once the rest
here made it into mainline.

Thanks,
-- Marco

> diff --git a/lib/kunit/Kconfig b/lib/kunit/Kconfig
> index 95d12e3d6d95..d6a912779816 100644
> --- a/lib/kunit/Kconfig
> +++ b/lib/kunit/Kconfig
> @@ -15,7 +15,8 @@ menuconfig KUNIT
> if KUNIT
>
> config KUNIT_DEBUGFS
> - bool "KUnit - Enable /sys/kernel/debug/kunit debugfs representation"
> + bool "KUnit - Enable /sys/kernel/debug/kunit debugfs representation" if !KUNIT_RUN_ALL
> + default KUNIT_RUN_ALL
> help
> Enable debugfs representation for kunit. Currently this consists
> of /sys/kernel/debug/kunit/<test_suite>/results files for each
> @@ -23,7 +24,8 @@ config KUNIT_DEBUGFS
> run that occurred.
>
> config KUNIT_TEST
> - tristate "KUnit test for KUnit"
> + tristate "KUnit test for KUnit" if !KUNIT_RUN_ALL
> + default KUNIT_RUN_ALL
> help
> Enables the unit tests for the KUnit test framework. These tests test
> the KUnit test framework itself; the tests are both written using
> @@ -32,7 +34,8 @@ config KUNIT_TEST
> expected.
>
> config KUNIT_EXAMPLE_TEST
> - tristate "Example test for KUnit"
> + tristate "Example test for KUnit" if !KUNIT_RUN_ALL
> + default KUNIT_RUN_ALL
> help
> Enables an example unit test that illustrates some of the basic
> features of KUnit. This test only exists to help new users understand
> @@ -41,4 +44,10 @@ config KUNIT_EXAMPLE_TEST
> is intended for curious hackers who would like to understand how to
> use KUnit for kernel development.
>
> +config KUNIT_RUN_ALL
> + tristate "KUnit run all test"
> + help
> + Enables all KUnit tests. If they can be enabled.
> + That depends on if KUnit is enabled as a module or builtin.
> +

s/tests. If/tests, if/ ?

> endif # KUNIT
> diff --git a/security/apparmor/Kconfig b/security/apparmor/Kconfig
> index 0fe336860773..c4648426ea5d 100644
> --- a/security/apparmor/Kconfig
> +++ b/security/apparmor/Kconfig
> @@ -70,8 +70,9 @@ config SECURITY_APPARMOR_DEBUG_MESSAGES
> the kernel message buffer.
>
> config SECURITY_APPARMOR_KUNIT_TEST
> - bool "Build KUnit tests for policy_unpack.c"
> + bool "Build KUnit tests for policy_unpack.c" if !KUNIT_RUN_ALL
> depends on KUNIT=y && SECURITY_APPARMOR
> + default KUNIT_RUN_ALL
> help
> This builds the AppArmor KUnit tests.
>
> --
> 2.20.1
>

2020-05-01 16:15:37

by Anders Roxell

[permalink] [raw]
Subject: Re: [PATCH] kunit: Kconfig: enable a KUNIT_RUN_ALL fragment

On Fri, 1 May 2020 at 11:57, Marco Elver <[email protected]> wrote:
>
> On Fri, 1 May 2020 at 10:35, Anders Roxell <[email protected]> wrote:
> >
> > Make it easier to enable all KUnit fragments. This is needed for kernel
> > test-systems, so its easy to get all KUnit tests enabled and if new gets
> > added they will be enabled as well. Fragments that has to be builtin
> > will be missed if CONFIG_KUNIT_RUN_ALL is set as a module.
> >
> > Adding 'if !KUNIT_RUN_ALL' so individual test can be turned of if
> > someone wants that even though KUNIT_RUN_ALL is enabled.
> >
> > Signed-off-by: Anders Roxell <[email protected]>
> > ---
> > drivers/base/Kconfig | 3 ++-
> > drivers/base/test/Kconfig | 3 ++-
> > fs/ext4/Kconfig | 3 ++-
> > lib/Kconfig.debug | 6 ++++--
> > lib/Kconfig.kcsan | 3 ++-
> > lib/kunit/Kconfig | 15 ++++++++++++---
> > security/apparmor/Kconfig | 3 ++-
> > 7 files changed, 26 insertions(+), 10 deletions(-)
> >
> > diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
> > index 5f0bc74d2409..c48e6e4ef367 100644
> > --- a/drivers/base/Kconfig
> > +++ b/drivers/base/Kconfig
> > @@ -149,8 +149,9 @@ config DEBUG_TEST_DRIVER_REMOVE
> > test this functionality.
> >
> > config PM_QOS_KUNIT_TEST
> > - bool "KUnit Test for PM QoS features"
> > + bool "KUnit Test for PM QoS features" if !KUNIT_RUN_ALL
> > depends on KUNIT=y
> > + default KUNIT_RUN_ALL
> >
> > config HMEM_REPORTING
> > bool
> > diff --git a/drivers/base/test/Kconfig b/drivers/base/test/Kconfig
> > index 305c7751184a..0d662d689f6b 100644
> > --- a/drivers/base/test/Kconfig
> > +++ b/drivers/base/test/Kconfig
> > @@ -9,5 +9,6 @@ config TEST_ASYNC_DRIVER_PROBE
> >
> > If unsure say N.
> > config KUNIT_DRIVER_PE_TEST
> > - bool "KUnit Tests for property entry API"
> > + bool "KUnit Tests for property entry API" if !KUNIT_RUN_ALL
> > depends on KUNIT=y
> > + default KUNIT_RUN_ALL
> > diff --git a/fs/ext4/Kconfig b/fs/ext4/Kconfig
> > index 2a592e38cdfe..76785143259d 100644
> > --- a/fs/ext4/Kconfig
> > +++ b/fs/ext4/Kconfig
> > @@ -103,9 +103,10 @@ config EXT4_DEBUG
> > echo 1 > /sys/module/ext4/parameters/mballoc_debug
> >
> > config EXT4_KUNIT_TESTS
> > - tristate "KUnit tests for ext4"
> > + tristate "KUnit tests for ext4" if !KUNIT_RUN_ALL
> > select EXT4_FS
> > depends on KUNIT
> > + default KUNIT_RUN_ALL
> > help
> > This builds the ext4 KUnit tests.
> >
> > diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> > index 8e4aded46281..993e0c5549bc 100644
> > --- a/lib/Kconfig.debug
> > +++ b/lib/Kconfig.debug
> > @@ -2123,8 +2123,9 @@ config TEST_SYSCTL
> > If unsure, say N.
> >
> > config SYSCTL_KUNIT_TEST
> > - tristate "KUnit test for sysctl"
> > + tristate "KUnit test for sysctl" if !KUNIT_RUN_ALL
> > depends on KUNIT
> > + default KUNIT_RUN_ALL
> > help
> > This builds the proc sysctl unit test, which runs on boot.
> > Tests the API contract and implementation correctness of sysctl.
> > @@ -2134,8 +2135,9 @@ config SYSCTL_KUNIT_TEST
> > If unsure, say N.
> >
> > config LIST_KUNIT_TEST
> > - tristate "KUnit Test for Kernel Linked-list structures"
> > + tristate "KUnit Test for Kernel Linked-list structures" if !KUNIT_RUN_ALL
> > depends on KUNIT
> > + default KUNIT_RUN_ALL
> > help
> > This builds the linked list KUnit test suite.
> > It tests that the API and basic functionality of the list_head type
> > diff --git a/lib/Kconfig.kcsan b/lib/Kconfig.kcsan
> > index ea28245c6c1d..91398300a1bc 100644
> > --- a/lib/Kconfig.kcsan
> > +++ b/lib/Kconfig.kcsan
> > @@ -46,8 +46,9 @@ config KCSAN_SELFTEST
> > works as intended.
> >
> > config KCSAN_TEST
> > - tristate "KCSAN test for integrated runtime behaviour"
> > + tristate "KCSAN test for integrated runtime behaviour" if !KUNIT_RUN_ALL
> > depends on TRACEPOINTS && KUNIT
> > + default KUNIT_RUN_ALL
> > select TORTURE_TEST
> > help
> > KCSAN test focusing on behaviour of the integrated runtime. Tests
>
> I think if you want this patch to be picked up you need to split it,
> with one patch for each test that is not yet in mainline or the tree
> that should pick this patch up.

OK, would it be ok to do one patch per subsystem if it's in the mainline tree,
and another patch if it's only in the next tree for the same subsystem?

>
> The KCSAN test is in the -rcu tree, but I don't expect it to be merged
> before 5.9. Most likely, we would only be able to pick up the patch
> that would make the chance to the KCSAN Kconfig entry once the rest
> here made it into mainline.
>
> Thanks,
> -- Marco
>
> > diff --git a/lib/kunit/Kconfig b/lib/kunit/Kconfig
> > index 95d12e3d6d95..d6a912779816 100644
> > --- a/lib/kunit/Kconfig
> > +++ b/lib/kunit/Kconfig
> > @@ -15,7 +15,8 @@ menuconfig KUNIT
> > if KUNIT
> >
> > config KUNIT_DEBUGFS
> > - bool "KUnit - Enable /sys/kernel/debug/kunit debugfs representation"
> > + bool "KUnit - Enable /sys/kernel/debug/kunit debugfs representation" if !KUNIT_RUN_ALL
> > + default KUNIT_RUN_ALL
> > help
> > Enable debugfs representation for kunit. Currently this consists
> > of /sys/kernel/debug/kunit/<test_suite>/results files for each
> > @@ -23,7 +24,8 @@ config KUNIT_DEBUGFS
> > run that occurred.
> >
> > config KUNIT_TEST
> > - tristate "KUnit test for KUnit"
> > + tristate "KUnit test for KUnit" if !KUNIT_RUN_ALL
> > + default KUNIT_RUN_ALL
> > help
> > Enables the unit tests for the KUnit test framework. These tests test
> > the KUnit test framework itself; the tests are both written using
> > @@ -32,7 +34,8 @@ config KUNIT_TEST
> > expected.
> >
> > config KUNIT_EXAMPLE_TEST
> > - tristate "Example test for KUnit"
> > + tristate "Example test for KUnit" if !KUNIT_RUN_ALL
> > + default KUNIT_RUN_ALL
> > help
> > Enables an example unit test that illustrates some of the basic
> > features of KUnit. This test only exists to help new users understand
> > @@ -41,4 +44,10 @@ config KUNIT_EXAMPLE_TEST
> > is intended for curious hackers who would like to understand how to
> > use KUnit for kernel development.
> >
> > +config KUNIT_RUN_ALL
> > + tristate "KUnit run all test"
> > + help
> > + Enables all KUnit tests. If they can be enabled.
> > + That depends on if KUnit is enabled as a module or builtin.
> > +
>
> s/tests. If/tests, if/ ?

correct, I will fix that.

Cheers,
Anders

>
> > endif # KUNIT
> > diff --git a/security/apparmor/Kconfig b/security/apparmor/Kconfig
> > index 0fe336860773..c4648426ea5d 100644
> > --- a/security/apparmor/Kconfig
> > +++ b/security/apparmor/Kconfig
> > @@ -70,8 +70,9 @@ config SECURITY_APPARMOR_DEBUG_MESSAGES
> > the kernel message buffer.
> >
> > config SECURITY_APPARMOR_KUNIT_TEST
> > - bool "Build KUnit tests for policy_unpack.c"
> > + bool "Build KUnit tests for policy_unpack.c" if !KUNIT_RUN_ALL
> > depends on KUNIT=y && SECURITY_APPARMOR
> > + default KUNIT_RUN_ALL
> > help
> > This builds the AppArmor KUnit tests.
> >
> > --
> > 2.20.1
> >

2020-05-01 20:32:15

by Brendan Higgins

[permalink] [raw]
Subject: Re: [PATCH] kunit: Kconfig: enable a KUNIT_RUN_ALL fragment

On Fri, May 1, 2020 at 1:35 AM Anders Roxell <[email protected]> wrote:
>
> Make it easier to enable all KUnit fragments. This is needed for kernel
> test-systems, so its easy to get all KUnit tests enabled and if new gets
> added they will be enabled as well. Fragments that has to be builtin
> will be missed if CONFIG_KUNIT_RUN_ALL is set as a module.
>
> Adding 'if !KUNIT_RUN_ALL' so individual test can be turned of if
> someone wants that even though KUNIT_RUN_ALL is enabled.

I would LOVE IT, if you could make this work! I have been trying to
figure out the best way to run all KUnit tests for a long time now.

That being said, I am a bit skeptical that this approach will be much
more successful than just using allyesconfig. Either way, there are
tests coming down the pipeline that are incompatible with each other
(the KASAN test and the KCSAN test will be incompatible). Even so,
tests like the apparmor test require a lot of non-default
configuration to compile. In the end, I am not sure how many tests we
will really be able to turn on this way.

Thoughts?

2020-05-02 02:12:10

by David Gow

[permalink] [raw]
Subject: Re: [PATCH] kunit: Kconfig: enable a KUNIT_RUN_ALL fragment

On Sat, May 2, 2020 at 4:31 AM Brendan Higgins
<[email protected]> wrote:
>
> On Fri, May 1, 2020 at 1:35 AM Anders Roxell <[email protected]> wrote:
> >
> > Make it easier to enable all KUnit fragments. This is needed for kernel
> > test-systems, so its easy to get all KUnit tests enabled and if new gets
> > added they will be enabled as well. Fragments that has to be builtin
> > will be missed if CONFIG_KUNIT_RUN_ALL is set as a module.
> >
> > Adding 'if !KUNIT_RUN_ALL' so individual test can be turned of if
> > someone wants that even though KUNIT_RUN_ALL is enabled.
>
> I would LOVE IT, if you could make this work! I have been trying to
> figure out the best way to run all KUnit tests for a long time now.
>
> That being said, I am a bit skeptical that this approach will be much
> more successful than just using allyesconfig. Either way, there are
> tests coming down the pipeline that are incompatible with each other
> (the KASAN test and the KCSAN test will be incompatible). Even so,
> tests like the apparmor test require a lot of non-default
> configuration to compile. In the end, I am not sure how many tests we
> will really be able to turn on this way.
>
> Thoughts?

I think there's still some value in this which the allyesconfig option
doesn't provide. As you point out, it's not possible to have a generic
"run all tests" option due to potential conflicting dependencies, but
this does provide a way to run all tests for things enabled in the
current config. This could be really useful for downstream developers
who want a way of running all tests relevant to their config without
the overhead of running irrelevant tests (e.g., for drivers they don't
build). Using allyesconfig doesn't make that distinction.

Ultimately, we'll probably still want something which enables a
broader set of tests for upstream development: whether that's based on
this, allyesconfig, or something else entirely remains to be seen, I
think. I suspect we're going to end up with something
subsystem-specific (having a kunitconfig per subsystem, or a testing
line in MAINTAINERS or similar are ideas which have been brought up in
the past).

This is a great looking tool to have in the toolbox, though.

-- David