LinuxLists.cc - [PATCH v3 00/20] kselftest install target feature

2014-12-24 16:28:08

Subject: [PATCH v3 00/20] kselftest install target feature

This patch series adds a new kselftest_install make target
to enable selftest install. When make kselftest_install is
run, selftests are installed on the system. A new install
target is added to selftests Makefile which will install
targets for the tests that are specified in INSTALL_TARGETS.
During install, a script is generated to run tests that are
installed. This script will be installed in the selftest install
directory. Individual test Makefiles are changed to add to the
script. This will allow new tests to add install and run test
commands to the generated kselftest script. kselftest target
now depends on kselftest_install and runs the generated kselftest
script to reduce duplicate work and for common look and feel when
running tests.

This approach leverages and extends the existing framework that
uses makefile targets to implement run_tests and adds install
target. This will scale well as new tests get added and makes
it easier for test writers to add install target at the same
time new test gets added.

This series is uploaded to the following experimental branch
for anybody that is interested in playing with it:
git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git devel

Some benchmark numbers: no relation to this patch series,
I just happened to get some timing numbers, this is nice
and low.
kselftest full run from install dir using kselftest.sh
9.41user 3.55system 0:24.86elapsed

This v3 series reduces duplicate code to generate script
in indiviual test Makefiles and consolidates support in
selftests main Makefile. In the main Makefile, it does
minimal work to set and export install path. In this
series exec and powerpc tests are not included in the
install, this work will be done in future patches. exec
and powerpc are still run when make kselftest is invoked.

v2 series: addressed the duplicate code in install and
run_tests targets in individual test Makefiles.
Reference: https://lkml.org/lkml/2014/11/4/707

Shuah Khan (20):
selftests/breakpoints: add install target to enable test install
selftests/cpu-hotplug: add install target to enable test install
selftests/efivarfs: add install target to enable test install
selftests/firmware: add install target to enable test install
selftests/ftrace: add install target to enable test install
selftests/ipc: add install target to enable test install
selftests/kcmp: add install target to enable test install
selftests/memfd: add install target to enable test install
selftests/memory-hotplug: add install target to enable test install
selftests/mount: add install target to enable test install
selftests/mqueue: add install target to enable test install
selftests/net: add install target to enable test install
selftests/ptrace: add install target to enable test install
selftests/size: add install target to enable test install
selftests/sysctl: add install target to enable test install
selftests/timers: add install target to enable test install
selftests/user: add install target to enable test install
selftests/vm: add install target to enable test install
selftests: add install target to enable test install
kbuild: add a new kselftest_install make target to install selftests

Makefile | 14 +++++-
tools/testing/selftests/Makefile | 54 +++++++++++++++++++++-
tools/testing/selftests/breakpoints/Makefile | 19 +++++++-
tools/testing/selftests/cpu-hotplug/Makefile | 14 +++++-
.../{on-off-test.sh => cpu-on-off-test.sh} | 0
tools/testing/selftests/efivarfs/Makefile | 16 ++++++-
tools/testing/selftests/firmware/Makefile | 43 ++++++++++-------
tools/testing/selftests/ftrace/Makefile | 11 ++++-
tools/testing/selftests/ipc/Makefile | 19 +++++++-
tools/testing/selftests/kcmp/Makefile | 13 +++++-
tools/testing/selftests/memfd/Makefile | 17 +++++--
tools/testing/selftests/memory-hotplug/Makefile | 14 +++++-
.../{on-off-test.sh => mem-on-off-test.sh} | 0
tools/testing/selftests/mount/Makefile | 12 ++++-
tools/testing/selftests/mqueue/Makefile | 18 ++++++--
tools/testing/selftests/net/Makefile | 20 ++++++--
tools/testing/selftests/ptrace/Makefile | 16 +++++--
tools/testing/selftests/size/Makefile | 12 ++++-
tools/testing/selftests/sysctl/Makefile | 17 ++++++-
tools/testing/selftests/timers/Makefile | 12 ++++-
tools/testing/selftests/user/Makefile | 12 ++++-
tools/testing/selftests/vm/Makefile | 11 ++++-
22 files changed, 315 insertions(+), 49 deletions(-)
rename tools/testing/selftests/cpu-hotplug/{on-off-test.sh => cpu-on-off-test.sh} (100%)
rename tools/testing/selftests/memory-hotplug/{on-off-test.sh => mem-on-off-test.sh} (100%)

--
2.1.0

2014-12-24 16:28:11

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 02/20] selftests/cpu-hotplug: add install target to enable test install

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <[email protected]>
---
tools/testing/selftests/cpu-hotplug/Makefile | 14 ++++++++++++--
.../cpu-hotplug/{on-off-test.sh => cpu-on-off-test.sh} | 0
2 files changed, 12 insertions(+), 2 deletions(-)
rename tools/testing/selftests/cpu-hotplug/{on-off-test.sh => cpu-on-off-test.sh} (100%)

diff --git a/tools/testing/selftests/cpu-hotplug/Makefile b/tools/testing/selftests/cpu-hotplug/Makefile
index e9c28d8..c9e15ee 100644
--- a/tools/testing/selftests/cpu-hotplug/Makefile
+++ b/tools/testing/selftests/cpu-hotplug/Makefile
@@ -1,9 +1,19 @@
+TEST_STR=/bin/bash ./cpu-on-off-test.sh || echo cpu-hotplug selftests: [FAIL]
+
all:

+install:
+ifdef INSTALL_KSFT_PATH
+ install ./cpu-on-off-test.sh $(INSTALL_KSFT_PATH)/cpu-on-off-test.sh
+ @echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+ @echo Run make kselftest_install in top level source directory
+endif
+
run_tests:
- @/bin/bash ./on-off-test.sh || echo "cpu-hotplug selftests: [FAIL]"
+ @$(TEST_STR)

run_full_test:
- @/bin/bash ./on-off-test.sh -a || echo "cpu-hotplug selftests: [FAIL]"
+ @/bin/bash ./cpu-on-off-test.sh -a || echo "cpu-hotplug selftests: [FAIL]"

clean:
diff --git a/tools/testing/selftests/cpu-hotplug/on-off-test.sh b/tools/testing/selftests/cpu-hotplug/cpu-on-off-test.sh
similarity index 100%
rename from tools/testing/selftests/cpu-hotplug/on-off-test.sh
rename to tools/testing/selftests/cpu-hotplug/cpu-on-off-test.sh
--
2.1.0

2014-12-24 16:28:15

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 11/20] selftests/mqueue: add install target to enable test install

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <[email protected]>
---
tools/testing/selftests/mqueue/Makefile | 18 +++++++++++++++---
1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/mqueue/Makefile b/tools/testing/selftests/mqueue/Makefile
index 8056e2e..27757b9 100644
--- a/tools/testing/selftests/mqueue/Makefile
+++ b/tools/testing/selftests/mqueue/Makefile
@@ -1,10 +1,22 @@
+MQ_OPEN_TEST_STR = ./mq_open_tests /test1 || echo mq_open_tests: [FAIL]
+MQ_PERF_TEST_STR = ./mq_perf_tests || echo mq_perf_tests: [FAIL]
+
all:
gcc -O2 mq_open_tests.c -o mq_open_tests -lrt
gcc -O2 -o mq_perf_tests mq_perf_tests.c -lrt -lpthread -lpopt

-run_tests:
- @./mq_open_tests /test1 || echo "mq_open_tests: [FAIL]"
- @./mq_perf_tests || echo "mq_perf_tests: [FAIL]"
+install:
+ifdef INSTALL_KSFT_PATH
+ install ./mq_open_tests ./mq_perf_tests $(INSTALL_KSFT_PATH)
+ @echo "$(MQ_OPEN_TEST_STR)" >> $(KSELFTEST)
+ @echo "$(MQ_PERF_TEST_STR)" >> $(KSELFTEST)
+else
+ @echo Run make kselftest_install in top level source directory
+endif
+
+run_tests: all
+ @$(MQ_OPEN_TEST_STR)
+ @$(MQ_PERF_TEST_STR)

clean:
rm -f mq_open_tests mq_perf_tests
--
2.1.0

2014-12-24 16:28:18

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 12/20] selftests/net: add install target to enable test install

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <[email protected]>
---
tools/testing/selftests/net/Makefile | 20 +++++++++++++++++---
1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index 62f22cc..2ffc96f 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -6,14 +6,28 @@ CFLAGS = -Wall -O2 -g
CFLAGS += -I../../../../usr/include/

NET_PROGS = socket psock_fanout psock_tpacket
+INSTALL_PROGS = run_netsocktests run_afpackettests test_bpf.sh $(NET_PROGS)
+NETSOCK_TEST_STR = /bin/sh ./run_netsocktests || echo sockettests: [FAIL]
+AFPKT_TEST_STR = /bin/sh ./run_afpackettests || echo afpackettests: [FAIL]
+BFP_TEST_STR = ./test_bpf.sh

all: $(NET_PROGS)
%: %.c
$(CC) $(CFLAGS) -o $@ $^

+install:
+ifdef INSTALL_KSFT_PATH
+ install $(INSTALL_PROGS) $(INSTALL_KSFT_PATH)
+ @echo "$(NETSOCK_TEST_STR)" >> $(KSELFTEST)
+ @echo "$(AFPKT_TEST_STR)" >> $(KSELFTEST)
+ @echo "$(BFP_TEST_STR)" >> $(KSELFTEST)
+else
+ @echo Run make kselftest_install in top level source directory
+endif
+
run_tests: all
- @/bin/sh ./run_netsocktests || echo "sockettests: [FAIL]"
- @/bin/sh ./run_afpackettests || echo "afpackettests: [FAIL]"
- ./test_bpf.sh
+ @$(NETSOCK_TEST_STR)
+ @$(AFPKT_TEST_STR)
+ @$(BFP_TEST_STR)
clean:
$(RM) $(NET_PROGS)
--
2.1.0

2014-12-24 16:28:25

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 17/20] selftests/user: add install target to enable test install

2014-12-24 16:28:23

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 18/20] selftests/vm: add install target to enable test install

2014-12-24 16:28:49

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 20/20] kbuild: add a new kselftest_install make target to install selftests

Add a new make target to install to install kernel selftests.
This new target will build and install selftests. kselftest
target now depends on kselftest_install and runs the generated
kselftest script to reduce duplicate work and for common look
and feel when running tests.

make kselftest_target:
-- exports kselftest INSTALL_KSFT_PATH
default $(INSTALL_MOD_PATH)/lib/kselftest/$(KERNELRELEASE)
-- exports INSTALL_KSFT_PATH
-- runs selftests make install target:

Signed-off-by: Shuah Khan <[email protected]>
---
Makefile | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index b1c3254..99fac47 100644
--- a/Makefile
+++ b/Makefile
@@ -1072,12 +1072,21 @@ headers_check: headers_install
$(Q)$(MAKE) $(hdr-inst)=arch/$(hdr-arch)/include/uapi/asm $(hdr-dst) HDRCHECK=1

# ---------------------------------------------------------------------------
-# Kernel selftest
+# Kernel selftest targets
+
+# Default base path for kselftest install
+INSTALL_KSFT_PATH = $(INSTALL_MOD_PATH)/lib/kselftest/$(KERNELRELEASE)
+export INSTALL_KSFT_PATH

PHONY += kselftest
kselftest:
$(Q)$(MAKE) -C tools/testing/selftests run_tests

+# Kernel selftest install
+PHONY += kselftest_install
+kselftest_install:
+ $(Q)$(MAKE) -C tools/testing/selftests install
+
# ---------------------------------------------------------------------------
# Modules

@@ -1286,6 +1295,9 @@ help:
@echo ' Build, install, and boot kernel before'
@echo ' running kselftest on it'
@echo ''
+ @echo ' kselftest_install - Install Kselftests to INSTALL_KSFT_PATH'
+ @echo ' default: $(INSTALL_MOD_PATH)/lib/kselftest/$(KERNELRELEASE)'
+ @echo ''
@echo 'Kernel packaging:'
@$(MAKE) $(build)=$(package-dir) help
@echo ''
--
2.1.0

2014-12-24 16:29:26

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 19/20] selftests: add install target to enable test install

Add a new make target to enable installing selftests. This
new target will call install targets for the tests that are
specified in INSTALL_TARGETS. During install, a script is
generated to run tests that are installed. This script will
be installed in the selftest install directory. Individual
test Makefiles are changed to add to the script. This will
allow new tests to add install and run test commands to the
generated kselftest script. run_tests target runs the
generated kselftest script to run tests when it is initiated
from from "make kselftest" from top level source directory.

Approach:

Add a new kselftest_install target:
-- exports kselftest INSTALL_KSFT_PATH
default $(INSTALL_MOD_PATH)/lib/kselftest/$(KERNELRELEASE)
-- exports INSTALL_KSFT_PATH
-- runs selftests make kselftest_install target:

selftests make install target
-- Sets up environment for sub-makefiles
-- creates kselftest.sh script in install install dir
-- runs install targets for INSTALL_TARGETS
-- install target can be run only from top level source dir.

Individual test make install targets:
-- install test programs and/or scripts in install dir
-- append to the ksefltest.sh file to add commands to run test
-- install target can be run only from top level source dir.

Adds the following new ways to initiate selftests:
-- Installing and running kselftest from install directory
by running "make kselftest"
-- Running kselftest script from install directory

Maintains the following ways to run tests:
-- make TARGETS=net kselftest
-- make -C tools/testing/selftests run_tests
-- make -C tools/testing/selftests TARGETS=target run_tests
Ability specify targets: e.g TARGETS=net
-- make run_tests from tools/testing/selftests
-- make run_tests from individual test directories:
e.g: make run_tests in tools/testing/selftests/net

Signed-off-by: Shuah Khan <[email protected]>
---
tools/testing/selftests/Makefile | 54 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 53 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 4e51122..e70cdc9 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -22,15 +22,67 @@ TARGETS += vm
TARGETS_HOTPLUG = cpu-hotplug
TARGETS_HOTPLUG += memory-hotplug

+# Used in only run_tests target when make kselftest is run in
+# top level source directory
+ifeq "$(origin TARGETS)" "command line"
+no_install_run=1
+endif
+
+ifdef INSTALL_KSFT_PATH
+KSELFTEST=$(INSTALL_KSFT_PATH)/kselftest.sh
+export KSELFTEST
+# TODO add install target for SKIP_INSTALL_TARGETS
+SKIP_INSTALL_TARGETS = exec powerpc
+INSTALL_TARGETS = $(filter-out $(SKIP_INSTALL_TARGETS),$(TARGETS)) ipc
+else
+no_install_run=1
+endif
+
+INSTALL_KSFT_ERR = Run make kselftest_install in top level source directory
+
all:
for TARGET in $(TARGETS); do \
make -C $$TARGET; \
done;

-run_tests: all
+install:
+ifdef INSTALL_KSFT_PATH
+ rm -rf $(INSTALL_KSFT_PATH)
+ mkdir -p $(INSTALL_KSFT_PATH)
+
+ make all
+ @echo "#!/bin/sh\n# Kselftest Run Tests ...." >> $(KSELFTEST)
+ @echo "# This file is generated by kselftest_install" >> $(KSELFTEST)
+ @echo "# Please don't change it !!\n" >> $(KSELFTEST)
+ @echo echo ============================== >> $(KSELFTEST)
+ for TARGET in $(INSTALL_TARGETS); do \
+ echo Installing $$TARGET; \
+ echo echo Start $$TARGET test .... >> $(KSELFTEST); \
+ make -C $$TARGET install; \
+ echo echo End $$TARGET test .... >> $(KSELFTEST); \
+ echo echo ============================== >> $(KSELFTEST); \
+ done;
+ chmod +x $(KSELFTEST)
+else
+ @echo $(INSTALL_KSFT_ERR)
+endif
+
+run_tests:
+ifndef no_install_run
+# ifdef INSTALL_KSFT_PATH
+ make install
+ @cd $(INSTALL_KSFT_PATH); ./kselftest.sh; cd -
+# invoke run_tests for SKIP_INSTALL_TARGETS
+ for TARGET in $(SKIP_INSTALL_TARGETS); do \
+ make -C $$TARGET run_tests; \
+ done;
+# endif
+else
+ make all
for TARGET in $(TARGETS); do \
make -C $$TARGET run_tests; \
done;
+endif

hotplug:
for TARGET in $(TARGETS_HOTPLUG); do \
--
2.1.0

2014-12-24 16:29:44

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 16/20] selftests/timers: add install target to enable test install

2014-12-24 16:30:06

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 15/20] selftests/sysctl: add install target to enable test install

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <[email protected]>
---
tools/testing/selftests/sysctl/Makefile | 17 +++++++++++++++--
1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/sysctl/Makefile b/tools/testing/selftests/sysctl/Makefile
index 0a92ada..dfbaa58 100644
--- a/tools/testing/selftests/sysctl/Makefile
+++ b/tools/testing/selftests/sysctl/Makefile
@@ -4,12 +4,25 @@
# No binaries, but make sure arg-less "make" doesn't trigger "run_tests".
all:

+INSTALL_PROGS = common_tests run_numerictests run_stringtests
+NUMERIC_TEST_STR = /bin/sh ./run_numerictests
+STRING_TEST_STR = /bin/sh ./run_stringtests
+
# Allow specific tests to be selected.
test_num:
- @/bin/sh ./run_numerictests
+ @$(NUMERIC_TEST_STR)

test_string:
- @/bin/sh ./run_stringtests
+ @$(STRING_TEST_STR)
+
+install: all
+ifdef INSTALL_KSFT_PATH
+ install $(INSTALL_PROGS) $(INSTALL_KSFT_PATH)
+ @echo "$(NUMERIC_TEST_STR)" >> $(KSELFTEST)
+ @echo "$(STRING_TEST_STR)" >> $(KSELFTEST)
+else
+ @echo Run make kselftest_install in top level source directory
+endif

run_tests: all test_num test_string

--
2.1.0

2014-12-24 16:30:43

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 14/20] selftests/size: add install target to enable test install

2014-12-24 16:30:58

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 13/20] selftests/ptrace: add install target to enable test install

2014-12-24 16:31:19

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 10/20] selftests/mount: add install target to enable test install

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <[email protected]>
---
tools/testing/selftests/mount/Makefile | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/mount/Makefile b/tools/testing/selftests/mount/Makefile
index 337d853..607de4c 100644
--- a/tools/testing/selftests/mount/Makefile
+++ b/tools/testing/selftests/mount/Makefile
@@ -1,5 +1,7 @@
# Makefile for mount selftests.

+TEST_STR = if [ -f /proc/self/uid_map ] ; then ./unprivileged-remount-test ; fi
+
all: unprivileged-remount-test

unprivileged-remount-test: unprivileged-remount-test.c
@@ -7,7 +9,15 @@ unprivileged-remount-test: unprivileged-remount-test.c

# Allow specific tests to be selected.
test_unprivileged_remount: unprivileged-remount-test
- @if [ -f /proc/self/uid_map ] ; then ./unprivileged-remount-test ; fi
+ @$(TEST_STR)
+
+install:
+ifdef INSTALL_KSFT_PATH
+ install ./unprivileged-remount-test $(INSTALL_KSFT_PATH)
+ @echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+ @echo Run make kselftest_install in top level source directory
+endif

run_tests: all test_unprivileged_remount

--
2.1.0

2014-12-24 16:31:43

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 09/20] selftests/memory-hotplug: add install target to enable test install

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <[email protected]>
---
tools/testing/selftests/memory-hotplug/Makefile | 14 ++++++++++++--
.../memory-hotplug/{on-off-test.sh => mem-on-off-test.sh} | 0
2 files changed, 12 insertions(+), 2 deletions(-)
rename tools/testing/selftests/memory-hotplug/{on-off-test.sh => mem-on-off-test.sh} (100%)

diff --git a/tools/testing/selftests/memory-hotplug/Makefile b/tools/testing/selftests/memory-hotplug/Makefile
index d46b8d4..561ee2b 100644
--- a/tools/testing/selftests/memory-hotplug/Makefile
+++ b/tools/testing/selftests/memory-hotplug/Makefile
@@ -1,9 +1,19 @@
+TEST_STR=/bin/bash ./mem-on-off-test.sh -r 2 || echo memory-hotplug selftests: [FAIL]
+
all:

+install:
+ifdef INSTALL_KSFT_PATH
+ install ./mem-on-off-test.sh $(INSTALL_KSFT_PATH)/mem-on-off-test.sh
+ @echo "$(TEST_STR)" >> $(KSELFTEST) >> $(KSELFTEST)
+else
+ @echo Run make kselftest_install in top level source directory
+endif
+
run_tests:
- @/bin/bash ./on-off-test.sh -r 2 || echo "memory-hotplug selftests: [FAIL]"
+ @$(TEST_STR)

run_full_test:
- @/bin/bash ./on-off-test.sh || echo "memory-hotplug selftests: [FAIL]"
+ @/bin/bash ./mem-on-off-test.sh || echo "memory-hotplug selftests: [FAIL]"

clean:
diff --git a/tools/testing/selftests/memory-hotplug/on-off-test.sh b/tools/testing/selftests/memory-hotplug/mem-on-off-test.sh
similarity index 100%
rename from tools/testing/selftests/memory-hotplug/on-off-test.sh
rename to tools/testing/selftests/memory-hotplug/mem-on-off-test.sh
--
2.1.0

2014-12-24 16:32:00

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 07/20] selftests/kcmp: add install target to enable test install

2014-12-24 16:31:58

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 08/20] selftests/memfd: add install target to enable test install

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <[email protected]>
---
tools/testing/selftests/memfd/Makefile | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/memfd/Makefile b/tools/testing/selftests/memfd/Makefile
index b80cd10..5f70b33 100644
--- a/tools/testing/selftests/memfd/Makefile
+++ b/tools/testing/selftests/memfd/Makefile
@@ -2,19 +2,30 @@ CFLAGS += -D_FILE_OFFSET_BITS=64
CFLAGS += -I../../../../include/uapi/
CFLAGS += -I../../../../include/

+INSTALL_PROGS = memfd_test fuse_test run_fuse_test.sh
+MEMFD_TEST_STR = ./memfd_test || echo memfd_test: [FAIL]
+FUSE_TEST_STR = ./run_fuse_test.sh || echo fuse_test: [FAIL]
+
all:
gcc $(CFLAGS) memfd_test.c -o memfd_test

+install:
+ifdef INSTALL_KSFT_PATH
+ install $(INSTALL_PROGS) $(INSTALL_KSFT_PATH)
+ @echo "$(MEMFD_TEST_STR)" >> $(KSELFTEST)
+else
+ @echo Run make kselftest_install in top level source directory
+endif
+
run_tests: all
- gcc $(CFLAGS) memfd_test.c -o memfd_test
- @./memfd_test || echo "memfd_test: [FAIL]"
+ @$(MEMFD_TEST_STR)

build_fuse:
gcc $(CFLAGS) fuse_mnt.c `pkg-config fuse --cflags --libs` -o fuse_mnt
gcc $(CFLAGS) fuse_test.c -o fuse_test

run_fuse: build_fuse
- @./run_fuse_test.sh || echo "fuse_test: [FAIL]"
+ @$(FUSE_TEST_STR)

clean:
$(RM) memfd_test fuse_test
--
2.1.0

2014-12-24 16:28:05

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 01/20] selftests/breakpoints: add install target to enable test install

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <[email protected]>
---
tools/testing/selftests/breakpoints/Makefile | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/breakpoints/Makefile b/tools/testing/selftests/breakpoints/Makefile
index e18b42b..b5a5fe8 100644
--- a/tools/testing/selftests/breakpoints/Makefile
+++ b/tools/testing/selftests/breakpoints/Makefile
@@ -8,6 +8,7 @@ ifeq ($(ARCH),x86_64)
ARCH := x86
endif

+TEST_STR = ./breakpoint_test || echo breakpoints selftests: [FAIL]

all:
ifeq ($(ARCH),x86)
@@ -16,8 +17,22 @@ else
echo "Not an x86 target, can't build breakpoints selftests"
endif

-run_tests:
- @./breakpoint_test || echo "breakpoints selftests: [FAIL]"
+install:
+ifdef INSTALL_KSFT_PATH
+ifeq ($(ARCH),x86)
+ install ./breakpoint_test $(INSTALL_KSFT_PATH)
+ @echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+ @echo Not an x86 target, unable to install breakpoints selftests
+endif
+else
+ @echo Run make kselftest_install in top level source directory
+endif
+
+run_tests: all
+ifeq ($(ARCH),x86)
+ @$(TEST_STR)
+endif

clean:
rm -fr breakpoint_test
--
2.1.0

2014-12-24 16:32:46

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 05/20] selftests/ftrace: add install target to enable test install

2014-12-24 16:32:44

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 06/20] selftests/ipc: add install target to enable test install

2014-12-24 16:34:59

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 04/20] selftests/firmware: add install target to enable test install

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <[email protected]>
---
tools/testing/selftests/firmware/Makefile | 43 +++++++++++++++++++------------
1 file changed, 26 insertions(+), 17 deletions(-)

diff --git a/tools/testing/selftests/firmware/Makefile b/tools/testing/selftests/firmware/Makefile
index e23cce0..7ac1cf3 100644
--- a/tools/testing/selftests/firmware/Makefile
+++ b/tools/testing/selftests/firmware/Makefile
@@ -1,25 +1,34 @@
# Makefile for firmware loading selftests

# No binaries, but make sure arg-less "make" doesn't trigger "run_tests"
+
+__fw_filesystem:
+fw_filesystem = if /bin/sh ./fw_filesystem.sh ; then
+fw_filesystem += echo fw_filesystem: ok;
+fw_filesystem += else echo fw_filesystem: [FAIL];
+fw_filesystem += fi
+
+__fw_userhelper:
+fw_userhelper = if /bin/sh ./fw_userhelper.sh ; then
+fw_userhelper += echo fw_userhelper: ok;
+fw_userhelper += else
+fw_userhelper += echo fw_userhelper: [FAIL];
+fw_userhelper += fi
+
all:

-fw_filesystem:
- @if /bin/sh ./fw_filesystem.sh ; then \
- echo "fw_filesystem: ok"; \
- else \
- echo "fw_filesystem: [FAIL]"; \
- exit 1; \
- fi
-
-fw_userhelper:
- @if /bin/sh ./fw_userhelper.sh ; then \
- echo "fw_userhelper: ok"; \
- else \
- echo "fw_userhelper: [FAIL]"; \
- exit 1; \
- fi
-
-run_tests: all fw_filesystem fw_userhelper
+install:
+ifdef INSTALL_KSFT_PATH
+ install ./fw_filesystem.sh ./fw_userhelper.sh $(INSTALL_KSFT_PATH)
+ @echo "$(fw_filesystem)" >> $(KSELFTEST)
+ @echo "$(fw_userhelper)" >> $(KSELFTEST)
+else
+ @echo Run make kselftest_install in top level source directory
+endif
+
+run_tests:
+ @$(fw_filesystem)
+ @$(fw_userhelper)

# Nothing to clean up.
clean:
--
2.1.0

2014-12-24 16:35:33

by Shuah Khan

[permalink] [raw]

Subject: [PATCH v3 03/20] selftests/efivarfs: add install target to enable test install

2014-12-29 05:00:49

by Michael Ellerman

[permalink] [raw]

Subject: Re: [PATCH v3 00/20] kselftest install target feature

On Wed, 2014-12-24 at 09:27 -0700, Shuah Khan wrote:
> This patch series adds a new kselftest_install make target
> to enable selftest install. When make kselftest_install is
> run, selftests are installed on the system. A new install
> target is added to selftests Makefile which will install
> targets for the tests that are specified in INSTALL_TARGETS.
> During install, a script is generated to run tests that are
> installed. This script will be installed in the selftest install
> directory. Individual test Makefiles are changed to add to the
> script. This will allow new tests to add install and run test
> commands to the generated kselftest script. kselftest target
> now depends on kselftest_install and runs the generated kselftest
> script to reduce duplicate work and for common look and feel when
> running tests.
>
> This approach leverages and extends the existing framework that
> uses makefile targets to implement run_tests and adds install
> target. This will scale well as new tests get added and makes
> it easier for test writers to add install target at the same
> time new test gets added.
>
> This v3 series reduces duplicate code to generate script
> in indiviual test Makefiles and consolidates support in
> selftests main Makefile. In the main Makefile, it does
> minimal work to set and export install path. In this
> series exec and powerpc tests are not included in the
> install, this work will be done in future patches. exec
> and powerpc are still run when make kselftest is invoked.

Any particular reason you excluded the powerpc tests? Going by a quick count,
powerpc has 32 of the 54 self tests, ie. more than half.

Sorry I didn't get a chance to review v1 or v2, but is this really the best
solution we can come up with? It seems to involve a lot of boiler plate getting
repeated in every Makefile.

I'm off this week so I can't immediately come up with something better, I'll
try in the new year.

cheers

2014-12-29 15:25:00

by Shuah Khan

[permalink] [raw]

Subject: Re: [PATCH v3 00/20] kselftest install target feature

On 12/28/2014 09:53 PM, Michael Ellerman wrote:
> On Wed, 2014-12-24 at 09:27 -0700, Shuah Khan wrote:
>> This patch series adds a new kselftest_install make target
>> to enable selftest install. When make kselftest_install is
>> run, selftests are installed on the system. A new install
>> target is added to selftests Makefile which will install
>> targets for the tests that are specified in INSTALL_TARGETS.
>> During install, a script is generated to run tests that are
>> installed. This script will be installed in the selftest install
>> directory. Individual test Makefiles are changed to add to the
>> script. This will allow new tests to add install and run test
>> commands to the generated kselftest script. kselftest target
>> now depends on kselftest_install and runs the generated kselftest
>> script to reduce duplicate work and for common look and feel when
>> running tests.
>>
>> This approach leverages and extends the existing framework that
>> uses makefile targets to implement run_tests and adds install
>> target. This will scale well as new tests get added and makes
>> it easier for test writers to add install target at the same
>> time new test gets added.
>>
>> This v3 series reduces duplicate code to generate script
>> in indiviual test Makefiles and consolidates support in
>> selftests main Makefile. In the main Makefile, it does
>> minimal work to set and export install path. In this
>> series exec and powerpc tests are not included in the
>> install, this work will be done in future patches. exec
>> and powerpc are still run when make kselftest is invoked.
>
> Any particular reason you excluded the powerpc tests? Going by a quick count,
> powerpc has 32 of the 54 self tests, ie. more than half.

No particular reason other than not having a good way to test the
changes I need to make. It does have sub-directory structure with
multiple makefiles underneath. I would like to work on this after
this patch series gets in or maybe you can help out on powerpc
changes if you like.

>
> Sorry I didn't get a chance to review v1 or v2, but is this really the best
> solution we can come up with? It seems to involve a lot of boiler plate getting
> repeated in every Makefile.

This approach extends the existing approach to use makefile targets as
a means to support running tests. Also it gives full control to the
individual test developer in making changes to the targets as needed
without conflicts with work that is in progress on other tests.

There isn't a whole lot of boiler plating code repeated as such in
individual makefiles. They all add their specific targets to the
main script.

-- Shuah

--
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
[email protected] | (970) 217-8978

2015-01-01 02:31:38

by Tim Bird

[permalink] [raw]

Subject: WIP alternative - was Re: [PATCH v3 14/20] selftests/size: add install target to enable test install

On 12/24/2014 08:27 AM, Shuah Khan wrote:
> Add a new make target to enable installing test. This target
> installs test in the kselftest install location and add to the
> kselftest script to run the test. Install target can be run
> only from top level kernel source directory.
>
> Signed-off-by: Shuah Khan <[email protected]>
> ---
> tools/testing/selftests/size/Makefile | 12 +++++++++++-
> 1 file changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/tools/testing/selftests/size/Makefile b/tools/testing/selftests/size/Makefile
> index 04dc25e..bb7113b 100644
> --- a/tools/testing/selftests/size/Makefile
> +++ b/tools/testing/selftests/size/Makefile
> @@ -1,12 +1,22 @@
> CC = $(CROSS_COMPILE)gcc
>
> +TEST_STR = ./get_size || echo get_size selftests: [FAIL]
> +
> all: get_size
>
> get_size: get_size.c
> $(CC) -static -ffreestanding -nostartfiles -s $< -o $@
>
> +install:
> +ifdef INSTALL_KSFT_PATH
> + install ./get_size $(INSTALL_KSFT_PATH)
> + @echo "$(TEST_STR)" >> $(KSELFTEST)
> +else
> + @echo Run make kselftest_install in top level source directory
> +endif
> +
> run_tests: all
> - ./get_size
> + @$(TEST_STR)
>
> clean:
> $(RM) get_size
>

The install phase is desperately needed for usage of kselftest in
cross-target situations (applicable to almost all embedded). So this
is great stuff.

I worked a bit on isolating the install stuff to a makefile include file.
This allows simplifying some of the sub-level Makefiles a bit, and allowing
control of some of the install and run logic in less places.

This is still a work in progress, but before I got too far along, I wanted
to post it for people to provide feedback. A couple of problems cropped
up that are worth discussing, IMHO.

1) I think it should be a requirement that each test has a single
"main" program to execute to run the tests. If multiple tests are supported
or more flexibility is desired for additional arguments, or that sort of
thing, then that's fine, but the automated script builder should be able
to run just a single program or script to have things work. This also
makes things more consistent. In the case of the firmware test, I created
a single fw_both.sh script to do this, instead of having two separate
blocks in the kselftest.sh script.

2) I've added a CROSS_INSTALL variable, which can call an arbitrary program
to place files on the target system (rather than just calling 'install').
In my case, I'd use my own 'ttc cp' command, which I can extend as necessary
to put things on a remote machine. This works for a single directory,
but things get dicier with sub-directory trees full of files (like
the ftrace test uses.)

If additional items need to be installed to the target, then maybe a setup
program should be used, rather than just copying files.

3) Some of the scripts were using /bin/bash to execute them, rather
than rely on the interpreter line in the script itself (and having
the script have executable privileges). Is there a reason for this?
I modified a few scripts to be executable, and got rid of the
explicit execution with /bin/bash.

The following is just a start... Let me know if this direction looks
OK, and I'll finish this up. The main item to look at is
kselftest.include file. Note that these patches are based on Shuah's
series - but if you want to use these ideas I can rebase them onto
mainline, and break them out per test sub-level like Shuah did.

Let me know what you think.
>From 14164fd3117c97799a050f8cf791dedc93aa5e82 Mon Sep 17 00:00:00 2001
From: Tim Bird <[email protected]>
Date: Wed, 31 Dec 2014 18:04:08 -0800
Subject: [PATCH] Switch to using an include file for common kselftest_install
actions

---
tools/testing/selftests/cpu-hotplug/Makefile | 15 +++-------
.../selftests/cpu-hotplug/cpu-on-off-test.sh | 0
tools/testing/selftests/efivarfs/Makefile | 21 ++++----------
tools/testing/selftests/efivarfs/efivarfs.sh | 0
tools/testing/selftests/firmware/Makefile | 32 ++++------------------
tools/testing/selftests/firmware/fw_both.sh | 13 +++++++++
tools/testing/selftests/kselftest.include | 26 ++++++++++++++++++
tools/testing/selftests/size/Makefile | 19 +++----------
9 files changed, 60 insertions(+), 69 deletions(-)
mode change 100644 => 100755 tools/testing/selftests/cpu-hotplug/cpu-on-off-test.sh
mode change 100644 => 100755 tools/testing/selftests/efivarfs/efivarfs.sh
create mode 100755 tools/testing/selftests/firmware/fw_both.sh
create mode 100644 tools/testing/selftests/kselftest.include

diff --git a/tools/testing/selftests/cpu-hotplug/Makefile b/tools/testing/selftests/cpu-hotplug/Makefile
index c9e15ee..d2e540d 100644
--- a/tools/testing/selftests/cpu-hotplug/Makefile
+++ b/tools/testing/selftests/cpu-hotplug/Makefile
@@ -1,18 +1,11 @@
-TEST_STR=/bin/bash ./cpu-on-off-test.sh || echo cpu-hotplug selftests: [FAIL]
+TEST_TITLE = cpu-hotplug
+TEST_MAIN_PROG = cpu-on-off-test.sh

all:

-install:
-ifdef INSTALL_KSFT_PATH
- install ./cpu-on-off-test.sh $(INSTALL_KSFT_PATH)/cpu-on-off-test.sh
- @echo "$(TEST_STR)" >> $(KSELFTEST)
-else
- @echo Run make kselftest_install in top level source directory
-endif
-
-run_tests:
- @$(TEST_STR)
+include ../kselftest.include

+# use -a to get all tests
run_full_test:
@/bin/bash ./cpu-on-off-test.sh -a || echo "cpu-hotplug selftests: [FAIL]"

diff --git a/tools/testing/selftests/cpu-hotplug/cpu-on-off-test.sh b/tools/testing/selftests/cpu-hotplug/cpu-on-off-test.sh
old mode 100644
new mode 100755
diff --git a/tools/testing/selftests/efivarfs/Makefile b/tools/testing/selftests/efivarfs/Makefile
index aaf404b..8d48e9c 100644
--- a/tools/testing/selftests/efivarfs/Makefile
+++ b/tools/testing/selftests/efivarfs/Makefile
@@ -1,24 +1,15 @@
-CC = $(CROSS_COMPILE)gcc
-CFLAGS = -Wall
+TEST_TITLE = efivarfs
+TEST_MAIN_PROG = efivarfs.sh

-test_objs = open-unlink create-read
+CFLAGS = -Wall

-TEST_STR = /bin/bash ./efivarfs.sh || echo efivarfs selftests: [FAIL]
+TEST_PROGS = open-unlink create-read

all:
gcc open-unlink.c -o open-unlink
gcc create-read.c -o create-read

-install:
-ifdef INSTALL_KSFT_PATH
- install ./efivarfs.sh $(test_objs) $(INSTALL_KSFT_PATH)
- @echo "$(TEST_STR)" >> $(KSELFTEST)
-else
- @echo Run make kselftest_install in top level source directory
-endif
-
-run_tests: all
- @$(TEST_STR)
+include ../kselftest.include

clean:
- rm -f $(test_objs)
+ rm -f $(TEST_PROGS)
diff --git a/tools/testing/selftests/efivarfs/efivarfs.sh b/tools/testing/selftests/efivarfs/efivarfs.sh
old mode 100644
new mode 100755
diff --git a/tools/testing/selftests/firmware/Makefile b/tools/testing/selftests/firmware/Makefile
index 7ac1cf3..b576308 100644
--- a/tools/testing/selftests/firmware/Makefile
+++ b/tools/testing/selftests/firmware/Makefile
@@ -1,36 +1,14 @@
# Makefile for firmware loading selftests
+TEST_MAIN_PROG = fw_both.sh
+TEST_TITLE = firmware
+TEST_FILES = ./fw_filesystem.sh ./fw_userhelper.sh

# No binaries, but make sure arg-less "make" doesn't trigger "run_tests"
-
-__fw_filesystem:
-fw_filesystem = if /bin/sh ./fw_filesystem.sh ; then
-fw_filesystem += echo fw_filesystem: ok;
-fw_filesystem += else echo fw_filesystem: [FAIL];
-fw_filesystem += fi
-
-__fw_userhelper:
-fw_userhelper = if /bin/sh ./fw_userhelper.sh ; then
-fw_userhelper += echo fw_userhelper: ok;
-fw_userhelper += else
-fw_userhelper += echo fw_userhelper: [FAIL];
-fw_userhelper += fi
-
all:

-install:
-ifdef INSTALL_KSFT_PATH
- install ./fw_filesystem.sh ./fw_userhelper.sh $(INSTALL_KSFT_PATH)
- @echo "$(fw_filesystem)" >> $(KSELFTEST)
- @echo "$(fw_userhelper)" >> $(KSELFTEST)
-else
- @echo Run make kselftest_install in top level source directory
-endif
-
-run_tests:
- @$(fw_filesystem)
- @$(fw_userhelper)
+include ../kselftest.include

# Nothing to clean up.
clean:

-.PHONY: all clean run_tests fw_filesystem fw_userhelper
+.PHONY: all clean run_tests
diff --git a/tools/testing/selftests/firmware/fw_both.sh b/tools/testing/selftests/firmware/fw_both.sh
new file mode 100755
index 0000000..fb7fa8d
--- /dev/null
+++ b/tools/testing/selftests/firmware/fw_both.sh
@@ -0,0 +1,13 @@
+#!/bin/sh
+
+if /bin/sh ./fw_filesystem.sh ; then
+ echo "fw_filesystem: ok";
+else
+ echo "fw_filesystem: [FAIL]";
+fi
+
+if /bin/sh ./fw_userhelper.sh ; then
+ echo "fw_userhelper: ok";
+else
+ echo "fw_userhelper: [FAIL]";
+fi
diff --git a/tools/testing/selftests/kselftest.include b/tools/testing/selftests/kselftest.include
new file mode 100644
index 0000000..7020a82
--- /dev/null
+++ b/tools/testing/selftests/kselftest.include
@@ -0,0 +1,26 @@
+# this make include script expects the following variables to be set:
+# TEST_MAIN_PROG - a single program to invoke to run the test
+# TEST_TITLE - the title of the test
+# TEST_FILES - other files that should be copied to the testing directory
+# INSTALL_KSFT_PATH - where to put test programs
+# KSELFTEST - the name of the generated script (which calls the sub-tests)
+# the 'all' target, which makes sure everything is built
+
+# this file defines the 'install' and 'run_tests' targets
+
+CC = $(CROSS_COMPILE)gcc
+INSTALL = $(CROSS_INSTALL)install
+export INSTALL
+
+TEST_STR = ./$(TEST_MAIN_PROG) || echo $(TEST_TITLE) selftests: [FAIL]
+
+install:
+ifdef INSTALL_KSFT_PATH
+ $(INSTALL) ./$(TEST_MAIN_PROG) $(TEST_FILES) $(INSTALL_KSFT_PATH)
+ @echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+ @echo Run make kselftest_install in the top level source directory
+endif
+
+run_tests: all
+ @$(TEST_STR)
diff --git a/tools/testing/selftests/size/Makefile b/tools/testing/selftests/size/Makefile
index bb7113b..c88819a 100644
--- a/tools/testing/selftests/size/Makefile
+++ b/tools/testing/selftests/size/Makefile
@@ -1,22 +1,11 @@
-CC = $(CROSS_COMPILE)gcc
+TEST_MAIN_PROG = get_size

-TEST_STR = ./get_size || echo get_size selftests: [FAIL]
+all: $(TEST_MAIN_PROG)

-all: get_size
+include ../kselftest.include

get_size: get_size.c
$(CC) -static -ffreestanding -nostartfiles -s $< -o $@

-install:
-ifdef INSTALL_KSFT_PATH
- install ./get_size $(INSTALL_KSFT_PATH)
- @echo "$(TEST_STR)" >> $(KSELFTEST)
-else
- @echo Run make kselftest_install in top level source directory
-endif
-
-run_tests: all
- @$(TEST_STR)
-
clean:
- $(RM) get_size
+ $(RM) $(TEST_MAIN_PROG)
--
1.8.2.2