Currently, after generating run_kselftest.sh, there is no way to choose
which test we could run. All the tests are listed together and we have
to run all every time. This patch enhanced the run_kselftest.sh to make
the tests individually selectable. e.g.
$ ./run_kselftest.sh -t "bpf size timers"
Before the patch:
================
$ cat run_kselftest.sh
\#!/bin/sh
BASE_DIR=$(realpath $(dirname $0))
cd $BASE_DIR
. ./kselftest/runner.sh
ROOT=$PWD
if [ "$1" = "--summary" ]; then
logfile=$BASE_DIR/output.log
cat /dev/null > $logfile
fi
[ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg
cd android
run_many \
"run.sh"
cd $ROOT
...<snip>...
[ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg
cd zram
run_many \
"zram.sh"
cd $ROOT
After the patch:
===============
$ cat run_kselftest.sh
\#!/bin/sh
BASE_DIR=$(realpath $(dirname $0))
. ./kselftest/runner.sh
TESTS="android ...<snip>... filesystems/binderfs ...<snip>... zram"
run_android()
{
[ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg
cd android
run_many \
"run.sh"
cd $ROOT
}
...<snip>...
run_filesystems_binderfs()
{
[ -w /dev/kmsg ] && echo "kselftest: Running tests in filesystems/binderfs" >> /dev/kmsg
cd filesystems/binderfs
run_many \
"binderfs_test"
cd $ROOT
}
...<snip>...
run_zram()
{
[ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg
cd zram
run_many \
"zram.sh"
cd $ROOT
}
usage()
{
cat <<EOF
usage: ${0##*/} OPTS
-s | --summary Only print summary info and put detailed log in output.log
-t | --tests Test name you want to run specifically
-h | --help Show this usage info
EOF
}
while true; do
case "$1" in
-s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;
-t | --tests ) TESTS=$2; shift 2 ;;
-l | --list ) echo $TESTS; exit 0 ;;
-h | --help ) usage; exit 0 ;;
"" ) break;;
* ) usage; exit 1;;
esac
done
cd $BASE_DIR
ROOT=$PWD
for folder in $TESTS; do
folder=$(echo $folder | tr -s '/-' '_')
run_$folder
done
Signed-off-by: Hangbin Liu <[email protected]>
---
v5:
Forgot to update commit description for the new added -l option
v4:
Add parameter -l to list available tests, suggested by Bird, Tim
v3:
1) rebase the patch to latest code
2) move `tr -s "/-" "_"` in for loop at the end so user could use test
folder name directly. Before the fix, user need to use
./run_kselftest.sh -t 'networking_forwarding'. Now they can just run
./run_kselftest.sh -t 'networking/forwarding' directly.
v2: update document and commit description.
---
Documentation/dev-tools/kselftest.rst | 8 +++++
tools/testing/selftests/Makefile | 51 +++++++++++++++++++++------
tools/testing/selftests/lib.mk | 2 +-
3 files changed, 50 insertions(+), 11 deletions(-)
diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
index 469d115a95f1..7b92f9c177f6 100644
--- a/Documentation/dev-tools/kselftest.rst
+++ b/Documentation/dev-tools/kselftest.rst
@@ -151,6 +151,14 @@ note some tests will require root privileges::
$ cd kselftest
$ ./run_kselftest.sh
+Or you can run some specific test cases in the installed Kselftests by::
+
+ $ ./run_kselftest.sh -t "bpf size timers"
+
+You can view the available tests to run with::
+
+ $ ./run_kselftest.sh -l
+
Packaging selftests
===================
diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 15c1c1359c50..4c8159dd2bd7 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -225,13 +225,9 @@ ifdef INSTALL_PATH
@# Ask all targets to emit their test scripts
echo "#!/bin/sh" > $(ALL_SCRIPT)
echo "BASE_DIR=\$$(realpath \$$(dirname \$$0))" >> $(ALL_SCRIPT)
- echo "cd \$$BASE_DIR" >> $(ALL_SCRIPT)
echo ". ./kselftest/runner.sh" >> $(ALL_SCRIPT)
- echo "ROOT=\$$PWD" >> $(ALL_SCRIPT)
- echo "if [ \"\$$1\" = \"--summary\" ]; then" >> $(ALL_SCRIPT)
- echo " logfile=\$$BASE_DIR/output.log" >> $(ALL_SCRIPT)
- echo " cat /dev/null > \$$logfile" >> $(ALL_SCRIPT)
- echo "fi" >> $(ALL_SCRIPT)
+ echo "TESTS=\"$(TARGETS)\"" >> $(ALL_SCRIPT)
+ echo "" >> $(ALL_SCRIPT);
@# While building run_kselftest.sh skip also non-existent TARGET dirs:
@# they could be the result of a build failure and should NOT be
@@ -239,15 +235,50 @@ ifdef INSTALL_PATH
for TARGET in $(TARGETS); do \
BUILD_TARGET=$$BUILD/$$TARGET; \
[ ! -d $(INSTALL_PATH)/$$TARGET ] && echo "Skipping non-existent dir: $$TARGET" && continue; \
- echo "[ -w /dev/kmsg ] && echo \"kselftest: Running tests in $$TARGET\" >> /dev/kmsg" >> $(ALL_SCRIPT); \
- echo "cd $$TARGET" >> $(ALL_SCRIPT); \
- echo -n "run_many" >> $(ALL_SCRIPT); \
+ echo "run_$$TARGET()" | tr -s "/-" "_" >> $(ALL_SCRIPT); \
+ echo "{" >> $(ALL_SCRIPT); \
+ echo -e "\t[ -w /dev/kmsg ] && echo \"kselftest: Running tests in $$TARGET\" >> /dev/kmsg" >> $(ALL_SCRIPT); \
+ echo -e "\tcd $$TARGET" >> $(ALL_SCRIPT); \
+ echo -en "\trun_many" >> $(ALL_SCRIPT); \
echo -n "Emit Tests for $$TARGET\n"; \
$(MAKE) -s --no-print-directory OUTPUT=$$BUILD_TARGET -C $$TARGET emit_tests >> $(ALL_SCRIPT); \
echo "" >> $(ALL_SCRIPT); \
- echo "cd \$$ROOT" >> $(ALL_SCRIPT); \
+ echo -e "\tcd \$$ROOT" >> $(ALL_SCRIPT); \
+ echo "}" >> $(ALL_SCRIPT); \
+ echo "" >> $(ALL_SCRIPT); \
done;
+ echo "usage()" >> $(ALL_SCRIPT);
+ echo "{" >> $(ALL_SCRIPT);
+ echo -e "\tcat <<EOF" >> $(ALL_SCRIPT);
+ echo "usage: \$${0##*/} OPTS" >> $(ALL_SCRIPT);
+ echo -e "\t-s | --summary\t\tOnly print summary info and put detailed log in output.log" >> $(ALL_SCRIPT);
+ echo -e "\t-t | --tests\t\tTest name you want to run specifically" >> $(ALL_SCRIPT);
+ echo -e "\t-l | --list\t\tList the available tests" >> $(ALL_SCRIPT);
+ echo -e "\t-h | --help\t\tShow this usage info" >> $(ALL_SCRIPT);
+ echo "EOF" >> $(ALL_SCRIPT);
+ echo "}" >> $(ALL_SCRIPT);
+ echo "" >> $(ALL_SCRIPT);
+
+ echo "while true; do" >> $(ALL_SCRIPT);
+ echo -e "\tcase \"\$$1\" in" >> $(ALL_SCRIPT);
+ echo -e "\t-s | --summary ) logfile=\$$BASE_DIR/output.log; cat /dev/null > \$$logfile; shift ;;" >> $(ALL_SCRIPT);
+ echo -e "\t-t | --tests ) TESTS=\$$2; shift 2 ;;" >> $(ALL_SCRIPT);
+ echo -e "\t-l | --list ) echo \$$TESTS; exit 0 ;;" >> $(ALL_SCRIPT);
+ echo -e "\t-h | --help ) usage; exit 0 ;;" >> $(ALL_SCRIPT);
+ echo -e "\t\"\" ) break;;" >> $(ALL_SCRIPT);
+ echo -e "\t* ) usage; exit 1;;" >> $(ALL_SCRIPT);
+ echo -e "\tesac" >> $(ALL_SCRIPT);
+ echo "done" >> $(ALL_SCRIPT);
+ echo "" >> $(ALL_SCRIPT);
+
+ echo "cd \$$BASE_DIR" >> $(ALL_SCRIPT)
+ echo "ROOT=\$$PWD" >> $(ALL_SCRIPT)
+
+ echo "for folder in \$$TESTS; do" >> $(ALL_SCRIPT); \
+ echo -e "\tfolder=\$$(echo \$$folder | tr -s '/-' '_')" >> $(ALL_SCRIPT); \
+ echo -e "\trun_\$$folder" >> $(ALL_SCRIPT); \
+ echo "done" >> $(ALL_SCRIPT); \
chmod u+x $(ALL_SCRIPT)
else
$(error Error: set INSTALL_PATH to use install)
diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
index 51124b962d56..3c4c94a5d184 100644
--- a/tools/testing/selftests/lib.mk
+++ b/tools/testing/selftests/lib.mk
@@ -108,7 +108,7 @@ emit_tests:
for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \
BASENAME_TEST=`basename $$TEST`; \
echo " \\"; \
- echo -n " \"$$BASENAME_TEST\""; \
+ echo -ne "\t\t\"$$BASENAME_TEST\""; \
done; \
# define if isn't already. It is undefined in make O= case.
--
2.25.4
On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <[email protected]> wrote:
>
> Currently, after generating run_kselftest.sh, there is no way to choose
> which test we could run. All the tests are listed together and we have
> to run all every time. This patch enhanced the run_kselftest.sh to make
> the tests individually selectable. e.g.
>
> $ ./run_kselftest.sh -t "bpf size timers"
My test run break on linux next
./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
./run_kselftest.sh: line 1331: `-e -s | --summary )
logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'
steps do run:
# run_kselftest.sh file generated by kselftest Makefile and included in tarball
./run_kselftest.sh 2>&1 | tee "${LOGFILE}"
ref:
https://github.com/nareshkamboju/test-definitions/blob/master/automated/linux/kselftest/kselftest.sh#L222
full test run log:
https://lkft.validation.linaro.org/scheduler/job/1786826#L1391
>
> Before the patch:
> ================
>
> $ cat run_kselftest.sh
> \#!/bin/sh
> BASE_DIR=$(realpath $(dirname $0))
> cd $BASE_DIR
> . ./kselftest/runner.sh
> ROOT=$PWD
> if [ "$1" = "--summary" ]; then
> logfile=$BASE_DIR/output.log
> cat /dev/null > $logfile
> fi
> [ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg
> cd android
> run_many \
> "run.sh"
> cd $ROOT
> ...<snip>...
> [ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg
> cd zram
> run_many \
> "zram.sh"
> cd $ROOT
>
> After the patch:
> ===============
>
> $ cat run_kselftest.sh
> \#!/bin/sh
> BASE_DIR=$(realpath $(dirname $0))
> . ./kselftest/runner.sh
> TESTS="android ...<snip>... filesystems/binderfs ...<snip>... zram"
>
> run_android()
> {
> [ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg
> cd android
> run_many \
> "run.sh"
> cd $ROOT
> }
>
> ...<snip>...
>
> run_filesystems_binderfs()
> {
> [ -w /dev/kmsg ] && echo "kselftest: Running tests in filesystems/binderfs" >> /dev/kmsg
> cd filesystems/binderfs
> run_many \
> "binderfs_test"
> cd $ROOT
> }
>
> ...<snip>...
>
> run_zram()
> {
> [ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg
> cd zram
> run_many \
> "zram.sh"
> cd $ROOT
> }
>
> usage()
> {
> cat <<EOF
> usage: ${0##*/} OPTS
> -s | --summary Only print summary info and put detailed log in output.log
> -t | --tests Test name you want to run specifically
> -h | --help Show this usage info
> EOF
> }
>
> while true; do
> case "$1" in
> -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;
> -t | --tests ) TESTS=$2; shift 2 ;;
> -l | --list ) echo $TESTS; exit 0 ;;
> -h | --help ) usage; exit 0 ;;
> "" ) break;;
> * ) usage; exit 1;;
> esac
> done
>
> cd $BASE_DIR
> ROOT=$PWD
> for folder in $TESTS; do
> folder=$(echo $folder | tr -s '/-' '_')
> run_$folder
> done
>
> Signed-off-by: Hangbin Liu <[email protected]>
Reported-by: Naresh Kamboju <[email protected]>
--
Linaro LKFT
https://lkft.linaro.org
On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote:
> On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <[email protected]> wrote:
> >
> > Currently, after generating run_kselftest.sh, there is no way to choose
> > which test we could run. All the tests are listed together and we have
> > to run all every time. This patch enhanced the run_kselftest.sh to make
> > the tests individually selectable. e.g.
> >
> > $ ./run_kselftest.sh -t "bpf size timers"
>
> My test run break on linux next
>
> ./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
> ./run_kselftest.sh: line 1331: `-e -s | --summary )
> logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'
Yes, please revert this patch. The resulting script is completely
trashed:
BASE_DIR=$(realpath $(dirname $0))
. ./kselftest/runner.sh
TESTS="seccomp"
run_seccomp()
{
-e [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
-e cd seccomp
-en run_many
\
-ne "seccomp_bpf"
\
-ne "seccomp_benchmark"
-e cd $ROOT
}
--
Kees Cook
On 9/25/20 3:16 PM, Kees Cook wrote:
> On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote:
>> On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <[email protected]> wrote:
>>>
>>> Currently, after generating run_kselftest.sh, there is no way to choose
>>> which test we could run. All the tests are listed together and we have
>>> to run all every time. This patch enhanced the run_kselftest.sh to make
>>> the tests individually selectable. e.g.
>>>
>>> $ ./run_kselftest.sh -t "bpf size timers"
>>
>> My test run break on linux next
>>
>> ./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
>> ./run_kselftest.sh: line 1331: `-e -s | --summary )
>> logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'
>
> Yes, please revert this patch. The resulting script is completely
> trashed:
>
> BASE_DIR=$(realpath $(dirname $0))
> . ./kselftest/runner.sh
> TESTS="seccomp"
>
> run_seccomp()
> {
> -e [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
> -e cd seccomp
> -en run_many
> \
> -ne "seccomp_bpf"
> \
> -ne "seccomp_benchmark"
>
> -e cd $ROOT
> }
>
>
>
Will do.
thanks,
-- Shuah
On 9/25/20 3:16 PM, Kees Cook wrote:
> On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote:
>> On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <[email protected]> wrote:
>>>
>>> Currently, after generating run_kselftest.sh, there is no way to choose
>>> which test we could run. All the tests are listed together and we have
>>> to run all every time. This patch enhanced the run_kselftest.sh to make
>>> the tests individually selectable. e.g.
>>>
>>> $ ./run_kselftest.sh -t "bpf size timers"
>>
>> My test run break on linux next
>>
>> ./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
>> ./run_kselftest.sh: line 1331: `-e -s | --summary )
>> logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'
>
> Yes, please revert this patch. The resulting script is completely
> trashed:
>
Thank you both. Now reverted.
thanks,
-- Shuah
On Fri, Sep 25, 2020 at 05:06:02PM -0600, Shuah Khan wrote:
> On 9/25/20 3:16 PM, Kees Cook wrote:
> > On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote:
> > > On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <[email protected]> wrote:
> > > >
> > > > Currently, after generating run_kselftest.sh, there is no way to choose
> > > > which test we could run. All the tests are listed together and we have
> > > > to run all every time. This patch enhanced the run_kselftest.sh to make
> > > > the tests individually selectable. e.g.
> > > >
> > > > $ ./run_kselftest.sh -t "bpf size timers"
> > >
> > > My test run break on linux next
> > >
> > > ./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
> > > ./run_kselftest.sh: line 1331: `-e -s | --summary )
> > > logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'
> >
> > Yes, please revert this patch. The resulting script is completely
> > trashed:
> >
>
> Thank you both. Now reverted.
I've sent an alternative that I think should do nicely. It will work
well with LAVA as well.
--
Kees Cook
On Fri, Sep 25, 2020 at 02:16:14PM -0700, Kees Cook wrote:
> On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote:
> > On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <[email protected]> wrote:
> > >
> > > Currently, after generating run_kselftest.sh, there is no way to choose
> > > which test we could run. All the tests are listed together and we have
> > > to run all every time. This patch enhanced the run_kselftest.sh to make
> > > the tests individually selectable. e.g.
> > >
> > > $ ./run_kselftest.sh -t "bpf size timers"
> >
> > My test run break on linux next
> >
> > ./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
> > ./run_kselftest.sh: line 1331: `-e -s | --summary )
> > logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'
>
> Yes, please revert this patch. The resulting script is completely
> trashed:
>
> BASE_DIR=$(realpath $(dirname $0))
> . ./kselftest/runner.sh
> TESTS="seccomp"
>
> run_seccomp()
> {
> -e [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
> -e cd seccomp
> -en run_many
> \
> -ne "seccomp_bpf"
> \
> -ne "seccomp_benchmark"
>
> -e cd $ROOT
> }
I'm really sorry to make this trouble. And I'm OK to revert the patch.
I just a little wondering how do you generate this script.
I tested with 'make -C tools/testing/selftests gen_tar FORMAT=.xz' before
post the patch and all looks good to me.
```
run_seccomp()
{
[ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
cd seccomp
run_many \
"seccomp_bpf" \
"seccomp_benchmark"
cd $ROOT
}
```
Thanks
Hangbin
On Sun, Sep 27, 2020 at 09:53:34AM +0800, Hangbin Liu wrote:
> On Fri, Sep 25, 2020 at 02:16:14PM -0700, Kees Cook wrote:
> > On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote:
> > > On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <[email protected]> wrote:
> > > >
> > > > Currently, after generating run_kselftest.sh, there is no way to choose
> > > > which test we could run. All the tests are listed together and we have
> > > > to run all every time. This patch enhanced the run_kselftest.sh to make
> > > > the tests individually selectable. e.g.
> > > >
> > > > $ ./run_kselftest.sh -t "bpf size timers"
> > >
> > > My test run break on linux next
> > >
> > > ./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
> > > ./run_kselftest.sh: line 1331: `-e -s | --summary )
> > > logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'
> >
> > Yes, please revert this patch. The resulting script is completely
> > trashed:
> >
> > BASE_DIR=$(realpath $(dirname $0))
> > . ./kselftest/runner.sh
> > TESTS="seccomp"
> >
> > run_seccomp()
> > {
> > -e [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
> > -e cd seccomp
> > -en run_many
> > \
> > -ne "seccomp_bpf"
> > \
> > -ne "seccomp_benchmark"
> >
> > -e cd $ROOT
> > }
>
> I'm really sorry to make this trouble. And I'm OK to revert the patch.
> I just a little wondering how do you generate this script.
This issue is with which shell is used. I suspect your /bin/sh is full
/bin/bash, where as Naresh's, the CI's, and mine are /bin/dash (which
lacks "-e" support for the built-in "echo").
--
Kees Cook
On Mon, Sep 28, 2020 at 01:06:15PM -0700, Kees Cook wrote:
> > I'm really sorry to make this trouble. And I'm OK to revert the patch.
> > I just a little wondering how do you generate this script.
>
> This issue is with which shell is used. I suspect your /bin/sh is full
> /bin/bash, where as Naresh's, the CI's, and mine are /bin/dash (which
> lacks "-e" support for the built-in "echo").
Ah, got it. Thanks for your explanation.
Regards
Hangbin