The default timeout for selftests tests is 45 seconds. Although
we already have 13 settings for tests of about 96 sefltests which
use a timeout greater than this, we want to try to avoid encouraging
more tests to forcing a higher test timeout as selftests strives to
run all tests quickly. Selftests also uses the timeout as a non-fatal
error. Only tests runners which have control over a system would know
if to treat a timeout as fatal or not.
To help with all this:
o Enhance documentation to avoid future increases of insane timeouts
o Add the option to allow overriding the default timeout with test
runners with a command line option
Suggested-by: Shuah Khan <[email protected]>
Signed-off-by: Luis Chamberlain <[email protected]>
---
Documentation/dev-tools/kselftest.rst | 22 +++++++++++++++++++++
tools/testing/selftests/kselftest/runner.sh | 11 ++++++++++-
tools/testing/selftests/run_kselftest.sh | 5 +++++
3 files changed, 37 insertions(+), 1 deletion(-)
diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
index 12b575b76b20..dd214af7b7ff 100644
--- a/Documentation/dev-tools/kselftest.rst
+++ b/Documentation/dev-tools/kselftest.rst
@@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times::
For other features see the script usage output, seen with the `-h` option.
+Timeout for selftests
+=====================
+
+Selftests are designed to be quick and so a default timeout is used of 45
+seconds for each test. Tests can override the default timeout by adding
+a settings file in their directory and set a timeout variable there to the
+configured a desired upper timeout for the test. Only a few tests override
+the timeout with a value higher than 45 seconds, selftests strives to keep
+it that way. Timeouts in selftests are not considered fatal because the
+system under which a test runs may change and this can also modify the
+expected time it takes to run a test. If you have control over the systems
+which will run the tests you can configure a test runner on those systems to
+use a greater or lower timeout on the command line as with the `-o` or
+the `--override-timeout` argument. For example to use 165 seconds instead
+one would use:
+
+ $ ./run_kselftest.sh --override-timeout 165
+
+You can look at the TAP output to see if you ran into the timeout. Test
+runners which know a test must run under a specific time can then optionally
+treat these timeouts then as fatal.
+
Packaging selftests
===================
diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh
index 294619ade49f..1c952d1401d4 100644
--- a/tools/testing/selftests/kselftest/runner.sh
+++ b/tools/testing/selftests/kselftest/runner.sh
@@ -8,7 +8,8 @@ export logfile=/dev/stdout
export per_test_logging=
# Defaults for "settings" file fields:
-# "timeout" how many seconds to let each test run before failing.
+# "timeout" how many seconds to let each test run before running
+# over our soft timeout limit.
export kselftest_default_timeout=45
# There isn't a shell-agnostic way to find the path of a sourced file,
@@ -90,6 +91,14 @@ run_one()
done < "$settings"
fi
+ # Command line timeout overrides the settings file
+ if [ -n "$kselftest_override_timeout" ]; then
+ kselftest_timeout="$kselftest_override_timeout"
+ echo "# overriding timeout to $kselftest_timeout" >> "$logfile"
+ else
+ echo "# timeout set to $kselftest_timeout" >> "$logfile"
+ fi
+
TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
echo "# $TEST_HDR_MSG"
if [ ! -e "$TEST" ]; then
diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh
index 97165a83df63..9a981b36bd7f 100755
--- a/tools/testing/selftests/run_kselftest.sh
+++ b/tools/testing/selftests/run_kselftest.sh
@@ -26,6 +26,7 @@ Usage: $0 [OPTIONS]
-l | --list List the available collection:test entries
-d | --dry-run Don't actually run any tests
-h | --help Show this usage info
+ -o | --override-timeout Number of seconds after which we timeout
EOF
exit $1
}
@@ -33,6 +34,7 @@ EOF
COLLECTIONS=""
TESTS=""
dryrun=""
+kselftest_override_timeout=""
while true; do
case "$1" in
-s | --summary)
@@ -51,6 +53,9 @@ while true; do
-d | --dry-run)
dryrun="echo"
shift ;;
+ -o | --override-timeout)
+ kselftest_override_timeout="$2"
+ shift 2 ;;
-h | --help)
usage 0 ;;
"")
--
2.39.2
On 4/15/23 12:38 AM, Luis Chamberlain wrote:
> The default timeout for selftests tests is 45 seconds. Although
> we already have 13 settings for tests of about 96 sefltests which
> use a timeout greater than this, we want to try to avoid encouraging
> more tests to forcing a higher test timeout as selftests strives to
> run all tests quickly. Selftests also uses the timeout as a non-fatal
> error. Only tests runners which have control over a system would know
> if to treat a timeout as fatal or not.
>
> To help with all this:
>
> o Enhance documentation to avoid future increases of insane timeouts
> o Add the option to allow overriding the default timeout with test
> runners with a command line option
>
> Suggested-by: Shuah Khan <[email protected]>
> Signed-off-by: Luis Chamberlain <[email protected]>
Reviewed-by: Muhammad Usama Anjum <[email protected]>
Tested-by:Muhammad Usama Anjum <[email protected]>
> ---
> Documentation/dev-tools/kselftest.rst | 22 +++++++++++++++++++++
> tools/testing/selftests/kselftest/runner.sh | 11 ++++++++++-
> tools/testing/selftests/run_kselftest.sh | 5 +++++
> 3 files changed, 37 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
> index 12b575b76b20..dd214af7b7ff 100644
> --- a/Documentation/dev-tools/kselftest.rst
> +++ b/Documentation/dev-tools/kselftest.rst
> @@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times::
>
> For other features see the script usage output, seen with the `-h` option.
>
> +Timeout for selftests
> +=====================
> +
> +Selftests are designed to be quick and so a default timeout is used of 45
> +seconds for each test. Tests can override the default timeout by adding
> +a settings file in their directory and set a timeout variable there to the
> +configured a desired upper timeout for the test. Only a few tests override
> +the timeout with a value higher than 45 seconds, selftests strives to keep
> +it that way. Timeouts in selftests are not considered fatal because the
> +system under which a test runs may change and this can also modify the
> +expected time it takes to run a test. If you have control over the systems
> +which will run the tests you can configure a test runner on those systems to
> +use a greater or lower timeout on the command line as with the `-o` or
> +the `--override-timeout` argument. For example to use 165 seconds instead
> +one would use:
> +
> + $ ./run_kselftest.sh --override-timeout 165
> +
> +You can look at the TAP output to see if you ran into the timeout. Test
> +runners which know a test must run under a specific time can then optionally
> +treat these timeouts then as fatal.
> +
> Packaging selftests
> ===================
>
> diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh
> index 294619ade49f..1c952d1401d4 100644
> --- a/tools/testing/selftests/kselftest/runner.sh
> +++ b/tools/testing/selftests/kselftest/runner.sh
> @@ -8,7 +8,8 @@ export logfile=/dev/stdout
> export per_test_logging=
>
> # Defaults for "settings" file fields:
> -# "timeout" how many seconds to let each test run before failing.
> +# "timeout" how many seconds to let each test run before running
> +# over our soft timeout limit.
> export kselftest_default_timeout=45
>
> # There isn't a shell-agnostic way to find the path of a sourced file,
> @@ -90,6 +91,14 @@ run_one()
> done < "$settings"
> fi
>
> + # Command line timeout overrides the settings file
> + if [ -n "$kselftest_override_timeout" ]; then
> + kselftest_timeout="$kselftest_override_timeout"
> + echo "# overriding timeout to $kselftest_timeout" >> "$logfile"
> + else
> + echo "# timeout set to $kselftest_timeout" >> "$logfile"
> + fi
> +
> TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
> echo "# $TEST_HDR_MSG"
> if [ ! -e "$TEST" ]; then
> diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh
> index 97165a83df63..9a981b36bd7f 100755
> --- a/tools/testing/selftests/run_kselftest.sh
> +++ b/tools/testing/selftests/run_kselftest.sh
> @@ -26,6 +26,7 @@ Usage: $0 [OPTIONS]
> -l | --list List the available collection:test entries
> -d | --dry-run Don't actually run any tests
> -h | --help Show this usage info
> + -o | --override-timeout Number of seconds after which we timeout
> EOF
> exit $1
> }
> @@ -33,6 +34,7 @@ EOF
> COLLECTIONS=""
> TESTS=""
> dryrun=""
> +kselftest_override_timeout=""
> while true; do
> case "$1" in
> -s | --summary)
> @@ -51,6 +53,9 @@ while true; do
> -d | --dry-run)
> dryrun="echo"
> shift ;;
> + -o | --override-timeout)
> + kselftest_override_timeout="$2"
> + shift 2 ;;
> -h | --help)
> usage 0 ;;
> "")
--
BR,
Muhammad Usama Anjum
On Fri, Apr 28, 2023 at 1:34 AM Muhammad Usama Anjum
<[email protected]> wrote:
>
> On 4/15/23 12:38 AM, Luis Chamberlain wrote:
> > The default timeout for selftests tests is 45 seconds. Although
> > we already have 13 settings for tests of about 96 sefltests which
> > use a timeout greater than this, we want to try to avoid encouraging
> > more tests to forcing a higher test timeout as selftests strives to
> > run all tests quickly. Selftests also uses the timeout as a non-fatal
> > error. Only tests runners which have control over a system would know
> > if to treat a timeout as fatal or not.
> >
> > To help with all this:
> >
> > o Enhance documentation to avoid future increases of insane timeouts
> > o Add the option to allow overriding the default timeout with test
> > runners with a command line option
> >
> > Suggested-by: Shuah Khan <[email protected]>
> > Signed-off-by: Luis Chamberlain <[email protected]>
> Reviewed-by: Muhammad Usama Anjum <[email protected]>
> Tested-by:Muhammad Usama Anjum <[email protected]>
Shuah, just a friendly poke! This is needed to allow me to enable full
automation for kdevops for selftests.
Luis
On Thu, May 11, 2023 at 08:26:42AM -0700, Luis Chamberlain wrote:
> On Fri, Apr 28, 2023 at 1:34 AM Muhammad Usama Anjum
> <[email protected]> wrote:
> >
> > On 4/15/23 12:38 AM, Luis Chamberlain wrote:
> > > The default timeout for selftests tests is 45 seconds. Although
> > > we already have 13 settings for tests of about 96 sefltests which
> > > use a timeout greater than this, we want to try to avoid encouraging
> > > more tests to forcing a higher test timeout as selftests strives to
> > > run all tests quickly. Selftests also uses the timeout as a non-fatal
> > > error. Only tests runners which have control over a system would know
> > > if to treat a timeout as fatal or not.
> > >
> > > To help with all this:
> > >
> > > o Enhance documentation to avoid future increases of insane timeouts
> > > o Add the option to allow overriding the default timeout with test
> > > runners with a command line option
> > >
> > > Suggested-by: Shuah Khan <[email protected]>
> > > Signed-off-by: Luis Chamberlain <[email protected]>
> > Reviewed-by: Muhammad Usama Anjum <[email protected]>
> > Tested-by:Muhammad Usama Anjum <[email protected]>
>
> Shuah, just a friendly poke! This is needed to allow me to enable full
> automation for kdevops for selftests.
Shuah, friendly re-poke.
Luis
On 5/24/23 16:58, Luis Chamberlain wrote:
> On Thu, May 11, 2023 at 08:26:42AM -0700, Luis Chamberlain wrote:
>> On Fri, Apr 28, 2023 at 1:34 AM Muhammad Usama Anjum
>> <[email protected]> wrote:
>>>
>>> On 4/15/23 12:38 AM, Luis Chamberlain wrote:
>>>> The default timeout for selftests tests is 45 seconds. Although
>>>> we already have 13 settings for tests of about 96 sefltests which
>>>> use a timeout greater than this, we want to try to avoid encouraging
>>>> more tests to forcing a higher test timeout as selftests strives to
>>>> run all tests quickly. Selftests also uses the timeout as a non-fatal
>>>> error. Only tests runners which have control over a system would know
>>>> if to treat a timeout as fatal or not.
>>>>
>>>> To help with all this:
>>>>
>>>> o Enhance documentation to avoid future increases of insane timeouts
>>>> o Add the option to allow overriding the default timeout with test
>>>> runners with a command line option
>>>>
>>>> Suggested-by: Shuah Khan <[email protected]>
>>>> Signed-off-by: Luis Chamberlain <[email protected]>
>>> Reviewed-by: Muhammad Usama Anjum <[email protected]>
>>> Tested-by:Muhammad Usama Anjum <[email protected]>
>>
>> Shuah, just a friendly poke! This is needed to allow me to enable full
>> automation for kdevops for selftests.
>
> Shuah, friendly re-poke.
>
> Luis
Thanks. I will apply this for next.
thanks,
-- Shuah