The current selftests infrastructure formats the results in TAP 13. This
version doesn't support subtests and only the end result of each
selftest is taken into account. It means that a single issue in a
subtest of a selftest containing multiple subtests forces the whole
selftest to be marked as failed. It also means that subtests results are
not tracked by CI executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important
to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels
with the last version of the test suite: tests might fail because a
feature is not supported but the test didn't skip that part. In this
case, if subtests are not tracked, the whole selftest will be marked as
failed making the other subtests useless because their results are
ignored.
Regarding this patch set:
- The two first patches modify connect and userspace_pm selftests to
continue executing other tests if there is an error before the end.
This is what is done in the other MPTCP selftests.
- Patches 3-5 are refactoring the code in userspace_pm selftest to
reduce duplicated code, suppress some shellcheck warnings and prepare
subtests' support by using new helpers.
- Patch 6 adds new helpers in mptcp_lib.sh to easily support printing
the subtests results in the different MPTCP selftests.
- Patch 7-13 format subtests results in TAP 13 in the different MPTCP
selftests.
Signed-off-by: Matthieu Baerts <[email protected]>
---
Matthieu Baerts (13):
selftests: mptcp: connect: don't stop if error
selftests: mptcp: userspace pm: don't stop if error
selftests: mptcp: userspace_pm: fix shellcheck warnings
selftests: mptcp: userspace_pm: uniform results printing
selftests: mptcp: userspace_pm: reduce dup code around printf
selftests: mptcp: lib: format subtests results in TAP
selftests: mptcp: connect: format subtests results in TAP
selftests: mptcp: pm_netlink: format subtests results in TAP
selftests: mptcp: join: format subtests results in TAP
selftests: mptcp: diag: format subtests results in TAP
selftests: mptcp: simult flows: format subtests results in TAP
selftests: mptcp: sockopt: format subtests results in TAP
selftests: mptcp: userspace_pm: format subtests results in TAP
tools/testing/selftests/net/mptcp/diag.sh | 7 +
tools/testing/selftests/net/mptcp/mptcp_connect.sh | 66 ++++++--
tools/testing/selftests/net/mptcp/mptcp_join.sh | 37 ++++-
tools/testing/selftests/net/mptcp/mptcp_lib.sh | 66 ++++++++
tools/testing/selftests/net/mptcp/mptcp_sockopt.sh | 20 ++-
tools/testing/selftests/net/mptcp/pm_netlink.sh | 6 +-
tools/testing/selftests/net/mptcp/simult_flows.sh | 4 +
tools/testing/selftests/net/mptcp/userspace_pm.sh | 181 +++++++++++++--------
8 files changed, 298 insertions(+), 89 deletions(-)
---
base-commit: 60cc1f7d0605598b47ee3c0c2b4b6fbd4da50a06
change-id: 20230712-upstream-net-next-20230712-selftests-mptcp-subtests-25d250d77886
Best regards,
--
Matthieu Baerts <[email protected]>
The current selftests infrastructure formats the results in TAP 13. This
version doesn't support subtests and only the end result of each
selftest is taken into account. It means that a single issue in a
subtest of a selftest containing multiple subtests forces the whole
selftest to be marked as failed. It also means that subtests results are
not tracked by CIs executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important
to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels
with the last version of the test suite: tests might fail because a
feature is not supported but the test didn't skip that part. In this
case, if subtests are not tracked, the whole selftest will be marked as
failed making the other subtests useless because their results are
ignored.
This patch adds some helpers in mptcp_lib.sh to be able to easily format
subtests results in TAP in the different MPTCP selftests.
Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/368
Acked-by: Paolo Abeni <[email protected]>
Signed-off-by: Matthieu Baerts <[email protected]>
---
tools/testing/selftests/net/mptcp/mptcp_lib.sh | 66 ++++++++++++++++++++++++++
1 file changed, 66 insertions(+)
diff --git a/tools/testing/selftests/net/mptcp/mptcp_lib.sh b/tools/testing/selftests/net/mptcp/mptcp_lib.sh
index f32045b23b89..b1a0fdd0408b 100644
--- a/tools/testing/selftests/net/mptcp/mptcp_lib.sh
+++ b/tools/testing/selftests/net/mptcp/mptcp_lib.sh
@@ -1,8 +1,12 @@
#! /bin/bash
# SPDX-License-Identifier: GPL-2.0
+readonly KSFT_PASS=0
readonly KSFT_FAIL=1
readonly KSFT_SKIP=4
+readonly KSFT_TEST=$(basename "${0}" | sed 's/\.sh$//g')
+
+MPTCP_LIB_SUBTESTS=()
# SELFTESTS_MPTCP_LIB_EXPECT_ALL_FEATURES env var can be set when validating all
# features using the last version of the kernel and the selftests to make sure
@@ -102,3 +106,65 @@ mptcp_lib_kversion_ge() {
mptcp_lib_fail_if_expected_feature "kernel version ${1} lower than ${v}"
}
+
+__mptcp_lib_result_add() {
+ local result="${1}"
+ shift
+
+ local id=$((${#MPTCP_LIB_SUBTESTS[@]} + 1))
+
+ MPTCP_LIB_SUBTESTS+=("${result} ${id} - ${KSFT_TEST}: ${*}")
+}
+
+# $1: test name
+mptcp_lib_result_pass() {
+ __mptcp_lib_result_add "ok" "${1}"
+}
+
+# $1: test name
+mptcp_lib_result_fail() {
+ __mptcp_lib_result_add "not ok" "${1}"
+}
+
+# $1: test name
+mptcp_lib_result_skip() {
+ __mptcp_lib_result_add "ok" "${1} # SKIP"
+}
+
+# $1: result code ; $2: test name
+mptcp_lib_result_code() {
+ local ret="${1}"
+ local name="${2}"
+
+ case "${ret}" in
+ "${KSFT_PASS}")
+ mptcp_lib_result_pass "${name}"
+ ;;
+ "${KSFT_FAIL}")
+ mptcp_lib_result_fail "${name}"
+ ;;
+ "${KSFT_SKIP}")
+ mptcp_lib_result_skip "${name}"
+ ;;
+ *)
+ echo "ERROR: wrong result code: ${ret}"
+ exit ${KSFT_FAIL}
+ ;;
+ esac
+}
+
+mptcp_lib_result_print_all_tap() {
+ local subtest
+
+ if [ ${#MPTCP_LIB_SUBTESTS[@]} -eq 0 ] ||
+ [ "${SELFTESTS_MPTCP_LIB_NO_TAP:-}" = "1" ]; then
+ return
+ fi
+
+ printf "\nTAP version 13\n"
+ printf "1..%d\n" "${#MPTCP_LIB_SUBTESTS[@]}"
+
+ for subtest in "${MPTCP_LIB_SUBTESTS[@]}"; do
+ printf "%s\n" "${subtest}"
+ done
+}
--
2.40.1
The current selftests infrastructure formats the results in TAP 13. This
version doesn't support subtests and only the end result of each
selftest is taken into account. It means that a single issue in a
subtest of a selftest containing multiple subtests forces the whole
selftest to be marked as failed. It also means that subtests results are
not tracked by CIs executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important
to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels
with the last version of the test suite: tests might fail because a
feature is not supported but the test didn't skip that part. In this
case, if subtests are not tracked, the whole selftest will be marked as
failed making the other subtests useless because their results are
ignored.
This patch formats subtests results in TAP in simult_flows.sh selftest.
Link: https://github.com/multipath-tcp/mptcp_net-next/issues/368
Acked-by: Paolo Abeni <[email protected]>
Signed-off-by: Matthieu Baerts <[email protected]>
---
tools/testing/selftests/net/mptcp/simult_flows.sh | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/tools/testing/selftests/net/mptcp/simult_flows.sh b/tools/testing/selftests/net/mptcp/simult_flows.sh
index 36a3c9d92e20..ce9203b817f8 100755
--- a/tools/testing/selftests/net/mptcp/simult_flows.sh
+++ b/tools/testing/selftests/net/mptcp/simult_flows.sh
@@ -261,6 +261,7 @@ run_test()
printf "%-60s" "$msg"
do_transfer $small $large $time
lret=$?
+ mptcp_lib_result_code "${lret}" "${msg}"
if [ $lret -ne 0 ]; then
ret=$lret
[ $bail -eq 0 ] || exit $ret
@@ -269,6 +270,7 @@ run_test()
printf "%-60s" "$msg - reverse direction"
do_transfer $large $small $time
lret=$?
+ mptcp_lib_result_code "${lret}" "${msg}"
if [ $lret -ne 0 ]; then
ret=$lret
[ $bail -eq 0 ] || exit $ret
@@ -305,4 +307,6 @@ run_test 10 10 1 50 "balanced bwidth with unbalanced delay"
run_test 30 10 0 0 "unbalanced bwidth"
run_test 30 10 1 50 "unbalanced bwidth with unbalanced delay"
run_test 30 10 50 1 "unbalanced bwidth with opposed, unbalanced delay"
+
+mptcp_lib_result_print_all_tap
exit $ret
--
2.40.1
The current selftests infrastructure formats the results in TAP 13. This
version doesn't support subtests and only the end result of each
selftest is taken into account. It means that a single issue in a
subtest of a selftest containing multiple subtests forces the whole
selftest to be marked as failed. It also means that subtests results are
not tracked by CIs executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important
to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels
with the last version of the test suite: tests might fail because a
feature is not supported but the test didn't skip that part. In this
case, if subtests are not tracked, the whole selftest will be marked as
failed making the other subtests useless because their results are
ignored.
This patch formats subtests results in TAP in diag.sh selftest.
Link: https://github.com/multipath-tcp/mptcp_net-next/issues/368
Acked-by: Paolo Abeni <[email protected]>
Signed-off-by: Matthieu Baerts <[email protected]>
---
tools/testing/selftests/net/mptcp/diag.sh | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/tools/testing/selftests/net/mptcp/diag.sh b/tools/testing/selftests/net/mptcp/diag.sh
index fa9e09ad97d9..85a8ee9395b3 100755
--- a/tools/testing/selftests/net/mptcp/diag.sh
+++ b/tools/testing/selftests/net/mptcp/diag.sh
@@ -65,12 +65,15 @@ __chk_nr()
if [ $nr != $expected ]; then
if [ $nr = "$skip" ] && ! mptcp_lib_expect_all_features; then
echo "[ skip ] Feature probably not supported"
+ mptcp_lib_result_skip "${msg}"
else
echo "[ fail ] expected $expected found $nr"
+ mptcp_lib_result_fail "${msg}"
ret=$test_cnt
fi
else
echo "[ ok ]"
+ mptcp_lib_result_pass "${msg}"
fi
test_cnt=$((test_cnt+1))
}
@@ -111,12 +114,15 @@ wait_msk_nr()
printf "%-50s" "$msg"
if [ $i -ge $timeout ]; then
echo "[ fail ] timeout while expecting $expected max $max last $nr"
+ mptcp_lib_result_fail "${msg} # timeout"
ret=$test_cnt
elif [ $nr != $expected ]; then
echo "[ fail ] expected $expected found $nr"
+ mptcp_lib_result_fail "${msg} # unexpected result"
ret=$test_cnt
else
echo "[ ok ]"
+ mptcp_lib_result_pass "${msg}"
fi
test_cnt=$((test_cnt+1))
}
@@ -276,4 +282,5 @@ flush_pids
chk_msk_inuse 0 "....chk 0 msk in use after flush"
+mptcp_lib_result_print_all_tap
exit $ret
--
2.40.1
Hello:
This series was applied to netdev/net-next.git (main)
by David S. Miller <[email protected]>:
On Mon, 17 Jul 2023 15:21:20 +0200 you wrote:
> The current selftests infrastructure formats the results in TAP 13. This
> version doesn't support subtests and only the end result of each
> selftest is taken into account. It means that a single issue in a
> subtest of a selftest containing multiple subtests forces the whole
> selftest to be marked as failed. It also means that subtests results are
> not tracked by CI executing selftests.
>
> [...]
Here is the summary with links:
- [net-next,01/13] selftests: mptcp: connect: don't stop if error
https://git.kernel.org/netdev/net-next/c/edbc16c43b27
- [net-next,02/13] selftests: mptcp: userspace pm: don't stop if error
https://git.kernel.org/netdev/net-next/c/e141c1e8e4c1
- [net-next,03/13] selftests: mptcp: userspace_pm: fix shellcheck warnings
https://git.kernel.org/netdev/net-next/c/8320b1387a15
- [net-next,04/13] selftests: mptcp: userspace_pm: uniform results printing
https://git.kernel.org/netdev/net-next/c/e198ad759273
- [net-next,05/13] selftests: mptcp: userspace_pm: reduce dup code around printf
https://git.kernel.org/netdev/net-next/c/d8463d81652d
- [net-next,06/13] selftests: mptcp: lib: format subtests results in TAP
https://git.kernel.org/netdev/net-next/c/c4192967e62f
- [net-next,07/13] selftests: mptcp: connect: format subtests results in TAP
https://git.kernel.org/netdev/net-next/c/dd350f46e35e
- [net-next,08/13] selftests: mptcp: pm_netlink: format subtests results in TAP
https://git.kernel.org/netdev/net-next/c/d85555ac11f9
- [net-next,09/13] selftests: mptcp: join: format subtests results in TAP
https://git.kernel.org/netdev/net-next/c/7f117cd37c61
- [net-next,10/13] selftests: mptcp: diag: format subtests results in TAP
https://git.kernel.org/netdev/net-next/c/ce9902573652
- [net-next,11/13] selftests: mptcp: simult flows: format subtests results in TAP
https://git.kernel.org/netdev/net-next/c/675d99338e7a
- [net-next,12/13] selftests: mptcp: sockopt: format subtests results in TAP
https://git.kernel.org/netdev/net-next/c/9e86a297796b
- [net-next,13/13] selftests: mptcp: userspace_pm: format subtests results in TAP
https://git.kernel.org/netdev/net-next/c/f589234e1af0
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html