Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <20200611210545.173886-1-urielguajardojr@gmail.com>
In-Reply-To: <20200611210545.173886-1-urielguajardojr@gmail.com>
From:   Brendan Higgins <brendanhiggins@google.com>
Date:   Thu, 11 Jun 2020 14:22:56 -0700
Message-ID: <CAFd5g44qv_GvF09jCS1KxhRRzi9qc9rJ_1n8Bnumd2LaoW2z7g@mail.gmail.com>
Subject: Re: [PATCH] kunit: show error if kunit results are not present
To:     Uriel Guajardo <urielguajardojr@gmail.com>,
        Will Chen <chenwi@google.com>
Cc:     Uriel Guajardo <urielguajardo@google.com>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@vger.kernel.org>,
        KUnit Development <kunit-dev@googlegroups.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

+Will Chen

On Thu, Jun 11, 2020 at 2:17 PM Uriel Guajardo
<urielguajardojr@gmail.com> wrote:
>
> From: Uriel Guajardo <urielguajardo@google.com>
>
> Currently, if the kernel is configured incorrectly or if it crashes before any
> kunit tests are run, kunit finishes without error, reporting
> that 0 test cases were run.
>
> To fix this, an error is shown when the tap header is not found, which
> indicates that kunit was not able to run at all.
>
> Signed-off-by: Uriel Guajardo <urielguajardo@google.com>
> ---
>  tools/testing/kunit/kunit_parser.py               |   8 ++++----
>  tools/testing/kunit/kunit_tool_test.py            |  11 +++++++++++
>  .../kunit/test_data/test_insufficient_memory.log  | Bin 0 -> 856 bytes
>  3 files changed, 15 insertions(+), 4 deletions(-)
>  create mode 100644 tools/testing/kunit/test_data/test_insufficient_memory.log
>
> diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
> index 64aac9dcd431..f13e0c0d6663 100644
> --- a/tools/testing/kunit/kunit_parser.py
> +++ b/tools/testing/kunit/kunit_parser.py
> @@ -265,11 +265,9 @@ def bubble_up_suite_errors(test_suite_list: List[TestSuite]) -> TestStatus:
>         return bubble_up_errors(lambda x: x.status, test_suite_list)
>
>  def parse_test_result(lines: List[str]) -> TestResult:
> -       if not lines:
> -               return TestResult(TestStatus.NO_TESTS, [], lines)
>         consume_non_diagnositic(lines)
> -       if not parse_tap_header(lines):
> -               return None
> +       if not lines or not parse_tap_header(lines):
> +               return TestResult(TestStatus.NO_TESTS, [], lines)
>         test_suites = []
>         test_suite = parse_test_suite(lines)
>         while test_suite:
> @@ -282,6 +280,8 @@ def parse_run_tests(kernel_output) -> TestResult:
>         failed_tests = 0
>         crashed_tests = 0
>         test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
> +       if test_result.status == TestStatus.NO_TESTS:
> +               print_with_timestamp(red('[ERROR] ') + 'no kunit output detected')
>         for test_suite in test_result.suites:
>                 if test_suite.status == TestStatus.SUCCESS:
>                         print_suite_divider(green('[PASSED] ') + test_suite.name)
> diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> index 984588d6ba95..4a7ef2eef831 100755
> --- a/tools/testing/kunit/kunit_tool_test.py
> +++ b/tools/testing/kunit/kunit_tool_test.py
> @@ -170,6 +170,17 @@ class KUnitParserTest(unittest.TestCase):
>                         result.status)
>                 file.close()
>
> +       def test_no_kunit_output(self):
> +               crash_log = get_absolute_path(
> +                       'test_data/test_insufficient_memory.log')
> +               file = open(crash_log)
> +               print_mock = mock.patch('builtins.print').start()
> +               result = kunit_parser.parse_run_tests(
> +                       kunit_parser.isolate_kunit_output(file.readlines()))
> +               print_mock.assert_any_call(StrContains("no kunit output detected"))
> +               print_mock.stop()
> +               file.close()
> +
>         def test_crashed_test(self):
>                 crashed_log = get_absolute_path(
>                         'test_data/test_is_test_passed-crash.log')
> diff --git a/tools/testing/kunit/test_data/test_insufficient_memory.log b/tools/testing/kunit/test_data/test_insufficient_memory.log
> new file mode 100644
> index 0000000000000000000000000000000000000000..6c9e4da7b6d763a03c0c83302b2faa6dc21a6ea6
> GIT binary patch
> literal 856
> zcmbu6Piw<448`yH6xx0L3mw^@hf#VOv}5$J+fZ#KHj8CrJK6H>=Z2=CgO$<hA}n~n
> z_av{ZCy=#qP-&xMfG4=6vR)i)5n%+={(WUX@f3QQc$@8JeI=@sWh+JEQ*#MXVFVXF
> z3Ic%)LLrv~p$(Btnp72VZT95Dcs$;|{9!uqlCTqzsVj`yGB}JvL1F#T<`SpJ4?L6&
> zvLJa#bUk?B5jWafHpCGc^cwdSg)SXJ+Sp0$Q$V&%X;`!D6P@>grwf=!Y9>J$&;ioE
> z(YWH`vCdK5Yv8@UZFe8*_w@>lK?j;o`2XCkM`zDc9)5TBu*ma#3i~#uEu#ge+Mv@N
> z!H}`OJ&aI@v}o*ZK;I-rL20Tal*zuYY-cSL^z~bR2OA<uaygqX^7-Rzeh;N?h?hs>
> F><3Em&*A_8
>
> literal 0
> HcmV?d00001
>
> --
> 2.27.0.278.ge193c7cf3a9-goog
>