Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <20210204225448.1393741-1-dlatypov@google.com>
In-Reply-To: <20210204225448.1393741-1-dlatypov@google.com>
From:   Brendan Higgins <brendanhiggins@google.com>
Date:   Fri, 5 Feb 2021 15:51:56 -0800
Message-ID: <CAFd5g448GebW-zYwRzfCkJqx-7VZZOa_7dfSuqX_pAJLk4+CyQ@mail.gmail.com>
Subject: Re: [PATCH v3 0/3] kunit: support running subsets of test suites from kunit.py
To:     Daniel Latypov <dlatypov@google.com>
Cc:     David Gow <davidgow@google.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@vger.kernel.org>,
        Shuah Khan <skhan@linuxfoundation.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Thu, Feb 4, 2021 at 2:54 PM Daniel Latypov <dlatypov@google.com> wrote:
>
> When using `kunit.py run` to run tests, users must populate a
> `kunitconfig` file to select the options the tests are hidden behind and
> all their dependencies.
>
> The patch [1] to allow specifying a path to kunitconfig promises to make
> this nicer as we can have checked in files corresponding to different
> sets of tests.
>
> But it's still annoying
> 1) when trying to run a subet of tests
> 2) when you want to run tests that don't have such a pre-existing
> kunitconfig and selecting all the necessary options is tricky.
>
> This patch series aims to alleviate both:
> 1) `kunit.py run 'my-suite-*'`
> I.e. use my current kunitconfig, but just run suites that match this glob
> 2) `kunit.py run --alltests 'my-suite-*'`
> I.e. use allyesconfig so I don't have to worry about writing a
> kunitconfig at all.
>
> See the first commit message for more details and discussion about
> future work.
>
> This patch series also includes a bugfix for a latent bug that can't be
> triggered right now but has worse consequences as a result of the
> changes needed to plumb in this suite name glob.
>
> [1] https://lore.kernel.org/linux-kselftest/20210201205514.3943096-1-dlatypov@google.com/

I tried applying this series on top of several of your other patches
over the past cycle and running the kunit_tool unit tests
(kunit_tool_test.py) and I got several failures:

======================================================================
FAIL: test_exec_builddir (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tools/testing/kunit/kunit_tool_test.py", line 417, in test_exec_builddir
    self.linux_source_mock.run_kernel.assert_called_once_with(build_dir=build_dir,
timeout=300)
  File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=300)
Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_exec_passes_args_pass (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tools/testing/kunit/kunit_tool_test.py", line 337, in
test_exec_passes_args_pass
    self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='.kunit',
timeout=300)
  File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=300)
Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_exec_timeout (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tools/testing/kunit/kunit_tool_test.py", line 385, in test_exec_timeout
    self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='.kunit',
timeout=timeout)
  File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=3453)
Actual: run_kernel(timeout=3453, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_run_builddir (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tools/testing/kunit/kunit_tool_test.py", line 400, in test_run_builddir
    self.linux_source_mock.run_kernel.assert_called_once_with(
  File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=300)
Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_run_passes_args_pass (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tools/testing/kunit/kunit_tool_test.py", line 344, in
test_run_passes_args_pass
    self.linux_source_mock.run_kernel.assert_called_once_with(
  File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=300)
Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_run_timeout (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tools/testing/kunit/kunit_tool_test.py", line 392, in test_run_timeout
    self.linux_source_mock.run_kernel.assert_called_once_with(
  File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=3453)
Actual: run_kernel(timeout=3453, filter_glob='', build_dir='.kunit')

----------------------------------------------------------------------
Ran 37 tests in 0.082s

FAILED (failures=6)