Date:   Mon, 27 Feb 2023 14:42:15 -0800
From:   Luis Chamberlain <mcgrof@kernel.org>
To:     Shuah Khan <skhan@linuxfoundation.org>
Cc:     shuah@kernel.org, linux-kselftest@vger.kernel.org,
        gregkh@linuxfoundation.org, tiwai@suse.de, tianfei.zhang@intel.com,
        russell.h.weight@intel.com, keescook@chromium.org,
        tweek@google.com, a.manzanares@samsung.com, dave@stgolabs.net,
        linux-modules@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH 1/2] selftests/kmod: increase the kmod timeout from 45 to
 165
Message-ID: <Y/0xx0cedxlRMKpH@bombadil.infradead.org>
References: <20230206234344.2433950-1-mcgrof@kernel.org>
 <20230206234344.2433950-2-mcgrof@kernel.org>
 <b094dc23-a96d-93c4-a350-8fb92476f431@linuxfoundation.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <b094dc23-a96d-93c4-a350-8fb92476f431@linuxfoundation.org>
Sender: Luis Chamberlain <mcgrof@infradead.org>
Precedence: bulk

On Mon, Feb 27, 2023 at 03:32:50PM -0700, Shuah Khan wrote:
> On 2/6/23 16:43, Luis Chamberlain wrote:
> > The default sefltests timeout is 45 seconds. If you run the kmod
> > selftests on your own with say:
> > 
> > ./tools/testings/selftests/kmod.sh
> > 
> > Then the default timeout won't be in effect.
> > 
> > I've never ran kmod selftests using the generic make wrapper
> > (./tools/testing/selftests/run_kselftest.sh -s) util now
> > that I have support for it on kdevops [0]. And with that the
> > test is limitted to the default timeout which we quickly run
> > into. Bump this up to what I see is required on 8GiB / 8 vcpu
> > libvirt q35 guest as can be easily created now with kdevops.
> > 
> > To run selftests with kdevops:
> > 
> > make menuconfig # enable dedicated selftests and kmod test
> > make
> > make bringup
> > make linux
> > make selftests-kmod
> > 
> > This ends up taking about 280 seconds now, give or take add
> > 50 seconds more more and we end up with 350. Document the
> > rationale.
> > 
> > [0] https://github.com/linux-kdevops/kdevops
> > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
> > ---
> >   tools/testing/selftests/kmod/settings | 4 ++++
> >   1 file changed, 4 insertions(+)
> >   create mode 100644 tools/testing/selftests/kmod/settings
> > 
> > diff --git a/tools/testing/selftests/kmod/settings b/tools/testing/selftests/kmod/settings
> > new file mode 100644
> > index 000000000000..6fca0f1a4594
> > --- /dev/null
> > +++ b/tools/testing/selftests/kmod/settings
> > @@ -0,0 +1,4 @@
> > +# measured from a manual run:
> > +# time ./tools/testing/selftests/kmod/kmod.sh
> > +# Then add ~50 seconds more gracetime.
> > +timeout=350
> 
> Adding timeouts like this for individual tests increases the overall kselftest
> run-time. I am not in favor of adding timeouts.
> 
> We have to find a better way to do this.

Well if folks don't have this the test will fail, and so a false
positive. If the goal is to have a low time timeout for "do not run
tests past this time and do not fail if we stopped the test" then
that seems to be likely one way to go and each test may need to be
modified to not fail fatally in case of a special signal.

  Luis