Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Received-SPF: Pass (protection.outlook.com: domain of sony.com designates
 160.33.194.228 as permitted sender) receiver=protection.outlook.com;
 client-ip=160.33.194.228; helo=usculsndmail01v.am.sony.com;
From:   <Tim.Bird@sony.com>
To:     <Tim.Bird@sony.com>, <keescook@chromium.org>, <shuah@kernel.org>
CC:     <anders.roxell@linaro.org>, <alexandre.belloni@bootlin.com>,
        <linux-kernel@vger.kernel.org>, <linux-kselftest@vger.kernel.org>
Subject: RE: [PATCH] selftests/kselftest/runner.sh: Add 45 second timeout
 per test
Thread-Topic: [PATCH] selftests/kselftest/runner.sh: Add 45 second timeout
 per test
Thread-Index: AQHVbxUXsNEjsnpJkUeVTYYfiT5hwaczb7LQgAAJVtA=
Date:   Thu, 19 Sep 2019 20:49:14 +0000
Message-ID: <ECADFF3FD767C149AD96A924E7EA6EAF977BA66E@USCULXMSG01.am.sony.com>
References: <201909191102.97FA56072@keescook>
 <ECADFF3FD767C149AD96A924E7EA6EAF977BA636@USCULXMSG01.am.sony.com>
In-Reply-To: <ECADFF3FD767C149AD96A924E7EA6EAF977BA636@USCULXMSG01.am.sony.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Sep 2019 20:49:19.6305
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 6abddc12-62e6-40b7-819b-08d73d42d7f4
X-MS-Exchange-CrossTenant-Id: 66c65d8a-9158-4521-a2d8-664963db48e4
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=66c65d8a-9158-4521-a2d8-664963db48e4;Ip=[160.33.194.228];Helo=[usculsndmail01v.am.sony.com]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR13MB3288
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> -----Original Message-----
> From Tim.Bird@sony.com
>=20
> > -----Original Message-----
> > From: Kees Cook
> >
> > Commit a745f7af3cbd ("selftests/harness: Add 30 second timeout per
> > test") solves the problem of kselftest_harness.h-using binary tests
> > possibly hanging forever. However, scripts and other binaries can still
> > hang forever. This adds a global timeout to each test script run.
> >
> > To make this configurable (e.g. as needed in the "rtc" test case),
> > include a new per-test-directory "settings" file (similar to "config")
> > that can contain kselftest-specific settings. The first recognized fiel=
d
> > is "timeout".
>=20
> OK - this is quite interesting.  I have had on my to-do list an action
> item to propose the creation of a file (or a standard kerneldoc string)
> to hold CI-related meta-data (of which timeout is one example).
>=20
> What other meta-data did you have in mind?
>=20
> I would like (that Fuego, and probably other CI systems would like) to ha=
ve
>  access to data like test dependencies, descriptions, and results
> interpretation
> that would be beneficial for both CI systems (using them to control test
> invocations and scheduling), as
> well as users who are trying to interpret and handle the test results.
> So this concept is a very welcome addition to kselftest.
>=20
> LTP is in the process of adopting a new system for expressing and handlin=
g
> their test meta-data.
> See the discussion at:
> https://lists.yoctoproject.org/pipermail/automated-testing/2019-
> August/000471.html
> and the prototype implementation at:
> https://github.com/metan-ucw/ltp/tree/master/docparse
>=20
> I realize that that system is coupled pretty tightly to LTP, but conceptu=
ally
> some of the same type of information would be valuable for kselftest test=
s.
> One example of a specific field that would be handy is 'need_root'.
>=20
> It would be nice to avoid proliferation of such meta-data schemas (that i=
s
> field names), so maybe we can have a discussion about this before adoptin=
g
> something?
>=20
> Just FYI, I'm OK with the name 'timeout'.  I think that's pretty much
> universally
> used by all CI runners I'm aware of to indicate the test timeout value.  =
But
> before adopting other fields it would be good to start comparing notes
> and not invent a bunch of new field names for concepts that are already i=
n
> other systems.
>=20
> >
> > Additionally, this splits the reporting for timeouts into a specific
> > "TIMEOUT" not-ok (and adds exit code reporting in the remaining case).
> >
> > Signed-off-by: Kees Cook <keescook@chromium.org>
> > ---
> >  tools/testing/selftests/kselftest/runner.sh | 36 +++++++++++++++++++--
> >  tools/testing/selftests/rtc/settings        |  1 +
> >  2 files changed, 34 insertions(+), 3 deletions(-)
> >  create mode 100644 tools/testing/selftests/rtc/settings
> >
> > diff --git a/tools/testing/selftests/kselftest/runner.sh
> > b/tools/testing/selftests/kselftest/runner.sh
> > index 00c9020bdda8..84de7bc74f2c 100644
> > --- a/tools/testing/selftests/kselftest/runner.sh
> > +++ b/tools/testing/selftests/kselftest/runner.sh
> > @@ -3,9 +3,14 @@
> >  #
> >  # Runs a set of tests in a given subdirectory.
> >  export skip_rc=3D4
> > +export timeout_rc=3D124
> what are the units here?  I presume seconds?

Nevermind.  I misread this.  This is the return code from the 'timeout' pro=
gram, right?
 -- Tim