Received-SPF: pass (google.com: best guess record for domain of linux-ext4-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Received-SPF: Pass (protection.outlook.com: domain of sony.com designates
 160.33.194.231 as permitted sender) receiver=protection.outlook.com;
 client-ip=160.33.194.231; helo=usculsndmail04v.am.sony.com;
From:   <Tim.Bird@sony.com>
To:     <tytso@mit.edu>, <skhan@linuxfoundation.org>
CC:     <brendanhiggins@google.com>, <yzaikin@google.com>,
        <linux-kselftest@vger.kernel.org>, <linux-ext4@vger.kernel.org>,
        <adilger.kernel@dilger.ca>, <kunit-dev@googlegroups.com>
Subject: RE: [PATCH linux-kselftest/test v2] ext4: add kunit test for
 decoding extended timestamps
Thread-Topic: [PATCH linux-kselftest/test v2] ext4: add kunit test for
 decoding extended timestamps
Thread-Index: AQHVfxQJt5KYyUfdIUCqjuPcGwlYBqdUYHYAgABUYYCAAMbxgIAANgMAgAhyYYCAABMAgIAA1OuAgABbSgA=
Date:   Thu, 17 Oct 2019 22:25:35 +0000
Message-ID: <ECADFF3FD767C149AD96A924E7EA6EAF977D0023@USCULXMSG01.am.sony.com>
References: <20191010023931.230475-1-yzaikin@google.com>
 <2f2ea7b0-f683-1cdd-f3f2-ecdf44cb4a97@linuxfoundation.org>
 <CAAXuY3qtSHENgy3S168_03ju_JwAucOAt5WEJGQ+pi5PfurP6g@mail.gmail.com>
 <CAFd5g46RcFV0FACuoF=jCSLzf7UFmEYn4gddaijUZ+zR_CFZBQ@mail.gmail.com>
 <20191011131902.GC16225@mit.edu>
 <CAFd5g45s1-=Z4JwJn4A1VDGu4oEGBisQ_0RFp4otUU3rKf1XpQ@mail.gmail.com>
 <1e6611e6-2fa6-6f7d-bc7f-0bc2243d9342@linuxfoundation.org>
 <20191017120833.GA25548@mit.edu>
In-Reply-To: <20191017120833.GA25548@mit.edu>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2019 22:25:40.2046
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 66fdbcb1-13e5-4eee-1812-08d75350f105
X-MS-Exchange-CrossTenant-Id: 66c65d8a-9158-4521-a2d8-664963db48e4
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=66c65d8a-9158-4521-a2d8-664963db48e4;Ip=[160.33.194.231];Helo=[usculsndmail04v.am.sony.com]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR13MB1816
Sender: linux-ext4-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org


> -----Original Message-----
> From: Theodore Y. Ts'o on October 17, 2019 2:09 AM
>=20
> On Wed, Oct 16, 2019 at 05:26:29PM -0600, Shuah Khan wrote:
> >
> > I don't really buy the argument that unit tests should be deterministic
> > Possibly, but I would opt for having the ability to feed test data.
>=20
> I strongly believe that unit tests should be deterministic.
> Non-deterministic tests are essentially fuzz tests.  And fuzz tests
> should be different from unit tests.

I'm not sure I have the entire context here, but I think deterministic
might not be the right word, or it might not capture the exact meaning
intended.

I think there are multiple issues here:
 1. Does the test enclose all its data, including working data and expected=
 results?
Or, does the test allow someone to provide working data?  This alternative
implies that either the some of testcases or the results might be different=
 depending on
the data that is provided.  IMHO the test would be deterministic if it alwa=
ys produced
the same results based on the same data inputs.  And if the input data was =
deterministic.
I would call this a data-driven test.

Since the results would be dependent on the data provided, the results
from tests using different data would not be comparable.  Essentially,
changing the input data changes the test so maybe it's best to consider
this a different test.  Like 'test-with-data-A' and 'test-with-data-B'.

2. Does the test automatically detect some attribute of the system, and adj=
ust
its operation based on that (does the test probe?)  This is actually quite =
common
if you include things like when a test requires root access to run.  Someti=
mes such tests,
when run without root privilege, run as many testcases as possible not as r=
oot, and skip
the testcases that require root.

In general, altering the test based on probed data is a form of data-driven=
 test,
except the data is not provided by the user.  Whether
this is deterministic in the sense of (1) depends on whether the data that
is probed is deterministic.  In the case or requiring root, then it should
not change from run to run (and it should probably be reflected in the char=
acterization
of the results).

Maybe neither of the above cases fall in the category of unit tests, but
they are not necessarily fuzzing tests.  IMHO a fuzzing test is one which r=
andomizes
the data for a data-driven test (hence using non-deterministic data).  Once=
 the fuzzer
has found a bug, and the data and code for a test is fixed into a reproduce=
r program,
then at that point it should be deterministic (modulo what I say about race=
 condition
tests below).

>=20
> We want unit tests to run quickly.  Fuzz tests need to be run for a
> large number of passes (perhaps hours) in order to be sure that we've
> hit any possible bad cases.  We want to be able to easily bisect fuzz
> tests --- preferably, automatically.  And any kind of flakey test is
> hell to bisect.
Agreed.
=20
> It's bad enough when a test is flakey because of the underlying code.
> But when a test is flakey because the test inputs are
> non-deterministic, it's even worse.
I very much agree on this as well.

I'm not sure how one classes a program that seeks to invoke a race conditio=
n.
This can take variable time, so in that sense it is not deterministic.   Bu=
t it should
produce the same result if the probabilities required for the race conditio=
n
to be hit are fulfilled.  Probably (see what I did there :-), one needs to =
take
a probabilistic approach to reproducing and bisecting such bugs.  The durat=
ion
or iterations required to reproduce the bug (to some confidence level) may
need to be included with the reproducer program.  I'm not sure if the syska=
ller
reproducers do this or not, or if they just run forever.  One I looked at r=
an forever.
But you would want to limit this in order to produce results with some conf=
idence
level (and not waste testing resources).

---
The reason I want get clarity on the issue of data-driven tests is that I t=
hink
data-driven tests and tests that probe are very much desirable.  This allow=
s a
test to be able to be more generalized and allows for specialization of the
test for more scenarios without re-coding it.
I'm not sure if this still qualifies as unit testing, but it's very useful =
as a means to
extend the value of a test.  We haven't trod into the mocking parts of kuni=
t,
but I'm hoping that it may be possible to have that be data-driven (dependi=
ng on
what's being mocked), to make it easier to test more things using the same =
code.

Finally, I think the issue of testing speed is orthogonal to whether a test=
 is self-enclosed
or data-driven.  Definitely fuzzers, which are experimenting with system in=
teraction
in a non-deterministic way, have speed problems.

Just my 2 cents.
 -- Tim
=20