From:   "Frank Filz" <ffilzlnx@mindspring.com>
To:     "'J. Bruce Fields'" <bfields@fieldses.org>,
        "'Jeff Layton'" <jlayton@kernel.org>
Cc:     <dai.ngo@oracle.com>, <linux-nfs@vger.kernel.org>
References: <20230222182043.155712-1-jlayton@kernel.org> <20230223151959.GC10456@fieldses.org>
In-Reply-To: <20230223151959.GC10456@fieldses.org>
Subject: RE: [pynfs RFC PATCH] testserver.py: special-case the "all" flag
Date:   Thu, 23 Feb 2023 08:20:40 -0800
Message-ID: <029801d947a2$c64a1c40$52de54c0$@mindspring.com>
MIME-Version: 1.0
Content-Type: text/plain;
        charset="us-ascii"
Content-Transfer-Encoding: 7bit
Thread-Index: AQFaakfSQa9czN/urkZcULRlvLq/BAI3dwcGr8ixkDA=
Content-Language: en-us
Precedence: bulk

> -----Original Message-----
> From: J. Bruce Fields [mailto:bfields@fieldses.org]
 
> On Wed, Feb 22, 2023 at 01:20:43PM -0500, Jeff Layton wrote:
> > The READMEs for v4.0 and v4.1 are inconsistent here. For v4.0, the "all"
> > flag is supposed to run all of the "standard" tests. For v4.1 "all" is
> > documented to run all of the tests, but it actually doesn't since not
> > every tests has "all" in its FLAGS: field.
> >
> > I move that we change this. If I say that I want to run "all", then I
> > really do want to run _all_ of the tests. Ensure that every test has
> > the "all" flag set.
> 
> In some (all?) cases where the "all" flag was left off, it was
intentional.
> 
> We try not to flag spec-compliant servers as failing, because people are
> sometimes a little careless about "fixing" failures that in their
particular case
> really shouldn't be fixed.  But sometimes it's still useful to have a test
that goes
> somewhat beyond the spec.
> 
> There might be other ways to handle that kind of test, but it would need
some
> more thought.

There are also tests that are server specific, for example, there are tests
that are Ganesha specific. For example, look at LOCK11g. LOCK11 attempts to
make a bad stateid, and it sort of "knows" how the server formats its
stateids. There is a function makeBadIDganesha that "knows" how Ganesha
forms its stateids and forms a bad one that will definitely result in an
NSF4ERR_BAD_STATEID.

There are also a number of tests that require specific setup bits to work
that someone who just fires up a server won't have set up. I think there are
some that expect specific exports, or second exports or other specific
aspects of the pseudofs. There are tests that deal with server reboots.

At the least, I want a way to specify to run all the tests that are expected
to pass on a server that has been set up without any special cases, and that
are expected to pass on Linux.

The Ganesha tag, in addition to specifying a few Ganesha only tests,
specifies several tests that may not pass on Linux in general (so they
aren't in all) but do pass on Ganesha.

On the flip side, Ganesha lives with a handful of known failures...

Frank

> > Signed-off-by: Jeff Layton <jlayton@kernel.org>
> > ---
> >  nfs4.1/testmod.py | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > If this is unacceptable, then an alternative could be to add a new
> > (similarly special-cased) "everything" flag.
> >
> > diff --git a/nfs4.1/testmod.py b/nfs4.1/testmod.py index
> > 11e759d673fd..7b3bac543084 100644
> > --- a/nfs4.1/testmod.py
> > +++ b/nfs4.1/testmod.py
> > @@ -386,6 +386,8 @@ def createtests(testdir):
> >      for t in tests:
> >  ##         if not t.flags_list:
> >  ##             raise RuntimeError("%s has no flags" % t.fullname)
> > +        if "all" not in t.flags_list:
> > +            t.flags_list.append("all")
> >          for f in t.flags_list:
> >              if f not in flag_dict:
> >                  flag_dict[f] = bit
> > --
> > 2.39.2