From:   "Frank Filz" <ffilzlnx@mindspring.com>
To:     "'Jeff Layton'" <jlayton@kernel.org>, <bfields@fieldses.org>,
        <dai.ngo@oracle.com>
Cc:     <linux-nfs@vger.kernel.org>
References: <20230222134952.32851-1-jlayton@kernel.org>
In-Reply-To: <20230222134952.32851-1-jlayton@kernel.org>
Subject: RE: [pynfs RFC PATCH] nfs4.0/testserver.py: don't return an error when tests fail
Date:   Thu, 23 Feb 2023 08:22:40 -0800
Message-ID: <029901d947a3$0dd00c00$29702400$@mindspring.com>
MIME-Version: 1.0
Content-Type: text/plain;
        charset="us-ascii"
Content-Transfer-Encoding: 7bit
Thread-Index: AQInBvp4ShtorjsaZdmfQzq7ERPYu65BNzBg
Content-Language: en-us
Precedence: bulk

> From: Jeff Layton [mailto:jlayton@kernel.org]
 
> This script was originally changed in eb3ba0b60055 ("Have testserver.py
have
> non-zero exit code if any tests fail"), but the same change wasn't made to
the
> 4.1 testserver.py script.
> 
> There also wasn't much explanation for it, and it makes it difficult to
tell
> whether the test harness itself failed, or whether there was a failure in
a
> requested test.
> 
> Stop the 4.0 testserver.py from exiting with an error code when a test
fails, so
> that a successful return means only that the test harness itself worked,
not that
> every requested test passed.
> 
> Cc: Frank Filz <ffilzlnx@mindspring.com>
> Signed-off-by: Jeff Layton <jlayton@kernel.org>
> ---
>  nfs4.0/testserver.py | 2 --
>  1 file changed, 2 deletions(-)
> 
> I'm not sure about this one. I've worked around this in kdevops for now,
but it
> would really be preferable if it worked this way, imo. If this isn't
acceptable,
> maybe we can add a new option that enables this behavior?
> 
> Frank, what was the original rationale for eb3ba0b60055 ?

We needed a way for CI to easily detect failure of pynfs. I'm not sure how
helpful it is since Ganesha does fail some tests...

It might be helpful to have some helpers for CI to use, or an option that
causes pynfs to report in a way that's much easier for CI to determine if
pynfs succeeded or not.

Hmm, one thing that would help is to be able to flag a set of tests that
should not constitute a CI failure (known errors) but we want to keep
running them because of what they exercise, or to more readily detect that
they have been fixed.

> diff --git a/nfs4.0/testserver.py b/nfs4.0/testserver.py index
> f2c41568e5c7..4f4286daa657 100755
> --- a/nfs4.0/testserver.py
> +++ b/nfs4.0/testserver.py
> @@ -387,8 +387,6 @@ def main():
> 
>      if nfail < 0:
>          sys.exit(3)
> -    if nfail > 0:
> -        sys.exit(2)
> 
>  if __name__ == "__main__":
>      main()
> --
> 2.39.2