Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7;
Message-ID: <24cacac8-e67e-47e6-89b4-131d64de786b@linuxfoundation.org>
Date:   Wed, 18 Oct 2023 13:06:04 -0600
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] selftests/clone3: Report descriptive test names
Content-Language: en-US
To:     Mark Brown <broonie@kernel.org>,
        Christian Brauner <brauner@kernel.org>,
        Shuah Khan <shuah@kernel.org>
Cc:     linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org,
        Shuah Khan <skhan@linuxfoundation.org>
References: <20231018-kselftest-clone3-output-v1-1-12b7c50ea2cf@kernel.org>
From:   Shuah Khan <skhan@linuxfoundation.org>
In-Reply-To: <20231018-kselftest-clone3-output-v1-1-12b7c50ea2cf@kernel.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 10/17/23 17:01, Mark Brown wrote:
> The clone3() selftests currently report test results in a format that does
> not mesh entirely well with automation. They log output for each test such
> as:
> 
>    # [1382411] Trying clone3() with flags 0 (size 0)
>    # I am the parent (1382411). My child's pid is 1382412
>    # I am the child, my PID is 1382412
>    # [1382411] clone3() with flags says: 0 expected 0
>    ok 1 [1382411] Result (0) matches expectation (0)
> 
> This is not ideal for automated parsers since the text after the "ok 1" is
> treated as the test name when comparing runs by a lot of automation (tests
> routinely get renumbered due to things like new tests being added based on
> logical groupings). The PID means that the test names will frequently vary
> and the rest of the name being a description of results means several tests
> have identical text there.
> 
> Address this by refactoring things so that we have a static descriptive
> name for each test which we use when logging passes, failures and skips
> and since we now have a stable name for the test to hand log that before
> starting the test to address the common issue reading logs where the test
> name is only printed after any diagnostics. The result is:
> 
>   # Running test 'simple clone3()'
>   # [1562777] Trying clone3() with flags 0 (size 0)
>   # I am the parent (1562777). My child's pid is 1562778
>   # I am the child, my PID is 1562778
>   # [1562777] clone3() with flags says: 0 expected 0
>   ok 1 simple clone3()
> 
> In order to handle skips a bit more neatly this is done in a moderately
> invasive fashion where we move from a sequence of function calls to having
> an array of test parameters. This hopefully also makes it a little easier
> to see what the tests are doing when looking at both the source and the
> logs.
> 


Good change. Thank you.

Applied to linux-kselftest next for Linux 6.7-rc1.
> Signed-off-by: Mark Brown <broonie@kernel.org>
> ---
>   tools/testing/selftests/clone3/clone3.c | 265 +++++++++++++++++++++++---------
>   1 file changed, 192 insertions(+), 73 deletions(-)
> 

thanks,
-- Shuah