Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <5a12536a-2e33-7a68-6cfd-fd991ddf875a@redhat.com>
Date:   Mon, 17 Jul 2023 19:27:13 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.12.0
Subject: Re: [PATCH v2 8/8] selftests/mm: Run all tests from run_vmtests.sh
Content-Language: en-US
To:     Ryan Roberts <ryan.roberts@arm.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Shuah Khan <shuah@kernel.org>,
        =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= <jglisse@redhat.com>,
        Mark Brown <broonie@kernel.org>,
        John Hubbard <jhubbard@nvidia.com>,
        Florent Revest <revest@chromium.org>,
        Peter Xu <peterx@redhat.com>
Cc:     linux-kernel@vger.kernel.org, linux-mm@kvack.org,
        linux-kselftest@vger.kernel.org
References: <20230717103152.202078-1-ryan.roberts@arm.com>
 <20230717103152.202078-9-ryan.roberts@arm.com>
From:   David Hildenbrand <david@redhat.com>
Organization: Red Hat
In-Reply-To: <20230717103152.202078-9-ryan.roberts@arm.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 17.07.23 12:31, Ryan Roberts wrote:
> It is very unclear to me how one is supposed to run all the mm selftests
> consistently and get clear results.
> 
> Most of the test programs are launched by both run_vmtests.sh and
> run_kselftest.sh:
> 
>    hugepage-mmap
>    hugepage-shm
>    map_hugetlb
>    hugepage-mremap
>    hugepage-vmemmap
>    hugetlb-madvise
>    map_fixed_noreplace
>    gup_test
>    gup_longterm
>    uffd-unit-tests
>    uffd-stress
>    compaction_test
>    on-fault-limit
>    map_populate
>    mlock-random-test
>    mlock2-tests
>    mrelease_test
>    mremap_test
>    thuge-gen
>    virtual_address_range
>    va_high_addr_switch
>    mremap_dontunmap
>    hmm-tests
>    madv_populate
>    memfd_secret
>    ksm_tests
>    ksm_functional_tests
>    soft-dirty
>    cow
> 
> However, of this set, when launched by run_vmtests.sh, some of the
> programs are invoked multiple times with different arguments. When
> invoked by run_kselftest.sh, they are invoked without arguments (and as
> a consequence, some fail immediately).
> 
> Some test programs are only launched by run_vmtests.sh:
> 
>    test_vmalloc.sh
> 
> And some test programs and only launched by run_kselftest.sh:
> 
>    khugepaged
>    migration
>    mkdirty
>    transhuge-stress
>    split_huge_page_test
>    mdwe_test
>    write_to_hugetlbfs
> 
> Furthermore, run_vmtests.sh is invoked by run_kselftest.sh, so in this
> case all the test programs invoked by both scripts are run twice!
> 
> Needless to say, this is a bit of a mess. In the absence of fully
> understanding the history here, it looks to me like the best solution is
> to launch ALL test programs from run_vmtests.sh, and ONLY invoke
> run_vmtests.sh from run_kselftest.sh. This way, we get full control over
> the parameters, each program is only invoked the intended number of
> times, and regardless of which script is used, the same tests get run in
> the same way.
> 
> The only drawback is that if using run_kselftest.sh, it's top-level tap
> result reporting reports only a single test and it fails if any of the
> contained tests fail. I don't see this as a big deal though since we
> still see all the nested reporting from multiple layers. The other issue
> with this is that all of run_vmtests.sh must execute within a single
> kselftest timeout period, so let's increase that to something more
> suitable.
> 
> In the Makefile, TEST_GEN_PROGS will compile and install the tests and
> will add them to the list of tests that run_kselftest.sh will run.
> TEST_GEN_FILES will compile and install the tests but will not add them
> to the test list. So let's move all the programs from TEST_GEN_PROGS to
> TEST_GEN_FILES so that they are built but not executed by
> run_kselftest.sh. Note that run_vmtests.sh is added to TEST_PROGS, which
> means it ends up in the test list. (the lack of "_GEN" means it won't be
> compiled, but simply copied).
> 
> Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>
> ---

Acked-by: David Hildenbrand <david@redhat.com>

-- 
Cheers,

David / dhildenb