Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Subject: Re: [PATCH v3] selftests/vm: enable running select groups of tests
To:     Joel Savitz <jsavitz@redhat.com>, Nico Pache <npache@redhat.com>
Cc:     Andrew Morton <akpm@linux-foundation.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Shuah Khan <shuah@kernel.org>, Linux MM <linux-mm@kvack.org>,
        linux-kselftest@vger.kernel.org,
        Shuah Khan <skhan@linuxfoundation.org>
References: <20220706205451.4133254-1-jsavitz@redhat.com>
 <CAA1CXcCHp+zt=34wyFrif+7QtAs9TBOLaL3Nqitz8f_tnR7u9g@mail.gmail.com>
 <CAL1p7m5pLQcvkGajOL8CQiunvye4-8QrDS8oWMShOHmOUFsB-w@mail.gmail.com>
From:   Shuah Khan <skhan@linuxfoundation.org>
Message-ID: <0469b7fb-a71f-854a-2a9f-3da8c6df27c4@linuxfoundation.org>
Date:   Tue, 26 Jul 2022 14:59:25 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.8.1
MIME-Version: 1.0
In-Reply-To: <CAL1p7m5pLQcvkGajOL8CQiunvye4-8QrDS8oWMShOHmOUFsB-w@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 7/19/22 4:27 PM, Joel Savitz wrote:
> Hello,
> 
> Sorry for the late reply.
> 
> Setting TEST_ITEMS="xx yy zz" and invoking the script has the same
> effect as passing -t "xx yy zz" to the script. VM_TEST_ITEMS may be a
> better env variable name however.
> 
> Best,
> Joel Savitz
> 

When you are respoding on kernel mailing lists, please bottom post.
Refer to the kernel mailing list communication guidelines.

> On Fri, Jul 15, 2022 at 3:55 PM Nico Pache <npache@redhat.com> wrote:
>>
>> Hi Andrew and Joel,
>>
>> I noticed a flaw in switching to the cmdline argument over the
>> environment variable. If you are utilizing the exerciser script (in
>> tools/testing/selftests/run_kselftest.sh), this is not designed to
>> pass cmdline variables to the individual TEST_PROGS. On the other hand
>> if we utilize Env. variables we can still specify which tests we'd
>> like to run. Our infrastructure is designed to utilize this wrapper,
>> so we inevitably will not be able to use this the way we thought.
>>
>> Would you be ok with switching back to the initial approach? If so I
>> believe making the name more specific would be ideal, ie)
>> VM_TEST_ITEMS.
>>
>> Cheers,
>> -- Nico
>>
>> On Wed, Jul 6, 2022 at 4:55 PM Joel Savitz <jsavitz@redhat.com> wrote:
>>>
>>> Our memory management kernel CI testing at Red Hat uses the VM
>>> selftests and we have run into two problems:
>>>
>>> First, our LTP tests overlap with the VM selftests.
>>>
>>> We want to avoid unhelpful redundancy in our testing practices.
>>>
>>> Second, we have observed the current run_vmtests.sh to report overall
>>> failure/ambiguous results in the case that a machine lacks the necessary
>>> hardware to perform one or more of the tests. E.g. ksm tests that
>>> require more than one numa node.
>>>
>>> We want to be able to run the vm selftests suitable to particular hardware.
>>>
>>> Add the ability to run one or more groups of vm tests via run_vmtests.sh
>>> instead of simply all-or-none in order to solve these problems.
>>>
>>> Preserve existing default behavior of running all tests when the script
>>> is invoked with no arguments.
>>>
>>> Documentation of test groups is included in the patch as follows:
>>>
>>>      # ./run_vmtests.sh [ -h || --help ]
>>>
>>>      usage: ./tools/testing/selftests/vm/run_vmtests.sh [ -h | -t "<categories>"]
>>>        -t: specify specific categories to tests to run
>>>        -h: display this message
>>>
>>>      The default behavior is to run all tests.
>>>
>>>      Alternatively, specific groups tests can be run by passing a string
>>>      to the -t argument containing one or more of the following categories
>>>      separated by spaces:
>>>      - mmap
>>>              tests for mmap(2)
>>>      - gup_test
>>>              tests for gup using gup_test interface
>>>      - userfaultfd
>>>              tests for  userfaultfd(2)
>>>      - compaction
>>>              a test for the patch "Allow compaction of unevictable pages"
>>>      - mlock
>>>              tests for mlock(2)
>>>      - mremap
>>>              tests for mremap(2)
>>>      - hugevm
>>>              tests for very large virtual address space
>>>      - vmalloc
>>>              vmalloc smoke tests
>>>      - hmm
>>>              hmm smoke tests
>>>      - madv_populate
>>>              test memadvise(2) MADV_POPULATE_{READ,WRITE} options
>>>      - memfd_secret
>>>              test memfd_secret(2)
>>>      - process_mrelease
>>>              test process_mrelease(2)
>>>      - ksm
>>>              ksm tests that do not require >=2 NUMA nodes
>>>      - ksm_numa
>>>              ksm tests that require >=2 NUMA nodes
>>>      - pkey
>>>              memory protection key tests
>>>      example: ./run_vmtests.sh -t "hmm mmap ksm"
>>>
>>> Changes from v2:
>>>          - rebase onto the mm-everyting branch in
>>>          https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git
>>>          - integrate this functionality with new the tests
>>>
>>> Changes from v1:
>>>          - use a command line argument to pass the test categories to the
>>>            script instead of an environmet variable
>>>          - remove novel prints to avoid messing with extant parsers of this
>>>            script
>>>          - update the usage text
>>>
>>> Signed-off-by: Joel Savitz <jsavitz@redhat.com>
>>> ---
>>>   tools/testing/selftests/vm/run_vmtests.sh | 241 +++++++++++++++-------
>>>   1 file changed, 161 insertions(+), 80 deletions(-)
>>>
>>> diff --git a/tools/testing/selftests/vm/run_vmtests.sh b/tools/testing/selftests/vm/run_vmtests.sh
>>> index d84fe0fa15e1..fb72a1338d07 100755
>>> --- a/tools/testing/selftests/vm/run_vmtests.sh
>>> +++ b/tools/testing/selftests/vm/run_vmtests.sh
>>> @@ -1,6 +1,6 @@
>>>   #!/bin/bash
>>>   # SPDX-License-Identifier: GPL-2.0
>>> -#please run as root
>>> +# Please run as root
>>>
>>>   # Kselftest framework requirement - SKIP code is 4.
>>>   ksft_skip=4
>>> @@ -8,15 +8,75 @@ ksft_skip=4
>>>   mnt=./huge
>>>   exitcode=0
>>>
>>> -#get huge pagesize and freepages from /proc/meminfo
>>> -while read -r name size unit; do
>>> -       if [ "$name" = "HugePages_Free:" ]; then
>>> -               freepgs="$size"
>>> -       fi
>>> -       if [ "$name" = "Hugepagesize:" ]; then
>>> -               hpgsize_KB="$size"
>>> +usage() {
>>> +       cat <<EOF
>>> +usage: ${BASH_SOURCE[0]:-$0} [ -h | -t "<categories>"]
>>> +  -t: specify specific categories to tests to run
>>> +  -h: display this message
>>> +
>>> +The default behavior is to run all tests.
>>> +
>>> +Alternatively, specific groups tests can be run by passing a string
>>> +to the -t argument containing one or more of the following categories
>>> +separated by spaces:
>>> +- mmap
>>> +       tests for mmap(2)
>>> +- gup_test
>>> +       tests for gup using gup_test interface
>>> +- userfaultfd
>>> +       tests for  userfaultfd(2)
>>> +- compaction
>>> +       a test for the patch "Allow compaction of unevictable pages"
>>> +- mlock
>>> +       tests for mlock(2)
>>> +- mremap
>>> +       tests for mremap(2)
>>> +- hugevm
>>> +       tests for very large virtual address space
>>> +- vmalloc
>>> +       vmalloc smoke tests
>>> +- hmm
>>> +       hmm smoke tests
>>> +- madv_populate
>>> +       test memadvise(2) MADV_POPULATE_{READ,WRITE} options
>>> +- memfd_secret
>>> +       test memfd_secret(2)
>>> +- process_mrelease
>>> +       test process_mrelease(2)
>>> +- ksm
>>> +       ksm tests that do not require >=2 NUMA nodes
>>> +- ksm_numa
>>> +       ksm tests that require >=2 NUMA nodes
>>> +- pkey
>>> +       memory protection key tests
>>> +example: ./run_vmtests.sh -t "hmm mmap ksm"
>>> +EOF
>>> +       exit 0
>>> +}
>>> +
>>> +
>>> +while getopts "ht:" OPT; do
>>> +       case ${OPT} in
>>> +               "h") usage ;;
>>> +               "t") TEST_ITEMS=${OPTARG} ;;
>>> +       esac
>>> +done
>>> +shift $((OPTIND -1))
>>> +
>>> +# default behavior: run all tests
>>> +TEST_ITEMS=${TEST_ITEMS:-default}
>>> +
>>> +test_selected() {
>>> +       if [ "$TEST_ITEMS" == "default" ]; then
>>> +               # If no TEST_ITEMS are specified, run all tests
>>> +               return 0
>>>          fi
>>> -done < /proc/meminfo
>>> +       echo ${TEST_ITEMS} | grep ${1} 2>&1 >/dev/null
>>> +       return ${?}
>>> +}
>>> +
>>> +# Hugepage setup only needed for hugetlb tests
>>> +if test_selected "hugetlb"; then
>>>
>>>   # Simple hugetlbfs tests have a hardcoded minimum requirement of
>>>   # huge pages totaling 256MB (262144KB) in size.  The userfaultfd
>>> @@ -28,7 +88,17 @@ hpgsize_MB=$((hpgsize_KB / 1024))
>>>   half_ufd_size_MB=$((((nr_cpus * hpgsize_MB + 127) / 128) * 128))
>>>   needmem_KB=$((half_ufd_size_MB * 2 * 1024))
>>>
>>> -#set proper nr_hugepages
>>> +# get huge pagesize and freepages from /proc/meminfo
>>> +while read -r name size unit; do
>>> +       if [ "$name" = "HugePages_Free:" ]; then
>>> +               freepgs="$size"
>>> +       fi
>>> +       if [ "$name" = "Hugepagesize:" ]; then
>>> +               hpgsize_KB="$size"
>>> +       fi
>>> +done < /proc/meminfo
>>> +
>>> +# set proper nr_hugepages
>>>   if [ -n "$freepgs" ] && [ -n "$hpgsize_KB" ]; then
>>>          nr_hugepgs=$(cat /proc/sys/vm/nr_hugepages)
>>>          needpgs=$((needmem_KB / hpgsize_KB))
>>> @@ -57,144 +127,155 @@ else
>>>          exit 1
>>>   fi
>>>
>>> -#filter 64bit architectures
>>> +fi # test_selected "hugetlb"
>>> +
>>> +# filter 64bit architectures
>>>   ARCH64STR="arm64 ia64 mips64 parisc64 ppc64 ppc64le riscv64 s390x sh64 sparc64 x86_64"
>>>   if [ -z "$ARCH" ]; then
>>>          ARCH=$(uname -m 2>/dev/null | sed -e 's/aarch64.*/arm64/')
>>>   fi
>>>   VADDR64=0
>>> -echo "$ARCH64STR" | grep "$ARCH" && VADDR64=1
>>> +echo "$ARCH64STR" | grep "$ARCH" &>/dev/null && VADDR64=1
>>>
>>>   # Usage: run_test [test binary] [arbitrary test arguments...]
>>>   run_test() {
>>> -       local title="running $*"
>>> -       local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -)
>>> -       printf "%s\n%s\n%s\n" "$sep" "$title" "$sep"
>>> -
>>> -       "$@"
>>> -       local ret=$?
>>> -       if [ $ret -eq 0 ]; then
>>> -               echo "[PASS]"
>>> -       elif [ $ret -eq $ksft_skip ]; then
>>> -               echo "[SKIP]"
>>> -               exitcode=$ksft_skip
>>> -       else
>>> -               echo "[FAIL]"
>>> -               exitcode=1
>>> -       fi
>>> +       if test_selected ${CATEGORY}; then
>>> +               local title="running $*"
>>> +               local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -)
>>> +               printf "%s\n%s\n%s\n" "$sep" "$title" "$sep"
>>> +
>>> +               "$@"
>>> +               local ret=$?
>>> +               if [ $ret -eq 0 ]; then
>>> +                       echo "[PASS]"
>>> +               elif [ $ret -eq $ksft_skip ]; then
>>> +                       echo "[SKIP]"
>>> +                       exitcode=$ksft_skip
>>> +               else
>>> +                       echo "[FAIL]"
>>> +                       exitcode=1
>>> +               fi
>>> +       fi # test_selected
>>>   }
>>>
>>> -mkdir "$mnt"
>>> -mount -t hugetlbfs none "$mnt"
>>> +# setup only needed for hugetlb tests
>>> +if test_selected "hugetlb"; then
>>> +       mkdir "$mnt"
>>> +       mount -t hugetlbfs none "$mnt"
>>> +fi
>>>
>>> -run_test ./hugepage-mmap
>>> +CATEGORY="hugetlb" run_test ./hugepage-mmap
>>>
>>>   shmmax=$(cat /proc/sys/kernel/shmmax)
>>>   shmall=$(cat /proc/sys/kernel/shmall)
>>>   echo 268435456 > /proc/sys/kernel/shmmax
>>>   echo 4194304 > /proc/sys/kernel/shmall
>>> -run_test ./hugepage-shm
>>> +CATEGORY="hugetlb" run_test ./hugepage-shm
>>>   echo "$shmmax" > /proc/sys/kernel/shmmax
>>>   echo "$shmall" > /proc/sys/kernel/shmall
>>>
>>> -run_test ./map_hugetlb
>>> +CATEGORY="hugetlb" run_test ./map_hugetlb
>>>
>>> -run_test ./hugepage-mremap "$mnt"/huge_mremap
>>> -rm -f "$mnt"/huge_mremap
>>> +CATEGORY="hugetlb" run_test ./hugepage-mremap "$mnt"/huge_mremap
>>> +test_selected "hugetlb" && rm -f "$mnt"/huge_mremap
>>>
>>> -run_test ./hugepage-vmemmap
>>> +CATEGORY="hugetlb" run_test ./hugepage-vmemmap
>>>
>>> -run_test ./hugetlb-madvise "$mnt"/madvise-test
>>> -rm -f "$mnt"/madvise-test
>>> +CATEGORY="hugetlb" run_test ./hugetlb-madvise "$mnt"/madvise-test
>>> +test_selected "hugetlb" && rm -f "$mnt"/madvise-test
>>>
>>> -echo "NOTE: The above hugetlb tests provide minimal coverage.  Use"
>>> -echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for"
>>> -echo "      hugetlb regression testing."
>>> +if test_selected "hugetlb"; then
>>> +       echo "NOTE: These hugetlb tests provide minimal coverage.  Use"
>>> +       echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for"
>>> +       echo "      hugetlb regression testing."
>>> +fi
>>>
>>> -run_test ./map_fixed_noreplace
>>> +CATEGORY="mmap" run_test ./map_fixed_noreplace
>>>
>>>   # get_user_pages_fast() benchmark
>>> -run_test ./gup_test -u
>>> +CATEGORY="gup_test" run_test ./gup_test -u
>>>   # pin_user_pages_fast() benchmark
>>> -run_test ./gup_test -a
>>> +CATEGORY="gup_test" run_test ./gup_test -a
>>>   # Dump pages 0, 19, and 4096, using pin_user_pages:
>>> -run_test ./gup_test -ct -F 0x1 0 19 0x1000
>>> +CATEGORY="gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000
>>>
>>> -run_test ./userfaultfd anon 20 16
>>> -run_test ./userfaultfd anon:dev 20 16
>>> +CATEGORY="userfaultfd" run_test ./userfaultfd anon 20 16
>>> +CATEGORY="userfaultfd" run_test ./userfaultfd anon:dev 20 16
>>>   # Hugetlb tests require source and destination huge pages. Pass in half the
>>>   # size ($half_ufd_size_MB), which is used for *each*.
>>> -run_test ./userfaultfd hugetlb "$half_ufd_size_MB" 32
>>> -run_test ./userfaultfd hugetlb:dev "$half_ufd_size_MB" 32
>>> -run_test ./userfaultfd hugetlb_shared "$half_ufd_size_MB" 32 "$mnt"/uffd-test
>>> +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb "$half_ufd_size_MB" 32
>>> +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb:dev "$half_ufd_size_MB" 32
>>> +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb_shared "$half_ufd_size_MB" 32 "$mnt"/uffd-test
>>>   rm -f "$mnt"/uffd-test
>>> -run_test ./userfaultfd hugetlb_shared:dev "$half_ufd_size_MB" 32 "$mnt"/uffd-test
>>> +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb_shared:dev "$half_ufd_size_MB" 32 "$mnt"/uffd-test
>>>   rm -f "$mnt"/uffd-test
>>> -run_test ./userfaultfd shmem 20 16
>>> -run_test ./userfaultfd shmem:dev 20 16
>>> -
>>> -#cleanup
>>> -umount "$mnt"
>>> -rm -rf "$mnt"
>>> -echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages
>>> +CATEGORY="userfaultfd" run_test ./userfaultfd shmem 20 16
>>> +CATEGORY="userfaultfd" run_test ./userfaultfd shmem:dev 20 16
>>> +
>>> +# cleanup (only needed when running hugetlb tests)
>>> +if test_selected "hugetlb"; then
>>> +       umount "$mnt"
>>> +       rm -rf "$mnt"
>>> +       echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages
>>> +fi
>>>
>>> -run_test ./compaction_test
>>> +CATEGORY="compaction" run_test ./compaction_test
>>>
>>> -run_test sudo -u nobody ./on-fault-limit
>>> +CATEGORY="mlock" run_test sudo -u nobody ./on-fault-limit
>>>
>>> -run_test ./map_populate
>>> +CATEGORY="mmap" run_test ./map_populate
>>>
>>> -run_test ./mlock-random-test
>>> +CATEGORY="mlock" run_test ./mlock-random-test
>>>
>>> -run_test ./mlock2-tests
>>> +CATEGORY="mlock" run_test ./mlock2-tests
>>>
>>> -run_test ./mrelease_test
>>> +CATEGORY="process_mrelease" run_test ./mrelease_test
>>>
>>> -run_test ./mremap_test
>>> +CATEGORY="mremap" run_test ./mremap_test
>>>
>>> -run_test ./thuge-gen
>>> +CATEGORY="hugetlb" run_test ./thuge-gen
>>>
>>>   if [ $VADDR64 -ne 0 ]; then
>>> -       run_test ./virtual_address_range
>>> +       CATEGORY="hugevm" run_test ./virtual_address_range
>>>
>>>          # virtual address 128TB switch test
>>> -       run_test ./va_128TBswitch.sh
>>> +       CATEGORY="hugevm" run_test ./va_128TBswitch.sh
>>>   fi # VADDR64
>>>
>>>   # vmalloc stability smoke test
>>> -run_test ./test_vmalloc.sh smoke
>>> +CATEGORY="vmalloc" run_test ./test_vmalloc.sh smoke
>>>
>>> -run_test ./mremap_dontunmap
>>> +CATEGORY="mremap" run_test ./mremap_dontunmap
>>>
>>> -run_test ./test_hmm.sh smoke
>>> +CATEGORY="hmm" run_test ./test_hmm.sh smoke
>>>
>>>   # MADV_POPULATE_READ and MADV_POPULATE_WRITE tests
>>> -run_test ./madv_populate
>>> +CATEGORY="madv_populate" run_test ./madv_populate
>>>
>>> -run_test ./memfd_secret
>>> +CATEGORY="memfd_secret" run_test ./memfd_secret
>>>
>>>   # KSM MADV_MERGEABLE test with 10 identical pages
>>> -run_test ./ksm_tests -M -p 10
>>> +CATEGORY="ksm" run_test ./ksm_tests -M -p 10
>>>   # KSM unmerge test
>>> -run_test ./ksm_tests -U
>>> +CATEGORY="ksm" run_test ./ksm_tests -U
>>>   # KSM test with 10 zero pages and use_zero_pages = 0
>>> -run_test ./ksm_tests -Z -p 10 -z 0
>>> +CATEGORY="ksm" run_test ./ksm_tests -Z -p 10 -z 0
>>>   # KSM test with 10 zero pages and use_zero_pages = 1
>>> -run_test ./ksm_tests -Z -p 10 -z 1
>>> +CATEGORY="ksm" run_test ./ksm_tests -Z -p 10 -z 1
>>>   # KSM test with 2 NUMA nodes and merge_across_nodes = 1
>>> -run_test ./ksm_tests -N -m 1
>>> +CATEGORY="ksm_numa" run_test ./ksm_tests -N -m 1
>>>   # KSM test with 2 NUMA nodes and merge_across_nodes = 0
>>> -run_test ./ksm_tests -N -m 0
>>> +CATEGORY="ksm_numa" run_test ./ksm_tests -N -m 0
>>>
>>>   # protection_keys tests
>>>   if [ -x ./protection_keys_32 ]
>>>   then
>>> -       run_test ./protection_keys_32
>>> +       CATEGORY="pkey" run_test ./protection_keys_32
>>>   fi
>>>
>>>   if [ -x ./protection_keys_64 ]
>>>   then
>>> -       run_test ./protection_keys_64
>>> +       CATEGORY="pkey" run_test ./protection_keys_64
>>>   fi
>>>
>>>   exit $exitcode
>>> --
>>> 2.31.1
>>>
>>
> 
>