Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220706205451.4133254-1-jsavitz@redhat.com> <CAA1CXcCHp+zt=34wyFrif+7QtAs9TBOLaL3Nqitz8f_tnR7u9g@mail.gmail.com>
 <CAL1p7m5pLQcvkGajOL8CQiunvye4-8QrDS8oWMShOHmOUFsB-w@mail.gmail.com>
In-Reply-To: <CAL1p7m5pLQcvkGajOL8CQiunvye4-8QrDS8oWMShOHmOUFsB-w@mail.gmail.com>
From:   Nico Pache <npache@redhat.com>
Date:   Tue, 26 Jul 2022 06:31:29 -0400
Message-ID: <CAA1CXcAQpO+T67RheV5Hv6BZ8n-xEUEwztVzkGe+Z2fEL+uJ2Q@mail.gmail.com>
Subject: Re: [PATCH v3] selftests/vm: enable running select groups of tests
To:     Joel Savitz <jsavitz@redhat.com>
Cc:     Andrew Morton <akpm@linux-foundation.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Shuah Khan <shuah@kernel.org>, Linux MM <linux-mm@kvack.org>,
        linux-kselftest@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

Ok thanks for pointing that out!

Andrew, would it be possible to modify the patch and make the variable
name VM_SELFTEST_ITEMS?

Cheers,
-- Nico

On Tue, Jul 19, 2022 at 6:28 PM Joel Savitz <jsavitz@redhat.com> wrote:
>
> Hello,
>
> Sorry for the late reply.
>
> Setting TEST_ITEMS="xx yy zz" and invoking the script has the same
> effect as passing -t "xx yy zz" to the script. VM_TEST_ITEMS may be a
> better env variable name however.
>
> Best,
> Joel Savitz
>
> On Fri, Jul 15, 2022 at 3:55 PM Nico Pache <npache@redhat.com> wrote:
> >
> > Hi Andrew and Joel,
> >
> > I noticed a flaw in switching to the cmdline argument over the
> > environment variable. If you are utilizing the exerciser script (in
> > tools/testing/selftests/run_kselftest.sh), this is not designed to
> > pass cmdline variables to the individual TEST_PROGS. On the other hand
> > if we utilize Env. variables we can still specify which tests we'd
> > like to run. Our infrastructure is designed to utilize this wrapper,
> > so we inevitably will not be able to use this the way we thought.
> >
> > Would you be ok with switching back to the initial approach? If so I
> > believe making the name more specific would be ideal, ie)
> > VM_TEST_ITEMS.
> >
> > Cheers,
> > -- Nico
> >
> > On Wed, Jul 6, 2022 at 4:55 PM Joel Savitz <jsavitz@redhat.com> wrote:
> > >
> > > Our memory management kernel CI testing at Red Hat uses the VM
> > > selftests and we have run into two problems:
> > >
> > > First, our LTP tests overlap with the VM selftests.
> > >
> > > We want to avoid unhelpful redundancy in our testing practices.
> > >
> > > Second, we have observed the current run_vmtests.sh to report overall
> > > failure/ambiguous results in the case that a machine lacks the necessary
> > > hardware to perform one or more of the tests. E.g. ksm tests that
> > > require more than one numa node.
> > >
> > > We want to be able to run the vm selftests suitable to particular hardware.
> > >
> > > Add the ability to run one or more groups of vm tests via run_vmtests.sh
> > > instead of simply all-or-none in order to solve these problems.
> > >
> > > Preserve existing default behavior of running all tests when the script
> > > is invoked with no arguments.
> > >
> > > Documentation of test groups is included in the patch as follows:
> > >
> > >     # ./run_vmtests.sh [ -h || --help ]
> > >
> > >     usage: ./tools/testing/selftests/vm/run_vmtests.sh [ -h | -t "<categories>"]
> > >       -t: specify specific categories to tests to run
> > >       -h: display this message
> > >
> > >     The default behavior is to run all tests.
> > >
> > >     Alternatively, specific groups tests can be run by passing a string
> > >     to the -t argument containing one or more of the following categories
> > >     separated by spaces:
> > >     - mmap
> > >             tests for mmap(2)
> > >     - gup_test
> > >             tests for gup using gup_test interface
> > >     - userfaultfd
> > >             tests for  userfaultfd(2)
> > >     - compaction
> > >             a test for the patch "Allow compaction of unevictable pages"
> > >     - mlock
> > >             tests for mlock(2)
> > >     - mremap
> > >             tests for mremap(2)
> > >     - hugevm
> > >             tests for very large virtual address space
> > >     - vmalloc
> > >             vmalloc smoke tests
> > >     - hmm
> > >             hmm smoke tests
> > >     - madv_populate
> > >             test memadvise(2) MADV_POPULATE_{READ,WRITE} options
> > >     - memfd_secret
> > >             test memfd_secret(2)
> > >     - process_mrelease
> > >             test process_mrelease(2)
> > >     - ksm
> > >             ksm tests that do not require >=2 NUMA nodes
> > >     - ksm_numa
> > >             ksm tests that require >=2 NUMA nodes
> > >     - pkey
> > >             memory protection key tests
> > >     example: ./run_vmtests.sh -t "hmm mmap ksm"
> > >
> > > Changes from v2:
> > >         - rebase onto the mm-everyting branch in
> > >         https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git
> > >         - integrate this functionality with new the tests
> > >
> > > Changes from v1:
> > >         - use a command line argument to pass the test categories to the
> > >           script instead of an environmet variable
> > >         - remove novel prints to avoid messing with extant parsers of this
> > >           script
> > >         - update the usage text
> > >
> > > Signed-off-by: Joel Savitz <jsavitz@redhat.com>
> > > ---
> > >  tools/testing/selftests/vm/run_vmtests.sh | 241 +++++++++++++++-------
> > >  1 file changed, 161 insertions(+), 80 deletions(-)
> > >
> > > diff --git a/tools/testing/selftests/vm/run_vmtests.sh b/tools/testing/selftests/vm/run_vmtests.sh
> > > index d84fe0fa15e1..fb72a1338d07 100755
> > > --- a/tools/testing/selftests/vm/run_vmtests.sh
> > > +++ b/tools/testing/selftests/vm/run_vmtests.sh
> > > @@ -1,6 +1,6 @@
> > >  #!/bin/bash
> > >  # SPDX-License-Identifier: GPL-2.0
> > > -#please run as root
> > > +# Please run as root
> > >
> > >  # Kselftest framework requirement - SKIP code is 4.
> > >  ksft_skip=4
> > > @@ -8,15 +8,75 @@ ksft_skip=4
> > >  mnt=./huge
> > >  exitcode=0
> > >
> > > -#get huge pagesize and freepages from /proc/meminfo
> > > -while read -r name size unit; do
> > > -       if [ "$name" = "HugePages_Free:" ]; then
> > > -               freepgs="$size"
> > > -       fi
> > > -       if [ "$name" = "Hugepagesize:" ]; then
> > > -               hpgsize_KB="$size"
> > > +usage() {
> > > +       cat <<EOF
> > > +usage: ${BASH_SOURCE[0]:-$0} [ -h | -t "<categories>"]
> > > +  -t: specify specific categories to tests to run
> > > +  -h: display this message
> > > +
> > > +The default behavior is to run all tests.
> > > +
> > > +Alternatively, specific groups tests can be run by passing a string
> > > +to the -t argument containing one or more of the following categories
> > > +separated by spaces:
> > > +- mmap
> > > +       tests for mmap(2)
> > > +- gup_test
> > > +       tests for gup using gup_test interface
> > > +- userfaultfd
> > > +       tests for  userfaultfd(2)
> > > +- compaction
> > > +       a test for the patch "Allow compaction of unevictable pages"
> > > +- mlock
> > > +       tests for mlock(2)
> > > +- mremap
> > > +       tests for mremap(2)
> > > +- hugevm
> > > +       tests for very large virtual address space
> > > +- vmalloc
> > > +       vmalloc smoke tests
> > > +- hmm
> > > +       hmm smoke tests
> > > +- madv_populate
> > > +       test memadvise(2) MADV_POPULATE_{READ,WRITE} options
> > > +- memfd_secret
> > > +       test memfd_secret(2)
> > > +- process_mrelease
> > > +       test process_mrelease(2)
> > > +- ksm
> > > +       ksm tests that do not require >=2 NUMA nodes
> > > +- ksm_numa
> > > +       ksm tests that require >=2 NUMA nodes
> > > +- pkey
> > > +       memory protection key tests
> > > +example: ./run_vmtests.sh -t "hmm mmap ksm"
> > > +EOF
> > > +       exit 0
> > > +}
> > > +
> > > +
> > > +while getopts "ht:" OPT; do
> > > +       case ${OPT} in
> > > +               "h") usage ;;
> > > +               "t") TEST_ITEMS=${OPTARG} ;;
> > > +       esac
> > > +done
> > > +shift $((OPTIND -1))
> > > +
> > > +# default behavior: run all tests
> > > +TEST_ITEMS=${TEST_ITEMS:-default}
> > > +
> > > +test_selected() {
> > > +       if [ "$TEST_ITEMS" == "default" ]; then
> > > +               # If no TEST_ITEMS are specified, run all tests
> > > +               return 0
> > >         fi
> > > -done < /proc/meminfo
> > > +       echo ${TEST_ITEMS} | grep ${1} 2>&1 >/dev/null
> > > +       return ${?}
> > > +}
> > > +
> > > +# Hugepage setup only needed for hugetlb tests
> > > +if test_selected "hugetlb"; then
> > >
> > >  # Simple hugetlbfs tests have a hardcoded minimum requirement of
> > >  # huge pages totaling 256MB (262144KB) in size.  The userfaultfd
> > > @@ -28,7 +88,17 @@ hpgsize_MB=$((hpgsize_KB / 1024))
> > >  half_ufd_size_MB=$((((nr_cpus * hpgsize_MB + 127) / 128) * 128))
> > >  needmem_KB=$((half_ufd_size_MB * 2 * 1024))
> > >
> > > -#set proper nr_hugepages
> > > +# get huge pagesize and freepages from /proc/meminfo
> > > +while read -r name size unit; do
> > > +       if [ "$name" = "HugePages_Free:" ]; then
> > > +               freepgs="$size"
> > > +       fi
> > > +       if [ "$name" = "Hugepagesize:" ]; then
> > > +               hpgsize_KB="$size"
> > > +       fi
> > > +done < /proc/meminfo
> > > +
> > > +# set proper nr_hugepages
> > >  if [ -n "$freepgs" ] && [ -n "$hpgsize_KB" ]; then
> > >         nr_hugepgs=$(cat /proc/sys/vm/nr_hugepages)
> > >         needpgs=$((needmem_KB / hpgsize_KB))
> > > @@ -57,144 +127,155 @@ else
> > >         exit 1
> > >  fi
> > >
> > > -#filter 64bit architectures
> > > +fi # test_selected "hugetlb"
> > > +
> > > +# filter 64bit architectures
> > >  ARCH64STR="arm64 ia64 mips64 parisc64 ppc64 ppc64le riscv64 s390x sh64 sparc64 x86_64"
> > >  if [ -z "$ARCH" ]; then
> > >         ARCH=$(uname -m 2>/dev/null | sed -e 's/aarch64.*/arm64/')
> > >  fi
> > >  VADDR64=0
> > > -echo "$ARCH64STR" | grep "$ARCH" && VADDR64=1
> > > +echo "$ARCH64STR" | grep "$ARCH" &>/dev/null && VADDR64=1
> > >
> > >  # Usage: run_test [test binary] [arbitrary test arguments...]
> > >  run_test() {
> > > -       local title="running $*"
> > > -       local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -)
> > > -       printf "%s\n%s\n%s\n" "$sep" "$title" "$sep"
> > > -
> > > -       "$@"
> > > -       local ret=$?
> > > -       if [ $ret -eq 0 ]; then
> > > -               echo "[PASS]"
> > > -       elif [ $ret -eq $ksft_skip ]; then
> > > -               echo "[SKIP]"
> > > -               exitcode=$ksft_skip
> > > -       else
> > > -               echo "[FAIL]"
> > > -               exitcode=1
> > > -       fi
> > > +       if test_selected ${CATEGORY}; then
> > > +               local title="running $*"
> > > +               local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -)
> > > +               printf "%s\n%s\n%s\n" "$sep" "$title" "$sep"
> > > +
> > > +               "$@"
> > > +               local ret=$?
> > > +               if [ $ret -eq 0 ]; then
> > > +                       echo "[PASS]"
> > > +               elif [ $ret -eq $ksft_skip ]; then
> > > +                       echo "[SKIP]"
> > > +                       exitcode=$ksft_skip
> > > +               else
> > > +                       echo "[FAIL]"
> > > +                       exitcode=1
> > > +               fi
> > > +       fi # test_selected
> > >  }
> > >
> > > -mkdir "$mnt"
> > > -mount -t hugetlbfs none "$mnt"
> > > +# setup only needed for hugetlb tests
> > > +if test_selected "hugetlb"; then
> > > +       mkdir "$mnt"
> > > +       mount -t hugetlbfs none "$mnt"
> > > +fi
> > >
> > > -run_test ./hugepage-mmap
> > > +CATEGORY="hugetlb" run_test ./hugepage-mmap
> > >
> > >  shmmax=$(cat /proc/sys/kernel/shmmax)
> > >  shmall=$(cat /proc/sys/kernel/shmall)
> > >  echo 268435456 > /proc/sys/kernel/shmmax
> > >  echo 4194304 > /proc/sys/kernel/shmall
> > > -run_test ./hugepage-shm
> > > +CATEGORY="hugetlb" run_test ./hugepage-shm
> > >  echo "$shmmax" > /proc/sys/kernel/shmmax
> > >  echo "$shmall" > /proc/sys/kernel/shmall
> > >
> > > -run_test ./map_hugetlb
> > > +CATEGORY="hugetlb" run_test ./map_hugetlb
> > >
> > > -run_test ./hugepage-mremap "$mnt"/huge_mremap
> > > -rm -f "$mnt"/huge_mremap
> > > +CATEGORY="hugetlb" run_test ./hugepage-mremap "$mnt"/huge_mremap
> > > +test_selected "hugetlb" && rm -f "$mnt"/huge_mremap
> > >
> > > -run_test ./hugepage-vmemmap
> > > +CATEGORY="hugetlb" run_test ./hugepage-vmemmap
> > >
> > > -run_test ./hugetlb-madvise "$mnt"/madvise-test
> > > -rm -f "$mnt"/madvise-test
> > > +CATEGORY="hugetlb" run_test ./hugetlb-madvise "$mnt"/madvise-test
> > > +test_selected "hugetlb" && rm -f "$mnt"/madvise-test
> > >
> > > -echo "NOTE: The above hugetlb tests provide minimal coverage.  Use"
> > > -echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for"
> > > -echo "      hugetlb regression testing."
> > > +if test_selected "hugetlb"; then
> > > +       echo "NOTE: These hugetlb tests provide minimal coverage.  Use"
> > > +       echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for"
> > > +       echo "      hugetlb regression testing."
> > > +fi
> > >
> > > -run_test ./map_fixed_noreplace
> > > +CATEGORY="mmap" run_test ./map_fixed_noreplace
> > >
> > >  # get_user_pages_fast() benchmark
> > > -run_test ./gup_test -u
> > > +CATEGORY="gup_test" run_test ./gup_test -u
> > >  # pin_user_pages_fast() benchmark
> > > -run_test ./gup_test -a
> > > +CATEGORY="gup_test" run_test ./gup_test -a
> > >  # Dump pages 0, 19, and 4096, using pin_user_pages:
> > > -run_test ./gup_test -ct -F 0x1 0 19 0x1000
> > > +CATEGORY="gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000
> > >
> > > -run_test ./userfaultfd anon 20 16
> > > -run_test ./userfaultfd anon:dev 20 16
> > > +CATEGORY="userfaultfd" run_test ./userfaultfd anon 20 16
> > > +CATEGORY="userfaultfd" run_test ./userfaultfd anon:dev 20 16
> > >  # Hugetlb tests require source and destination huge pages. Pass in half the
> > >  # size ($half_ufd_size_MB), which is used for *each*.
> > > -run_test ./userfaultfd hugetlb "$half_ufd_size_MB" 32
> > > -run_test ./userfaultfd hugetlb:dev "$half_ufd_size_MB" 32
> > > -run_test ./userfaultfd hugetlb_shared "$half_ufd_size_MB" 32 "$mnt"/uffd-test
> > > +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb "$half_ufd_size_MB" 32
> > > +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb:dev "$half_ufd_size_MB" 32
> > > +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb_shared "$half_ufd_size_MB" 32 "$mnt"/uffd-test
> > >  rm -f "$mnt"/uffd-test
> > > -run_test ./userfaultfd hugetlb_shared:dev "$half_ufd_size_MB" 32 "$mnt"/uffd-test
> > > +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb_shared:dev "$half_ufd_size_MB" 32 "$mnt"/uffd-test
> > >  rm -f "$mnt"/uffd-test
> > > -run_test ./userfaultfd shmem 20 16
> > > -run_test ./userfaultfd shmem:dev 20 16
> > > -
> > > -#cleanup
> > > -umount "$mnt"
> > > -rm -rf "$mnt"
> > > -echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages
> > > +CATEGORY="userfaultfd" run_test ./userfaultfd shmem 20 16
> > > +CATEGORY="userfaultfd" run_test ./userfaultfd shmem:dev 20 16
> > > +
> > > +# cleanup (only needed when running hugetlb tests)
> > > +if test_selected "hugetlb"; then
> > > +       umount "$mnt"
> > > +       rm -rf "$mnt"
> > > +       echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages
> > > +fi
> > >
> > > -run_test ./compaction_test
> > > +CATEGORY="compaction" run_test ./compaction_test
> > >
> > > -run_test sudo -u nobody ./on-fault-limit
> > > +CATEGORY="mlock" run_test sudo -u nobody ./on-fault-limit
> > >
> > > -run_test ./map_populate
> > > +CATEGORY="mmap" run_test ./map_populate
> > >
> > > -run_test ./mlock-random-test
> > > +CATEGORY="mlock" run_test ./mlock-random-test
> > >
> > > -run_test ./mlock2-tests
> > > +CATEGORY="mlock" run_test ./mlock2-tests
> > >
> > > -run_test ./mrelease_test
> > > +CATEGORY="process_mrelease" run_test ./mrelease_test
> > >
> > > -run_test ./mremap_test
> > > +CATEGORY="mremap" run_test ./mremap_test
> > >
> > > -run_test ./thuge-gen
> > > +CATEGORY="hugetlb" run_test ./thuge-gen
> > >
> > >  if [ $VADDR64 -ne 0 ]; then
> > > -       run_test ./virtual_address_range
> > > +       CATEGORY="hugevm" run_test ./virtual_address_range
> > >
> > >         # virtual address 128TB switch test
> > > -       run_test ./va_128TBswitch.sh
> > > +       CATEGORY="hugevm" run_test ./va_128TBswitch.sh
> > >  fi # VADDR64
> > >
> > >  # vmalloc stability smoke test
> > > -run_test ./test_vmalloc.sh smoke
> > > +CATEGORY="vmalloc" run_test ./test_vmalloc.sh smoke
> > >
> > > -run_test ./mremap_dontunmap
> > > +CATEGORY="mremap" run_test ./mremap_dontunmap
> > >
> > > -run_test ./test_hmm.sh smoke
> > > +CATEGORY="hmm" run_test ./test_hmm.sh smoke
> > >
> > >  # MADV_POPULATE_READ and MADV_POPULATE_WRITE tests
> > > -run_test ./madv_populate
> > > +CATEGORY="madv_populate" run_test ./madv_populate
> > >
> > > -run_test ./memfd_secret
> > > +CATEGORY="memfd_secret" run_test ./memfd_secret
> > >
> > >  # KSM MADV_MERGEABLE test with 10 identical pages
> > > -run_test ./ksm_tests -M -p 10
> > > +CATEGORY="ksm" run_test ./ksm_tests -M -p 10
> > >  # KSM unmerge test
> > > -run_test ./ksm_tests -U
> > > +CATEGORY="ksm" run_test ./ksm_tests -U
> > >  # KSM test with 10 zero pages and use_zero_pages = 0
> > > -run_test ./ksm_tests -Z -p 10 -z 0
> > > +CATEGORY="ksm" run_test ./ksm_tests -Z -p 10 -z 0
> > >  # KSM test with 10 zero pages and use_zero_pages = 1
> > > -run_test ./ksm_tests -Z -p 10 -z 1
> > > +CATEGORY="ksm" run_test ./ksm_tests -Z -p 10 -z 1
> > >  # KSM test with 2 NUMA nodes and merge_across_nodes = 1
> > > -run_test ./ksm_tests -N -m 1
> > > +CATEGORY="ksm_numa" run_test ./ksm_tests -N -m 1
> > >  # KSM test with 2 NUMA nodes and merge_across_nodes = 0
> > > -run_test ./ksm_tests -N -m 0
> > > +CATEGORY="ksm_numa" run_test ./ksm_tests -N -m 0
> > >
> > >  # protection_keys tests
> > >  if [ -x ./protection_keys_32 ]
> > >  then
> > > -       run_test ./protection_keys_32
> > > +       CATEGORY="pkey" run_test ./protection_keys_32
> > >  fi
> > >
> > >  if [ -x ./protection_keys_64 ]
> > >  then
> > > -       run_test ./protection_keys_64
> > > +       CATEGORY="pkey" run_test ./protection_keys_64
> > >  fi
> > >
> > >  exit $exitcode
> > > --
> > > 2.31.1
> > >
> >
>