Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.117.160 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C
Message-ID: <6f845148-e180-822d-a54f-e2bb844d54f8@nvidia.com>
Date:   Fri, 2 Jun 2023 13:38:21 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.2
Subject: Re: [PATCH 05/12] selftests/mm: fix invocation of tests that are run
 via shell scripts
Content-Language: en-US
To:     David Hildenbrand <david@redhat.com>,
        Andrew Morton <akpm@linux-foundation.org>
CC:     Peter Xu <peterx@redhat.com>, Shuah Khan <shuah@kernel.org>,
        "Nathan Chancellor" <nathan@kernel.org>, <linux-mm@kvack.org>,
        <linux-kselftest@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>
References: <20230602013358.900637-1-jhubbard@nvidia.com>
 <20230602013358.900637-6-jhubbard@nvidia.com>
 <ead7d0db-6104-1d62-b3d0-f9ebb767af4d@redhat.com>
From:   John Hubbard <jhubbard@nvidia.com>
In-Reply-To: <ead7d0db-6104-1d62-b3d0-f9ebb767af4d@redhat.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jun 2023 20:38:30.2407
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: c3b8d0f0-40f5-4dd3-80f2-08db63a9537b
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT068.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8755
X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,
        SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 6/2/23 03:05, David Hildenbrand wrote:
> On 02.06.23 03:33, John Hubbard wrote:
>> We cannot depend upon git to reliably retain the executable bit on shell
>> scripts, or so I was told several years ago while working on this same
>> run_vmtests.sh script. And sure enough, things such as test_hmm.sh are
>> lately failing to run, due to lacking execute permissions.
>>
>> A nice clean way to fix this would have been to use TEST_PROGS instead
>> of TEST_FILES for the .sh scripts here. That tells the selftest
>> framework to run these (and emit a warning if the files are not
>> executable, but still run them anyway).

Actually, for the record (and I'll update this in v2), the above is
inaccurate, because run_vmtests.sh aspires to be the only TEST_PROGS
item here. And I see that the framework does already work if-and-only-if
invoked via Make, as in "make run_tests".

However,

a) Many people naturally expect to run test scripts without
(unnecessarily!) involving Make, and

b) Based on some experience in building and using various test
frameworks over many years, I'd claim that it's better to use shell
scripts to collect and manage tests and test scripts, rather than
involving Make. Make is a limited, specialized language and is better at
handling builds and dependencies.

So the "make run_tests" is a convenience, but it should not be the only
way to launch a test run. So we still want to fix this up.

>>
>> Unfortunately, run_vmtests.sh has its own run_test() routine, which does
>> *not* do the right thing for shell scripts.
>>
>> Fix this by explicitly adding "bash" to each of the shell script
>> invocations. Leave fixing the overall approach to another day.
>>
>> Signed-off-by: John Hubbard <jhubbard@nvidia.com>
>> ---
>>   tools/testing/selftests/mm/run_vmtests.sh | 6 +++---
>>   1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
>> index 4893eb60d96d..8f81432e4bac 100644
>> --- a/tools/testing/selftests/mm/run_vmtests.sh
>> +++ b/tools/testing/selftests/mm/run_vmtests.sh
>> @@ -242,18 +242,18 @@ if [ $VADDR64 -ne 0 ]; then
>>       if [ "$ARCH" == "$ARCH_ARM64" ]; then
>>           echo 6 > /proc/sys/vm/nr_hugepages
>>       fi
>> -    CATEGORY="hugevm" run_test ./va_high_addr_switch.sh
>> +    CATEGORY="hugevm" run_test bash ./va_high_addr_switch.sh
>>       if [ "$ARCH" == "$ARCH_ARM64" ]; then
>>           echo $prev_nr_hugepages > /proc/sys/vm/nr_hugepages
>>       fi
>>   fi # VADDR64
>>   # vmalloc stability smoke test
>> -CATEGORY="vmalloc" run_test ./test_vmalloc.sh smoke
>> +CATEGORY="vmalloc" run_test bash ./test_vmalloc.sh smoke
>>   CATEGORY="mremap" run_test ./mremap_dontunmap
>> -CATEGORY="hmm" run_test ./test_hmm.sh smoke
>> +CATEGORY="hmm" run_test bash ./test_hmm.sh smoke
>>   # MADV_POPULATE_READ and MADV_POPULATE_WRITE tests
>>   CATEGORY="madv_populate" run_test ./madv_populate
> 
> Sounds hacky, but if it gets the job done
> 

Yes. It's also hacky that we can't just invoke shell scripts like normal
programs. This limitation hurts my sense of "things should be more
perfect!". :)

> Acked-by: David Hildenbrand <david@redhat.com>
> 

Thanks for the ack.

-- 
John Hubbard
NVIDIA