Message-ID: <6d4eb508-f551-4c12-2e15-34ab9b1dc49f@intel.com>
Date:   Wed, 1 Dec 2021 16:18:47 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.3.2
Subject: Re: [PATCH 1/3] selftests/resctrl: Make resctrl_tests run using
 kselftest framework
Content-Language: en-US
To:     "tan.shaopeng@fujitsu.com" <tan.shaopeng@fujitsu.com>,
        Fenghua Yu <fenghua.yu@intel.com>,
        Shuah Khan <shuah@kernel.org>
CC:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>
References: <20211110093315.3219191-1-tan.shaopeng@jp.fujitsu.com>
 <20211110093315.3219191-2-tan.shaopeng@jp.fujitsu.com>
 <08708d7a-4d34-3817-032f-25698a30551c@intel.com>
 <TYAPR01MB633090B5A9DE4480D12CC9268B689@TYAPR01MB6330.jpnprd01.prod.outlook.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <TYAPR01MB633090B5A9DE4480D12CC9268B689@TYAPR01MB6330.jpnprd01.prod.outlook.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VDhLVWVJb1ZReVEyMG55Tmx0d3VOZmNNMDMyTkFNQVk2ZmtaTFN0Tno2QTl4?=
 =?utf-8?B?YmQwaTd6RVFkelU4WHVZU2J5RFVjWWtaSStEdHZlWUYxbjlwQ1FScmF0Zkcw?=
 =?utf-8?B?ZklRSjBIZFZ0WnhHSlZ4UXUrRGtUSEtIL0FmQ1Q0NXoyaEk1MjZmYVB5ZVZF?=
 =?utf-8?B?M2xQNG4wSjRBZUwzRVhoRm9JVmxlc2YyMkQzSy9SMkRPelJSeGp3NnA1aElM?=
 =?utf-8?B?b3pETEx6UTVXai9jZVZPYU1HRjdITGVCNjhGSUw5bkZ0aEExM1BNSzhQbHpl?=
 =?utf-8?B?enFqUHhmYmhTSFVrSE0venFUdE1aNHVHNGF0SDU0OHhpMkhTZEl0ZnF3cE1v?=
 =?utf-8?B?YitDMjlCYlk5Q3hiTWJtS2gvL25vRGtvMWlpaGk3UjVnRVJ0Z2tYdzBwVklR?=
 =?utf-8?B?SDFCUFhWdXkyby9jVUN3ZmljTk9TVDZXWGp1cmR0OUI5RXFnWGV0VTQ3am1M?=
 =?utf-8?B?QnRqUEcxbW1YdlI1MTBpM09OYjlHMzVrc3Y5V1pTQzBWbHRpYmZKeXNoWjFj?=
 =?utf-8?B?L1hnM3pqN1lFYnN1c0pXY2IxOUVPeGh1THdBSXlPbHhScE51VDI3L2tSNDQy?=
 =?utf-8?B?RXR0bldKTGZPWGtZdXlOSjl2YWo5WTdPVGpvSTZoZGxra1NwemNaclNYend5?=
 =?utf-8?B?ays0TXJaTVFucXllODkzZHhmRUFaVkxobEluRVJ2SzVZbFJOYVZVVWdSeVBC?=
 =?utf-8?B?ZENnK21HUU9QUjdHWFE4d2xQbGo4VnU4aHNTeE53a3RRdmdnNGdQbDhLZGRF?=
 =?utf-8?B?VmVHeS9xbmtCbk05R2dWT0ZDS2pUM3c4VUpOcG41RTZMaXc0TGNYYmlhT2lm?=
 =?utf-8?B?Qzl2bzQrZWE0NEpmaW9FcnAwbVNsemd4V1lpUG9KcFo0RGpjREVvRGY2czlZ?=
 =?utf-8?B?YWFGL0dwS1ArOWN6VmcvRm96SThON1NSYU1jc1pPUEZsSnNVVUY3VmNYSEJw?=
 =?utf-8?B?Q2dMRlVNU0VhWUNlRTVxWmtJdG1wa3l4WnVQelpkd01NWE1iNG8zMDd1VUF1?=
 =?utf-8?B?cStzb0ZhNmhha0JMZFdpTHpuM3ZINVBzbHYyRXJJSUVuWmdrUTdpT0F5UVNu?=
 =?utf-8?B?cHJFVTc1WGxLRno2ZU45eVMrd2hSVG50OGpTODJLQXNRZmhOWGdNcVp4YXVl?=
 =?utf-8?B?YUIwQStKZ1J4eGxpT1RZdnZSYUZoWnlTNmxycmJPdFEzU0ZaUmVwNzlFbVlx?=
 =?utf-8?B?S3JoZ2xBSGx4NG1KOWdJVUtyWEtzZFFPd2cvQUU0VlRiRWw3dW5jMVFEY2lk?=
 =?utf-8?B?TzJ2dnRYZnUwbDdWSXN1cjJ6UnQ3eUtKY3pzaEh0M1htWWFaSzVaL1R0NUtn?=
 =?utf-8?B?RjhwRHhVZnZFendCT1NBeElXWWNGaFRCVC84ekh1b0VyZ3E2TStXdi9temlq?=
 =?utf-8?B?STNVeGttSXhTQzJGZlBodWhNMWhZSTZ0UURqOWhPQ0JqNVVaQjkzZDNNekR4?=
 =?utf-8?B?MkdBN0ZnR2ttTnRFZ0hHcjdSVkV6YzV3NDE3OHdUV1ZzNVF3R1RPaTNEVVlh?=
 =?utf-8?B?bzJjRTR3bjlYQ1FvRDlGYmszTmNjU1JuMGRseVpBeUh2Q2pJbUpmVFBFTllk?=
 =?utf-8?B?NEMvZEdOV1BEQ01GUTRuNE5rSEVpSzNvMk4rL0JHMzJXSlpLSDFFTzBOVXpt?=
 =?utf-8?B?R2FXMWJNZkFvdEJ0eGwwUU4yMUk4aENJMlBvdlg1cFBDYnZMNnB5N25wdnpu?=
 =?utf-8?B?L0s0YU5LVloreStnbkExdHY0SkhmUGl3Tmdnb2FGNTFBS2IvKzdaRUt5K05F?=
 =?utf-8?B?bzJVdWJwZlY3VlNVVFE4MERFMEdYRDFiKytiUmVhWklZeVR0RXBrek9PajZ2?=
 =?utf-8?B?dzRrT3R0cWdFbE5LOG9nVldlQVhBZm9reHd5U2ZJOTNmZ1RzaDQ5YVl2V3dy?=
 =?utf-8?B?bkRHYWlZV2Q5dk1kZktWQXhQMnRCSVdtNUEyeFVoU1BNN29FdEQxd2lqeFZO?=
 =?utf-8?B?eWJESllta1AwUkZNbVNKMHI5L1BOcktrM2lzRW4vcmthRHp6RUlzTUVKWFFa?=
 =?utf-8?B?VjNOYzhPbXRJSDRydmlJdVg1NzlNVnltT01SRDE4QllxSXFPT2FjMTlSUjBK?=
 =?utf-8?B?Mklsd1BtY1RpbUhneXB6RUovbFdwOVFqYjZhSFBKTHJtYVEwY1NPNFRsR3pZ?=
 =?utf-8?B?ZDgxR255Sk9TUzhmT0ozbDM2WnhkT05RbmNUWGYwL0JMZ2hXajQ4cHZmdjNj?=
 =?utf-8?B?N25BTE1QNW1URGkwZi9lVG1TdHVSNmJWRnlzcmw0MHhBWVJCNnpVeDdEdVFu?=
 =?utf-8?Q?LKkIGgIIvexEPz/rzggGIKMs6Xkp3vVn+Vt4HULMzw=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: abf11139-2cb3-4926-a52f-08d9b5295169
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Dec 2021 00:18:52.1018
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 8oCTRwUv0QD89QESMew5d5TaXRLX/ePyD4FpQuW+VAo/DeEKF69jDG4VuJGiFMtrxbw24SLY4zlX+XXDGsx5QqAt8eZlW1aqidbpK2Z+S0A=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR11MB2675
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Shaopeng Tan,

On 11/30/2021 6:36 PM, tan.shaopeng@fujitsu.com wrote:
> Hi Reinette
> 
>> On 11/10/2021 1:33 AM, Shaopeng Tan wrote:
>>> From: "Tan, Shaopeng" <tan.shaopeng@jp.fujitsu.com>
>>>
>>> This commit enables kselftest to be built/run in resctrl framework.
>>> Build/run resctrl_tests by build/run all tests of kselftest, or by
>>> using the "TARGETS" variable on the make command line to specify
>> resctrl_tests.
>>> To make resctrl_tests run using kselftest framework, this commit
>>> modified the Makefile of kernel kselftest set and the Makefile of
>> resctrl_tests.
>>
>> The above sentence mentions that changes were made to the resctrl selftest
>> Makefile but it does not describe what the change accomplish or why they are
>> needed. Could you please elaborate?
> 
> Before these changes of Makefile, when we run resctrl test,
> we need to goto tools/testing/selftests/resctrl/ directory,
> run "make" to build executable file "resctrl_tests",
> and run "sudo ./resctrl_tests" to execute the test.
> 
> With this patch, we can resctrl test in selftest framwork as follow.
> Run all tests include resctrl:
>   $ make -C tools/testing/selftests run_tests
> Run a subset(resctrl) of selftests:
>   $ make -C tools/testing/selftests TARGETS=resctrl run_tests
> 
> Linux Kernel Selftests :
> https://www.kernel.org/doc/html/latest/dev-tools/kselftest.html

Understood and this is a reasonable change. What you wrote above would 
be a great addition to the changelog. I think it is also important to 
add to the changelog that the changes do not change the existing 
behavior and users can continue to build and run the tests as before.

Also, please follow the guidance found in "Separate your changes" in
Documentation/process/submitting-patches.rst: Logical changes should be 
in separate patches. This patch does too many things. Please consider:
1) the license addition is not relevant to this work
2) the new comment seems unnecessary as it states the obvious
3) where is the "LDLIBS += -lnuma" change coming from and why is it needed?

When logical changes are isolated into separate patches it really makes 
the patches easier to understand.

>>> To ensure the resctrl_tests finish in limited time, this commit
>>> changed the default limited time(45s) to 120 seconds for resctrl_tests
>>> by adding "setting" file.
>>
>> How is changing the timeout related to the resctrl framework changes? Is it not
>> a separate change?
> 
> In selftest framwork, the default limited time of all tests is 45 seconds
> which is specified by common file tools/testing/selftests/kselftest/runner.sh.
> Each test can change the limited time individually by adding a "setting"
> file into its own directory. I changed the limited time of resctrl to120s
> because 45s was not enough in my environment.

Understood. My question was if this can be a separate change with its 
own patch? It seems to me that this deserves its own patch ... but 
actually it also raises an important issue that the resctrl tests are 
taking a long time.

I do see a rule for tests in Documentation/dev-tools/kselftest.rst: 
"Don't take too long". This may be a motivation _not_ to include the 
resctrl tests in the regular kselftest targets because when a user wants 
to run all tests on a system it needs to be quick and the resctrl tests 
are not quick.

Reinette