Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Message-ID: <6f9083e2-d633-d483-702e-f974317133b7@intel.com>
Date:   Mon, 31 Jan 2022 13:20:18 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.5.1
Subject: Re: [PATCH] docs/kselftest: clarify running mainline tests on stables
Content-Language: en-US
To:     Shuah Khan <skhan@linuxfoundation.org>, <shuah@kernel.org>,
        <corbet@lwn.net>
CC:     <linux-kselftest@vger.kernel.org>, <linux-doc@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>
References: <20220126201341.55771-1-skhan@linuxfoundation.org>
 <7ba5e99a-9169-75c4-2324-f9a3ce9a506e@intel.com>
 <bb1ba6f5-4cd8-742c-62b7-a62a6f4cef91@linuxfoundation.org>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <bb1ba6f5-4cd8-742c-62b7-a62a6f4cef91@linuxfoundation.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cERkeEUvQnB1cjlDMVNJR2ZGZGhYcWNrUVUrZEgraGUzbmtZS3c4WkpkZlRu?=
 =?utf-8?B?ZlBUQU4yV0lSTElMd0pZVEp0ZlNzQWoxNk5ROHRZWStpeVVIWndyWEhTMkNI?=
 =?utf-8?B?SlltVjF0VmowZDA4b0xoUVJJOU5pWER1ZEFzYWt4SndUaDZ6R1Bac3ZOdEYr?=
 =?utf-8?B?VUl6VTZna0M4WmFvblBDbzJTdEFiMHl2cEF3TktaRE1SWmRJQ3FCdS9yTkVJ?=
 =?utf-8?B?ck5uMndZY1ZyYkg5bUtNSk1YVEFkc3l1NVpYOFV0bkZHMzFwdnErdmg5azIv?=
 =?utf-8?B?QWpUWHkxQTBKY3VQeFFwRmFXRjgwVnM1aVRxUjhrZktNem5iK21SbkVOT0Zh?=
 =?utf-8?B?Rnc1elZkTXB3Zm5VRUtQVGZPOGpYNHBRRVNndE1JMlJOQ2htcmRaL2dFcU1Q?=
 =?utf-8?B?cE5JV3VUMHRXaGNKaWtRNUJDcWhtcFMxdFQ2SHdFckVqZnozcXlsRnIxV2lI?=
 =?utf-8?B?N1lRcVJ0MGdjTWQ1bzZvNERvbmx4ZUZyV3F1b3lqcjlXdEFhdWgrWTJNSFc2?=
 =?utf-8?B?eXZEMmpzUTVwQ1ptWGpuTi9aTnVnZkxxY3h6RDRPVTBCSDFoeWRwS1NPbVBw?=
 =?utf-8?B?ZmJGMnlFRFU3VlZBTEY3cjRnVmZ0d3hQYlhuNXRMM01hb2NMMFJPR0JnNXlt?=
 =?utf-8?B?RWZhY29yNmUvZ05lVlgvdkJNenN2N0xzQVplV05KVG5zVVJNUXlEaUVJTmQ0?=
 =?utf-8?B?QlRsYUF4YUlFQ3U2UTRRRkhUT210STBUVGIxakFIYXkwZW9uNnR5cW52K01D?=
 =?utf-8?B?RS9zMWNzeEovelhLejZzNmlaQkFVVzNPV25EdXREcERNT0ViMzFtMEZEZzIw?=
 =?utf-8?B?RUNFQWdZNWdLbktkeGJSRUlpSE16a0xJOW9hK1Q5YlBrWHQxOFhpQjRLTlVw?=
 =?utf-8?B?b1Z4QTBXVzJTUGZNOGV4MXplS2llQ2tlcFkxMjZ1YkRtMmVpSXBCYkpuTzBU?=
 =?utf-8?B?am11blRaZlFpTjA2bnZEa1BBZlZhMG1QT2dRKzRhSWRraW9qbGhLa0N6Njg5?=
 =?utf-8?B?c2tBRXQxc21DTUE0NFpFbG9oZCt6YUxOQWw0M0w0R0xqNzlqUmduMHhYY0Vr?=
 =?utf-8?B?WlJnUmJ0NnJ1RGIzMS80cVlZdzkxVU1MSFVKWEUvSWI4THN4c0p5eHpQVW5Y?=
 =?utf-8?B?MDR0TENyeGV1UU1lMDBBSlRQSzlzb3pSMTJFZEdwZ3RmVjduNEFFcXZSSVhw?=
 =?utf-8?B?dWNXSDBDbWVtUHU5VU1wNUYzdTRIajVPeGJ1OVEwMSt5RTNqZXRJQ1p0RGJX?=
 =?utf-8?B?eTc2cWVka2hVWVVBVjc2UzRIcmlyK0d2M3c0aWpZeVYrRXVlZ1VTSllHcm05?=
 =?utf-8?B?NVp5b29XTmV3RTE4d1hEUWNoMnZHZGR6OFE4WmszNkdtZS9Xdm04ckwxamx5?=
 =?utf-8?B?MzVKVmUrMit0aTZoU2V5SkhYSjVPSXBpbXdqM3Bxa1JLb3JjR3g2b3Fwek9O?=
 =?utf-8?B?VGZ0YUlrQTJ5TFFaa3hKZlZPMHNBQXlIejdyR0ovMnUwTEx1RVVXREdxOHh6?=
 =?utf-8?B?YTNmdFZyZmV5U0Y3NUExOVAwNElVZnA3RlpiN245MW1Ibm1tSVA1YnpKejNU?=
 =?utf-8?B?NGtmRHM0U2hWU0hmR0xNb3U1RVhzZVNNQjAxVFFieksxeEVKY1FOYXd5eXVn?=
 =?utf-8?B?M1gyZ253VzVBcHE0dVhzZGt4ajlvdmdnbHNrWElqYnV5TFpZcytzYTEvOHZN?=
 =?utf-8?B?MkMxRUZuTCtkdWlOY0I1Q0RMMHp0bm0xTGxsUlRWTkFaRFFkQ1JmT2dVMFNR?=
 =?utf-8?B?SWtVODJQdUN0WGJZZThyaW53eFMzK1FqYUJpRndtZUxtKzVuT05iVC8zQVQ3?=
 =?utf-8?B?OVZnT2hIcmJDYjNzSEd4TVVYSkN6cXMrdm9jYUVaVDdSbnZDbTJqR1lEVEtw?=
 =?utf-8?B?WGZJMjByWkRRVEgvMldwckZMdmZHRXFUbjVjcFlzR1hnemFoZnZERVBWM0l5?=
 =?utf-8?B?d0E4RFE2UUZPMUtSdGg5YjE4cVcvbXlrWmxRSkJSYUFXSmhiYTZOMnpaMzJP?=
 =?utf-8?B?K2p5TjdyR0xQTWg2VitsNmFBdHRSeVMwZElaekVNZ1dKWkVNS3lRanpBMWVx?=
 =?utf-8?B?YW9wemE1VE1pVVp2bmVuVjg5Yklpb1pxUEp5OXAzaTFnMUVxWjRtdTZUN3BW?=
 =?utf-8?B?aDNQNGhYMSsrZnpQMnllL1NYNkRqZGdrQnU0K29JMUJCUzA2dENJWFoyV1Q0?=
 =?utf-8?B?MVN4UGVlaDdCL0lTYzBLV2dXOUNWTnBYUDRrS0FNUmpqNmdnZWVnU040MnBi?=
 =?utf-8?Q?VRztf8F3+89Mf9WL0y5OenkYF8TRVza4697NpzvG5k=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: bd8eef9a-9504-4afb-5fb7-08d9e4ff7d72
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Jan 2022 21:20:22.3964
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: KMpk0H6qJn7pSCg8bchpaKsYNHgJZ7Cxyi62kiTcPqAkZP2JIts83+mlBMsw1FkO0bH02m6C3ZCrteFyyHP2agiyv7VF5P7oCBAmS3/Bo48=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB5753
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Shuah,

On 1/31/2022 12:34 PM, Shuah Khan wrote:
> On 1/31/22 12:37 PM, Reinette Chatre wrote:
>> On 1/26/2022 12:13 PM, Shuah Khan wrote:
>>> Update the document to clarifiy support for running mainline
>>> kselftest on stable releases and the reasons for not removing
>>> test code that can test older kernels.
>>>
>>> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
>>> ---
>>>   Documentation/dev-tools/kselftest.rst | 8 ++++++++
>>>   1 file changed, 8 insertions(+)
>>>
>>> diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
>>> index dcefee707ccd..a833ecf12fbc 100644
>>> --- a/Documentation/dev-tools/kselftest.rst
>>> +++ b/Documentation/dev-tools/kselftest.rst
>>> @@ -7,6 +7,14 @@ directory. These are intended to be small tests to exercise individual code
>>>   paths in the kernel. Tests are intended to be run after building, installing
>>>   and booting a kernel.
>>>   +Kselftest from mainline can be run on older stable kernels. Running tests
>>> +from mainline offers the best coverage. Several test rings run mainline
>>> +kselftest suite on stable releases. The reason is that when a new test
>>> +gets added to test existing code to regression test a bug, we should be
>>> +able to run that test on an older kernel. Hence, it is important to keep
>>> +code that can still test an older kernel and make sure it skips the test
>>> +gracefully on newer releases.
>>> +
>>>   You can find additional information on Kselftest framework, how to
>>>   write new tests using the framework on Kselftest wiki:
>>>   
>>
>> (My apologies if this is already documented, I was not able to find this guidance
>> in Documentation/dev-tools/kselftest.rst nor when looking at the
>> "Kselftest use-cases..." slides linked from https://kselftest.wiki.kernel.org/)
>>
>> Could you please clarify what the requirement/expectation is regarding fixes
>> to tests? Since the recommendation in the above change is that Kselftest from
>> mainline should be run on older stable kernels, is it required to backport
>> fixes to the tests themselves to stable kernels?
>>
> 
> Couple of things to consider.
> 
> - A new test gets added to regression test a bug in stable and mainline
> - A new test gets added to test a kernel module/feature/API that has been
>   supported by stable and mainline releases
> 
> In both of these cases, running mainline kselftest on stables gives you the
> best coverage.
> 
> Kselftest fixes get pulled into stables like any other kernel fixes. If a few
> fixes are missing, it is a good idea to back-port if they fall into above two
> categories. If the test is for a new feature then, it doesn't make sense to
> back-port.
> 
> Hope this is helpful.

This is helpful, thank you very much. In summary I understand this to mean that
when testing a stable kernel it is recommended to run tests from mainline, but
running the tests from the same stable kernel version as the kernel being tested
is also a supported use case and thus fixes to tests should be back-ported. 

Thank you

Reinette