Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 30 Sep 2022 11:11:56 +0200
From:   =?utf-8?Q?Micha=C5=82?= Winiarski <michal.winiarski@intel.com>
To:     David Gow <davidgow@google.com>
CC:     =?utf-8?B?TWHDrXJh?= Canal <mcanal@igalia.com>,
        David Airlie <airlied@linux.ie>,
        Daniel Vetter <daniel@ffwll.ch>, Melissa Wen <mwen@igalia.com>,
        =?utf-8?B?QW5kcsOp?= Almeida <andrealmeid@igalia.com>,
        Javier Martinez Canillas <javierm@redhat.com>,
        Daniel Latypov <dlatypov@google.com>,
        Arthur Grillo <arthur.grillo@usp.br>,
        Isabella Basso <isabbasso@riseup.net>,
        <magalilemes00@gmail.com>,
        Tales Aparecida <tales.aparecida@gmail.com>,
        <dri-devel@lists.freedesktop.org>,
        "Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v2 2/2] drm/tests: Split
 drm_test_dp_mst_sideband_msg_req_decode into parameterized tests
Message-ID: <20220930091156.zt7w74g5axfozlts@nostramo>
References: <20220927221206.55930-1-mcanal@igalia.com>
 <20220927221206.55930-2-mcanal@igalia.com>
 <20220929223333.vh6wy45mfx6kccds@nostramo>
 <CABVgOSkx7KYNRKCN5h=37zQGR0qu+BDCb6cQeqbCwX8UxC3knw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CABVgOSkx7KYNRKCN5h=37zQGR0qu+BDCb6cQeqbCwX8UxC3knw@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?WE90NWN0enpadDdPaFhZUnBpT2s1UHZtcEg5aWEwbHVTK3NQeEpadGg1c0li?=
 =?utf-8?B?WkhQVDM1TVo0QnFSWjVsNmVBYlk3YSt4dy9STnMyQVBwVmk5SHo0OE43aFlQ?=
 =?utf-8?B?eUJGZFZ5cm5IdVF1bjVHMkVQZkJaZnpJbFdxOWZDNG9xNXpYckRKWG5SZ0kx?=
 =?utf-8?B?YjRXZWQzQlFGcXhObUllYXp4dzMwU091Tk1TdVczUTRPTlBnckUxUzNad3dB?=
 =?utf-8?B?TWdkU2ZFWUZGTHgxUVk4c0Q4RGFRVllRcmFOYjBTaTRWMzZ3ejNNcHd2dkNH?=
 =?utf-8?B?MlY1dHI4OVF4N0VFMWQ1UkNUUU16RDJxeWN4L2tiSERiOFZuYUR3b3J5b2VT?=
 =?utf-8?B?ZzhZRVFwQ3NhZnpnRjl5cEovUXVvYlpPR241Mk9rdzZRSFFxdGpndFN4Z1lJ?=
 =?utf-8?B?R0hkaEppYXg5blBtRjN0VXBYUVV1dlA4bXpvTVpvTWswMFF4Wkp6NFZxZHVs?=
 =?utf-8?B?dnliSW9KM2lCcHJLS2k3TUpsdTRYc296Um40UHZBMWRoZi9lQWpQdEgrRU5j?=
 =?utf-8?B?QndsR2JhYVVXaUNwWnVocDV0aWtRa0VKMGVCNEhJd0luOVllZm5TR2pJck1h?=
 =?utf-8?B?a3lGKzVlZld3WTdjeFRUcmxOT1ZwU3dVa2FVVUZwU0ZZZEpzYU5xeVVaelEr?=
 =?utf-8?B?bTV2QXFtUm5Rdmd3MEwvVm1zMFBjNzd0WC93S0ZkSHU0dHNCMjJMTk5hM3NC?=
 =?utf-8?B?eDZlL1JucU5xU2FsSk1CRnM5am5xa1VpTHBFUWlwRXhJK3dJbkhKQzl5R2dl?=
 =?utf-8?B?Q3RrT1g1OXBETy9xT2VLeDcvRkxianZDMTAzWGU2bFVDQm5WR0wyTUtLK052?=
 =?utf-8?B?SkFBTElKM0NJc2JneS8xN2h6M2NObXV1YUVVWkdENFdaN0ttT1VTVllXUVFJ?=
 =?utf-8?B?Vm8vYVFaeVR4ZDM2YWxVWmkzdy9UanZ4bnZSRmxMMVVEWS9GblF1aVBweVZw?=
 =?utf-8?B?Qk0zQkUzMCtIQUhLc2drdDQ0MFdaY2xDTTVwMHQreE1zcHZ0Ky8rajRwU2tB?=
 =?utf-8?B?U3pFWnhOU3hjVWpxKzNCWkFjWWd2VmhUcEdtNmdLR1p2REgyOGE5WWloV0dn?=
 =?utf-8?B?UFp2NlU4MVRTRXpFbFFyajdmem4yOU9sYXNUaWdtMG5WMjVENzJPMGZqOEVh?=
 =?utf-8?B?RWxLcC9qdjY4aGpXQmozU3kyNjV2UXdDWW1HVG13SWJwekpvT01CYjczRlN3?=
 =?utf-8?B?bE1ETFd4ZzMyWGpGamdEYVYxdkdkMC9yODNhU0xuamN0YXduNjUzdGx0TUgr?=
 =?utf-8?B?dVpnbkdETFZ3ZS9EK0dtZ0NHYXNKdCtQYUxHdk5vVGw2bEtLUDVlV0QxWncv?=
 =?utf-8?B?blpMTHZDeDhma2pSRWp2eXNNanoyOGRsSDJqcitnWlRKNUx0QlpPOEkwdkhI?=
 =?utf-8?B?cGNXb1JaYlRydHNDMThNZWp6MnlSaklYM0xobFBPWDQ2ZnplcFZZeUEwM25y?=
 =?utf-8?B?eXEyNkorRTU2RzJrQytLaVVSRWplNUwzd3Q4cmM2NFBXaWFzME0rbUJvTkVw?=
 =?utf-8?B?VjUyVDNDR21jRGdRVzJTUHlTM2QvYmVwNGdWOFFWRDRzcXFXUUR5VVZHTytn?=
 =?utf-8?B?ejF6NkJqcndjVzdVQThINGNmMGczVkhwcVB1NDVsZC9POGgxUUhGUEsxSjZk?=
 =?utf-8?B?Z1lreVRtam9LYXdlUzVXOE9KTmdxR3N1MjlhQzNGLzB3dEdtQzhsa0g4WVdF?=
 =?utf-8?B?VVFKckVoZ1pkcHVPL3JWbzR3T1oxTTlDMVBqd01xSWVJaTFMaWRJYWVCaW4x?=
 =?utf-8?B?akg2YUdWeUJjYXJrNXZHTy9PR2RJb1NrVEI5cUpkdEE5RGY1NkdseU9oalpi?=
 =?utf-8?B?TGcxVndFaDdORUtwT3B3S0pwVlBDNy9Da2c5dncwS0txb1QyNG1FV25xTlRi?=
 =?utf-8?B?RVh2UVhPc2VoVk9nR2ZJZkRhVkl0N2xYYUVoZG53ZEdJV0lhMUhLVnZTd2pi?=
 =?utf-8?B?M2V6TnBOcUhwc0FXaWxTTzA5SVhOM0pNMXJTajl4MlBwRXlqOXRpR08zTVVn?=
 =?utf-8?B?MTV6bWJRaUVTUHkxWmJydWw3Mm9uVmlUakxSK0l5SFNwRnZ4T214RUFaU2xQ?=
 =?utf-8?B?S2NUZEtObi8zYll4ZkRFbTkwUEdsa21tVjNWR1FTM1hKUmRzMFJ3dVR3dlhW?=
 =?utf-8?B?ZVpqR0xvMTBWL1ovTVpENCtRamFvR2hoWW45RTRpd2ZXamduVU84QlppRDVm?=
 =?utf-8?B?eUE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 79810f82-fc74-4386-82d4-08daa2c3d504
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5373.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Sep 2022 09:12:00.5636
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: B0+Gx4P/cBHHvXz42uY6FcpRw6Af0kN+UOsySvBOhZXzM7B8LDUGNgOE/3lhtjY61Gm5HMCAKDWezD3wDs9LauufH5YYNqxpbs9H/flZZy0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB6695
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Sep 30, 2022 at 02:50:43PM +0800, David Gow wrote:
> On Fri, Sep 30, 2022 at 6:33 AM Michał Winiarski
> <michal.winiarski@intel.com> wrote:
> >
> > On Tue, Sep 27, 2022 at 07:12:06PM -0300, Maíra Canal wrote:
> > > The drm_test_dp_mst_sideband_msg_req_decode repeats the same test
> > > structure with different parameters. This could be better represented
> > > by parameterized tests, provided by KUnit.
> > >
> > > In order to convert the tests to parameterized tests, the test case for
> > > the client ID was changed: instead of using get_random_bytes to generate
> > > the client ID, the client ID is now hardcoded in the test case.
> >
> > Generally "random" usage is not incompatible with parameterized tests, we can
> > create parameterized tests that use random data.
> > The idea is to pass a function that generates the actual param (where we have a
> > pointer to function as one of the members in "params" struct).
> >
> > For example, see "random_dp_query_enc_client_id" usage here:
> > https://lore.kernel.org/dri-devel/20220117232259.180459-7-michal.winiarski@intel.com/
> >
> > In this case, we just compare data going in with data going out (and the data
> > itself is not transformed in any way), so it doesn't really matter for coverage
> > and we can hardcode.
> >
> > -Michał
> 
> FWIW, while the uses of randomness in DRM tests so far haven't
> concerned me much, I think we'll eventually want to have some way of
> ensuring the inputs to tests are deterministic.
> 
> My thoughts are that (at some point) we'll add a kunit_random()
> function or similar, which will use a pseudorandom number generator
> which can be set to a deterministic seed before each test case. That
> way, there'd be a way to reproduce an error easily if it occurred. (Of
> course, there'd be a way of setting different or random seeds to
> preserve the extra coverage you'd otherwise get.)

That's exactly what DRM tests do (well... most DRM tests, this one being the
exception, and those other tests also seem to have lost a printk with seed value
after being refactored into kunit).
See the usage of DRM_RND_STATE in drm_mm_test and drm_buddy_test.
Having kunit_random() would definitely be useful and let us remove bunch of
boilerplate from the tests, but it doesn't prevent using reproducible random
data in existing tests.

> I don't think this is something worth holding up or changing existing
> tests at the moment, but having tests behave deterministically is
> definitely desirable, so +1 to avoiding get_random_bytes() if it's not
> giving you any real benefit.

Yeah - all I was refering to in my previous message was the wording of the
commit message. We're just removing it because it is desirable in this
particular case, not because of the fact that the test is now parameterized and
that's somehow preventing get_random_bytes() usage.

-Michał

> We've also had a few requests in the past for being able to pass in a
> custom set of parameters from userspace, which opens up some other
> interesting possibilities, though it's not a priority at the moment.
> 
> Cheers,
> -- David