Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35;
Message-ID: <c25165891e0628b3a3a50cc528a86b2b2a65b7ae.camel@collabora.com>
Subject: Re: [RFC PATCH] drm/ci: add helper script update-xfails.py
From:   Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
To:     Helen Koike <helen.koike@collabora.com>,
        dri-devel@lists.freedesktop.org
Cc:     airlied@gmail.com, daniel@ffwll.ch, linux-kernel@vger.kernel.org,
        robdclark@gmail.com, daniel@fooishbar.org,
        vignesh.raman@collabora.com, jani.nikula@linux.intel.com,
        mripard@kernel.org, michel.daenzer@mailbox.org,
        quic_abhinavk@quicinc.com, quic_jesszhan@quicinc.com,
        david.heidelberg@collabora.com, guilherme.gallo@collabora.com
Date:   Thu, 28 Sep 2023 09:10:58 +0200
In-Reply-To: <785cbfee-f4d3-3d53-a3e6-2f9af8fa05fb@collabora.com>
References: <20230925195556.106090-1-helen.koike@collabora.com>
         <8f946d0a5632d580b593cf731091126d59feb77f.camel@collabora.com>
         <785cbfee-f4d3-3d53-a3e6-2f9af8fa05fb@collabora.com>
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.46.4-2 
MIME-Version: 1.0
Precedence: bulk

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Helen,

On Wed, 2023-09-27 at 19:28 -0300, Helen Koike wrote:
> > > +def get_unit_test_name_and_results(unit_test):
> > > +=C2=A0=C2=A0=C2=A0 if "Artifact results/failures.csv not found" in u=
nit_test:
> > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return None, None
> > > +=C2=A0=C2=A0=C2=A0 unit_test_name, unit_test_result =3D
> > > unit_test.strip().split(",")
> > > +=C2=A0=C2=A0=C2=A0 return unit_test_name, unit_test_result
> >=20
> > Suggestion: it is not managing empty lines or comments. By now,
> > there
> > aren't, but they could be found.
>=20
> Indeed.

Just add a new if statement to discard if the strings start with # or
strip the line and check the length. Perhaps we can think of other
assertions to sanitise the string.

> > Suggestion: Sometimes tests fails with different status ("Fail" to
> > "Crash" for example) and the expectations should be updated with
> > the
> > newer status.
>=20
> The status is only present in the fails and not in the flakes list,
> so I=20
> update it with add_unit_test_or_update_result_to_fails_if_present()=20
> function below, make sense?

Absolutely, sorry that I didn't see this was a process included in the
last if statement. If it is present in the fails' file (that includes
the test name and its state) you do exactly what's necessary: add if
not present, update if it was already in the file.

>=20
Regards,


- --=20
Sergi Blanch Torn=C3=A9
Senior Software Engineer

Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

-----BEGIN PGP SIGNATURE-----

iHUEARYIAB0WIQQwWRK68l+taJfhwqAto5bHyTm9RwUCZRUnAgAKCRAto5bHyTm9
R53NAP9T2OCiwbnEjv+H0CQg/eK1xGe7yS/3cqjaPFRvvZPp1wD/V1H9NuhpRR6M
8+QZgbsS/swSPdwYABtcz+75CKpuJwo=3D
=3DXRRO
-----END PGP SIGNATURE-----