Date:   Thu, 9 Mar 2023 08:32:15 +0100
From:   Johan Hovold <johan@kernel.org>
To:     Cyril Brulebois <kibi@debian.org>
Cc:     Johan Hovold <johan+linaro@kernel.org>,
        Marc Zyngier <maz@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>, x86@kernel.org,
        platform-driver-x86@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org,
        linux-kernel@vger.kernel.org, stable@vger.kernel.org,
        Dmitry Torokhov <dtor@chromium.org>,
        Jon Hunter <jonathanh@nvidia.com>,
        Hsin-Yi Wang <hsinyi@chromium.org>,
        Mark-PK Tsai <mark-pk.tsai@mediatek.com>
Subject: Re: [PATCH v6 06/20] irqdomain: Fix mapping-creation race
Message-ID: <ZAmLf3YKxI5fE9w5@hovoldconsulting.com>
References: <20230213104302.17307-1-johan+linaro@kernel.org>
 <20230213104302.17307-7-johan+linaro@kernel.org>
 <20230308144105.di552lbogqv2s7fk@mraw.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
        protocol="application/pgp-signature"; boundary="1pYx6IuXSbcDEQEb"
Content-Disposition: inline
In-Reply-To: <20230308144105.di552lbogqv2s7fk@mraw.org>
Precedence: bulk


--1pYx6IuXSbcDEQEb
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Wed, Mar 08, 2023 at 03:41:05PM +0100, Cyril Brulebois wrote:
> Hi Johan,
>=20
> And thanks so much for this patch series.
>=20
> Johan Hovold <johan+linaro@kernel.org> (2023-02-13):
> > Parallel probing of devices that share interrupts (e.g. when a driver
> > uses asynchronous probing) can currently result in two mappings for the
> > same hardware interrupt to be created due to missing serialisation.
> >=20
> > Make sure to hold the irq_domain_mutex when creating mappings so that
> > looking for an existing mapping before creating a new one is done
> > atomically.
>=20
> Just for information: This patch fixes a long-standing regression
> regarding Raspberry Pi devices, which have been failing to boot (at
> least reliably) due to MMC timeouts for a long while; I think that
> started between v5.17 and v5.19, but I couldn't bisect at the time
> (I was already chasing some other regression).
>=20
> Example bug report:
>   https://bugs.debian.org/1019700
>=20
> Before trying to pinpoint when the regression appeared, I've checked
> these versions, with a Debian testing userspace as of 2023-03-07:
>  - v6.1.12: affected.
>  - v6.2: affected.
>  - v6.3-rc1: not affected.
>=20
> A bisect between v6.2 and v6.3-rc1 led me to this patch specifically.
> Seeing how it's part of a patch series, and how previous patches are
> preliminary ones, I've checked that cherry-picking the first 6 patches
> on top of v6.1.15 indeed fixes the problem there too, and it does
> (git cherry-pick v6.2-rc4..601363cc08da25747feb87c55573dd54de91d66a).

That's good to hear, thanks for reporting.

> With the following systems:
>  - Pi 4 B, using external storage (SD card),
>  - CM4 Lite on CM4 IO Board, using external storage (SD card),
>  - CM4 on CM4 IO Board, using internal storage (eMMC),
>=20
> I've been able to verify that v6.1.12 (baseline in Debian testing)
> triggers this MMC timeout issue, while v6.1.15 + the aforementioned
> range of cherry-picked commits no longer triggers this issue.
>=20
> (Methodology: cold boot then reboot 20 times, monitoring via serial
> console to keep HDMI output of the equation; affected systems stop
> booting after 1-4 boots; unaffected systems boot and reboot just fine
> all the time.)
>=20
>=20
> This looks like a critical bugfix for Raspberry Pi users.
>=20
> Seeing the stable@ mention is about 4.8, I suppose this is going to be
> considered for a wide range of kernels already=E2=80=A6 but I'm happy to =
dig
> into this further to pinpoint when the regression appeared, if that's
> helpful.

I took a quick look at the rpi-4 devicetree and it looks like the emmc
indeed is sharing an irq line with another device which should make is
susceptible to the mapping race.

The corresponding drivers also use asynchronous probing since commits

	7320915c8861 ("mmc: Set PROBE_PREFER_ASYNCHRONOUS for drivers that existed=
 in v4.14")
	21b2cec61c04 ("mmc: Set PROBE_PREFER_ASYNCHRONOUS for drivers that existed=
 in v4.4"

that went into 5.10. So the issue should have been there for longer but
perhaps only manifested itself around 5.17 due to changes in timing.

Johan

--1pYx6IuXSbcDEQEb
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iHUEABYIAB0WIQQHbPq+cpGvN/peuzMLxc3C7H1lCAUCZAmLeAAKCRALxc3C7H1l
CG9UAP9ztoxOSXxHCUZXg63cjsFhvy7S6lrLVZ5RvHvjmjv2CwD/XUbPSo+2Fd5H
Z4bA7fuXNrGMT4z40fDurbv+sBWmogw=
=aPds
-----END PGP SIGNATURE-----

--1pYx6IuXSbcDEQEb--