Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Date:   Sun, 26 Nov 2023 08:14:15 +0700
From:   Bagas Sanjaya <bagasdotme@gmail.com>
To:     Joey Gouly <joey.gouly@arm.com>,
        Linux Kernel Hardening <linux-hardening@vger.kernel.org>,
        Linux Networking <netdev@vger.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Linux Regressions <regression@lists.linux.dev>
Cc:     "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>, linux-kernel@vger.kernel.org,
        Kees Cook <keescook@chromium.org>,
        "Gustavo A. R. Silva" <gustavoars@kernel.org>,
        Bill Wendling <morbo@google.com>
Subject: Re: [BUG] Boot crash on v6.7-rc2
Message-ID: <ZWKb56E2kKngcWW5@archie.me>
References: <20231124102458.GB1503258@e124191.cambridge.arm.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
        protocol="application/pgp-signature"; boundary="5Hmfvs539k+85674"
Content-Disposition: inline
In-Reply-To: <20231124102458.GB1503258@e124191.cambridge.arm.com>
Precedence: bulk


--5Hmfvs539k+85674
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Fri, Nov 24, 2023 at 10:24:58AM +0000, Joey Gouly wrote:
> Hi all,
>=20
> I just hit a boot crash on v6.7-rc2 (arm64, FVP model):
>=20
> [    1.418845] Unable to handle kernel NULL pointer dereference at virtua=
l address 0000000000000517
> [    1.418855] Mem abort info:
> [    1.418860]   ESR =3D 0x0000000096000004
> [    1.418867]   EC =3D 0x25: DABT (current EL), IL =3D 32 bits
> [    1.418876]   SET =3D 0, FnV =3D 0
> [    1.418882]   EA =3D 0, S1PTW =3D 0
> [    1.418889]   FSC =3D 0x04: level 0 translation fault
> [    1.418897] Data abort info:
> [    1.418902]   ISV =3D 0, ISS =3D 0x00000004, ISS2 =3D 0x00000000
> [    1.418910]   CM =3D 0, WnR =3D 0, TnD =3D 0, TagAccess =3D 0
> [    1.418919]   GCS =3D 0, Overlay =3D 0, DirtyBit =3D 0, Xs =3D 0
> [    1.418928] user pgtable: 4k pages, 48-bit VAs, pgdp=3D0000000881835000
> [    1.418938] [0000000000000517] pgd=3D0000000000000000, p4d=3D000000000=
0000000
> [    1.418952] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
> [    1.418961] Modules linked in:
> [    1.418969] CPU: 0 PID: 8 Comm: kworker/0:0 Tainted: G                =
T  6.7.0-rc2-dirty #4191 40d10cdc812c74fd5dc5d91e2452ff6f1e5f4b4a
> [    1.418984] Hardware name: FVP Base RevC (DT)
> [    1.418992] Workqueue: mld mld_ifc_work
> [    1.419003] pstate: 101402005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTY=
PE=3D--)
> [    1.419016] pc : ___neigh_create+0x790/0x9c8
> [    1.419028] lr : ___neigh_create+0x270/0x9c8
> [    1.419041] sp : ffff8000800c3a20
> [    1.419048] x29: ffff8000800c3a20 x28: ffffd7c64c921078 x27: ffff00080=
188bd50
> [    1.419066] x26: ffff00080183a30c x25: ffff00080188bda0 x24: ffff00080=
183a300
> [    1.419084] x23: 0000000000000000 x22: 0000000000000010 x21: ffff00080=
188bcc0
> [    1.419102] x20: 0000000000000000 x19: ffff0008003ef000 x18: 000000000=
0000014
> [    1.419119] x17: 00000000cf0f2572 x16: 0000000080faa78d x15: 00000000b=
79921ac
> [    1.419137] x14: ffff00087ff332c0 x13: 1600000000000000 x12: 000000000=
00002ff
> [    1.419155] x11: 000000007c2c4dbd x10: 0000000000000003 x9 : 000000000=
0000000
> [    1.419172] x8 : ffff00080188bd80 x7 : 00000000be3df655 x6 : 00000000f=
1691d6f
> [    1.419190] x5 : 000000007c2c4dbd x4 : 0000000000000000 x3 : 000000008=
eb8ab5b
> [    1.419207] x2 : 000000000000050f x1 : 000000000000001d x0 : 000000000=
00002ff
> [    1.419225] Call trace:
> [    1.419230]  ___neigh_create+0x790/0x9c8
> [    1.419243]  __neigh_create+0x18/0x20
> [    1.419255]  ip6_finish_output2+0x5f8/0x8c4
> [    1.419267]  ip6_finish_output+0x1f0/0x258
> [    1.419279]  ip6_output+0x70/0x1cc
> [    1.419291]  NF_HOOK.constprop.0+0x4c/0xd8
> [    1.419302]  mld_sendpack+0x1b4/0x394
> [    1.419313]  mld_ifc_work+0x1d4/0x4b4
>=20
> I tracked it down to the following line in net/core/neighbour.c ___neigh_=
create:
> 	memcpy(n->primary_key, pkey, key_len);
>=20
> I did this by surrounding the memcpy with BUG():
> 	BUG_ON(n->tbl !=3D tbl);
> 	memcpy(n->primary_key, pkey, key_len);
> 	BUG_ON(n->tbl !=3D tbl);
>=20
> And it was crashing on the second one.
>=20
> Checking `struct neighbour`:
>=20
> 	struct neighbour {
> 		struct neighbour __rcu	*next;
> 		struct neigh_table	*tbl;
> 	.. fields ..
> 		u8			primary_key[0];
> 	} __randomize_layout;
>=20
> Due to the `__randomize_layout`, `primary_key` field is being placed befo=
re `tbl` (actually it's the same address since it's a 0 length array). That=
 means the memcpy() corrupts the tbl pointer.
>=20
> I think I just got unlucky with my CONFIG_RANDSTRUCT seed (I can provide =
it if needed), it doesn't look as if it's a new issue.
>=20
> I couldn't reproduce directly on v6.6 (the offsets for `tbl` and `primary=
_key` didn't overlap).
> However I tried changing the zero-length-array to a flexible one:
>=20
> 	+	DECLARE_FLEX_ARRAY(u8, primary_key);
> 	+	u8		primary_key[0];
>=20
> Then the field offsets ended up overlapping, and I also got the same cras=
h on v6.6.
>=20

Thanks for the well-handled regression report. I'm adding it to regzbot
for tracking:

#regzbot ^introduced: 1ee60356c2dca9
#regzbot title: Boot crash caused by true flexible array warning
#regzbot fix: neighbour: Fix __randomize_layout crash in struct neighbour

--=20
An old man doll... just what I always wanted! - Clara

--5Hmfvs539k+85674
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iHUEABYKAB0WIQSSYQ6Cy7oyFNCHrUH2uYlJVVFOowUCZWKb4wAKCRD2uYlJVVFO
o3v8AQCzt7rgIarKQqzj/B/2EhfxNDrdWb72QbQTgt3iu7rZfAEAw5hbmG8YNctp
YSjTe8uJ2MlH9+1YPx3hqxUFlr4Fogw=
=/ICY
-----END PGP SIGNATURE-----

--5Hmfvs539k+85674--