Received-SPF: pass (google.com: domain of linux-kernel+bounces-13574-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <0a03664f3adba63600bddebae5b5fd299d4ebbf1.camel@sapience.com>
Subject: Re: 6.6.8 stable: crash in folio_mark_dirty
From: Genes Lists <lists@sapience.com>
To: Matthew Wilcox <willy@infradead.org>
Cc: linux-kernel@vger.kernel.org, Andrew Morton <akpm@linux-foundation.org>,
  linux-fsdevel@vger.kernel.org, linux-mm@kvack.org
Date: Sat, 30 Dec 2023 14:16:36 -0500
In-Reply-To: <ZZBbNm5RRSGEDlqk@casper.infradead.org>
References: <8bb29431064fc1f70a42edef75a8788dd4a0eecc.camel@sapience.com>
	 <ZZBbNm5RRSGEDlqk@casper.infradead.org>
Autocrypt: addr=lists@sapience.com; prefer-encrypt=mutual;
 keydata=mQENBFx5lqsBCACnoqPWNljBVS+eM+AiVpFn25csowDVqjLKEmCxJ0CWD/jj4h0opv2278TgyPrAny9Zw0Vrtb38ho6i0LqKE4mptFDxp/aRP9v+ywE8Ax/XCaOYeQ2u8eKKI4iLyjFdlzETpMfF4Xs5HKklugmhPxqcQs1S+4pzrQoch5z0aBi8A3MiGc9QVZEjfiK5NmRgibg3cHXMGcuODeijoYWDjcU/Y+yTyQ9MG4p95pUMPYBEVufR+Gaksk4xULurOlciphQC8oO9uV9X9f8CvEOBilM1eo98NL0egnOWg4hJgcBgHJ6vjalW7v+3gjoyLxbqzRDTjp63nMAF08C5RAY+ylCdABEBAAG0IEdlbmVzIExpc3RzIDxsaXN0c0BzYXBpZW5jZS5jb20+iQFUBBMBCAA+FiEEc/TBPYz5PhmLF5p8HgSPILb7+v0FAlx5lqwCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQHgSPILb7+v38fgf/T0t7OSNSi6AfSdZrt1Uy04mY7A9MqLhRn00gJMXQMiZRdlkFROUXLkLYLiZUVJRLCsXXDGHMl44EsWddwwTqx0cVGBjErT2AkKnN+o72Uc+fyI5XCMjgQafFIHXVcOBmbM9QjruL6zgbmPO8EtndHGsQ7pVhN3ysZxQfcN+wxdaufXXi8ZQzNvCMyE9dVnVWBpKRSxgN1L33SyjPnuJIdGCYz1ZIgfyUyB3AF/dK8ZQ05NW8TxFcsOF1lsku4WINNBMkd+WaSaxQ+lsCPhRiw+aL87HWccUI4wnNOdvo76f1Hork6Abd6S0UKlBDrknaNsFERvErSguAmqFMmIxhd5gzBF0mPRkWCSsGAQQB2kcPAQEHQMMxX5qfptJXZdr4Jm7Twv1ze/5Ob7HKQA6twZkcuUe8tC1NYWlsIExpc3RzIChMMCAyMDE5MDcxMCkgPGxpc3RzQHNhcGllb
	mNlLmNvbT6IlgQTFggAPgIbAQULCQgHAgYVCgkICwIEFgIDAQIeAQIXgBYhBOWDKFMXGxIEDrzjCnPnZQr+j8UABQJjyX6KBQkZb0RxAAoJEHPnZQr+j8UAHP8BAPw25oPdQzhl5ZdDltRimmUBkA5e3x4mzDikRgul/3pqAPsEEeRSMwZ1fz6122qPC0v4dLUpHl4D7GMa8732SJyAArgzBF0mPVgWCSsGAQQB2kcPAQEHQB+8yWRF3SUA9RiXZR3wH7WZJ3IH09ZiCLamrdomXGXmiPUEGBYIACYCGwIWIQTlgyhTFxsSBA684wpz52UK/o/FAAUCY8l+sQUJF44Q2QCBdiAEGRYIAB0WIQRByXNdQO2KDRJ2iXo5BdB0L6Ze2wUCXSY9WAAKCRA5BdB0L6Ze2+tHAQDDg1To6YZVGo2AorjOaX4H54E08Avh+qttW+8Ly8YwGAD/eEZvKAxdDj9SvNGqtJh+76Q/fso19FpN+CRYWc+9wgMJEHPnZQr+j8UAwo4A/184tkQWY8y8K1kumSuhiSl6tbXSHNNKA3g/dsxns0UgAQC1MAJo2MJzCrIV5pyVnmHfiNDFPctA2G1RiLo/9TPnB7g4BF0mPYwSCisGAQQBl1UBBQEBB0BAXvdh6o1D7dZqc5tLM3t5KMVbxdtbTY07YFCtLbCNCgMBCAeIfgQYFggAJgIbDBYhBOWDKFMXGxIEDrzjCnPnZQr+j8UABQJjyX6xBQkXjhClAAoJEHPnZQr+j8UAgpIA/0trNf9NrQszYGZ1cq27doPaDeWDbp5HwQFdCcxTuIIRAP9x4Ia8td4dbauGMK343i+xViosCxWwh49r1gqZULEtDg==
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.50.2 
Precedence: bulk
MIME-Version: 1.0

On Sat, 2023-12-30 at 18:02 +0000, Matthew Wilcox wrote:
>=20
> Thanks for the report.=C2=A0 Apologies, I'm on holiday until the middle o=
f
> the week so this will be extremely terse.
>=20

Enjoy =F0=9F=99=82

> >=20
> Dec 30 07:00:36 s6 kernel: CPU: 0 PID: 521524 Comm: rsync Not tainted
> So rsync is exiting.=C2=A0 Do you happen to know what rsync is doing?
> .

There are 2 rsyncs I can think of:

 (a) rsync from another server (s8) pushing files over the local
network to this machine (s6). rsync writes to the raid drives on s6.

s8 says the rsync completed successfully at 3:04 am (about 4 hours
prior to this error at 7.00 am).=20

 (b) There is also a script running inotify which uses rsync to keep
the spare root drive sync'ed. System had update at 5:48 am of a few
packages, and that would have caused an rsync from root on nvme to
sapre on sdg. Most likely this is this one that triggered around 7 am.

  This one runs:=20

    /usr/bin/rsync --open-noatime --no-specials --delete --atimes -
axHAX --times  <src> <dst>


> t looks llike rsync has a page from the block device mmaped?=C2=A0 I'll
> have
> to investigate this properly when I'm back.=C2=A0 If you haven't heard
> from
> me in a week, please ping me.

Thank you.

>=20
> (I don't think I caused this, but I think I stand a fighting chance
> of
> tracking down what the problem is, just not right now).


This may or may not be related, but this same machine crashed during an
rsync same as (a) above (i.e. s8 pushing files to the raid6 disks on
s6) about 3 weeks ago - then was on 6.6.4 kernel. In that case the
error was in md code.

https://lore.kernel.org/lkml/e2d47b6c-3420-4785-8e04-e5f217d09a46@leemhuis.=
info/T/

Thank you again,


gene