Received-SPF: pass (google.com: domain of linux-kernel+bounces-143921-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: Andreas Dilger <adilger@dilger.ca>
Message-Id: <F8B008EE-22F4-4A3C-A80D-545104520D7D@dilger.ca>
Content-Type: multipart/signed;
 boundary="Apple-Mail=_C000EAC3-7C8A-4B66-8179-94F2E4BA6CB0";
 protocol="application/pgp-signature"; micalg=pgp-sha256
Precedence: bulk
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: [PATCH v2] jbd2: avoid mount failed when commit block is partial
 submitted
Date: Sat, 13 Apr 2024 17:27:54 -0600
In-Reply-To: <20240413013056.1830515-1-yebin10@huawei.com>
Cc: Theodore Ts'o <tytso@mit.edu>,
 Ext4 Developers List <linux-ext4@vger.kernel.org>,
 Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
 Jan Kara <jack@suse.cz>
To: Ye Bin <yebin10@huawei.com>
References: <20240413013056.1830515-1-yebin10@huawei.com>


--Apple-Mail=_C000EAC3-7C8A-4B66-8179-94F2E4BA6CB0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

On Apr 12, 2024, at 7:30 PM, Ye Bin <yebin10@huawei.com> wrote:
>=20
> We encountered a problem that the file system could not be mounted in
> the power-off scenario. The analysis of the file system mirror shows =
that
> only part of the data is written to the last commit block.
> The valid data of the commit block is concentrated in the first =
sector.
> However, the data of the entire block is involved in the checksum =
calculation.
> For different hardware, the minimum atomic unit may be different.
> If the checksum of a committed block is incorrect, clear the data =
except the
> 'commit_header' and then calculate the checksum. If the checkusm is =
correct,
> it is considered that the block is partially committed.

I think this is a clever solution to the problem, thanks for submitting
the patch.

> However, if there are valid description/revoke blocks, it is =
considered
> that the data is abnormal and the log replay is stopped.

It would be possible to use the r_count of records in the revoke block
to determine how much of the revoke block is unused and could be zeroed
out to recompute the partial checksum?  That should be relatively safe
to try, as long as r_count is itself checked to fit within the block
before the memory is zeroed, to avoid overflowing the temporary buffer =
size:

       r_count <=3D journal_revoke_records_per_block(journal)


It is open for discussion how much corruption should be allowed in the
journal, since it can be very destructive to copy corrupted blocks from
one place in the journal exactly into important metadata blocks across
the whole filesystem.  That said, the checksums *should* avoid this kind
of problem, and revoke blocks do not contain "metadata" that is copied
into the filesystem but only block numbers to skip.  It is "less bad" if
this was wrong, and having an incomplete journal replay due to minor
corruption that is causing boot failure is also a problem that should be
avoided if it can safely be done.


Additional comments inline below:

> Signed-off-by: Ye Bin <yebin10@huawei.com>
> ---
> fs/jbd2/recovery.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 48 insertions(+)
>=20
> diff --git a/fs/jbd2/recovery.c b/fs/jbd2/recovery.c
> index 1f7664984d6e..eb0e026f3109 100644
> --- a/fs/jbd2/recovery.c
> +++ b/fs/jbd2/recovery.c
> @@ -443,6 +443,27 @@ static int =
jbd2_commit_block_csum_verify(journal_t *j, void *buf)
> 	return provided =3D=3D cpu_to_be32(calculated);
> }
>=20
> +static bool jbd2_commit_block_csum_partial_verify(journal_t *j, void =
*buf)
> +{

(style) if this is named jbd2_commit_block_csum_verify_partial() then
it would sort together with jbd2_commit_block_csum_verify() and would
be easier to find with tag completion and grep in the future.

> +	struct commit_header *h;
> +	__be32 provided;
> +	__u32 calculated;
> +	void *tmpbuf;
> +
> +	tmpbuf =3D kzalloc(j->j_blocksize, GFP_KERNEL);
> +	if (!tmpbuf)
> +		return false;
> +
> +	memcpy(tmpbuf, buf, sizeof(struct commit_header));
> +	h =3D tmpbuf;
> +	provided =3D h->h_chksum[0];
> +	h->h_chksum[0] =3D 0;
> +	calculated =3D jbd2_chksum(j, j->j_csum_seed, tmpbuf, =
j->j_blocksize);
> +	kfree(tmpbuf);
> +
> +	return provided =3D=3D cpu_to_be32(calculated);
> +}
> +
> static int jbd2_block_tag_csum_verify(journal_t *j, =
journal_block_tag_t *tag,
> 				      journal_block_tag3_t *tag3,
> 				      void *buf, __u32 sequence)
> @@ -479,6 +500,7 @@ static int do_one_pass(journal_t *journal,
> 	int			descr_csum_size =3D 0;
> 	int			block_error =3D 0;
> 	bool			need_check_commit_time =3D false;
> +	bool                    has_partial_commit =3D false;
> 	__u64			last_trans_commit_time =3D 0, =
commit_time;
>=20
> 	/*
> @@ -590,6 +612,14 @@ static int do_one_pass(journal_t *journal,
> 					next_log_block);
> 			}
>=20
> +			if (pass =3D=3D PASS_SCAN && has_partial_commit) =
{
> +				pr_err("JBD2: Detect validate descriptor =
block %lu after incomplete commit block\n",

(minor) it isn't clear to me what this error message is trying to say?
Should it be something like "detected invalid descriptor block ..."?

> +				       next_log_block);
> +				err =3D -EFSBADCRC;
> +				brelse(bh);
> +				goto failed;
> +			}
> +
> 			/* If it is a valid descriptor block, replay it
> 			 * in pass REPLAY; if journal_checksums enabled, =
then
> 			 * calculate checksums in PASS_SCAN, otherwise,
> @@ -810,6 +840,14 @@ static int do_one_pass(journal_t *journal,
> 			if (pass =3D=3D PASS_SCAN &&
> 			    !jbd2_commit_block_csum_verify(journal,
> 							   bh->b_data)) =
{
> +				if =
(jbd2_commit_block_csum_partial_verify(

If this function was restructured a bit then the code flow would not =
need
to get more complex than it already is.  Something like:

			if (pass =3D=3D PASS_SCAN &&
			    !(jbd2_commit_block_csum_verify(journal,
							    bh->b_data) =
||
			      (has_partial_commit =3D
			       =
jbd2_commit_block_csum_verify_partial(journal,
							    =
bh->b_data))) {

The pr_notice() can be printed by =
jbd2_commit_block_csum_partial_verify()
if the partial checksum is valid, so no need for goto and chksum_ok =
label.

> +					pr_notice("JBD2: Find incomplete =
commit block in transaction %u block %lu\n",
> +						  next_commit_ID, =
next_log_block);
> +					has_partial_commit =3D true;
> +					goto chksum_ok;
> +				}
> 			chksum_error:
> 				if (commit_time < =
last_trans_commit_time)
> 					goto ignore_crc_mismatch;
> @@ -824,6 +862,7 @@ static int do_one_pass(journal_t *journal,
> 				}
> 			}
> 			if (pass =3D=3D PASS_SCAN) {
> +			chksum_ok:
> 				last_trans_commit_time =3D commit_time;
> 				head_block =3D next_log_block;
> 			}
> @@ -843,6 +882,15 @@ static int do_one_pass(journal_t *journal,
> 					  next_log_block);
> 				need_check_commit_time =3D true;
> 			}
> +
> +			if (pass =3D=3D PASS_SCAN && has_partial_commit) =
{
> +				pr_err("JBD2: Detect validate revoke =
block %lu after incomplete commit block\n",

Similarly, I find this error message hard to understand.  Maybe =
"detected invalid revoke block ..."?

> +				       next_log_block);
> +				err =3D -EFSBADCRC;
> +				brelse(bh);
> +				goto failed;
> +			}
> +
> 			/* If we aren't in the REVOKE pass, then we can
> 			 * just skip over this block. */
> 			if (pass !=3D PASS_REVOKE) {
> --
> 2.31.1
>=20


Cheers, Andreas


--Apple-Mail=_C000EAC3-7C8A-4B66-8179-94F2E4BA6CB0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIzBAEBCAAdFiEEDb73u6ZejP5ZMprvcqXauRfMH+AFAmYbFPoACgkQcqXauRfM
H+B/wQ/9EfXuI6o89DuxifUWUohCT2zbh6lphzZNJXBSYGdhBI6VrbbDlAnoyGMl
Rl1bNGq4kbypDkEeDjOefeN8Oydm/JacN355Dm7O+B3mABTRCkws3N5EOKyPo9pS
qtxyIGUqq2nQ09lkP5v5mt4DZTNH6d7cGm+axlWJUaVaVBs6APoJkNmTxqHcK7wc
GxAhNCHsE+Lgejgl8IGfBkkMLWMqyKksa3t/KpIl0x4djq7oihFGMMooHLAq6DAN
YyMWsin++Ady+PaNGNqffjNlL30kjT/xfSr8MGYHnGFC0dOfkgmWxNV51+gGWtXb
HzS41noOsE1MffwC66sempbfTYFuXCe7qIcCKVSu3MHyoE9CYHRHnJZqOiuyHVUH
fvyMDqI7t5WLLrATs3ROD7oCdMEAaxqiS7FXXDmH14Y9xa+53rYIZK80ZCLQ63aO
dBPyqOM+SU/BWn0AA45Ao90w0twSQExPuBT9v/mC67DE3GZVUZIl6l/tqNa86RxB
3/6Za+Bto7ueMUnsq6GFnUkeI+GbwY3FFoEgDG2OqSeDCSC78s8gyNR5xStqveue
TCq/lEgBDhPRCXIlooL7OGBpaMJi9VmRbVMMugU1WcLDSpMJKeD0xAUJwTijGdOf
WdMNboRVwioXLMcJ6iOl/wUFiUE/rVMl1zc8iVo192RjVGQcBus=
=mSJJ
-----END PGP SIGNATURE-----

--Apple-Mail=_C000EAC3-7C8A-4B66-8179-94F2E4BA6CB0--