Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33;
From:   Wu Bo <bo.wu@vivo.com>
To:     Alasdair Kergon <agk@redhat.com>,
        Mike Snitzer <snitzer@kernel.org>,
        Mikulas Patocka <mpatocka@redhat.com>
Cc:     dm-devel@lists.linux.dev, linux-kernel@vger.kernel.org,
        Wu Bo <wubo.oduw@gmail.com>, Wu Bo <bo.wu@vivo.com>
Subject: [PATCH 2/2] dm verity: don't verity if readahead failed
Date:   Tue, 21 Nov 2023 01:55:29 -0700
Message-Id: <e18ec7ad7449f2aba885b93467005848745f4853.1700555778.git.bo.wu@vivo.com>
In-Reply-To: <cover.1700555778.git.bo.wu@vivo.com>
References: <cover.1700555778.git.bo.wu@vivo.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?kHQxOKA//8tBgx5jtF6Nh5l34U+UqkCdi3+Hl95+xS5aLZ7LWjUCsSPRAMRc?=
 =?us-ascii?Q?fkShwsMuz5H20H4KjP3Gcn93q4uLXdCH9DPbpdWmp7lHw4LWPk2b0fm2QJtv?=
 =?us-ascii?Q?zD9mjSkQ32c6Mcs0lnbmQgHULXlDXmIRVL4lHmrCOXO2mRnJy6lnLVM1sbwF?=
 =?us-ascii?Q?wPD0JWz+SAyD+j/mIrjao/tVeJtw9TcXsznG0kKokJFVdtzRTJXi+L5wZw/z?=
 =?us-ascii?Q?RhCtdbb+XhHD8GSE1roBWduVn76Mmz2VKLQp1jxf9wvwWwXqoF4+cA9ibTYv?=
 =?us-ascii?Q?FmqpSdRLJdYa4oF4d1RYb6aytVrjNiAmYnO28tqPQrgYneZJalxQ1mr6vWt3?=
 =?us-ascii?Q?AMmYePXma896htF9i1SjaxAhmchEJeoj5hH2OWz88BE17kRJtI73pFI1yiff?=
 =?us-ascii?Q?QVydgbYGiBuwHnZgcVfwY3/DcqUFljnGpU2M/VaEZ7B40s0puD5cXsyDA4gF?=
 =?us-ascii?Q?ApVgPsuHsoCFaDEBymn/oArDNv2G8+VkguhaqQg+QRTPZ5Kpw/kILYJs86UI?=
 =?us-ascii?Q?yq8JWAKGhK2A3JWcC073NKOGs5VoEu2BAgV8pnO+bWNkbfFo3mYO/DzO2r6M?=
 =?us-ascii?Q?CEQ9I0t3ytICNs6JaF00jC93qjUzFvSdS3HcyWUPYSqhtyBOPROPt0xTw8wN?=
 =?us-ascii?Q?nD2daCjxhw8oirRtzMb/ArCtEk2LtyfTYWVPp8UkmWYT0dfAk671rDQACpBw?=
 =?us-ascii?Q?vTr4+KVpcC2A3xwAWpACwrcZX631jDEMf3LFq6vdDA10cLtXt7V3VAHxZMda?=
 =?us-ascii?Q?bN1lNmz/mdBslq6OOVzkOp3KjZ0770bvZDIq6CQssY6hagLQXyTzkFCO20jb?=
 =?us-ascii?Q?HYgtyuX5a3rAr7zKHlyQBAm5T+sDq2/AVC3aZiLZeGJZQpi62UrowRZnygIp?=
 =?us-ascii?Q?XqUd6HanZqW351FZ/tS4gHViQJhZ6PK85N+cxWfONpuj9aL5wQ6JsMZnAM+p?=
 =?us-ascii?Q?OXlosaW+AhH3CxpAgRFrTzaayv53cxaRrIQrLECI073QxpZ5ViVQW9tksVuG?=
 =?us-ascii?Q?nDx2KWb4QVjkzTxROoZG/8con0WEYBLaYi4eQCyyZWQr40ikrZ5dxm2iQquS?=
 =?us-ascii?Q?66/1DD6KLpgshaOnM4n3pjQp6mwXfIRBkD34nKYFmejWtatdhXWNl5LjK1Lw?=
 =?us-ascii?Q?aAA85Dd0CokY4V50tOlOvB0vvNwPkJON48K5IEvyUHeAKc7SyqFeo6E8Bod1?=
 =?us-ascii?Q?36oxaFAiwvUVCOA+AJGrtyadRCkETxIYChMno+hwQ63/HXKD4eQOoKldt6Zq?=
 =?us-ascii?Q?YNjIUdnkFOqA9uXQy8iVixpZ1iAMVvWq3OsCHkNSj76WzL1ANNDD4ffA5M9i?=
 =?us-ascii?Q?pFcO4aQj3D+QBTAixR7VDL81j7vn3juUgDKslxKtLEpztzLV1efEPlhGLUpt?=
 =?us-ascii?Q?KvKTVQj2xBpOnlohJ84A020swExpsW9CgPruddv1Aoi9qwe8EhZo9i0ouJLj?=
 =?us-ascii?Q?81UZUYn7ZjBrkVu3HyvOjiE+cdCLcDd04ZatMZlBnSSuuxAmhZQFtLJk1kJC?=
 =?us-ascii?Q?Am78i+yjx4x0cR3IoFiiIVrYZGKN8pUWykP3lEU88nZQKEpUDHLgwnpblnSu?=
 =?us-ascii?Q?KWXzEpj2R9/E8YcTcB7PqjuG4Mb8quhoOWu1bivd?=
X-OriginatorOrg: vivo.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5d432032-10af-4a18-d72e-08dbea6e6d88
X-MS-Exchange-CrossTenant-AuthSource: PS2PR06MB3509.apcprd06.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Nov 2023 08:47:01.0043
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 923e42dc-48d5-4cbe-b582-1a797a6412ed
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: S0jWMWJjjtI5Yfp/n9IkXjl6gxZ4TG9i+WCBbtZG9tUxVI7owFufb3B1AOfrpOjs7PlE/7YO81PzSQZwzp0yNw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYZPR06MB4127
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Tue, 21 Nov 2023 00:47:45 -0800 (PST)

We found an issue under Android OTA scenario that many BIOs have to do
FEC where the data under dm-verity is 100% complete and no corruption.

Android OTA has many dm-block layers, from upper to lower:
dm-verity
dm-snapshot
dm-origin & dm-cow
dm-linear
ufs

Dm tables have to change 2 times during Android OTA merging process.
When doing table change, the dm-snapshot will be suspended for a while.
During this interval, we found there are many readahead IOs are
submitted to dm_verity from filesystem. Then the kverity works are busy
doing FEC process which cost too much time to finish dm-verity IO. And
cause system stuck.

We add some debug log and find that each readahead IO need around 10s to
finish when this situation occurred. Because here has a IO
amplification:

dm-snapshot suspend
erofs_readahead     // 300+ io is submitted
	dm_submit_bio (dm_verity)
		dm_submit_bio (dm_snapshot)
		bio return EIO
		bio got nothing, it's empty
	verity_end_io
	verity_verify_io
	forloop range(0, io->n_blocks)    // each io->nblocks ~= 20
		verity_fec_decode
		fec_decode_rsb
		fec_read_bufs
		forloop range(0, v->fec->rsn) // v->fec->rsn = 253
			new_read
			submit_bio (dm_snapshot)
		end loop
	end loop
dm-snapshot resume

Readahead BIO got nothing during dm-snapshot suspended. So all of them
will do FEC.
Each readahead BIO need to do io->n_blocks ~= 20 times verify.
Each block need to do fec, and every block need to do v->fec->rsn = 253
times read.
So during the suspend interval(~200ms), 300 readahead BIO make
300*20*253 IOs on dm-snapshot.

As readahead IO is not required by user space, and to fix this issue,
I think it would be better to pass it to upper layer to handle it.

Signed-off-by: Wu Bo <bo.wu@vivo.com>
---
 drivers/md/dm-verity-target.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
index 42b2483eb08c..d242e50ec869 100644
--- a/drivers/md/dm-verity-target.c
+++ b/drivers/md/dm-verity-target.c
@@ -668,7 +668,9 @@ static void verity_end_io(struct bio *bio)
 
 	verity_fec_init_io(io);
 	if (bio->bi_status &&
-	    (!verity_fec_is_enabled(io->v) || verity_is_system_shutting_down())) {
+	    (!verity_fec_is_enabled(io->v) ||
+	     verity_is_system_shutting_down() ||
+	     (bio->bi_opf & REQ_RAHEAD))) {
 		verity_finish_io(io, bio->bi_status);
 		return;
 	}
-- 
2.25.1