Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35;
From:   Wu Bo <bo.wu@vivo.com>
To:     Alasdair Kergon <agk@redhat.com>,
        Mike Snitzer <snitzer@kernel.org>,
        Mikulas Patocka <mpatocka@redhat.com>
Cc:     dm-devel@lists.linux.dev, linux-kernel@vger.kernel.org,
        Wu Bo <wubo.oduw@gmail.com>,
        Eric Biggers <ebiggers@kernel.org>, Wu Bo <bo.wu@vivo.com>,
        stable@vger.kernel.org
Subject: [PATCH v2 2/2] dm verity: don't verity if readahead failed
Date:   Tue, 21 Nov 2023 20:51:50 -0700
Message-Id: <b23a4fc8baba99010c16059a236d2f72087199a1.1700623691.git.bo.wu@vivo.com>
In-Reply-To: <cover.1700623691.git.bo.wu@vivo.com>
References: <cover.1700623691.git.bo.wu@vivo.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ptU2ksE2HAoQme9mEZ77rdVetDbehL6GAPQA8IdWnxCCvJZ4dEDycMSm2Lt8?=
 =?us-ascii?Q?3aichP6rRFaLkk0JHoMgbdTXUpitSuqyHAStQdsdrdfHpcr6kbHBdhoZ1Xs1?=
 =?us-ascii?Q?k1gozUDPr87vZyqI+WeF1Rd5seQD0j6RXGMFA6RHgeGXgpllEVfBzsSGeWD9?=
 =?us-ascii?Q?xgpNjh2YQxM+o5YFin3XhF4fX2yybIgbFBM7Dar4Y0JpbtCIZ37Rwt6OiJHy?=
 =?us-ascii?Q?elz6yYquuRetiWBR/h3fvv8zOUByhtVnGijTYHgjcByr/vzQNlBgf7tWGXMv?=
 =?us-ascii?Q?0I44lrAoPxd4xuhDyM/KBzRg/Txc9iPdQkVuB0GwM3OEhg9Fh2kbHvUQ1UgD?=
 =?us-ascii?Q?KJcMnaHma/SA5lX7i+NUs5NBLJCIwprs4VW4NA2Y/Fa+e0Bk2bEu/moWo7UW?=
 =?us-ascii?Q?ASGiliVVzZUnVS7Z1R/XNZcw6g7wH3B+e47WqNYaW5zUZzRa3d1K69xuqFul?=
 =?us-ascii?Q?euZeqZAOhRrxad+EPBKF1bipsv0aaBSP1Okr/asXPp4fLdHNqUVBLQAj8sZr?=
 =?us-ascii?Q?2wOJ48ose4vg/FY/GMgn8iAkiTvctHTt52edZ5YLXTJMZlOJNt6Tni5jLSOd?=
 =?us-ascii?Q?BE5xYsTOPytCcA2Gzsvacm3FggRL8xGkenrsjJ3F0OaCzhrDlqK7aKxwIe8P?=
 =?us-ascii?Q?D3c0oEK9lCG7AIeogcWPVDaecE13pJnxV8jaIpyKZ9Ovw5Tw2rPKrRGOv1ng?=
 =?us-ascii?Q?uA+mWS5+GhAyb+ctcUDbvLJIEm07G2BHcc1CWe12M+gXxolfWS4eZaAAm14t?=
 =?us-ascii?Q?8cQ0KaW7HK7U7hzsEie1oWDcxeD/efj2G2toV2Shf1R2+9UZjUs3+R79eVuI?=
 =?us-ascii?Q?hnbiGRfWFXoknM1D248m9kQeobZiLTeE0NTi64cdIdeFtEmddPvx9vZDFTlE?=
 =?us-ascii?Q?5k0jPX06SgD7RQREaz5lZpqs81eRulPq8g/HMb5F+8oQpbKt+1epCws3D+Xi?=
 =?us-ascii?Q?D4wU1tidKSVV1kIdJQ7QyMyMcL3XUjD7uLajG+CNevqpl26zuwhDspvDyn9C?=
 =?us-ascii?Q?/iIs5JP7YzMAaIkOtJArGt75yGEVQOfov1XAq9RKCErBtyK726HbK9qiJ75g?=
 =?us-ascii?Q?KCikSL7p7EuOUghhk4NUcpMRcXRptbM+nD8OYPk8pMFvSt7yeGsflVZksj7v?=
 =?us-ascii?Q?BeIrmZg++KYcXCXJxeYgoyBD52bnOpsjblFYHUFXyxitC9iMcpUo4xZTcy8b?=
 =?us-ascii?Q?jtyjOEmzH97iNDtVxH0kyF+mBlvTndFEIGfMzlzOFlvDVVSG8lkxDUJ13saD?=
 =?us-ascii?Q?8m89Rd/9MmQ1oBIEaEckrsdugIm8qwQ6yVcG8xBNrnUqIWKr895OM259n5kh?=
 =?us-ascii?Q?HnLNLASGlOZ7KuCSWhZ8tsHttloKOL/3aQIrhIeUm+wT2pHFG0rttN0Mh90F?=
 =?us-ascii?Q?yEe8LJAp28mY2DLMkQ7By63+tAT5BreuCVbfn6ndzXR9tFzDQ0WBFRrL/lyZ?=
 =?us-ascii?Q?Zi64Uhlhzd80PPkJ70Iokfs6oEg3JdVjfWmTQtvQrYh/URmNGRxdrz0jZgxS?=
 =?us-ascii?Q?KfYRaJmritrjrBCWOHhTFpMFnCalUQHpJiNoSj8LCZ/4tDS2HuJ+F2CQzKmf?=
 =?us-ascii?Q?zL9gQWGYvO/py6njVkpVmFLZMdfl+bBzcLWTGcFe?=
X-OriginatorOrg: vivo.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 40514c59-c666-4e35-6059-08dbeb0d3966
X-MS-Exchange-CrossTenant-AuthSource: PS2PR06MB3509.apcprd06.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Nov 2023 03:43:43.4965
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 923e42dc-48d5-4cbe-b582-1a797a6412ed
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: S4U1xo8kz7MxwC+C4P9D6TMi5qpVWI9sp55pn/W8DXbjm00E/nCI2svgZEGRSNDAL/INf/SYuXt2vEV66nw5vw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: KL1PR0601MB3891
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Tue, 21 Nov 2023 19:44:17 -0800 (PST)

We found an issue under Android OTA scenario that many BIOs have to do
FEC where the data under dm-verity is 100% complete and no corruption.

Android OTA has many dm-block layers, from upper to lower:
dm-verity
dm-snapshot
dm-origin & dm-cow
dm-linear
ufs

Dm tables have to change 2 times during Android OTA merging process.
When doing table change, the dm-snapshot will be suspended for a while.
During this interval, we found there are many readahead IOs are
submitted to dm_verity from filesystem. Then the kverity works are busy
doing FEC process which cost too much time to finish dm-verity IO. And
cause system stuck.

We add some debug log and find that each readahead IO need around 10s to
finish when this situation occurred. Because here has a IO
amplification:

dm-snapshot suspend
erofs_readahead     // 300+ io is submitted
	dm_submit_bio (dm_verity)
		dm_submit_bio (dm_snapshot)
		bio return EIO
		bio got nothing, it's empty
	verity_end_io
	verity_verify_io
	forloop range(0, io->n_blocks)    // each io->nblocks ~= 20
		verity_fec_decode
		fec_decode_rsb
		fec_read_bufs
		forloop range(0, v->fec->rsn) // v->fec->rsn = 253
			new_read
			submit_bio (dm_snapshot)
		end loop
	end loop
dm-snapshot resume

Readahead BIO got nothing during dm-snapshot suspended. So all of them
will do FEC.
Each readahead BIO need to do io->n_blocks ~= 20 times verify.
Each block need to do fec, and every block need to do v->fec->rsn = 253
times read.
So during the suspend interval(~200ms), 300 readahead BIO make
300*20*253 IOs on dm-snapshot.

As readahead IO is not required by user space, and to fix this issue,
I think it would be better to pass it to upper layer to handle it.

Cc: stable@vger.kernel.org
Fixes: a739ff3f543a ("dm verity: add support for forward error correction")
Signed-off-by: Wu Bo <bo.wu@vivo.com>
---
 drivers/md/dm-verity-target.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
index beec14b6b044..14e58ae70521 100644
--- a/drivers/md/dm-verity-target.c
+++ b/drivers/md/dm-verity-target.c
@@ -667,7 +667,9 @@ static void verity_end_io(struct bio *bio)
 	struct dm_verity_io *io = bio->bi_private;
 
 	if (bio->bi_status &&
-	    (!verity_fec_is_enabled(io->v) || verity_is_system_shutting_down())) {
+	    (!verity_fec_is_enabled(io->v) ||
+	     verity_is_system_shutting_down() ||
+	     (bio->bi_opf & REQ_RAHEAD))) {
 		verity_finish_io(io, bio->bi_status);
 		return;
 	}
-- 
2.25.1