Received: by 2002:ac0:946b:0:0:0:0:0 with SMTP id j40csp2717157imj; Mon, 11 Feb 2019 07:26:17 -0800 (PST) X-Google-Smtp-Source: AHgI3IbkOrLOfSi6h493aFLBj8SjOMv5B+84EzMpKsxOsKGo9n/IgFXs6va8flhXYwdmbD6p3zXJ X-Received: by 2002:a62:4549:: with SMTP id s70mr36872929pfa.233.1549898777678; Mon, 11 Feb 2019 07:26:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1549898777; cv=none; d=google.com; s=arc-20160816; b=GwhRS06LKPkQmpxSiE9odEYLFMyygDTHA/TX/DGqE40A4BDay/OoyOhZdKY112Diip 0+CEVnoubO7ltT9JehvnoGouSzClKhPgJiYenyo0UQfKwj09Ctz7xpq8jPNgC2xLUsyV KSjHxV0zP/xE4l5A0Ol2fYsnc6GsN/g3LVQM1WXy9KAMFFGw6VidGt1c3MEqSKkbAt1I KJ+CMzG5FSeb3I9pFraD4+RX6wJEqbGBrM2uGeZe6fvvhCtx6Zd1nRmyhWV+iZ4AvxQP kDNDUXGf9lqnHhDdDzsmkR5X3GRAeQPIJb3A7BdQmPm2Mn5eJFM9cL8aVU3z0dBlnoqy f8Vw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=413POKDW3TBLp8K7S2ICoQgD0r2C7hqqq4OvKfxMlKQ=; b=AOqHXmP15z6a2myoB7/VokacexgNHaieH3U1z6/yGclG071RFakE40ChD7oP6i/rze UBlfKIAgzUI/MmhtRkTCaYDy3cEdVHg4fkE4kX8BX33BvL7duf1VAwQauRo9MTS2D8n4 1ePRRecksjopGZht8BqEEoeUY36KnQcuCaV0BHOzmFeLFUO51/WvQQbsvkCb55EFTK3N XrNZLTd5AC9KVF/lmd2lQ5c86rVsnvW5hz0UG7bTb5HVLxAdwmePd/T1zw+m3rvvyQFC APrBOq6LWKaIlreFSCgtG4r534/fREgcpl9skQVX/bjgIDzh2uVRsjAbrC/h5PZRK/Rn 3eNA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=Y7CAffON; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id s129si9616092pgs.121.2019.02.11.07.26.01; Mon, 11 Feb 2019 07:26:17 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=Y7CAffON; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387786AbfBKPZK (ORCPT + 99 others); Mon, 11 Feb 2019 10:25:10 -0500 Received: from mail.kernel.org ([198.145.29.99]:49622 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390206AbfBKPBf (ORCPT ); Mon, 11 Feb 2019 10:01:35 -0500 Received: from localhost (5356596B.cm-6-7b.dynamic.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id ACA0E222A7; Mon, 11 Feb 2019 15:01:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1549897295; bh=vF6Ija9CafkHWozTnBAD4APHeZENXMdXbplwgIlWYq4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Y7CAffONXh6sOGGrR+LRaIvz5/tUtyxUTvicbnfD2M9eokwau8KPEKT48/k4u7JUz 1zbgnGnqzn/gErYr6r0QnvHBFFlyHrhNYEDgaoB4UJj450uCbEYSfHB/bj+8+ym1bg 5QKc6YUuoJufjfqfyoa9o866VHMiI8PZ/NPTZiZE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Guoqing Jiang , Shaohua Li , Sasha Levin , =?UTF-8?q?Sini=C5=A1a=20Bandin?= Subject: [PATCH 4.14 123/205] md: fix raid10 hang issue caused by barrier Date: Mon, 11 Feb 2019 15:18:41 +0100 Message-Id: <20190211141836.497307516@linuxfoundation.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190211141827.214852402@linuxfoundation.org> References: <20190211141827.214852402@linuxfoundation.org> User-Agent: quilt/0.65 X-stable: review X-Patchwork-Hint: ignore MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 4.14-stable review patch. If anyone has any objections, please let me know. ------------------ [ Upstream commit e820d55cb99dd93ac2dc949cf486bb187e5cd70d ] When both regular IO and resync IO happen at the same time, and if we also need to split regular. Then we can see tasks hang due to barrier. 1. resync thread [ 1463.757205] INFO: task md1_resync:5215 blocked for more than 480 seconds. [ 1463.757207] Not tainted 4.19.5-1-default #1 [ 1463.757209] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1463.757212] md1_resync D 0 5215 2 0x80000000 [ 1463.757216] Call Trace: [ 1463.757223] ? __schedule+0x29a/0x880 [ 1463.757231] ? raise_barrier+0x8d/0x140 [raid10] [ 1463.757236] schedule+0x78/0x110 [ 1463.757243] raise_barrier+0x8d/0x140 [raid10] [ 1463.757248] ? wait_woken+0x80/0x80 [ 1463.757257] raid10_sync_request+0x1f6/0x1e30 [raid10] [ 1463.757265] ? _raw_spin_unlock_irq+0x22/0x40 [ 1463.757284] ? is_mddev_idle+0x125/0x137 [md_mod] [ 1463.757302] md_do_sync.cold.78+0x404/0x969 [md_mod] [ 1463.757311] ? wait_woken+0x80/0x80 [ 1463.757336] ? md_rdev_init+0xb0/0xb0 [md_mod] [ 1463.757351] md_thread+0xe9/0x140 [md_mod] [ 1463.757358] ? _raw_spin_unlock_irqrestore+0x2e/0x60 [ 1463.757364] ? __kthread_parkme+0x4c/0x70 [ 1463.757369] kthread+0x112/0x130 [ 1463.757374] ? kthread_create_worker_on_cpu+0x40/0x40 [ 1463.757380] ret_from_fork+0x3a/0x50 2. regular IO [ 1463.760679] INFO: task kworker/0:8:5367 blocked for more than 480 seconds. [ 1463.760683] Not tainted 4.19.5-1-default #1 [ 1463.760684] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1463.760687] kworker/0:8 D 0 5367 2 0x80000000 [ 1463.760718] Workqueue: md submit_flushes [md_mod] [ 1463.760721] Call Trace: [ 1463.760731] ? __schedule+0x29a/0x880 [ 1463.760741] ? wait_barrier+0xdd/0x170 [raid10] [ 1463.760746] schedule+0x78/0x110 [ 1463.760753] wait_barrier+0xdd/0x170 [raid10] [ 1463.760761] ? wait_woken+0x80/0x80 [ 1463.760768] raid10_write_request+0xf2/0x900 [raid10] [ 1463.760774] ? wait_woken+0x80/0x80 [ 1463.760778] ? mempool_alloc+0x55/0x160 [ 1463.760795] ? md_write_start+0xa9/0x270 [md_mod] [ 1463.760801] ? try_to_wake_up+0x44/0x470 [ 1463.760810] raid10_make_request+0xc1/0x120 [raid10] [ 1463.760816] ? wait_woken+0x80/0x80 [ 1463.760831] md_handle_request+0x121/0x190 [md_mod] [ 1463.760851] md_make_request+0x78/0x190 [md_mod] [ 1463.760860] generic_make_request+0x1c6/0x470 [ 1463.760870] raid10_write_request+0x77a/0x900 [raid10] [ 1463.760875] ? wait_woken+0x80/0x80 [ 1463.760879] ? mempool_alloc+0x55/0x160 [ 1463.760895] ? md_write_start+0xa9/0x270 [md_mod] [ 1463.760904] raid10_make_request+0xc1/0x120 [raid10] [ 1463.760910] ? wait_woken+0x80/0x80 [ 1463.760926] md_handle_request+0x121/0x190 [md_mod] [ 1463.760931] ? _raw_spin_unlock_irq+0x22/0x40 [ 1463.760936] ? finish_task_switch+0x74/0x260 [ 1463.760954] submit_flushes+0x21/0x40 [md_mod] So resync io is waiting for regular write io to complete to decrease nr_pending (conf->barrier++ is called before waiting). The regular write io splits another bio after call wait_barrier which call nr_pending++, then the splitted bio would continue with raid10_write_request -> wait_barrier, so the splitted bio has to wait for barrier to be zero, then deadlock happens as follows. resync io regular io raise_barrier wait_barrier generic_make_request wait_barrier To resolve the issue, we need to call allow_barrier to decrease nr_pending before generic_make_request since regular IO is not issued to underlying devices, and wait_barrier is called again to ensure no internal IO happening. Fixes: fc9977dd069e ("md/raid10: simplify the splitting of requests.") Reported-and-tested-by: SiniĊĦa Bandin Signed-off-by: Guoqing Jiang Signed-off-by: Shaohua Li Signed-off-by: Sasha Levin --- drivers/md/raid10.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 52ddfa0fca94..2ce079a0b0bd 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -1190,7 +1190,9 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, struct bio *split = bio_split(bio, max_sectors, gfp, conf->bio_split); bio_chain(split, bio); + allow_barrier(conf); generic_make_request(bio); + wait_barrier(conf); bio = split; r10_bio->master_bio = bio; r10_bio->sectors = max_sectors; @@ -1479,7 +1481,9 @@ retry_write: struct bio *split = bio_split(bio, r10_bio->sectors, GFP_NOIO, conf->bio_split); bio_chain(split, bio); + allow_barrier(conf); generic_make_request(bio); + wait_barrier(conf); bio = split; r10_bio->master_bio = bio; } -- 2.19.1