Received: by 2002:a25:8b12:0:0:0:0:0 with SMTP id i18csp3360605ybl; Mon, 19 Aug 2019 17:29:13 -0700 (PDT) X-Google-Smtp-Source: APXvYqzTTDEWWdZOLjiADV/s+1VTu0DsrCLQdp4qOXnEW7KGyOifrxEL7GstfTUq8Gvjld+RdyDF X-Received: by 2002:a17:902:d210:: with SMTP id t16mr2262063ply.11.1566260953823; Mon, 19 Aug 2019 17:29:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566260953; cv=none; d=google.com; s=arc-20160816; b=LLp40Ms9nqabvOVJYhHn+9QceNc/sNEls/ZAxpS+WGxc+SiirZZW53kiKFuqWKFFm+ uaMguuFremBvKtMkeoumJk10TBYpsuCI4KowA/lhsuKm6pnFPdF1qoyeGofZcaKB0c/h NXWNfZf+gw3RCe26dw4ub8zABinZR2NgpADm303RWLewIFNsIc38wetCHucUj9Tf6M2n 2MRc/LJXWZeW34PjTC5So2GNYqh2ZP2yVI7ZH/bIW32awpaHFQFt+gzfwpPhtPkd/SSL 9Na3cbAsSGZUa6u+sKdAMEoFltzPOIbvkaZki3nSOgZo5scltOa1SPCeyERytaJu8C1g NHyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:message-id:references :in-reply-to:subject:cc:date:to:from; bh=0hvrLIyHHhc1vuR7u3ib7zXt1nEqzSJJq0glUJPt0IY=; b=fm6HQlPABe04xh2SJ7opFfSJJfT+h9k0l6MRcq1e/Ipubwa/lIcE9ZYHn26WhZrBwU Rhl5qClIXzVK7AA/3Ae2Yxy/CLVWUsiAxfSMf/vuqK04PWFcf/rPXc0acIoYbRrOEMz9 GyaWCd3yToYEq4wFBXvXLQQTP6exeiVVBzW+uZYmb8CPAE/Qa913OUQcXjXxGfBJGmKM 6agSeAZHqGcimKIo9oy+JNCemwGzWjsTr2IgC7o2iCypApGnU2Mhxh0l7Io5XbQAiE1W of/mKw4tojvsXac5SVer583fnIB+y8u15Jn6Ms44GmZFhX1LGTS3Dj1HYVIKLs8OgXaL 3uiA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a21si10714198pgv.185.2019.08.19.17.28.57; Mon, 19 Aug 2019 17:29:13 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728772AbfHTA2I (ORCPT + 99 others); Mon, 19 Aug 2019 20:28:08 -0400 Received: from mx2.suse.de ([195.135.220.15]:49344 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728578AbfHTA2H (ORCPT ); Mon, 19 Aug 2019 20:28:07 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id BF64AAC28; Tue, 20 Aug 2019 00:28:05 +0000 (UTC) From: NeilBrown To: Jinpu Wang Date: Tue, 20 Aug 2019 10:27:58 +1000 Cc: Alexandr Iarygin , Guoqing Jiang , Paul Menzel , Neil F Brown , linux-kernel@vger.kernel.org, linux-raid Subject: Re: Bisected: Kernel 4.14 + has 3 times higher write IO latency than Kernel 4.4 with raid1 In-Reply-To: References: <0a83fde3-1a74-684c-0d70-fb44b9021f96@molgen.mpg.de> <87h86vjhv0.fsf@notabene.neil.brown.name> <87blx1kglx.fsf@notabene.neil.brown.name> Message-ID: <87pnl0he9d.fsf@notabene.neil.brown.name> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --=-=-= Content-Type: text/plain On Fri, Aug 16 2019, Jinpu Wang wrote: > On Wed, Aug 7, 2019 at 2:35 PM Jinpu Wang wrote: >> >> On Wed, Aug 7, 2019 at 8:36 AM Jinpu Wang wrote: >> > >> > On Wed, Aug 7, 2019 at 1:40 AM NeilBrown wrote: >> > > >> > > On Tue, Aug 06 2019, Jinpu Wang wrote: >> > > >> > > > On Tue, Aug 6, 2019 at 9:54 AM Jinpu Wang wrote: >> > > >> >> > > >> On Tue, Aug 6, 2019 at 1:46 AM NeilBrown wrote: >> > > >> > >> > > >> > On Mon, Aug 05 2019, Jinpu Wang wrote: >> > > >> > >> > > >> > > Hi Neil, >> > > >> > > >> > > >> > > For the md higher write IO latency problem, I bisected it to these commits: >> > > >> > > >> > > >> > > 4ad23a97 MD: use per-cpu counter for writes_pending >> > > >> > > 210f7cd percpu-refcount: support synchronous switch to atomic mode. >> > > >> > > >> > > >> > > Do you maybe have an idea? How can we fix it? >> > > >> > >> > > >> > Hmmm.... not sure. >> > > >> Hi Neil, >> > > >> >> > > >> Thanks for reply, detailed result in line. >> > > >> > > Thanks for the extra testing. >> > > ... >> > > > [ 105.133299] md md0 in_sync is 0, sb_flags 2, recovery 3, external >> > > > 0, safemode 0, recovery_cp 524288 >> > > ... >> > > >> > > ahh - the resync was still happening. That explains why set_in_sync() >> > > is being called so often. If you wait for sync to complete (or create >> > > the array with --assume-clean) you should see more normal behaviour. >> > I've updated my tests accordingly, thanks for the hint. >> > > >> > > This patch should fix it. I think we can do better but it would be more >> > > complex so no suitable for backports to -stable. >> > > >> > > Once you confirm it works, I'll send it upstream with a >> > > Reported-and-Tested-by from you. >> > > >> > > Thanks, >> > > NeilBrown >> > >> > Thanks a lot, Neil, my quick test show, yes, it fixed the problem for me. >> > >> > I will run more tests to be sure, will report back the test result. >> Hi Neil, >> >> I've run our regression tests with your patch, everything works fine >> as expected. >> >> So Reported-and-Tested-by: Jack Wang >> >> Thank you for your quick fix. >> >> The patch should go to stable 4.12+ > > Hi Neil, > > I hope you're doing well, just a soft ping? do you need further > testing from my side? Thanks for the reminder. I've sent out the patch now. NeilBrown > > Please let me know how can we move the fix forward. > > Thanks, > Jack Wang --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEG8Yp69OQ2HB7X0l6Oeye3VZigbkFAl1bPo4ACgkQOeye3VZi gblYVg/8Dexb3Ia8ZpTQaD7fgTUi9kZ89sfPTzDBTxpcXiuYbGm2F2cCk3Uf12R4 Zfdr1EFajzmPo98CVCNcB7pzkbHqvtH5rx1XwxV6gQ0pIGJzWp3pnDSnfoLalj4k BeKs/zWjo5NHyUH0VTXQJmOhKuSk7RCSkEfNJVdV0q07ShK1uegy2khv7PEfmhdl 5e8vsf3aXNDfZnparqaY6fJanrMvv+Psq0lQUtQYVTLgk8Ty9NOTDh07aRoj2ZjJ u3Pxyt80jMIgkVOiBghdziDnapxCixPydXs8Pdj5y9IbSiii8noRDiHDnN3dLgg3 6VWy1uk3cp83Op0Nm+6Fe/IgPTRFZN8K9UGb1woOTsP4Abpe6K275k41CkgZ5HQF TsFD0icnEnjHH6O5yYn4oGLWf/LEl1H2B1sWN7K/HP+RzQNvwKistYTuxpacShwa L4xKm3b3anaQoMNm+rC+SfsVWjY0lFcFcy+sfZytcr2BrBfSgV5efD1aMeEKhtbm jLkI15kvfmPbl+mqMOc3A4B7jT3I/Y5J2Vgv5KIWCrm/4zrDN5TZ6AU4Yy4B+rMi QxrMcKYkXTvKF3jz56s3rHtgyLs6kopXVcX3Krqw4n7SIhV8Q0G/4nTLV0DPryAr lV1dRxQi4BuL5adBWDd6TDxXqPO6ukPXVF3lqbVz0e4x6duIjK0= =TyQL -----END PGP SIGNATURE----- --=-=-=--