Received: by 2002:a05:6a10:5bc5:0:0:0:0 with SMTP id os5csp19783pxb; Thu, 14 Oct 2021 23:00:19 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwsjPDWJ7eXbxW+4ke2Y3nCK5m0XJnFF67CbyNdGqZCfj46FlUqRGXyzhvzLMveZsXl3szm X-Received: by 2002:a17:906:82c8:: with SMTP id a8mr4504512ejy.384.1634277619222; Thu, 14 Oct 2021 23:00:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1634277619; cv=none; d=google.com; s=arc-20160816; b=rbBzzS48eZmRqqLllDihrs6uE7o/5jPFAQjEzX/z0TS7Nt5zXWCnZUGmhkKKpEOnhW U8T0KbBuaCuPXnSsU8lBx+HrxPp3mYGtZvVP/uTlkNeZMiUjwffWXs4WNHa67xttWOo3 8/4RZWN5BUSV25Gg9J3C73/D8yWmylZ60CwfbEx70ktUA6BuOMqyCMxTPDIyhttRX/T0 RAGz0YneZuOfx7fhJAwPMukxxXS/ftW1Krc6ToBhHXVhzb8tC1tvjS1gpnsSO2ZbVxar ObY2hqME+JpSWD6x4fbKFpWrsHeudL4ZT19Lm/yxezHIn758YbfOtP28aIzeKYWsAwoo OX1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:message-id:date:references:in-reply-to:subject :cc:to:from:mime-version:content-transfer-encoding:dkim-signature :dkim-signature; bh=rlXcAuuDIvW3z2VorqCFxSCxEKvxyRCe+a9pWvJPKl0=; b=iwcyQztPT3xB3Mf/oZoO2S/vu+wbzGFW/7lunQ0CF7lGMZVWpNTmG8eiprPu2qu9w1 6uHZModCSAsA5VFL8VqCkJBrEuuoTlLhnss57OEsGO80M0IYGZFScU5Pc792Lj4LQCPS nxmLrBuWkIj9ra0KB5cilyk9NNMq4lFRb6vYoUmEbw8TJPZHpqxQxaGEAK0+WNtU/dJ0 YWdESj5C1XTXknLrSk3io1c+zThK5miz/C9f+T55W7yFI2uqwYQB2tF14t11dyllHdAM uBxMxFr1ElnRe74Qqv02G+sPbRNI2kPrW/s1j1nL0u086+hNsO+llFQa0hUvB0vvL7Pl QAlw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b="X/FkL1hc"; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=ZP3ZmPIU; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id gg25si7059184ejb.307.2021.10.14.22.59.53; Thu, 14 Oct 2021 23:00:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b="X/FkL1hc"; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=ZP3ZmPIU; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230015AbhJNWxY (ORCPT + 99 others); Thu, 14 Oct 2021 18:53:24 -0400 Received: from smtp-out1.suse.de ([195.135.220.28]:43936 "EHLO smtp-out1.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229818AbhJNWxY (ORCPT ); Thu, 14 Oct 2021 18:53:24 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 534252196F; Thu, 14 Oct 2021 22:51:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1634251878; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rlXcAuuDIvW3z2VorqCFxSCxEKvxyRCe+a9pWvJPKl0=; b=X/FkL1hce1jZo89F4w4pYvTG64yZd41tDI/XCY5af2n7Pi++6vdsikhz0eeG9pR03mrpys BEobTmuNE2xd8dPrpe9ruOvtyfJx5qK6ODs2tsxaueEV1Mac7eNkdeC3YSkQf4+jsoKasZ LxNbERbbkRFAj7iv/BgVVRRQlDtFGqI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1634251878; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rlXcAuuDIvW3z2VorqCFxSCxEKvxyRCe+a9pWvJPKl0=; b=ZP3ZmPIUFFTbgsaZXx1kroIBUJAZ3CtJVUifqkkLX8l5EA2iEeDcmbYzxfrRJyYl/TAFGl nGutCE1auIJ79PBQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 0029A13522; Thu, 14 Oct 2021 22:51:15 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 37jHJ2O0aGEWRAAAMHmgww (envelope-from ); Thu, 14 Oct 2021 22:51:15 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 From: "NeilBrown" To: "Trond Myklebust" Cc: "chuck.lever@oracle.com" , "bfields@fieldses.org" , "smayhew@redhat.com" , "tpearson@raptorengineering.com" , "linux-nfs@vger.kernel.org" Subject: Re: CPU stall, eventual host hang with BTRFS + NFS under heavy load In-reply-to: References: <281642234.3818.1625478269194.JavaMail.zimbra@raptorengineeringinc.com>, <162855621114.22632.14151019687856585770@noble.neil.brown.name>, <20210812144428.GA9536@fieldses.org>, <162880418532.15074.7140645794203395299@noble.neil.brown.name>, , <589AFA4F-DF8E-45A3-8299-54E820E33169@oracle.com>, <20211011143028.GB22387@fieldses.org>, <34A4C7EB-2798-4482-A786-90161F5F9E34@oracle.com>, <163398946770.17149.14605560717849885454@noble.neil.brown.name>, Date: Fri, 15 Oct 2021 09:51:12 +1100 Message-id: <163425187213.17149.4082212844733494965@noble.neil.brown.name> Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org T24gRnJpLCAxNSBPY3QgMjAyMSwgVHJvbmQgTXlrbGVidXN0IHdyb3RlOgo+IE9uIFR1ZSwgMjAy MS0xMC0xMiBhdCAwODo1NyArMTEwMCwgTmVpbEJyb3duIHdyb3RlOgo+ID4gT24gVHVlLCAxMiBP Y3QgMjAyMSwgQ2h1Y2sgTGV2ZXIgSUlJIHdyb3RlOgo+ID4gPiAKPiA+ID4gU2NvdHQgc2VlbXMg d2VsbCBwb3NpdGlvbmVkIHRvIGlkZW50aWZ5IGEgcmVwcm9kdWNlci4gTWF5YmUgd2UKPiA+ID4g Y2FuIGdpdmUgaGltIHNvbWUgbGlrZWx5IGNhbmRpZGF0ZXMgZm9yIHBvc3NpYmxlIGJ1Z3MgdG8g ZXhwbG9yZQo+ID4gPiBmaXJzdC4KPiA+IAo+ID4gSGFzIHRoaXMgcGF0Y2ggYmVlbiB0cmllZD8K PiA+IAo+ID4gTmVpbEJyb3duCj4gPiAKPiA+IAo+ID4gZGlmZiAtLWdpdCBhL25ldC9zdW5ycGMv c2NoZWQuYyBiL25ldC9zdW5ycGMvc2NoZWQuYwo+ID4gaW5kZXggYzA0NWY2M2QxMWZhLi4zMDhm NTk2MWNiNzggMTAwNjQ0Cj4gPiAtLS0gYS9uZXQvc3VucnBjL3NjaGVkLmMKPiA+ICsrKyBiL25l dC9zdW5ycGMvc2NoZWQuYwo+ID4gQEAgLTgxNCw2ICs4MTQsNyBAQCBycGNfcmVzZXRfdGFza19z dGF0aXN0aWNzKHN0cnVjdCBycGNfdGFzayAqdGFzaykKPiA+IMKgewo+ID4gwqDCoMKgwqDCoMKg wqDCoHRhc2stPnRrX3RpbWVvdXRzID0gMDsKPiA+IMKgwqDCoMKgwqDCoMKgwqB0YXNrLT50a19m bGFncyAmPSB+KFJQQ19DQUxMX01BSk9SU0VFTnxSUENfVEFTS19TRU5UKTsKPiA+ICvCoMKgwqDC oMKgwqDCoGNsZWFyX2JpdChSUENfVEFTS19TSUdOQUxMRUQsICZ0YXNrLT50a19ydW5zdGF0ZSk7 Cj4gPiDCoMKgwqDCoMKgwqDCoMKgcnBjX2luaXRfdGFza19zdAo+IAo+IFdlIHNob3VsZG4ndCBh dXRvbWF0aWNhbGx5ICJ1bnNpZ25hbCIgYSB0YXNrIG9uY2UgaXQgaGFzIGJlZW4gdG9sZCB0bwo+ IGRpZS4gVGhlIGNvcnJlY3QgdGhpbmcgdG8gZG8gaGVyZSBzaG91bGQgcmF0aGVyIGJlIHRvIGNo YW5nZQo+IHJwY19yZXN0YXJ0X2NhbGwoKSB0byBleGl0IGVhcmx5IGlmIHRoZSB0YXNrIHdhcyBz aWduYWxsZWQuCj4gCgpNYXliZS4gIEl0IGRlcGVuZHMgb24gZXhhY3RseSB3aGF0IHRoZSBzaWdu YWwgbWVhbnQgKHJwY19raWxsYWxsX3Rhc2tzKCkKaXMgYSBiaXQgZGlmZmVyZW50IGZyb20gZ2V0 dGluZyBhIFNJR0tJTEwpLCBhbmQgZXhhY3RseSB3aGF0IHRoZSB0YXNrIGlzCnRyeWluZyB0byBh Y2hpZXZlLgoKQmVmb3JlIENvbW1pdCBhZTY3YmQzODIxYmIgKCJTVU5SUEM6IEZpeCB1cCB0YXNr IHNpZ25hbGxpbmciKQp0aGF0IGlzIGV4YWN0bHkgd2hhdCB3ZSBkaWQuCklmIHdlIHdhbnQgdG8g Y2hhbmdlIHRoZSBiZWhhdmlvdXIgb2YgYSB0YXNrIHJlc3BvbmRpbmcgdG8KcnBjX2tpbGxhbGxf dGFza3MoKSwgd2Ugc2hvdWxkIGNsZWFybHkganVzdGlmeSBpdCBpbiBhIHBhdGNoIGRvaW5nCmV4 YWN0bHkgdGhhdC4KCk5laWxCcm93bgo=