Received: by 2002:a05:7208:9594:b0:7e:5202:c8b4 with SMTP id gs20csp1190123rbb; Mon, 26 Feb 2024 01:19:02 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXHGdbFr9uSnfDYRCLs+kOTXRpmPL9xrTXKstfyN25w3auQtsSYfbJ1uX63Cz4bJlL+fhkJdEqLY5TC1J4WtJ5ponmQybAClCAnID4MZw== X-Google-Smtp-Source: AGHT+IFGvavOMEgt6umLoy8zfy4guRNSaJnFnpHlMb1+kLZb4zh99i1jQVyQYMv+4AH9JXD4QlaR X-Received: by 2002:a05:6358:9894:b0:17b:7a21:7f58 with SMTP id q20-20020a056358989400b0017b7a217f58mr7606821rwa.22.1708939141971; Mon, 26 Feb 2024 01:19:01 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708939141; cv=pass; d=google.com; s=arc-20160816; b=TX39KYhRDSTvEbA7yLU8owLQq6BaZtGNOeDbX4AHRTXIIst8UMBe0CN2GpuUfVt4Zg 58MvmZekiRc6hnEvY2gePVWBI08jyT4lqlQP864UYc52xBY+W/x4SWd0rTgczNc//yKT waHtQD84ZyTd20A6g83XTNTPc7KHRHz2pyk4GbCBD6jrzDl/3fo8xp19uRi0SubzUKPW AuAj4/UeQSwh/ss0d/REnPjzdewFoKVQsghrWoughZ+5TxgDTUP0phe8b3U0lVPueTpW 4YroJlHzk7TDEJlCWMV6xoAh+ZzHgVczzBvqWKUY/zoHT0it18aZAm34Lzp96hD7aoON s5vg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:dkim-signature; bh=FJkjXTBzXY+2D9xPxXL1fp+CqJ2v05a2sL5VHAnqrUM=; fh=8v6KXyXvobuQelc1y7Wd9IvaRllOwvIXTSfcNwHX6Ks=; b=LpVhfG8NRp4V8hCcVWLs6fUAVV51SdW1SPpjUsOvnaqw03U21XbqK+FvYqnihm1JKX QNqKjB+Rq2bcz5cx7YAyM1hknhNWOxcRMWGPRrsQZiB04t3Azukw++NUqxVCmpMqPbEY YPipU3d7JZV7YkR4tBtSrL98/sJ9IqQIAiVVSEFXTmO7tMfhInWgqSP7ip3PEN/l9ojd D/60OuOx+7heVUHhBbj3UTIJv59MsvG7h19bJVZwLDbDxeRzViMtAjPqEvUxeiiKGlLK k7UlXiybB1Brjtei8DT+EEmJzMrNganUO5vkGSLz1mDp/cyuAg6owPxmUMu7AlqAsVQ+ DEug==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=c9RcqYty; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-nfs+bounces-2086-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-nfs+bounces-2086-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id u6-20020a170902e80600b001dbf6a7e767si3413458plg.59.2024.02.26.01.19.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 01:19:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-nfs+bounces-2086-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=c9RcqYty; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-nfs+bounces-2086-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-nfs+bounces-2086-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 871F6285E00 for ; Mon, 26 Feb 2024 09:19:01 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1A95253393; Mon, 26 Feb 2024 08:39:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="c9RcqYty" X-Original-To: linux-nfs@vger.kernel.org Received: from mail-ed1-f51.google.com (mail-ed1-f51.google.com [209.85.208.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5321353392 for ; Mon, 26 Feb 2024 08:39:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.51 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708936781; cv=none; b=N+cUIZHOckuQDZoqnhbjMr6NfIO2MxqqU1o0DKuDdVwnagqZZlx6jgt3DczRDOoRWRjvjWz0wu8IPf0Fg96V2WNxIpW+iHtWyrIWD8rwU3cR47MSd+4gyKZLEzYeXEtf4fRV5ENLx/VtUPr7uBQrLkmyz9hoQsGX8ySbhZRJUTU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708936781; c=relaxed/simple; bh=9MzM/EfUjg6Q2/7K2j8SB8iHqM0nd1jWYatIjQwvVXU=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Content-Type; b=C2Wt4dZedFkt/GOsxDVoCutdjoh0gjuRB7hjauRpGngwx2GwxtlEd1WXyLye1ZBp2y0F2En+fKzOz5TS1Vu9VRXNBmdx9r2uLsOLK/mx4DWCT3vEVaDcLbASUI/EjiPUUtvbLFa9Y9GsGv4w6PrjNKemdr2Aa59bxo6Zxx0NIXE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=c9RcqYty; arc=none smtp.client-ip=209.85.208.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-ed1-f51.google.com with SMTP id 4fb4d7f45d1cf-564372fb762so3987897a12.0 for ; Mon, 26 Feb 2024 00:39:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708936777; x=1709541577; darn=vger.kernel.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=FJkjXTBzXY+2D9xPxXL1fp+CqJ2v05a2sL5VHAnqrUM=; b=c9RcqYtynRgdn+MZr/4Qw7UwS5ut1SLNpeecWmGUbSyp75a/BdXetbisV9FT72KmfC MS1dXYrG3q1fu+efGNHmsu1Vq/axNZt2yjZADic22opNzOv5XcyjoALc/Wqpicre5lLn 7mSKjesxk947f/OeJ5F3m7JWc7ZEnV5FJoPPdo+c+X/9rYQrjgGe1GCz2O6J3PEiEDn0 gA2NdZedL4bfpDuvhp1ONPxHiFhgKw0RNjSfg4L1lAKPS+eKu0TQKVnznM8lhMvwXwfx PQezmye7YrUJHdtJXszj4ha3AoMlhYns+Dgc0KmQ16TgSSg42uBTUvt8QqqjVH4uT6hu QIrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708936777; x=1709541577; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FJkjXTBzXY+2D9xPxXL1fp+CqJ2v05a2sL5VHAnqrUM=; b=nb/A4KpEWrhxPjxwewumRa6XjNmTdyyU6wbfGZC4zcuAtsYq+58OqqHinG9ept57IH yIkdty/qxkLGre1W/a06WLR3mDduQjGyBQq+OHW4PX0ikO1ItICCQPy13Xo5FiCJYSmZ Ro4KCXS6kOM+qC1jnEzYR/3d5gCO50B+U0zMBepp5hW2+0lqxwy7WvN9YfHbChT1Sb2R 7dWHAndyMDF/QOno12K2KoGF7ecBTWecVtUakZ7hSp2QyAcCUlS9ODotP7aPK/yxfBPI dgAvWzhVBOg/0FfIm0QLI+PDgVtWFJ2vpGRspiKnuoa+fmjWcF5betZish4NffsSwQhq 8HMw== X-Forwarded-Encrypted: i=1; AJvYcCU3QoHg5xD7ssKDZXaxM/RZTuXeZMgAJRRvc1x4pbuTn4gb/i0F3BB4ibLajLFW8UojrCf/J5cKMTKwqsGxqij5FkAEY54dAetJ X-Gm-Message-State: AOJu0YwzhwNZSBddQiwbZH7WMF0JGJT0sF5BE55WD2l0OGJt698b8hMM CSV9UQqo8rGibmuyOr3ShdvbjMmTZhWes9vNzRuIMB9skwmP487RVjWKv2Zc79n9D842GHiGVY2 YT8yEOpcZENKP5UbPq/Sfzf3kbtc= X-Received: by 2002:a05:6402:e9a:b0:565:af1d:7416 with SMTP id h26-20020a0564020e9a00b00565af1d7416mr3415808eda.5.1708936777279; Mon, 26 Feb 2024 00:39:37 -0800 (PST) Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <170890214013.24797.3257981274610636720@noble.neil.brown.name> <170890314859.24797.16728369357798855399@noble.neil.brown.name> In-Reply-To: <170890314859.24797.16728369357798855399@noble.neil.brown.name> From: Cedric Blancher Date: Mon, 26 Feb 2024 09:39:00 +0100 Message-ID: Subject: Re: NFS data corruption on congested network To: NeilBrown , Jacek Tomaka , trond.myklebust@hammerspace.com, anna.schumaker@netapp.com, linux-nfs@vger.kernel.org Content-Type: text/plain; charset="UTF-8" On Mon, 26 Feb 2024 at 00:19, NeilBrown wrote: > > On Mon, 26 Feb 2024, NeilBrown wrote: > > On Fri, 23 Feb 2024, Jacek Tomaka wrote: > > > Hello, > > > I ran into an issue where the NFS file ends up being corrupted on disk. We started noticing it on certain, quite old hardware after upgrading OS from Centos 6 to Rocky 9.2. We do see it on Rocky 9.3 but not on 9.1. > > > > > > After some investigation we have reasons to believe that the change was introduced by the following commit: > > > https://github.com/torvalds/linux/commit/6df25e58532be7a4cd6fb15bcd85805947402d91 > > > > Thanks for the report. > > Can you try a change to your kernel? > > > > diff --git a/fs/nfs/write.c b/fs/nfs/write.c > > index bb79d3a886ae..08a787147bd2 100644 > > --- a/fs/nfs/write.c > > +++ b/fs/nfs/write.c > > @@ -668,8 +668,10 @@ static int nfs_writepage_locked(struct folio *folio, > > int err; > > > > if (wbc->sync_mode == WB_SYNC_NONE && > > - NFS_SERVER(inode)->write_congested) > > + NFS_SERVER(inode)->write_congested) { > > + folio_redirty_for_writepage(wbc, folio); > > return AOP_WRITEPAGE_ACTIVATE; > > + } > > > > nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGE); > > nfs_pageio_init_write(&pgio, inode, 0, false, > > Actually this is only needed before linux 6.8 as only nfs_writepage() > can call nfs_writepage_locked() with sync_mode of WB_SYNC_NONE. > So v5.18 through v6.7 might need fixing. Please do not forget the Linux 6.6-stable branch!! Ced -- Cedric Blancher [https://plus.google.com/u/0/+CedricBlancher/] Institute Pasteur