Received-SPF: pass (google.com: domain of linux-nfs+bounces-3511-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Fri, 31 May 2024 16:12:35 +0200
From: Jan Kara <jack@suse.cz>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: NeilBrown <neilb@suse.de>, Jan Kara <jack@suse.cz>,
	Trond Myklebust <trondmy@hammerspace.com>,
	"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
	"anna@kernel.org" <anna@kernel.org>
Subject: Re: NFS write congestion size
Message-ID: <20240531141235.gthl6nvyrb354rfc@quack3>
References: <>
 <a86fdd86-7c3a-44e9-9d49-4b3edfab66e6@grimberg.me>
 <171706807360.14261.8929224868643154972@noble.neil.brown.name>
 <faf945a9-d1bc-45db-9543-057b7bc0ed8d@grimberg.me>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <faf945a9-d1bc-45db-9543-057b7bc0ed8d@grimberg.me>

On Thu 30-05-24 21:19:55, Sagi Grimberg wrote:
> > > btw, I think you meant that *slower* devices may need a larger queue to
> > > saturate,
> > > because if the device is fast, 256MB inflight is probably enough... So
> > > you are solving
> > > for the "consumer grade contemporary disks".
> > No, faster devices.  The context here is random writes.  The more of
> > those that are in the queue on the server, the more chance it has to
> > re-order or coalesce requests.
> 
> Umm, what would reorder help with if writes are random?

Well, even though writes are random, they are still from a limited space so
the more of them you have, the larger contiguous chunks you are actually
going to get. But that's not really the main helping factor here. The main
benefit of allowing more than 256MB to be fed into the server is indeed
that the writeback on the server can be more contiguous. With 256MB limit
what I see is that the server gets the data, then commit which triggers
writeout of the data on the server, the server then signals completion and
idles before the it gets more data from the client. With say 1GB or larger
limit, the server has some data to write most of the time so overall
throughput and disk utilization is much better.

So it is all about how latency of the network, the speed of the storage and
batching of writeback completions plays together. And yes, we can just
increase the limit but my opinion still is that limiting the number of MB
of writeback we have outstanding against the server is a wrong way of
throttling things because that number is difficult to get right without
knowing storage speeds, network speeds, load from other clients and
internals of page writeback...

I understand Trond's concerns about latency increase as well as your
concerns about overloading the server with RPC traffic so perhaps we need
*some* mechanism by which the client can detect contention. But I think we
need to come up with something better than fixed number of MB...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR