Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 19 Jan 2021 17:22:29 -0500
From:   "bfields@fieldses.org" <bfields@fieldses.org>
To:     Trond Myklebust <trondmy@hammerspace.com>
Cc:     "chuck.lever@oracle.com" <chuck.lever@oracle.com>,
        "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
        "aglo@umich.edu" <aglo@umich.edu>,
        "bcodding@redhat.com" <bcodding@redhat.com>,
        Jacob Shivers <jshivers@redhat.com>,
        Frank Sorenson <fsorenso@redhat.com>
Subject: Re: unsharing tcp connections from different NFS mounts
Message-ID: <20210119222229.GA29488@fieldses.org>
References: <95542179-0C20-4A1F-A835-77E73AD70DB8@redhat.com>
 <CAN-5tyGDC0VQqjqUNzs_Ka+-G_1eCScVxuXvWsp7xe7QYj69Ww@mail.gmail.com>
 <20201007001814.GA5138@fieldses.org>
 <57E3293C-5C49-4A80-957B-E490E6A9B32E@redhat.com>
 <5B5CF80C-494A-42D3-8D3F-51C0277D9E1B@redhat.com>
 <8ED5511E-25DE-4C06-9E26-A1947383C86A@oracle.com>
 <20201007140502.GC23452@fieldses.org>
 <85F496CD-9AAC-451C-A224-FCD138BDC591@oracle.com>
 <20201007160556.GE23452@fieldses.org>
 <e06c31e4211cefda52091c7710d871f44dc9160e.camel@hammerspace.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <e06c31e4211cefda52091c7710d871f44dc9160e.camel@hammerspace.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: bulk

On Wed, Oct 07, 2020 at 04:50:26PM +0000, Trond Myklebust wrote:
> As far as I can tell, this thread started with a complaint that
> performance suffers when we don't allow setups that hack the client by
> pretending that a multi-homed server is actually multiple different
> servers.
> 
> AFAICS Tom Talpey's question is the relevant one. Why is there a
> performance regression being seen by these setups when they share the
> same connection? Is it really the connection, or is it the fact that
> they all share the same fixed-slot session?
> 
> I did see Igor's claim that there is a QoS issue (which afaics would
> also affect NFSv3), but why do I care about QoS as a per-mountpoint
> feature?

Sorry for being slow to get back to this.

Some more details:

Say an NFS server exports /data1 and /data2.

A client mounts both.  Process 'large' starts creating 10G+ files in
/data1, queuing up a lot of nfs WRITE rpc_tasks.

Process 'small' creates a lot of small files in /data2, which requires a
lot of synchronous rpc_tasks, each of which wait in line with the large
WRITE tasks.

The 'small' process makes painfully slow progress.

The customer previously made things work for them by mounting two
different server IP addresses, so the "small" and "large" processes
effectively end up with their own queues.

Frank Sorenson has a test showing the difference; see

	https://bugzilla.redhat.com/show_bug.cgi?id=1703850#c42
	https://bugzilla.redhat.com/show_bug.cgi?id=1703850#c43

In that test, the "small" process creates files at a rate thousands of
times slower when the "large" process is also running.

Any suggestions?

--b.