Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <39bf58c7-47d9-744b-6d26-d672aa713024@gmx.ch> <8cd63730f7b5f3e2aa3bde98587de0c6a42b384b.camel@kernel.org>
 <8550c032-2ef8-4ddb-19a0-307777bd9645@gmx.ch> <c163b299164cd9abe21325a947c9efe908a04331.camel@kernel.org>
 <e3842fe7-da95-98e2-b0e8-fa3aede673cc@gmx.ch>
In-Reply-To: <e3842fe7-da95-98e2-b0e8-fa3aede673cc@gmx.ch>
From:   Olga Kornievskaia <aglo@umich.edu>
Date:   Mon, 3 Oct 2022 10:18:38 -0400
Message-ID: <CAN-5tyEj97UO-K=ZcgkhzNr2H3wTd5tq8u+TdPj19-syc81=rw@mail.gmail.com>
Subject: Re: nfs4.1+: workaround for defunct clientaddr?
To:     Manfred Schwarb <manfred99@gmx.ch>
Cc:     Jeff Layton <jlayton@kernel.org>, linux-nfs@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

 Hi Manfred,

What's the purpose of segregating your connections? You don't want
your backup traffic "interfering" with your regular operation.
However, the assumption is that between 2 NICs the backup traffic and
regular traffic could co-exists, correct? In that case why not use
session trunking?  What you are correctly experiencing is that with
4.1+ the 2nd mount discovers that in the 2nd mount it's the same
server the client is talking to (even if it's thru a different IPs)
and the client will drop the new connection.

For session trunking, you can configure your linux server (I'm
assuming it is, if not that might be a problem) to support session
trunking (by using replica=<> option). Then you can also add
"discovertrunking" option to your mount command and then the client
will discover the 2 available paths to the server. You wouldn't need 2
mounts and you'd have both NICs available to serve your combined
regular and backup traffic. This would be the solution to utilize both
of the NICs (network paths) you have available between the client and
the server.


On Mon, Oct 3, 2022 at 9:27 AM Manfred Schwarb <manfred99@gmx.ch> wrote:
>
> Am 03.10.22 um 14:26 schrieb Jeff Layton:
> > On Mon, 2022-10-03 at 13:55 +0200, Manfred Schwarb wrote:
> >> Am 03.10.22 um 13:39 schrieb Jeff Layton:
> >>> On Sun, 2022-10-02 at 14:35 +0200, Manfred Schwarb wrote:
> >>>> Hi,
> >>>>
> >>>> I have 2 boxes connected with 2 network cards each, one
> >>>> crossover connection and one connection via LAN.
> >>>> I want to use the crossover connection for backup,
> >>>> so I want to be able to select exactly this wire when
> >>>> doing my NFS backup transfers. Everything interconnected via NFS4.1
> >>>> and automount.
> >>>>
> >>>> Now the thing is, if there is an already existing connection
> >>>> via LAN, I am not able to select the crossover connection,
> >>>> there is some session reuse against my will.
> >>>>
> >>>> automount config:
> >>>> /net/192.168.99.1  -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.99.100   /  192.168.99.1:/
> >>>> /net2/192.168.98.1 -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.98.100   /  192.168.98.1:/
> >>>>
> >>>> mount -l:
> >>>> 192.168.99.1:/data on /net/192.168.99.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
> >>>> 192.168.99.1:/data on /net2/192.168.98.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
> >>>>
> >>>> As you see, both connections are on "192.168.99.1:/data", and the backup runs
> >>>> over the same wire as all user communication, which is not desired.
> >>>> This even happens if I explicitly set some clientaddr= option.
> >>>>
> >>>> Now I found two workarounds:
> >>>> - downgrade to NFS 4.0, clientaddr seems to work with it
> >>>> - choose different NFS versions, i.e. one connection with
> >>>>   minorversion=1 and the other with minorversion=2
> >>>>
> >>>> Both possibilities seem a bit lame to me.
> >>>> Are there some other (recommended) variants which do what I want?
> >>>>
> >>>> It seems different minor versions result in different "nfs4_unique_id" values,
> >>>> and therefore no session sharing occurs. But why do different network
> >>>> interfaces (via explicitly set clientaddr= by user) not result in different
> >>>> "nfs4_unique_id" values?
> >>>>
> >>>> Thanks for any comments and advice,
> >>>> Manfred
> >>>
> >>> That sounds like a bug. We probably need to compare the clientaddr
> >>> values in nfs_compare_super or nfs_compare_mount_options so that it
> >>> doesn't match if the clientaddrs are different.
> >>>
> >
> >
> > Actually, I take it back, clientaddr is specifically advertised as being
> > for NFSv4.0 only. The workaround for you is "nosharecache", which will
> > force the mount under /net2 to get a new superblock altogether.
>
> But clientaddr is silently accepted on NFS4.1+, and seemingly silently does nothing.
>
> The point is, RFC5661 explicitely tells
> "NFS minor version 1 is deemed superior to NFS minor version 0 with no loss of functionality".
>
> So this behavior comes as a surprise.
>
> >
> >>> As a workaround, you can probably mount the second mount with
> >>> -o nosharecache and get what you want.
> >>
> >> Indeed, nosharecache works. But the man page has some scary words for it:
> >>   "This is considered a data risk".
> >>
> >
> > Yeah, it does sound scary but it's not a huge issue unless you're doing
> > I/O to the same files at the same time via both mounts. With
> > "sharecache" (the default) you get better cache coherency in that
> > situation since the inode and its pagecache are the same.
> >
>
> So I guess this is equivalent to the minorversion=1/minorversion=2 trick
> cache coherency wise then?
>
>
> > With "nosharecache" you need to be more careful to flush caches, etc. if
> > you are doing reads and writes to the same files via different paths. If
> > you need careful coordination there, then you probably want to use file
> > locking.
>
> Thanks for these explanations, it is appreciated!
> Manfred
>
> > --
> > Jeff Layton <jlayton@kernel.org>
>