Received-SPF: pass (google.com: domain of linux-nfs+bounces-3335-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Precedence: bulk
MIME-Version: 1.0
References: <CAN-5tyFBn3C_CTrsftuYeWJHe7KWxd82YFCyrN9t=az8J4RU0w@mail.gmail.com>
 <2C80B5BC-AAEC-41F8-BEB6-C920F88C89BB@oracle.com> <0b1101daa646$d26a6300$773f2900$@mindspring.com>
 <CAN-5tyGECFmtzFsYNSZicPcH4SMKF0yovk6V20sWJ1LrZKzzyA@mail.gmail.com> <0b1401daa64b$f5831ee0$e0895ca0$@mindspring.com>
In-Reply-To: <0b1401daa64b$f5831ee0$e0895ca0$@mindspring.com>
From: Olga Kornievskaia <aglo@umich.edu>
Date: Wed, 22 May 2024 09:57:39 -0400
Message-ID: <CAN-5tyHWeQykx0cFpOFf2hTBRk9_NaZarzeeAFdSu2NW0zqobA@mail.gmail.com>
Subject: Re: sm notify (nlm) question
To: Frank Filz <ffilzlnx@mindspring.com>
Cc: Chuck Lever III <chuck.lever@oracle.com>, 
	Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Tue, May 14, 2024 at 6:13=E2=80=AFPM Frank Filz <ffilzlnx@mindspring.com=
> wrote:
>
>
>
> > -----Original Message-----
> > From: Olga Kornievskaia [mailto:aglo@umich.edu]
> > Sent: Tuesday, May 14, 2024 2:50 PM
> > To: Frank Filz <ffilzlnx@mindspring.com>
> > Cc: Chuck Lever III <chuck.lever@oracle.com>; Linux NFS Mailing List <l=
inux-
> > nfs@vger.kernel.org>
> > Subject: Re: sm notify (nlm) question
> >
> > On Tue, May 14, 2024 at 5:36=E2=80=AFPM Frank Filz <ffilzlnx@mindspring=
.com> wrote:
> > >
> > > > > On May 14, 2024, at 2:56=E2=80=AFPM, Olga Kornievskaia <aglo@umic=
h.edu>
> > wrote:
> > > > >
> > > > > Hi folks,
> > > > >
> > > > > Given that not everything for NFSv3 has a specification, I post a
> > > > > question here (as it concerns linux v3 (client) implementation)
> > > > > but I ask a generic question with respect to NOTIFY sent by an NF=
S server.
> > > >
> > > > There is a standard:
> > > >
> > > > https://pubs.opengroup.org/onlinepubs/9629799/chap11.htm
> > > >
> > > >
> > > > > A NOTIFY message that is sent by an NFS server upon reboot has a
> > > > > monitor name and a state. This "state" is an integer and is
> > > > > modified on each server reboot. My question is: what about state
> > > > > value uniqueness? Is there somewhere some notion that this value
> > > > > has to be unique (as in say a random value).
> > > > >
> > > > > Here's a problem. Say a client has 2 mounts to ip1 and ip2 (both
> > > > > representing the same DNS name) and acquires a lock per mount. No=
w
> > > > > say each of those servers reboot. Once up they each send a NOTIFY
> > > > > call and each use a timestamp as basis for their "state" value --
> > > > > which very likely is to produce the same value for 2 servers
> > > > > rebooted at the same time (or for the linux server that looks lik=
e
> > > > > a counter). On the client side, once the client processes the 1st
> > > > > NOTIFY call, it updates the "state" for the monitor name (ie a
> > > > > client monitors based on a DNS name which is the same for ip1 and
> > > > > ip2) and then in the current code, because the 2nd NOTIFY has the
> > > > > same "state" value this NOTIFY call would be ignored. The linux
> > > > > client would never reclaim the 2nd lock (but the application
> > > > > obviously would never know it's missing a lock)
> > > > > --- data corruption.
> > > > >
> > > > > Who is to blame: is the server not allowed to send "non-unique"
> > > > > state value? Or is the client at fault here for some reason?
> > > >
> > > > The state value is supposed to be specific to the monitored host. I=
f
> > > > the client is indeed ignoring the second reboot notification, that'=
s incorrect
> > behavior, IMO.
> > >
> > > If you are using multiple server IP addresses with the same DNS name,=
 you
> > may want to set:
> > >
> > > sysctl fs.nfs.nsm_use_hostnames=3D0
> > >
> > > The NLM will register with statd using the IP address as name instead=
 of host
> > name. Then your two IP addresses will each have a separate monitor entr=
y and
> > state value monitored.
> >
> > In my setup I already have this set to 0. But I'll look around the code=
 to see what
> > it is supposed to do.
>
> Hmm, maybe it doesn't work on the client side. I don't often test NLM cli=
ents with my Ganesha work because I only run one VM and NLM clients can=E2=
=80=99t function on the same host as any server other than knfsd...

I've been staring and tracing the code and here's what I conclude: the
use of nsm_use_hostname toggles nothing that helps. No matter what
statd always stores whatever it is monitoring based on the DSN name
(looks like git blame says it's due to nfs-utils's commit
0da56f7d359475837008ea4b8d3764fe982ef512 "statd - use dnsname to
ensure correct matching of NOTIFY requests". Now what's worse is that
when statd receives a 2nd monitoring request from lockd for something
that maps to the same DNS name, statd overwrites the previous
monitoring information it had. When a NOTIFY arrives from an IP
matching the DNS name, the statd does the downcall and it will send
whatever the last monitoring information lockd gave it. Therefore all
the other locks will never be recovered.

What I struggle with is how to solve this problem. Say ip1 and ip2 run
an NFS server and both are known under the same DNS name: foo.bar.com.
Does it mean that they represent the "same" server? Can we assume that
if one of them "rebooted" then the other rebooted as well?  It seems
like we can't go backwards and go back to monitoring by IP. In that
case I can see that we'll get in trouble if the rebooted server indeed
comes back up with a different IP (same DNS name) and then it would
never match the old entry and the lock would never be recovered (but
then also I think lockd will only send the lock to the IP is stored
previously which in this case would be unreachable). If statd
continues to monitor by DNS name and then matches either ips to the
stored entry, then the problem comes with "state" update. Once statd
processes one NOTIFY which matched the DNS name its state "should" be
updated but then it would leads us back into the problem if ignoring
the 2nd NOTIFY call. If statd were to be changed to store multiple
monitor handles lockd asked to monitor, then when the 1st NOTIFY call
comes we can ask lockd to recover "all" the store handles. But then it
circles back to my question: can we assume that if one IP rebooted
does it imply all IPs rebooted?

Perhaps it's lockd that needs to change in how it keeps track of
servers that hold locks. The behaviour seems to have changed in 2010
(with commit 8ea6ecc8b0759756a766c05dc7c98c51ec90de37 "lockd: Create
client-side nlm_host cache") when nlm_host cache was introduced
written to be based on hash of IP. It seems that before things were
based on a DNS name making it in line with statd.

Anybody has any thoughts as to whether statd or lockd needs to change?