Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CAD15GZf0FtU81hwQ+brhnt+sv895=TpAuz-YrMtjfx__FJ28Gg@mail.gmail.com>
 <8ae13798a15c69cf16272579f49768ec92484584.camel@hammerspace.com>
 <CAD15GZfmpdzMhXquciebs5M4e2kewu+yTUcTx9c-jeSXgZS+XQ@mail.gmail.com>
 <668b5de2f3951f0d64aa10e910a8aa3d626bec91.camel@hammerspace.com>
 <ec225634-44cd-aeeb-4919-956e68b249f0@talpey.com> <CAD15GZdsPtLuhRSWYkLgfVzwhp8QEDN2VQ89AeyUwjJRTshT7g@mail.gmail.com>
In-Reply-To: <CAD15GZdsPtLuhRSWYkLgfVzwhp8QEDN2VQ89AeyUwjJRTshT7g@mail.gmail.com>
From:   Jan Kasiak <j.kasiak@gmail.com>
Date:   Tue, 9 Aug 2022 10:16:26 -0400
Message-ID: <CAD15GZdMAjdSwgV6hMopF1L1jJFRr+h8gm41MjLRr=PVKeTrwg@mail.gmail.com>
Subject: Re: Question about nlmclnt_lock
To:     Tom Talpey <tom@talpey.com>
Cc:     Trond Myklebust <trondmy@hammerspace.com>,
        "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

Sorry, I meant FreeBSD uses the caller field as well.

-Jan

On Tue, Aug 9, 2022 at 10:14 AM Jan Kasiak <j.kasiak@gmail.com> wrote:
>
> Thanks for all of the resources!
>
> I was trying to implement an NFS server, and v3 sounded like an easier
> place to start :-)
>
> I think I'll move on to v4.
>
> If we're revisiting the past, maybe just one last historical question:
>
> Do either of you know why the Linux Kernel only uses the IP
> address/svid to identify the caller?
>
> FreeBSD uses the owner field as well.
>
> Jan
>
> On Sun, Aug 7, 2022 at 8:01 AM Tom Talpey <tom@talpey.com> wrote:
> >
> > On 8/6/2022 3:49 PM, Trond Myklebust wrote:
> > > On Sat, 2022-08-06 at 11:03 -0400, Jan Kasiak wrote:
> > >> Hi Trond,
> > >>
> > >> The v4 RFCs do mention protocol design flaws, but don't go into more
> > >> detail.
> > >>
> > >> I was trying to understand those flaws in order to understand how and
> > >> why v3 was problematic.
> > >>
> > >>
> > >
> > > The main issues derive from the fact that NLM is a side band protocol,
> > > meaning that it has no ability to influence the NFS protocol
> > > operations. In particular, there is no way to ensure safe ordering of
> > > locks and I/O. e.g. if your readahead code kicks in while you are
> > > unlocking the file, then there is nothing that guarantees the page
> > > reads happened while the lock was in place on the server.
> > > The same weakness also causes problems for reboots: if your client
> > > doesn't notice that the server rebooted (and lost your locks) because
> > > the statd callback mechanism failed, then you're SOL. Your I/O may
> > > succeed, but can end up causing problems for another client that has
> > > since grabbed the lock and assumes it now has exclusive access to the
> > > file.
> > >
> > > NLM also suffers from intrinsic problems of its own such as lack of
> > > only-once semantics. If you send a blocking LOCK request, and
> > > subsequently send a CANCEL operation, then who knows whether or not the
> > > lock or the cancel get processed first by the server? Many servers will
> > > reply LCK_GRANTED to the CANCEL even if they did not find the lock
> > > request. Sending an UNLOCK can also cause issues if the lock was
> > > granted via a blocking lock callback (NLM_GRANTED) since there is no
> > > ordering between the reply to the NLM_GRANTED and the UNLOCK.
> > >
> > > Finally, as already mentioned, there are multiple issues associated
> > > with client or server reboot. The NLM mechanism is pretty dependent on
> > > yet another side band mechanism (STATD) to tell you when this occurs,
> > > but that mechanism does not work to release the locks held by a client
> > > if it fails to come back after reboot. Even if the client does come
> > > back, it might forget to invoke the statd process, or it might use a
> > > different identifier than it did during the last boot instance (e.g.
> > > because DHCP allocated a different IP address, or the IP address it not
> > > unique due to use of NAT, or a hostname was used that is non-unique,
> > > ...).
> > > If the server reboots, then it may fail to notify the client of that
> > > reboot through the callback mechanism. Reasons may include the
> > > existence of a NAT, failure of the rpcbind/portmapper process on the
> > > client, firewalls,...
> >
> > That brought back memories.
> >
> > http://www.nfsv4bat.org/Documents/ConnectAThon/2006/talpey-cthon06-nsm.pdf
> >
> > Here's an even older issues list for nlm on Solaris circa 1996.
> > The portrait-mode slides are in reverse order. :)
> >
> > http://www.nfsv4bat.org/Documents/ConnectAThon/1996/lockmgr.pdf
> >
> > The NLM protocol is an antique and hasn't been looked at in well
> > over a decade (or two!). NLMv4 (circa 1995) widened offsets to
> > 64-bit, which was the last innovation it got. None of the RPC
> > sideband protocols were ever standardized, btw.
> >
> > Jan, what are you planning to use it for? Personally I'd advise
> > against pretty much anything.
> >
> > Tom.
> >
> > >
> > >> -Jan
> > >>
> > >>
> > >> On Fri, Aug 5, 2022 at 10:27 PM Trond Myklebust
> > >> <trondmy@hammerspace.com> wrote:
> > >>>
> > >>> On Fri, 2022-08-05 at 19:17 -0400, Jan Kasiak wrote:
> > >>>> Hi,
> > >>>>
> > >>>> I was looking at the code for nlmclnt_lock and wanted to ask a
> > >>>> question about how the Linux kernel client and the NLM 4 protocol
> > >>>> handle some errors around certain edge cases.
> > >>>>
> > >>>> Specifically, I think there is a race condition around two
> > >>>> threads of
> > >>>> the same program acquiring a lock, one of the threads being
> > >>>> interrupted, and the NFS client sending an unlock when none of
> > >>>> the
> > >>>> program threads called unlock.
> > >>>>
> > >>>> On NFS server machine S:
> > >>>> there exists an unlocked file F
> > >>>>
> > >>>> On NFS client machine C:
> > >>>> in program P:
> > >>>> thread 1 tries to lock(F) with fd A
> > >>>> thread 2 tries to lock(F) with fd B
> > >>>>
> > >>>> The Linux client will issue two NLM_LOCK calls with the same svid
> > >>>> and
> > >>>> same range, because it uses the program id to map to an svid.
> > >>>>
> > >>>> For whatever reason, assume the connection is broken (cable gets
> > >>>> pulled etc...)
> > >>>> and `status = nlmclnt_call(cred, req, NLMPROC_LOCK);` fails.
> > >>>>
> > >>>> The Linux client will retry the request, but at some point thread
> > >>>> 1
> > >>>> receives a signal and nlmclnt_lock breaks out of its loop.
> > >>>> Because
> > >>>> the
> > >>>> Linux client request failed, it will fall through and go to the
> > >>>> out_unlock label, where it will want to send an unlock request.
> > >>>>
> > >>>> Assume that at some point the connection is reestablished.
> > >>>>
> > >>>> The Linux kernel client now has two outstanding lock requests to
> > >>>> send
> > >>>> to the remote server: one for a lock that thread 2 is still
> > >>>> trying to
> > >>>> acquire, and one for an unlock of thread 1 that failed and was
> > >>>> interrupted.
> > >>>>
> > >>>> I'm worried that the Linux client may first send the lock
> > >>>> request,
> > >>>> and
> > >>>> tell thread 2 that it acquired the lock, and then send an unlock
> > >>>> request from the cancelled thread 1 request.
> > >>>>
> > >>>> The server will successfully process both requests, because the
> > >>>> svid
> > >>>> is the same for both, and the true server side state will be that
> > >>>> the
> > >>>> file is unlocked.
> > >>>>
> > >>>> One can talk about the wisdom of using multiple threads to
> > >>>> acquire
> > >>>> the
> > >>>> same file lock, but this behavior is weird, because none of the
> > >>>> threads called unlock.
> > >>>>
> > >>>> I have experimented with reproducing this, but have not been
> > >>>> successful in triggering this ordering of events.
> > >>>>
> > >>>> I've also looked at the code of in clntproc.c and I don't see a
> > >>>> spot
> > >>>> where outstanding failed lock/unlock requests are checked while
> > >>>> processing lock requests?
> > >>>>
> > >>>> Thanks,
> > >>>> -Jan
> > >>>
> > >>> Nobody here is likely to want to waste much time trying to 'fix'
> > >>> the
> > >>> NLM locking protocol. The protocol itself is known to be extremely
> > >>> fragile, and the endemic problems constitute some of the main
> > >>> motivations for the development of the NFSv4 protocol
> > >>> (See https://datatracker.ietf.org/doc/html/rfc2624#section-8
> > >>> and https://datatracker.ietf.org/doc/html/rfc7530#section-9).
> > >>>
> > >>> If you need more reliable support for POSIX locks beyond what
> > >>> exists
> > >>> today for NLM, then please consider NFSv4.
> > >>>
> > >>> --
> > >>> Trond Myklebust
> > >>> Linux NFS client maintainer, Hammerspace
> > >>> trond.myklebust@hammerspace.com
> > >>>
> > >>>
> > >