Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Frank Filz" <ffilzlnx@mindspring.com>
To:     "'Daire Byrne'" <daire@dneg.com>,
        "'Bruce Fields'" <bfields@fieldses.org>
Cc:     "'Chuck Lever III'" <chuck.lever@oracle.com>,
        "'Linux NFS Mailing List'" <linux-nfs@vger.kernel.org>
References: <20210921143259.GB21704@fieldses.org> <37851433-48C9-4585-9B68-834474AA6E06@oracle.com> <20210921160030.GC21704@fieldses.org> <CAPt2mGOf8orCkOj9hCM_sSu2uucPiRFPEK+yep+kufv-cDvhSA@mail.gmail.com>
In-Reply-To: <CAPt2mGOf8orCkOj9hCM_sSu2uucPiRFPEK+yep+kufv-cDvhSA@mail.gmail.com>
Subject: RE: [PATCH] nfs: reexport documentation
Date:   Wed, 22 Sep 2021 07:32:02 -0700
Message-ID: <01c901d7afbe$9cb5dbd0$d6219370$@mindspring.com>
MIME-Version: 1.0
Content-Type: text/plain;
        charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Thread-Index: AQHaPmxrNtjQ+kQkvcxH7DV+BmtjZwK315yQATRhzjcB4dNh36t8iYGA
Content-Language: en-us
Precedence: bulk


> -----Original Message-----
> From: Daire Byrne [mailto:daire@dneg.com]
> Sent: Wednesday, September 22, 2021 2:48 AM
> To: Bruce Fields <bfields@fieldses.org>
> Cc: Chuck Lever III <chuck.lever@oracle.com>; Linux NFS Mailing List =
<linux-
> nfs@vger.kernel.org>
> Subject: Re: [PATCH] nfs: reexport documentation
>=20
> On Tue, 21 Sept 2021 at 17:00, Bruce Fields <bfields@fieldses.org> =
wrote:
> >
> > > Any recommended workarounds? Or does this simply mean that
> > > administrators need to notify client users to unmount (or at least
> > > stop their workloads) before rebooting the proxy?
> >
> > I think so.
> >
> > If you don't use any file locking or delegations I suppose you're =
also
> > OK.  Delegations might be useful, though.
> >
> > I'd expect reexport to be useful mainly for data that changes very
> > rarely, if that helps.
> >
> > --b.
>=20
> Firstly, it's great to see this documentation and the well maintained =
wiki page
> for something we use in production (a lot) - thanks Bruce!
>=20
> I can only draw on our experience to say:
> * if the nfs re-export server doesn't crash, we rarely have cause to =
reboot it.
> * we re-export read-only software repositories to WAN/cloud instances =
(an
> ideal use case).
> * we also re-export read/write production storage but every client =
process is
> writing unique files - there are no writes to the same
> file(s) from any clients of the re-export server.
>=20
> We don't use or need crossmnt functionality, but I know from chatting =
to others
> within our industry that the fsid/crossmnt limitation causes them the =
most grief
> and confusion. I think in the case of Netapps, there are similar =
problems with
> trying to re-export a namespace made up of different volumes?
>=20
> As noted on the wiki, the only way around that is probably to have a =
"proxy"
> mode (similar to what ganesha does?).

I'm glad to see this documentation also. It helps to have a common =
understanding of the challenges of re-export.

Ganesha's proxy mode would be different from what Bruce is proposing due =
to how Ganesha constructs handles. It adds an export ID to the export =
configuration which leads to the specific back end (FSAL) that is =
exporting that handle. It may or may not also encode an fsid into the =
handle (proxy does so only to the extent the re-exported server put an =
fsid into the handle, Ganesha's proxy doesn't distinguish between the =
filesystems on the re-exported server). Ganesha can proxy and serve =
local file systems because it's proxy isn't a re-export of an NFS mount, =
but a separate back end module (two actually, one for NFSv3 and one for =
NFSv4) that talk directly to the proxied server while local file systems =
resolve handles pretty much the same way knfsd does (use the fsid to =
find the right filesystem and then use open_by_handle to find the inode =
within that filesystem - open_by_handle is a user space interface to the =
same exportfs interface that knfsd uses, with all the same limitations).

What Bruce is proposing is a bit more like a proxy mode I have =
considered for Ganesha that would allow both a proxied Ganesha server =
and the Ganesha proxy to use the same handles. Basically the export IDs =
would be shared between the two servers, though even that could be =
constructed in a way to allow multiple proxied servers (they just would =
be required to have non-overlapping export IDs) as well as still export =
local file systems.

Frank