Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
From:   Pradeep <pradeepthomas@gmail.com>
Date:   Mon, 12 Sep 2022 07:29:14 -0700
Message-ID: <CAD8zhTCcHTXsrhqYWZRAK9NZLFg4oZUzia8Uh-quGgZSjscMaw@mail.gmail.com>
Subject: EREMOTE error on NFSv4 CREATE RPC with latest kernel.
To:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Hello,

We have created a referral-based =E2=80=9Cdistributed=E2=80=9D namespace sp=
read across
multiple Linux nodes using the local file system. The server
implementation uses nfs-ganesha. Within an NFSv4 export, we place
top-level directories (under export root) on different nodes for =E2=80=9Cl=
oad
balancing=E2=80=9D. These directories simulate separate filesystems (separa=
te
fsids) and return fs_locations when queried by the client. Note that
currently, the Linux NFS client only queries for fs_locations
attribute (handling of NFS4ERR_MOVED) on a =E2=80=9Clookup=E2=80=9D VFS ope=
ration.

To allow the client to be able to create top-level directories, we
must ensure that the mkdir succeeds (since the client cannot handle
NFS4ERR_MOVED on CREATE), but the directory created to be potentially
placed on a different node than the one client is connecting to. To do
this, we internally forward the mkdir request to another node where it
gets created.

Until recently, we were able to return a zero-byte filehandle in the
GETFH response for CREATE (a compound of PUTFH, CREATE, GETFH,
GETATTR). This forces the Linux client to issue a LOOKUP on the
directory name, get an NFS4ERR_MOVED in response and subsequently get
redirected to the correct node.

See code here for reference that forces a LOOKUP on zero byte FH:
https://elixir.bootlin.com/linux/latest/source/fs/nfs/dir.c#L2237

Of course, this isn=E2=80=99t documented in the protocol, although the
protocol does not explicitly disallow =E2=80=9Cjunctions=E2=80=9D to be cre=
ated by the
client.

Trond=E2=80=99s recent change on file handle validation in RPC decoding lay=
er
now makes the zero-byte file handle invalid - see
https://git.linux-nfs.org/?p=3Dtrondmy/linux-nfs.git;a=3Dcommitdiff;h=3Deb3=
d58c68e39fad68d8054e0324eb06d82dcedbb;hp=3D3d66bae156a652be18e278f3c88bc3e0=
69ae824b.

This is probably done (correctly) for other reasons. However, it does
prevent the client from =E2=80=9Crecovering=E2=80=9D the filehandle of a ne=
wly created
directory through a subsequent lookup. Other options like returning a
correct filehandle (which we could obtain from the remote node) would
not work, since the client is not informed of fs_locations. Similarly,
NFS4ERR_BADHANDLE or NFS4ERR_STALE will not perform any =E2=80=9Crecovery=
=E2=80=9D
since the directory isn=E2=80=99t known yet to have a separate fsid.

Suggestions on how/if we can support junction creates from the client?

Thanks,
Pradeep