Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CAD8zhTCcHTXsrhqYWZRAK9NZLFg4oZUzia8Uh-quGgZSjscMaw@mail.gmail.com>
In-Reply-To: <CAD8zhTCcHTXsrhqYWZRAK9NZLFg4oZUzia8Uh-quGgZSjscMaw@mail.gmail.com>
From:   Pradeep <pradeepthomas@gmail.com>
Date:   Tue, 20 Sep 2022 06:41:34 -0700
Message-ID: <CAD8zhTCEfUycDT2jKJG4TNVGFUqe2J-=7On=vyypkA=eGFcGZw@mail.gmail.com>
Subject: Re: EREMOTE error on NFSv4 CREATE RPC with latest kernel.
To:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Any thoughts or suggestions on this issue?

Thanks

On Mon, Sep 12, 2022 at 7:29 AM Pradeep <pradeepthomas@gmail.com> wrote:
>
> Hello,
>
> We have created a referral-based =E2=80=9Cdistributed=E2=80=9D namespace =
spread across
> multiple Linux nodes using the local file system. The server
> implementation uses nfs-ganesha. Within an NFSv4 export, we place
> top-level directories (under export root) on different nodes for =E2=80=
=9Cload
> balancing=E2=80=9D. These directories simulate separate filesystems (sepa=
rate
> fsids) and return fs_locations when queried by the client. Note that
> currently, the Linux NFS client only queries for fs_locations
> attribute (handling of NFS4ERR_MOVED) on a =E2=80=9Clookup=E2=80=9D VFS o=
peration.
>
> To allow the client to be able to create top-level directories, we
> must ensure that the mkdir succeeds (since the client cannot handle
> NFS4ERR_MOVED on CREATE), but the directory created to be potentially
> placed on a different node than the one client is connecting to. To do
> this, we internally forward the mkdir request to another node where it
> gets created.
>
> Until recently, we were able to return a zero-byte filehandle in the
> GETFH response for CREATE (a compound of PUTFH, CREATE, GETFH,
> GETATTR). This forces the Linux client to issue a LOOKUP on the
> directory name, get an NFS4ERR_MOVED in response and subsequently get
> redirected to the correct node.
>
> See code here for reference that forces a LOOKUP on zero byte FH:
> https://elixir.bootlin.com/linux/latest/source/fs/nfs/dir.c#L2237
>
> Of course, this isn=E2=80=99t documented in the protocol, although the
> protocol does not explicitly disallow =E2=80=9Cjunctions=E2=80=9D to be c=
reated by the
> client.
>
> Trond=E2=80=99s recent change on file handle validation in RPC decoding l=
ayer
> now makes the zero-byte file handle invalid - see
> https://git.linux-nfs.org/?p=3Dtrondmy/linux-nfs.git;a=3Dcommitdiff;h=3De=
b3d58c68e39fad68d8054e0324eb06d82dcedbb;hp=3D3d66bae156a652be18e278f3c88bc3=
e069ae824b.
>
> This is probably done (correctly) for other reasons. However, it does
> prevent the client from =E2=80=9Crecovering=E2=80=9D the filehandle of a =
newly created
> directory through a subsequent lookup. Other options like returning a
> correct filehandle (which we could obtain from the remote node) would
> not work, since the client is not informed of fs_locations. Similarly,
> NFS4ERR_BADHANDLE or NFS4ERR_STALE will not perform any =E2=80=9Crecovery=
=E2=80=9D
> since the directory isn=E2=80=99t known yet to have a separate fsid.
>
> Suggestions on how/if we can support junction creates from the client?
>
> Thanks,
> Pradeep