Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Bruce Fields <bfields@fieldses.org>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH] nfs: reexport documentation
Thread-Topic: [PATCH] nfs: reexport documentation
Thread-Index: AQHXrvWaAWSTDh3zBEqQtD2/lsKJM6uujxeA
Date:   Tue, 21 Sep 2021 14:39:39 +0000
Message-ID: <37851433-48C9-4585-9B68-834474AA6E06@oracle.com>
References: <20210921143259.GB21704@fieldses.org>
In-Reply-To: <20210921143259.GB21704@fieldses.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?c3jrwxSbxWXGRlxdrCabH6YhIc+lKSdEEWRXM6fadfbV9LQQALm4xGkHZ13N?=
 =?us-ascii?Q?z7KbkHykEtQzaetiyavG9HKh+vbYFoB/igYKZZ+9wk+c1I6T4ajcf1oiuTEv?=
 =?us-ascii?Q?M2OCXFuKQUlcGBtvNU0LLX/HsiDElN7M3R7wOmIZ+sjtNCorNVhFXnoelfV4?=
 =?us-ascii?Q?dh5jrvU6zwee3/GWTiGIwoansuRfb4umXt0E14eub5XHml48ZUQAz/HOpumD?=
 =?us-ascii?Q?eYqBYCbW0sYRNIxwd7oNZJ6dLUONh95E+SoVNqcVoM7FA7ths90x5bKGkl+g?=
 =?us-ascii?Q?+1JgSfBkVNA9xSkSjJ65qNbLwFRHqaSubmqinVXf+J20ZZ2LFN6e9vssYwDy?=
 =?us-ascii?Q?ZaZyRUauU+acSBhodgd1Hm/h7aQwDpVe++JvIwFZhZiUXyaRGh7QP8QKZJM4?=
 =?us-ascii?Q?bJMDbbJ/VA9aOi4/vHTM+1FiJ69t34tM9euiFjk2Be1uIfvYw8CVvSneE2W6?=
 =?us-ascii?Q?6T7xsIEEFyaPf7eFF6Bxf2OJ+1H7rc6uKBb1f3jVc61n98Lj9Ue9sFl+Owbc?=
 =?us-ascii?Q?n/D2OGKfVv8er8DhOyFmqJUT78A5gpxgE6wEh2jDskLqN5agmodu8MCr3xMW?=
 =?us-ascii?Q?sohCDnSJxEKzO+zq+3mUKNYZct10KvkfMuyyF+cc7NsQhph427e351/SMohe?=
 =?us-ascii?Q?3okJL9maDhjHTOaQQXeIx/1D8YjpEwMm3+etjfm7ohNp5tPb0zbPOM/RHHJ0?=
 =?us-ascii?Q?7wPwE5mpuGBPE+th1kCoB3pntcgE2HZvxObPGLZY1eSuAzEJXMXugcUKoNVU?=
 =?us-ascii?Q?9SvnPGed0Z4rdiG0of+62lrvzb26h0f7FSCVnhBDYGla2In2qQb8k6xtJCuw?=
 =?us-ascii?Q?cOTKF5gmZZaK3A0De9Z34pGvFieyGTvDu3az5xB5hHF3+/aIG8pFP/RTnhPB?=
 =?us-ascii?Q?GToRozvdGz2TysYLeHRUi4jB0oYB9kJg6iIJ5OUsn+HBilhARJ5dLpe7IJM5?=
 =?us-ascii?Q?l9eOuTIX/mkSZRCBBvrAzg2iBC8xO7CbAfDYYzuyBjimIjUfZAIAJZCgNKXN?=
 =?us-ascii?Q?At24DQtFUUDXEo295ZoUi3B5G/Lwva41pmXmVoxauTuGXLAE2j+OJE9VU1j0?=
 =?us-ascii?Q?+XAac8hF+3FhNX5lUvQr8lKs6apgqmE9AC4U3+2KutjusSdKS6GVFhG0iqnI?=
 =?us-ascii?Q?1aIHOcLbBHkFnNmG4LU4dpprdfBEjG+6zPf5XoKMKL+bscUs8SPfdi7muHKD?=
 =?us-ascii?Q?AOFOlz0CU4ow9OE64JjYRBaU0zcsbh7jXw3Pj1ximA+KbQ04JUcvfpUeYv8C?=
 =?us-ascii?Q?C2kSvSMDwnvl3Jzupo5jyuDTkuxRygLwSfiihZ8/8Rv+vTkOC+xGamNrJ+gV?=
 =?us-ascii?Q?Orh+2ysECo0Y9l+eF8BVO53l?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="us-ascii"
Content-ID: <3600289FBFAA774CAE1E6E8BBF433E8F@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6c3628eb-3895-4f33-42a7-08d97d0da4a0
X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Sep 2021 14:39:39.8663
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: qlXFJtMifWD/w+lZ0ZSUJiOYT/Ngjp+7h5GeJOUYnnQ/cQixsTHm0wOQH8f+Wmq5qXX8NGTBAGO3GRSv09PAPw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3953
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10114 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 adultscore=0
 phishscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109030001
 definitions=main-2109210089
X-Proofpoint-GUID: wuPNyfoBsM_y0DTHaZ2OrDojqk1ZdBVi
X-Proofpoint-ORIG-GUID: wuPNyfoBsM_y0DTHaZ2OrDojqk1ZdBVi
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Sep 21, 2021, at 10:32 AM, J. Bruce Fields <bfields@fieldses.org> wrot=
e:
>=20
> From: "J. Bruce Fields" <bfields@redhat.com>
>=20
> We've supported reexport for a while but documentation is limited.  This
> is mainly a simplified version of the text I wrote for the linux-nfs
> wiki at https://wiki.linux-nfs.org/wiki/index.php/NFS_re-export.
>=20
> Signed-off-by: J. Bruce Fields <bfields@redhat.com>

Thanks for posting this, Bruce! Comments inline.


> ---
> Documentation/filesystems/nfs/index.rst    |   1 +
> Documentation/filesystems/nfs/reexport.rst | 110 +++++++++++++++++++++
> 2 files changed, 111 insertions(+)
> create mode 100644 Documentation/filesystems/nfs/reexport.rst
>=20
> diff --git a/Documentation/filesystems/nfs/index.rst b/Documentation/file=
systems/nfs/index.rst
> index 65805624e39b..288d8ddb2bc6 100644
> --- a/Documentation/filesystems/nfs/index.rst
> +++ b/Documentation/filesystems/nfs/index.rst
> @@ -11,3 +11,4 @@ NFS
>    rpc-server-gss
>    nfs41-server
>    knfsd-stats
> +   reexport
> diff --git a/Documentation/filesystems/nfs/reexport.rst b/Documentation/f=
ilesystems/nfs/reexport.rst
> new file mode 100644
> index 000000000000..892cb1e9c45c
> --- /dev/null
> +++ b/Documentation/filesystems/nfs/reexport.rst
> @@ -0,0 +1,110 @@
> +Reexporting NFS filesystems
> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D
> +
> +Overview
> +--------
> +
> +It is possible to reexport an NFS filesystem over NFS.  However, this
> +feature comes with a number of limitations.  Before trying it, we
> +recommend some careful research to determine wether it will work for
> +your purposes.

^wether^whether


> +
> +A discussion of current known limitations follows.
> +
> +"fsid=3D" required, crossmnt broken
> +---------------------------------
> +
> +We require the "fsid=3D" export option on any reexport of an NFS
> +filesystem.

Recommended approach? I would just say use 'uuidgen -r'


> +The "crossmnt" export option does not work in the reexport case.

Can you expand on this a little? Consequences? Risks?


> +Reboot recovery
> +---------------
> +
> +The NFS protocol's normal reboot recovery mechanisms don't work for the
> +case when the reexport server reboots.  Clients will lose any locks
> +they held before the reboot, and further IO will result in errors.
> +Closing and reopening files should clear the errors.

Any recommended workarounds? Or does this simply mean that
administrators need to notify client users to unmount (or
at least stop their workloads) before rebooting the proxy?


> +Filehandle limits
> +-----------------
> +
> +If the original server uses an X byte filehandle for a given object, the
> +reexport server's filehandle for the reexported object will be X+22
> +bytes, rounded up to the nearest multiple of four bytes.
> +
> +The result must fit into the RFC-mandated filehandle size limits:
> +
> ++-------+-----------+
> +| NFSv2 |  32 bytes |
> ++-------+-----------+
> +| NFSv3 |  64 bytes |
> ++-------+-----------+
> +| NFSv4 | 128 bytes |
> ++-------+-----------+
> +
> +So, for example, you will only be able to reexport a filesystem over
> +NFSv2 if the original server gives you filehandles that fit in 10
> +bytes--which is unlikely.
> +
> +In general there's no way to know the maximum filehandle size given out
> +by an NFS server without asking the server vendor.
> +
> +But the following table gives a few examples.  The first column is the
> +typical length of the filehandle from a Linux server exporting the given
> +filesystem, the second is the length after that nfs export is reexported
> +by another Linux host:
> +
> ++--------+-------------------+----------------+
> +|        | filehandle length | after reexport |
> ++=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+
> +| ext4:  | 28 bytes          | 52 bytes       |
> ++--------+-------------------+----------------+
> +| xfs:   | 32 bytes          | 56 bytes       |
> ++--------+-------------------+----------------+
> +| btrfs: | 40 bytes          | 64 bytes       |
> ++--------+-------------------+----------------+
> +
> +All will therefore fit in an NFSv3 or NFSv4 filehandle after reexport,
> +but none are reexportable over NFSv2.
> +
> +Linux server filehandles are a bit more complicated than this, though;
> +for example:
> +
> +        - The (non-default) "subtreecheck" export option generally
> +          requires another 4 to 8 bytes in the filehandle.
> +        - If you export a subdirectory of a filesystem (instead of
> +          exporting the filesystem root), that also usually adds 4 to 8
> +          bytes.
> +        - If you export over NFSv2, knfsd usually uses a shorter
> +          filesystem identifier that saves 8 bytes.
> +        - The root directory of an export uses a filehandle that is
> +          shorter.
> +
> +As you can see, the 128-byte NFSv4 filehandle is large enough that
> +you're unlikely to have trouble using NFSv4 to reexport any filesystem
> +exported from a Linux server.  In general, if the original server is
> +something that also supports NFSv3, you're *probably* OK.  Re-exporting
> +over NFSv3 may be dicier, and reexporting over NFSv2 will probably
> +never work.
> +
> +For more details of Linux filehandle structure, the best reference is
> +the source code and comments; see in particular:
> +
> +        - include/linux/exportfs.h:enum fid_type
> +        - include/uapi/linux/nfsd/nfsfh.h:struct nfs_fhbase_new
> +        - fs/nfsd/nfsfh.c:set_version_and_fsid_type
> +        - fs/nfs/export.c:nfs_encode_fh
> +
> +Open DENY bits ignored
> +----------------------
> +
> +NFS since NFSv4 supports ALLOW and DENY bits taken from Windows, which
> +allow you, for example, to open a file in a mode which forbids other
> +read opens or write opens. The Linux client doesn't use them, and the
> +server's support has always been incomplete: they are enforced only
> +against other NFS users, not against processes accessing the exported
> +filesystem locally. A reexport server will also not pass them along to
> +the original server, so they will not be enforced between clients of
> +different reexport servers.
> --=20
> 2.31.1
>=20

--
Chuck Lever