Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   "crispyduck@outlook.at" <crispyduck@outlook.at>
To:     "J. Bruce Fields" <bfields@fieldses.org>,
        Rick Macklem <rmacklem@uoguelph.ca>
CC:     Chuck Lever III <chuck.lever@oracle.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: AW: Problems with NFS4.1 on ESXi
Thread-Topic: Problems with NFS4.1 on ESXi
Thread-Index: AQHYVOLTJWvLLebuJE+8fGO0gJ9faaz5zdx5gACo0QCAAAJi6IAAGkiAgAAlUYCAAE6KrIABBqgAgAA5RS4=
Date:   Fri, 22 Apr 2022 18:43:23 +0000
Message-ID: <AM9P191MB1665D6D3AF49AF0DFCAD31698EF79@AM9P191MB1665.EURP191.PROD.OUTLOOK.COM>
References: <AM9P191MB1665484E1EFD2088D22C2E2F8EF59@AM9P191MB1665.EURP191.PROD.OUTLOOK.COM>
 <AM9P191MB16655E3D5F3611D1B40457F08EF49@AM9P191MB1665.EURP191.PROD.OUTLOOK.COM>
 <4D814D23-58D6-4EF0-A689-D6DD44CB2D56@oracle.com>
 <AM9P191MB16651F3A158CAED8F358602A8EF49@AM9P191MB1665.EURP191.PROD.OUTLOOK.COM>
 <20220421164049.GB18620@fieldses.org> <20220421185423.GD18620@fieldses.org>
 <YT2PR01MB973028EFA90F153C446798C1DDF49@YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM>
 <20220422151534.GA29913@fieldses.org>
In-Reply-To: <20220422151534.GA29913@fieldses.org>
Accept-Language: de-AT, en-US
Content-Language: de-AT
suggested_attachment_session_id: 3b2abf01-1213-80f2-2049-384f91ed5c66
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn:  [dpq3kt2Ul89QrvoB67Ej40NkDyzR1tsr]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 6248630d-8b4b-46bf-3a48-08da248ffadd
x-ms-traffictypediagnostic: AM9P191MB1794:EE_
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: OFvH3jlTU25SfXMpWWfE/RIdI1e1LJrZR+U5xWdFbqkedi6FbZzILp1wZ7R0DB8EcJGbdb0oP06y4oce7Ckhrys5Fl2mIbzFfB5ltnxADFgRxZXMis7SM4+ppnRQG24tGQqoqMsiTpeTtUw9BtwKGJontl6O0J0oaOQlqO7/ylVu0F5zGewMA+T9XNH5mdWqFV9GrcEjxhFQzAwQKB9sc5BLb4fDvYkgn2Y3Mi6tNZX/KQDYhq7oBHEJVEBogFncItszgRCnZpcxmMTEZrYBRZKLATNNxJufXglV9TZupZPSPetJRWcTd1tJR7UZBYJTwHc+mA+yO3DwjVcf0/2aU1oPmcovPXOaXFB4Z+VeSQX2w2AMqio1Q3iE0Gl5l2KAApRwIYiFsaDj6Okt4oehQSppxW8o72xFFDYRhvfSkRTvx6RYMyndRMYDPhyzwIm/oqr2y4+hpof8s42h2tZ0Ddbtfd2LpA4G1d87T8FNbwHFmTJVeT1DTHU6PrcpU+SeX4YJpN2qJiM4hwVpmNoHnVlz3ti1o67yqVdzFJWEMT4+aJ8eBYk2AlwUfkMSCnjgR8x5NP44XnnKXOxEkWDgOz9yyFS7YVyBP1CwUp+qfiy/MgxuJpsT0MCe30YvC0tV
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?1OwYdOe7/DD1TKABcePCxxEKsGwyZAG+mv/YuWtoCmzQ1mXLv7/WZ2jZqx?=
 =?iso-8859-1?Q?k49wdTnaNLjvcMfumTC+hPB9FtduJ1oM25JeU53otYrJkUYvonTqfF/76N?=
 =?iso-8859-1?Q?4QVmXzdqJ55mvluYC2jFg1yFmpG6o33oriJas3+QhJ7KkE11x0Zdtld+32?=
 =?iso-8859-1?Q?LNgN00PRJLWeNTriAXFcvROWAos0bHm2xmkNP3SINTaM5Li6keJFwHXIc9?=
 =?iso-8859-1?Q?4CAyzcY3jDwxc+simojAwZ9dRBvNr3x6t2h69YLCneZSvYJSN6EsK7wUdS?=
 =?iso-8859-1?Q?Xmkrb3zBUTe5KEtLiAyO77/fhidkzqndoFwaZKRQX9otVi2k41WBjyb+tY?=
 =?iso-8859-1?Q?yaTVpqWCVaZJYQ4RNG/5hJYUqd5WIc7YiXxjplFRWEqDAiZ+HMSxf6SfWR?=
 =?iso-8859-1?Q?lf5X6j2ZE/IScTfuaOpR21g4pA9apzmEGSelQNBnspce+w3qi3GZQidgw2?=
 =?iso-8859-1?Q?5eEOniZSsDppPpImUwwWNJ5rEK8dsn4S0Mv7s8W1bRJs7ffJwb8T2VTGmo?=
 =?iso-8859-1?Q?g6VwXF/GZJYxdg5elHeR4sGLpT4dUJ2T6/5aJpBj4VtbgkkbjsUyLdfAkl?=
 =?iso-8859-1?Q?WDSdg+X7EzMclfiyEtl0mLD6sZawZAdvQnx8ajBAhniy3Ha51IrZKMcCKF?=
 =?iso-8859-1?Q?IlbZQA06YsIr4N+dWN2uVo99u5Crxb/+3P1TPJS+p0OAomp1mtsClydzhu?=
 =?iso-8859-1?Q?x8SOQvVvYJEPFtb8pnP7JndbUktEMgTP9kd7klUGcnYCJc9tS3VaCFGO3M?=
 =?iso-8859-1?Q?U6i95DydPi2WMUn1J8gHurqdt5hNxCaJ6Uv1m0gW8cWcJ/GKcqJ6Afa7c8?=
 =?iso-8859-1?Q?bCbXwICMP9N7EyLcPopd3m32rX0eX7j9ESnVtOB7hEqWZuGpo5OjRg43l5?=
 =?iso-8859-1?Q?DJNZorkeRTSuvt/vVY5BOOTt+aci/uSlrq716cfDaINV9rw7vQTALZK7wo?=
 =?iso-8859-1?Q?winI0hP0mcZTZ8upFNfb3KyQ1PbuQXYflh6SdV3xYtPeIElFPARnKvUdf9?=
 =?iso-8859-1?Q?u1vc7nVxSjrJ4LhKTOsYVqhFK2cv7S/5NIAAs/oszAOQUradbeSzpIh2IP?=
 =?iso-8859-1?Q?LrVNu96Eav/5wcyvzgdaGahhrXpl1gvkhstVVWhY8/AuFEuvWYCdztk2z7?=
 =?iso-8859-1?Q?W13g/FtHK2ZJhYAUCKDpqHaXibhh2crtLpy5C8BNa2vJaX8MD5QWgQgvqX?=
 =?iso-8859-1?Q?aUVDDqO0GiCbVurHLN6QVQJ+lf4qSAUmxCnEU5cHeOGrKq4ETXytzWau/T?=
 =?iso-8859-1?Q?qwouNG7Ehsp2oCFpA8XxHI2D+i+zinb+w5FR5rgmGi2A05Iq2Qia8rJKRj?=
 =?iso-8859-1?Q?u+yoYlPSi07UstVK2iPLmRqc/zrRGbZ1ci3wJGwVDcmpuJEHdLkomObaAt?=
 =?iso-8859-1?Q?MCXHlOSXaYsZkkzU0ogFri7eZRCnPe3Fdqt8s1BlwfVJf0c1XivVGVT+2a?=
 =?iso-8859-1?Q?OlVo92b6EydYc/vSixK2QDw7FD27Rn6OAvnPWURmi/u0gPidoy9Mpk+WXl?=
 =?iso-8859-1?Q?E=3D?=
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: 8BIT
MIME-Version: 1.0
X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-50200.templateTenant
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: AM9P191MB1665.EURP191.PROD.OUTLOOK.COM
X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-CrossTenant-Network-Message-Id: 6248630d-8b4b-46bf-3a48-08da248ffadd
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Apr 2022 18:43:23.3603
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM9P191MB1794
X-Spam-Status: No, score=-1.6 required=5.0 tests=BAYES_00,
	FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,SPOOFED_FREEMAIL_NO_RDNS
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Output of exportfs -v:
sync,wdelay,hide,crossmnt,no_subtree_check,fsid=74345722,mountpoint,sec=sys,rw,secure,no_root_squash,no_all_squash

ESXi only supports NFS3 and NFS4.1, NFS4 is not supported, no idea why, think thy only somehow implemented 4.1 for session trunking and kerberos.

Br,
Andreas


Von: J. Bruce Fields <bfields@fieldses.org>
Gesendet: Freitag, 22. April 2022 17:15
An: Rick Macklem <rmacklem@uoguelph.ca>
Cc: crispyduck@outlook.at <crispyduck@outlook.at>; Chuck Lever III <chuck.lever@oracle.com>; Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Betreff: Re: Problems with NFS4.1 on ESXi 
?
On Thu, Apr 21, 2022 at 11:52:32PM +0000, Rick Macklem wrote:
> J. Bruce Fields <bfields@fieldses.org> wrote:
> [stuff snipped]
> > On Thu, Apr 21, 2022 at 12:40:49PM -0400, bfields wrote:
> > >
> > >
> > > Stale filehandles aren't normal, and suggest some bug or
> > > misconfiguration on the server side, either in NFS or the exported
> > > filesystem.
> > 
> > Actually, I should take that back: if one client removes files while a
> > second client is using them, it'd be normal for applications on that
> > second client to see ESTALE.
> I took a look at crispyduck's packet trace and here's what I saw:
> Packet#
> 48 Lookup of test-ovf.vmx
> 49 NFS_OK FH is 0x7c9ce14b (the hash)
> ...
> 51 Open Claim_FH fo 0x7c9ce14b
> 52 NFS_OK Open Stateid 0x35be
> ...
> 138 Rename test-ovf.vmx~ to test-ovf.vmx
> 139 NFS_OK
> ...
> 141 Close with PutFH 0x7c9ce14b
> 142 NFS4ERR_STALE for the PutFH
> 
> So, it seems that the Rename will delete the file (names another file to the
> same name "test-ovf.vmx".? Then the subsequent Close's PutFH fails,
> because the file for the FH has been deleted.

Actually (sorry I'm slow to understand this)--why would our 4.1 server
ever be returning STALE on a close?? We normally hold a reference to the
file.

Oh, wait, is subtree_check set on the export?? You don't want to do
that.? (The freebsd server probably doesn't even give that as an
option?)

--b.

> 
> Looks like yet another ESXi client bug to me?
> (I've seen assorted other ones, but not this one. I have no idea how this
>? might work on a FreeBSD server. I can only assume the RPC sequence
>? ends up different for FreeBSD for some reason? Maybe the Close gets
>? processed before the Rename? I didn't look at the Sequence args for
>? these RPCs to see if they use different slots.)
> 
> 
> > So it might be interesting to know what actually happens when VM
> > templates are imported.
> If you look at the packet trace, somewhat weird, like most things for this
> client. It does a Lookup of the same file name over and over again, for
> example.
> 
> > I suppose you could also try NFSv4.0 or try varying kernel versions to
> > try to narrow down the problem.
> I think it only does NFSv4.1.
> I've tried to contact the VMware engineers, but never had any luck.
> I wish they'd show up at a bakeathon, but...
> 
> > No easy ideas off the top of my head, sorry.
> I once posted a list of problems I had found with ESXi 6.5 to a FreeBSD
> mailing list and someone who worked for VMware cut/pasted it into their
> problem database.? They responded to him with "might be fixed in a future
> release" and, indeed, they were fixed in ESXi 6.7, so if you can get this to
> them, they might fix it?
> 
> rick
> 
> --b.
> 
> > Figuring out more than that would require more
> > investigation.
> >
> > --b.
> >
> > >
> > > Br,
> > > Andi
> > >
> > >
> > >
> > >
> > >
> > >
> > > Von: Chuck Lever III <chuck.lever@oracle.com>
> > > Gesendet: Donnerstag, 21. April 2022 16:58
> > > An: Andreas Nagy <crispyduck@outlook.at>
> > > Cc: Linux NFS Mailing List <linux-nfs@vger.kernel.org>
> > > Betreff: Re: Problems with NFS4.1 on ESXi
> > >
> > > Hi Andreas-
> > >
> > > > On Apr 21, 2022, at 12:55 AM, Andreas Nagy <crispyduck@outlook.at> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I hope this mailing list is the right place to discuss some problems with nfs4.1.
> > >
> > > Well, yes and no. This is an upstream developer mailing list,
> > > not really for user support.
> > >
> > > You seem to be asking about products that are currently supported,
> > > and I'm not sure if the Debian kernel is stock upstream 5.13 or
> > > something else. ZFS is not an upstream Linux filesystem and the
> > > ESXi NFS client is something we have little to no experience with.
> > >
> > > I recommend contacting the support desk for your products. If
> > > they find a specific problem with the Linux NFS server's
> > > implementation of the NFSv4.1 protocol, then come back here.
> > >
> > >
> > > > Switching from FreeBSD host as NFS server to a Proxmox environment also serving NFS I see some strange issues in combination with VMWare ESXi.
> > > >
> > > > After first thinking it works fine, I started to realize that there are problems with ESXi datastores on NFS4.1 when trying to import VMs (OVF).
> > > >
> > > > Importing ESXi OVF VM Templates fails nearly every time with a ESXi error message "postNFCData failed: Not Found". With NFS3 it is working fine.
> > > >
> > > > NFS server is running on a Proxmox host:
> > > >
> > > >? root@sepp-sto-01:~# hostnamectl
> > > >? Static hostname: sepp-sto-01
> > > >? Icon name: computer-server
> > > >? Chassis: server
> > > >? Machine ID: 028da2386e514db19a3793d876fadf12
> > > >? Boot ID: c5130c8524c64bc38994f6cdd170d9fd
> > > >? Operating System: Debian GNU/Linux 11 (bullseye)
> > > >? Kernel: Linux 5.13.19-4-pve
> > > >? Architecture: x86-64
> > > >
> > > >
> > > > File system is ZFS, but also tried it with others and it is the same behaivour.
> > > >
> > > >
> > > > ESXi version 7.2U3
> > > >
> > > > ESXi vmkernel.log:
> > > > 2022-04-19T17:46:38.933Z cpu0:262261)cswitch: L2Sec_EnforcePortCompliance:209: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]client vmk1 requested promiscuous mode on port 0x4000010, disallowed by vswitch policy
> > > > 2022-04-19T17:46:40.897Z cpu10:266351 opID=936118c3)World: 12075: VC opID esxui-d6ab-f678 maps to vmkernel opID 936118c3
> > > > 2022-04-19T17:46:40.897Z cpu10:266351 opID=936118c3)WARNING: NFS41: NFS41FileDoCloseFile:3128: file handle close on obj 0x4303fce02850 failed: Stale file handle
> > > > 2022-04-19T17:46:40.897Z cpu10:266351 opID=936118c3)WARNING: NFS41: NFS41FileOpCloseFile:3718: NFS41FileCloseFile failed: Stale file handle
> > > > 2022-04-19T17:46:41.164Z cpu4:266351 opID=936118c3)WARNING: NFS41: NFS41FileDoCloseFile:3128: file handle close on obj 0x4303fcdaa000 failed: Stale file handle
> > > > 2022-04-19T17:46:41.164Z cpu4:266351 opID=936118c3)WARNING: NFS41: NFS41FileOpCloseFile:3718: NFS41FileCloseFile failed: Stale file handle
> > > > 2022-04-19T17:47:25.166Z cpu18:262376)ScsiVmas: 1074: Inquiry for VPD page 00 to device mpx.vmhba32:C0:T0:L0 failed with error Not supported
> > > > 2022-04-19T17:47:25.167Z cpu18:262375)StorageDevice: 7059: End path evaluation for device mpx.vmhba32:C0:T0:L0
> > > > 2022-04-19T17:47:30.645Z cpu4:264565 opID=9529ace7)World: 12075: VC opID esxui-6787-f694 maps to vmkernel opID 9529ace7
> > > > 2022-04-19T17:47:30.645Z cpu4:264565 opID=9529ace7)VmMemXfer: vm 264565: 2465: Evicting VM with path:/vmfs/volumes/9f10677f-697882ed-0000-000000000000/test-ovf/test-ovf.vmx
> > > > 2022-04-19T17:47:30.645Z cpu4:264565 opID=9529ace7)VmMemXfer: 209: Creating crypto hash
> > > > 2022-04-19T17:47:30.645Z cpu4:264565 opID=9529ace7)VmMemXfer: vm 264565: 2479: Could not find MemXferFS region for /vmfs/volumes/9f10677f-697882ed-0000-000000000000/test-ovf/test-ovf.vmx
> > > > 2022-04-19T17:47:30.693Z cpu4:264565 opID=9529ace7)VmMemXfer: vm 264565: 2465: Evicting VM with path:/vmfs/volumes/9f10677f-697882ed-0000-000000000000/test-ovf/test-ovf.vmx
> > > > 2022-04-19T17:47:30.693Z cpu4:264565 opID=9529ace7)VmMemXfer: 209: Creating crypto hash
> > > > 2022-04-19T17:47:30.693Z cpu4:264565 opID=9529ace7)VmMemXfer: vm 264565: 2479: Could not find MemXferFS region for /vmfs/volumes/9f10677f-697882ed-0000-000000000000/test-ovf/test-ovf.vmx
> > > >
> > > > tcpdump taken on the esxi with filter on the nfs server ip is attached here:
> > > > https://easyupload.io/xvtpt1
> > > >
> > > > I tried to analyze, but have no idea what exactly the problem is. Maybe it is some issue with the VMWare implementation?
> > > > Would be nice if someone with better NFS knowledge could have a look on the traces.
> > > >
> > > > Best regards,
> > > > cd
> > >
> > > --
> > > Chuck Lever
> > >