Received-SPF: pass (google.com: domain of linux-nfs+bounces-3151-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Fri, 3 May 2024 14:44:42 -0400
From: Chuck Lever <chuck.lever@oracle.com>
To: Scott Mayhew <smayhew@redhat.com>
Cc: Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: NFSv3 and xprtsec policies
Message-ID: <ZjUwmthoBkBLaW/I@tissot.1015granger.net>
References: <ZjO3Qwf_G87yNXb2@aion>
 <100A1A35-2B53-46CA-A448-F82A95CA1EFA@oracle.com>
 <ZjPPZmBZJZVmBuA6@aion>
 <38C9B493-2A43-4691-A19A-8998F0DFAED9@oracle.com>
 <ZjPgq-xA1G6Z2_aQ@aion>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <ZjPgq-xA1G6Z2_aQ@aion>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?TFpKSWpnK2tid0d1cHRmbS9iTTBkUmRzOGM5eGhKRXdlbldtT2g3NE9QRUQz?=
 =?utf-8?B?bmVNUUF2Q0RJVGRFNk9MUWI5OVlaN0lnNWduWm1lK1F3VWN3YU9RNDFkOW92?=
 =?utf-8?B?R3pUckdUMGpOYTJFWE1VbDFPU0M5djZYaTUyWTZYWmZxRndCODJmQndPOVV1?=
 =?utf-8?B?WXBsMS9OUllpdGl3RkF0eEZtV0YreEluWnEzWlBZNWZkOGJnMVZxaEk1dmF3?=
 =?utf-8?B?SE5Ib1NySnJjWFdHdEtjc1ZCMnk2dFlyOWUyQUdpemJsUDArR3RodXA0UCtJ?=
 =?utf-8?B?UjlvNGo0aWlTbHNxMWIrcVVhdzN3N0VHMUdEcTk5U1J5YjQzU2xqN1k4Q0lM?=
 =?utf-8?B?WHJnTGdRdE9VUWpaNko2TG50UTJJSjdnVG5KUzNNZXJGS2ZlQXl5eFpmd1dC?=
 =?utf-8?B?KzFXZnRqSzBMMVEydUtoQ3doME1FYzlMVVppcVBnNzZpejVDMDZUQWtMTEZj?=
 =?utf-8?B?N1VtaGVRQUFQSGZPd0VxemVFUkhuZVBtUHZVanVkTzJ5KzlCQU92Rm5iV3A3?=
 =?utf-8?B?clVaVWJ5MmpPZjJEZDJNaGNnTUI1OVYyZ1g3V2lEaHVrTE45M3NtbGczSnI2?=
 =?utf-8?B?bTBRWFNvOVRHU1dlb2xqWGVvQnZGNDBueEZ1MTRaa0YvWVQ1dnRJYVlDbTdC?=
 =?utf-8?B?QkJ4c055SkZobmpZYVAralZuMHpCeHJMNE5SaWxTUCtmczBiMVRudm51S1hK?=
 =?utf-8?B?NjVsQVdYMUlLQlRxL3dpWnExWndYZWdxYmFCU1ZEb1EwTS9jWFZTbzMrRlBD?=
 =?utf-8?B?Tjd1VzVrODFBNk91RE1oUGxyWVkvYmYwd0Eva3oyTUZWRDRKYm1mOUpZeGxF?=
 =?utf-8?B?RkJBcnkrWGxlYVBmYVRyWmh6THQ2dU14aTB2K0JwZFNZcFY1M1IrbjhTTWJO?=
 =?utf-8?B?bHhOalFQeGg5YU16SlJFUGJUWGJRNVlYQ1lqTDUzWk1VQTRrdlMyN1U0RG5v?=
 =?utf-8?B?RGZSelE3TDE4OGV1VWtqZm5WVGlJVlVkOE14SzN3dG5oanNKbzZMVlNSa1I0?=
 =?utf-8?B?RkNwaWxteHp6Ti9yQUZrMFFmdHpLZlBvbGV5UW5JeDd4Q0N0enVXalNNeVNk?=
 =?utf-8?B?L05KdnFqUVpIcjdwSGJrdVQ3UDFSck1zT1ZWUFpnVzNmaHdCZzVIUXVkb3Ax?=
 =?utf-8?B?V2RWR2NLVXR4OGhzbUxqRTJzZ05iWUJmbTB6M0lGNnRxZ3RhREx5TUlEQ092?=
 =?utf-8?B?ekhSd2dDdXczZE9PVkx0elBjVXc5ZmhpVHBTY0JzU2F4ZHgxcmlXTCtEbkhX?=
 =?utf-8?B?MVBiaXpqUThPditmanovZ08zb3pMR1hWWmJYc3g5YThuc2dEUUx3Q08rWkhY?=
 =?utf-8?B?THNnUmdKTlpqSmhPMm1YVTd2YXFpS2FwU0Jxa1pkTVpNa2gzUjg0SkxOWDBS?=
 =?utf-8?B?SzR0ZmprT1JrL0JVMFl4SWlUbXNaV1JCRFhRK2srNzVteHFQRHJXNDBqa0dI?=
 =?utf-8?B?aFFVV0RHb281bHVMdEZXTnZ4WHJXUjR0YjRNU05lR0JXcnZlaitOeThFL1My?=
 =?utf-8?B?WURNSEZrNUhxRzNieG5RNlZzTmZVZytldThDUFpKL0xZL055Z0kwVzV0NmlS?=
 =?utf-8?B?Q3JyR041VlRMVnNSZzE5VCtTSjhlOExFNmU0Rm5vQVZORDZkeXNZNTh0NE9U?=
 =?utf-8?B?MTl0YU1ybGNqRkVPUzQ1SXYvbHF3V0dHVkpNMGMxcGJYYm1LbVFhWUl4Yncz?=
 =?utf-8?B?MUZuYXVYclRWMkFBQjQ0VTNoMTZBa1p3QVhCNzAwSk9QTnFLR2d1WHRMRThz?=
 =?utf-8?B?UmhxcTE1b1YrOTZCc0NjY3RGcVdRN2dYYVpOZ0htUXNIZHFFeFFWZXBZcFl0?=
 =?utf-8?B?U3hveTBLanVxQ01XanlUTVBuNUZGRkt5cHlmT2RXL2VxMGxXZXJaOVJLQisz?=
 =?utf-8?B?Y09BcEw2TzNMbXFOREQwQ2I4MnJaZ0JEcXBmcUowRUVwNnVqaHB6MTN1Undl?=
 =?utf-8?B?eHF6YklOUGtYUmVDUUVqcnZZRXZrTHVJK2xkUXdTd24wYm0yYUJTcStyNXhJ?=
 =?utf-8?B?bUUvVVM2VDBQRElEZGF6K3puYnpJcVVmTUNNUjBoRUlzOXlIbXpnaGhhRVl4?=
 =?utf-8?B?Q2hrdWZuZmdUWHM1Q0h5M0pFMGJ3Sld2d1VBell4U0Q0Ukh4VVR3YWNRdzR5?=
 =?utf-8?B?TDhIY1diSDQ4eUhXQktHVjIxUWNTd0FMVFhXcTk2SVFQVUFEYVptN2FoWjgx?=
 =?utf-8?B?QWc9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	bqk/OLvdSiYs1NEzgj8tkMYlLMAGRjLkpzxEaYW8JNxDeVoH8qyxlmBkQbPAnAwOkvQwQTh3WtdwY5L553DlD9WpXNOjboUoskfmXKDf4vLV6xl/DW6KmlOvIVpJBlXe1rwItzA15QrBsNsGGg3jlfwpaFFul7Y4w+Dz1YALrDft/kS2qQgahvij8sutK/COQ0KbxmMzLBEyePJ4YIYsgdjpOW/PmScrPHY7TIghjR1gGWrnBKI2ELLxC7YbLjRowwyikywx8oJlfBddFfjt5wi0yBBxcNWskBBpnN9bWCc2zpQmCIbU9VRwMR1C51Cp5BgaNfQo4F8Ke7Eh1hnhT5H3mSRYwQWb/nRX28R8Z7T4u6FVd351guEl9HwaPsXctfKS4MuWBpJEXwcyvGVfuS417Q7O3HHgoHzO3F+XfjacdlFH99w0Ca/sup2vJTcv2IBuCW7h7iLFhrCFM314BxdVLbgC6QZdJWA97hgzunZuKG2haJ4iUvYnQllBDBc0VBrXYJkDBxFF0gd09TcU64WMoBnwYsLKwXBrPTLMt+8n/Wz3TeCMhnk3if2jLRn01fN9Ta3y7XM/1auRQ/Q7tEgMwwedib1jVD910ZnXnCM=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 195c261a-c26f-4c5a-4f5e-08dc6ba119d4
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 May 2024 18:44:44.8715
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: C6f4uPICiIjp0HKrYJGwin3+MnUlQSSDkjhAYeYh4MkdsWvZPSGvDi23NWilqsEo1FQVvsY4mWn3BQNyRcJ7Tg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB5804
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1011,Hydra:6.0.650,FMLib:17.11.176.26
 definitions=2024-05-03_12,2024-05-03_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 malwarescore=0
 phishscore=0 spamscore=0 bulkscore=0 suspectscore=0 adultscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2404010000 definitions=main-2405030132
X-Proofpoint-GUID: lgf-mXDhGi2gR1j0Vx0wBIc15doWGB9t
X-Proofpoint-ORIG-GUID: lgf-mXDhGi2gR1j0Vx0wBIc15doWGB9t

On Thu, May 02, 2024 at 02:51:23PM -0400, Scott Mayhew wrote:
> On Thu, 02 May 2024, Chuck Lever III wrote:
> 
> > 
> > 
> > > On May 2, 2024, at 1:37 PM, Scott Mayhew <smayhew@redhat.com> wrote:
> > > 
> > > On Thu, 02 May 2024, Chuck Lever III wrote:
> > > 
> > >>> On May 2, 2024, at 11:54 AM, Scott Mayhew <smayhew@redhat.com> wrote:
> > >>> 
> > >>> Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an
> > >>> NFSv3 client can mount with "xprtsec=none" a filesystem exported with
> > >>> "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a
> > >>> mount to its mount table - it can't actually access the mount).
> > >>> 
> > >>> Here's an example using machines from the recent Bakeathon.
> > >>> 
> > >>> Mounting a server with TLS enabled:
> > >>> 
> > >>> # mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > >>> # umount /mnt
> > >>> 
> > >>> Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
> > >>> 
> > >>> # mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > >>> mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
> > >>> 
> > >>> Yet a v3 mount without "xprtsec=tls" works:
> > >>> 
> > >>> # mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > >>> # umount /mnt
> > >>> 
> > >>> and a mount with no explicit version and without "xprtsec=tls" falls back to
> > >>> v3 and also "works":
> > >>> 
> > >>> # mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > >>> # grep ora /proc/mounts
> > >>> oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
> > >>> +rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
> > >>> 
> > >>> Even though the filesystem is mounted, the client can't do anything with it:
> > >>> 
> > >>> # ls /mnt
> > >>> ls: cannot open directory '/mnt': Permission denied
> > >>> 
> > >>> When krb5 is used with NFSv3, the server returns a list of pseudoflavors in
> > >>> mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
> > >>> The client compares that list with its own list of auth flavors parsed from the
> > >>> mount request and returns -EACCES if no match is found (see
> > >>> nfs_verify_authflavors()).
> > >>> 
> > >>> Perhaps we should be doing something similar with xprtsec policies?
> > >> 
> > >> The problem might be in how you've set up the exports. With NFSv3,
> > >> the parent export needs the "crossmnt" export option in order for
> > >> NFSv3 to behave like NFSv4 in this regard, although I could have
> > >> missed something.
> > > 
> > > I was mounting your server though :)
> > 
> > OK, then not the same bug that Olga found last year.
> > 
> > We should find out what FreeBSD does in this case.
> 
> I thought about that.  Rick's servers from the BAT are offline, and I
> don't think he was exporting v3 anyway.
> 
> > 
> > 
> > >>> Should
> > >>> there be an errata to RFC 9289 and a request from IANA for assigned numbers for
> > >>> pseudo-flavors corresponding to xprtsec policies?
> > >> 
> > >> No. Transport-layer security is not an RPC security flavor or
> > >> pseudo-flavor. These two things are not related.
> > >> 
> > >> (And in fact, I proposed something like this for NFSv4 SECINFO,
> > >> but it was rejected).
> > > 
> > > I thought it might be a stretch to try to use mountres3.auth_flavors for
> > > this, but since RFC 9289 does refer to AUTH_TLS as an authentication
> > > flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
> > > also lists TLS under the Flavor Name column I thought it might make
> > > sense to treat xprtsec policies as if they were pseudo-flavors even
> > > though they're not, if only to give the client a way to determine that
> > > the mount should fail.
> > 
> > RPC_AUTH_TLS is used only when a client probes a server to see if
> > it supports RPC-with-TLS. At all other times, the client uses one
> > of the normal, legitimate flavors. It does not represent a security
> > flavor that can be used during regular operation.
> > 
> > NFSv3 mount failover logic is still open for discussion (ie, incomplete).
> > 
> > Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors
> > list? I think clients that don't recognize it should ignore it,
> > but I'm not sure. What should a client do if it sees that flavor in
> > the list? It's not one that can be used for any other procedure than
> > a NULL RPC.
> 
> Maybe?  After the client gets the filehandle it's calling FSINFO and
> PATHCONF.  The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't
> checking for a negative return code from the PATHCONF operation.  If it
> did, it could maybe use the -EACCES coupled with the knowledge that the
> server had RPC_AUTH_TLS enabled to emit an error message saying to check
> the xprtsec policies (but I don't think that would be as definitive as
> what I had in mind) and to fail the mount.

If Linux is the only implementation of NFSv3 with TLS so far, then
we have some latitude for innovation.

I would like to hear from the client maintainers about what they
would prefer the client user experience to look like. Then NFSD's
behavior can be adjusted to accommodate.

In this case, Steve would have to sign off on an rpc.mountd change
to return AUTH_TLS in the auth_flavors list.


-- 
Chuck Lever