From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Wang Yugui <wangyugui@e16-tech.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Olga Kornievskaia <kolga@netapp.com>
Subject: Re: question about the performance impact of sec=krb5
Thread-Topic: question about the performance impact of sec=krb5
Thread-Index: AQHZPqeMez/BkWTmQkuEvdu9Nhcyhq7LltYAgABTYICAACd8AA==
Date:   Mon, 13 Feb 2023 01:07:13 +0000
Message-ID: <84ACDC95-5354-468F-A5F4-AB934AF61728@oracle.com>
References: <20230212140148.4F0D.409509F4@e16-tech.com>
 <A2C070C8-3B50-4B11-B07B-99FE93F9BA16@oracle.com>
 <20230213064552.3960.409509F4@e16-tech.com>
In-Reply-To: <20230213064552.3960.409509F4@e16-tech.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?4z9RQJ2g5uuZE/zNhC0c16KUauEuC4jbQE+g16ebxBtGOxtkaEZkGIj1/M/i?=
 =?us-ascii?Q?0SyjhTfOshwSD2CVDDSFb5qDGrffxiWOlSa5VTlnyYPrdFjr9gQ8bjL3FdfR?=
 =?us-ascii?Q?rotMCj/uXiVDw4fEuv6Bjd30hDTgz+gBtDe43gEOCjrLe3B8tVCKAZgKszys?=
 =?us-ascii?Q?mbq8Co8qn3OAN6B9oH4HczCG9ffWwBTPw5av98jGpHR3UFoMryqAc26zOj8x?=
 =?us-ascii?Q?lgcc2j75FLTNugEbDYDGd0nwnvEQGiBs9Gtpvn7JQBxkCAlpGmkldtUQr6az?=
 =?us-ascii?Q?/HKIXCQFpofmXZgSXggDDu91sq8um/yGBywsuI80UCLFymLeGJya93bpC5Uu?=
 =?us-ascii?Q?U+IJSvuHXz2ms6k+UANDcClT/95lrzv0zZyi/FmjEAjFtseB0cZWyfA8Qf34?=
 =?us-ascii?Q?iBdJKwsdk6pJ8xiD+6jVgMwWNlBG+Wf3NDgL4yKZs+109fMUjaqXes00tMoe?=
 =?us-ascii?Q?EF50UMN4u/0hU8Cdd7qg7smyNWkQIzNnwK4De8gkAQNLaEP8myn1MisVfj7Y?=
 =?us-ascii?Q?6M19GV9uKy53uSZMbwB0VE7MzecSjwj+6PbF8dGh/ZS3S5sFYfyYN6V+jIIv?=
 =?us-ascii?Q?llcT/1eq9ijPSW4b+kUzu5aBlNVuUuaSZdY28O7f30EMidWo+qb+hgpbj1N9?=
 =?us-ascii?Q?zLdrnbm0e9LXFXAeu+xUlCZQcW0TtdzrSSmQA9sRH2c6TkCHEELTYL8XI/Cs?=
 =?us-ascii?Q?xgGGf4IRIFSjWhCfziJzzBIWgKwofocnWsTI77azJvFDLtDAjH0Mr8SHAohX?=
 =?us-ascii?Q?mRYokHONKkQSqCcceH6tLyNoCE9fbB4j9YJugYGlEpScTFnbZxht3R6x+JM4?=
 =?us-ascii?Q?odTvvi20ifisIJgt1j8b9j6R920wDOSy1RnDNly5R8LtEPIvZK8t54YuPAgr?=
 =?us-ascii?Q?s3CX44NFFmMF6Ay6xhmYV6CNau7arURzYhq6rZpW1KC8jx3dZS/KKLAzU9m6?=
 =?us-ascii?Q?eqga6OKFlCAMu9bHW3LBe2t8Uqt00AWkOaLlVSISfsqGjj8xk3wnK0S009lE?=
 =?us-ascii?Q?WZkjrFVYpTy8pokeGCfGFyKaafJdznJPUp4cmhDs+20G4i9H9ENLu1DUd3nO?=
 =?us-ascii?Q?TppsIplSsbLATUGo14T5A3FVJc5wRnZHPQKCdXeWtYokok8tVNYv3R6Mjbvc?=
 =?us-ascii?Q?rrDy9XcL3MmcR2SvHhx1HkatTR9N91ZQ8KapMgyRmlc0dYuN2W12PIZ2c2Lc?=
 =?us-ascii?Q?/iSaRMfy0OS38B8J3rgy7u+ypHkRXqezIR3+ALcymlnZKciPQ0CoVP65iObh?=
 =?us-ascii?Q?7Kc6KH4ecTQ7PPVi/pKO7Rgc9adtIb01s4nsDBHmNA2ZlWYLldUudt92D2WI?=
 =?us-ascii?Q?q5MnsmZqfhEuvEObp3ULQNjZ3NNlEW9L8Qt0CBiwQf7Z3R/Y+bq2m9cljqRu?=
 =?us-ascii?Q?nJkmgGx9O0XkNdEWT5OTonE9w2yVRssa0sshObZ/VOXZtfS/HbRXI/wIWBgW?=
 =?us-ascii?Q?PLJ3RDvydGOzcTk70aMo5HlnVWwf8rP2j/SN+R/RpMaEWb120a/ZEr8M+75e?=
 =?us-ascii?Q?+VteLfvZ0XbnemSOZIVBKDc3wdN7N8LEnWUBsOUw+2NUnODjuO2eECDvRXbJ?=
 =?us-ascii?Q?eTaWVi3JTC0rN2vBgcbePwEu7ukQBa5vJn6JmPmxVEudNiAcQRvaMM5lgKZn?=
 =?us-ascii?Q?Ww=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <C2F7DAEAA1853F4BA7A7127DC247C9CB@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: NHQXKSw1P91WPwPxi4NuXia5FVaFWIlGfqANhRb/r1oHEQtitrqwl/+jG34v/4xpQsaMQV46M1Y3o95eQrWNAgC3f9OdyMoxFzod9ifA/nInVFVG6YWokNxjabWABlfQTMdzBZAFnGJFdpcQKVLYJ1Ag2zw+wv0EgyRY181a+QfGVB6VfIsFuyo7E9T990HLcfVjX9LLbDeyeIyF5m4n+SLShkfPXyEI4mKkPtbFz/OjmhDg2zsQBdITpkydxfSvWBZfQpOKrjp07lJvUL9vObN5p2oIfpjFLE3veRUlvRWUoIiq/61hQ0dAvnTfgZ39im2OgYaY9wt1WZzLdYxUO8ueCuDiFRcthVHJJtciMpRWYN1KjFcIVKew8xFcP4dSIUYpMO+6k6/oqBWVD6ib3l2ttJZis0BohwAs6BvbE2lyMpGYx5yqOHvcTcYHpyTVHm87TtzoNQac8/oRgzVkiBkUR5Np9E9XS0BFPLBW0Va6pi285FhaHifZTxqz2DQXOGus08XBqF69eGB3dtQ6vYe83fNrPvyTaCxginP+SFp7QNosJgXbP8cjnrNGzEi92WYkpFggojVIsEa1gvygbedNViQSRGYnnaiMZA9npW3ekiQGoeVmJBf5j7cB+HYFTNLzbCvqkBxf00ZtD36xKk3B8uUsjmpNy1HU0qiqTQfbhj+rrfGnkmaMKByFNFyk9FseMPSTJBjVDHXk7V9mLzXazDuvczTWnDEKmBkFxDSQ2r38M4aGBb1stCK1YJMSMliHUlHU3cshn2bMSpCobO0YjmHiTuDbt5TxI+EPmdSJrOGQJnCEPI1GYvoXmH4hyb8fJoMquAhZynFrhGXhGw==
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1da5e73e-1ed0-4a40-1458-08db0d5ea459
X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Feb 2023 01:07:13.7820
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Mf3bSa5BmRQYadcFxk8N03AbSjcQJKvZi4iwvypHW89EDz6ERhI8/aGOTcGGAzc6XA4fuTBmOPLhI8FHv/LVrg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6609
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22
 definitions=2023-02-12_12,2023-02-09_03,2023-02-09_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0
 phishscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2212070000 definitions=main-2302130008
X-Proofpoint-GUID: Ai1jdOyouLBywn33Uj5mbycBIQwtwOUi
X-Proofpoint-ORIG-GUID: Ai1jdOyouLBywn33Uj5mbycBIQwtwOUi
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Feb 12, 2023, at 5:45 PM, Wang Yugui <wangyugui@e16-tech.com> wrote:
>=20
> Hi,
>=20
>>=20
>>=20
>>> On Feb 12, 2023, at 1:01 AM, Wang Yugui <wangyugui@e16-tech.com> wrote:
>>>=20
>>> Hi,
>>>=20
>>> question about the performance of sec=3Dkrb5.
>>>=20
>>> https://learn.microsoft.com/en-us/azure/azure-netapp-files/performance-=
impact-kerberos
>>> Performance impact of krb5:
>>> 	Average IOPS decreased by 53%
>>> 	Average throughput decreased by 53%
>>> 	Average latency increased by 3.2 ms
>>=20
>> Looking at the numbers in this article... they don't
>> seem quite right. Here are the others:
>>=20
>>> Performance impact of krb5i:
>>> 	? Average IOPS decreased by 55%
>>> 	? Average throughput decreased by 55%
>>> 	? Average latency increased by 0.6 ms
>>> Performance impact of krb5p:
>>> 	? Average IOPS decreased by 77%
>>> 	? Average throughput decreased by 77%
>>> 	? Average latency increased by 1.6 ms
>>=20
>> I would expect krb5p to be the worst in terms of
>> latency. And I would like to see round-trip numbers
>> reported: what part of the increase in latency is
>> due to server versus client processing?
>>=20
>> This is also remarkable:
>>=20
>>> When nconnect is used in Linux, the GSS security context is shared betw=
een all the nconnect connections to a particular server. TCP is a reliable =
transport that supports out-of-order packet delivery to deal with out-of-or=
der packets in a GSS stream, using a sliding window of sequence numbers.?Wh=
en packets not in the sequence window are received, the security context is=
 discarded, and?a new security context is negotiated. All messages sent wit=
h in the now-discarded context are no longer valid, thus requiring the mess=
ages to be sent again. Larger number of packets in an nconnect setup cause =
frequent out-of-window packets, triggering the described behavior. No speci=
fic degradation percentages can be stated with this behavior.
>>=20
>>=20
>> So, does this mean that nconnect makes the GSS sequence
>> window problem worse, or that when a window underrun
>> occurs it has broader impact because multiple connections
>> are affected?
>>=20
>> Seems like maybe nconnect should set up a unique GSS
>> context for each xprt. It would be helpful to file a bug.
>>=20
>>=20
>>> and then in 'man 5 nfs'
>>> sec=3Dkrb5  provides cryptographic proof of a user's identity in each R=
PC request.
>>=20
>> Kerberos has performance impacts due to the crypto-
>> graphic operations that are performed on even small
>> fixed-sized sections of each RPC message, when using
>> sec=3Dkrb5 (no 'i' or 'p').
>>=20
>>=20
>>> Is there a option of better performance to check krb5 only when mount.n=
fs4,
>>> not when file acess?
>>=20
>> If you mount with NFSv4 and sec=3Dsys from a Linux NFS
>> client that has a keytab, the client will attempt to
>> use krb5i for lease management operations (such as
>> EXCHANGE_ID) but it will continue to use sec=3Dsys for
>> user authentication. That's not terribly secure.
>=20
> I noticed this feature in this case
> - the nfs client joined the windows AD(then have a keytab)
> - the windows AD server is shutdown.
> then 'mount.nfs4 -o sec=3Dsys' will take about 3 min.
> because there are 60s timeout  *3 inside.
> but 'sec=3Dsys' does not need any krb5 operations?

I would expect some bad behavior in this case: the
client is using Kerberos while part of the network
service infrastructure is not available to it. It's
going to hang.

If you don't want sec=3Dsys to hang, then either don't
take the AD offline, don't put a keytab on the client,
or don't use NFSv4.


> maybe we can have another krb5 mode, such as 'krb5l'
> - the nfs client must have a keytab.
> - krb5 must be used only when mount.nfs4

It's not that simple.

All mount points on that client of that server share the
same lease, whether they are sec=3Dsys or sec=3Dkrb5*. The
krb5 mounts must use krb5 for lease management, the
sec=3Dsys mounts may use it, but don't have to.

What's more, when the client reboots, it needs to re-
identify itself to the server using the same credential,
no matter which order the mounts are re-established --
sys first or krb5 first.

Or, more generally speaking, when a keytab is present,
even if the client has only sec=3Dsys mounts at this moment,
it might establish a sec=3Dkrb5 mount at any time in the
future. For instance, consider the case where only sec=3Dsys
mounts reside in /etc/fstab that get mounted at boot time,
but there are sec=3Dkrb5 mounts in an automounter map that
get pulled in when a user accesses them.

In other words, it's not a per-mount setting, and it has
to be the same principal and security flavor after every
client reboot. We picked an appropriate level of security
for lease management that meets these requirements. The
only choice is to use Kerberos if there is even the
possibility that sec=3Dkrb5* can be used.

It might be surprising behavior, until you realize this
is kind of the only way it can work with a single lease
per client. Plus it encourages better security.


> It would be more secure than IP address check in /etc/exorts?

Well, it would provide some degree of peer authentication
based on whatever principal is available on the client
(a host service principal or some user that wants to
provide a password for this purpose).

But then user I/O requests would use AUTH_SYS, which is
trivial to alter while the RPC messages transit an open
network. That's what I meant by not terribly secure. But
better than all AUTH_SYS, sure.


> Best Regards
> Wang Yugui (wangyugui@e16-tech.com)
> 2023/02/13
>=20
>=20
>>=20
>> A better answer would be to make Kerberos faster.
>> I've done some recent work on improving the overhead
>> of using message digest algorithms with GSS-API, but
>> haven't done any specific measurement. I'm sure
>> there's more room for optimization.
>>=20
>> Even better would be to use a transport layer security
>> service. Amazon has EFS and Oracle Cloud has something
>> similar, but we're working on a standard approach that
>> uses TLSv1.3.
>>=20
>>=20
>> --
>> Chuck Lever

--
Chuck Lever