MIME-Version: 1.0
References: <20230212140148.4F0D.409509F4@e16-tech.com> <A2C070C8-3B50-4B11-B07B-99FE93F9BA16@oracle.com>
In-Reply-To: <A2C070C8-3B50-4B11-B07B-99FE93F9BA16@oracle.com>
From:   Rick Macklem <rick.macklem@gmail.com>
Date:   Sun, 12 Feb 2023 20:30:25 -0800
Message-ID: <CAM5tNy4hXW_YUExyncJAFU07yVhnvAG7dnNM+YFROsLOEM9qBQ@mail.gmail.com>
Subject: Re: question about the performance impact of sec=krb5
To:     Chuck Lever III <chuck.lever@oracle.com>
Cc:     Wang Yugui <wangyugui@e16-tech.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Olga Kornievskaia <kolga@netapp.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Sun, Feb 12, 2023 at 9:47 AM Chuck Lever III <chuck.lever@oracle.com> wr=
ote:
>
> CAUTION: This email originated from outside of the University of Guelph. =
Do not click links or open attachments unless you recognize the sender and =
know the content is safe. If in doubt, forward suspicious emails to IThelp@=
uoguelph.ca
>
>
>
>
> > On Feb 12, 2023, at 1:01 AM, Wang Yugui <wangyugui@e16-tech.com> wrote:
> >
> > Hi,
> >
> > question about the performance of sec=3Dkrb5.
> >
> > https://learn.microsoft.com/en-us/azure/azure-netapp-files/performance-=
impact-kerberos
> > Performance impact of krb5:
> >       Average IOPS decreased by 53%
> >       Average throughput decreased by 53%
> >       Average latency increased by 3.2 ms
>
> Looking at the numbers in this article... they don't
> seem quite right. Here are the others:
>
> > Performance impact of krb5i:
> >       =E2=80=A2 Average IOPS decreased by 55%
> >       =E2=80=A2 Average throughput decreased by 55%
> >       =E2=80=A2 Average latency increased by 0.6 ms
> > Performance impact of krb5p:
> >       =E2=80=A2 Average IOPS decreased by 77%
> >       =E2=80=A2 Average throughput decreased by 77%
> >       =E2=80=A2 Average latency increased by 1.6 ms
>
> I would expect krb5p to be the worst in terms of
> latency. And I would like to see round-trip numbers
> reported: what part of the increase in latency is
> due to server versus client processing?
>
> This is also remarkable:
>
> > When nconnect is used in Linux, the GSS security context is shared betw=
een all the nconnect connections to a particular server. TCP is a reliable =
transport that supports out-of-order packet delivery to deal with out-of-or=
der packets in a GSS stream, using a sliding window of sequence numbers.=E2=
=80=AFWhen packets not in the sequence window are received, the security co=
ntext is discarded, and=E2=80=AFa new security context is negotiated. All m=
essages sent with in the now-discarded context are no longer valid, thus re=
quiring the messages to be sent again. Larger number of packets in an nconn=
ect setup cause frequent out-of-window packets, triggering the described be=
havior. No specific degradation percentages can be stated with this behavio=
r.
>
>
> So, does this mean that nconnect makes the GSS sequence
> window problem worse, or that when a window underrun
> occurs it has broader impact because multiple connections
> are affected?
>
> Seems like maybe nconnect should set up a unique GSS
> context for each xprt. It would be helpful to file a bug.
>
Here's a snippet from RFC2203:
   In a successful response, the seq_window field is set to the sequence
   window length supported by the server for this context.  This window
   specifies the maximum number of client requests that may be
   outstanding for this context. The server will accept "seq_window"
   requests at a time, and these may be out of order.  The client may
   use this number to determine the number of threads that can
   simultaneously send requests on this context.

It would be interesting to know what size of window Netapp filers specify
in the reply when context initialization completes.
A simple fix might be to get Netapp to increase the window, since they
have observed the problem.
FreeBSD servers use 128.  I have no idea what other servers use.

rick

>
> > and then in 'man 5 nfs'
> > sec=3Dkrb5  provides cryptographic proof of a user's identity in each R=
PC request.
>
> Kerberos has performance impacts due to the crypto-
> graphic operations that are performed on even small
> fixed-sized sections of each RPC message, when using
> sec=3Dkrb5 (no 'i' or 'p').
>
>
> > Is there a option of better performance to check krb5 only when mount.n=
fs4,
> > not when file acess?
>
> If you mount with NFSv4 and sec=3Dsys from a Linux NFS
> client that has a keytab, the client will attempt to
> use krb5i for lease management operations (such as
> EXCHANGE_ID) but it will continue to use sec=3Dsys for
> user authentication. That's not terribly secure.
>
> A better answer would be to make Kerberos faster.
> I've done some recent work on improving the overhead
> of using message digest algorithms with GSS-API, but
> haven't done any specific measurement. I'm sure
> there's more room for optimization.
>
> Even better would be to use a transport layer security
> service. Amazon has EFS and Oracle Cloud has something
> similar, but we're working on a standard approach that
> uses TLSv1.3.
>
>
> --
> Chuck Lever
>
>
>