We have an issue with the Ganesha server with very short (2 second)
lease times. Ganesha uses a 1 second granularity for lease management,
and considers a time since last renewed equal to the lease time as too
long. The result is that the lease period may be short close to 2
seconds depending on when within a given second things actually happened
(so a last renew at 0.99 with a subsequent renew at 2.01 which is just
over one second looks like 2 seconds to Ganesha and thus is >= the 2
second lease time and not good enough. A simple change would be to
change the >= to a >, which gives one more second, but it still could
result in the lease time being almost 1 second too short which is
significant with a 2 second lease time. But if the minimum reasonable
lease time is more like 5 or 10 seconds, that 1 second becomes less
significant.
The bigger fix would be to use a finer grained time, but that adds
complexity, but if people really want to run with 2 second lease times
and it makes any kind of sense, we would need to make that change.
Thanks
Frank
Hi Frank,
IMHO, very short leases can trigger a massive state recovery on network hiccups.
Our server offers 90 seconds to the client. The client usually renew lease (sequence) once in a minute.
During high IO periods lease is not required and when idle, then once in a minute is sufficient to
keep the mount alive. In a worst case, when a client got a lock and dispersal, a competing lock/open
will block only for 90s. Whatever number you have, it should be
BTW, IETF mailing list probably a better place for this question.
Regards,
Tigran.
----- Original Message -----
> From: "Frank Filz" <[email protected]>
> To: "linux-nfs" <[email protected]>
> Sent: Wednesday, December 12, 2018 10:14:59 PM
> Subject: What is a reasonable minimum lease time?
> We have an issue with the Ganesha server with very short (2 second)
> lease times. Ganesha uses a 1 second granularity for lease management,
> and considers a time since last renewed equal to the lease time as too
> long. The result is that the lease period may be short close to 2
> seconds depending on when within a given second things actually happened
> (so a last renew at 0.99 with a subsequent renew at 2.01 which is just
> over one second looks like 2 seconds to Ganesha and thus is >= the 2
> second lease time and not good enough. A simple change would be to
> change the >= to a >, which gives one more second, but it still could
> result in the lease time being almost 1 second too short which is
> significant with a 2 second lease time. But if the minimum reasonable
> lease time is more like 5 or 10 seconds, that 1 second becomes less
> significant.
>
> The bigger fix would be to use a finer grained time, but that adds
> complexity, but if people really want to run with 2 second lease times
> and it makes any kind of sense, we would need to make that change.
>
> Thanks
>
>
> Frank
On Wed, 2018-12-12 at 23:11 +0100, Mkrtchyan, Tigran wrote:
> Hi Frank,
>
> IMHO, very short leases can trigger a massive state recovery on
> network hiccups.
>
> Our server offers 90 seconds to the client. The client usually renew
> lease (sequence) once in a minute.
> During high IO periods lease is not required and when idle, then once
> in a minute is sufficient to
> keep the mount alive. In a worst case, when a client got a lock and
> dispersal, a competing lock/open
> will block only for 90s. Whatever number you have, it should be
>
Given that TCP SYN timeouts generally follow an exponential law, and
can easily reach > 20s, then I'd argue a 2s lease is on the extremely
short end of tolerable. You probably want to do something a lot larger
if you want lock recovery after a server crash to be reliable.
>
> BTW, IETF mailing list probably a better place for this question.
>
> Regards,
> Tigran.
>
>
> ----- Original Message -----
> > From: "Frank Filz" <[email protected]>
> > To: "linux-nfs" <[email protected]>
> > Sent: Wednesday, December 12, 2018 10:14:59 PM
> > Subject: What is a reasonable minimum lease time?
> > We have an issue with the Ganesha server with very short (2 second)
> > lease times. Ganesha uses a 1 second granularity for lease
> > management,
> > and considers a time since last renewed equal to the lease time as
> > too
> > long. The result is that the lease period may be short close to 2
> > seconds depending on when within a given second things actually
> > happened
> > (so a last renew at 0.99 with a subsequent renew at 2.01 which is
> > just
> > over one second looks like 2 seconds to Ganesha and thus is >= the
> > 2
> > second lease time and not good enough. A simple change would be to
> > change the >= to a >, which gives one more second, but it still
> > could
> > result in the lease time being almost 1 second too short which is
> > significant with a 2 second lease time. But if the minimum
> > reasonable
> > lease time is more like 5 or 10 seconds, that 1 second becomes less
> > significant.
> >
> > The bigger fix would be to use a finer grained time, but that adds
> > complexity, but if people really want to run with 2 second lease
> > times
> > and it makes any kind of sense, we would need to make that change.
> >
> > Thanks
> >
> >
> > Frank
--
Trond Myklebust
CTO, Hammerspace Inc
4300 El Camino Real, Suite 105
Los Altos, CA 94022
http://www.hammer.space