Bug Fixes:
The /proc/net/rpc/use-gss-proxy file can not be used
as a start up trigger for rpc.svcsgssd since it always
exists, with or without gss-proxy installed.
Adding the Wants= to the nfs server unit cause a systemd ordering
cycle which caused reboots to take forever.
Comment One:
Even though nfs-client does conditionally start the rpc.gssd
service when /etc/krb5.keytab exists (which good), but that's
all it does. Meaning 'systemctl status' does not show that rpc.gssd
is running and 'systemctl restart' does not restart rpc.gssd
and 'systemctl stop' does not stop the daemon.
It just seems odd to me that we will be using one target unit
to enable a daemon then another service unit (rpc-gssd) to
control it.
I thinking we should have one service unit, when enabled, control
both the rpc.gssd and the rpc.svcgssd daemons. The start up trigger
for both daemons will be the existence of /etc/krb5.keytab and
rpc.svcgssd will only be started if the nfs server is
enabled (if that is possible).
Comment Two:
How about renaming the nfs-utils unit to nfs-services since a
'systemctl restart' of the unit start all the server and client
daemons (even when the server is not enabled, which is probably a bug).
Since a 'systemctl restart nfs-utils' starts all the daemons shouldn't
'systemctl stop nfs-utils' bring them all down as well?
Another oddity, going a 'systemctl restart nfs-utils' causes v3
mounts to go stale... Meaning going a ls on a v3 mount point
after the restart errors out with ESTALE... Not sure why...
Comment Three:
I'm not seeing how the nfs-utils_env.sh file, called by each unit,
is all that useful. The main reason is you can not tell which
unit its being called from so how do know what should be done?
I guess I'm just missing the concept on how and what it should
be used for.
Steve Dickson (2):
rpc-svcgssd.service: removed a the start up triggers
systemd: Removed the "ordering cycle" from nfs-server.service
systemd/nfs-server.service | 2 --
systemd/rpc-svcgssd.service | 1 -
2 files changed, 3 deletions(-)
--
1.8.5.3
The file /proc/net/rpc/use-gss-proxy exists whether
gss-proxy is or is not installed. So is existence
can not be used as start up trigger.
Signed-off-by: Steve Dickson <[email protected]>
---
systemd/rpc-svcgssd.service | 1 -
1 file changed, 1 deletion(-)
diff --git a/systemd/rpc-svcgssd.service b/systemd/rpc-svcgssd.service
index 6bd4588..8e10b98 100644
--- a/systemd/rpc-svcgssd.service
+++ b/systemd/rpc-svcgssd.service
@@ -7,7 +7,6 @@ PartOf=nfs-utils.service
After=gssproxy.service
ConditionPathExists=|!@localstatedir@/run/gssproxy.pid
-ConditionPathExists=|!/proc/net/rpc/use-gss-proxy
ConditionPathExists=/etc/krb5.keytab
[Service]
--
1.8.5.3
On Wed, 2014-02-19 at 14:17 +1100, NeilBrown wrote:
> On Tue, 18 Feb 2014 03:48:48 -0500 Steve Dickson <[email protected]> wrote:
>
> > The file /proc/net/rpc/use-gss-proxy exists whether
> > gss-proxy is or is not installed. So is existence
> > can not be used as start up trigger.
> >
> > Signed-off-by: Steve Dickson <[email protected]>
> > ---
> > systemd/rpc-svcgssd.service | 1 -
> > 1 file changed, 1 deletion(-)
> >
> > diff --git a/systemd/rpc-svcgssd.service b/systemd/rpc-svcgssd.service
> > index 6bd4588..8e10b98 100644
> > --- a/systemd/rpc-svcgssd.service
> > +++ b/systemd/rpc-svcgssd.service
> > @@ -7,7 +7,6 @@ PartOf=nfs-utils.service
> >
> > After=gssproxy.service
> > ConditionPathExists=|!@localstatedir@/run/gssproxy.pid
> > -ConditionPathExists=|!/proc/net/rpc/use-gss-proxy
> > ConditionPathExists=/etc/krb5.keytab
> >
> > [Service]
>
> This is testing for non-existence, not existence.
>
> We need rpc-svcgssd unless gssproxy can perform that task for us, in which
> case we don't want rpc-svcgssd.
>
> gssproxy can only perform the task for us if:
> 1/ it is running (hence the test on the pid file)
could be stale
> 2/ the kernel knows how to talk to it (hence the test on use-gss-proxy)
use-gss-proxy is always available whether gss-proxy is being used or not
as it is the interface used by gss-proxy to tell the kernel it is in
use.
> With you version, if gssproxy were running on a kernel which did not include
> support for nfsd talking to gssproxy, then rpc-svcgssd would fail to start,
> which isn't what we want.
True the right way to handle this would probably be to
make /proc/net/rpc/use-gss-proxy readable and start rpc.svcgssd if the
value is 0 ?
> This may be an unusual configuration but unless it is an impossible
> configuration I would rather the code remained as it was.
> (This should work on *all* distros, not just sane distros).
I agree with the sentiment.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
On Wed, 19 Feb 2014 11:03:36 -0500 Simo Sorce <[email protected]> wrote:
> On Wed, 2014-02-19 at 14:17 +1100, NeilBrown wrote:
> > On Tue, 18 Feb 2014 03:48:48 -0500 Steve Dickson <[email protected]> wrote:
> >
> > > The file /proc/net/rpc/use-gss-proxy exists whether
> > > gss-proxy is or is not installed. So is existence
> > > can not be used as start up trigger.
> > >
> > > Signed-off-by: Steve Dickson <[email protected]>
> > > ---
> > > systemd/rpc-svcgssd.service | 1 -
> > > 1 file changed, 1 deletion(-)
> > >
> > > diff --git a/systemd/rpc-svcgssd.service b/systemd/rpc-svcgssd.service
> > > index 6bd4588..8e10b98 100644
> > > --- a/systemd/rpc-svcgssd.service
> > > +++ b/systemd/rpc-svcgssd.service
> > > @@ -7,7 +7,6 @@ PartOf=nfs-utils.service
> > >
> > > After=gssproxy.service
> > > ConditionPathExists=|!@localstatedir@/run/gssproxy.pid
> > > -ConditionPathExists=|!/proc/net/rpc/use-gss-proxy
> > > ConditionPathExists=/etc/krb5.keytab
> > >
> > > [Service]
> >
> > This is testing for non-existence, not existence.
> >
> > We need rpc-svcgssd unless gssproxy can perform that task for us, in which
> > case we don't want rpc-svcgssd.
> >
> > gssproxy can only perform the task for us if:
> > 1/ it is running (hence the test on the pid file)
>
> could be stale
Agreed. But it is the best we have. Systemd doesn't support
ConditionUnitActive
or similar.
>
> > 2/ the kernel knows how to talk to it (hence the test on use-gss-proxy)
>
> use-gss-proxy is always available whether gss-proxy is being used or not
> as it is the interface used by gss-proxy to tell the kernel it is in
> use.
It is not available in linux 3.9 or earlier.
>
> > With you version, if gssproxy were running on a kernel which did not include
> > support for nfsd talking to gssproxy, then rpc-svcgssd would fail to start,
> > which isn't what we want.
>
> True the right way to handle this would probably be to
> make /proc/net/rpc/use-gss-proxy readable and start rpc.svcgssd if the
> value is 0 ?
Pity systemd doesn't have
ConditionPathIsFileContaining
Even udev has that!
>
> > This may be an unusual configuration but unless it is an impossible
> > configuration I would rather the code remained as it was.
> > (This should work on *all* distros, not just sane distros).
>
> I agree with the sentiment.
Thanks,
NeilBrown
>
> Simo.
>
On Tue, 18 Feb 2014 03:48:48 -0500 Steve Dickson <[email protected]> wrote:
> The file /proc/net/rpc/use-gss-proxy exists whether
> gss-proxy is or is not installed. So is existence
> can not be used as start up trigger.
>
> Signed-off-by: Steve Dickson <[email protected]>
> ---
> systemd/rpc-svcgssd.service | 1 -
> 1 file changed, 1 deletion(-)
>
> diff --git a/systemd/rpc-svcgssd.service b/systemd/rpc-svcgssd.service
> index 6bd4588..8e10b98 100644
> --- a/systemd/rpc-svcgssd.service
> +++ b/systemd/rpc-svcgssd.service
> @@ -7,7 +7,6 @@ PartOf=nfs-utils.service
>
> After=gssproxy.service
> ConditionPathExists=|!@localstatedir@/run/gssproxy.pid
> -ConditionPathExists=|!/proc/net/rpc/use-gss-proxy
> ConditionPathExists=/etc/krb5.keytab
>
> [Service]
This is testing for non-existence, not existence.
We need rpc-svcgssd unless gssproxy can perform that task for us, in which
case we don't want rpc-svcgssd.
gssproxy can only perform the task for us if:
1/ it is running (hence the test on the pid file)
2/ the kernel knows how to talk to it (hence the test on use-gss-proxy)
With you version, if gssproxy were running on a kernel which did not include
support for nfsd talking to gssproxy, then rpc-svcgssd would fail to start,
which isn't what we want.
This may be an unusual configuration but unless it is an impossible
configuration I would rather the code remained as it was.
(This should work on *all* distros, not just sane distros).
Thanks,
NeilBrown
Hi Steve-
On Feb 18, 2014, at 1:44 PM, Steve Dickson <[email protected]> wrote:
>
>
> On 02/18/2014 09:29 AM, Chuck Lever wrote:
>> On Feb 18, 2014, at 3:48 AM, Steve Dickson <[email protected]> wrote:
>>
>>>> Bug Fixes:
>>>>
>>>> The /proc/net/rpc/use-gss-proxy file can not be used
>>>> as a start up trigger for rpc.svcsgssd since it always
>>>> exists, with or without gss-proxy installed.
>>>>
>>>> Adding the Wants= to the nfs server unit cause a systemd ordering
>>>> cycle which caused reboots to take forever.
>>>>
>>>> Comment One:
>>>>
>>>> Even though nfs-client does conditionally start the rpc.gssd
>>>> service when /etc/krb5.keytab exists (which good),
>> No, that's bad. rpc.gssd has to run in cases where there is no /etc/krb5.ktab. See
>>
>> http://www.spinics.net/lists/linux-nfs/msg41585.html
> At this point its a pipe dream for rpc.gssd to run with no keytab.
This is not a pipe dream. I?m talking about the common use case where a user kinit?s as root then uses the ?-n? option on gssd so that root?s credential is used as the client?s machine credential, instead of using the keytab to establish a GSS context.
With the exception of kernels 3.9 - 3.12, this has always worked, does require gssd to be running, and does not need to have a keytab on the client to operate correctly. When 3.9 broke this feature, people (including NeilB!) complained loudly.
> It logs a ton of errors messages on every upcall which means
> on every mount these days.
> We either have to tone down the error messages or check for the
> existence of the keytab before processing the upcall.
> I think the latter would better?
gssd default verbosity is a legacy of the days when Kerberized NFS was new and we wanted verbose logging to monitor gssd activity. It seems like a harmless step forward to eliminate them or hide some or all of them behind a gssd command line option.
The bug here is gssd log diarrhea.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On 02/18/2014 02:36 PM, Chuck Lever wrote:
>> At this point its a pipe dream for rpc.gssd to run with no keytab.
> This is not a pipe dream.
I meant its a pipe dream in the current state due to, your nicely
termed, "gssd log diarrhea" ;-)
steved.
> I?m talking about the common use case where a user kinit?s as root then uses the ?-n? option on gssd so that root?s credential is used as the client?s machine credential, instead of using the keytab to establish a GSS context.
>
> With the exception of kernels 3.9 - 3.12, this has always worked, does require gssd to be running, and does not need to have a keytab on the client to operate correctly. When 3.9 broke this feature, people (including NeilB!) complained loudly.
>
>> > It logs a ton of errors messages on every upcall which means
>> > on every mount these days.
>> > We either have to tone down the error messages or check for the
>> > existence of the keytab before processing the upcall.
>> > I think the latter would better?
> gssd default verbosity is a legacy of the days when Kerberized NFS was new and we wanted verbose logging to monitor gssd activity. It seems like a harmless step forward to eliminate them or hide some or all of them behind a gssd command line option.
>
> The bug here is gssd log diarrhea.
On Feb 18, 2014, at 3:48 AM, Steve Dickson <[email protected]> wrote:
> Bug Fixes:
>
> The /proc/net/rpc/use-gss-proxy file can not be used
> as a start up trigger for rpc.svcsgssd since it always
> exists, with or without gss-proxy installed.
>
> Adding the Wants= to the nfs server unit cause a systemd ordering
> cycle which caused reboots to take forever.
>
> Comment One:
>
> Even though nfs-client does conditionally start the rpc.gssd
> service when /etc/krb5.keytab exists (which good),
No, that's bad. rpc.gssd has to run in cases where there is no /etc/krb5.ktab. See
http://www.spinics.net/lists/linux-nfs/msg41585.html
The existence of /etc/krb5.conf is a better choice.
> but that's
> all it does. Meaning 'systemctl status' does not show that rpc.gssd
> is running and 'systemctl restart' does not restart rpc.gssd
> and 'systemctl stop' does not stop the daemon.
>
> It just seems odd to me that we will be using one target unit
> to enable a daemon then another service unit (rpc-gssd) to
> control it.
>
> I thinking we should have one service unit, when enabled, control
> both the rpc.gssd and the rpc.svcgssd daemons. The start up trigger
> for both daemons will be the existence of /etc/krb5.keytab and
> rpc.svcgssd will only be started if the nfs server is
> enabled (if that is possible).
>
> Comment Two:
>
> How about renaming the nfs-utils unit to nfs-services since a
> 'systemctl restart' of the unit start all the server and client
> daemons (even when the server is not enabled, which is probably a bug).
>
> Since a 'systemctl restart nfs-utils' starts all the daemons shouldn't
> 'systemctl stop nfs-utils' bring them all down as well?
>
> Another oddity, going a 'systemctl restart nfs-utils' causes v3
> mounts to go stale... Meaning going a ls on a v3 mount point
> after the restart errors out with ESTALE... Not sure why...
>
> Comment Three:
>
> I'm not seeing how the nfs-utils_env.sh file, called by each unit,
> is all that useful. The main reason is you can not tell which
> unit its being called from so how do know what should be done?
> I guess I'm just missing the concept on how and what it should
> be used for.
>
> Steve Dickson (2):
> rpc-svcgssd.service: removed a the start up triggers
> systemd: Removed the "ordering cycle" from nfs-server.service
>
> systemd/nfs-server.service | 2 --
> systemd/rpc-svcgssd.service | 1 -
> 2 files changed, 3 deletions(-)
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Commit 88ff0417 introduce 'Wants=' statements that
conditionally started other services. Those services
need to be removed from the 'After=' statements
otherwise a systemd ordering cycle is created.
Signed-off-by: Steve Dickson <[email protected]>
---
systemd/nfs-server.service | 2 --
1 file changed, 2 deletions(-)
diff --git a/systemd/nfs-server.service b/systemd/nfs-server.service
index 26df656..43be4aa 100644
--- a/systemd/nfs-server.service
+++ b/systemd/nfs-server.service
@@ -7,8 +7,6 @@ Wants=rpc-statd.service nfs-idmapd.service rpc-gssd.service rpc-svcgssd.service
Wants=rpc-statd-notify.service
After= network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
-After= nfs-idmapd.service rpc-statd.service
-After= rpc-gssd.service rpc-svcgssd.service
Before= rpc-statd-notify.service
[Service]
--
1.8.5.3
On Tue, 18 Feb 2014 03:48:47 -0500 Steve Dickson <[email protected]> wrote:
> Bug Fixes:
>
> The /proc/net/rpc/use-gss-proxy file can not be used
> as a start up trigger for rpc.svcsgssd since it always
> exists, with or without gss-proxy installed.
>
> Adding the Wants= to the nfs server unit cause a systemd ordering
> cycle which caused reboots to take forever.
>
> Comment One:
>
> Even though nfs-client does conditionally start the rpc.gssd
> service when /etc/krb5.keytab exists (which good), but that's
> all it does. Meaning 'systemctl status' does not show that rpc.gssd
> is running and 'systemctl restart' does not restart rpc.gssd
> and 'systemctl stop' does not stop the daemon.
Well it doesn't *only* start rpc.gssd, it also starts e.g. sm-notify. But
maybe you meant that. It only starts things, it doesn't stop them.
This how systemd works. Certainly there is no way I can see for
systemctl status foo
to report on anything but a single service: foo. It might be nice if it
could report on a collection of services, but that doesn't seem to be
supported.
We could arrange that when you 'stop' nfs-client it would also stop rpc-gssd
(unless the server was running and wanted it) by setting
StopWhenUnneeded=yes
That isn't the default in systemd so I'm cautious of setting it without a
good reason. However I don't object if it turns out that it works.
Restart is a bit tricky - again because both the client and the server want
some services.
You can enable Restart propagation by using PartOf=. However that also
enables Stop propagation and we don't want that. i.e. we don't want
systemctl stop nfs-client
to unilaterally stop rpc.gssd - in particular not if nfs-server is running.
In short: I think that the inter-relationships in nfs-utils are too complex to
allow the simplicity that you are seeking. Or systemd isn't sophisticated
enough to represent them.
>
> It just seems odd to me that we will be using one target unit
> to enable a daemon then another service unit (rpc-gssd) to
> control it.
>
> I thinking we should have one service unit, when enabled, control
> both the rpc.gssd and the rpc.svcgssd daemons. The start up trigger
> for both daemons will be the existence of /etc/krb5.keytab and
> rpc.svcgssd will only be started if the nfs server is
> enabled (if that is possible).
I originally thought that would not practical as the requirements for server
and client are different. However I now see (thanks Bruce) that both the
client and the server need both rpc.gssd and rpc.svcgssd. So we can have an
nfs-secure
(or similar) which Wants both rpc-gssd and rpc-svcgssd. Then both server and
client Want it.
However I don't think it should be "enabled". It should automatically start
whenever nfs-server or nfs-client is started, providing the relevant files
exist.
(It can be 'masked' if someone has installed krb5 but really doesn't want the
gss daemons running).
So you would still use a different service to enable it than to restart it.
Also "systemctl status nfs-secure" would not be able to show you the status
of both daemons.
>
> Comment Two:
>
> How about renaming the nfs-utils unit to nfs-services since a
> 'systemctl restart' of the unit start all the server and client
> daemons (even when the server is not enabled, which is probably a bug).
>
I don't object to a renaming. The name "nfs-utils" was simply the first
thing I thought of.
systemctl restart nfs-utils
should *not* start any service that is not already started. I think I tested
that. However if I'm wrong we could argue that "try-restart" should be used
if you don't want to start services that aren't already running.... though
I'm not sure that would work. Must test....
systemctl stop nfs-utils
should stop services. It uses "PartOf" which is supposed to apply to
'restart' and 'stop' equally.
> Since a 'systemctl restart nfs-utils' starts all the daemons shouldn't
> 'systemctl stop nfs-utils' bring them all down as well?
Doesn't it?
>
> Another oddity, going a 'systemctl restart nfs-utils' causes v3
> mounts to go stale... Meaning going a ls on a v3 mount point
> after the restart errors out with ESTALE... Not sure why...
>
That is weird and certainly needs looking in to. I could be due to
ExecStopPost=/usr/sbin/exportfs -au
I wonder if we really need that.
I hope to get some time tomorrow to experiment with all this and I will
definitely look into this issue (unless you explain it first :-)
> Comment Three:
>
> I'm not seeing how the nfs-utils_env.sh file, called by each unit,
> is all that useful. The main reason is you can not tell which
> unit its being called from so how do know what should be done?
> I guess I'm just missing the concept on how and what it should
> be used for.
I've got a patch which changes this so it is only called once in a separate
service. However the principle remains unchanged.
The purpose of the script is to read /etc/sysconfig/nfs (or similar) and
write out /run/sysconfig/nfs-utils containing *all* the environment variables
used in *any* nfs-utils unit file. It allows a translation from what is
expected in the sysconfig file to what the daemons expect as their arguments.
I see this as a transitional thing. Ultimately the various daemons should be
able to inspect their environment, and expect exactly the things that might
be in the config file. So systemd just reads /etc/sysconfig/nfs and then runs
the daemons. But that goal is a little ways off yet.
One thing we have in our sysconfig file is "NFS3_SERVER_SUPPORT". If that is
set to "no", then as well as constructing appropriate arguments for rpc.mountd
and rpc.nfs we don't run rpc.statd (unless a v3 filesystem is mounted).
Handling the first part with nfs-utils_env.sh is trivial. Handling the
second bit is something I haven't figured out yet.
Thanks for your comments.
NeilBrown
>
> Steve Dickson (2):
> rpc-svcgssd.service: removed a the start up triggers
> systemd: Removed the "ordering cycle" from nfs-server.service
>
> systemd/nfs-server.service | 2 --
> systemd/rpc-svcgssd.service | 1 -
> 2 files changed, 3 deletions(-)
>
On Tue, 18 Feb 2014 03:48:49 -0500 Steve Dickson <[email protected]> wrote:
> Commit 88ff0417 introduce 'Wants=' statements that
> conditionally started other services. Those services
> need to be removed from the 'After=' statements
> otherwise a systemd ordering cycle is created.
>
> Signed-off-by: Steve Dickson <[email protected]>
> ---
> systemd/nfs-server.service | 2 --
> 1 file changed, 2 deletions(-)
>
> diff --git a/systemd/nfs-server.service b/systemd/nfs-server.service
> index 26df656..43be4aa 100644
> --- a/systemd/nfs-server.service
> +++ b/systemd/nfs-server.service
> @@ -7,8 +7,6 @@ Wants=rpc-statd.service nfs-idmapd.service rpc-gssd.service rpc-svcgssd.service
> Wants=rpc-statd-notify.service
>
> After= network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
> -After= nfs-idmapd.service rpc-statd.service
> -After= rpc-gssd.service rpc-svcgssd.service
> Before= rpc-statd-notify.service
>
> [Service]
"Wants" does not imply ordering. If A Wants B, then A and B can be started
at the same time.
But we really want these various services to be running before nfsd is
started. So the "After" are important. Please leave them.
The lack of ordering is explicit in the documentation for Requires:
If a unit foo.service requires a unit bar.service as
configured with Requires= and no ordering is configured with After=
or Before=, then both units will be started simultaneously and
without any delay between them if foo.service is activated.
and implicit in the documentation for Wants:
Wants=
A weaker version of Requires=.
Thanks,
NeilBrown
On 02/18/2014 09:29 AM, Chuck Lever wrote:
> On Feb 18, 2014, at 3:48 AM, Steve Dickson <[email protected]> wrote:
>
>> > Bug Fixes:
>> >
>> > The /proc/net/rpc/use-gss-proxy file can not be used
>> > as a start up trigger for rpc.svcsgssd since it always
>> > exists, with or without gss-proxy installed.
>> >
>> > Adding the Wants= to the nfs server unit cause a systemd ordering
>> > cycle which caused reboots to take forever.
>> >
>> > Comment One:
>> >
>> > Even though nfs-client does conditionally start the rpc.gssd
>> > service when /etc/krb5.keytab exists (which good),
> No, that's bad. rpc.gssd has to run in cases where there is no /etc/krb5.ktab. See
>
> http://www.spinics.net/lists/linux-nfs/msg41585.html
At this point its a pipe dream for rpc.gssd to run with no keytab.
It logs a ton of errors messages on every upcall which means
on every mount these days.
We either have to tone down the error messages or check for the
existence of the keytab before processing the upcall.
I think the latter would better...
>
> The existence of /etc/krb5.conf is a better choice.
>
Fine... If there is no objection... I'm ok with it..
steved.