Hi all,
it appears that systemd doesn't play well with NFS "bg" mounts.
I can see a few options for how to address this and wonder if anyone
has any opinions.
"bg" mounts will try to mount the filesystem just like normal, but
if the server cannot be contacted before a "major timeout" (4.5 minutes
by default for TCP), mount.nfs will fork and continue in the
background. Meanwhile the original mount process reports success (even
though the filesystem wasn't mounted). This allows the boot process to
continue and succeed.
Currently if you specify the "bg" option in /etc/fstab and are using
systemd, the "bg" has no useful effect.
systemd imposes its own timeout of 90 seconds (which is less than 4.5
minutes).
After 90 seconds, systemd will kill the mount process and decide that
the mount failed. This will lead to remote-fs.target not being reached,
and boot not completing.
If you set TimeoutSec=0 (aka "infinity") for the mount unit, either by
hacking fstab-generator or adding "x-systemd.mount-timeout=infinity"
if you have systemd 233 or later, then systemd won't kill the mount
process, and after 4.5 minutes it will exit.
(to quote a comment in systemd/src/core/mount.c
" /bin/mount lies to us and is broken"
:-)
This is better, but the background mount.nfs can persist for a long
time. I don't think it persists forever, but at least half an hour I
think.
When the foo.mount unit is stopped, the mount.nfs process isn't killed.
I don't think this is a major problem, but it is unfortunate and could
be confusing. During testing I've had multiple mount.nfs background
processes all attached to the one .mount unit.
What should we do about bg NFS mounts with systemd?
Some options:
- declare "bg" to be not-supported. If you don't need the filesystem
to be present for boot, then use x-systemd.automount, or some other
automount scheme.
If we did this, we probably need to make it very obvious that "bg"
mounts aren't supported - maybe a log message that appears when
you do "systemctl status ..." ??
- decide that "bg" is really just a lame attempt at automounting, and
that now we have real automounting, "bg" can be translated to that.
So systemd-fstab-generator would treat "bg" like
"x-systemd.automount" and otherwise strip it from the list of
options.
- do our best to really support "bg". That means, at least, applying
a much larger timeout to "bg" mounts, and preferably killing any
background processes when a mount unit is stopped. Below is a
little patch which does this last bit, but I'm not sure it is generally
safe.
A side question is: should this knowledge about NFS be encoded in
systemd, or should nfs-utils add the necessary knowledge?
i.e. we could add an nfs-fstab-generator to nfs-utils which creates
drop-ins to modify the behaviour of the drop-ins provided by
systemd-fstab-generator.
Adding the TimeoutSec= would be easy. Stripping the "bg" would be
possible.
Changing the remote-fs.target.requires/foo.mount symlink to be
remote-fs.target.requires/foo.automount would be problematic though.
Could we teach systemd-fstab-generator to ignore $TYPE filesystems
if TYPE-fstab-generator existed?
Or should we just build all this filesystem-specific knowledge into
systemd?
Thanks for your thoughts,
NeilBrown
hackish patch to kill backgrounded mount.nfs processes:
diff --git a/src/core/mount.c b/src/core/mount.c
index ca0c4b0d5eed..91939b48d11a 100644
--- a/src/core/mount.c
+++ b/src/core/mount.c
@@ -883,6 +883,18 @@ static void mount_enter_unmounting(Mount *m) {
MOUNT_UNMOUNTING_SIGKILL))
m->n_retry_umount = 0;
+ if (m->result == MOUNT_SUCCESS &&
+ !m->from_proc_self_mountinfo) {
+ /* There is no mountpoint, but mount seemed to succeed.
+ * Could be a bg mount.nfs.
+ * In any case, kill any processes that might be hanging
+ * around, they cannot be doing anything useful.
+ */
+ sd_bus_error error = SD_BUS_ERROR_NULL;
+ unit_kill_common(UNIT(m), KILL_ALL, SIGTERM, -1, -1, &error);
+ }
+
+
m->control_command_id = MOUNT_EXEC_UNMOUNT;
m->control_command = m->exec_command + MOUNT_EXEC_UNMOUNT;
On Fri, 26.05.17 12:46, NeilBrown ([email protected]) wrote:
>
> Hi all,
> it appears that systemd doesn't play well with NFS "bg" mounts.
> I can see a few options for how to address this and wonder if anyone
> has any opinions.
Yeah, this has come up before. Long story short: "bg" is simply not
compatible with systemd. We assume that the /bin/mount's effect is
visible in /proc/self/mountinfo, and everything else is considered a
bug, i.e. /bin/mount lying to us. And I think that's a pretty rational
assumption and requirement to make.
I am not particularly convinced the "bg" usecase is really such a
great idea, since it is necessarily racy: you never know whether mount
is actually in effect or not, even though /bin/mount claims so. I am
pretty sure other options (such as autofs mounts, which are dead-easy
to use in system: just replace the "bg" mount option in fstab by
"x-systemd.automount") are much better approaches to the problem at
hand: they also make your local system less dependent on remote
network access, but they do give proper guarantees about their
runtime: when the autofs mount is established the path is available.
Hence I am tempted to treat the issue as a documentation and warning
issue: accept that "bg" is not supported, but document this better. In
addition, we should probably log about "bg" being broken in the
fstab-generator. I file a bug about that now:
https://github.com/systemd/systemd/issues/6046
> This is better, but the background mount.nfs can persist for a long
> time. I don't think it persists forever, but at least half an hour I
> think.
>
> When the foo.mount unit is stopped, the mount.nfs process isn't
> killed.
Hmm, if you see this, then this would be a bug: mount units that are
stopped should not leave processes around.
> I don't think this is a major problem, but it is unfortunate and could
> be confusing. During testing I've had multiple mount.nfs background
> processes all attached to the one .mount unit.
Humpf, could you file a bug?
While I think the "bg" concept is broken, as discussed above, having
FUSE mounts with processes in the background is actually supported,
and we should clean them up properly when the mount unit is stopped.
Hmm, maybe mount.nfs isn't properly killable? i.e. systemd tries to
kill it, but it refuses to be killed?
> What should we do about bg NFS mounts with systemd?
> Some options:
> - declare "bg" to be not-supported. If you don't need the filesystem
> to be present for boot, then use x-systemd.automount, or some other
> automount scheme.
> If we did this, we probably need to make it very obvious that "bg"
> mounts aren't supported - maybe a log message that appears when
> you do "systemctl status ..." ??
I am all for this, as suggested above. I'd only log from
fstab-generator though. (That said, if we want something stronger, we
could also add the fact that we stripped "bg" from the mount optoins
to the DEscription= of the generated mount unit.)
> - decide that "bg" is really just a lame attempt at automounting, and
> that now we have real automounting, "bg" can be translated to that.
> So systemd-fstab-generator would treat "bg" like
> "x-systemd.automount" and otherwise strip it from the list of
> options.
I am a bit afraid of this I must say. The semantics are different
enough to likely cause more problems then we'd solve with this. Not
supporting this at all sounds like the much better approach here:
let's strip "bg" when specified.
> - do our best to really support "bg". That means, at least, applying
> a much larger timeout to "bg" mounts, and preferably killing any
> background processes when a mount unit is stopped. Below is a
> little patch which does this last bit, but I'm not sure it is generally
> safe.
As mentioned I think this would just trade one race for a couple of
new ones, and that appears to be a bad idea to me.
> A side question is: should this knowledge about NFS be encoded in
> systemd, or should nfs-utils add the necessary knowledge?
I am pretty sure we should keep special understanding of NFS at a
minimum in PID 1, but I think we can be less strict in
fstab-generator, as its primary job is compat with UNIX anyway.
>
> i.e. we could add an nfs-fstab-generator to nfs-utils which creates
> drop-ins to modify the behaviour of the drop-ins provided by
> systemd-fstab-generator.
> Adding the TimeoutSec= would be easy. Stripping the "bg" would be
> possible.
> Changing the remote-fs.target.requires/foo.mount symlink to be
> remote-fs.target.requires/foo.automount would be problematic
> though.
Well, I'd be fine with letting NFS do its own handling of the NFS
/etc/fstab entries, but I think the special casing of "bg" is fine to
simply add to the existing generator in systemd.
> Could we teach systemd-fstab-generator to ignore $TYPE filesystems
> if TYPE-fstab-generator existed?
Well, generators can override each other in very limited ways (as
there are three different output directories a gnerator can write to,
which are inserted at different places in the unit file search path),
we could build on that. That said, I think adding this to
fstab-generator in systemd is OK.
> Or should we just build all this filesystem-specific knowledge into
> systemd?
For now, I think adding this to systemd's fstab-generator would be fine.
Hope this makes sense,
Lennart
--
Lennart Poettering, Red Hat
On Mon, May 29 2017, Lennart Poettering wrote:
> On Fri, 26.05.17 12:46, NeilBrown ([email protected]) wrote:
>
>>
>> Hi all,
>> it appears that systemd doesn't play well with NFS "bg" mounts.
>> I can see a few options for how to address this and wonder if anyone
>> has any opinions.
>
> Yeah, this has come up before. Long story short: "bg" is simply not
> compatible with systemd. We assume that the /bin/mount's effect is
> visible in /proc/self/mountinfo, and everything else is considered a
> bug, i.e. /bin/mount lying to us. And I think that's a pretty rational
> assumption and requirement to make.
>
> I am not particularly convinced the "bg" usecase is really such a
> great idea, since it is necessarily racy: you never know whether mount
> is actually in effect or not, even though /bin/mount claims so. I am
> pretty sure other options (such as autofs mounts, which are dead-easy
> to use in system: just replace the "bg" mount option in fstab by
> "x-systemd.automount") are much better approaches to the problem at
> hand: they also make your local system less dependent on remote
> network access, but they do give proper guarantees about their
> runtime: when the autofs mount is established the path is available.
>
> Hence I am tempted to treat the issue as a documentation and warning
> issue: accept that "bg" is not supported, but document this better. In
> addition, we should probably log about "bg" being broken in the
> fstab-generator. I file a bug about that now:
>
> https://github.com/systemd/systemd/issues/6046
There is a weird distorted sort of justice here. When NFS first
appeared, it broke various long-standing Unix practices, such as
O_EXCL|O_CREAT for lock files. Now systemd appears and breaks a
long-standing NFS practice: bg mounts.
I hoped we could find a way to make them work, but I won't cry over
their demise. I much prefer automount .... I think all NFS mounts
should be automounts.
I see this is already documented in systemd.mount:
The NFS mount option bg for NFS background mounts as documented in nfs(5) is
not supported in /etc/fstab entries.
I wonder how many people actually read that.
We should probably add symmetric documentation to nfs(5)
Both should give clear pointers to x-systemd.automount.
>
>> This is better, but the background mount.nfs can persist for a long
>> time. I don't think it persists forever, but at least half an hour I
>> think.
>>
>> When the foo.mount unit is stopped, the mount.nfs process isn't
>> killed.
>
> Hmm, if you see this, then this would be a bug: mount units that are
> stopped should not leave processes around.
>
>> I don't think this is a major problem, but it is unfortunate and could
>> be confusing. During testing I've had multiple mount.nfs background
>> processes all attached to the one .mount unit.
>
> Humpf, could you file a bug?
https://github.com/systemd/systemd/issues/6048
>
> While I think the "bg" concept is broken, as discussed above, having
> FUSE mounts with processes in the background is actually supported,
> and we should clean them up properly when the mount unit is stopped.
>
> Hmm, maybe mount.nfs isn't properly killable? i.e. systemd tries to
> kill it, but it refuses to be killed?
mount.nfs responds cleanly to SIGTERM.
>
>> What should we do about bg NFS mounts with systemd?
>> Some options:
>> - declare "bg" to be not-supported. If you don't need the filesystem
>> to be present for boot, then use x-systemd.automount, or some other
>> automount scheme.
>> If we did this, we probably need to make it very obvious that "bg"
>> mounts aren't supported - maybe a log message that appears when
>> you do "systemctl status ..." ??
>
> I am all for this, as suggested above. I'd only log from
> fstab-generator though. (That said, if we want something stronger, we
> could also add the fact that we stripped "bg" from the mount optoins
> to the DEscription= of the generated mount unit.)
That last bit sounds like a very good idea. Stripping "bg" could be seen
as a "surprising" thing for fstab-generator to do. Making it as obvious
as possible to the sys-admin would be a good thing (and would probably
make support personnel happy too).
>
>> - decide that "bg" is really just a lame attempt at automounting, and
>> that now we have real automounting, "bg" can be translated to that.
>> So systemd-fstab-generator would treat "bg" like
>> "x-systemd.automount" and otherwise strip it from the list of
>> options.
>
> I am a bit afraid of this I must say. The semantics are different
> enough to likely cause more problems then we'd solve with this. Not
> supporting this at all sounds like the much better approach here:
> let's strip "bg" when specified.
>
>> - do our best to really support "bg". That means, at least, applying
>> a much larger timeout to "bg" mounts, and preferably killing any
>> background processes when a mount unit is stopped. Below is a
>> little patch which does this last bit, but I'm not sure it is generally
>> safe.
>
> As mentioned I think this would just trade one race for a couple of
> new ones, and that appears to be a bad idea to me.
>
>> A side question is: should this knowledge about NFS be encoded in
>> systemd, or should nfs-utils add the necessary knowledge?
>
> I am pretty sure we should keep special understanding of NFS at a
> minimum in PID 1, but I think we can be less strict in
> fstab-generator, as its primary job is compat with UNIX anyway.
I was thinking about which source package the knowledge would be in, and
hence which set of maintainers would over-see it. I don't expect
systemd maintainers to be fully in-touch with the details of NFS, but
then NFS developers cannot be fully in-touch with how systemd works.
Apart from some documentation changes, we probably don't need to put
anything new in nfs-utils at the moment.
>
>>
>> i.e. we could add an nfs-fstab-generator to nfs-utils which creates
>> drop-ins to modify the behaviour of the drop-ins provided by
>> systemd-fstab-generator.
>> Adding the TimeoutSec= would be easy. Stripping the "bg" would be
>> possible.
>> Changing the remote-fs.target.requires/foo.mount symlink to be
>> remote-fs.target.requires/foo.automount would be problematic
>> though.
>
> Well, I'd be fine with letting NFS do its own handling of the NFS
> /etc/fstab entries, but I think the special casing of "bg" is fine to
> simply add to the existing generator in systemd.
>
>> Could we teach systemd-fstab-generator to ignore $TYPE filesystems
>> if TYPE-fstab-generator existed?
>
> Well, generators can override each other in very limited ways (as
> there are three different output directories a gnerator can write to,
> which are inserted at different places in the unit file search path),
> we could build on that. That said, I think adding this to
> fstab-generator in systemd is OK.
Ahh, of course. That is what normal-dir / early-dir / late-dir is for.
I'll keep that in mind in case I do ever need it.
>
>> Or should we just build all this filesystem-specific knowledge into
>> systemd?
>
> For now, I think adding this to systemd's fstab-generator would be fine.
>
> Hope this makes sense,
Yes it does. It wasn't the outcome I was hoping for, but it is hard to
argue against it.
Thanks a lot,
NeilBrown
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.
Explain this.
See also https://github.com/systemd/systemd/issues/6046
Signed-off-by: NeilBrown <[email protected]>
---
utils/mount/nfs.man | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
index cc6e992ed807..7e76492d454f 100644
--- a/utils/mount/nfs.man
+++ b/utils/mount/nfs.man
@@ -372,6 +372,21 @@ Alternatively these issues can be addressed
using an automounter (refer to
.BR automount (8)
for details).
+.IP
+When
+.B systemd
+is used to mount the filesystems listed in
+.IR /etc/fstab ,
+the
+.B bg
+option is not supported, and may be stripped from the option list.
+Similar functionality can be achieved by providing the
+.B x-system.automount
+option. This will cause
+.B systemd
+to attempt to mount the filesystem when the mountpoint is first
+accessed, rather than during system boot. The mount still happens in
+the "background", though in a different way.
.TP 1.5i
.BR rdirplus " / " nordirplus
Selects whether to use NFS v3 or v4 READDIRPLUS requests.
@@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
.BR rpc.idmapd (8),
.BR rpc.gssd (8),
.BR rpc.svcgssd (8),
-.BR kerberos (1)
+.BR kerberos (1),
+.BR systemd.mount (5) .
.sp
RFC 768 for the UDP specification.
.br
--
2.12.2
On Tue, May 30, 2017 at 08:19:16AM +1000, NeilBrown wrote:
>
> Systemd does not, and will not, support "bg" correctly.
> It has other, better, ways to handle "background" mounting.
>
> Explain this.
>
> See also https://github.com/systemd/systemd/issues/6046
>
> Signed-off-by: NeilBrown <[email protected]>
> ---
> utils/mount/nfs.man | 18 +++++++++++++++++-
> 1 file changed, 17 insertions(+), 1 deletion(-)
>
> diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
> index cc6e992ed807..7e76492d454f 100644
> --- a/utils/mount/nfs.man
> +++ b/utils/mount/nfs.man
> @@ -372,6 +372,21 @@ Alternatively these issues can be addressed
> using an automounter (refer to
> .BR automount (8)
> for details).
> +.IP
> +When
> +.B systemd
> +is used to mount the filesystems listed in
> +.IR /etc/fstab ,
> +the
> +.B bg
> +option is not supported, and may be stripped from the option list.
> +Similar functionality can be achieved by providing the
> +.B x-system.automount
> +option. This will cause
> +.B systemd
> +to attempt to mount the filesystem when the mountpoint is first
> +accessed, rather than during system boot. The mount still happens in
> +the "background", though in a different way.
> .TP 1.5i
> .BR rdirplus " / " nordirplus
> Selects whether to use NFS v3 or v4 READDIRPLUS requests.
> @@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
> .BR rpc.idmapd (8),
> .BR rpc.gssd (8),
> .BR rpc.svcgssd (8),
> -.BR kerberos (1)
> +.BR kerberos (1),
> +.BR systemd.mount (5) .
> .sp
> RFC 768 for the UDP specification.
> .br
> --
> 2.12.2
>
I like this, it makes it so much easier for users to find the better
alternative.
FWIW,
Reviewed-by: Niels de Vos <[email protected]>
2017-05-30 0:19 GMT+02:00 NeilBrown <[email protected]>:
> +.B bg
> +option is not supported, and may be stripped from the option list.
Either systemd is updated to actually strip the bg option or not. The
documentation should reflect that.
I don't think we should be vague about this, as it would only be confusing.
Michael
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
On Tue, May 30 2017, Michael Biebl wrote:
> 2017-05-30 0:19 GMT+02:00 NeilBrown <[email protected]>:
>
>> +.B bg
>> +option is not supported, and may be stripped from the option list.
>
> Either systemd is updated to actually strip the bg option or not. The
> documentation should reflect that.
> I don't think we should be vague about this, as it would only be confusing.
It depends on which version of systemd is in use.
Even if/when systemd is updated to strip the 'bg', it would not
be correct to say "and will be stripped..." as that isn't true for all
versions.
We could spell it out "and may be stripped from the option list,
depending on which version of systemd is installed", but I'm not sure
that really helps...
"... and some versions of systemd will strip 'bg' from the option list"
??
Thanks,
NeilBrown
2017-05-30 10:55 GMT+02:00 NeilBrown <[email protected]>:
> On Tue, May 30 2017, Michael Biebl wrote:
>
>> 2017-05-30 0:19 GMT+02:00 NeilBrown <[email protected]>:
>>
>>> +.B bg
>>> +option is not supported, and may be stripped from the option list.
>>
>> Either systemd is updated to actually strip the bg option or not. The
>> documentation should reflect that.
>> I don't think we should be vague about this, as it would only be confusing.
>
> It depends on which version of systemd is in use.
> Even if/when systemd is updated to strip the 'bg', it would not
> be correct to say "and will be stripped..." as that isn't true for all
> versions.
> We could spell it out "and may be stripped from the option list,
> depending on which version of systemd is installed", but I'm not sure
> that really helps...
> "... and some versions of systemd will strip 'bg' from the option list"
> ??
So far, no version of systemd strips bg from the option list.
If in version X systemd actually starts stripping bg, we should
explicitly mention that version in the man page imho.
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
On Tue, 30.05.17 08:19, NeilBrown ([email protected]) wrote:
>
> Systemd does not, and will not, support "bg" correctly.
> It has other, better, ways to handle "background" mounting.
>
> Explain this.
>
> See also https://github.com/systemd/systemd/issues/6046
>
> Signed-off-by: NeilBrown <[email protected]>
> ---
> utils/mount/nfs.man | 18 +++++++++++++++++-
> 1 file changed, 17 insertions(+), 1 deletion(-)
>
> diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
> index cc6e992ed807..7e76492d454f 100644
> --- a/utils/mount/nfs.man
> +++ b/utils/mount/nfs.man
> @@ -372,6 +372,21 @@ Alternatively these issues can be addressed
> using an automounter (refer to
> .BR automount (8)
> for details).
> +.IP
> +When
> +.B systemd
> +is used to mount the filesystems listed in
> +.IR /etc/fstab ,
> +the
> +.B bg
> +option is not supported, and may be stripped from the option list.
> +Similar functionality can be achieved by providing the
> +.B x-system.automount
The option is called "x-systemd.automount", i.e. one more "d".
> +option. This will cause
> +.B systemd
> +to attempt to mount the filesystem when the mountpoint is first
> +accessed, rather than during system boot. The mount still happens in
> +the "background", though in a different way.
Looks good to me otherwise!
Lennart
--
Lennart Poettering, Red Hat
On Tue, 30.05.17 11:15, Michael Biebl ([email protected]) wrote:
> 2017-05-30 10:55 GMT+02:00 NeilBrown <[email protected]>:
> > On Tue, May 30 2017, Michael Biebl wrote:
> >
> >> 2017-05-30 0:19 GMT+02:00 NeilBrown <[email protected]>:
> >>
> >>> +.B bg
> >>> +option is not supported, and may be stripped from the option list.
> >>
> >> Either systemd is updated to actually strip the bg option or not. The
> >> documentation should reflect that.
> >> I don't think we should be vague about this, as it would only be confusing.
> >
> > It depends on which version of systemd is in use.
> > Even if/when systemd is updated to strip the 'bg', it would not
> > be correct to say "and will be stripped..." as that isn't true for all
> > versions.
> > We could spell it out "and may be stripped from the option list,
> > depending on which version of systemd is installed", but I'm not sure
> > that really helps...
> > "... and some versions of systemd will strip 'bg' from the option list"
> > ??
>
> So far, no version of systemd strips bg from the option list.
> If in version X systemd actually starts stripping bg, we should
> explicitly mention that version in the man page imho.
I figure it's up to the NFS folks to decide on this, but if I were
them I would not mention any version numbers. That just gets out of
date. Instead, document the new behaviour and indicate in the README
file that the documentation assumes systemd with a certain version is
used, and then let the packers figure out the rest, i.e. if they want
to place some kind of weak package version dependency or not.
Lennart
--
Lennart Poettering, Red Hat
On Thu, Jun 08, 2017 at 03:16:52PM +1000, NeilBrown wrote:
> So I think I've found a solution for systemd to handle "bg" nfs mounts
> correctly. I'll submit some pull requests for consideration.
Out of curiosity, after that change is there still any reason you'd
recommend any new user actually use "bg" (as opposed to an automount)?
I appreciate the effort to keep existing systems working, I'm just
curious.
--b.
On 06/08/2017 01:16 AM, NeilBrown wrote:
> On Wed, Jun 07 2017, Steve Dickson wrote:
>
>> On 06/07/2017 08:02 AM, Lennart Poettering wrote:
>>> On Wed, 07.06.17 06:08, Steve Dickson ([email protected]) wrote:
>>>
>>>>
>>>>
>>>> On 06/06/2017 05:49 PM, NeilBrown wrote:
>>>>> On Tue, Jun 06 2017, Steve Dickson wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> On 05/29/2017 06:19 PM, NeilBrown wrote:
>>>>>>>
>>>>>>> Systemd does not, and will not, support "bg" correctly.
>>>>>>> It has other, better, ways to handle "background" mounting.
>>>>>> The only problem with this is bg mounts still work at least
>>>>>> up to 4.11 kernel...
>>>>>
>>>>> I don't think this is correct.
>>>>> In the default configuration, "mount -t nfs -o bg ...."
>>>>> runs for longer than 90 seconds, so systemd kill it.
>>>> I must be missing something... it seems to be working for me
>>>>
>>>> # mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
>>>> mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
>>>> mount.nfs: mount(2): Connection refused
>>>> mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
>>>> mount.nfs: prog 100003, trying vers=3, prot=6
>>>> mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
>>>> mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
>>>> mount.nfs: backgrounding "rhel7srv:/home/tmp"
>>>> mount.nfs: mount options: "rw,bg"
>>>
>>> We are talking about mounts done through /etc/fstab, i.e. the ones
>>> systemd actually manages.
>> I guess I was not clear... After adding a bg mount to fstab and
>> reboot, mounting a server that is not up, there is a mount in
>> background that looks like
>>
>> # ps ax | grep mount
>> 1104 ? Ss 0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
>>
>> Looking at the remote-fs.target status:
>>
>> # systemctl status remote-fs.target
>> * remote-fs.target - Remote File Systems
>> Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
>> Active: active since Tue 2017-06-06 12:36:51 EDT; 12min ago
>> Docs: man:systemd.special(7)
>>
>> Jun 06 12:36:51 f26.boston.devel.redhat.com systemd[1]: Reached target Remote File Systems.
>>
>> It appears to be successful
>
> Strange ... not for me.
>
> I have a recent compiled-from-source upstream systemd and a very recent
> upstream kernel.
>
> I add a line to /etc/fstab
>
> 192.168.1.111:/nowhere /mnt nfs bg 0 0
>
> and reboot (192.168.1.111 is on a different subnet to the VM I am
> testing in, but no host responds to the address).
>
> There is a 1m30s wait while trying to mount /mnt, then boot completes
> (I was wrong when I said that it didn't).
>
> ● mnt.mount - /mnt
> Loaded: loaded (/etc/fstab; generated; vendor preset: enabled)
> Active: failed (Result: timeout) since Thu 2017-06-08 11:13:52 AEST; 1min 24s ago
> Where: /mnt
> What: 192.168.1.111:/nowhere
> Docs: man:fstab(5)
> man:systemd-fstab-generator(8)
> Process: 333 ExecMount=/bin/mount 192.168.1.111:/nowhere /mnt -t nfs -o bg (code=killed, signal=TERM)
>
> Jun 08 11:12:22 debian systemd[1]: Mounting /mnt...
> Jun 08 11:13:52 debian systemd[1]: mnt.mount: Mounting timed out. Stopping.
> Jun 08 11:13:52 debian systemd[1]: mnt.mount: Mount process exited, code=killed status=15
> Jun 08 11:13:52 debian systemd[1]: Failed to mount /mnt.
> Jun 08 11:13:52 debian systemd[1]: mnt.mount: Unit entered failed state.
>
>
> The /bin/mount process has been killed (SIGTERM) after the 90 second
> timeout
>
> ● remote-fs.target - Remote File Systems
> Loaded: loaded (/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
> Drop-In: /run/systemd/generator/remote-fs.target.d
> └─50-insserv.conf.conf
> Active: inactive (dead)
> Docs: man:systemd.special(7)
>
> Jun 08 11:13:52 debian systemd[1]: Dependency failed for Remote File Systems.
> Jun 08 11:13:52 debian systemd[1]: remote-fs.target: Job remote-fs.target/start failed with result 'dependency'.
>
>
> remote-fs.target has not been reached.
I'm seeing this now... the server has to be up to cause this hang.
>
> Because remote-fs.target is WantedBy multi-user.target, but need
> RequiredBy it, boot completes.
> But if anything did Require remote-fs.target, it would fail if "bg"
> mounts were not mounted within 90 seconds.
>
>
> Looking back over your log messages:
>
>>>> mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
>>>> mount.nfs: backgrounding "rhel7srv:/home/tmp"
>
> it appears that the fg mount attempt failed quickly (ECONNREFUSED), so it
> background the process immediately. Systemd sees that as success
> (despite the fact that the filesystem isn't actually mounted) and
> doesn't clean up the background processes (which is arguably a bug).
No... Systemd is doing the right thing in this case... Letting
bg mounts work...
>
> If you try to mount from a server which doesn't even return ECONNREFUSED or
> EHOSTUNREACH, such that the first mount attempt takes longer than 90
> seconds, then you should get the same results as me.
This is clearly a bug in systemd! The 'bg' mounts work with the server
down but hangs when the server is up. How can't this be a bug in systemd??
Why is systemd even looking at or interpreting the mount options???
This clear an overreach of systemd, IMHO. Then just to blindly rip
them out... WOW!
I'm only assuming... but it appears, do to this overreach, that systemd
actually think it know how to mount a file system better than the
actual mount command tailor for particular that file system and
if it doesn't like an mount option it just going to rip it
out! Again, I hope this is not the case because if its...
that's just crazy!!!
>
> Your results go some way to explaining why Lennart hasn't received many
> complaints, but I'm convinced that the current situation is not ideal
> (because SUSE has received some complaints).
It just proves bg mounts are being used.
>
> I've been pondering the possibility of making "bg" work properly with
> systemd and I think I've found a promising approach. It involves having
> systemd take responsibility for the "run in the background" part.
>
> If we get systemd-fstab-generator to translate "bg" to "retry=10000",
> then "mount.nfs" will behave like the background version of
> "mount.nfs -o bg". i.e. it will retry for one month (nearly). If there is
> already a 'retry=' option, we just strip the "bg".
>
> For this to work, we would need to add
> TimeoutSec=infinity
> to the .mount unit file so that systemd doesn't kill the mount.
> We would also need to add (the effect of) "nofail", so that systemd
> doesn't wait for the mount to complete...
> Except that the current implementation of "nofail" is faulty.
> It removes the default "Before=remote-fs.mount", which has the unwanted
> consequence of unmounting the filesystem too early at shutdown.
>
> I have a solution for that too (which I'll submit a push request for
> shortly).
> My solution to "nofail" is to treat it much like "automount", but
> instead of using an automount unit to trigger the mount, we use
> a timer unit (with OnActiveSec=0).
> By triggering the mount unit with a timer instead of Wanting it
> directly, it gets run in a separate transaction. This means that the
> "Before=remote-fs.target" doesn't have the effect of delaying
> remote-fs.target. Before/After only order jobs within the one
> transaction.
> When it comes time to shutdown, remote-fs.target and the foo.mount will
> be in the same transaction, so the Before= will ensure foo.mount
> isn't unmounted until after remote-fs.target has been allows to finish,
> which is after any services that might be using the filesystem.
>
> So I think I've found a solution for systemd to handle "bg" nfs mounts
> correctly. I'll submit some pull requests for consideration.
Neil, your an excellent engineer so I'm sure you will craft
something very cool... but I truly feel you are breaking one
of the NFS communities long standing golden rule... We never add
fix to client that fixes a bug in the server... Fix the server...
That is the case here, IMHO... There is a bug in systemd that is not
letting a mount command succeed... It's clearly their bug,
let them fix it... Again, why systemd looking at mount options
is just mind boggling...
steved.
On 06/07/2017 12:08 PM, J. Bruce Fields wrote:
> On Wed, Jun 07, 2017 at 06:04:12AM -0400, Steve Dickson wrote:
>> # ps ax | grep mount
>> 980 ? Ss 0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
>
> Right, but I think we also need to see a "systemctl status
> remote-fs.target", or something, to verify whether that's the forked
> background process or just the foreground process that's still hanging
> up some part of the boot process (even though it's gotten far enough
> along that you can log in--unless logins aren't permitted till remote
> fs's are mounted, I don't know.)
It succeeds...
# systemctl status remote-fs.target
* remote-fs.target - Remote File Systems
Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
Active: active since Tue 2017-06-06 12:36:51 EDT; 12min ago
Docs: man:systemd.special(7)
Jun 06 12:36:51 f26 systemd[1]: Reached target Remote File Systems.
The reason being, as Neil pointed out, the mount.nfs gets the
ECONNREFUSED right away because the server is down. So a
child is quickly forked that continues to try the mount...
Basically sneaking around systemd back... Which is hard
to do... these day 8-)
steved.
On Thu, Jun 08 2017, J. Bruce Fields wrote:
> On Thu, Jun 08, 2017 at 03:16:52PM +1000, NeilBrown wrote:
>> So I think I've found a solution for systemd to handle "bg" nfs mounts
>> correctly. I'll submit some pull requests for consideration.
>
> Out of curiosity, after that change is there still any reason you'd
> recommend any new user actually use "bg" (as opposed to an automount)?
Me? Recommend? Who would listen? Who would even hear?
For the last few years I've been recommending that automount should
be used for *all* NFS mounts at every opportunity. I think I've had two
opportunities.
But no, I would not recommend "bg". I would recommend automount and
then when they reported problems, I would help fix them.
I would be much happier recommending automount if it were easier.
Setting up /etc/auto.direct with automountd is fairly easy, but you need
to actually enable it but modifying auto.master or auto.master.d, which
is slightly annoying.
systemd does make it easier to do direct mounts, but it is ugly.
You need to include "comment=systemd.automount" or "x-systemd.automount"
in /etc/fstab instead of just "automount" or "ondemand". I understand
exactly why they did that and I cannot fault the logic. But it still
looks clumsy.
With systemd, you cannot divorce the timeout that an application has to
wait when accessing the mountpoint while the server is down, from the
timeout imposed on the mount program. i.e., mount cannot keep trying
in the background. - that could be useful if you want really-short
timeouts... at least it seems to me that they should be separate.
The timeout is configured differently if mounting from a device, or
mounting from anything else such as NFS. The first uses
x-systemd.device-timeout. The other needs x-systemd.mount-timeout.
But I'm ranting... I should probably shut up and send patches.
A generator for /etc/fstab.auto??
>
> I appreciate the effort to keep existing systems working, I'm just
> curious.
Compatibility with existing practice is certainly the main driver.
Thanks,
NeilBrown
Hello,
On 05/29/2017 06:19 PM, NeilBrown wrote:
>
> Systemd does not, and will not, support "bg" correctly.
> It has other, better, ways to handle "background" mounting.
The only problem with this is bg mounts still work at least
up to 4.11 kernel...
It appears there is a problem with a 4.12 kernel. The mount no
longer errors out with ECONNREFUSED it just hangs in the
kernel trying forever... It sounds like a bug to me but
maybe that change was intentional.. Anna?? Trond???
So I'm a bit hesitant to commit this since not accurate, yet.
Finally, the whole idea of systemd randomly/silently
strip off mount options is crazy... IMHO...
Just because a concept that has been around for years
does not fix well in the systemd world it gets
rip out??? IDK... but I think we can do better than that.
Note, the 'bg' is used by clients that do want their
booting to hang by servers that are down so if the
option is rip out, boots will start hang. This
will make it very difficult to debug since the bg
will still exist in fstab.
Again, the whole concept of systemd messing with mounts options
is just not a good one... IMHO..
steved.
>
> Explain this.
>
> See also https://github.com/systemd/systemd/issues/6046
>
> Signed-off-by: NeilBrown <[email protected]>
> ---
> utils/mount/nfs.man | 18 +++++++++++++++++-
> 1 file changed, 17 insertions(+), 1 deletion(-)
>
> diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
> index cc6e992ed807..7e76492d454f 100644
> --- a/utils/mount/nfs.man
> +++ b/utils/mount/nfs.man
> @@ -372,6 +372,21 @@ Alternatively these issues can be addressed
> using an automounter (refer to
> .BR automount (8)
> for details).
> +.IP
> +When
> +.B systemd
> +is used to mount the filesystems listed in
> +.IR /etc/fstab ,
> +the
> +.B bg
> +option is not supported, and may be stripped from the option list.
> +Similar functionality can be achieved by providing the
> +.B x-system.automount
> +option. This will cause
> +.B systemd
> +to attempt to mount the filesystem when the mountpoint is first
> +accessed, rather than during system boot. The mount still happens in
> +the "background", though in a different way.
> .TP 1.5i
> .BR rdirplus " / " nordirplus
> Selects whether to use NFS v3 or v4 READDIRPLUS requests.
> @@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
> .BR rpc.idmapd (8),
> .BR rpc.gssd (8),
> .BR rpc.svcgssd (8),
> -.BR kerberos (1)
> +.BR kerberos (1),
> +.BR systemd.mount (5) .
> .sp
> RFC 768 for the UDP specification.
> .br
>
2017-06-06 20:07 GMT+02:00 Steve Dickson <[email protected]>:
> Finally, the whole idea of systemd randomly/silently
> strip off mount options is crazy... IMHO...
Personally, I would prefer if systemd simply logged a warning/error
message but would *not* strip the bg option.
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
On Tue, Jun 06 2017, Steve Dickson wrote:
> Hello,
>
> On 05/29/2017 06:19 PM, NeilBrown wrote:
>>
>> Systemd does not, and will not, support "bg" correctly.
>> It has other, better, ways to handle "background" mounting.
> The only problem with this is bg mounts still work at least
> up to 4.11 kernel...
I don't think this is correct.
In the default configuration, "mount -t nfs -o bg ...."
runs for longer than 90 seconds, so systemd kill it.
A working "bg" mount would successfully mount the filesystem is the
server came back after 5 minutes. If you use current systemd in the
default configuration, it won't.
bg mounts do work sometimes, but I don't think they are reliable, and
there seems to be no interest in changing this.
Maybe the text should say "Systemd does not, and will not, reliably
support "bg" mounts...".
>
> It appears there is a problem with a 4.12 kernel. The mount no
> longer errors out with ECONNREFUSED it just hangs in the
> kernel trying forever... It sounds like a bug to me but
> maybe that change was intentional.. Anna?? Trond???
Might be related to my patch
[PATCH] SUNRPC: ensure correct error is reported by xs_tcp_setup_socket()
sent 25th May.
>
> So I'm a bit hesitant to commit this since not accurate, yet.
>
> Finally, the whole idea of systemd randomly/silently
> strip off mount options is crazy... IMHO...
It isn't exactly systemd, it is systemd-fstab-generator.
The options lists in /etc/fstab are not all equal. Some
are relevant to /bin/mount, some to mount.nfs, some to the kernel.
I think /bin/mount processes the option lists before passing it
to mount.nfs. There is no intrinsic reason that systemd-fstab-generator
shouldn't do the same thing.
>
> Just because a concept that has been around for years
> does not fix well in the systemd world it gets
> rip out??? IDK... but I think we can do better than that.
I could suggest that automount is uniformly better than bg. Give how
long automount has been around, and how easy it is to use with systemd,
it might be time to start encouraging people to stop using the inferior
technology.
I could say that, but I'm not 100% sure. The difference between
automount and bg is that with bg it is easy to see if the mount has
succeeded yet - just look for an empty directory. With automount,
you'll typically get a delay at that point. We could possibly wind down
that delay...
>
> Note, the 'bg' is used by clients that do want their
> booting to hang by servers that are down so if the
> option is rip out, boots will start hang. This
> will make it very difficult to debug since the bg
> will still exist in fstab.
Not exactly.
Current default behaviour is that systemd will wait 90 seconds, then
kill the mount program and fail the boot. If we strip out "bg", the
same thing will happen.
I'm OK with the patch not being applied just yet. I think we need to
resolve this issue, but it isn't 100% clear to me what the best
resolution would be. So I'm happy to see a conversation happening and
opinions being shared.
I'd be particularly pleased if you could double check how "bg" is
currently handled on some systemd-enabled system that you use.
Does the mount program get killed like I see? Does boot succeed if
there is a bg mount from an unresponsive server?
Thanks,
NeilBrown
>
> Again, the whole concept of systemd messing with mounts options
> is just not a good one... IMHO..
>
> steved.
>
>>
>> Explain this.
>>
>> See also https://github.com/systemd/systemd/issues/6046
>>
>> Signed-off-by: NeilBrown <[email protected]>
>> ---
>> utils/mount/nfs.man | 18 +++++++++++++++++-
>> 1 file changed, 17 insertions(+), 1 deletion(-)
>>
>> diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
>> index cc6e992ed807..7e76492d454f 100644
>> --- a/utils/mount/nfs.man
>> +++ b/utils/mount/nfs.man
>> @@ -372,6 +372,21 @@ Alternatively these issues can be addressed
>> using an automounter (refer to
>> .BR automount (8)
>> for details).
>> +.IP
>> +When
>> +.B systemd
>> +is used to mount the filesystems listed in
>> +.IR /etc/fstab ,
>> +the
>> +.B bg
>> +option is not supported, and may be stripped from the option list.
>> +Similar functionality can be achieved by providing the
>> +.B x-system.automount
>> +option. This will cause
>> +.B systemd
>> +to attempt to mount the filesystem when the mountpoint is first
>> +accessed, rather than during system boot. The mount still happens in
>> +the "background", though in a different way.
>> .TP 1.5i
>> .BR rdirplus " / " nordirplus
>> Selects whether to use NFS v3 or v4 READDIRPLUS requests.
>> @@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
>> .BR rpc.idmapd (8),
>> .BR rpc.gssd (8),
>> .BR rpc.svcgssd (8),
>> -.BR kerberos (1)
>> +.BR kerberos (1),
>> +.BR systemd.mount (5) .
>> .sp
>> RFC 768 for the UDP specification.
>> .br
>>
On Tue, 06.06.17 14:07, Steve Dickson ([email protected]) wrote:
> Hello,
>
> On 05/29/2017 06:19 PM, NeilBrown wrote:
> >
> > Systemd does not, and will not, support "bg" correctly.
> > It has other, better, ways to handle "background" mounting.
> The only problem with this is bg mounts still work at least
> up to 4.11 kernel...
>
> It appears there is a problem with a 4.12 kernel. The mount no
> longer errors out with ECONNREFUSED it just hangs in the
> kernel trying forever... It sounds like a bug to me but
> maybe that change was intentional.. Anna?? Trond???
>
> So I'm a bit hesitant to commit this since not accurate, yet.
>
> Finally, the whole idea of systemd randomly/silently
> strip off mount options is crazy... IMHO...
>
> Just because a concept that has been around for years
> does not fix well in the systemd world it gets
> rip out??? IDK... but I think we can do better than that.
Well "bg" doesn't really work on systemd systems, and this was never
different, hence I think it's only fair to document that it is
incompatible with systemd. In addition, I have the suspicion it is not
used very widely, since I never actually got complaints about it.
Lennart
--
Lennart Poettering, Red Hat
On Tue, 06.06.17 21:57, Michael Biebl ([email protected]) wrote:
> 2017-06-06 20:07 GMT+02:00 Steve Dickson <[email protected]>:
> > Finally, the whole idea of systemd randomly/silently
> > strip off mount options is crazy... IMHO...
>
> Personally, I would prefer if systemd simply logged a warning/error
> message but would *not* strip the bg option.
What good does that do if "bg" doesn't actually work properly?
Lennart
--
Lennart Poettering, Red Hat
On 06/07/2017 04:13 AM, Lennart Poettering wrote:
> On Tue, 06.06.17 21:57, Michael Biebl ([email protected]) wrote:
>
>> 2017-06-06 20:07 GMT+02:00 Steve Dickson <[email protected]>:
>>> Finally, the whole idea of systemd randomly/silently
>>> strip off mount options is crazy... IMHO...
>>
>> Personally, I would prefer if systemd simply logged a warning/error
>> message but would *not* strip the bg option.
>
> What good does that do if "bg" doesn't actually work properly?
At the moment it works just fine...
steved.
>
> Lennart
>
On 06/07/2017 04:12 AM, Lennart Poettering wrote:
> On Tue, 06.06.17 14:07, Steve Dickson ([email protected]) wrote:
>
>> Hello,
>>
>> On 05/29/2017 06:19 PM, NeilBrown wrote:
>>>
>>> Systemd does not, and will not, support "bg" correctly.
>>> It has other, better, ways to handle "background" mounting.
>> The only problem with this is bg mounts still work at least
>> up to 4.11 kernel...
>>
>> It appears there is a problem with a 4.12 kernel. The mount no
>> longer errors out with ECONNREFUSED it just hangs in the
>> kernel trying forever... It sounds like a bug to me but
>> maybe that change was intentional.. Anna?? Trond???
>>
>> So I'm a bit hesitant to commit this since not accurate, yet.
>>
>> Finally, the whole idea of systemd randomly/silently
>> strip off mount options is crazy... IMHO...
>>
>> Just because a concept that has been around for years
>> does not fix well in the systemd world it gets
>> rip out??? IDK... but I think we can do better than that.
>
> Well "bg" doesn't really work on systemd systems, and this was never
> different, hence I think it's only fair to document that it is
> incompatible with systemd.
I'm seeing it work just fine...
/etc/fstab: nfssrv:/home/tmp /mnt/tmp nfs bg 0 0
nfssrv is down
reboot client
login to client
# ps ax | grep mount
980 ? Ss 0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
> In addition, I have the suspicion it is not
> used very widely, since I never actually got complaints about it.
Since it seems to be still working we probably would not hear
any complaints about it...
steved.
On 06/06/2017 05:49 PM, NeilBrown wrote:
> On Tue, Jun 06 2017, Steve Dickson wrote:
>
>> Hello,
>>
>> On 05/29/2017 06:19 PM, NeilBrown wrote:
>>>
>>> Systemd does not, and will not, support "bg" correctly.
>>> It has other, better, ways to handle "background" mounting.
>> The only problem with this is bg mounts still work at least
>> up to 4.11 kernel...
>
> I don't think this is correct.
> In the default configuration, "mount -t nfs -o bg ...."
> runs for longer than 90 seconds, so systemd kill it.
I must be missing something... it seems to be working for me
# mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
mount.nfs: mount(2): Connection refused
mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
mount.nfs: backgrounding "rhel7srv:/home/tmp"
mount.nfs: mount options: "rw,bg"
# ps ax | grep mount.nfs
2270 ? Ss 0:00 /sbin/mount.nfs rhel7srv:/home/tmp /mnt/tmp -v -o rw,bg
>
> A working "bg" mount would successfully mount the filesystem is the
> server came back after 5 minutes. If you use current systemd in the
> default configuration, it won't.
When I add a bg mount to fstab again... it works just fine. This
is with the latest upstream nfs-utils.
>
> bg mounts do work sometimes, but I don't think they are reliable, and
> there seems to be no interest in changing this.
> Maybe the text should say "Systemd does not, and will not, reliably
> support "bg" mounts...".
not reliable maybe... I'm just doing very simple testing...
>
>
>>
>> It appears there is a problem with a 4.12 kernel. The mount no
>> longer errors out with ECONNREFUSED it just hangs in the
>> kernel trying forever... It sounds like a bug to me but
>> maybe that change was intentional.. Anna?? Trond???
>
> Might be related to my patch
> [PATCH] SUNRPC: ensure correct error is reported by xs_tcp_setup_socket()
>
> sent 25th May.
I'll take a look.. thanks!
>
>>
>> So I'm a bit hesitant to commit this since not accurate, yet.
>>
>> Finally, the whole idea of systemd randomly/silently
>> strip off mount options is crazy... IMHO...
>
> It isn't exactly systemd, it is systemd-fstab-generator.
> The options lists in /etc/fstab are not all equal. Some
> are relevant to /bin/mount, some to mount.nfs, some to the kernel.
> I think /bin/mount processes the option lists before passing it
> to mount.nfs. There is no intrinsic reason that systemd-fstab-generator
> shouldn't do the same thing.
OK.
>
>>
>> Just because a concept that has been around for years
>> does not fix well in the systemd world it gets
>> rip out??? IDK... but I think we can do better than that.
>
> I could suggest that automount is uniformly better than bg. Give how
> long automount has been around, and how easy it is to use with systemd,
> it might be time to start encouraging people to stop using the inferior
> technology.
Yes... bg mounts are a poor man's automount...
>
> I could say that, but I'm not 100% sure. The difference between
> automount and bg is that with bg it is easy to see if the mount has
> succeeded yet - just look for an empty directory. With automount,
> you'll typically get a delay at that point. We could possibly wind down
> that delay...
>
>>
>> Note, the 'bg' is used by clients that do want their
>> booting to hang by servers that are down so if the
>> option is rip out, boots will start hang. This
>> will make it very difficult to debug since the bg
>> will still exist in fstab.
>
> Not exactly.
> Current default behaviour is that systemd will wait 90 seconds, then
> kill the mount program and fail the boot. If we strip out "bg", the
> same thing will happen.
Again.. I'm not seeing this 90 sec delay when I add a bg mount
to /etc/fstab.
>
> I'm OK with the patch not being applied just yet. I think we need to
> resolve this issue, but it isn't 100% clear to me what the best
> resolution would be. So I'm happy to see a conversation happening and
> opinions being shared.
> I'd be particularly pleased if you could double check how "bg" is
> currently handled on some systemd-enabled system that you use.
> Does the mount program get killed like I see?
No. after adding a bg mount to fstab and rebooting (with the server
down) I see the following mount in backgroup
1104 ? Ss 0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
> Does boot succeed if there is a bg mount from an unresponsive server?
Yes. Then when I bring up the server the mount succeeds
steved.
P.S. I'm taking some PTO today so I will not be back in the office
until later today or tomorrow...
steved.
>
> Thanks,
> NeilBrown
>
>
>>
>> Again, the whole concept of systemd messing with mounts options
>> is just not a good one... IMHO..
>>
>> steved.
>>
>>>
>>> Explain this.
>>>
>>> See also https://github.com/systemd/systemd/issues/6046
>>>
>>> Signed-off-by: NeilBrown <[email protected]>
>>> ---
>>> utils/mount/nfs.man | 18 +++++++++++++++++-
>>> 1 file changed, 17 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
>>> index cc6e992ed807..7e76492d454f 100644
>>> --- a/utils/mount/nfs.man
>>> +++ b/utils/mount/nfs.man
>>> @@ -372,6 +372,21 @@ Alternatively these issues can be addressed
>>> using an automounter (refer to
>>> .BR automount (8)
>>> for details).
>>> +.IP
>>> +When
>>> +.B systemd
>>> +is used to mount the filesystems listed in
>>> +.IR /etc/fstab ,
>>> +the
>>> +.B bg
>>> +option is not supported, and may be stripped from the option list.
>>> +Similar functionality can be achieved by providing the
>>> +.B x-system.automount
>>> +option. This will cause
>>> +.B systemd
>>> +to attempt to mount the filesystem when the mountpoint is first
>>> +accessed, rather than during system boot. The mount still happens in
>>> +the "background", though in a different way.
>>> .TP 1.5i
>>> .BR rdirplus " / " nordirplus
>>> Selects whether to use NFS v3 or v4 READDIRPLUS requests.
>>> @@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
>>> .BR rpc.idmapd (8),
>>> .BR rpc.gssd (8),
>>> .BR rpc.svcgssd (8),
>>> -.BR kerberos (1)
>>> +.BR kerberos (1),
>>> +.BR systemd.mount (5) .
>>> .sp
>>> RFC 768 for the UDP specification.
>>> .br
>>>
On Wed, 07.06.17 06:08, Steve Dickson ([email protected]) wrote:
>
>
> On 06/06/2017 05:49 PM, NeilBrown wrote:
> > On Tue, Jun 06 2017, Steve Dickson wrote:
> >
> >> Hello,
> >>
> >> On 05/29/2017 06:19 PM, NeilBrown wrote:
> >>>
> >>> Systemd does not, and will not, support "bg" correctly.
> >>> It has other, better, ways to handle "background" mounting.
> >> The only problem with this is bg mounts still work at least
> >> up to 4.11 kernel...
> >
> > I don't think this is correct.
> > In the default configuration, "mount -t nfs -o bg ...."
> > runs for longer than 90 seconds, so systemd kill it.
> I must be missing something... it seems to be working for me
>
> # mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
> mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
> mount.nfs: mount(2): Connection refused
> mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
> mount.nfs: prog 100003, trying vers=3, prot=6
> mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
> mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
> mount.nfs: backgrounding "rhel7srv:/home/tmp"
> mount.nfs: mount options: "rw,bg"
We are talking about mounts done through /etc/fstab, i.e. the ones
systemd actually manages.
Lennart
--
Lennart Poettering, Red Hat
On Wed, Jun 07, 2017 at 06:04:12AM -0400, Steve Dickson wrote:
>
>
> On 06/07/2017 04:12 AM, Lennart Poettering wrote:
> > On Tue, 06.06.17 14:07, Steve Dickson ([email protected]) wrote:
> >
> >> Hello,
> >>
> >> On 05/29/2017 06:19 PM, NeilBrown wrote:
> >>>
> >>> Systemd does not, and will not, support "bg" correctly.
> >>> It has other, better, ways to handle "background" mounting.
> >> The only problem with this is bg mounts still work at least
> >> up to 4.11 kernel...
> >>
> >> It appears there is a problem with a 4.12 kernel. The mount no
> >> longer errors out with ECONNREFUSED it just hangs in the
> >> kernel trying forever... It sounds like a bug to me but
> >> maybe that change was intentional.. Anna?? Trond???
> >>
> >> So I'm a bit hesitant to commit this since not accurate, yet.
> >>
> >> Finally, the whole idea of systemd randomly/silently
> >> strip off mount options is crazy... IMHO...
> >>
> >> Just because a concept that has been around for years
> >> does not fix well in the systemd world it gets
> >> rip out??? IDK... but I think we can do better than that.
> >
> > Well "bg" doesn't really work on systemd systems, and this was never
> > different, hence I think it's only fair to document that it is
> > incompatible with systemd.
> I'm seeing it work just fine...
>
> /etc/fstab: nfssrv:/home/tmp /mnt/tmp nfs bg 0 0
> nfssrv is down
> reboot client
> login to client
> # ps ax | grep mount
> 980 ? Ss 0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
Right, but I think we also need to see a "systemctl status
remote-fs.target", or something, to verify whether that's the forked
background process or just the foreground process that's still hanging
up some part of the boot process (even though it's gotten far enough
along that you can log in--unless logins aren't permitted till remote
fs's are mounted, I don't know.)
--b.
On 06/07/2017 08:02 AM, Lennart Poettering wrote:
> On Wed, 07.06.17 06:08, Steve Dickson ([email protected]) wrote:
>
>>
>>
>> On 06/06/2017 05:49 PM, NeilBrown wrote:
>>> On Tue, Jun 06 2017, Steve Dickson wrote:
>>>
>>>> Hello,
>>>>
>>>> On 05/29/2017 06:19 PM, NeilBrown wrote:
>>>>>
>>>>> Systemd does not, and will not, support "bg" correctly.
>>>>> It has other, better, ways to handle "background" mounting.
>>>> The only problem with this is bg mounts still work at least
>>>> up to 4.11 kernel...
>>>
>>> I don't think this is correct.
>>> In the default configuration, "mount -t nfs -o bg ...."
>>> runs for longer than 90 seconds, so systemd kill it.
>> I must be missing something... it seems to be working for me
>>
>> # mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
>> mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
>> mount.nfs: mount(2): Connection refused
>> mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
>> mount.nfs: prog 100003, trying vers=3, prot=6
>> mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
>> mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
>> mount.nfs: backgrounding "rhel7srv:/home/tmp"
>> mount.nfs: mount options: "rw,bg"
>
> We are talking about mounts done through /etc/fstab, i.e. the ones
> systemd actually manages.
I guess I was not clear... After adding a bg mount to fstab and
reboot, mounting a server that is not up, there is a mount in
background that looks like
# ps ax | grep mount
1104 ? Ss 0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
Looking at the remote-fs.target status:
# systemctl status remote-fs.target
* remote-fs.target - Remote File Systems
Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
Active: active since Tue 2017-06-06 12:36:51 EDT; 12min ago
Docs: man:systemd.special(7)
Jun 06 12:36:51 f26.boston.devel.redhat.com systemd[1]: Reached target Remote File Systems.
It appears to be successful
steved.
On Wed, Jun 07 2017, Steve Dickson wrote:
> On 06/07/2017 08:02 AM, Lennart Poettering wrote:
>> On Wed, 07.06.17 06:08, Steve Dickson ([email protected]) wrote:
>>
>>>
>>>
>>> On 06/06/2017 05:49 PM, NeilBrown wrote:
>>>> On Tue, Jun 06 2017, Steve Dickson wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> On 05/29/2017 06:19 PM, NeilBrown wrote:
>>>>>>
>>>>>> Systemd does not, and will not, support "bg" correctly.
>>>>>> It has other, better, ways to handle "background" mounting.
>>>>> The only problem with this is bg mounts still work at least
>>>>> up to 4.11 kernel...
>>>>
>>>> I don't think this is correct.
>>>> In the default configuration, "mount -t nfs -o bg ...."
>>>> runs for longer than 90 seconds, so systemd kill it.
>>> I must be missing something... it seems to be working for me
>>>
>>> # mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
>>> mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
>>> mount.nfs: mount(2): Connection refused
>>> mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
>>> mount.nfs: prog 100003, trying vers=3, prot=6
>>> mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
>>> mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
>>> mount.nfs: backgrounding "rhel7srv:/home/tmp"
>>> mount.nfs: mount options: "rw,bg"
>>
>> We are talking about mounts done through /etc/fstab, i.e. the ones
>> systemd actually manages.
> I guess I was not clear... After adding a bg mount to fstab and
> reboot, mounting a server that is not up, there is a mount in
> background that looks like
>
> # ps ax | grep mount
> 1104 ? Ss 0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
>
> Looking at the remote-fs.target status:
>
> # systemctl status remote-fs.target
> * remote-fs.target - Remote File Systems
> Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
> Active: active since Tue 2017-06-06 12:36:51 EDT; 12min ago
> Docs: man:systemd.special(7)
>
> Jun 06 12:36:51 f26.boston.devel.redhat.com systemd[1]: Reached target Remote File Systems.
>
> It appears to be successful
Strange ... not for me.
I have a recent compiled-from-source upstream systemd and a very recent
upstream kernel.
I add a line to /etc/fstab
192.168.1.111:/nowhere /mnt nfs bg 0 0
and reboot (192.168.1.111 is on a different subnet to the VM I am
testing in, but no host responds to the address).
There is a 1m30s wait while trying to mount /mnt, then boot completes
(I was wrong when I said that it didn't).
● mnt.mount - /mnt
Loaded: loaded (/etc/fstab; generated; vendor preset: enabled)
Active: failed (Result: timeout) since Thu 2017-06-08 11:13:52 AEST; 1min 24s ago
Where: /mnt
What: 192.168.1.111:/nowhere
Docs: man:fstab(5)
man:systemd-fstab-generator(8)
Process: 333 ExecMount=/bin/mount 192.168.1.111:/nowhere /mnt -t nfs -o bg (code=killed, signal=TERM)
Jun 08 11:12:22 debian systemd[1]: Mounting /mnt...
Jun 08 11:13:52 debian systemd[1]: mnt.mount: Mounting timed out. Stopping.
Jun 08 11:13:52 debian systemd[1]: mnt.mount: Mount process exited, code=killed status=15
Jun 08 11:13:52 debian systemd[1]: Failed to mount /mnt.
Jun 08 11:13:52 debian systemd[1]: mnt.mount: Unit entered failed state.
The /bin/mount process has been killed (SIGTERM) after the 90 second
timeout
● remote-fs.target - Remote File Systems
Loaded: loaded (/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
Drop-In: /run/systemd/generator/remote-fs.target.d
└─50-insserv.conf.conf
Active: inactive (dead)
Docs: man:systemd.special(7)
Jun 08 11:13:52 debian systemd[1]: Dependency failed for Remote File Systems.
Jun 08 11:13:52 debian systemd[1]: remote-fs.target: Job remote-fs.target/start failed with result 'dependency'.
remote-fs.target has not been reached.
Because remote-fs.target is WantedBy multi-user.target, but need
RequiredBy it, boot completes.
But if anything did Require remote-fs.target, it would fail if "bg"
mounts were not mounted within 90 seconds.
Looking back over your log messages:
>>> mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
>>> mount.nfs: backgrounding "rhel7srv:/home/tmp"
it appears that the fg mount attempt failed quickly (ECONNREFUSED), so it
background the process immediately. Systemd sees that as success
(despite the fact that the filesystem isn't actually mounted) and
doesn't clean up the background processes (which is arguably a bug).
If you try to mount from a server which doesn't even return ECONNREFUSED or
EHOSTUNREACH, such that the first mount attempt takes longer than 90
seconds, then you should get the same results as me.
Your results go some way to explaining why Lennart hasn't received many
complaints, but I'm convinced that the current situation is not ideal
(because SUSE has received some complaints).
I've been pondering the possibility of making "bg" work properly with
systemd and I think I've found a promising approach. It involves having
systemd take responsibility for the "run in the background" part.
If we get systemd-fstab-generator to translate "bg" to "retry=10000",
then "mount.nfs" will behave like the background version of
"mount.nfs -o bg". i.e. it will retry for one month (nearly). If there is
already a 'retry=' option, we just strip the "bg".
For this to work, we would need to add
TimeoutSec=infinity
to the .mount unit file so that systemd doesn't kill the mount.
We would also need to add (the effect of) "nofail", so that systemd
doesn't wait for the mount to complete...
Except that the current implementation of "nofail" is faulty.
It removes the default "Before=remote-fs.mount", which has the unwanted
consequence of unmounting the filesystem too early at shutdown.
I have a solution for that too (which I'll submit a push request for
shortly).
My solution to "nofail" is to treat it much like "automount", but
instead of using an automount unit to trigger the mount, we use
a timer unit (with OnActiveSec=0).
By triggering the mount unit with a timer instead of Wanting it
directly, it gets run in a separate transaction. This means that the
"Before=remote-fs.target" doesn't have the effect of delaying
remote-fs.target. Before/After only order jobs within the one
transaction.
When it comes time to shutdown, remote-fs.target and the foo.mount will
be in the same transaction, so the Before= will ensure foo.mount
isn't unmounted until after remote-fs.target has been allows to finish,
which is after any services that might be using the filesystem.
So I think I've found a solution for systemd to handle "bg" nfs mounts
correctly. I'll submit some pull requests for consideration.
Thanks,
NeilBrown
On Tue, May 30 2017, NeilBrown wrote:
> Systemd does not, and will not, support "bg" correctly.
> It has other, better, ways to handle "background" mounting.
For those who aren't closely watching systemd development, a
patch was recently accepted which causes systemd to work correctly with
NFS bg mounts. So the above "and will not" was, happily, not correct.
So this patch to the nfs documentation is no longer relevant.
Thanks everyone,
NeilBrown
Hey Neil,
On 07/04/2017 06:20 PM, NeilBrown wrote:
> On Tue, May 30 2017, NeilBrown wrote:
>
>> Systemd does not, and will not, support "bg" correctly.
>> It has other, better, ways to handle "background" mounting.
>
> For those who aren't closely watching systemd development, a
> patch was recently accepted which causes systemd to work correctly with
> NFS bg mounts. So the above "and will not" was, happily, not correct.
Could you please post a pointer to the thread?
tia,
steved.
On Mon, Jul 10 2017, Steve Dickson wrote:
> Hey Neil,
>
> On 07/04/2017 06:20 PM, NeilBrown wrote:
>> On Tue, May 30 2017, NeilBrown wrote:
>>
>>> Systemd does not, and will not, support "bg" correctly.
>>> It has other, better, ways to handle "background" mounting.
>>
>> For those who aren't closely watching systemd development, a
>> patch was recently accepted which causes systemd to work correctly with
>> NFS bg mounts. So the above "and will not" was, happily, not correct.
> Could you please post a pointer to the thread?
The main commmit is
https://github.com/systemd/systemd/commit/65e1dee7dcf1668c25c32f0238c935708dbffbcf
The link in the title leads to
https://github.com/systemd/systemd/pull/6103
which is the discussion of the pull request.
The "Fixes" link at the bottom leads to
https://github.com/systemd/systemd/issues/6046
which is an 'issue' that started out as "we should warn if 'bg' is used"
but became "here is a fix so bg works".
I've been wondering if we should add text to nfs.5 suggesting that
automount is often a good match for NFS, and can particularly be useful
where "bg" is currently used. Not sure what to do about timeouts
though.
systemd currently waits for the mount command to fail, or kills it after
90 seconds. It doesn't return an error to accesses of the mount point
(typically ENODEV) until mount exits. I think I would prefer a shorter
timeout before the error, but a longer timeout before killing mount.
But I'm not really sure. Does anyone have opinions?
Thanks,
NeilBrown