2006-11-20 19:47:00

by Open Source

[permalink] [raw]
Subject: Re: [NFS] "mount: File exists" when trying to mount a second krb5 volume!

Hi Trond,

I can confirm that I also see a crash with your patch file in place!
I'm not sure if that's because it didn't apply cleanly to begin with
and my massaging might have caused a problem. I don't think
so though.

Basically if you mount a couple NFSv3+krb5 partitions and
then unmount one of them, the system locks up after about 2
seconds. I didn't see this before since I didn't use the
automounter utility.

Thanks,
Paarvai


----- Original Message ----
From: Kostas Georgiou <[email protected]>
To: Open Source <[email protected]>
Cc: Trond Myklebust <[email protected]>; [email protected]; [email protected]
Sent: Saturday, November 18, 2006 7:15:04 AM
Subject: Re: [NFS] "mount: File exists" when trying to mount a second krb5 volume!

On Fri, Nov 17, 2006 at 11:59:25AM -0800, Open Source wrote:

> Thanks Trond. Saving the patch did the trick. I had
> to do some manual hacking to get the patch to
> apply because you must have done your diff
> against a different kernel tree revision. I'm using
> 2.6.18 from FC5 updates.
>
> Once I massaged things, the patch compiled and it
> works. I am able to mount more than one partition
> with no problems now. However, I cannot speak
> to the finer points (like memory leaks, etc.) since
> I don't know the code so well. I trust someone else
> will independently verify it (i.e., code review) before
> it gets into the kernel tree.

I did the same with the FC6 2.6.18 kernel and while it does allow
me to have more than one mount from the server autofs decides that
my home is not in use any more and it unmounts it, somehow this
causes the system to freeze as well.

Nov 17 16:28:14 thufir automount[2483]: mount still busy /home
Nov 17 16:29:29 thufir automount[2483]: expiring path /home/georgiou
Nov 17 16:29:29 thufir automount[2483]: unmounting dir = /home/georgiou
Nov 17 16:29:32 thufir automount[2483]: expired /home/georgiou

Here is the patch with my changes to apply in FC6 which most likely is
bad since I really don't know the code at all :(

Kostas Georgiou

diff -u a/include/linux/sunrpc/clnt.h b/include/linux/sunrpc/clnt.h
--- a/include/linux/sunrpc/clnt.h
+++ b/include/linux/sunrpc/clnt.h
@@ -64,6 +64,7 @@
struct dentry * cl_dentry; /* inode */
struct rpc_clnt * cl_parent; /* Points to parent of clones */
struct rpc_rtt cl_rtt_default;
+ struct rpc_program * cl_program;
struct rpc_portmap cl_pmap_default;
char cl_inline_name[32];
};
diff -u a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -154,6 +154,7 @@
clnt->cl_prot = xprt->prot;
clnt->cl_stats = program->stats;
clnt->cl_metrics = rpc_alloc_iostats(clnt);
+ clnt->cl_program = program;
rpc_init_wait_queue(&clnt->cl_pmap_default.pm_bindwait, "bindwait");

if (!clnt->cl_port)
@@ -235,6 +236,7 @@
rpc_clone_client(struct rpc_clnt *clnt)
{
struct rpc_clnt *new;
+ int err;

new = kmalloc(sizeof(*new), GFP_KERNEL);
if (!new)
@@ -242,6 +244,11 @@
memcpy(new, clnt, sizeof(*new));
atomic_set(&new->cl_count, 1);
atomic_set(&new->cl_users, 0);
+ err = rpc_setup_pipedir(new, clnt->cl_program->pipe_dir_name);
+ if (err != 0) {
+ kfree(new);
+ return ERR_PTR(err);
+ }
new->cl_parent = clnt;
atomic_inc(&clnt->cl_count);
/* Duplicate portmapper */
@@ -250,8 +257,6 @@
new->cl_autobind = 0;
new->cl_oneshot = 0;
new->cl_dead = 0;
- if (!IS_ERR(new->cl_dentry))
- dget(new->cl_dentry);
rpc_init_rtt(&new->cl_rtt_default, clnt->cl_xprt->timeout.to_initval);
if (new->cl_auth)
atomic_inc(&new->cl_auth->au_count);
@@ -313,12 +318,6 @@
rpcauth_destroy(clnt->cl_auth);
clnt->cl_auth = NULL;
}
- if (clnt->cl_parent != clnt) {
- if (!IS_ERR(clnt->cl_dentry))
- dput(clnt->cl_dentry);
- rpc_destroy_client(clnt->cl_parent);
- goto out_free;
- }
if (!IS_ERR(clnt->cl_dentry)) {
rpc_rmdir(clnt->cl_dentry);
rpc_put_mount();
@@ -327,6 +326,10 @@
xprt_destroy(clnt->cl_xprt);
clnt->cl_xprt = NULL;
}
+ if (clnt->cl_parent != clnt) {
+ rpc_destroy_client(clnt->cl_parent);
+ goto out_free;
+ }
if (clnt->cl_server != clnt->cl_inline_name)
kfree(clnt->cl_server);
out_free:


2006-11-27 11:58:14

by Ian Kent

[permalink] [raw]
Subject: Re: "mount: File exists" when trying to mount a second krb5 volume!

On Wed, 2006-11-22 at 14:43 -0800, Paarvai Naai wrote:
> Hi Trond,
>
> I'm certain the problem is causing a crash on my machine.
> When I am on the console and just do an umount the machine is okay
> for about 2 seconds and then it crashes. If I instead just try to do
> /sbin/reboot then I see a big OOPS when the initscripts end up
> unmounting the NFS filesystems, although it's hard to make sense
> of it or write any info down at that point.

Doesn't it get logged.
syslog should shutdown after autofs and NFS.

Ian



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2006-11-29 03:50:05

by Paarvai Naai

[permalink] [raw]
Subject: Re: [NFS] "mount: File exists" when trying to mount a second krb5 volume!

Hi,

I bit the bullet and compiled 2.6.19-rc6 (which was a rather
frustrating experience for a number of unrelated reasons). In
any case, I don't see the crash there. Hence, I'm fairly confident
that the crash had to do with an incompatibility between the
patch and 2.6.18.

Now that the patch has been verified can we do something to
get it into 2.6.19 before the final release. This is a very serious
bug that prevents people from using krb5 on a server with two
such exports. It needs to be fixed!

Thanks again to all the help on this list. Hopefully we can get this
thing into the upstream kernel ASAP.

Thoughts?

Paarvai


On 11/27/06, Ian Kent <[email protected]> wrote:
>
> On Wed, 2006-11-22 at 14:43 -0800, Paarvai Naai wrote:
> > Hi Trond,
> >
> > I'm certain the problem is causing a crash on my machine.
> > When I am on the console and just do an umount the machine is okay
> > for about 2 seconds and then it crashes. If I instead just try to do
> > /sbin/reboot then I see a big OOPS when the initscripts end up
> > unmounting the NFS filesystems, although it's hard to make sense
> > of it or write any info down at that point.
>
> Doesn't it get logged.
> syslog should shutdown after autofs and NFS.
>
> Ian
>
>
>


Attachments:
(No filename) (1.21 kB)
(No filename) (1.61 kB)
(No filename) (138.00 B)
Download all attachments

2006-11-22 20:43:37

by Trond Myklebust

[permalink] [raw]
Subject: Re: "mount: File exists" when trying to mount a second krb5 volume!

On Mon, 2006-11-20 at 11:47 -0800, Open Source wrote:
> Hi Trond,
>
> I can confirm that I also see a crash with your patch file in place!
> I'm not sure if that's because it didn't apply cleanly to begin with
> and my massaging might have caused a problem. I don't think
> so though.
>
> Basically if you mount a couple NFSv3+krb5 partitions and
> then unmount one of them, the system locks up after about 2
> seconds. I didn't see this before since I didn't use the
> automounter utility.

I'm unable to reproduce whether using the automounter or not. Can you be
a bit more specific about what hangs? Are you seeing an Oops or anything
like that?

Trond


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2006-11-22 22:43:30

by Paarvai Naai

[permalink] [raw]
Subject: Re: [NFS] "mount: File exists" when trying to mount a second krb5 volume!

Hi Trond,

I'm certain the problem is causing a crash on my machine.
When I am on the console and just do an umount the machine is okay
for about 2 seconds and then it crashes. If I instead just try to do
/sbin/reboot then I see a big OOPS when the initscripts end up
unmounting the NFS filesystems, although it's hard to make sense
of it or write any info down at that point.

There is clearly something wrong but I'm starting to suspect it's
because the patch might not be appropriate for the kernels
that Kostas and I are using. What version of the kernel (and
CITI patch, if applicable) are you making the patch against?
I'm using:

http://mirrors.kernel.org/fedora/core/updates/5/SRPMS/kernel-2.6.18-1.2200.fc5.src.rpm

Regards,
Paarvai


On 11/22/06, Trond Myklebust <[email protected]> wrote:
>
> On Mon, 2006-11-20 at 11:47 -0800, Open Source wrote:
> > Hi Trond,
> >
> > I can confirm that I also see a crash with your patch file in place!
> > I'm not sure if that's because it didn't apply cleanly to begin with
> > and my massaging might have caused a problem. I don't think
> > so though.
> >
> > Basically if you mount a couple NFSv3+krb5 partitions and
> > then unmount one of them, the system locks up after about 2
> > seconds. I didn't see this before since I didn't use the
> > automounter utility.
>
> I'm unable to reproduce whether using the automounter or not. Can you be
> a bit more specific about what hangs? Are you seeing an Oops or anything
> like that?
>
> Trond
>
> _______________________________________________
> NFSv4 mailing list
> [email protected]
> http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
>


Attachments:
(No filename) (1.62 kB)
(No filename) (2.24 kB)
(No filename) (138.00 B)
Download all attachments

2006-12-01 18:34:31

by Paarvai Naai

[permalink] [raw]
Subject: Re: [NFS] "mount: File exists" when trying to mount a second krb5 volume!

Hi all,

I really do appreciate the high quality technical assistance
I have gotten to date from this list. But at the same time,
I am getting somewhat frustrated with the very spotty
attention I am getting for a *critical* bug that somehow
made it into the upstream kernel sources. This is directly
affecting the ability to use krb5 on either NFSv3 or NFSv4.

First I had to actually dig through the source and find the
bug. Then I was offered a patch but when it didn't work
there was very little response/assistance to help with my
investigation into which kernel that patch is compatible with.
Finally, I have been insisting that we get the fix back into the
upstream sources and it appears that no one appreciates
the importance.

Can someone please help me attend to this? I'm happy to
assist with open source projects, but given that I have other
industry responsibilities it's hard to stay totally on top of a
problem that never should have been mine to contend with.

Thanks for understanding.

Best regards,
Paarvai



On 11/28/06, Paarvai Naai <[email protected]> wrote:
>
> Hi,
>
> I bit the bullet and compiled 2.6.19-rc6 (which was a rather
> frustrating experience for a number of unrelated reasons). In
> any case, I don't see the crash there. Hence, I'm fairly confident
> that the crash had to do with an incompatibility between the
> patch and 2.6.18.
>
> Now that the patch has been verified can we do something to
> get it into 2.6.19 before the final release. This is a very serious
> bug that prevents people from using krb5 on a server with two
> such exports. It needs to be fixed!
>
> Thanks again to all the help on this list. Hopefully we can get this
> thing into the upstream kernel ASAP.
>
> Thoughts?
>
> Paarvai
>
>
> On 11/27/06, Ian Kent <[email protected]> wrote:
> >
> > On Wed, 2006-11-22 at 14:43 -0800, Paarvai Naai wrote:
> > > Hi Trond,
> > >
> > > I'm certain the problem is causing a crash on my machine.
> > > When I am on the console and just do an umount the machine is okay
> > > for about 2 seconds and then it crashes. If I instead just try to do
> > > /sbin/reboot then I see a big OOPS when the initscripts end up
> > > unmounting the NFS filesystems, although it's hard to make sense
> > > of it or write any info down at that point.
> >
> > Doesn't it get logged.
> > syslog should shutdown after autofs and NFS.
> >
> > Ian
> >
> >
> >
>


Attachments:
(No filename) (2.35 kB)
(No filename) (3.21 kB)
(No filename) (138.00 B)
Download all attachments

2006-12-01 18:43:28

by Trond Myklebust

[permalink] [raw]
Subject: Re: [NFS] "mount: File exists" when trying to mount a second krb5 volume!

On Fri, 2006-12-01 at 10:34 -0800, Paarvai Naai wrote:
> Hi all,
>
> I really do appreciate the high quality technical assistance
> I have gotten to date from this list. But at the same time,
> I am getting somewhat frustrated with the very spotty
> attention I am getting for a *critical* bug that somehow
> made it into the upstream kernel sources. This is directly
> affecting the ability to use krb5 on either NFSv3 or NFSv4.
>
> First I had to actually dig through the source and find the
> bug. Then I was offered a patch but when it didn't work
> there was very little response/assistance to help with my
> investigation into which kernel that patch is compatible with.
> Finally, I have been insisting that we get the fix back into the
> upstream sources and it appears that no one appreciates
> the importance.

2.6.19 is already out (and by the time this bug was found, the 2.6.19-rc
series was way into injury time).

The patch will go into 2.6.20, then we can look into sending it in as a
stable update to 2.6.19.x. Not before.

Trond

2006-12-01 19:04:21

by Paarvai Naai

[permalink] [raw]
Subject: Re: [NFS] "mount: File exists" when trying to mount a second krb5 volume!

Thank you for your quick response. I'm not too familiar with the
term "injury time" and the like. But regardless, what is the
purpose of a rc series if we can't get an important bug fix into it.

Here's what I propose and I am happy to help facilitate to
some degree. We should definitely get it into 2.6.19.x ASAP
and obviously into 2.6.20. And Fedora often adds extra
patches into their source RPMs. This is the most important
for my company. Even if we don't have it in 2.6.19 vanilla,
if I had a "certified" patch from you, we can try to take it
to Fedora and get it into their source RPM.

Thoughts?

Thanks,
Paarvai




On 12/1/06, Trond Myklebust <[email protected]> wrote:
>
> On Fri, 2006-12-01 at 10:34 -0800, Paarvai Naai wrote:
> > Hi all,
> >
> > I really do appreciate the high quality technical assistance
> > I have gotten to date from this list. But at the same time,
> > I am getting somewhat frustrated with the very spotty
> > attention I am getting for a *critical* bug that somehow
> > made it into the upstream kernel sources. This is directly
> > affecting the ability to use krb5 on either NFSv3 or NFSv4.
> >
> > First I had to actually dig through the source and find the
> > bug. Then I was offered a patch but when it didn't work
> > there was very little response/assistance to help with my
> > investigation into which kernel that patch is compatible with.
> > Finally, I have been insisting that we get the fix back into the
> > upstream sources and it appears that no one appreciates
> > the importance.
>
> 2.6.19 is already out (and by the time this bug was found, the 2.6.19-rc
> series was way into injury time).
>
> The patch will go into 2.6.20, then we can look into sending it in as a
> stable update to 2.6.19.x. Not before.
>
> Trond
>
>


Attachments:
(No filename) (1.76 kB)
(No filename) (2.24 kB)
(No filename) (138.00 B)
Download all attachments

2006-12-01 19:16:47

by Trond Myklebust

[permalink] [raw]
Subject: Re: [NFS] "mount: File exists" when trying to mount a second krb5 volume!

On Fri, 2006-12-01 at 11:04 -0800, Paarvai Naai wrote:
> Thank you for your quick response. I'm not too familiar with the
> term "injury time" and the like. But regardless, what is the
> purpose of a rc series if we can't get an important bug fix into it.

That depends entirely on the nature of the bug. This one does not
corrupt data, it does not cause an Oops, so we have time to look
carefully at a solution.

> Here's what I propose and I am happy to help facilitate to
> some degree. We should definitely get it into 2.6.19.x ASAP
> and obviously into 2.6.20. And Fedora often adds extra
> patches into their source RPMs. This is the most important
> for my company. Even if we don't have it in 2.6.19 vanilla,
> if I had a "certified" patch from you, we can try to take it
> to Fedora and get it into their source RPM.

The procedures for dealing with kernel regressions are already well
established. I will be following them as usual.

Trond

2006-12-01 20:57:33

by Paarvai Naai

[permalink] [raw]
Subject: Re: [NFS] "mount: File exists" when trying to mount a second krb5 volume!

On 12/1/06, Trond Myklebust <[email protected]> wrote:

> The procedures for dealing with kernel regressions are already well
> established. I will be following them as usual.
>
> Trond
>
>
That's good to know. What is the well-established procedure for Fedora?
I really want to make sure it gets in ASAP. The importance of this
(basically that the current bug *nearly completely breaks* krb5 with
NFS) needs to be impressed on them.

Thanks,
Paarvai


Attachments:
(No filename) (462.00 B)
(No filename) (864.00 B)
(No filename) (138.00 B)
Download all attachments

2006-12-04 18:55:38

by Paarvai Naai

[permalink] [raw]
Subject: Re: [NFS] "mount: File exists" when trying to mount a second krb5 volume!

I have added this bug to bugzilla.redhat.com and requested that they
push Trond's patch into their source RPMs as soon as possible. Sorry if
I'm stepping on anyone toes, but I just felt that I had to proceed since I
didn't receive a response to my previous question (see below).

Thanks,
Paarvai

On 12/1/06, Paarvai Naai <[email protected]> wrote:
>
> On 12/1/06, Trond Myklebust <[email protected] > wrote:
>
> > The procedures for dealing with kernel regressions are already well
> > established. I will be following them as usual.
> >
> > Trond
> >
> >
> That's good to know. What is the well-established procedure for Fedora?
> I really want to make sure it gets in ASAP. The importance of this
> (basically that the current bug *nearly completely breaks* krb5 with
> NFS) needs to be impressed on them.
>
> Thanks,
> Paarvai
>
>
>
>


Attachments:
(No filename) (858.00 B)
(No filename) (1.60 kB)
(No filename) (138.00 B)
Download all attachments

2006-12-05 02:36:57

by Ian Kent

[permalink] [raw]
Subject: Re: "mount: File exists" when trying to mount a second krb5 volume!

On Mon, 4 Dec 2006, Paarvai Naai wrote:

> I have added this bug to bugzilla.redhat.com and requested that they
> push Trond's patch into their source RPMs as soon as possible. Sorry if
> I'm stepping on anyone toes, but I just felt that I had to proceed since I
> didn't receive a response to my previous question (see below).

Sorry, I meant to say to log a bug.

But, looking at the history in this thread, it appears that a solution
hasn't been reached.

>
> Thanks,
> Paarvai
>
> On 12/1/06, Paarvai Naai <[email protected]> wrote:
> >
> > On 12/1/06, Trond Myklebust <[email protected] > wrote:
> >
> > > The procedures for dealing with kernel regressions are already well
> > > established. I will be following them as usual.
> > >
> > > Trond
> > >
> > >
> > That's good to know. What is the well-established procedure for Fedora?
> > I really want to make sure it gets in ASAP. The importance of this
> > (basically that the current bug *nearly completely breaks* krb5 with
> > NFS) needs to be impressed on them.
> >
> > Thanks,
> > Paarvai
> >
> >
> >
> >
>

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2006-12-05 19:06:26

by Paarvai Naai

[permalink] [raw]
Subject: Re: [NFS] "mount: File exists" when trying to mount a second krb5 volume!

Hi Ian,

Regarding the resolution of the issue, Trond sent out a patch. Please see:

http://linux-nfs.org/pipermail/nfsv4/2006-November/005315.html

I've verified this patch on my system and Trond had it working as well.
The initial confusion was that the patch did not apply to 2.6.18 and
when I hacked it up it was causing segfaults on umount. But that's not
really a concern since we can get the patch into 2.6.19 and up.

Thanks,
Paarvai


On 12/4/06, Ian Kent <[email protected]> wrote:
> On Mon, 4 Dec 2006, Paarvai Naai wrote:
>
> > I have added this bug to bugzilla.redhat.com and requested that they
> > push Trond's patch into their source RPMs as soon as possible. Sorry if
> > I'm stepping on anyone toes, but I just felt that I had to proceed since I
> > didn't receive a response to my previous question (see below).
>
> Sorry, I meant to say to log a bug.
>
> But, looking at the history in this thread, it appears that a solution
> hasn't been reached.
>
> >
> > Thanks,
> > Paarvai
> >
> > On 12/1/06, Paarvai Naai <[email protected]> wrote:
> > >
> > > On 12/1/06, Trond Myklebust <[email protected] > wrote:
> > >
> > > > The procedures for dealing with kernel regressions are already well
> > > > established. I will be following them as usual.
> > > >
> > > > Trond
> > > >
> > > >
> > > That's good to know. What is the well-established procedure for Fedora?
> > > I really want to make sure it gets in ASAP. The importance of this
> > > (basically that the current bug *nearly completely breaks* krb5 with
> > > NFS) needs to be impressed on them.
> > >
> > > Thanks,
> > > Paarvai
> > >
> > >
> > >
> > >
> >
>

2007-01-23 22:52:32

by Paarvai Naai

[permalink] [raw]
Subject: Re: "mount: File exists" when trying to mount a second krb5 volume!

Hi all,

Sadly, Trond's patch has *still* not made it to the upstream sources.
For example, Fedora Core 6 has released a kernel package based on
2.6.19.2 and it doesn't appear to have the fix.

I'm confused as to why this fairly serious regression has not been
corrected by either 2.6.19.1 or 2.6.19.2. What can be done to get
this remedied asap?

Incidentally, the folks at bugzilla.redhat.com are now on top of this
problem for the upcoming RHEL 5, but that doesn't help non-RHEL users
any!

Thanks,
Paarvai


> > > > On 12/1/06, Trond Myklebust <[email protected] > wrote:
> > > >
> > > > > The procedures for dealing with kernel regressions are already well
> > > > > established. I will be following them as usual.
> > > > >
> > > > > Trond

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs