2007-11-13 22:44:47

by Roland

[permalink] [raw]
Subject: uuid/blkid performance problem with large number of mounts - was: Re: stale nfs file handle with exported loopback mounts

since i posted 2 problems in one mail - and one problem is solved now, for =
the second problem i have opened a ticket at =


http://sourceforge.net/tracker/index.php?func=3Ddetail&aid=3D1831403&group_=
id=3D14&atid=3D100014 =

( uuid/blkid performance problem with large number of mounts )

so this won`t get lost.

regards
roland


> -----Urspr=FCngliche Nachricht-----
> Von: [email protected]
> Gesendet: 02.11.07 20:06:58
> An: Neil Brown <[email protected]>
> CC: "J. Bruce Fields" <[email protected]>, [email protected]
> Betreff: Re: [NFS] stale nfs file handle with exported loopback mounts


> =

> hi!
> =

> it seems i was having weird mail problems with sending mails trough my we=
bmailer - at least two followups with attachments seem to be lost on sendin=
g and are not in my sent folder anymore....
> =

> anyway - here is a second try, but probably worse than what i have writte=
n before :)
> =

> =

> first off, thanks for the patch Neil, things look _much_ better now and e=
xporting loopback mounts now basiscally works again.
> nice to see that my posting helped finding bugs.
> =

> maybe i have two more bugs for you :)
> =

> i have loopback mounts on the server and exported the parent dir with cro=
ssmnt option.
> =

> after mounting for the first time on the client, i`m getting "Invalid arg=
ument" for each loopback-mounted dir, if i do an ls -la on /mnt.
> this only happens _once_ and seems to be a server problem, because i can =
reboot the client and remount , i never see that errors again.
> =

> besides that, all seems to work fine.
> =

> as neil suggested, i have made a tcpdump of this available at:
> http://82.141.46.148/bugs/nfs/tcpdump.out.bz2
> =

> =

> furthermore, there is a very strange performance issue i was able to trac=
k down to uuid/blkid support.
> =

> i recognized this issue when i exported a directory containing a very lar=
ge number of loopback mounts via crossmnt export option.
> ls -la on the clients mountpoint seemed to hung and i could see mountd be=
ing busy, eating 100% cpu for quite a while.
> =

> the time needed for ls to finish seems to grow exponentially with the num=
ber of loopback-mounts inside the exported directory - i also tried with 10=
00 loopback mounts and mountd being busy for several minutes with this.
> =

> i have made a strace of mountd available at:
> http://82.141.46.148/bugs/nfs/mountd.strace.txt.bz2
> =

> you can see that mountd seems to be busy doing the same things over and o=
ver again, looks that it does stat64() for all devices in /etc/blkid.tab fo=
r each loopback mount, weird.
> =

> here is some "strace -c -p $PID_OF_MOUNTD" for comparison - without uuid=
/blkid support compiled in it looks like this:
> =

> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 73.23 0.147722 2 66313 stat64
> 10.37 0.020923 20 1031 write
> 5.54 0.011179 23 494 select
> 3.82 0.007699 5 1546 read
> 3.04 0.006137 8 773 time
> 2.18 0.004393 6 769 lstat64
> 1.08 0.002182 4 519 munmap
> 0.40 0.000797 1 1035 close
> 0.29 0.000594 1 1034 open
> 0.04 0.000089 0 1036 fstat64
> 0.00 0.000000 0 2 alarm
> 0.00 0.000000 0 3 _llseek
> 0.00 0.000000 0 1 fdatasync
> 0.00 0.000000 0 2 poll
> 0.00 0.000000 0 2 rt_sigaction
> 0.00 0.000000 0 521 mmap2
> 0.00 0.000000 0 2 fcntl64
> 0.00 0.000000 0 1 socket
> 0.00 0.000000 0 1 connect
> 0.00 0.000000 0 1 accept
> 0.00 0.000000 0 2 send
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.201715 75088 total
> =

> =

> =

> this is an strace -c when uuid/blkid support is being compiled in:
> =

> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 61.64 1.008158 2 550916 stat64
> 21.67 0.354441 9 37662 read
> 5.65 0.092476 15 6377 getdents64
> 4.06 0.066381 3 21395 8232 open
> 1.62 0.026485 2 13169 fstat64
> 1.36 0.022237 2 13164 close
> 1.22 0.020025 2 8414 lstat64
> 1.15 0.018805 4 4415 munmap
> 0.27 0.004382 17 258 rename
> 0.26 0.004329 17 258 unlink
> 0.26 0.004305 2 2101 write
> 0.23 0.003786 1 4380 fcntl64
> 0.18 0.002899 11 262 select
> 0.18 0.002883 11 258 access
> 0.11 0.001857 0 4417 mmap2
> 0.11 0.001765 0 4652 time
> 0.01 0.000237 1 258 link
> 0.00 0.000041 0 258 lseek
> 0.00 0.000000 0 2 alarm
> 0.00 0.000000 0 2 brk
> 0.00 0.000000 0 1 gettimeofday
> 0.00 0.000000 0 258 fchmod
> 0.00 0.000000 0 265 _llseek
> 0.00 0.000000 0 1 fdatasync
> 0.00 0.000000 0 2 poll
> 0.00 0.000000 0 1 prctl
> 0.00 0.000000 0 2 rt_sigaction
> 0.00 0.000000 0 1 getuid32
> 0.00 0.000000 0 1 getgid32
> 0.00 0.000000 0 1 geteuid32
> 0.00 0.000000 0 1 getegid32
> 0.00 0.000000 0 1 futex
> 0.00 0.000000 0 1 socket
> 0.00 0.000000 0 1 connect
> 0.00 0.000000 0 1 accept
> 0.00 0.000000 0 2 send
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 1.635492 673158 8232 total
> =

> =

> as you can see there is an unusual high number of stat64() calls
> =

> server is opensuse 10.3 , client is suse 9.3 professional
> =

> if i can help resolving this issue, tell me what to do :)
> =

> regards
> roland
> =

> =

> =

> > -----Urspr=FCngliche Nachricht-----
> > Von: Neil Brown <[email protected]>
> > Gesendet: 01.11.07 05:26:50
> > An: [email protected]
> > CC: "J. Bruce Fields" <[email protected]>, [email protected]
> > Betreff: Re: [NFS] stale nfs file handle with exported loopback mounts
> =

> =

> > =

> > On Wednesday October 31, [email protected] wrote:
> > > ok, i just wanted to tell that this isn`t the right way to go imho.
> > > =

> > > some time ago i have tested exporting a parent dir containing
> > > several loopback mounted iso images with some pre-1.1.0 nfs-utils
> > > version and it worked - so =EC wonder why it now seems to have issues
> > > as things should have gone stable..... =

> > =

> > We have a way of breaking things sometimes.... It's called
> > "progress". :-)
> > =

> > The short answer is that there is a bug in mountd which is fixed by
> > this patch:
> > =

> > diff --git a/utils/mountd/cache.c b/utils/mountd/cache.c
> > index ce1a5a9..fd317cd 100644
> > --- a/utils/mountd/cache.c
> > +++ b/utils/mountd/cache.c
> > @@ -508,7 +508,7 @@ void nfsd_fh(FILE *f)
> > */
> > qword_printint(f, 0x7fffffff);
> > if (found)
> > - qword_print(f, found->e_path);
> > + qword_print(f, found_path);
> > qword_eol(f);
> > out:
> > free(found_path);
> > =

> > =

> > The longer answer is that there is also a bug in "mount.nfs" which is
> > unrelated but was slowing me down in chasing this bug, and there is
> > also a bug in the NFS client which was causing my client oops and need
> > a reboot every time I triggered this bug in mountd, which further
> > slowed me down.
> > =

> > The effect of this bug in mountd is that when the NFS client calls
> > GETATTR on the root of the subordinate filesystem (e.g. your
> > loop-mounted isos), it got attr information about the parent. ie. the
> > top-level exported filesystem (/export in your case I think).
> > This has a different 'fsid' than the nfs client was expecting and
> > the NFS client got confused in various ways.
> > =

> > Thanks for your problem report - it helped find 3 bugs!
> > =

> > I'll get proper patches or bug report off to the relevant maintainers
> > shortly.
> > =

> > NeilBrown
> > =

> =

> =



______________________________________________________________________
XXL-Speicher, PC-Virenschutz, Spartarife & mehr: Nur im WEB.DE Club! =

Jetzt testen! http://produkte.web.de/club/?mc=3D021130


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs


2007-11-15 05:26:53

by NeilBrown

[permalink] [raw]
Subject: Re: uuid/blkid performance problem with large number of mounts - was: Re: stale nfs file handle with exported loopback mounts

On Tuesday November 13, [email protected] wrote:
> since i posted 2 problems in one mail - and one problem is solved now, for the second problem i have opened a ticket at
>
> http://sourceforge.net/tracker/index.php?func=detail&aid=1831403&group_id=14&atid=100014
> ( uuid/blkid performance problem with large number of mounts )
>
> so this won`t get lost.

Unfortuantely that is a bit like saying "I've placed in the nice round
filling cabinet so it won't get lost", not realising that the nice
round filing cabinet is actually a garbage be.

No one (that I know of) looks at the issue tracker on sourceforge at
all.

I have it on my personal "list of things to look at one day", so maybe
it won't get forgotten.

thanks,
NeilBrown

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-11-15 07:54:24

by Roland

[permalink] [raw]
Subject: Re: uuid/blkid performance problem with large number of mounts - was: Re: stale nfs file handle with exported loopback mounts

> No one (that I know of) looks at the issue tracker on sourceforge at all.
ah -ok, sorry - i didn`t know. since there were recent entries from octob=
er, i thought it was an official way to report bugs.

> I have it on my personal "list of things to look at one day", so maybe
> it won't get forgotten.
thanks. =

if i`m allowed to add a comment:
a public bugtracker has one advantage: others can easily see that this or a=
nother bug exists and maybe they can provide (parts of) a solution or don`t=
need to ask for such.
i`m just too used to open source having open bugtrackers, so i thought it w=
as the right way to go. =


if bugs not being tracked there and and you explicitly discourage using it =
- what about disabling the bugtracker entirely ?

roland



> -----Urspr=FCngliche Nachricht-----
> Von: Neil Brown <[email protected]>
> Gesendet: 15.11.07 06:27:07
> An: [email protected]
> CC: "J. Bruce Fields" <[email protected]>, [email protected]
> Betreff: Re: [NFS] uuid/blkid performance problem with large number of mo=
unts - was: Re: stale nfs file handle with exported loopback mounts


> =

> On Tuesday November 13, [email protected] wrote:
> > since i posted 2 problems in one mail - and one problem is solved now, =
for the second problem i have opened a ticket at =

> > =

> > http://sourceforge.net/tracker/index.php?func=3Ddetail&aid=3D1831403&gr=
oup_id=3D14&atid=3D100014 =

> > ( uuid/blkid performance problem with large number of mounts )
> > =

> > so this won`t get lost.
> =

> Unfortuantely that is a bit like saying "I've placed in the nice round
> filling cabinet so it won't get lost", not realising that the nice
> round filing cabinet is actually a garbage be.
> =

> No one (that I know of) looks at the issue tracker on sourceforge at
> all.
> =

> I have it on my personal "list of things to look at one day", so maybe
> it won't get forgotten.
> =

> thanks,
> NeilBrown
> =



_____________________________________________________________________
Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
http://smartsurfer.web.de/?mc=3D100071&distributionid=3D000000000066


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-11-15 17:23:52

by Trond Myklebust

[permalink] [raw]
Subject: Re: uuid/blkid performance problem with large number of mounts - was: Re: stale nfs file handle with exported loopback mounts


On Thu, 2007-11-15 at 16:26 +1100, Neil Brown wrote:
> On Tuesday November 13, [email protected] wrote:
> > since i posted 2 problems in one mail - and one problem is solved now, for the second problem i have opened a ticket at
> >
> > http://sourceforge.net/tracker/index.php?func=detail&aid=1831403&group_id=14&atid=100014
> > ( uuid/blkid performance problem with large number of mounts )
> >
> > so this won`t get lost.
>
> Unfortuantely that is a bit like saying "I've placed in the nice round
> filling cabinet so it won't get lost", not realising that the nice
> round filing cabinet is actually a garbage be.
>
> No one (that I know of) looks at the issue tracker on sourceforge at
> all.
>
> I have it on my personal "list of things to look at one day", so maybe
> it won't get forgotten.

Is it possible to turn this feature off on sourceforge? It is rather
unfortunate that people take the time to type these things up, without
realising that the tracker is unmonitored. It would be even better if we
could display a message saying "please submit this to the mailing lists
and/or bugzilla".

Cheers
Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-11-15 16:53:04

by J. Bruce Fields

[permalink] [raw]
Subject: Re: uuid/blkid performance problem with large number of mounts - was: Re: stale nfs file handle with exported loopback mounts

On Thu, Nov 15, 2007 at 08:54:24AM +0100, [email protected] wrote:
> > No one (that I know of) looks at the issue tracker on sourceforge at
> > all.
> ah -ok, sorry - i didn`t know. since there were recent entries from
> october, i thought it was an official way to report bugs.
>
> > I have it on my personal "list of things to look at one day", so
> > maybe it won't get forgotten.
> thanks. if i`m allowed to add a comment: a public bugtracker has one
> advantage: others can easily see that this or another bug exists and
> maybe they can provide (parts of) a solution or don`t need to ask for
> such. i`m just too used to open source having open bugtrackers, so i
> thought it was the right way to go.

The bug tracker at http://bugzilla.kernel.org/ is actually used, at
least by some people.

There's also http://bugzilla.linux-nfs.org/. I think the kernel.org one
is probably be preferred at this point, especially for anything in a
mainstream kernel.

> if bugs not being tracked there and and you explicitly discourage
> using it - what about disabling the bugtracker entirely ?

That would make sense to me if there's an easy way to do it.

--b.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-11-15 20:07:20

by Roland

[permalink] [raw]
Subject: Re: uuid/blkid performance problem with large number of mounts - was: Re: stale nfs file handle with exported loopback mounts

> Is it possible to turn this feature off on sourceforge? I

yes - iirc, i have seen projects without a tracker or without other option=
, being on by default.

if you want to disable it, maybe this one helps:

https://sourceforge.net/docs/G03/en/#tracker

Tracker

Trackers may be flagged as hidden by project administrators. No means is pr=
ovided to completely remove a Tracker. All data will be preserved within hi=
dden Trackers, but it will not be accessible by site users.

To hide a Tracker as a project administrator:

1. Click on the "Tracker" link for the project.
2. Click on the "Admin" link.
3. Click on the name of the desired Tracker.
4. Click on the "Update Preferences" link.
5. Change the "Visibility of this Tracker" pulldown menu value as desire=
d. Default is "Visible to all site users". A change to "Visible to project =
members only" will restrict view of the Tracker to members of your project =
team. A change to "Not visible (hidden)" will prevent view of the Tracker b=
y any site user, including members of your project team; only Tracker admin=
s and project admins will then be able to see this Tracker when in the admi=
n view.
6. Click on the "SUBMIT" button to complete the change.

Tracker items may be removed by SourceForge.net staff, if given suitable re=
ason. The "Deleted" Tracker item "Status" value does not actually delete Tr=
acker items or remove them from view; this status option is provided for so=
rting purposes. No means is provided for project administrators and site us=
ers to remove Tracker items on their own. If you need a Tracker item remove=
d, please contact the SourceForge.net team. Similar is true of comments on =
Tracker items.

regards
roland


> -----Urspr=FCngliche Nachricht-----
> Von: "Trond Myklebust" <[email protected]>
> Gesendet: 15.11.07 18:24:24
> An: Neil Brown <[email protected]>
> CC: [email protected], [email protected]
> Betreff: Re: [NFS] uuid/blkid performance problem with large number of mo=
unts - was: Re: stale nfs file handle with exported loopback mounts


> =

> =

> On Thu, 2007-11-15 at 16:26 +1100, Neil Brown wrote:
> > On Tuesday November 13, [email protected] wrote:
> > > since i posted 2 problems in one mail - and one problem is solved now=
, for the second problem i have opened a ticket at =

> > > =

> > > http://sourceforge.net/tracker/index.php?func=3Ddetail&aid=3D1831403&=
group_id=3D14&atid=3D100014 =

> > > ( uuid/blkid performance problem with large number of mounts )
> > > =

> > > so this won`t get lost.
> > =

> > Unfortuantely that is a bit like saying "I've placed in the nice round
> > filling cabinet so it won't get lost", not realising that the nice
> > round filing cabinet is actually a garbage be.
> > =

> > No one (that I know of) looks at the issue tracker on sourceforge at
> > all.
> > =

> > I have it on my personal "list of things to look at one day", so maybe
> > it won't get forgotten.
> =

> Is it possible to turn this feature off on sourceforge? It is rather
> unfortunate that people take the time to type these things up, without
> realising that the tracker is unmonitored. It would be even better if we
> could display a message saying "please submit this to the mailing lists
> and/or bugzilla".
> =

> Cheers
> Trond
> =

> =



___________________________________________________________________________=
___
Jetzt neu! Im riesigen WEB.DE Club SmartDrive Dateien freigeben und mit =

Freunden teilen! http://www.freemail.web.de/club/smartdrive_ttc.htm/?mc=3D0=
21134


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-11-15 20:12:40

by Trond Myklebust

[permalink] [raw]
Subject: Re: uuid/blkid performance problem with large number of mounts - was: Re: stale nfs file handle with exported loopback mounts


On Thu, 2007-11-15 at 21:07 +0100, [email protected] wrote:
> > Is it possible to turn this feature off on sourceforge? I
>
> yes - iirc, i have seen projects without a tracker or without other
> option, being on by default.
>
> if you want to disable it, maybe this one helps:
>
> https://sourceforge.net/docs/G03/en/#tracker

That would have to be up to Neil since he is the only one with
administrator rights on that list.

Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs