On Sat, Feb 16, 2008 at 12:18:58AM +0100, Ferenc Wagner wrote:
> Hi,
>
> 5 days ago I pulled the git tree (HEAD was
> 25f666300625d894ebe04bac2b4b3aadb907c861), added two minor patches
> (the vmsplice fix and the GFS1 exports), compiled and booted the
> kernel. Things are working OK, but I noticed that inode usage has
> been steadily rising since then (see attached graph, unless lost in
> transit). The real filesystems used by the machine are XFS. I wonder
> if it may be some kind of bug and if yes, whether it has been fixed
> already. Feel free to ask for any missing information.
Output of /proc/slabinfo, please. If you could get a sample when the
machine has just booted, one at the typical post-boot steady state
as well as one after it has increased well past normal, it would
help identify what type of inode is increasing differently.
Also, can you tell us what metrics you are graphing (i.e. where
in /proc or /sys you are getting them from)?
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
David Chinner <[email protected]> writes:
> On Sat, Feb 16, 2008 at 12:18:58AM +0100, Ferenc Wagner wrote:
>
>> 5 days ago I pulled the git tree (HEAD was
>> 25f666300625d894ebe04bac2b4b3aadb907c861), added two minor patches
>> (the vmsplice fix and the GFS1 exports), compiled and booted the
>> kernel. Things are working OK, but I noticed that inode usage has
>> been steadily rising since then (see attached graph, unless lost in
>> transit). The real filesystems used by the machine are XFS. I wonder
>> if it may be some kind of bug and if yes, whether it has been fixed
>> already. Feel free to ask for any missing information.
>
> Output of /proc/slabinfo, please. If you could get a sample when the
> machine has just booted, one at the typical post-boot steady state
> as well as one after it has increased well past normal, it would
> help identify what type of inode is increasing differently.
Ugh. Your message came just a little bit too late, I rebooted the
machine a couple of hours ago for applying an IPv6 patch, without
saving /proc/slabinfo. The currently running kernel is 2.6.24.2 +
GFS1 exports + IPv6 fix, and I snapshotted /proc/slabinfo approx. 3
hours after reboot. At least we will see whether this version also
produces the problem, it isn't too different after all (for some
"too").
Btw. there was no steady-state with the previous kernel, the increase
started right after reboot, which means that by tomorrow I'll be able
to tell whether it's increasing again or this kernel doesn't exhibit
such effect.
> Also, can you tell us what metrics you are graphing (i.e. where
> in /proc or /sys you are getting them from)?
I wonder why I assumed everybody knows Munin's graphs by heart...
In short: "inode table size" is the first value from
/proc/sys/fs/inode-nr; "open inodes" is the same minus the second
value. In other words:
awk '{print "used.value " $1-$2 "\nmax.value " $1}' < /proc/sys/fs/inode-nr
I'll come back shortly with the new findings. If nothing turns up,
it's possible to boot up the previous kernel (or -- if needed --
current git) with this IPv6 fix added and check that again.
--
Thanks,
Feri.
On Tue, Feb 19, 2008 at 05:57:08PM +0100, Ferenc Wagner wrote:
> David Chinner <[email protected]> writes:
> > On Sat, Feb 16, 2008 at 12:18:58AM +0100, Ferenc Wagner wrote:
> So, I loaded the same kernel on a different machine, but that seems to
> exhibit a very similar behaviour. The machine is absolutely idle,
> nobody logged in during this period, though an updatedb ran during
> this period. However, the increase seems steady, not correlated to
> cron.daily.
>
> The contents of /proc/sys/fs/inode-nr after reboot was:
> 4421 95
>
> and now, 13h35m later it's:
> 146182 0
>
> Find the two slabinfo outputs attached.
Content-Description: slabinfo output immediately after reboot
> xfs_inode 791 800 384 10 1 : tunables 54 27 8 : slabdata 80 80 0
> xfs_vnode 790 790 384 10 1 : tunables 54 27 8 : slabdata 79 79 0
> dentry 5133 5133 132 29 1 : tunables 120 60 8 : slabdata 177 177 0
Content-Description: slabinfo output 13h35m after reboot
> xfs_inode 142548 142550 384 10 1 : tunables 54 27 8 : slabdata 14255 14255 0
> xfs_vnode 142548 142550 384 10 1 : tunables 54 27 8 : slabdata 14255 14255 0
> dentry 148003 148074 132 29 1 : tunables 120 60 8 : slabdata 5106 5106 0
> The xfs_inode, xfs_vnode and dentry lines show significant increase.
> The machine indeed uses XFS as its root filesystem. Hope this gives
> enough clues to narrow down the problem. I can try other kernels if
> needed.
The xfs inodes are clearly pinned by the dentry cache, so the issue
is dentries, not inodes. What's causing dentries not to be
reclaimed? I can't see anything that cold pin them (e.g. no filp's
that would indicate open files being responsible), so my initial
thoughts are that memory reclaim may have changed behaviour.
I guess the first thing to find out is whether memory pressure
results in freeing the dentries. To simulate memory pressure causing
slab cache reclaim, can you run:
# echo 2 > /proc/sys/vm/drop_caches
and see if the number of dentries and inodes drops. If the number
goes down significantly, then we aren't leaking dentries and there's
been a change in memoy reclaim behaviour. If it stays the same, then
we probably are leaking dentries....
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
David Chinner <[email protected]> writes:
> The xfs inodes are clearly pinned by the dentry cache, so the issue
> is dentries, not inodes. What's causing dentries not to be
> reclaimed? I can't see anything that cold pin them (e.g. no filp's
> that would indicate open files being responsible), so my initial
> thoughts are that memory reclaim may have changed behaviour.
>
> I guess the first thing to find out is whether memory pressure
> results in freeing the dentries. To simulate memory pressure causing
> slab cache reclaim, can you run:
>
> # echo 2 > /proc/sys/vm/drop_caches
>
> and see if the number of dentries and inodes drops. If the number
> goes down significantly, then we aren't leaking dentries and there's
> been a change in memoy reclaim behaviour. If it stays the same, then
> we probably are leaking dentries....
Hi Dave,
Thanks for looking into this. There's no real conclusion yet: the
simulated memory pressure sent the numbers down allright, but
meanwhile it turned out that this is a different case: on this machine
the increase wasn't a constant growth, but related to the daily
updatedb job. I'll reload the original kernel on the original
machine, and collect the same info if the problem reappers.
--
Regards,
Feri.
On Wed, Feb 20, 2008 at 03:36:53PM +0100, Ferenc Wagner wrote:
> David Chinner <[email protected]> writes:
> > The xfs inodes are clearly pinned by the dentry cache, so the issue
> > is dentries, not inodes. What's causing dentries not to be
> > reclaimed? I can't see anything that cold pin them (e.g. no filp's
> > that would indicate open files being responsible), so my initial
> > thoughts are that memory reclaim may have changed behaviour.
> >
> > I guess the first thing to find out is whether memory pressure
> > results in freeing the dentries. To simulate memory pressure causing
> > slab cache reclaim, can you run:
> >
> > # echo 2 > /proc/sys/vm/drop_caches
> >
> > and see if the number of dentries and inodes drops. If the number
> > goes down significantly, then we aren't leaking dentries and there's
> > been a change in memoy reclaim behaviour. If it stays the same, then
> > we probably are leaking dentries....
>
> Hi Dave,
>
> Thanks for looking into this. There's no real conclusion yet: the
> simulated memory pressure sent the numbers down allright, but
> meanwhile it turned out that this is a different case: on this machine
> the increase wasn't a constant growth, but related to the daily
> updatedb job. I'll reload the original kernel on the original
> machine, and collect the same info if the problem reappers.
Ok, let me know how it goes when you get a chance.
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group