Hi,
I upgraded from 2.2.14 to 2.2.16 after the security
bug was discovered. Ever since, I have two boxes here
that keep falling over. Box A will randomly lock without
warning and box B will die and start printing this message
repeatedly on the screen until I physically hit reset:
VM: do_try_to_free_pages failed for cron...
...or something similar (could be apache etc.)
I tried upgrading to 2.2.17 for a week and the problem
didn't go away, installed 2.2.18 last night and the problem
is still there. Don't know what to do since anything
> 2.2.14 is very unstable on these machines, and anything
< 2.2.14 has the security bug.
Both machines have lots of these in the logs, just as others
have reported it happens in spurts:
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for kupdate...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for eggdrop...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for mysqld...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for apache...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for eggdrop...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for klogd...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for eggdrop...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for init...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for traceroute...
...etc.
Sometimes this will happen 10 minutes after a reboot,
or 45 mins, but neither machine will stay up for more
than 30 hours or so (usually *much* less).
Something else I noticed is that the Load average
is usually around 0.08, but when I let it idle for a
few mins, just tapping the spacebar in a terminal will
cause it to stop responding for 10 or so seconds with
the load average skyrocketing to over 6. After that the
system sometimes recovers and starts responding normally,
other times it will die.
Is there a patch out there that I can apply to 2.2.14
against the security bug? The machines were very stable
on that kernel.
Anything I can do from userland to help, let me know.
--
Mark
grep me no patterns and I'll tell you no lines.
> bug was discovered. Ever since, I have two boxes here
> that keep falling over. Box A will randomly lock without
> warning and box B will die and start printing this message
> repeatedly on the screen until I physically hit reset:
What are these two boxes doing ?
> Is there a patch out there that I can apply to 2.2.14
> against the security bug? The machines were very stable
> on that kernel.
Andrea's VM-global patch seems to be a wonder cure for those who have tried
it. Give it a shot and let folks know.
On Thu, Dec 14, 2000 at 09:57:28AM +0000, Alan Cox wrote:
> > bug was discovered. Ever since, I have two boxes here
> > that keep falling over. Box A will randomly lock without
> > warning and box B will die and start printing this message
> > repeatedly on the screen until I physically hit reset:
>
> What are these two boxes doing ?
>
> > Is there a patch out there that I can apply to 2.2.14
> > against the security bug? The machines were very stable
> > on that kernel.
>
> Andrea's VM-global patch seems to be a wonder cure for those who have tried
> it. Give it a shot and let folks know.
My experience:
2.2.18pre25 erroneously booted with mem=64M:
slrnpull --expire on a news-spool of about 600 Mb in 200,000 files gave
a lot of 'trying_to_free..' errors.
2.2.18 + VM-global, booted with mem=32M:
slrnpull --expire on the same spool worked fine.
Good luck,
Jurriaan
--
proof by reference to inaccessible literature:
The author cites a simple corollary of a theorem to be found
in a privately circulated memoir of the Slovenian
Philological Society, 1883 (second edition).
GNU/Linux 2.2.18 SMP 2x1117 bogomips load av: 0.56 0.15 0.05
> slrnpull --expire on a news-spool of about 600 Mb in 200,000 files gave
> a lot of 'trying_to_free..' errors.
>
> 2.2.18 + VM-global, booted with mem=32M:
>
> slrnpull --expire on the same spool worked fine.
I think Andrea just earned his official God status ;)
Alan Cox wrote:
> > slrnpull --expire on a news-spool of about 600 Mb in 200,000 files gave
> > a lot of 'trying_to_free..' errors.
> >
> > 2.2.18 + VM-global, booted with mem=32M:
> >
> > slrnpull --expire on the same spool worked fine.
>
> I think Andrea just earned his official God status ;)
So, maybe his divine VM patches will make it into 2.2.19?
jjs
On 2000/12/14 Alan Cox wrote:
> > slrnpull --expire on a news-spool of about 600 Mb in 200,000 files gave
> > a lot of 'trying_to_free..' errors.
> >
> > 2.2.18 + VM-global, booted with mem=32M:
> >
> > slrnpull --expire on the same spool worked fine.
>
> I think Andrea just earned his official God status ;)
>
How about a 2.2.19-pre1 == 2.2.18-aa2 ?
--
Juan Antonio Magallon Lacarta #> cd /pub
mailto:[email protected] #> more beer
Linux werewolf 2.2.18-aa2 #1 SMP Thu Dec 14 21:22:40 CET 2000 i686
> > I think Andrea just earned his official God status ;)
> So, maybe his divine VM patches will make it into 2.2.19?
The question is merely 'in what form' . I wanted to keep them seperate from
the other large changes in 2.2.18 for obvious reasons.
Andrea - can we have the core VM changes you did without adopting the
change in semaphore semantics for file system locking which will give third
party fs maintainers headaches and doesnt match 2.4 behaviour either ?
Alan
On Thu, Dec 14, 2000 at 11:17:11PM +0000, Alan Cox wrote:
> Andrea - can we have the core VM changes you did without adopting the
> change in semaphore semantics for file system locking which will give third
> party fs maintainers headaches and doesnt match 2.4 behaviour either ?
The changes in semaphore semantics are necessary to fix the spurious out of
memory with MAP_SHARED mappings and they came together with the removal of the
always-asynchronous kpiod. While it's certainly possible to remove it I don't
think removing the fix for MAP_SHARED stuff is a good idea.
Basically it's always safe to replace:
down(&inode->i_sem);
/* critical section */
up(&inode->isem);
with the new fs-semaphore:
fs_down(&inode->i_sem);
/* critical section */
fs_up(&inode->i_sem);
While it's always safe to use fs_down() in place of down(), it should be done
only with inodes that can be memory mapped using MAP_SHARED if in the
/* critical section */ there is any kind of not-GFP_ATOMIC or not-GFP_BUFFER
allocation like any kind of user-copy (as in every f_ops->write callback).
Most of those down are handled in the VFS and 99% of the time no changes are
necessary in the lowlevel fs so it should not even generate headaches to third
party fs maintainers (I think infact reiserfs patch didn't need any change).
Note: directory i_sem doesn't need the fs_down since a directory can't be
mapped in memory.
2.4.x doesn't need this per-process fs_locks information to avoid recursing on
the i_sem while flushing MAP_SHARED mappings to disk because the lowlevel fs
is required to implement a_ops->writepage to support MAP_SHARED, and the
page-flush gets synchronized only via the per-page lock (while in 2.2.x we need
the i_sem on the inode to call the non-zero-copy f_ops->write to flush the
MAP_SHARED dirty pages).
Andrea
> The changes in semaphore semantics are necessary to fix the spurious out of
> memory with MAP_SHARED mappings and they came together with the removal of the
> always-asynchronous kpiod. While it's certainly possible to remove it I don't
> think removing the fix for MAP_SHARED stuff is a good idea.
How hard is it to seperate losing kpiod (optimisation) from the MAP_SHARED
changes ? I am assuming they are two seperate issues, possibly wrongly
> Basically it's always safe to replace:
>
> down(&inode->i_sem);
> /* critical section */
> up(&inode->isem);
>
> with the new fs-semaphore:
>
> fs_down(&inode->i_sem);
> /* critical section */
> fs_up(&inode->i_sem);
Providing no inode semaphore is upped from a different task , which seems
currently quite a valid legal thing to do (ditto doing the up on completion of
something in bh or irq context)
Alan
On Fri, Dec 15, 2000 at 05:57:18PM +0000, Alan Cox wrote:
> How hard is it to seperate losing kpiod (optimisation) from the MAP_SHARED
> changes ? I am assuming they are two seperate issues, possibly wrongly
Losing kpiod isn't an optimization ;(. Losing kpiod is the MAP_SHARED bugfix.
The problem was:
o swap_out
o wants to flush a MAP_SHARED dirty page to disk
o so allocate kpiod-struct
o sumbit the page-flush request to kpiod
o don't wait I/O completion to avoid deadlocking on the i_sem
o swap_out returns 1 and memory balancing code so thinks we did progress
in freeing memory and goes to allocate memory from the freelist
without waiting I/O completion
o repeat N times the above
o in the meantime kpiod has a big queue but it's blocked slowly writing
those pages to disk
o while it writes a few pages swap_out floods again the queue
without waiting and it empties the freelist (task killed)
The problem was the lack of write throttling due the kpiod async-only nature.
> Providing no inode semaphore is upped from a different task , which seems
> currently quite a valid legal thing to do (ditto doing the up on completion of
> something in bh or irq context)
Yes, the same `current' context must run the down/up pair of calls and as you
said it is legal to rely on it on all the places it's used.
Andrea
----- Original Message -----
From: "Alan Cox" <[email protected]>
To: "Mark Symonds" <[email protected]>
Cc: <[email protected]>
Sent: Thursday, December 14, 2000 1:57 AM
Subject: Re: VM problems still in 2.2.18
Hihi,
>> Box A will randomly lock without
>> warning and box B will die and start printing this message
>> repeatedly on the screen until I physically hit reset:
>
>What are these two boxes doing ?
One is a P233 32MB and the other is a P100 24MB. We do
run *alot* of different things on it and the load has
grown substantially since the switch from 2.2.14. For
example one of the web developers did a webmail which uses
php-mysql so we have apache-ssl's and mysqld's sitting
there, then there is the guy who likes jserv, then the
mailserver (around 2000 a day) etc. etc.
> Andrea's VM-global patch seems to be a wonder cure for
> those who have tried it. Give it a shot and let folks know.
OK, will do. Just booted the new patched kernel about
five minutes ago so should know before the end of the
weekend if that does it.
Thanks much. :)
--
Mark
grep me no patterns and I'll tell you no lines.
> o don't wait I/O completion to avoid deadlocking on the i_sem
> o swap_out returns 1 and memory balancing code so thinks we did progress
> in freeing memory and goes to allocate memory from the freelist
> without waiting I/O completion
> o repeat N times the above
so the actual problem is either - the returning 1 when it is the wrong answer
- or the failure to block somewhere else (where its safe) based on a kpiod
maintained semaphore ?
> Yes, the same `current' context must run the down/up pair of calls and as you
> said it is legal to rely on it on all the places it's used.
I assume thats not an issue to reiserfs ?
On Fri, Dec 15, 2000 at 06:46:32PM +0000, Alan Cox wrote:
> so the actual problem is either - the returning 1 when it is the wrong answer
> - or the failure to block somewhere else (where its safe) based on a kpiod
> maintained semaphore ?
The problem is not to find a safe place where to wait, the problem is to know
"when" we can block. That's the only thing current->fs_locks tells us. Sometime
we simply can't wait because I/O can't start until we return (it would deadlock
regardless we wait on a kpiod semaphore or in down(i_sem) ourselfs).
Once we know "if" we can't wait or not the whole point of kpiod is lost
as kpiod exists only because we didn't know that and so we were not able
to wait.
Now we know when we can block so we can run f_ops->write ourselfs that's also
more efficient in terms of both performance and also memory pressure during
swap of course.
> I assume thats not an issue to reiserfs ?
As said reiserfs AFIK didn't need any change, so only VFS is using
fs_down/fs_up from the point of view of reiserfs.
Andrea
> Now we know when we can block so we can run f_ops->write ourselfs that's also
> more efficient in terms of both performance and also memory pressure during
> swap of course.
Yep
> As said reiserfs AFIK didn't need any change, so only VFS is using
> fs_down/fs_up from the point of view of reiserfs.
Ok. Im not keen on adding fs_down but it does look like the right bandage
(and a nice speed up) for 2.2.
I hate that kind of bug
On Fri, 15 Dec 2000, Alan Cox wrote:
> > Yes, the same `current' context must run the down/up pair of calls and as you
> > said it is legal to rely on it on all the places it's used.
>
> I assume thats not an issue to reiserfs ?
>
I don't think so. There are two places reiserfs calls down/up. In our
file->release() callback, and in the ioctl we added for lilo so it can
upack files with tails.
Neither calls copy_from/to_user or does an allocation other than
GFP_BUFFER. As far as I know that should be safe, but the change is
trivial if we need to make it.
-chris
On Sat, Dec 16, 2000 at 02:49:40AM -0800, Chris Mason wrote:
> GFP_BUFFER. As far as I know that should be safe, but the change is
Yes that's ok.
Andrea
----- Original Message -----
From: "Alan Cox" <[email protected]>
To: <[email protected]>
Cc: <[email protected]>
Sent: Thursday, December 14, 2000 2:38 PM
Subject: Re: [lkml]Re: VM problems still in 2.2.18
>
> I think Andrea just earned his official God status ;)
Just wanted to quickly report that after applying
Andreas patch, the problem has been fixed here as
well. I've got two machines here that were previously
unable to stay up more than a day, been pounding on
them since Friday without any problems.
Great! :)
--
Mark
grep me no patterns and I'll tell you no lines.