2004-04-25 14:29:42

by Sven Geggus

[permalink] [raw]
Subject: 2.6.6-rc2-mm1 pdflush eats my CPU

Hi there,

my Desktop machine 2.6.6-rc2-mm1 got almost unusable starting with Kernel
2.6.6-rc2-mm1. The machine starts up with pdflush eating up all CPU.
2.6.6-rc2 without mm1 has teh same behavour.

--top output--
top - 16:27:07 up 21 min, 6 users, load average: 1.07, 1.14, 0.87
Tasks: 94 total, 2 running, 91 sleeping, 0 stopped, 1 zombie
Cpu(s): 6.7% user, 90.4% system, 0.0% nice, 1.8% idle, 1.1% IO-wait
Mem: 510596k total, 281644k used, 228952k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 114684k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8 root 25 0 0 0 0 R 93.2 0.0 18:15.08 pdflush
--cut--

Its not happening when I use something like init=/bin/bash, but it does
happen on Normal startup (KDM login).

The only thing which is somewhat exotic with this machine, is the fact, that
there are no filesystems mounted besides NFS (root-NFS).

It did not happen with 2.6.5-rc3 which I have been using before.

I will be able to provide further debugging Information, when somebody tells
me what will be of interest (output of readprofile etc.)

Regards

Sven

--
"The term "any key" does not refer to a particular key on the keyboard. It
simply means to strike any one of the keys on your keyboard or handheld
screen." (Compaq FAQ Entry 2859)
/me is giggls@ircnet, http://sven.gegg.us/ on the Web


2004-04-25 17:42:18

by jurriaan

[permalink] [raw]
Subject: Re: 2.6.6-rc2-mm1 pdflush eats my CPU

From: Sven Geggus <[email protected]>
Date: Sun, Apr 25, 2004 at 02:29:35PM +0000
> Hi there,
>
> my Desktop machine 2.6.6-rc2-mm1 got almost unusable starting with Kernel
> 2.6.6-rc2-mm1. The machine starts up with pdflush eating up all CPU.
> 2.6.6-rc2 without mm1 has teh same behavour.
>
> --top output--
> top - 16:27:07 up 21 min, 6 users, load average: 1.07, 1.14, 0.87
> Tasks: 94 total, 2 running, 91 sleeping, 0 stopped, 1 zombie
> Cpu(s): 6.7% user, 90.4% system, 0.0% nice, 1.8% idle, 1.1% IO-wait
> Mem: 510596k total, 281644k used, 228952k free, 0k buffers
> Swap: 0k total, 0k used, 0k free, 114684k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 8 root 25 0 0 0 0 R 93.2 0.0 18:15.08 pdflush
> --cut--
>
> Its not happening when I use something like init=/bin/bash, but it does
> happen on Normal startup (KDM login).
>
> The only thing which is somewhat exotic with this machine, is the fact, that
> there are no filesystems mounted besides NFS (root-NFS).
>
> It did not happen with 2.6.5-rc3 which I have been using before.
>
> I will be able to provide further debugging Information, when somebody tells
> me what will be of interest (output of readprofile etc.)
>
I had that too, but without NFS. Andrew Morton thought it might have to
do with reiserfs, and send me a patch. Perhaps you can try this patch
and report back what happened (to the list and Andrew).

> From: Andrew Morton <[email protected]>
> Date: Tue, Apr 20, 2004 at 07:15:33PM -0700
> > Jurriaan <[email protected]> wrote:
> > >
> > > at times, pdflush is taking over my system:
> >
> > yup, there's some logic error in there. If I could reproduce it, it would
> > be fixed in a jiffy :(
> >
> > Which filesystems are in active use at the time? reiserfs?
> >
>
> mostly, yes, also ext3, and raid-1 and linear raid. I can't always
> reproduce it either, unfortunately :-(

I'm suspecting that reiserfs_writepage() is redirty pages rather than
writing them and writeback is livelocking.

Could you try this?
---

25-akpm/fs/buffer.c | 2 +-
25-akpm/fs/ext3/inode.c | 6 +++---
25-akpm/fs/ntfs/aops.c | 15 +++------------
25-akpm/fs/reiserfs/inode.c | 2 +-
25-akpm/include/linux/mm.h | 2 ++
25-akpm/mm/page-writeback.c | 12 ++++++++++++
6 files changed, 22 insertions(+), 17 deletions(-)

diff -puN fs/buffer.c~writeback-livelock-fix fs/buffer.c
--- 25/fs/buffer.c~writeback-livelock-fix 2004-04-20 23:46:12.979432024 -0700
+++ 25-akpm/fs/buffer.c 2004-04-20 23:46:13.000428832 -0700
@@ -1826,7 +1826,7 @@ static int __block_write_full_page(struc
if (wbc->sync_mode != WB_SYNC_NONE || !wbc->nonblocking) {
lock_buffer(bh);
} else if (test_set_buffer_locked(bh)) {
- __set_page_dirty_nobuffers(page);
+ redirty_page_for_writepage(wbc, page);
continue;
}
if (test_clear_buffer_dirty(bh)) {
diff -puN fs/ext3/inode.c~writeback-livelock-fix fs/ext3/inode.c
--- 25/fs/ext3/inode.c~writeback-livelock-fix 2004-04-20 23:46:12.981431720 -0700
+++ 25-akpm/fs/ext3/inode.c 2004-04-20 23:46:12.998429136 -0700
@@ -1311,7 +1311,7 @@ static int ext3_ordered_writepage(struct
return ret;

out_fail:
- __set_page_dirty_nobuffers(page);
+ redirty_page_for_writepage(wbc, page);
unlock_page(page);
return ret;
}
@@ -1340,7 +1340,7 @@ static int ext3_writeback_writepage(stru
return ret;

out_fail:
- __set_page_dirty_nobuffers(page);
+ redirty_page_for_writepage(wbc, page);
unlock_page(page);
return ret;
}
@@ -1396,7 +1396,7 @@ out:
return ret;

no_write:
- __set_page_dirty_nobuffers(page);
+ redirty_page_for_writepage(wbc, page);
out_unlock:
unlock_page(page);
goto out;
diff -puN fs/ntfs/aops.c~writeback-livelock-fix fs/ntfs/aops.c
--- 25/fs/ntfs/aops.c~writeback-livelock-fix 2004-04-20 23:46:12.983431416 -0700
+++ 25-akpm/fs/ntfs/aops.c 2004-04-20 23:46:12.995429592 -0700
@@ -499,10 +499,7 @@ static int ntfs_write_block(struct page
* Put the page back on mapping->dirty_pages, but leave its
* buffer's dirty state as-is.
*/
- // FIXME: Once Andrew's -EAGAIN patch goes in, remove the
- // __set_page_dirty_nobuffers(page) and return -EAGAIN instead
- // of zero.
- __set_page_dirty_nobuffers(page);
+ redirty_page_for_writepage(wbc, page);
unlock_page(page);
return 0;
}
@@ -733,10 +730,7 @@ lock_retry_remap:
* Put the page back on mapping->dirty_pages, but
* leave its buffer's dirty state as-is.
*/
- // FIXME: Once Andrew's -EAGAIN patch goes in, remove
- // the __set_page_dirty_nobuffers(page) and set err to
- // -EAGAIN instead of zero.
- __set_page_dirty_nobuffers(page);
+ redirty_page_for_writepage(wbc, page);
err = 0;
} else
SetPageError(page);
@@ -986,10 +980,7 @@ err_out:
* Put the page back on mapping->dirty_pages, but leave its
* buffer's dirty state as-is.
*/
- // FIXME: Once Andrew's -EAGAIN patch goes in, remove the
- // __set_page_dirty_nobuffers(page) and set err to -EAGAIN
- // instead of zero.
- __set_page_dirty_nobuffers(page);
+ redirty_page_for_writepage(wbc, page);
err = 0;
} else {
ntfs_error(vi->i_sb, "Resident attribute write failed with "
diff -puN fs/reiserfs/inode.c~writeback-livelock-fix fs/reiserfs/inode.c
--- 25/fs/reiserfs/inode.c~writeback-livelock-fix 2004-04-20 23:46:12.985431112 -0700
+++ 25-akpm/fs/reiserfs/inode.c 2004-04-20 23:46:12.993429896 -0700
@@ -2112,7 +2112,7 @@ static int reiserfs_write_full_page(stru
lock_buffer(bh);
} else {
if (test_set_buffer_locked(bh)) {
- __set_page_dirty_nobuffers(page);
+ redirty_page_for_writepage(wbc, page);
continue;
}
}
diff -puN include/linux/mm.h~writeback-livelock-fix include/linux/mm.h
--- 25/include/linux/mm.h~writeback-livelock-fix 2004-04-20 23:46:12.986430960 -0700
+++ 25-akpm/include/linux/mm.h 2004-04-20 23:46:12.992430048 -0700
@@ -499,6 +499,8 @@ int get_user_pages(struct task_struct *t

int __set_page_dirty_buffers(struct page *page);
int __set_page_dirty_nobuffers(struct page *page);
+int redirty_page_for_writepage(struct writeback_control *wbc,
+ struct page *page);
int FASTCALL(set_page_dirty(struct page *page));
int set_page_dirty_lock(struct page *page);
int clear_page_dirty_for_io(struct page *page);
diff -puN mm/page-writeback.c~writeback-livelock-fix mm/page-writeback.c
--- 25/mm/page-writeback.c~writeback-livelock-fix 2004-04-20 23:46:12.988430656 -0700
+++ 25-akpm/mm/page-writeback.c 2004-04-20 23:46:13.001428680 -0700
@@ -581,6 +581,18 @@ int __set_page_dirty_nobuffers(struct pa
EXPORT_SYMBOL(__set_page_dirty_nobuffers);

/*
+ * When a writepage implementation decides that it doesn't want to write this
+ * page for some reason, it should redirty the locked page via
+ * redirty_page_for_writepage() and it should then unlock the page and return 0
+ */
+int redirty_page_for_writepage(struct writeback_control *wbc, struct page *page)
+{
+ wbc->pages_skipped++;
+ return __set_page_dirty_nobuffers(page);
+}
+EXPORT_SYMBOL(redirty_page_for_writepage);
+
+/*
* If the mapping doesn't provide a set_page_dirty a_op, then
* just fall through and assume that it wants buffer_heads.
*/

_


Good luck,
Jurriaan
--
"Bother!" said Pooh as he fished his diskettes out of the honey jar.
Debian (Unstable) GNU/Linux 2.6.6-rc2-mm1 2x6062 bogomips 1.09 1.02