2004-03-19 10:11:37

by Holger Kiehl

[permalink] [raw]
Subject: 2.6.5-rc1-mm2 very slow

Hello

I am testing 2.6.5-rc1-mm2 and find it very slow when I do a bonnie
test, also the system itself feels very sluggish. Looking at dmesg
I get the following:

Badness in elv_remove_request at drivers/block/elevator.c:249
Call Trace:
[<c028b28f>] elv_remove_request+0x8d/0x8f
[<c02b6a4e>] scsi_request_fn+0x289/0x333
[<c028b174>] elv_next_request+0x3d/0xcb
[<c028c8da>] generic_unplug_device+0x43/0x45
[<c02f1bfc>] md_unplug_mddev+0x32/0x34
[<c02f07e4>] make_request+0x22/0x29f
[<c01a5523>] ext3_get_branch+0x4d/0xca
[<c028ddc7>] generic_make_request+0xf2/0x165
[<c0138a41>] mempool_alloc+0x63/0x114
[<c011d42b>] autoremove_wake_function+0x0/0x38
[<c028dea6>] submit_bio+0x6c/0x11b
[<c0154c42>] bio_alloc+0xba/0x182
[<c015355e>] block_read_full_page+0x19d/0x2a2
[<c01a5d77>] ext3_get_block+0x0/0x9e
[<c0154cb3>] bio_alloc+0x12b/0x182
[<c01a5dcb>] ext3_get_block+0x54/0x9e
[<c016d08e>] mpage_alloc+0x25/0x8a
[<c016d3b8>] do_mpage_readpage+0x24b/0x314
[<c0246894>] radix_tree_node_alloc+0x10/0x4d
[<c0246b00>] radix_tree_insert+0xdd/0xfd
[<c0135dfa>] add_to_page_cache+0x5a/0xd5
[<c016d59b>] mpage_readpages+0x11a/0x15e
[<c01a5d77>] ext3_get_block+0x0/0x9e
[<c01a6a59>] ext3_readpages+0x0/0x15
[<c013bb8d>] read_pages+0x136/0x13f
[<c01a5d77>] ext3_get_block+0x0/0x9e
[<c0139a98>] buffered_rmqueue+0xf6/0x1d3
[<c0139c20>] __alloc_pages+0xab/0x2e0
[<c013be48>] do_page_cache_readahead+0xfc/0x165
[<c013c039>] page_cache_readahead+0x188/0x1d7
[<c01364f0>] do_generic_mapping_read+0xc9/0x38e
[<c01367b5>] file_read_actor+0x0/0xca
[<c0136a4c>] __generic_file_aio_read+0x1cd/0x1ff
[<c01367b5>] file_read_actor+0x0/0xca
[<c0136ac5>] generic_file_aio_read+0x47/0x61
[<c015031e>] do_sync_read+0x6f/0xa4
[<c011a214>] load_balance+0x3c/0x156
[<c01503fc>] vfs_read+0xa9/0xf5
[<c0150619>] sys_read+0x38/0x59
[<c034b8a6>] sysenter_past_esp+0x43/0x65

The system is a dual Xeon with HT enabled and 2GB memory. Bonnie is
running on a software raid10 (six SCSI disks) with ext3.

It is compiled with CONFIG_SCHED_SMT and CONFIG_HIGHMEM4G. CONFIG_PREEMPT
is not set. I can also post the full .config if necessary.

I get more of these badness messages when I boot.

Under 2.6.3 there are no problems. Also have tried 2.6.4-mm2 but that would
oops during boot (when starting the SW raid).

Holger


2004-03-19 10:15:02

by Jens Axboe

[permalink] [raw]
Subject: Re: 2.6.5-rc1-mm2 very slow

On Fri, Mar 19 2004, Holger Kiehl wrote:
> Hello
>
> I am testing 2.6.5-rc1-mm2 and find it very slow when I do a bonnie
> test, also the system itself feels very sluggish. Looking at dmesg
> I get the following:
>
> Badness in elv_remove_request at drivers/block/elevator.c:249
> Call Trace:
> [<c028b28f>] elv_remove_request+0x8d/0x8f
> [<c02b6a4e>] scsi_request_fn+0x289/0x333
> [<c028b174>] elv_next_request+0x3d/0xcb
> [<c028c8da>] generic_unplug_device+0x43/0x45

Does it still complain with this patch?

--- drivers/block/elevator.c~ 2004-03-18 10:56:34.494431670 +0100
+++ drivers/block/elevator.c 2004-03-18 11:05:00.115063157 +0100
@@ -153,7 +153,7 @@
* it already went through dequeue, we need to decrement the
* in_flight count again
*/
- if (blk_rq_started(rq)) {
+ if (blk_account_rq(rq)) {
WARN_ON(q->in_flight == 0);
q->in_flight--;
}
@@ -244,7 +244,7 @@
* driver has seen (REQ_STARTED set), to avoid false accounting
* for request-request merges
*/
- if (blk_rq_started(rq)) {
+ if (blk_account_rq(rq)) {
q->in_flight++;
WARN_ON(q->in_flight > 2 * q->nr_requests);
}
@@ -341,7 +341,7 @@
/*
* request is released from the driver, io must be done
*/
- if (blk_rq_started(rq)) {
+ if (blk_account_rq(rq)) {
WARN_ON(q->in_flight == 0);
q->in_flight--;
}
--- include/linux/blkdev.h~ 2004-03-18 11:03:59.431584757 +0100
+++ include/linux/blkdev.h 2004-03-18 11:05:16.980250506 +0100
@@ -381,6 +381,8 @@
#define blk_noretry_request(rq) ((rq)->flags & REQ_FAILFAST)
#define blk_rq_started(rq) ((rq)->flags & REQ_STARTED)

+#define blk_account_rq(rq) (blk_rq_started(rq) && blk_fs_request(rq))
+
#define blk_pm_suspend_request(rq) ((rq)->flags & REQ_PM_SUSPEND)
#define blk_pm_resume_request(rq) ((rq)->flags & REQ_PM_RESUME)
#define blk_pm_request(rq) \


--
Jens Axboe

2004-03-19 11:39:10

by Holger Kiehl

[permalink] [raw]
Subject: Re: 2.6.5-rc1-mm2 very slow

On Fri, 19 Mar 2004, Jens Axboe wrote:

> On Fri, Mar 19 2004, Holger Kiehl wrote:
> > Hello
> >
> > I am testing 2.6.5-rc1-mm2 and find it very slow when I do a bonnie
> > test, also the system itself feels very sluggish. Looking at dmesg
> > I get the following:
> >
> > Badness in elv_remove_request at drivers/block/elevator.c:249
> > Call Trace:
> > [<c028b28f>] elv_remove_request+0x8d/0x8f
> > [<c02b6a4e>] scsi_request_fn+0x289/0x333
> > [<c028b174>] elv_next_request+0x3d/0xcb
> > [<c028c8da>] generic_unplug_device+0x43/0x45
>
> Does it still complain with this patch?
>
No, this also fixes the slowness. Performance is back to normal.

That was the quickest patch I have ever received :)

Many thanks!

Holger