Hi,
using a recent hard-/software setup, I observed that continuous
read/write operations severely degrade the overall system responsiveness
in typical desktop-PC use cases.
Merely doing write/read operations on a data volume leads to stuck text
and mouse cursors, seconds-long delays for simple window-context
switches in X11, dropouts in low-resolution video playback etc.
Test case:
- 64-bit dual-core PC, SATA harddrive, plenty of free RAM
- vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit
How to reproduce:
- start KDE/GNOME-session
- open a terminal window and do as a non-root user:
dd if=/dev/zero of=/home/john-doe/testfile
(or dd if=/home/john-doe/big-testfile of=/dev/null)
- a real use scenario would be a daily disk-backup or the
simple extraction of a tarball containing slightly bigger files
Observation:
- The system becomes _really_ slow as described above; unusable for
any multimedia tasks.
- Using an encrypted (dm-crypt/LUKS) /home (e.g. on mobile computers)
compounds the issue to a painful extent.
Possible culprits (I'm guessing) are the Linux I/O- or CPU scheduler.
I realize that a single heavyweight transfer slows down I/O for the
corresponding transactions/processes/volumes etc.
But there needs to be a fair distribution of I/O or CPU time which
leaves enough for other basic operations. And this doesn't seem to be
the case with recent Linux versions.
On a side note, I've tried the BFS-patches (bfs-221 on 2.6.31): This
yields significantly higher throughput when using disk encryption (50%
improvement with dm-crypt/LUKS, 512 bit aes-xts-plain cipher mode). But
with these patches, the responsiveness was even worse during my quick
test. Switching to a text-mode console: several /minutes/ delay...
I'm attaching my .config for linux 2.6.31 (grep ^C and bzip2-ed)
On Sat September 19 2009, Ulrich Lukas wrote:
> Hi,
>
>
> using a recent hard-/software setup, I observed that continuous
> read/write operations severely degrade the overall system responsiveness
> in typical desktop-PC use cases.
>
> Merely doing write/read operations on a data volume leads to stuck text
> and mouse cursors, seconds-long delays for simple window-context
> switches in X11, dropouts in low-resolution video playback etc.
>
>
>
> Test case:
> - 64-bit dual-core PC, SATA harddrive, plenty of free RAM
> - vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit
>
>
> How to reproduce:
> - start KDE/GNOME-session
> - open a terminal window and do as a non-root user:
> dd if=/dev/zero of=/home/john-doe/testfile
> (or dd if=/home/john-doe/big-testfile of=/dev/null)
>
> - a real use scenario would be a daily disk-backup or the
> simple extraction of a tarball containing slightly bigger files
>
>
> Observation:
> - The system becomes _really_ slow as described above; unusable for
> any multimedia tasks.
>
> - Using an encrypted (dm-crypt/LUKS) /home (e.g. on mobile computers)
> compounds the issue to a painful extent.
>
>
>
> Possible culprits (I'm guessing) are the Linux I/O- or CPU scheduler.
>
> I realize that a single heavyweight transfer slows down I/O for the
> corresponding transactions/processes/volumes etc.
>
> But there needs to be a fair distribution of I/O or CPU time which
> leaves enough for other basic operations. And this doesn't seem to be
> the case with recent Linux versions.
>
>
>
>
> On a side note, I've tried the BFS-patches (bfs-221 on 2.6.31): This
> yields significantly higher throughput when using disk encryption (50%
> improvement with dm-crypt/LUKS, 512 bit aes-xts-plain cipher mode). But
> with these patches, the responsiveness was even worse during my quick
> test. Switching to a text-mode console: several /minutes/ delay...
>
>
> I'm attaching my .config for linux 2.6.31 (grep ^C and bzip2-ed)
>
I've seen similar things with my Intel p35 (q6600) system, especially under
64bit. Heavy i/o makes the rest of the system perform very badly. I have
however not noticed nearly as much of a problem with my AMD (SB700&SB750)
systems.
--
Thomas Fjellstrom
[email protected]
On Sun, 20 Sep 2009 05:08:43 +0200
Ulrich Lukas <[email protected]> wrote:
>
> Observation:
> - The system becomes _really_ slow as described above; unusable for
> any multimedia tasks.
>
> - Using an encrypted (dm-crypt/LUKS) /home (e.g. on mobile computers)
> compounds the issue to a painful extent.
>
>
>
> Possible culprits (I'm guessing) are the Linux I/O- or CPU scheduler.
this has nothing to do with the cpu scheduler....
can you try the following two things?
echo 4096 > /sys/block/sda/queue/nr_requests
for i in `pidof kjournald` ; do ionice -c1 -p $i ; done
to see if they help?
they tweak the IO scheduler... I assume you're using CFQ.
Arjan van de Ven wrote:
> can you try the following two things?
>
> echo 4096 > /sys/block/sda/queue/nr_requests
> for i in `pidof kjournald` ; do ionice -c1 -p $i ; done
I tried this just now, but it doesn't change things much. I can do a
more testing later today.
> I assume you're using CFQ.
Yes, it's CFQ. (see .config of my previous posting)
Filesystem is EXT4.
Arjan van de Ven wrote:
> > can you try the following two things?
> >
> > echo 4096 > /sys/block/sda/queue/nr_requests
> > for i in `pidof kjournald` ; do ionice -c1 -p $i ; done
No, just tested again. Only a marginal improvement if there is any.
(Please don't strip the CC list when you reply. Use the "Reply to all"
functionality of your mail client and keep all CCs.)
On 09/20/2009 11:50 AM, Ulrich Lukas wrote:
> Arjan van de Ven wrote:
>> can you try the following two things?
>>
>> echo 4096> /sys/block/sda/queue/nr_requests
>> for i in `pidof kjournald` ; do ionice -c1 -p $i ; done
>
> I tried this just now, but it doesn't change things much. I can do a
> more testing later today.
>
>
>> I assume you're using CFQ.
>
> Yes, it's CFQ. (see .config of my previous posting)
>
> Filesystem is EXT4.
I can also confirm total lock-up when copying a big file (Intel Core 2
Duo, Intel X38 chipset). Everything locks-up for several seconds during
the copying. I'm also on ext4. I've been told on various mailing lists
that this is normal, expected and unavoidable so I pretty much think the
only solution is to avoid heavy I/O :P
Arjan van de Ven <[email protected]> wrote:
> can you try the following two things?
>
> echo 4096 > /sys/block/sda/queue/nr_requests
> for i in `pidof kjournald` ; do ionice -c1 -p $i ; done
>
> to see if they help?
That helped here. The hardware is a Thinkpad T60 with 1.5GB RAM and
this hard drive:
ata1.00: ATA-7: Hitachi HTE541616J9SA00, SB4OC7KP, max UDMA/100
ata1.00: 312581808 sectors, multi 16: LBA48 NCQ (depth 31/32)
When the 'locate' daily cron job runs, or I run 'aptitude
dist-upgrade' (after all packages are downloaded), or I back up /home
to a USB hard drive using rsync, the desktop response badly degrades.
I'm not using a fancy desktop---just twm. An rxvt can take up to 10
seconds to start up, and I often see emacs repainting its window when it
comes to the foreground. Using your settings, rxvt takes about 1 second
to start up, which is a large improvement.
(The kernel is 2.6.30 with debian patches)
-Sanjoy
On Sun, 2009-09-20 at 10:50 +0200, Ulrich Lukas wrote:
> Arjan van de Ven wrote:
> > can you try the following two things?
> >
> > echo 4096 > /sys/block/sda/queue/nr_requests
> > for i in `pidof kjournald` ; do ionice -c1 -p $i ; done
>
> I tried this just now, but it doesn't change things much. I can do a
> more testing later today.
Try ionice -c3 for your dd.
> > I assume you're using CFQ.
>
> Yes, it's CFQ. (see .config of my previous posting)
>
> Filesystem is EXT4.
Mine is ext3.
I can easily reproduce pain. The attached cheezy test script, when run
back to back with no args (stock IO), regularly produces beauties:
Performance counter stats for 'sh -c konsole -e exit':
257.670601 task-clock-msecs # 0.004 CPUs
1036 context-switches # 0.004 M/sec
30 CPU-migrations # 0.000 M/sec
6317 page-faults # 0.025 M/sec
471734719 cycles # 1830.767 M/sec
393365060 instructions # 0.834 IPC
7849464 cache-references # 30.463 M/sec
1038976 cache-misses # 4.032 M/sec
65.766937233 seconds time elapsed
Yup, a full minute to get a binary loaded and fired up. Seems it and
others get evicted by the VM. Getting whatever back into ram is painful
to listen to (poor disk/me.. seek latency eating MUCH time).
Calling script back to back with args 1 (realtime foreground) 3 (idle
background) does best:
Performance counter stats for 'sh -c konsole -e exit':
169.915036 task-clock-msecs # 0.077 CPUs
439 context-switches # 0.003 M/sec
9 CPU-migrations # 0.000 M/sec
6314 page-faults # 0.037 M/sec
363128252 cycles # 2137.117 M/sec
367410945 instructions # 1.012 IPC
5822302 cache-references # 34.266 M/sec
437189 cache-misses # 2.573 M/sec
2.213079979 seconds time elapsed
Args 2 (best effort foreground) and 3 (idle background);
Performance counter stats for 'sh -c konsole -e exit':
176.689856 task-clock-msecs # 0.015 CPUs
357 context-switches # 0.002 M/sec
15 CPU-migrations # 0.000 M/sec
6314 page-faults # 0.036 M/sec
374861947 cycles # 2121.582 M/sec
368929476 instructions # 0.984 IPC
5806286 cache-references # 32.861 M/sec
496651 cache-misses # 2.811 M/sec
12.036165968 seconds time elapsed
-Mike
On Sun, 20 Sep 2009, Ulrich Lukas wrote:
> Test case:
> - 64-bit dual-core PC, SATA harddrive, plenty of free RAM
> - vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit
>
>
> How to reproduce:
> - start KDE/GNOME-session
> - open a terminal window and do as a non-root user:
> dd if=/dev/zero of=/home/john-doe/testfile
> (or dd if=/home/john-doe/big-testfile of=/dev/null)
>
> - a real use scenario would be a daily disk-backup or the
> simple extraction of a tarball containing slightly bigger files
>
>
> Observation:
> - The system becomes _really_ slow as described above; unusable for
> any multimedia tasks.
I guess that switching from CFQ to deadline I/O scheduler improves the
situation, right?
--
Jiri Kosina
SUSE Labs, Novell Inc.
On Sun 20-09-09 22:22:20, Jiri Kosina wrote:
> On Sun, 20 Sep 2009, Ulrich Lukas wrote:
>
> > Test case:
> > - 64-bit dual-core PC, SATA harddrive, plenty of free RAM
> > - vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit
> >
> >
> > How to reproduce:
> > - start KDE/GNOME-session
> > - open a terminal window and do as a non-root user:
> > dd if=/dev/zero of=/home/john-doe/testfile
> > (or dd if=/home/john-doe/big-testfile of=/dev/null)
> >
> > - a real use scenario would be a daily disk-backup or the
> > simple extraction of a tarball containing slightly bigger files
> >
> >
> > Observation:
> > - The system becomes _really_ slow as described above; unusable for
> > any multimedia tasks.
>
> I guess that switching from CFQ to deadline I/O scheduler improves the
> situation, right?
For example on my desktop, switching to deadline slightly improves the
situation but the machine is still mostly unusable while dd is running...
I'll try to debug it some more to see whether it can be somehow helped.
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR
Mike Galbraith wrote:
> On Sun, 2009-09-20 at 10:50 +0200, Ulrich Lukas wrote:
>
>> Arjan van de Ven wrote:
>>
>>> can you try the following two things?
>>>
>>> echo 4096> /sys/block/sda/queue/nr_requests
>>> for i in `pidof kjournald` ; do ionice -c1 -p $i ; done
>>>
>> I tried this just now, but it doesn't change things much. I can do a
>> more testing later today.
>>
>
> Try ionice -c3 for your dd.
>
>
>>> I assume you're using CFQ.
>>>
>> Yes, it's CFQ. (see .config of my previous posting)
>>
>> Filesystem is EXT4.
>>
>
> Mine is ext3.
>
> I can easily reproduce pain. The attached cheezy test script, when run
> back to back with no args (stock IO), regularly produces beauties:
>
> Performance counter stats for 'sh -c konsole -e exit':
>
> 257.670601 task-clock-msecs # 0.004 CPUs
> 1036 context-switches # 0.004 M/sec
> 30 CPU-migrations # 0.000 M/sec
> 6317 page-faults # 0.025 M/sec
> 471734719 cycles # 1830.767 M/sec
> 393365060 instructions # 0.834 IPC
> 7849464 cache-references # 30.463 M/sec
> 1038976 cache-misses # 4.032 M/sec
>
> 65.766937233 seconds time elapsed
>
> Yup, a full minute to get a binary loaded and fired up. Seems it and
> others get evicted by the VM. Getting whatever back into ram is painful
> to listen to (poor disk/me.. seek latency eating MUCH time).
>
> Calling script back to back with args 1 (realtime foreground) 3 (idle
> background) does best:
>
> Performance counter stats for 'sh -c konsole -e exit':
>
> 169.915036 task-clock-msecs # 0.077 CPUs
> 439 context-switches # 0.003 M/sec
> 9 CPU-migrations # 0.000 M/sec
> 6314 page-faults # 0.037 M/sec
> 363128252 cycles # 2137.117 M/sec
> 367410945 instructions # 1.012 IPC
> 5822302 cache-references # 34.266 M/sec
> 437189 cache-misses # 2.573 M/sec
>
> 2.213079979 seconds time elapsed
>
> Args 2 (best effort foreground) and 3 (idle background);
>
> Performance counter stats for 'sh -c konsole -e exit':
>
> 176.689856 task-clock-msecs # 0.015 CPUs
> 357 context-switches # 0.002 M/sec
> 15 CPU-migrations # 0.000 M/sec
> 6314 page-faults # 0.036 M/sec
> 374861947 cycles # 2121.582 M/sec
> 368929476 instructions # 0.984 IPC
> 5806286 cache-references # 32.861 M/sec
> 496651 cache-misses # 2.811 M/sec
>
> 12.036165968 seconds time elapsed
>
> -Mike
>
Wow, at first I did not notice anything
from your *.sh script, but then a few moments
later, the system became vary sluggish. firefox
takes about 15 seconds to load. OS is mint linux
with the latest HEAD on an imac9,1.
Justin P. Mattock
Jiri Kosina wrote:
> I guess that switching from CFQ to deadline I/O scheduler improves the
> situation, right?
Indeed, there is an improvement. But, especially in the case with an
encrypted /home, copying a large file from a fast source means I better
go get another cup of coffee rather than continue working on something
else in the meantime.
BTW, I've tested this with a default install of
OpenSUSE 11.1, and I get basically the same behaviour.
On Sun, 2009-09-20 at 17:22 -0700, Justin P. Mattock wrote:
> Wow, at first I did not notice anything
> from your *.sh script, but then a few moments
> later, the system became vary sluggish. firefox
> takes about 15 seconds to load. OS is mint linux
> with the latest HEAD on an imac9,1.
CFQ inherits IO priority from the parent on fork(), so nicing a shell or
the dd should (and does) help a LOT. It also inherits the CPU scheduler
class. I just checked, and IO does indeed inherit SCHED_IDLE as well as
the RT classes (SCHED_BATCH does nothing). Setting the dd to SCHED_IDLE
makes a big difference, as it should, so there's a bigger hammer than
nice + ionice. SCHED_IDLE is CPU_adorable + IO_nicest.
Anyway, the problem seems to have to have at least two facets:
1. streaming IO causing pagecache reclaim. That problem doesn't seem
to want to go away. Fixing that won't help 2 one bit though.
2. reads are sync, more heavily affected by seek latency than writes.
If the disk your binaries live on is seeking a lot, you pay a LOT.
Hohum, IO + VM = haaaaard ;-)
-Mike
On Mon, 2009-09-21 at 00:04 +0200, Jan Kara wrote:
> On Sun 20-09-09 22:22:20, Jiri Kosina wrote:
> > On Sun, 20 Sep 2009, Ulrich Lukas wrote:
> >
> > > Test case:
> > > - 64-bit dual-core PC, SATA harddrive, plenty of free RAM
> > > - vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit
> > >
> > >
> > > How to reproduce:
> > > - start KDE/GNOME-session
> > > - open a terminal window and do as a non-root user:
> > > dd if=/dev/zero of=/home/john-doe/testfile
> > > (or dd if=/home/john-doe/big-testfile of=/dev/null)
> > >
> > > - a real use scenario would be a daily disk-backup or the
> > > simple extraction of a tarball containing slightly bigger files
> > >
> > >
> > > Observation:
> > > - The system becomes _really_ slow as described above; unusable for
> > > any multimedia tasks.
> >
> > I guess that switching from CFQ to deadline I/O scheduler improves the
> > situation, right?
> For example on my desktop, switching to deadline slightly improves the
> situation but the machine is still mostly unusable while dd is running...
> I'll try to debug it some more to see whether it can be somehow helped.
Datapoint:
echo 1 > /sys/block/$disk/queue/iosched/quantum seems to make a huge
difference. Stock is 4. (wisdom of setting it to 1? no idea)
This...
marge:/root/tmp # ./testo.sh
quantum is 4, setting to 1
dd if=3DMark2000.mkv of=/tmp/3DMark2000.mkv
204529+1 records in
204529+1 records out
104718895 bytes (105 MB) copied, 1.62672 s, 64.4 MB/s
perf sched record -o /tmp/mplayer-c2-c2.data (tmpfs)&
mplayer /tmp/3DMark2000.mkv&
dd if=/dev/zero of=crud-25859&
sleep 180
timing konsole -e exit
Performance counter stats for 'sh -c konsole -e exit':
173.615837 task-clock-msecs # 0.031 CPUs
431 context-switches # 0.002 M/sec
10 CPU-migrations # 0.000 M/sec
6314 page-faults # 0.036 M/sec
368422027 cycles # 2122.053 M/sec
376283291 instructions # 1.021 IPC
5883195 cache-references # 33.886 M/sec
444333 cache-misses # 2.559 M/sec
5.597628358 seconds time elapsed
...is pretty typical, and a LOT better than the 60-70 seconds I was
seeing, or the minutes I sometimes saw when playing with evolution.
I still occasionally see some largish numbers though, and suspect that
depends on how much cache was evicted.
-Mike
On Mon, 21 Sep 2009 09:25:08 +0200
Mike Galbraith <[email protected]> wrote:
> I still occasionally see some largish numbers though, and suspect that
> depends on how much cache was evicted.
if you run latencytop it'll at least tell you what kind of operating
was going on that hit the delay...
--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Mon, 2009-09-21 at 09:33 +0200, Arjan van de Ven wrote:
> On Mon, 21 Sep 2009 09:25:08 +0200
> Mike Galbraith <[email protected]> wrote:
> > I still occasionally see some largish numbers though, and suspect that
> > depends on how much cache was evicted.
>
> if you run latencytop it'll at least tell you what kind of operating
> was going on that hit the delay...
One of the really bad ones for evolution is below. It was so bad that I
had _plenty_ of time to get to a GUI free shell, and poke SysRq-W before
killing the darn dd.
[ 5112.427012] SysRq : Show Blocked State
[ 5112.430995] task PC stack pid father
[ 5112.430995] kjournald D 0000000000000002 0 1230 2 0x00000000
[ 5112.430995] ffff8800378d7c50 0000000000000046 0000000000000000 ffff8800a5ed53b0
[ 5112.430995] ffff8800bd2c0000 ffff8800378d7fd8 000000000000dd38 0000000000013480
[ 5112.430995] 0000000000004000 0000000000013480 ffff8800378d7bf0 00001000810e0c0a
[ 5112.430995] Call Trace:
[ 5112.430995] [<ffffffff812cacb4>] io_schedule+0x80/0xc6
[ 5112.430995] [<ffffffff8110df99>] sync_buffer+0x40/0x44
[ 5112.430995] [<ffffffff812cb373>] __wait_on_bit+0x4c/0x7e
[ 5112.430995] [<ffffffff8110df59>] ? sync_buffer+0x0/0x44
[ 5112.430995] [<ffffffff8110df59>] ? sync_buffer+0x0/0x44
[ 5112.430995] [<ffffffff812cb414>] out_of_line_wait_on_bit+0x6f/0x7c
[ 5112.430995] [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.430995] [<ffffffff8110deff>] __wait_on_buffer+0x24/0x26
[ 5112.430995] [<ffffffffa0077e91>] journal_commit_transaction+0xca4/0x138b [jbd]
[ 5112.430995] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.430995] [<ffffffffa007ba02>] kjournald+0x168/0x397 [jbd]
[ 5112.430995] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.430995] [<ffffffffa007b89a>] ? kjournald+0x0/0x397 [jbd]
[ 5112.430995] [<ffffffff81065acb>] kthread+0x82/0x8a
[ 5112.625071] [<ffffffff8100ccca>] child_rip+0xa/0x20
[ 5112.625071] [<ffffffff8100c669>] ? restore_args+0x0/0x30
[ 5112.625071] [<ffffffff81065a49>] ? kthread+0x0/0x8a
[ 5112.625071] [<ffffffff8100ccc0>] ? child_rip+0x0/0x20
[ 5112.625071] flush-8:16 D ffff8800bd2097f0 0 1236 2 0x00000000
[ 5112.625071] ffff8800bd209790 0000000000000046 0000000000000000 ffffffff810beb58
[ 5112.625071] ffff8800bd209740 ffff8800bd209fd8 000000000000dd38 0000000000013480
[ 5112.625071] 0000000000004000 0000000000013480 0000000001080001 0000000000000080
[ 5112.625071] Call Trace:
[ 5112.625071] [<ffffffff810beb58>] ? get_page_from_freelist+0x33d/0x4a6
[ 5112.625071] [<ffffffff81174cb6>] ? cfq_may_queue+0x60/0xe1
[ 5112.625071] [<ffffffff812cacb4>] io_schedule+0x80/0xc6
[ 5112.625071] [<ffffffff81169184>] get_request_wait+0xf3/0x18f
[ 5112.625071] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071] [<ffffffff8115f142>] ? elv_merge+0x37/0x19d
[ 5112.625071] [<ffffffff8116954d>] __make_request+0x32d/0x45b
[ 5112.625071] [<ffffffff811678ae>] generic_make_request+0x339/0x38b
[ 5112.625071] [<ffffffff811679d4>] submit_bio+0xd4/0xdd
[ 5112.625071] [<ffffffff8110ccfc>] submit_bh+0x106/0x129
[ 5112.625071] [<ffffffff8110f7a4>] __block_write_full_page+0x1eb/0x2d1
[ 5112.625071] [<ffffffff8110e197>] ? end_buffer_async_write+0x0/0x13f
[ 5112.625071] [<ffffffff8110f94f>] block_write_full_page_endio+0xc5/0xca
[ 5112.625071] [<ffffffff8110f969>] block_write_full_page+0x15/0x17
[ 5112.625071] [<ffffffffa0095e1f>] ext3_writeback_writepage+0x71/0x142 [ext3]
[ 5112.625071] [<ffffffff810bf647>] __writepage+0x17/0x34
[ 5112.625071] [<ffffffff810bfd9f>] write_cache_pages+0x250/0x393
[ 5112.625071] [<ffffffff810bf630>] ? __writepage+0x0/0x34
[ 5112.625071] [<ffffffff810bff06>] generic_writepages+0x24/0x2a
[ 5112.625071] [<ffffffff810bff34>] do_writepages+0x28/0x2a
[ 5112.625071] [<ffffffff81106b94>] writeback_single_inode+0x181/0x395
[ 5112.625071] [<ffffffff811077fc>] writeback_inodes_wb+0x341/0x42f
[ 5112.625071] [<ffffffff81107a2b>] wb_writeback+0x141/0x194
[ 5112.625071] [<ffffffff81095031>] ? call_rcu_sched+0x15/0x17
[ 5112.625071] [<ffffffff81095041>] ? call_rcu+0xe/0x10
[ 5112.625071] [<ffffffff81107bb9>] wb_do_writeback+0x7a/0x16b
[ 5112.625071] [<ffffffff81107ce9>] bdi_writeback_task+0x3f/0xad
[ 5112.625071] [<ffffffff810cca5d>] ? bdi_start_fn+0x0/0xd7
[ 5112.625071] [<ffffffff810ccad3>] bdi_start_fn+0x76/0xd7
[ 5112.625071] [<ffffffff810cca5d>] ? bdi_start_fn+0x0/0xd7
[ 5112.625071] [<ffffffff81065acb>] kthread+0x82/0x8a
[ 5112.625071] [<ffffffff8100ccca>] child_rip+0xa/0x20
[ 5112.625071] [<ffffffff8100c669>] ? restore_args+0x0/0x30
[ 5112.625071] [<ffffffff81065a49>] ? kthread+0x0/0x8a
[ 5112.625071] [<ffffffff8100ccc0>] ? child_rip+0x0/0x20
[ 5112.625071] console-kit-d D ffff8800b788c480 0 2963 1 0x00000000
[ 5112.625071] ffff8800bc59f6b8 0000000000000082 0000000000000000 ffffffffa0076655
[ 5112.625071] ffff8800bc59f628 ffff8800bc59ffd8 000000000000dd38 0000000000013480
[ 5112.625071] 0000000000004000 0000000000013480 ffff8800bd2f3c00 0000000000000000
[ 5112.625071] Call Trace:
[ 5112.625071] [<ffffffffa0076655>] ? do_get_write_access+0x443/0x48b [jbd]
[ 5112.625071] [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd]
[ 5112.625071] [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071] [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd]
[ 5112.625071] [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3]
[ 5112.625071] [<ffffffffa0092901>] ext3_new_blocks+0x333/0x669 [ext3]
[ 5112.625071] [<ffffffffa0096654>] ext3_get_blocks_handle+0x3ec/0x92d [ext3]
[ 5112.625071] [<ffffffff8110c866>] ? alloc_buffer_head+0x1e/0x51
[ 5112.625071] [<ffffffffa0096c3a>] ext3_get_block+0xa5/0xe3 [ext3]
[ 5112.625071] [<ffffffff8110ec0a>] __block_prepare_write+0x1f0/0x3d2
[ 5112.625071] [<ffffffffa0076e49>] ? start_this_handle+0x37d/0x474 [jbd]
[ 5112.625071] [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071] [<ffffffffa007709a>] ? journal_start+0x60/0xca [jbd]
[ 5112.625071] [<ffffffff8110ef7c>] block_write_begin+0x85/0xd5
[ 5112.625071] [<ffffffffa0098130>] ext3_write_begin+0xf0/0x1e7 [ext3]
[ 5112.625071] [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071] [<ffffffff810ba9e8>] generic_file_buffered_write+0x11f/0x277
[ 5112.625071] [<ffffffff810bafe1>] __generic_file_aio_write+0x351/0x385
[ 5112.625071] [<ffffffff810bb07b>] generic_file_aio_write+0x66/0xad
[ 5112.625071] [<ffffffff810e75b4>] do_sync_write+0xec/0x132
[ 5112.625071] [<ffffffff810c176a>] ? lru_cache_add_lru+0x2b/0x2d
[ 5112.625071] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071] [<ffffffff8117c94f>] ? __up_read+0x9e/0xa7
[ 5112.625071] [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18
[ 5112.625071] [<ffffffff810e8053>] vfs_write+0xb3/0x16f
[ 5112.625071] [<ffffffff810e81dd>] sys_write+0x4c/0x74
[ 5112.625071] [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
[ 5112.625071] dd D ffff8800b788c480 0 8587 6686 0x00000000
[ 5112.625071] ffff8800a5e2faa8 0000000000000086 0000000000000000 ffffffff812cab8c
[ 5112.625071] ffff8800a5e2fbe0 ffff8800a5e2ffd8 000000000000dd38 0000000000013480
[ 5112.625071] 0000000000004000 0000000000013480 ffff8800a5e2fa28 ffffffff810b9b1c
[ 5112.625071] Call Trace:
[ 5112.625071] [<ffffffff812cab8c>] ? thread_return+0x60/0x108
[ 5112.625071] [<ffffffff810b9b1c>] ? find_get_page+0x28/0x89
[ 5112.625071] [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd]
[ 5112.625071] [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071] [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd]
[ 5112.625071] [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3]
[ 5112.625071] [<ffffffffa00953f5>] ext3_reserve_inode_write+0x44/0x80 [ext3]
[ 5112.625071] [<ffffffffa0095460>] ext3_mark_inode_dirty+0x2f/0x4c [ext3]
[ 5112.625071] [<ffffffffa0095605>] ext3_dirty_inode+0xab/0xc2 [ext3]
[ 5112.625071] [<ffffffff81107392>] __mark_inode_dirty+0x34/0x15d
[ 5112.625071] [<ffffffff810fc6af>] file_update_time+0xbe/0x102
[ 5112.625071] [<ffffffff810baef7>] __generic_file_aio_write+0x267/0x385
[ 5112.625071] [<ffffffff810b8427>] ? perf_swcounter_ctx_event+0x16c/0x1c6
[ 5112.625071] [<ffffffff810bb07b>] generic_file_aio_write+0x66/0xad
[ 5112.625071] [<ffffffff810e75b4>] do_sync_write+0xec/0x132
[ 5112.625071] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071] [<ffffffff8117ffa0>] ? __clear_user+0x17/0x55
[ 5112.625071] [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18
[ 5112.625071] [<ffffffff810e8053>] vfs_write+0xb3/0x16f
[ 5112.625071] [<ffffffff810e81dd>] sys_write+0x4c/0x74
[ 5112.625071] [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
[ 5112.625071] perf D ffff8800b788c480 0 8588 6686 0x00000000
[ 5112.625071] ffff8800a5f115d8 0000000000000082 0000000000000000 000004a57bce3b78
[ 5112.625071] ffff8800be035f40 ffff8800a5f11fd8 000000000000dd38 0000000000013480
[ 5112.625071] 0000000000004000 0000000000013480 ffff8800a5f11558 ffffffff810b9b1c
[ 5112.625071] Call Trace:
[ 5112.625071] [<ffffffff810b9b1c>] ? find_get_page+0x28/0x89
[ 5112.625071] [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd]
[ 5112.625071] [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071] [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd]
[ 5112.625071] [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3]
[ 5112.625071] [<ffffffffa00953f5>] ext3_reserve_inode_write+0x44/0x80 [ext3]
[ 5112.625071] [<ffffffffa0095460>] ext3_mark_inode_dirty+0x2f/0x4c [ext3]
[ 5112.625071] [<ffffffffa0095605>] ext3_dirty_inode+0xab/0xc2 [ext3]
[ 5112.625071] [<ffffffff81107392>] __mark_inode_dirty+0x34/0x15d
[ 5112.625071] [<ffffffffa0092672>] ext3_new_blocks+0xa4/0x669 [ext3]
[ 5112.625071] [<ffffffff81110195>] ? __bread+0x13/0x92
[ 5112.625071] [<ffffffffa00959c4>] ? ext3_get_branch+0x7b/0xee [ext3]
[ 5112.625071] [<ffffffffa0096654>] ext3_get_blocks_handle+0x3ec/0x92d [ext3]
[ 5112.625071] [<ffffffffa0096c3a>] ext3_get_block+0xa5/0xe3 [ext3]
[ 5112.625071] [<ffffffff8110ec0a>] __block_prepare_write+0x1f0/0x3d2
[ 5112.625071] [<ffffffffa0076e49>] ? start_this_handle+0x37d/0x474 [jbd]
[ 5112.625071] [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071] [<ffffffff8110ef7c>] block_write_begin+0x85/0xd5
[ 5112.625071] [<ffffffffa0098130>] ext3_write_begin+0xf0/0x1e7 [ext3]
[ 5112.625071] [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071] [<ffffffff810ba9e8>] generic_file_buffered_write+0x11f/0x277
[ 5112.625071] [<ffffffff810bafe1>] __generic_file_aio_write+0x351/0x385
[ 5112.625071] [<ffffffff8100c669>] ? restore_args+0x0/0x30
[ 5112.625071] [<ffffffff812cb922>] ? mutex_lock+0x1c/0x3b
[ 5112.625071] [<ffffffff810bb07b>] generic_file_aio_write+0x66/0xad
[ 5112.625071] [<ffffffff810e75b4>] do_sync_write+0xec/0x132
[ 5112.625071] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071] [<ffffffff812cab6a>] ? thread_return+0x3e/0x108
[ 5112.625071] [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18
[ 5112.625071] [<ffffffff810e8053>] vfs_write+0xb3/0x16f
[ 5112.625071] [<ffffffff810e81dd>] sys_write+0x4c/0x74
[ 5112.625071] [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
[ 5112.625071] evolution.bin D 0000000000000082 0 8638 6627 0x00000000
[ 5112.625071] ffff880099339c08 0000000000000086 0000000000000000 ffffffff811157a6
[ 5112.625071] ffffffffa0096b95 ffff880099339fd8 000000000000dd38 0000000000013480
[ 5112.625071] 0000000000004000 0000000000013480 ffffea0001c5ae08 00000000009ff29b
[ 5112.625071] Call Trace:
[ 5112.625071] [<ffffffff811157a6>] ? mpage_readpages+0x10a/0x120
[ 5112.625071] [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071] [<ffffffff812cacb4>] io_schedule+0x80/0xc6
[ 5112.625071] [<ffffffff810b9d22>] sync_page+0x46/0x4a
[ 5112.625071] [<ffffffff810b9d34>] sync_page_killable+0xe/0x35
[ 5112.625071] [<ffffffff812cb263>] __wait_on_bit_lock+0x4a/0x92
[ 5112.625071] [<ffffffff810b9d26>] ? sync_page_killable+0x0/0x35
[ 5112.625071] [<ffffffff810b9c54>] __lock_page_killable+0x68/0x6f
[ 5112.625071] [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071] [<ffffffff810bb480>] generic_file_aio_read+0x3be/0x5bb
[ 5112.625071] [<ffffffff810e76e6>] do_sync_read+0xec/0x132
[ 5112.625071] [<ffffffff810c176a>] ? lru_cache_add_lru+0x2b/0x2d
[ 5112.625071] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071] [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18
[ 5112.625071] [<ffffffff810e82b5>] vfs_read+0xb0/0x16c
[ 5112.625071] [<ffffffff810e843f>] sys_read+0x4c/0x75
[ 5112.625071] [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
[ 5112.625071] hal-acl-tool D ffff8800b788c480 0 8675 2964 0x00000000
[ 5112.625071] ffff880037b3db78 0000000000000082 ffff88007af907e0 ffff880062b7f7e0
[ 5112.625071] ffff880037b3dac8 ffff880037b3dfd8 000000000000dd38 0000000000013480
[ 5112.625071] 0000000000004000 0000000000013480 ffff880037b3db18 ffffffff8110d414
[ 5112.625071] Call Trace:
[ 5112.625071] [<ffffffff8110d414>] ? __getblk+0x2d/0x2ae
[ 5112.625071] [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd]
[ 5112.625071] [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071] [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd]
[ 5112.625071] [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3]
[ 5112.625071] [<ffffffffa0093ef8>] ext3_new_inode+0x4d4/0x96d [ext3]
[ 5112.625071] [<ffffffffa009afca>] ext3_create+0x76/0xf3 [ext3]
[ 5112.625071] [<ffffffff810f2be9>] vfs_create+0xa2/0x117
[ 5112.625071] [<ffffffff810f50fb>] do_filp_open+0x327/0xa39
[ 5112.625071] [<ffffffff810f2e69>] ? getname+0x31/0x1b3
[ 5112.625071] [<ffffffff810e513e>] do_sys_open+0x62/0x109
[ 5112.625071] [<ffffffff810e5218>] sys_open+0x20/0x22
[ 5112.625071] [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
On Mon, 2009-09-21 at 09:41 +0200, Mike Galbraith wrote:
> On Mon, 2009-09-21 at 09:33 +0200, Arjan van de Ven wrote:
> > On Mon, 21 Sep 2009 09:25:08 +0200
> > Mike Galbraith <[email protected]> wrote:
> > > I still occasionally see some largish numbers though, and suspect that
> > > depends on how much cache was evicted.
> >
> > if you run latencytop it'll at least tell you what kind of operating
> > was going on that hit the delay...
>
> One of the really bad ones for evolution is below. It was so bad that I
> had _plenty_ of time to get to a GUI free shell, and poke SysRq-W before
> killing the darn dd.
P.S. evolution is always journal journal and more journal.
Hi and thanks for your reply!
Mike Galbraith wrote:
> nicing a shell or the dd should (and does) help a LOT.
If this is the only way to influence this, maybe the default settings
for the niceness of interactive and non-interactive tasks are not the
best choice. (Maybe a distribution problem in this case)
> reads are sync, more heavily affected by seek latency than writes.
But how does this explain the seconds-long delays?
If an interactive process causes a lot of seeks because of reads/writes
which "are sync", I see how this can greatly slow down otherwise
pipelined write operations, but the other way around?
On Mon, 2009-09-21 at 09:48 +0200, Ulrich Lukas wrote:
> Hi and thanks for your reply!
Hi.
> Mike Galbraith wrote:
> > nicing a shell or the dd should (and does) help a LOT.
>
> If this is the only way to influence this, maybe the default settings
> for the niceness of interactive and non-interactive tasks are not the
> best choice. (Maybe a distribution problem in this case)
There is no knowledge in the CPU nor IO scheduler wrt interactive vs
non-interactive. I've tinkered many times with a SCHED_INTERACTIVE
class, but it's not at all an easy problem, so keeps landing on the
trash heap. I could ramble on _for ever_ about that subject, but it's
thankfully irrelevant to this thread ;-)
> > reads are sync, more heavily affected by seek latency than writes.
>
> But how does this explain the seconds-long delays?
Seek latencies are cumulative is my (wild arsed) theory.
> If an interactive process causes a lot of seeks because of reads/writes
> which "are sync", I see how this can greatly slow down otherwise
> pipelined write operations, but the other way around?
If you seek between tiny reads, the more little reads you do, the more
seeks hurt. Readahead is supposed to help, but.. the pain is there.
-Mike
FWIW, I did not notice any problems with the last kernel I had compiled
in the -- IIRC -- 29-rc timeframe. Or maybe it was the 30-rc timeframe.
I had that kernel up for soemthing like five weeks, on a busy, low-ram
laptop, w/o and long pauses or other noticeable pain. For at least two
years before that the best this laptop could manage before latency
became horrific was a fortnight, and weekly reboots were desirable.
Unfortunately, before I could send out a congratulatory note (I wanted
to first confirm that the result remained) I pulled up and started see-
ing even worse latency then ever.
Turning ext4 barriers off helped avoid some of the new latency, and my
current compile (master as of the 17th) is much better than .31 was.
But long pauses still occur whenever paging and backgound i/o-bound
tasks interact.
There was a major regression in paging performance either .30 or .31
(see above). Interestingly, top(1) also shows less swap usage for
the same usage patterns than with the older kernels. I don't know
which is cause or which is effect.
A git pull on a clone of Linus' tree is a great way to trigger the
problem. As is using Gentoo's portage to install/upgrade anything.
(Bad python; no alligator!?) (Yes, that is a burlesque of the no
doughnut joke.)
[1] Pythons which have invaded Florida have reportedly been
witnessed eating alligators, according to a recent wire
service article.
-JimC
--
James Cloos <[email protected]> OpenPGP: 1024D/ED7DAEA6
On 09/21/2009 10:47 PM, James Cloos wrote:
>[...]
> I had that kernel up for soemthing like five weeks, on a busy, low-ram
> laptop, w/o and long pauses or other noticeable pain. For at least two
> years before that the best this laptop could manage before latency
> became horrific was a fortnight, and weekly reboots were desirable.
>
> Unfortunately, before I could send out a congratulatory note (I wanted
> to first confirm that the result remained) I pulled up and started see-
> ing even worse latency then ever.
>[...]
> A git pull on a clone of Linus' tree is a great way to trigger the
> problem. As is using Gentoo's portage to install/upgrade anything.
> (Bad python; no alligator!?)
Fortunately, this is completely (at least here) solved by putting those
in make.conf:
PORTAGE_NICENESS=19
PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"
>>>>> "Nikos" == Nikos Chantziaras <[email protected]> writes:
Nikos> On 09/21/2009 10:47 PM, James Cloos wrote:
>> is a great way to trigger the problem. As is using Gentoo's portage
>> to install/upgrade anything.
Nikos> Fortunately, this is completely (at least here) solved by putting
Nikos> those in make.conf:
Nikos> PORTAGE_NICENESS=19
Nikos> PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"
I had NICENESS set, but had never noticed that PORTAGE_IONICE_COMMAND
was available. I'll give that a test. Thanks.
-JimC
--
James Cloos <[email protected]> OpenPGP: 1024D/ED7DAEA6
On Mon, 2009-09-21 at 15:47 -0400, James Cloos wrote:
> FWIW, I did not notice any problems with the last kernel I had compiled
> in the -- IIRC -- 29-rc timeframe. Or maybe it was the 30-rc timeframe.
>
> I had that kernel up for soemthing like five weeks, on a busy, low-ram
> laptop, w/o and long pauses or other noticeable pain. For at least two
> years before that the best this laptop could manage before latency
> became horrific was a fortnight, and weekly reboots were desirable.
>
> Unfortunately, before I could send out a congratulatory note (I wanted
> to first confirm that the result remained) I pulled up and started see-
> ing even worse latency then ever.
>
> Turning ext4 barriers off helped avoid some of the new latency, and my
> current compile (master as of the 17th) is much better than .31 was.
>
> But long pauses still occur whenever paging and backgound i/o-bound
> tasks interact.
>
> There was a major regression in paging performance either .30 or .31
> (see above). Interestingly, top(1) also shows less swap usage for
> the same usage patterns than with the older kernels. I don't know
> which is cause or which is effect.
If someone has a definite good/bad cutoff for the problem they're
seeing, a git bisect (yeah, takes time) would be a good thing to try.
The "interactivity while saturating root disk" thing isn't a new problem
though, I can recall that being a problem to varying degrees forever.
(forever being defined as feb '93 -> today;)
> A git pull on a clone of Linus' tree is a great way to trigger the
> problem. As is using Gentoo's portage to install/upgrade anything.
> (Bad python; no alligator!¹) (Yes, that is a burlesque of the no
> doughnut joke.)
Git doesn't cause me any woes whatsoever, but my box's primary drive
isn't a slow laptop drive, ram isn't tight, and it has 4 cores.
Dunno.
If you're using the CFQ IO scheduler, IO bandwidth follows CPU nice
level, so tweaking nice tweaks both CPU and IO in one go. ionice can
also be used to tweak IO independently.
I don't know if distros provide a tool that supports SCHED_IDLE, if so,
you can use that to set both CPU and IO usage to IDLE class in one go.
If you don't have such a tool, you can try the attached for background
loads. Someone posted it to LKML in '98, and I've been updating/using
it for experiments ever since. Just do schedctl -I background-job, and
the job will be as polite as CPU and IO schedulers can make it.
-Mike
On Tue, 2009-09-22 at 09:06 +0200, Mike Galbraith wrote:
> I don't know if distros provide a tool that supports SCHED_IDLE, if so,
> you can use that to set both CPU and IO usage to IDLE class in one go.
P.S. I just remembered, don't use SCHED_IDLE with kernels < 2.6.28.
There are two patches wrt this scheduling class which did not go to the
stable trees less than 2.6.28.
-Mike
On Tue, 22 Sep 2009, Mike Galbraith wrote:
> On Tue, 2009-09-22 at 09:06 +0200, Mike Galbraith wrote:
> > I don't know if distros provide a tool that supports SCHED_IDLE, if so,
> > you can use that to set both CPU and IO usage to IDLE class in one go.
>
> P.S. I just remembered, don't use SCHED_IDLE with kernels < 2.6.28.
> There are two patches wrt this scheduling class which did not go to the
> stable trees less than 2.6.28.
Since 2.6.27 is long-term, maybe they should be sent to 2.6.27? It is weird
to have a long-term stable kernel for which the recommendation is "don't use
SCHED_IDLE".
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh
On Tue, 2009-09-22 at 08:22 -0300, Henrique de Moraes Holschuh wrote:
> On Tue, 22 Sep 2009, Mike Galbraith wrote:
> > On Tue, 2009-09-22 at 09:06 +0200, Mike Galbraith wrote:
> > > I don't know if distros provide a tool that supports SCHED_IDLE, if so,
> > > you can use that to set both CPU and IO usage to IDLE class in one go.
> >
> > P.S. I just remembered, don't use SCHED_IDLE with kernels < 2.6.28.
> > There are two patches wrt this scheduling class which did not go to the
> > stable trees less than 2.6.28.
>
> Since 2.6.27 is long-term, maybe they should be sent to 2.6.27? It is weird
> to have a long-term stable kernel for which the recommendation is "don't use
> SCHED_IDLE".
Yeah, that's what I thought too, but apparently maintainers didn't think
SCHED_IDLE was common/important enough.
-Mike
>>>>> "Mike" == Mike Galbraith <[email protected]> writes:
Mike> If someone has a definite good/bad cutoff for the problem they're
Mike> seeing, a git bisect (yeah, takes time) would be a good thing to try.
Yes, I wanted to try that for this, but given how long it takes to
compile a kernel on this box, and how long it takes to get everything
up so that I can test my real-life load, added to the difference in
performance after each day of uptime, I could do at best one kernel
per day; it would take weeks to finish.
That said, I jsut took a look at the git log for my /boot/grub repo
and can confirm that the last really good kernel -- the one I had up
for more than a month for the first time in years -- was pulled
between 2.6.30-rc6 and 2.6.30-rc7 and installed on 2009/05/17 which,
since I keep .config in git in my compile repo, lets me narrow down
the make oldconfig to 2009/05/17 18:37:24Z.
So, whatever Linus' tree looked like on that date is GOOD. His last
commit before that was 0f6f49a8cd0163fdb1723ed29f01fc65177108dc.
My next make oldconfig was on 2009/06/25 04:32:03Z, when I started
testing the radeon kms. (I'm back to non-KMS now.) There were too
many issues directly caused by KMS itself for me to comment on non-
KMS kernel issues from the kernels I ran with KMS enabled.
My return from KMS as on 2009/08/31; Linus' last commit before I did
that was adda766193ea1cf3137484a9521972d080d0b7af.
To give an idea of how little that narrows things:
:; git log 0f6f49a8cd..adda766193|egrep -c ^commit
12094
For those who use the tars, that means 30-rc6 was fast and 31-rc8 slow.
Mike> Git doesn't cause me any woes whatsoever, but my box's primary
Mike> drive isn't a slow laptop drive, ram isn't tight,
That does make a difference. Git is *great* when it fits into ram, but
a pull or merge on a clone of Linus' tree takes a quarter gig of VM on
x86-32, which guarantees paging, including to the swap partitions. (The
laptop is old enough to support three platters — including the optical —
so I have my primary swap partition on the second drive to reduce some
of the load to the / /var /boot drive. The first disk's swap partition
only gets hit when I boot w/o the second drive. It helps.)
The problem isn't that paging occurs, but rather that now, when
paging occurs, said paging is much slower than it had been.
IOW, the priority of paging vs other disk I/O has changed for the worse,
at some point after 0f6f49a8cd and before adda766193. But, as I wrote
above, to actually test kernels between those to determine where would
take weeks.
-JimC
--
James Cloos <[email protected]> OpenPGP: 1024D/ED7DAEA6