100% reproducible with elevator=cfq
Unable to handle kernel NULL pointer dereference at virtual address 00000000
printing eip:
c013f2c7
*pde = 00000000
Oops: 0000 [#1]
PREEMPT
Modules linked in: ipv6 dm_mod emu10k1 sound soundcore ac97_codec unix
CPU: 0
EIP: 0060:[put_page+7/144] Not tainted VLI
EFLAGS: 00010282 (2.6.9-mm1)
EIP is at put_page+0x7/0x90
eax: 00000000 ebx: df047400 ecx: 00000000 edx: 00000000
esi: 00000000 edi: 00000000 ebp: df047400 esp: cadd0c80
ds: 007b es: 007b ss: 0068
Process diotest4 (pid: 4498, threadinfo=cadd0000 task=de712a40)
Stack: df047400 c0178d50 00000000 00000000 c0179c91 df047400 00000001 c01b391b
c86b6b00 cb0c9b80 cadd0d1c cb0c9c38 c019d31e cee0b0b0 00000000 00000000
00003000 00000000 df047400 00000000 08051000 0000000c c017a182 00000001
Call Trace:
[dio_cleanup+48/64] dio_cleanup+0x30/0x40
[direct_io_worker+817/1568] direct_io_worker+0x331/0x620
[journal_put_journal_head+59/192] journal_put_journal_head+0x3b/0xc0
[ext3_do_update_inode+366/944] ext3_do_update_inode+0x16e/0x3b0
[__blockdev_direct_IO+514/752] __blockdev_direct_IO+0x202/0x2f0
[ext3_direct_io_get_blocks+0/240] ext3_direct_io_get_blocks+0x0/0xf0
[ext3_direct_IO+201/576] ext3_direct_IO+0xc9/0x240
[ext3_direct_io_get_blocks+0/240] ext3_direct_io_get_blocks+0x0/0xf0
[generic_file_direct_IO+121/160] generic_file_direct_IO+0x79/0xa0
[generic_file_direct_write+134/416] generic_file_direct_write+0x86/0x1a0
[generic_file_aio_write_nolock+689/1200] generic_file_aio_write_nolock+0x2b1/0x4b0
[generic_file_aio_write+131/256] generic_file_aio_write+0x83/0x100
[ext3_file_write+68/224] ext3_file_write+0x44/0xe0
[do_sync_write+190/240] do_sync_write+0xbe/0xf0
[autoremove_wake_function+0/96] autoremove_wake_function+0x0/0x60
[dnotify_parent+162/224] dnotify_parent+0xa2/0xe0
[do_sync_write+0/240] do_sync_write+0x0/0xf0
[vfs_write+184/304] vfs_write+0xb8/0x130
[sys_write+81/128] sys_write+0x51/0x80
[syscall_call+7/11] syscall_call+0x7/0xb
Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 83 ec 04 8b 54 24 08 <8b> 02 a9 00 80 00 00 75 44 8b 02 f6 c4 08 75 1f 8b 02 89 d1 a9
<1>Unable to handle kernel paging request at virtual address 4e6c9214
printing eip:
c0178939
*pde = 00000000
Oops: 0000 [#2]
PREEMPT
Modules linked in: ipv6 dm_mod emu10k1 sound soundcore ac97_codec unix
CPU: 0
EIP: 0060:[dio_get_page+25/96] Not tainted VLI
EFLAGS: 00010216 (2.6.9-mm1)
EIP is at dio_get_page+0x19/0x60
eax: 62615a59 ebx: c4e72800 ecx: 00000000 edx: 00000000
esi: 00000000 edi: 00000000 ebp: c4e72800 esp: de868c7c
ds: 007b es: 007b ss: 0068
Process diotest1 (pid: 4493, threadinfo=de868000 task=d1c435d0)
Stack: dffe0520 c4e72800 c0178d48 c4e72800 00000000 c0179c91 c4e72800 00000001
c01b391b c86b6380 c08349fc de868d1c c0834ab4 c019d31e cee0b0b0 00000000
00000000 00040000 00000000 c4e72800 00000000 08050000 0000000c c017a182
Call Trace:
[dio_cleanup+40/64] dio_cleanup+0x28/0x40
[direct_io_worker+817/1568] direct_io_worker+0x331/0x620
[journal_put_journal_head+59/192] journal_put_journal_head+0x3b/0xc0
[ext3_do_update_inode+366/944] ext3_do_update_inode+0x16e/0x3b0
[__blockdev_direct_IO+514/752] __blockdev_direct_IO+0x202/0x2f0
[ext3_direct_io_get_blocks+0/240] ext3_direct_io_get_blocks+0x0/0xf0
[ext3_direct_IO+201/576] ext3_direct_IO+0xc9/0x240
[ext3_direct_io_get_blocks+0/240] ext3_direct_io_get_blocks+0x0/0xf0
[generic_file_direct_IO+121/160] generic_file_direct_IO+0x79/0xa0
[generic_file_direct_write+134/416] generic_file_direct_write+0x86/0x1a0
[generic_file_aio_write_nolock+689/1200] generic_file_aio_write_nolock+0x2b1/0x4b0
[generic_file_aio_write+131/256] generic_file_aio_write+0x83/0x100
[ext3_file_write+68/224] ext3_file_write+0x44/0xe0
[do_sync_write+190/240] do_sync_write+0xbe/0xf0
[autoremove_wake_function+0/96] autoremove_wake_function+0x0/0x60
[dnotify_parent+162/224] dnotify_parent+0xa2/0xe0
[do_sync_write+0/240] do_sync_write+0x0/0xf0
[vfs_write+184/304] vfs_write+0xb8/0x130
[schedule+770/1376] schedule+0x302/0x560
[sys_write+81/128] sys_write+0x51/0x80
[syscall_call+7/11] syscall_call+0x7/0xb
Code: 00 89 87 b0 00 00 00 eb a5 89 f6 8d bc 27 00 00 00 00 83 ec 08 89 5c 24 04 8b 5c 24 0c 8b 83 b0 01 00 00 39 83 b4 01 00 00 74 18 <8b> 94 83 b0 00 00 00 40 89 83 b0 01 00 00 89 d0 8b 5c 24 04 83
--
I route therefore you are
On Sat, Oct 23 2004, Lorenzo Allegrucci wrote:
>
> 100% reproducible with elevator=cfq
but not with the other io schedulers?
> Unable to handle kernel NULL pointer dereference at virtual address 00000000
> printing eip:
> c013f2c7
> *pde = 00000000
> Oops: 0000 [#1]
> PREEMPT
> Modules linked in: ipv6 dm_mod emu10k1 sound soundcore ac97_codec unix
> CPU: 0
> EIP: 0060:[put_page+7/144] Not tainted VLI
> EFLAGS: 00010282 (2.6.9-mm1)
> EIP is at put_page+0x7/0x90
> eax: 00000000 ebx: df047400 ecx: 00000000 edx: 00000000
> esi: 00000000 edi: 00000000 ebp: df047400 esp: cadd0c80
> ds: 007b es: 007b ss: 0068
> Process diotest4 (pid: 4498, threadinfo=cadd0000 task=de712a40)
> Stack: df047400 c0178d50 00000000 00000000 c0179c91 df047400 00000001 c01b391b
> c86b6b00 cb0c9b80 cadd0d1c cb0c9c38 c019d31e cee0b0b0 00000000 00000000
> 00003000 00000000 df047400 00000000 08051000 0000000c c017a182 00000001
> Call Trace:
> [dio_cleanup+48/64] dio_cleanup+0x30/0x40
> [direct_io_worker+817/1568] direct_io_worker+0x331/0x620
> [journal_put_journal_head+59/192] journal_put_journal_head+0x3b/0xc0
> [ext3_do_update_inode+366/944] ext3_do_update_inode+0x16e/0x3b0
> [__blockdev_direct_IO+514/752] __blockdev_direct_IO+0x202/0x2f0
> [ext3_direct_io_get_blocks+0/240] ext3_direct_io_get_blocks+0x0/0xf0
> [ext3_direct_IO+201/576] ext3_direct_IO+0xc9/0x240
> [ext3_direct_io_get_blocks+0/240] ext3_direct_io_get_blocks+0x0/0xf0
> [generic_file_direct_IO+121/160] generic_file_direct_IO+0x79/0xa0
> [generic_file_direct_write+134/416] generic_file_direct_write+0x86/0x1a0
> [generic_file_aio_write_nolock+689/1200] generic_file_aio_write_nolock+0x2b1/0x4b0
> [generic_file_aio_write+131/256] generic_file_aio_write+0x83/0x100
> [ext3_file_write+68/224] ext3_file_write+0x44/0xe0
> [do_sync_write+190/240] do_sync_write+0xbe/0xf0
> [autoremove_wake_function+0/96] autoremove_wake_function+0x0/0x60
> [dnotify_parent+162/224] dnotify_parent+0xa2/0xe0
> [do_sync_write+0/240] do_sync_write+0x0/0xf0
> [vfs_write+184/304] vfs_write+0xb8/0x130
> [sys_write+81/128] sys_write+0x51/0x80
> [syscall_call+7/11] syscall_call+0x7/0xb
Please provide a test case, or at least a description of the problem.
--
Jens Axboe
On Sunday 24 October 2004 14:30, Jens Axboe wrote:
> On Sat, Oct 23 2004, Lorenzo Allegrucci wrote:
> >
> > 100% reproducible with elevator=cfq
>
> but not with the other io schedulers?
No, now I can reproduce it with the anticipatory scheduler too.
> > Unable to handle kernel NULL pointer dereference at virtual address 00000000
> > printing eip:
> > c013f2c7
> > *pde = 00000000
> > Oops: 0000 [#1]
> > PREEMPT
> > Modules linked in: ipv6 dm_mod emu10k1 sound soundcore ac97_codec unix
> > CPU: 0
> > EIP: 0060:[put_page+7/144] Not tainted VLI
> > EFLAGS: 00010282 (2.6.9-mm1)
> > EIP is at put_page+0x7/0x90
> > eax: 00000000 ebx: df047400 ecx: 00000000 edx: 00000000
> > esi: 00000000 edi: 00000000 ebp: df047400 esp: cadd0c80
> > ds: 007b es: 007b ss: 0068
> > Process diotest4 (pid: 4498, threadinfo=cadd0000 task=de712a40)
> > Stack: df047400 c0178d50 00000000 00000000 c0179c91 df047400 00000001 c01b391b
> > c86b6b00 cb0c9b80 cadd0d1c cb0c9c38 c019d31e cee0b0b0 00000000 00000000
> > 00003000 00000000 df047400 00000000 08051000 0000000c c017a182 00000001
> > Call Trace:
> > [dio_cleanup+48/64] dio_cleanup+0x30/0x40
> > [direct_io_worker+817/1568] direct_io_worker+0x331/0x620
> > [journal_put_journal_head+59/192] journal_put_journal_head+0x3b/0xc0
> > [ext3_do_update_inode+366/944] ext3_do_update_inode+0x16e/0x3b0
> > [__blockdev_direct_IO+514/752] __blockdev_direct_IO+0x202/0x2f0
> > [ext3_direct_io_get_blocks+0/240] ext3_direct_io_get_blocks+0x0/0xf0
> > [ext3_direct_IO+201/576] ext3_direct_IO+0xc9/0x240
> > [ext3_direct_io_get_blocks+0/240] ext3_direct_io_get_blocks+0x0/0xf0
> > [generic_file_direct_IO+121/160] generic_file_direct_IO+0x79/0xa0
> > [generic_file_direct_write+134/416] generic_file_direct_write+0x86/0x1a0
> > [generic_file_aio_write_nolock+689/1200] generic_file_aio_write_nolock+0x2b1/0x4b0
> > [generic_file_aio_write+131/256] generic_file_aio_write+0x83/0x100
> > [ext3_file_write+68/224] ext3_file_write+0x44/0xe0
> > [do_sync_write+190/240] do_sync_write+0xbe/0xf0
> > [autoremove_wake_function+0/96] autoremove_wake_function+0x0/0x60
> > [dnotify_parent+162/224] dnotify_parent+0xa2/0xe0
> > [do_sync_write+0/240] do_sync_write+0x0/0xf0
> > [vfs_write+184/304] vfs_write+0xb8/0x130
> > [sys_write+81/128] sys_write+0x51/0x80
> > [syscall_call+7/11] syscall_call+0x7/0xb
>
> Please provide a test case, or at least a description of the problem.
I can 100% reproduce it running "runalltests.sh -x 100" of
ltp-full-20040804.
The test cases seem to be diotest1 and diotest4 because they always
appear in the oops, but I couldn't reproduce it running diotest1 and
diotest4 alone.
--
I route therefore you are
On Sunday 24 October 2004 16:20, Lorenzo Allegrucci wrote:
> On Sunday 24 October 2004 14:30, Jens Axboe wrote:
> > On Sat, Oct 23 2004, Lorenzo Allegrucci wrote:
> > >
> > > 100% reproducible with elevator=cfq
> >
> > but not with the other io schedulers?
>
> No, now I can reproduce it with the anticipatory scheduler too.
I've just got the same oops using XFS instead of ext3.
Unable to handle kernel NULL pointer dereference at virtual address 0000007a
printing eip:
c013f2c7
*pde = 00000000
Oops: 0000 [#1]
PREEMPT
Modules linked in: xfs ipv6 dm_mod emu10k1 sound soundcore ac97_codec unix
CPU: 0
EIP: 0060:[put_page+7/144] Not tainted VLI
EFLAGS: 00010296 (2.6.9-mm1)
EIP is at put_page+0x7/0x90
eax: 0000007a ebx: de3f3400 ecx: 00000000 edx: 0000007a
esi: 00000000 edi: 00000000 ebp: de3f3400 esp: dd93ac5c
ds: 007b es: 007b ss: 0068
Process diotest1 (pid: 4795, threadinfo=dd93a000 task=dd923a00)
Stack: de3f3400 c0178d50 0000007a 00000000 c0179c91 de3f3400 00000001 d59075ec
00000001 00000008 e0a96eb5 d8e63610 d8e636dc 00000000 00000000 00000000
00000000 00000000 de3f3400 00000000 08050000 0000000c c017a182 00000001
Call Trace:
[dio_cleanup+48/64] dio_cleanup+0x30/0x40
[direct_io_worker+817/1568] direct_io_worker+0x331/0x620
[pg0+543870645/1069630464] xfs_ilock_map_shared+0x25/0x40 [xfs]
[__blockdev_direct_IO+514/752] __blockdev_direct_IO+0x202/0x2f0
[pg0+544038528/1069630464] linvfs_get_blocks_direct+0x0/0x50 [xfs]
[pg0+544032512/1069630464] linvfs_unwritten_convert_direct+0x0/0x70 [xfs]
[pg0+544038830/1069630464] linvfs_direct_IO+0xde/0xe0 [xfs]
[pg0+544038528/1069630464] linvfs_get_blocks_direct+0x0/0x50 [xfs]
[pg0+544032512/1069630464] linvfs_unwritten_convert_direct+0x0/0x70 [xfs]
[generic_file_direct_IO+121/160] generic_file_direct_IO+0x79/0xa0
[generic_file_direct_write+134/416] generic_file_direct_write+0x86/0x1a0
[pg0+544067824/1069630464] xfs_write+0x3b0/0xc00 [xfs]
[pg0+544049612/1069630464] linvfs_write+0x8c/0xa0 [xfs]
[do_sync_write+190/240] do_sync_write+0xbe/0xf0
[autoremove_wake_function+0/96] autoremove_wake_function+0x0/0x60
[do_sync_write+0/240] do_sync_write+0x0/0xf0
[vfs_write+184/304] vfs_write+0xb8/0x130
[sys_write+81/128] sys_write+0x51/0x80
[syscall_call+7/11] syscall_call+0x7/0xb
Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 83 ec 04 8b 54 24 08 <8b> 02 a9 00 80 00 00 75 44 8b 02 f6 c4 08 75 1f 8b 02 89 d1 a9
<1>Unable to handle kernel paging request at virtual address 5a3f30b0
printing eip:
c0178939
*pde = 00000000
Oops: 0000 [#2]
PREEMPT
Modules linked in: xfs ipv6 dm_mod emu10k1 sound soundcore ac97_codec unix
CPU: 0
EIP: 0060:[dio_get_page+25/96] Not tainted VLI
EFLAGS: 00010a02 (2.6.9-mm1)
EIP is at dio_get_page+0x19/0x60
eax: 5f000000 ebx: de3f3000 ecx: 00000000 edx: 00000000
esi: 00000000 edi: 00000000 ebp: de3f3000 esp: dc930c58
ds: 007b es: 007b ss: 0068
Process diotest4 (pid: 4947, threadinfo=dc930000 task=dcde9aa0)
Stack: 0000000e de3f3000 c0178d48 de3f3000 00000000 c0179c91 de3f3000 00000001
0000000e 0000003f 00000008 e0a96eb5 d14b65d0 d14b669c 00000000 00000000
00000000 0000f000 00000000 de3f3000 00000000 08051000 0000000c c017a182
Call Trace:
[dio_cleanup+40/64] dio_cleanup+0x28/0x40
[direct_io_worker+817/1568] direct_io_worker+0x331/0x620
[pg0+543870645/1069630464] xfs_ilock_map_shared+0x25/0x40 [xfs]
[__blockdev_direct_IO+514/752] __blockdev_direct_IO+0x202/0x2f0
[pg0+544038528/1069630464] linvfs_get_blocks_direct+0x0/0x50 [xfs]
[pg0+544032512/1069630464] linvfs_unwritten_convert_direct+0x0/0x70 [xfs]
[pg0+544038830/1069630464] linvfs_direct_IO+0xde/0xe0 [xfs]
[pg0+544038528/1069630464] linvfs_get_blocks_direct+0x0/0x50 [xfs]
[pg0+544032512/1069630464] linvfs_unwritten_convert_direct+0x0/0x70 [xfs]
[filemap_fdatawait+115/128] filemap_fdatawait+0x73/0x80
[generic_file_direct_IO+121/160] generic_file_direct_IO+0x79/0xa0
[generic_file_direct_write+134/416] generic_file_direct_write+0x86/0x1a0
[pg0+544067824/1069630464] xfs_write+0x3b0/0xc00 [xfs]
[pg0+544049612/1069630464] linvfs_write+0x8c/0xa0 [xfs]
[do_sync_write+190/240] do_sync_write+0xbe/0xf0
[autoremove_wake_function+0/96] autoremove_wake_function+0x0/0x60
[dnotify_parent+162/224] dnotify_parent+0xa2/0xe0
[do_sync_write+0/240] do_sync_write+0x0/0xf0
[vfs_write+184/304] vfs_write+0xb8/0x130
[sys_write+81/128] sys_write+0x51/0x80
[syscall_call+7/11] syscall_call+0x7/0xb
Code: 00 89 87 b0 00 00 00 eb a5 89 f6 8d bc 27 00 00 00 00 83 ec 08 89 5c 24 04 8b 5c 24 0c 8b 83 b0 01 00 00 39 83 b4 01 00 00 74 18 <8b> 94 83 b0 00 00 00 40 89 83 b0 01 00 00 89 d0 8b 5c 24 04 83
--
I route therefore you are
On Sunday 24 October 2004 22:02, Lorenzo Allegrucci wrote:
> On Sunday 24 October 2004 16:20, Lorenzo Allegrucci wrote:
> > On Sunday 24 October 2004 14:30, Jens Axboe wrote:
> > > On Sat, Oct 23 2004, Lorenzo Allegrucci wrote:
> > > >
> > > > 100% reproducible with elevator=cfq
> > >
> > > but not with the other io schedulers?
> >
> > No, now I can reproduce it with the anticipatory scheduler too.
>
> I've just got the same oops using XFS instead of ext3.
And disabling CONFIG_PREEMPT_BKL doesn't help.
--
I route therefore you are
On Mon, Oct 25 2004, Lorenzo Allegrucci wrote:
> On Sunday 24 October 2004 22:02, Lorenzo Allegrucci wrote:
> > On Sunday 24 October 2004 16:20, Lorenzo Allegrucci wrote:
> > > On Sunday 24 October 2004 14:30, Jens Axboe wrote:
> > > > On Sat, Oct 23 2004, Lorenzo Allegrucci wrote:
> > > > >
> > > > > 100% reproducible with elevator=cfq
> > > >
> > > > but not with the other io schedulers?
> > >
> > > No, now I can reproduce it with the anticipatory scheduler too.
> >
> > I've just got the same oops using XFS instead of ext3.
>
> And disabling CONFIG_PREEMPT_BKL doesn't help.
I'll try to reproduce it here now.
--
Jens Axboe
On Mon, Oct 25 2004, Jens Axboe wrote:
> On Mon, Oct 25 2004, Lorenzo Allegrucci wrote:
> > On Sunday 24 October 2004 22:02, Lorenzo Allegrucci wrote:
> > > On Sunday 24 October 2004 16:20, Lorenzo Allegrucci wrote:
> > > > On Sunday 24 October 2004 14:30, Jens Axboe wrote:
> > > > > On Sat, Oct 23 2004, Lorenzo Allegrucci wrote:
> > > > > >
> > > > > > 100% reproducible with elevator=cfq
> > > > >
> > > > > but not with the other io schedulers?
> > > >
> > > > No, now I can reproduce it with the anticipatory scheduler too.
> > >
> > > I've just got the same oops using XFS instead of ext3.
> >
> > And disabling CONFIG_PREEMPT_BKL doesn't help.
>
> I'll try to reproduce it here now.
Something fishy is going on here - it seems to happen in the if (ret)
cleanup in direct_io_worker(), last error was -EFAULT for this case. And
reverting the dio eof patch 'fixes' the bug, I don't know why yet.
--
Jens Axboe
On Mon, Oct 25 2004, Jens Axboe wrote:
> On Mon, Oct 25 2004, Jens Axboe wrote:
> > On Mon, Oct 25 2004, Lorenzo Allegrucci wrote:
> > > On Sunday 24 October 2004 22:02, Lorenzo Allegrucci wrote:
> > > > On Sunday 24 October 2004 16:20, Lorenzo Allegrucci wrote:
> > > > > On Sunday 24 October 2004 14:30, Jens Axboe wrote:
> > > > > > On Sat, Oct 23 2004, Lorenzo Allegrucci wrote:
> > > > > > >
> > > > > > > 100% reproducible with elevator=cfq
> > > > > >
> > > > > > but not with the other io schedulers?
> > > > >
> > > > > No, now I can reproduce it with the anticipatory scheduler too.
> > > >
> > > > I've just got the same oops using XFS instead of ext3.
> > >
> > > And disabling CONFIG_PREEMPT_BKL doesn't help.
> >
> > I'll try to reproduce it here now.
>
> Something fishy is going on here - it seems to happen in the if (ret)
> cleanup in direct_io_worker(), last error was -EFAULT for this case. And
> reverting the dio eof patch 'fixes' the bug, I don't know why yet.
->head and ->tail were not initialized in the cleanup path, I'm guessing
this happens if we adjust the read to zero. Seems best to simply check
for that condition and bail early, instead of initing ->head and tail
earlier and go through the whole path.
--- /opt/kernel/BK/linux-2.6/fs/direct-io.c 2004-10-19 20:40:38.000000000 +0200
+++ linux-2.6.9-mm1/fs/direct-io.c 2004-10-25 12:27:31.283064376 +0200
@@ -987,6 +987,8 @@
isize = i_size_read(inode);
if (bytes_todo > (isize - offset))
bytes_todo = isize - offset;
+ if (!bytes_todo)
+ return 0;
for (seg = 0; seg < nr_segs && bytes_todo; seg++) {
user_addr = (unsigned long)iov[seg].iov_base;
--
Jens Axboe