Hi,
I'm using Reiser4 for my filesystems on disk (/dev/sda) , and it works
just fine. Recently I bought a second disk (/dev/sdb) for RAID-1
mirroring. With mdadm I created a degraded raid-1 array on /dev/md/0,
devices missing,/dev/sdb1. After that I created a Reiser4 filesystem on
/dev/md/0 and mounted it at /mnt. Then I copied the data from /dev/sda1
to /mnt.
All goes well until I umount /mnt, umount simply hangs without any
error. Syslog doesn't report any error. In /proc/mdstat, the array
remains "active sync". Shutting down linux fails because the umount is
still waiting and seems to block other umounts. The umount process
cannot be killed by the root. A hard reset is the only resolution to get
my system functioning normally, but without the raid-1 of course.
The problem seems to emerge only with the combination of RAID and
Reiser4. I've created an ext-2 filesystem on /dev/md/0, and after that
mount ; cp -ax ; umount works without a problem, and the hanging umount
re-appears when using Reiser4 again.
My questions:
- how can I find the cause of the hanging umount?
- how can I fix it?
A few details of my linux-box:
Gentoo Linux, 2.6.16 kernel with Reiser4-for-2.6.16-2.patch.gz
i686 AMD Athlon(tm) XP 2400+
DC4300 SATA-II controller (Silicon Image 3124, libata + sata_sil24 drivers)
2 x Samsung SP2504C hard disk
Thanx in advance,
Arend
On Sunday May 28, [email protected] wrote:
>
> My questions:
> - how can I find the cause of the hanging umount?
# echo t > /proc/sysrq-trigger
and look at the resulting kernel messages, particularly for the
unmount process. If they don't make sense to you, post them.
> - how can I fix it?
Understand the problems, change the code :-)
Probably we can manage it together...
NeilBrown
Neil Brown wrote:
> # echo t > /proc/sysrq-trigger
>
> and look at the resulting kernel messages, particularly for the
> unmount process. If they don't make sense to you, post them.
>
>
Tnx for your response. After recompiling the kernel with magic_sysrq
enabled, rebooting, and repeating umount with the reiser4 md device, the
following trace contains a reiser4-entry:
> syslog-ng R running 0 7581 1 7197 (NOTLB)
> umount D C011B591 0 7588 7200 (NOTLB)
> f6643c98 00000000 c1808320 c011b591 f7db5ad0 f4d18c00 003d092a 00000000
> 00000000 f7db5ad0 c1808320 00000000 f4d18c00 003d092a f6b33540 c1808320
> 00000000 f4d18c00 003d092a f6b33540 f6b33668 c1808320 00000000 f6643cfc
> Call Trace:
> [<c011b591>] __wake_up_common+0x41/0x70
> [<c0318346>] io_schedule+0x26/0x30
> [<c01469fb>] sync_page+0x4b/0x60
> [<c03184d5>] __wait_on_bit+0x45/0x70
> [<c01469b0>] sync_page+0x0/0x60
> [<c014726d>] wait_on_page_bit+0xad/0xc0
> [<c0136a30>] wake_bit_function+0x0/0x60
> [<c0136a30>] wake_bit_function+0x0/0x60
> [<c01ac2f9>] jwait_io+0x59/0x80
> [<c01c1763>] update_journal_header+0x83/0xb0
> [<c01c272a>] commit_tx+0xca/0x110
> [<c01c29a1>] reiser4_write_logs+0x141/0x1e0
> [<c01b9d91>] commit_current_atom+0x171/0x2c0
> [<c01baabf>] try_commit_txnh+0x13f/0x1e0
> [<c01bab94>] commit_txnh+0x34/0xd0
> [<c01b91bc>] txn_end+0x2c/0x30
> [<c01b91d0>] txn_restart+0x10/0x30
> [<c01b920a>] txn_restart_current+0x1a/0x20
> [<c01b9f1f>] force_commit_atom+0x3f/0x70
> [<c01ba03a>] txnmgr_force_commit_all+0xea/0x130
> [<c01fcb0e>] release_format40+0x7e/0x150
> [<c01b5ea8>] init_context+0x58/0x80
> [<c01c8b89>] reiser4_put_super+0x89/0xf0
> [<c01857ed>] invalidate_inodes+0x5d/0x80
> [<c0170fb6>] generic_shutdown_super+0x156/0x160
> [<c0171b2d>] kill_block_super+0x2d/0x50
> [<c0170d90>] deactivate_super+0x60/0x80
> [<c0188e1f>] sys_umount+0x3f/0x90
> [<c01171b0>] do_page_fault+0x1c0/0x5a8
> [<c0159bc1>] sys_munmap+0x51/0x80
> [<c0188e87>] sys_oldumount+0x17/0x20
> [<c010306b>] sysenter_past_esp+0x54/0x75
Freely inerpreting this trace, I'd say that the umount causes the
reiser4 filesystem to do an extra commit, and the sync_page seems to be
waiting for IO.
Are there other traces of interrest?
On Sunday May 28, [email protected] wrote:
> Neil Brown wrote:
> > # echo t > /proc/sysrq-trigger
> >
> > and look at the resulting kernel messages, particularly for the
> > unmount process. If they don't make sense to you, post them.
> >
> >
> Tnx for your response. After recompiling the kernel with magic_sysrq
> enabled, rebooting, and repeating umount with the reiser4 md device, the
> following trace contains a reiser4-entry:
This looks very much like a reiser4 problem rather than a raid
problem, or at least you will need someone very familiar with reiser4
to understand what is going on here.
I recommend you post this information to
[email protected]
(and cc: to linux-kernel too if you like) to make sure someone
knowledgeable sees it.
I suspect reiser4 has some kernel threads that it uses. They might
have 'obvious' names like
ent:....
ktxnmgrd
from looking at the source. Including the stack trace of these and
anything else with 'reiser4' in the trace would probably be helpful.
NeilBrown
>
> > syslog-ng R running 0 7581 1 7197 (NOTLB)
> > umount D C011B591 0 7588 7200 (NOTLB)
> > f6643c98 00000000 c1808320 c011b591 f7db5ad0 f4d18c00 003d092a 00000000
> > 00000000 f7db5ad0 c1808320 00000000 f4d18c00 003d092a f6b33540 c1808320
> > 00000000 f4d18c00 003d092a f6b33540 f6b33668 c1808320 00000000 f6643cfc
> > Call Trace:
> > [<c011b591>] __wake_up_common+0x41/0x70
> > [<c0318346>] io_schedule+0x26/0x30
> > [<c01469fb>] sync_page+0x4b/0x60
> > [<c03184d5>] __wait_on_bit+0x45/0x70
> > [<c01469b0>] sync_page+0x0/0x60
> > [<c014726d>] wait_on_page_bit+0xad/0xc0
> > [<c0136a30>] wake_bit_function+0x0/0x60
> > [<c0136a30>] wake_bit_function+0x0/0x60
> > [<c01ac2f9>] jwait_io+0x59/0x80
> > [<c01c1763>] update_journal_header+0x83/0xb0
> > [<c01c272a>] commit_tx+0xca/0x110
> > [<c01c29a1>] reiser4_write_logs+0x141/0x1e0
> > [<c01b9d91>] commit_current_atom+0x171/0x2c0
> > [<c01baabf>] try_commit_txnh+0x13f/0x1e0
> > [<c01bab94>] commit_txnh+0x34/0xd0
> > [<c01b91bc>] txn_end+0x2c/0x30
> > [<c01b91d0>] txn_restart+0x10/0x30
> > [<c01b920a>] txn_restart_current+0x1a/0x20
> > [<c01b9f1f>] force_commit_atom+0x3f/0x70
> > [<c01ba03a>] txnmgr_force_commit_all+0xea/0x130
> > [<c01fcb0e>] release_format40+0x7e/0x150
> > [<c01b5ea8>] init_context+0x58/0x80
> > [<c01c8b89>] reiser4_put_super+0x89/0xf0
> > [<c01857ed>] invalidate_inodes+0x5d/0x80
> > [<c0170fb6>] generic_shutdown_super+0x156/0x160
> > [<c0171b2d>] kill_block_super+0x2d/0x50
> > [<c0170d90>] deactivate_super+0x60/0x80
> > [<c0188e1f>] sys_umount+0x3f/0x90
> > [<c01171b0>] do_page_fault+0x1c0/0x5a8
> > [<c0159bc1>] sys_munmap+0x51/0x80
> > [<c0188e87>] sys_oldumount+0x17/0x20
> > [<c010306b>] sysenter_past_esp+0x54/0x75
>
> Freely inerpreting this trace, I'd say that the umount causes the
> reiser4 filesystem to do an extra commit, and the sync_page seems to be
> waiting for IO.
> Are there other traces of interrest?
>
Neil Brown wrote:
> On Sunday May 28, Arend Freije wrote:
>
>> Neil Brown wrote:
>>
>>> # echo t > /proc/sysrq-trigger
>>>
>>> and look at the resulting kernel messages, particularly for the
>>> unmount process. If they don't make sense to you, post them.
>>>
>> Tnx for your response. After recompiling the kernel with magic_sysrq
>> enabled, rebooting, and repeating umount with the reiser4 md device, the
>> following trace contains a reiser4-entry:
>>
>
> This looks very much like a reiser4 problem rather than a raid
> problem, or at least you will need someone very familiar with reiser4
> to understand what is going on here.
>
> I recommend you post this information to
> [email protected]
>
That would be [email protected]
> (and cc: to linux-kernel too if you like) to make sure someone
> knowledgeable sees it.
>
> I suspect reiser4 has some kernel threads that it uses. They might
> have 'obvious' names like
> ent:....
> ktxnmgrd
> from looking at the source. Including the stack trace of these and
> anything else with 'reiser4' in the trace would probably be helpful.
>
> NeilBrown
>
>
Alexander Zarochentsev wrote:
>>> would it work better with "no_write_barrier" mount option?
>>>
>> It would indeed. What's the purpose of this option?
>>
>
> It disables write barrier support in reiser4 which may be buggy. It is
> not necessary that reiser4 code has a bug, but the elevator, md device
> or disk driver code.
Alexander,
You're probably right. I found several posts on the linux-kernel list
involving problems with write barrier support in combination with SATA
and ext3 . So I tried:
# mkfs -t ext3 /dev/md/0
# mount -o barrier=1 /dev/md/0 /mnt
# cp -a $src /mnt
# umount /mnt
And indeed, umount hangs now as well.
So it seems to be a linux-kernel issue after all...
The call traces of interrest with the ext3 case seem to be:
> kjournald D 00000000 0 6790 1 6745 (L-TLB)
> f104ddcc 00000000 c1808320 00000000 00000000 f76cb2b0 c011b591 dfcc3a90
> 5b666c00 003d092c 00000000 00000000 dfcc3a90 c1808320 00000000 c1808320
> 00000000 5b666c00 003d092c f7c77a90 f7c77bb8 c1808320 00000000 f104de38
> Call Trace:
> [<c011b591>] __wake_up_common+0x41/0x70
> [<c0315766>] io_schedule+0x26/0x30
> [<c016af13>] sync_buffer+0x53/0x60
> [<c03158f5>] __wait_on_bit+0x45/0x70
> [<c016aec0>] sync_buffer+0x0/0x60
> [<c016aec0>] sync_buffer+0x0/0x60
> [<c03159bd>] out_of_line_wait_on_bit+0x9d/0xc0
> [<c0136a30>] wake_bit_function+0x0/0x60
> [<c0136a30>] wake_bit_function+0x0/0x60
> [<c016ee0e>] sync_dirty_buffer+0x5e/0xd0
> [<f9d57055>] journal_write_commit_record+0x95/0x140 [jbd]
> [<f9d5cd18>] journal_free_journal_head+0x18/0x20 [jbd]
> [<f9d5d266>] journal_put_journal_head+0x96/0x100 [jbd]
> [<f9d56207>] journal_unfile_buffer+0x67/0xb0 [jbd]
> [<f9d57ca7>] journal_commit_transaction+0xba7/0x13a0 [jbd]
> [<c0314f85>] preempt_schedule+0x45/0x70
> [<c012ab24>] lock_timer_base+0x24/0x50
> [<c012ad91>] try_to_del_timer_sync+0x51/0x60
> [<f9d5aa45>] kjournald+0xa5/0x220 [jbd]
> [<c01369d0>] autoremove_wake_function+0x0/0x60
> [<c01369d0>] autoremove_wake_function+0x0/0x60
> [<c0102fa2>] ret_from_fork+0x6/0x14
> [<f9d5a990>] commit_timeout+0x0/0x10 [jbd]
> [<f9d5a9a0>] kjournald+0x0/0x220 [jbd]
> [<c0101225>] kernel_thread_helper+0x5/0x10
and
> umount D C014761A 0 6806 6779 (NOTLB)
> f0d0fe60 00000000 c1808320 c014761a f05efc9c f0d0fe6c 00000000 0000000e
> f548d48c f548d484 00000000 00000001 f0d0fe60 c011b600 f548d484 c1808320
> 0007a120 f80c8400 003d0930 dfcd1a70 dfcd1b98 f548d400 f548d464 f0d0fe88
> Call Trace:
> [<c014761a>] find_get_pages+0x5a/0x70
> [<c011b600>] __wake_up+0x40/0x60
> [<f9d5ad35>] journal_kill_thread+0xc5/0x110 [jbd]
> [<c01369d0>] autoremove_wake_function+0x0/0x60
> [<c01369d0>] autoremove_wake_function+0x0/0x60
> [<f9eb2510>] ext3_clear_inode+0x20/0x40 [ext3]
> [<f9d5c181>] journal_destroy+0x11/0x1e0 [jbd]
> [<c018567d>] dispose_list+0xcd/0xf0
> [<f9eb21f9>] ext3_put_super+0x29/0x1b0 [ext3]
> [<c01857ad>] invalidate_inodes+0x5d/0x80
> [<c0170f76>] generic_shutdown_super+0x156/0x160
> [<c0171aed>] kill_block_super+0x2d/0x50
Kind Regards,
Arend Freije
On Wed, May 31 2006, Arend Freije wrote:
> You're probably right. I found several posts on the linux-kernel list
> involving problems with write barrier support in combination with SATA
> and ext3 . So I tried:
>
> # mkfs -t ext3 /dev/md/0
> # mount -o barrier=1 /dev/md/0 /mnt
> # cp -a $src /mnt
> # umount /mnt
>
> And indeed, umount hangs now as well.
> So it seems to be a linux-kernel issue after all...
Can you please try 2.6.17-rc5?
--
Jens Axboe