Hello.
> The mm-of-the-moment snapshot 2008-11-19-02-19 has been uploaded to
Recent mmotm randomly freezes on /sbin/modprobe and read(). 2.6.28-rc2-mm1 was OK.
Config is at http://I-love.SAKURA.ne.jp/tmp/config-2.6.28-rc5-mm1
----- Quoting from http://I-love.SAKURA.ne.jp/tmp/messages4.txt (2008-11-19-02-19) -----
kjournald starting. Commit interval 5 seconds
EXT3-fs warning: checktime reached, running e2fsck is recommended
EXT3 FS on sdb1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
EXT3 FS on sdc1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
/dev/sdb1 on /usr/src/vanilla type ext3 (rw,noatime,nodiratime)
/dev/sdc1 on /usr/src/distro type ext3 (rw,noatime,nodiratime)
Cleaning /tmp /var/run /var/lock.
Detecting hardware: open_sock(): No such device
agpgart pcnet32 piix BusLogic
Skipping unavailable/built-in agpgart module.
pcnet32 disabled in configuration.
Skipping unavailable/built-in piix module.
Skipping already loaded module BusLogic.
Running 0dns-down to make sure resolv.conf is ok...done.
Setting up networking...done.
Starting hotplug subsystem:
pci
ignoring pci display device 00:0f.0
BusLogic: already loaded
SysRq : Changing Loglevel
Loglevel set to 8
CPA self-test:
4k 227326 large 0 gb 0 x 227326[c0000000-f77fd000] miss 0
4k 227326 large 0 gb 0 x 227326[c0000000-f77fd000] miss 0
4k 227326 large 0 gb 0 x 227326[c0000000-f77fd000] miss 0
ok.
SysRq : Show Blocked State
task PC stack pid father
kjournald D f6a43a40 5944 807 2
f6359e3c 00000046 c0207ff6 f6a43a40 c21a4740 f6a229c0 00201488 00000000
f6688a00 00000000 f6358000 c21a4740 f65e4108 f65e3ea0 f6a43a40 000047bc
c1f0525c f6359e34 c023e48b c21a4740 c21a4740 00000000 c1f0525c f6359e48
Call Trace:
[<c0207ff6>] ? read_tsc+0x9/0x26
[<c023e48b>] ? ktime_get_ts+0x43/0x47
[<c05b5a3e>] io_schedule+0x47/0x79
[<c02ba64e>] sync_buffer+0x37/0x3b
[<c05b5c0d>] __wait_on_bit+0x30/0x56
[<c02ba617>] ? sync_buffer+0x0/0x3b
[<c05b5c9a>] out_of_line_wait_on_bit+0x67/0x6f
[<c02ba617>] ? sync_buffer+0x0/0x3b
[<c023c3f3>] ? wake_bit_function+0x0/0x42
[<c023c3f3>] ? wake_bit_function+0x0/0x42
[<c02ba6a9>] __wait_on_buffer+0x19/0x1c
[<c02f6a14>] journal_commit_transaction+0x6e4/0xbf4
[<c0247f75>] ? lock_release_holdtime+0x91/0x97
[<c024ab0b>] ? trace_hardirqs_on_caller+0x120/0x148
[<c024ab3e>] ? trace_hardirqs_on+0xb/0xd
[<c0233247>] ? try_to_del_timer_sync+0x54/0x5c
[<c02f8972>] kjournald+0xb8/0x229
[<c023c3c0>] ? autoremove_wake_function+0x0/0x33
[<c023c3c0>] ? autoremove_wake_function+0x0/0x33
[<c02252d2>] ? complete+0x36/0x3e
[<c02f88ba>] ? kjournald+0x0/0x229
[<c023bf0a>] kthread+0x37/0x59
[<c023bed3>] ? kthread+0x0/0x59
[<c0203d23>] kernel_thread_helper+0x7/0x10
modprobe D f6a43ca0 6104 1403 1401
f6765d48 00000046 c0207ff6 f6a43ca0 c2059740 c079cba0 00201488 00000000
f6652500 00000000 f6764000 c2059740 f6a3eac8 f6a3e860 f6a43ca0 00004792
c1f0e2bc f6765d40 c023e48b c2059740 c2059740 00000000 c1f0e2bc f6765d54
Call Trace:
[<c0207ff6>] ? read_tsc+0x9/0x26
[<c023e48b>] ? ktime_get_ts+0x43/0x47
[<c05b5a3e>] io_schedule+0x47/0x79
[<c0279c46>] sync_page+0x45/0x49
[<c05b5cd2>] __wait_on_bit_lock+0x30/0x57
[<c0279c01>] ? sync_page+0x0/0x49
[<c027a2f6>] __lock_page+0x62/0x6a
[<c023c3f3>] ? wake_bit_function+0x0/0x42
[<c027a4a4>] ? find_get_page+0xcf/0xd9
[<c023c3f3>] ? wake_bit_function+0x0/0x42
[<c027a4d9>] find_lock_page+0x2b/0x50
[<c027b1ab>] filemap_fault+0x86/0x29d
[<c028ab8b>] ? __do_fault+0x2df/0x325
[<c028a8fc>] __do_fault+0x50/0x325
[<c024bd04>] ? __lock_acquire+0x72a/0x77c
[<c028abfd>] do_linear_fault+0x2c/0x33
[<c028ad66>] handle_mm_fault+0xf9/0x1d9
[<c05b9586>] do_page_fault+0x1b6/0x427
[<c05b93d0>] ? do_page_fault+0x0/0x427
[<c05b7b17>] error_code+0x77/0x7c
[<c033e62a>] ? __copy_from_user_ll+0x35/0x4c
[<c033e787>] copy_from_user+0x31/0x56
[<c0253bec>] load_module+0x72/0x9d7
[<c0254577>] ? sys_init_module+0x26/0x18c
[<c03181c0>] ? security_capable+0x12/0x15
[<c0254592>] sys_init_module+0x41/0x18c
[<c0202fae>] syscall_call+0x7/0xb
Sched Debug Version: v0.07, 2.6.28-rc5-mm1 #1
now at 31572.891772 msecs
.sysctl_sched_latency : 40.000000
.sysctl_sched_min_granularity : 8.000000
.sysctl_sched_wakeup_granularity : 10.000000
.sysctl_sched_child_runs_first : 0.000001
.sysctl_sched_features : 24191
----- Quoting from http://I-love.SAKURA.ne.jp/tmp/messages5.txt (2008-11-19-02-19 + TOMOYO)-----
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
umount: devfs: not mounted
Calling /sbin/tomoyo-init to load policy. Please wait.
SysRq : Changing Loglevel
Loglevel set to 8
SysRq : Show Blocked State
task PC stack pid father
chroot D 0000004d 5904 1 0
f704fe38 00000046 00000000 0000004d c21a4740 f6a229c0 7d2e883b 0000000c
f65ee500 00000000 f704e000 c21a4740 f7050268 f7050000 c0247f75 f70504f0
00000063 00000000 00000063 00000001 7fffffff f704fea0 f704fec0 f704fe78
Call Trace:
[<c0247f75>] ? lock_release_holdtime+0x91/0x97
[<c05ba9dc>] schedule_timeout+0x16/0x92
[<c024a9ae>] ? mark_held_locks+0x4d/0x6c
[<c05bc739>] ? _spin_unlock_irq+0x22/0x26
[<c024ab0b>] ? trace_hardirqs_on_caller+0x120/0x148
[<c024ab3e>] ? trace_hardirqs_on+0xb/0xd
[<c05ba699>] wait_for_common+0xcd/0x121
[<c022518f>] ? default_wake_function+0x0/0xd
[<c022518f>] ? default_wake_function+0x0/0xd
[<c0239034>] ? queue_work_on+0x35/0x41
[<c05ba6ff>] wait_for_completion+0x12/0x14
[<c0238ed5>] call_usermodehelper_exec+0x76/0x8f
[<c05ba5fc>] ? wait_for_common+0x30/0x121
[<c031c1cf>] tomoyo_load_policy+0x9a/0x100
[<c031cca0>] tomoyo_bprm_set_creds+0x23/0x35
[<c03184a7>] security_bprm_set_creds+0xc/0xe
[<c02a4d47>] prepare_binprm+0xac/0xe1
[<c02a5180>] do_execve+0x10f/0x232
[<c0201f61>] sys_execve+0x29/0x4c
[<c0202fae>] syscall_call+0x7/0xb
cat D f6b463b8 6124 823 815
f6741da8 00000046 c0207ff6 f6b463b8 c2059740 c07a3ba0 00201487 00000000
f6726500 00000000 f6740000 c2059740 f62a1748 f62a14e0 f6b463b8 000021f2
c1f0edbc f6741da0 c023e48b c2059740 c2059740 00000000 c1f0edbc f6741db4
Call Trace:
[<c0207ff6>] ? read_tsc+0x9/0x26
[<c023e48b>] ? ktime_get_ts+0x43/0x47
[<c05ba8fe>] io_schedule+0x47/0x79
[<c0279c46>] sync_page+0x45/0x49
[<c0279c52>] sync_page_killable+0x8/0x30
[<c05bab92>] __wait_on_bit_lock+0x30/0x57
[<c0279c4a>] ? sync_page_killable+0x0/0x30
[<c027a363>] __lock_page_killable+0x65/0x6d
[<c023c3f3>] ? wake_bit_function+0x0/0x42
[<c027a4a4>] ? find_get_page+0xcf/0xd9
[<c023c3f3>] ? wake_bit_function+0x0/0x42
[<c027ab96>] do_generic_file_read+0x244/0x3c6
[<c027afa3>] generic_file_aio_read+0x132/0x15a
[<c027ad18>] ? file_read_actor+0x0/0xe8
[<c02a04d2>] do_sync_read+0xb7/0xf5
[<c020777a>] ? native_sched_clock+0x42/0x92
[<c023c3c0>] ? autoremove_wake_function+0x0/0x33
[<c024c6d7>] ? __lock_release+0x4b/0x51
[<c02d4b47>] ? dnotify_parent+0x64/0x6d
[<c0318b51>] ? security_file_permission+0xf/0x11
[<c02a03cc>] ? rw_verify_area+0x96/0xb2
[<c02a059c>] vfs_read+0x8c/0x12e
[<c02a089f>] sys_read+0x3b/0x60
[<c0202fae>] syscall_call+0x7/0xb
Sched Debug Version: v0.07, 2.6.28-rc5-mm1 #2
now at 25613.058324 msecs
.sysctl_sched_latency : 40.000000
.sysctl_sched_min_granularity : 8.000000
.sysctl_sched_wakeup_granularity : 10.000000
.sysctl_sched_child_runs_first : 0.000001
.sysctl_sched_features : 24191
On Thu, 20 Nov 2008 15:09:35 +0900, Tetsuo Handa said:
> Hello.
>
> > The mm-of-the-moment snapshot 2008-11-19-02-19 has been uploaded to
> Recent mmotm randomly freezes on /sbin/modprobe and read(). 2.6.28-rc2-mm1 was OK.
I'm seeing very similar hangs on -mmotm-11-17 as well. I've hit it 3 times
today, all while disk activity was moderately heavy (things like 'yum update',
or a 'find . | xargs grep', and so on).
Managed to catch one while netconsole was active - I didn't have any messages
for bugs/warns/oopsen. Apparently, somebody is holding a lock. (I also have an
alt-sysrq-t from this incident, but that's about 10 times as big, didn't want
to abuse vger too much.. ;)
[ 3932.912494] SysRq : Show Blocked State
[ 3932.913465] task PC stack pid father
[ 3932.913465] pdflush D ffff88007e247cf0 5776 303 2
[ 3932.913465] ffff88007e247c50 0000000000000002 ffff88007e247bb0 ffff88007dcf54d8
[ 3932.913465] ffff88007e247c00 ffffffff8081c780 ffffffff8081c780 ffff88007f269040
[ 3932.913465] ffff88007f232040 ffff88007f269398 000000007e247be0 ffff88007f269398
[ 3932.913465] Call Trace:
[ 3932.913465] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3932.913465] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3932.913465] [<ffffffff8027e44d>] sync_page+0x78/0x7f
[ 3932.913465] [<ffffffff80568380>] __wait_on_bit+0x47/0x79
[ 3932.913465] [<ffffffff8027e3d5>] ? sync_page+0x0/0x7f
[ 3932.913465] [<ffffffff8027e5ed>] wait_on_page_bit+0x6e/0x75
[ 3932.913465] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3932.913465] [<ffffffff80286343>] ? pagevec_lookup_tag+0x22/0x2b
[ 3932.913465] [<ffffffff8027eed9>] wait_on_page_writeback_range+0x75/0x13d
[ 3932.913465] [<ffffffff8027efc1>] filemap_fdatawait+0x20/0x22
[ 3932.913465] [<ffffffff8027f0cc>] filemap_write_and_wait+0x27/0x33
[ 3932.913465] [<ffffffff802c7c32>] sync_blockdev+0x1b/0x1d
[ 3932.913465] [<ffffffff802c2242>] __sync_inodes+0x74/0xbf
[ 3932.913465] [<ffffffff802c22a6>] sync_inodes+0x19/0x33
[ 3932.913465] [<ffffffff802c5378>] do_sync+0x1a/0x77
[ 3932.913465] [<ffffffff80285a4c>] pdflush+0x145/0x1f8
[ 3932.913465] [<ffffffff802c535e>] ? do_sync+0x0/0x77
[ 3932.913465] [<ffffffff80285907>] ? pdflush+0x0/0x1f8
[ 3932.913465] [<ffffffff8024ba10>] kthread+0x49/0x76
[ 3932.913465] [<ffffffff8020cb79>] child_rip+0xa/0x11
[ 3932.913465] [<ffffffff8020bfe5>] ? restore_args+0x0/0x30
[ 3932.913465] [<ffffffff8024b9c7>] ? kthread+0x0/0x76
[ 3932.913465] [<ffffffff8020cb6f>] ? child_rip+0x0/0x11
[ 3932.913465] pdflush D 0000000000000246 5056 304 2
[ 3932.913465] ffff88007e275d50 0000000000000002 ffffe200016b1698 ffffe20001612f48
[ 3932.913465] ffffe20000d66ea8 ffffffff8081c780 ffffffff8081c780 ffff88007f25c040
[ 3932.913465] ffff88007f2697f0 ffff88007f25c398 000000007e275cf0 ffff88007f25c398
[ 3932.913465] Call Trace:
[ 3932.913465] [<ffffffff8022f968>] ? account_group_exec_runtime+0x26/0x64
[ 3932.913465] [<ffffffff8022fa79>] ? update_curr+0xd3/0xdc
[ 3932.913465] [<ffffffff8056883e>] __mutex_lock_common+0x280/0x406
[ 3932.913465] [<ffffffff80569f44>] ? _spin_unlock_irq+0x47/0x52
[ 3932.913465] [<ffffffff80568a56>] __mutex_lock_slowpath+0x14/0x16
[ 3932.913465] [<ffffffff80568a7d>] mutex_lock+0x25/0x29
[ 3932.913465] [<ffffffff802a8172>] lock_super+0x27/0x29
[ 3932.913465] [<ffffffff802a85db>] sync_supers+0x48/0xb6
[ 3932.913465] [<ffffffff80284ebe>] wb_kupdate+0x35/0x128
[ 3932.913465] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3932.913465] [<ffffffff80285a4c>] pdflush+0x145/0x1f8
[ 3932.913465] [<ffffffff80284e89>] ? wb_kupdate+0x0/0x128
[ 3932.913465] [<ffffffff80285907>] ? pdflush+0x0/0x1f8
[ 3932.913465] [<ffffffff8024ba10>] kthread+0x49/0x76
[ 3932.913465] [<ffffffff8020cb79>] child_rip+0xa/0x11
[ 3932.913465] [<ffffffff8020bfe5>] ? restore_args+0x0/0x30
[ 3932.913465] [<ffffffff8024b9c7>] ? kthread+0x0/0x76
[ 3932.913465] [<ffffffff8020cb6f>] ? child_rip+0x0/0x11
[ 3932.913465] kjournald D ffff88007e08c480 5696 1508 2
[ 3932.913465] ffff88007dc47c60 0000000000000002 ffff88007dcf54d8 ffff88007dcf54d8
[ 3932.913465] ffff88007dc47bd0 ffffffff8081c780 ffffffff8081c780 ffff88007e1a2040
[ 3932.913465] ffff88007906d040 ffff88007e1a2398 000000017dc47c00 ffff88007e1a2398
[ 3932.913465] Call Trace:
[ 3932.913465] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3932.913465] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3932.913465] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3932.913465] [<ffffffff80568380>] __wait_on_bit+0x47/0x79
[ 3932.913465] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3932.913465] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3932.913465] [<ffffffff8056841c>] out_of_line_wait_on_bit+0x6a/0x77
[ 3932.913465] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3932.913465] [<ffffffff803636fb>] ? submit_bio+0xc4/0xcd
[ 3932.913465] [<ffffffff802c7dc0>] __wait_on_buffer+0x36/0x3a
[ 3932.913465] [<ffffffff802c7e01>] wait_on_buffer+0x3d/0x41
[ 3932.913465] [<ffffffff802c8477>] sync_dirty_buffer+0x84/0xd3
[ 3932.913465] [<ffffffff80315357>] journal_commit_transaction+0xd22/0x11cb
[ 3932.913465] [<ffffffff80569fad>] ? _spin_unlock_irqrestore+0x5e/0x6c
[ 3932.913465] [<ffffffff80241178>] ? try_to_del_timer_sync+0x5e/0x6a
[ 3932.913465] [<ffffffff80318cf4>] kjournald+0xe2/0x23c
[ 3932.913465] [<ffffffff8024be3f>] ? autoremove_wake_function+0x0/0x38
[ 3933.879285] [<ffffffff80318c12>] ? kjournald+0x0/0x23c
[ 3933.879285] [<ffffffff8024ba10>] kthread+0x49/0x76
[ 3933.879285] [<ffffffff8020cb79>] child_rip+0xa/0x11
[ 3933.879285] [<ffffffff8020bfe5>] ? restore_args+0x0/0x30
[ 3933.879285] [<ffffffff805680f2>] ? preempt_schedule+0x43/0x81
[ 3933.879285] [<ffffffff8024b9c7>] ? kthread+0x0/0x76
[ 3933.879285] [<ffffffff8020cb6f>] ? child_rip+0x0/0x11
[ 3933.879285] kjournald D ffff880077c51d10 5264 1509 2
[ 3933.879285] ffff880077c51c80 0000000000000002 ffff88007dcf54d8 ffff88007dcf54d8
[ 3933.879285] ffff880077c51bf0 ffffffff8081c780 ffffffff8081c780 ffff88007e2dc7f0
[ 3933.879285] ffff88007f30a7f0 ffff88007e2dcb48 0000000177c51c20 ffff88007e2dcb48
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3933.879285] [<ffffffff80568380>] __wait_on_bit+0x47/0x79
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff8056841c>] out_of_line_wait_on_bit+0x6a/0x77
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff803636fb>] ? submit_bio+0xc4/0xcd
[ 3933.879285] [<ffffffff802c7dc0>] __wait_on_buffer+0x36/0x3a
[ 3933.879285] [<ffffffff80314536>] wait_on_buffer+0x3d/0x41
[ 3933.879285] [<ffffffff80314c27>] journal_commit_transaction+0x5f2/0x11cb
[ 3933.879285] [<ffffffff80569fad>] ? _spin_unlock_irqrestore+0x5e/0x6c
[ 3933.879285] [<ffffffff80241178>] ? try_to_del_timer_sync+0x5e/0x6a
[ 3933.879285] [<ffffffff80318cf4>] kjournald+0xe2/0x23c
[ 3933.879285] [<ffffffff8024be3f>] ? autoremove_wake_function+0x0/0x38
[ 3933.879285] [<ffffffff80318c12>] ? kjournald+0x0/0x23c
[ 3933.879285] [<ffffffff8024ba10>] kthread+0x49/0x76
[ 3933.879285] [<ffffffff8020cb79>] child_rip+0xa/0x11
[ 3933.879285] [<ffffffff8020bfe5>] ? restore_args+0x0/0x30
[ 3933.879285] [<ffffffff805680f2>] ? preempt_schedule+0x43/0x81
[ 3933.879285] [<ffffffff8024b9c7>] ? kthread+0x0/0x76
[ 3933.879285] [<ffffffff8020cb6f>] ? child_rip+0x0/0x11
[ 3933.879285] kjournald D ffff88007e41b080 5264 1511 2
[ 3933.879285] ffff8800770adc80 0000000000000002 ffff88007dcf54d8 ffff88007dcf54d8
[ 3933.879285] ffff8800770adbf0 ffffffff8081c780 ffffffff8081c780 ffff88007e1a4040
[ 3933.879285] ffff88007d5897f0 ffff88007e1a4398 00000000770adc20 ffff88007e1a4398
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3933.879285] [<ffffffff80568380>] __wait_on_bit+0x47/0x79
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff8056841c>] out_of_line_wait_on_bit+0x6a/0x77
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff803636fb>] ? submit_bio+0xc4/0xcd
[ 3933.879285] [<ffffffff802c7dc0>] __wait_on_buffer+0x36/0x3a
[ 3933.879285] [<ffffffff80314536>] wait_on_buffer+0x3d/0x41
[ 3933.879285] [<ffffffff80315125>] journal_commit_transaction+0xaf0/0x11cb
[ 3933.879285] [<ffffffff80569fad>] ? _spin_unlock_irqrestore+0x5e/0x6c
[ 3933.879285] [<ffffffff80241178>] ? try_to_del_timer_sync+0x5e/0x6a
[ 3933.879285] [<ffffffff80318cf4>] kjournald+0xe2/0x23c
[ 3933.879285] [<ffffffff8024be3f>] ? autoremove_wake_function+0x0/0x38
[ 3933.879285] [<ffffffff80318c12>] ? kjournald+0x0/0x23c
[ 3933.879285] [<ffffffff8024ba10>] kthread+0x49/0x76
[ 3933.879285] [<ffffffff8020cb79>] child_rip+0xa/0x11
[ 3933.879285] [<ffffffff8020bfe5>] ? restore_args+0x0/0x30
[ 3933.879285] [<ffffffff8024b9c7>] ? kthread+0x0/0x76
[ 3933.879285] [<ffffffff8020cb6f>] ? child_rip+0x0/0x11
[ 3933.879285] syslog-ng D 00000000ffffffff 4616 2021 1
[ 3933.879285] ffff88007e67fae8 0000000000000046 ffff88007dcf54d8 ffff88007dcf54d8
[ 3933.879285] ffff88007e67fa58 ffffffff8081c780 ffffffff8081c780 ffff88007e0d57f0
[ 3933.879285] ffffffff807093b0 ffff88007e0d5b48 000000007e67fa88 ffff88007e0d5b48
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3933.879285] [<ffffffff8056828e>] __wait_on_bit_lock+0x45/0x79
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff8056832c>] out_of_line_wait_on_bit_lock+0x6a/0x77
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff802c7001>] ? __find_get_block+0x1c7/0x1d9
[ 3933.879285] [<ffffffff802c8364>] __lock_buffer+0x25/0x27
[ 3933.879285] [<ffffffff8031283d>] lock_buffer+0x44/0x48
[ 3933.879285] [<ffffffff80313511>] do_get_write_access+0x72/0x4c7
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80317a12>] ? bit_spin_unlock+0x6b/0x79
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff80302ca4>] ext3_reserve_inode_write+0x3f/0x76
[ 3933.879285] [<ffffffff80302d11>] ext3_mark_inode_dirty+0x36/0x53
[ 3933.879285] [<ffffffff80302e7c>] ext3_dirty_inode+0x71/0x88
[ 3933.879285] [<ffffffff802c23e7>] __mark_inode_dirty+0x33/0x1a4
[ 3933.879285] [<ffffffff802ba2f0>] inode_setattr+0x11b/0x125
[ 3933.879285] [<ffffffff8030302d>] ext3_setattr+0x19a/0x201
[ 3933.879285] [<ffffffff802ba510>] notify_change+0x216/0x35e
[ 3933.879285] [<ffffffff802a52dc>] chown_common+0x69/0x80
[ 3933.879285] [<ffffffff802a525f>] ? audit_inode+0x43/0x57
[ 3933.879285] [<ffffffff802a5342>] sys_fchown+0x4f/0x6f
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] console-kit-d D ffff88007e08c480 4952 2109 1
[ 3933.879285] ffff88007d505a78 0000000000000046 ffff88007dcf54d8 ffff88007dcf54d8
[ 3933.879285] ffff88007d5059e8 ffffffff8081c780 ffffffff8081c780 ffff88007e1a37f0
[ 3933.879285] ffff88007906d040 ffff88007e1a3b48 000000007d505a18 ffff88007e1a3b48
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3933.879285] [<ffffffff8056828e>] __wait_on_bit_lock+0x45/0x79
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff8056832c>] out_of_line_wait_on_bit_lock+0x6a/0x77
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff802c667f>] ? put_bh+0x49/0x56
[ 3933.879285] [<ffffffff802c8364>] __lock_buffer+0x25/0x27
[ 3933.879285] [<ffffffff8031283d>] lock_buffer+0x44/0x48
[ 3933.879285] [<ffffffff80313511>] do_get_write_access+0x72/0x4c7
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80317a12>] ? bit_spin_unlock+0x6b/0x79
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff80301676>] ext3_new_inode+0x4d9/0x9fd
[ 3933.879285] [<ffffffff803088a4>] ext3_create+0x99/0x11b
[ 3933.879285] [<ffffffff802b0cbf>] vfs_create+0x79/0xa0
[ 3933.879285] [<ffffffff802b10c6>] do_filp_open+0x27c/0x806
[ 3933.879285] [<ffffffff802aa156>] ? cp_new_stat+0xe2/0xef
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] e16 D 00000000ffffffff 3752 2699 2640
[ 3933.879285] ffff880077173b88 0000000000000046 ffff88007f4a2da8 0000000000000001
[ 3933.879285] ffff880077173b08 ffffffff8081c780 ffffffff8081c780 ffff880077d6a040
[ 3933.879285] ffffffff807093b0 ffff880077d6a398 0000000077173b38 ffff880077d6a398
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569fad>] ? _spin_unlock_irqrestore+0x5e/0x6c
[ 3933.879285] [<ffffffff8031371b>] do_get_write_access+0x27c/0x4c7
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff8030177d>] ext3_new_inode+0x5e0/0x9fd
[ 3933.879285] [<ffffffff803088a4>] ext3_create+0x99/0x11b
[ 3933.879285] [<ffffffff802b0cbf>] vfs_create+0x79/0xa0
[ 3933.879285] [<ffffffff802b10c6>] do_filp_open+0x27c/0x806
[ 3933.879285] [<ffffffff8024be3f>] ? autoremove_wake_function+0x0/0x38
[ 3933.879285] [<ffffffff8024f26a>] ? ktime_get_ts+0x59/0x5e
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] festival_serv D 00000000ffffffff 4920 2918 2699
[ 3933.879285] ffff880077dddb68 0000000000000046 000000030006dac8 0000000000000001
[ 3933.879285] ffff880077dddad8 ffffffff8081c780 ffffffff8081c780 ffff88007f2617f0
[ 3933.879285] ffff88007f2327f0 ffff88007f261b48 0000000100000000 ffff88007f261b48
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff8034092d>] ? avc_has_perm+0x4e/0x60
[ 3933.879285] [<ffffffff8056883e>] __mutex_lock_common+0x280/0x406
[ 3933.879285] [<ffffffff80314062>] ? start_this_handle+0x3b3/0x3d0
[ 3933.879285] [<ffffffff80568a56>] __mutex_lock_slowpath+0x14/0x16
[ 3933.879285] [<ffffffff80568a7d>] mutex_lock+0x25/0x29
[ 3933.879285] [<ffffffff802a8172>] lock_super+0x27/0x29
[ 3933.879285] [<ffffffff80308d65>] ext3_orphan_add+0x28/0x17d
[ 3933.879285] [<ffffffff8031423e>] ? journal_start+0xb7/0xef
[ 3933.879285] [<ffffffff80302ff5>] ext3_setattr+0x162/0x201
[ 3933.879285] [<ffffffff802ba510>] notify_change+0x216/0x35e
[ 3933.879285] [<ffffffff802a5ae0>] do_truncate+0x63/0x81
[ 3933.879285] [<ffffffff802ae39d>] ? vfs_dq_init+0x5e/0x68
[ 3933.879285] [<ffffffff802b0c0b>] may_open+0x19f/0x1da
[ 3933.879285] [<ffffffff802b1259>] do_filp_open+0x40f/0x806
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] firefox-bin D 00000000ffffffff 4888 4737 4680
[ 3933.879285] ffff880079c09668 0000000000000046 0000000000000000 0002005079c095e8
[ 3933.879285] ffff880079c095d8 ffffffff8081c780 ffffffff8081c780 ffff88007afc4040
[ 3933.879285] ffffffff807093b0 ffff88007afc4398 0000000079c09618 ffff88007afc4398
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569fad>] ? _spin_unlock_irqrestore+0x5e/0x6c
[ 3933.879285] [<ffffffff8031371b>] do_get_write_access+0x27c/0x4c7
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff80300063>] ext3_new_blocks+0x33f/0x5b9
[ 3933.879285] [<ffffffff805689b3>] ? __mutex_lock_common+0x3f5/0x406
[ 3933.879285] [<ffffffff803044fc>] ext3_get_blocks_handle+0x3bc/0x891
[ 3933.879285] [<ffffffff8036867f>] ? blk_rq_map_sg+0x1cf/0x2ed
[ 3933.879285] [<ffffffff80367cdb>] ? blk_recalc_rq_segments+0x36/0x1a3
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff80304a94>] ext3_get_block+0xc3/0x101
[ 3933.879285] [<ffffffff802c8be5>] __block_prepare_write+0x21b/0x3ca
[ 3933.879285] [<ffffffff803049d1>] ? ext3_get_block+0x0/0x101
[ 3933.879285] [<ffffffff802c8f06>] block_write_begin+0x9e/0x102
[ 3933.879285] [<ffffffff80303f96>] ext3_write_begin+0xeb/0x1e1
[ 3933.879285] [<ffffffff803049d1>] ? ext3_get_block+0x0/0x101
[ 3933.879285] [<ffffffff8027d13e>] ? iov_iter_fault_in_readable+0x3d/0x83
[ 3933.879285] [<ffffffff8027f21b>] generic_file_buffered_write+0x143/0x35b
[ 3933.879285] [<ffffffff803817a1>] ? _raw_spin_lock+0xcf/0x18a
[ 3933.879285] [<ffffffff8027f860>] __generic_file_aio_write_nolock+0x274/0x2a8
[ 3933.879285] [<ffffffff802b8ab7>] ? file_update_time+0xc6/0x101
[ 3933.879285] [<ffffffff802800f9>] generic_file_aio_write+0x78/0xd4
[ 3933.879285] [<ffffffff80300cd4>] ext3_file_write+0x1e/0x9e
[ 3933.879285] [<ffffffff802a6714>] do_sync_write+0xe7/0x12d
[ 3933.879285] [<ffffffff8024be3f>] ? autoremove_wake_function+0x0/0x38
[ 3933.879285] [<ffffffff8033c97f>] ? security_file_permission+0x11/0x13
[ 3933.879285] [<ffffffff802a696b>] ? rw_verify_area+0xe4/0x10c
[ 3933.879285] [<ffffffff802a7123>] vfs_write+0xd9/0x137
[ 3933.879285] [<ffffffff802a7242>] sys_write+0x47/0x6c
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] yum-updatesd- D 00000000ffffffff 4536 6218 2539
[ 3933.879285] ffff880079e23bb8 0000000000000046 ffff880079e23b18 ffff88007dcf54d8
[ 3933.879285] ffff880079e23b68 ffffffff8081c780 ffffffff8081c780 ffff8800784f97f0
[ 3933.879285] ffffffff807093b0 ffff8800784f9b48 0000000079e23b48 ffff8800784f9b48
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff8027e44d>] sync_page+0x78/0x7f
[ 3933.879285] [<ffffffff8027e462>] sync_page_killable+0xe/0x56
[ 3933.879285] [<ffffffff8056828e>] __wait_on_bit_lock+0x45/0x79
[ 3933.879285] [<ffffffff8027e454>] ? sync_page_killable+0x0/0x56
[ 3933.879285] [<ffffffff8027e2d2>] __lock_page_killable+0x63/0x6a
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff8027e1ec>] ? find_get_page+0xe8/0x101
[ 3933.879285] [<ffffffff8027e31f>] lock_page_killable+0x46/0x4e
[ 3933.879285] [<ffffffff8027fc77>] generic_file_aio_read+0x3e3/0x605
[ 3933.879285] [<ffffffff80257b2e>] ? debug_mutex_free_waiter+0x2a/0x64
[ 3933.879285] [<ffffffff802a6841>] do_sync_read+0xe7/0x12d
[ 3933.879285] [<ffffffff80376b5e>] ? _atomic_dec_and_lock+0x5a/0x98
[ 3933.879285] [<ffffffff8024be3f>] ? autoremove_wake_function+0x0/0x38
[ 3933.879285] [<ffffffff8033c97f>] ? security_file_permission+0x11/0x13
[ 3933.879285] [<ffffffff802a696b>] ? rw_verify_area+0xe4/0x10c
[ 3933.879285] [<ffffffff802a733d>] vfs_read+0xd6/0x133
[ 3933.879285] [<ffffffff802a73f6>] sys_pread64+0x5c/0x7a
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] GrabWeather D 00000000ffffffff 5624 6239 3250
[ 3933.879285] ffff880079e2fb28 0000000000000046 ffff88007f2e3cc0 00000000000006d0
[ 3933.879285] ffff880079e2fb38 ffffffff8081c780 ffffffff8081c780 ffff880079dbf040
[ 3933.879285] ffffffff807093b0 ffff880079dbf398 0000000079e2fad8 ffff880079dbf398
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569fad>] ? _spin_unlock_irqrestore+0x5e/0x6c
[ 3933.879285] [<ffffffff8031371b>] do_get_write_access+0x27c/0x4c7
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff80308dde>] ext3_orphan_add+0xa1/0x17d
[ 3933.879285] [<ffffffff8031423e>] ? journal_start+0xb7/0xef
[ 3933.879285] [<ffffffff80302ff5>] ext3_setattr+0x162/0x201
[ 3933.879285] [<ffffffff802ba510>] notify_change+0x216/0x35e
[ 3933.879285] [<ffffffff802a5ae0>] do_truncate+0x63/0x81
[ 3933.879285] [<ffffffff802ae39d>] ? vfs_dq_init+0x5e/0x68
[ 3933.879285] [<ffffffff802b0c0b>] may_open+0x19f/0x1da
[ 3933.879285] [<ffffffff802b1259>] do_filp_open+0x40f/0x806
[ 3933.879285] [<ffffffff802b5dc6>] ? __d_free+0x76/0x7b
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] refile D 00000000ffffffff 5992 6248 6247
[ 3933.879285] ffff880079d3bb68 0000000000000046 000000030006bac8 0000000000000001
[ 3933.879285] ffff880079d3bad8 ffffffff8081c780 ffffffff8081c780 ffff880079dbf7f0
[ 3933.879285] ffff88007f2327f0 ffff880079dbfb48 0000000100000000 ffff880079dbfb48
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff8034092d>] ? avc_has_perm+0x4e/0x60
[ 3933.879285] [<ffffffff8056883e>] __mutex_lock_common+0x280/0x406
[ 3933.879285] [<ffffffff80314062>] ? start_this_handle+0x3b3/0x3d0
[ 3933.879285] [<ffffffff80568a56>] __mutex_lock_slowpath+0x14/0x16
[ 3933.879285] [<ffffffff80568a7d>] mutex_lock+0x25/0x29
[ 3933.879285] [<ffffffff802a8172>] lock_super+0x27/0x29
[ 3933.879285] [<ffffffff80308d65>] ext3_orphan_add+0x28/0x17d
[ 3933.879285] [<ffffffff8031423e>] ? journal_start+0xb7/0xef
[ 3933.879285] [<ffffffff80302ff5>] ext3_setattr+0x162/0x201
[ 3933.879285] [<ffffffff802ba510>] notify_change+0x216/0x35e
[ 3933.879285] [<ffffffff802a5ae0>] do_truncate+0x63/0x81
[ 3933.879285] [<ffffffff802ae39d>] ? vfs_dq_init+0x5e/0x68
[ 3933.879285] [<ffffffff802b0c0b>] may_open+0x19f/0x1da
[ 3933.879285] [<ffffffff802b1259>] do_filp_open+0x40f/0x806
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] sendmail D ffff88007e755c00 5624 6266 2416
[ 3933.879285] ffff880079e5fa78 0000000000000046 ffff88007dcf54d8 ffff88007dcf54d8
[ 3933.879285] ffff880079e5f9e8 ffffffff8081c780 ffffffff8081c780 ffff880077bb9040
[ 3933.879285] ffff8800790a3040 ffff880077bb9398 0000000079e5fa18 ffff880077bb9398
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3933.879285] [<ffffffff8056828e>] __wait_on_bit_lock+0x45/0x79
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff8056832c>] out_of_line_wait_on_bit_lock+0x6a/0x77
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff802c8364>] __lock_buffer+0x25/0x27
[ 3933.879285] [<ffffffff8031283d>] lock_buffer+0x44/0x48
[ 3933.879285] [<ffffffff80313511>] do_get_write_access+0x72/0x4c7
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80317a12>] ? bit_spin_unlock+0x6b/0x79
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff8030177d>] ext3_new_inode+0x5e0/0x9fd
[ 3933.879285] [<ffffffff803088a4>] ext3_create+0x99/0x11b
[ 3933.879285] [<ffffffff802b0cbf>] vfs_create+0x79/0xa0
[ 3933.879285] [<ffffffff802b10c6>] do_filp_open+0x27c/0x806
[ 3933.879285] [<ffffffff802aa156>] ? cp_new_stat+0xe2/0xef
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] hal-acl-tool D 00000000ffffffff 5896 6295 2110
[ 3933.879285] ffff8800773fda78 0000000000000046 ffff88007dcf54d8 ffff88007dcf54d8
[ 3933.879285] ffff8800773fd9e8 ffffffff8081c780 ffffffff8081c780 ffff880079118040
[ 3933.879285] ffff88007f2327f0 ffff880079118398 00000001773fda18 ffff880079118398
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3933.879285] [<ffffffff8056828e>] __wait_on_bit_lock+0x45/0x79
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff8056832c>] out_of_line_wait_on_bit_lock+0x6a/0x77
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff802c667f>] ? put_bh+0x49/0x56
[ 3933.879285] [<ffffffff802c8364>] __lock_buffer+0x25/0x27
[ 3933.879285] [<ffffffff8031283d>] lock_buffer+0x44/0x48
[ 3933.879285] [<ffffffff80313511>] do_get_write_access+0x72/0x4c7
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80317a12>] ? bit_spin_unlock+0x6b/0x79
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff80301676>] ext3_new_inode+0x4d9/0x9fd
[ 3933.879285] [<ffffffff803088a4>] ext3_create+0x99/0x11b
[ 3933.879285] [<ffffffff802b0cbf>] vfs_create+0x79/0xa0
[ 3933.879285] [<ffffffff802b10c6>] do_filp_open+0x27c/0x806
[ 3933.879285] [<ffffffff8023a6ea>] ? do_wait+0x33c/0x464
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
On Thu, 20 Nov 2008 18:05:47 -0500
[email protected] wrote:
> On Thu, 20 Nov 2008 15:09:35 +0900, Tetsuo Handa said:
> > Hello.
> >
> > > The mm-of-the-moment snapshot 2008-11-19-02-19 has been uploaded to
> > Recent mmotm randomly freezes on /sbin/modprobe and read(). 2.6.28-rc2-mm1 was OK.
>
> I'm seeing very similar hangs on -mmotm-11-17 as well. I've hit it 3 times
> today, all while disk activity was moderately heavy (things like 'yum update',
> or a 'find . | xargs grep', and so on).
>
> Managed to catch one while netconsole was active - I didn't have any messages
> for bugs/warns/oopsen. Apparently, somebody is holding a lock. (I also have an
> alt-sysrq-t from this incident, but that's about 10 times as big, didn't want
> to abuse vger too much.. ;)
>
> [ 3932.912494] SysRq : Show Blocked State
> [ 3932.913465] task PC stack pid father
> [ 3932.913465] pdflush D ffff88007e247cf0 5776 303 2
> [ 3932.913465] ffff88007e247c50 0000000000000002 ffff88007e247bb0 ffff88007dcf54d8
> [ 3932.913465] ffff88007e247c00 ffffffff8081c780 ffffffff8081c780 ffff88007f269040
> [ 3932.913465] ffff88007f232040 ffff88007f269398 000000007e247be0 ffff88007f269398
> [ 3932.913465] Call Trace:
> [ 3932.913465] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
> [ 3932.913465] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
> [ 3932.913465] [<ffffffff8027e44d>] sync_page+0x78/0x7f
> [ 3932.913465] [<ffffffff80568380>] __wait_on_bit+0x47/0x79
> [ 3932.913465] [<ffffffff8027e3d5>] ? sync_page+0x0/0x7f
> [ 3932.913465] [<ffffffff8027e5ed>] wait_on_page_bit+0x6e/0x75
> [ 3932.913465] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
> [ 3932.913465] [<ffffffff80286343>] ? pagevec_lookup_tag+0x22/0x2b
> [ 3932.913465] [<ffffffff8027eed9>] wait_on_page_writeback_range+0x75/0x13d
> [ 3932.913465] [<ffffffff8027efc1>] filemap_fdatawait+0x20/0x22
> [ 3932.913465] [<ffffffff8027f0cc>] filemap_write_and_wait+0x27/0x33
> [ 3932.913465] [<ffffffff802c7c32>] sync_blockdev+0x1b/0x1d
> [ 3932.913465] [<ffffffff802c2242>] __sync_inodes+0x74/0xbf
> [ 3932.913465] [<ffffffff802c22a6>] sync_inodes+0x19/0x33
> [ 3932.913465] [<ffffffff802c5378>] do_sync+0x1a/0x77
> [ 3932.913465] [<ffffffff80285a4c>] pdflush+0x145/0x1f8
> [ 3932.913465] [<ffffffff802c535e>] ? do_sync+0x0/0x77
> [ 3932.913465] [<ffffffff80285907>] ? pdflush+0x0/0x1f8
> [ 3932.913465] [<ffffffff8024ba10>] kthread+0x49/0x76
> [ 3932.913465] [<ffffffff8020cb79>] child_rip+0xa/0x11
> [ 3932.913465] [<ffffffff8020bfe5>] ? restore_args+0x0/0x30
> [ 3932.913465] [<ffffffff8024b9c7>] ? kthread+0x0/0x76
> [ 3932.913465] [<ffffffff8020cb6f>] ? child_rip+0x0/0x11
The traditional cause of the above trace is that someone mucked up the
block/driver/irq-routing layer and we lost an IO completion.
It's also of course possible (but less common) that someone mucked up
the VFS. It would be interesting to revert
do_mpage_readpage-dont-submit-lots-of-small-bios-on-boundary.patch.
On Thu, 20 Nov 2008 15:20:54 PST, Andrew Morton said:
> The traditional cause of the above trace is that someone mucked up the
> block/driver/irq-routing layer and we lost an IO completion.
Yes, that would explain all the symptoms and tracebacks - everybody comes
to a screeching halt the next time they try to go to disk, while the actual
disk drive is showing zero activity.
> It's also of course possible (but less common) that someone mucked up
> the VFS. It would be interesting to revert
> do_mpage_readpage-dont-submit-lots-of-small-bios-on-boundary.patch.
I'm seeing an MTBF of about 2-3 hours when actually applying an I/O load to the
system. I'll try reverting that patch, and if it survives an entire day or
two it will be pretty strong circumstantial evidence that patch is the culprit...
Hi.
On Fri, 21 Nov 2008 00:23:18 -0500, [email protected] wrote:
> On Thu, 20 Nov 2008 15:20:54 PST, Andrew Morton said:
>
> > The traditional cause of the above trace is that someone mucked up the
> > block/driver/irq-routing layer and we lost an IO completion.
>
> Yes, that would explain all the symptoms and tracebacks - everybody comes
> to a screeching halt the next time they try to go to disk, while the actual
> disk drive is showing zero activity.
>
> > It's also of course possible (but less common) that someone mucked up
> > the VFS. It would be interesting to revert
> > do_mpage_readpage-dont-submit-lots-of-small-bios-on-boundary.patch.
>
> I'm seeing an MTBF of about 2-3 hours when actually applying an I/O load to the
> system. I'll try reverting that patch, and if it survives an entire day or
> two it will be pretty strong circumstantial evidence that patch is the culprit...
>
Just FYI, I had seen similar errors with recent mmotms,
but current mmotm(2008-11-20-17-03) seems more stable in my environment.
Thanks,
Daisuke Nishimura.
On Thu, 20 Nov 2008 15:20:54 PST, Andrew Morton said:
> It's also of course possible (but less common) that someone mucked up
> the VFS. It would be interesting to revert
> do_mpage_readpage-dont-submit-lots-of-small-bios-on-boundary.patch.
Reverted that one, managed to provoke a wedge-up this morning anyhow.
So it's something else.
I may have to go and find a reproducer good enough to make bisection plausible.
Hello.
mmotm 2008-11-24-16-52 booted fine. But a patch for below BUG is missing.
http://kerneltrap.org/mailarchive/linux-netdev/2008/11/22/4204804
----------
INIT: version 2.86 booting
BUG: using smp_processor_id() in preemptible [00000000] code: hotplug/1159
caller is sock_prot_inuse_add+0x1c/0x40
Pid: 1159, comm: hotplug Not tainted 2.6.28-rc6-mm1 #1
Call Trace:
[<c0200ba4>] debug_smp_processor_id+0xc4/0xe0
[<c027cb7c>] sock_prot_inuse_add+0x1c/0x40
[<f8047821>] unix_create1+0x101/0x120 [unix]
[<f8047875>] unix_create+0x35/0x70 [unix]
[<c0279077>] __sock_create+0xf7/0x250
[<c0279208>] sock_create+0x38/0x50
[<c0279298>] sys_socket+0x38/0x70
[<c027a1fd>] sys_socketcall+0x5d/0x270
[<c0191fec>] sys_getcwd+0x11c/0x150
[<c011af30>] do_page_fault+0x0/0x450
[<c01034e6>] syscall_call+0x7/0xb
BUG: using smp_processor_id() in preemptible [00000000] code: hotplug/1159
caller is sock_prot_inuse_add+0x1c/0x40
BUG: using smp_processor_id() in preemptible [00000000] code: hotplug/1156
caller is sock_prot_inuse_add+0x1c/0x40
Pid: 1156, comm: hotplug Not tainted 2.6.28-rc6-mm1 #1
Call Trace:
[<c0200ba4>] debug_smp_processor_id+0xc4/0xe0
[<c027cb7c>] sock_prot_inuse_add+0x1c/0x40
[<f8047821>] unix_create1+0x101/0x120 [unix]
[<f8047875>] unix_create+0x35/0x70 [unix]
[<c0279077>] __sock_create+0xf7/0x250
[<c0279208>] sock_create+0x38/0x50
[<c0279298>] sys_socket+0x38/0x70
[<c027a1fd>] sys_socketcall+0x5d/0x270
[<c0191fec>] sys_getcwd+0x11c/0x150
[<c011af30>] do_page_fault+0x0/0x450
[<c01034e6>] syscall_call+0x7/0xb