The bug manifests as a soft lockup in kswapd:
[ 155.759084] netconsole: network logging started
[ 598.920430] BUG: soft lockup - CPU#1 stuck for 67s! [kswapd0:46]
[ 598.920472] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole]
[ 598.920834] CPU 1
[ 598.920843] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole]
[ 598.926818]
[ 598.928043] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 598.929299] RIP: 0010:[<ffffffffa007ff9b>] [<ffffffffa007ff9b>] i915_gem_inactive_shrink+0x78/0x194 [i915]
[ 598.930603] RSP: 0018:ffff8800709ebd50 EFLAGS: 00000216
[ 598.931867] RAX: ffff88006ec5c6b0 RBX: 00000000000000c0 RCX: 0000000000000000
[ 598.933135] RDX: ffff880037e59638 RSI: 0000000000000000 RDI: ffff880037876020
[ 598.934408] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 000000000001bd90
[ 598.935680] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e
[ 598.936944] R13: ffff8800709ebcf0 R14: ffff8800709ebcf8 R15: ffffffff810dd64d
[ 598.938221] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
[ 598.939525] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 598.940813] CR2: 00007fe16a34d380 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 598.942114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 598.943441] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 598.944744] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006df8ae00)
[ 598.946044] Stack:
[ 598.947317] ffff8800709ebd90 ffff880037e59638 ffff8800709ebd60 ffff880037e595f0
[ 598.948654] 0000000000000000 0000000000000000 00000000000000d0 000000000004c24e
[ 598.949986] ffff8800709ebde0 ffffffff810e1f89 000000000000003d 0000000000000080
[ 598.951329] Call Trace:
[ 598.952673] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 598.954003] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 598.955331] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 598.956683] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 598.958022] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 598.959379] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 598.960709] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 598.962032] Code: 31 ed 48 83 c3 48 48 2d b0 00 00 00 eb 0a 48 8d 82 50 ff ff ff 41 ff c5 48 8b 90 b0 00 00 00 48 05 b0 00 00 00 48 39 d8 0f 18 0a
[ 598.962192] e1 e9 da 00 00 00 4c 89 f7 e8 c6 fd ff ff 48 8b 43 48 4c 8b
[ 598.965009] Call Trace:
[ 598.966421] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 598.966423] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 598.966426] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 598.966429] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 598.966432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 598.966435] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 598.966438] [<ffffffff81483760>] ? gs_change+0x13/0x13
The traces are slightly different each reboot cycle, but it's always in
kswapd and usually in shrink_slab. Once it happens, anything that
touches the filesystem hangs in D wait, so the machine is basically
toast.
The box is a Lenovo T420s sandybridge core i5 based laptop with 2GB of
memory.
There is a corresponding Red Hat bugzilla report here:
https://bugzilla.redhat.com/show_bug.cgi?id=694818
And I've verified that the bug also shows in the 2.6.38.3 stable kernel.
If anyone wants me to try anything, please let me know.
James
Excerpts from James Bottomley's message of 2011-04-27 12:09:13 -0400:
> The bug manifests as a soft lockup in kswapd:
>
> [ 155.759084] netconsole: network logging started
> [ 598.920430] BUG: soft lockup - CPU#1 stuck for 67s! [kswapd0:46]
> [ 598.920472] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole]
> [ 598.920834] CPU 1
> [ 598.920843] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole]
> [ 598.926818]
Probably easier to debug with a sysrq-l and sysrq-w. If you get stuck
on the filesystem, it is probably waiting on ram, which it probably
can't get because kswapd is spinning. Eventually everyone backs up
waiting for the transaction that never ends. If we're really lucky it
is just GFP_KERNEL where it should NOFS.
Since you're often stuck in different spots inside shrink_slab, we're
probably not stuck on a lock. But, trying with lock debugging, lockdep
enabled and preempt on is a good idea to rule out locking mistakes.
Does the fedora debug kernel enable preempt?
-chris
On Wed, 2011-04-27 at 12:33 -0400, Chris Mason wrote:
> Probably easier to debug with a sysrq-l and sysrq-w.
Yes, just enabling them for the repro ... I didn't echo 1
> /proc/sys/kernel/sysrq first time around
> If you get stuck
> on the filesystem, it is probably waiting on ram, which it probably
> can't get because kswapd is spinning. Eventually everyone backs up
> waiting for the transaction that never ends. If we're really lucky it
> is just GFP_KERNEL where it should NOFS.
>
> Since you're often stuck in different spots inside shrink_slab, we're
> probably not stuck on a lock. But, trying with lock debugging, lockdep
> enabled and preempt on is a good idea to rule out locking mistakes.
>
> Does the fedora debug kernel enable preempt?
No ... I've taken the Fedora 15 kernel config directly from their
install CD for this (with a bit of munging to get it to work on -rc4)
James
Excerpts from James Bottomley's message of 2011-04-27 12:50:00 -0400:
> On Wed, 2011-04-27 at 12:33 -0400, Chris Mason wrote:
> > Probably easier to debug with a sysrq-l and sysrq-w.
>
> Yes, just enabling them for the repro ... I didn't echo 1
> > /proc/sys/kernel/sysrq first time around
>
> > If you get stuck
> > on the filesystem, it is probably waiting on ram, which it probably
> > can't get because kswapd is spinning. Eventually everyone backs up
> > waiting for the transaction that never ends. If we're really lucky it
> > is just GFP_KERNEL where it should NOFS.
> >
> > Since you're often stuck in different spots inside shrink_slab, we're
> > probably not stuck on a lock. But, trying with lock debugging, lockdep
> > enabled and preempt on is a good idea to rule out locking mistakes.
> >
> > Does the fedora debug kernel enable preempt?
>
> No ... I've taken the Fedora 15 kernel config directly from their
> install CD for this (with a bit of munging to get it to work on -rc4)
Ok, I'd try turning it on so we catch the sleeping with a spinlock held
case better.
-chris
On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> case better.
Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
This is the trace with sysrq-l and sysrq-w
The repro this time doesn't have a soft lockup, just the tar is hung and
one of my CPUs is in 99% system.
James
---
[ 453.351255] SysRq : Show backtrace of all active CPUs
[ 453.352601] sending NMI to all CPUs:
[ 453.353849] NMI backtrace for cpu 3
[ 453.355545] CPU 3
[ 453.355560] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.363188]
[ 453.365162] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.367133] RIP: 0010:[<ffffffff8147af8c>] [<ffffffff8147af8c>] mutex_trylock+0x16/0x38
[ 453.369122] RSP: 0018:ffff88006dfc1d40 EFLAGS: 00000246
[ 453.371098] RAX: 0000000000000001 RBX: ffff880037de15f0 RCX: 0000000000000001
[ 453.373099] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037875820
[ 453.375097] RBP: ffff88006dfc1d40 R08: 0000000000000000 R09: 00000000000074ad
[ 453.377079] R10: 0000000000000002 R11: ffffffff81a44e50 R12: 0000000000000000
[ 453.379052] R13: 0000000000000000 R14: ffff880037875800 R15: ffff880037875820
[ 453.381015] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
[ 453.382985] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.384944] CR2: 00007fbf8ea8d090 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 453.386920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.388887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.390789] Process kswapd0 (pid: 46, threadinfo ffff88006dfc0000, task ffff88006dfb8000)
[ 453.392649] Stack:
[ 453.394473] ffff88006dfc1d90 ffffffffa007ff52 ffff88006dfc1d90 ffffffff811613b5
[ 453.396337] ffff88006dfc1d60 ffff880037de15f0 0000000000000000 0000000000000000
[ 453.398174] 00000000000000d0 000000000006366c ffff88006dfc1de0 ffffffff810e1f89
[ 453.400015] Call Trace:
[ 453.401845] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
[ 453.403702] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
[ 453.405499] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 453.407234] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 453.408952] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 453.410690] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 453.412432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 453.414187] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 453.415945] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 453.417671] Code: 00 48 c7 47 18 00 00 00 00 f0 ff 07 7f 05 e8 ed 02 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 b9 01 00 00 00 31 d2 89 c8 f0 0f b1 17
[ 453.417905] c1 31 c0 ff c9 75 18 65 48 8b 04 25 c8 cc 00 00 48 2d d8 1f
[ 453.421534] Call Trace:
[ 453.423337] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
[ 453.425172] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
[ 453.426997] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 453.428818] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 453.430639] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 453.432485] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 453.434298] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 453.436112] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 453.437894] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 453.439654] NMI backtrace for cpu 2
[ 453.441508] CPU 2
[ 453.441525] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.449517]
[ 453.451516] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.453704] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 453.455772] RSP: 0018:ffff8800715c9e68 EFLAGS: 00000046
[ 453.457827] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
[ 453.459903] RDX: 0000000000000000 RSI: ffff8800715c9fd8 RDI: ffffffff81a0e640
[ 453.461999] RBP: ffff8800715c9eb8 R08: 000000000000006d R09: 00000000000003e5
[ 453.464039] R10: ffffffff00000002 R11: ffff880100293b40 R12: 0000000000000030
[ 453.466013] R13: 12187898d4512537 R14: 0000000000000004 R15: 0000000000000002
[ 453.467918] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 453.469797] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.471644] CR2: 0000000000452630 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 453.473512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.475356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.477185] Process kworker/0:1 (pid: 0, threadinfo ffff8800715c8000, task ffff8800715a1700)
[ 453.479024] Stack:
[ 453.480855] ffff8800715c9e88 ffffffff810731c0 ffff880100291290 0000000000011290
[ 453.482739] ffff8800715c9eb8 000000028139c97a ffffe8ffffc80170 ffffe8ffffc80170
[ 453.484631] ffffe8ffffc80300 0000000000000000 ffff8800715c9ef8 ffffffff8139b868
[ 453.486518] Call Trace:
[ 453.488365] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.490226] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.492096] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.493947] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.495791] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 453.499958] Call Trace:
[ 453.501895] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.503837] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.505775] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.507687] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.509598] NMI backtrace for cpu 1
[ 453.511390] CPU 1
[ 453.511405] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.519288]
[ 453.521289] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.523353] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 453.525377] RSP: 0018:ffff880071587e68 EFLAGS: 00000046
[ 453.527353] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
[ 453.529332] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640
[ 453.531303] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5
[ 453.533276] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010
[ 453.535249] R13: 12187898d4512ee3 R14: 0000000000000002 R15: 0000000000000001
[ 453.537229] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
[ 453.539224] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.541210] CR2: 00000037d9071fa0 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 453.543220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.545233] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.547239] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700)
[ 453.549274] Stack:
[ 453.551289] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290
[ 453.553301] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170
[ 453.555257] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868
[ 453.557156] Call Trace:
[ 453.558980] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.560817] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.562635] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.564438] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.566227] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 453.570292] Call Trace:
[ 453.572196] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.574117] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.576031] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.577924] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.579811] NMI backtrace for cpu 0
[ 453.581279] CPU 0
[ 453.581289] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.587576]
[ 453.589160] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.590777] RIP: 0010:[<ffffffff8100f0fa>] [<ffffffff8100f0fa>] native_read_tsc+0x1/0x14
[ 453.592390] RSP: 0018:ffff880100203b98 EFLAGS: 00000883
[ 453.594001] RAX: 00000000f9ab6980 RBX: 0000000000002710 RCX: 0000000000000040
[ 453.595624] RDX: 000000000026066c RSI: 0000000000000100 RDI: 000000000026066d
[ 453.597250] RBP: ffff880100203ba8 R08: 000000008b000052 R09: 0000000000000000
[ 453.598872] R10: 0000000000000000 R11: 0000000000000003 R12: 000000000026066d
[ 453.600437] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000001
[ 453.601924] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000
[ 453.603434] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.604942] CR2: 0000000000440360 CR3: 0000000001a03000 CR4: 00000000000406f0
[ 453.606460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.607981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.609597] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020)
[ 453.611134] Stack:
[ 453.612655] ffff880100203ba8 ffffffff81232dfe ffff880100203bd8 ffffffff81232ecb
[ 453.614176] 0000000000002710 0000000000000008 000000000000006c 0000000000000002
[ 453.615656] ffff880100203be8 ffffffff81232e49 ffff880100203bf8 ffffffff81232e77
[ 453.617101] Call Trace:
[ 453.618492] <IRQ>
[ 453.619874] [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
[ 453.621251] [<ffffffff81232ecb>] delay_tsc+0x27/0x74
[ 453.622613] [<ffffffff81232e49>] __delay+0xf/0x11
[ 453.623973] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
[ 453.625322] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 453.626673] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 453.628027] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 453.629372] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 453.630706] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 453.632028] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 453.633342] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 453.634655] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 453.635978] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 453.637283] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 453.638563] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 453.639830] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 453.641092] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 453.642354] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 453.643598] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 453.644832] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 453.646069] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 453.647299] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 453.648527] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 453.649695] <EOI>
[ 453.650794] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 453.651911] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 453.653025] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 453.654129] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.655231] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.656329] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 453.657417] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 453.658514] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 453.659576] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 453.660589] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 453.661559] Code: 21 00 00 e8 74 3d 22 00 5d c3 90 90 90 55 40 88 f8 48 89 e5 e6 70 e4 71 5d c3 55 40 88 f0 48 89 e5 e6 70 40 88 f8 e6 71 5d c3 55
[ 453.661721] 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 55 b2
[ 453.663728] Call Trace:
[ 453.664713] <IRQ> [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
[ 453.665710] [<ffffffff81232ecb>] delay_tsc+0x27/0x74
[ 453.666698] [<ffffffff81232e49>] __delay+0xf/0x11
[ 453.667686] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
[ 453.668675] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 453.669676] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 453.670681] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 453.671676] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 453.672662] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 453.673646] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 453.674627] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 453.675614] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 453.676587] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 453.677550] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 453.678498] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 453.679431] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 453.680351] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 453.681270] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 453.682186] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 453.683100] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 453.684014] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 453.684924] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 453.685831] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 453.686740] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 453.687655] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 453.688573] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 453.689480] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.690387] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.691293] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 453.692195] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 453.693095] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 453.693996] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 453.694905] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 454.680802] SysRq : Show Blocked State
[ 454.683427] task PC stack pid father
[ 454.686058] systemd D 0000000000000000 0 1 0 0x00000000
[ 454.688752] ffff8801003bdcd8 0000000000000082 ffff8801003bdc88 ffffffff00000000
[ 454.691491] ffff880100370000 ffff8801003bdfd8 ffff8801003bdfd8 0000000000013b40
[ 454.694228] ffff8800715e1700 ffff880100370000 ffff8801003bdcb8 00000001811329db
[ 454.696969] Call Trace:
[ 454.699683] [<ffffffff8147ac4b>] schedule_timeout+0x34/0xde
[ 454.702447] [<ffffffff810ad618>] ? __call_rcu+0x123/0x12c
[ 454.705184] [<ffffffff810ad64d>] ? call_rcu_sched+0x15/0x17
[ 454.707906] [<ffffffff8147aa14>] wait_for_common+0xac/0x101
[ 454.710656] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226
[ 454.713445] [<ffffffff8147ab1d>] wait_for_completion+0x1d/0x1f
[ 454.716199] [<ffffffff810ada5a>] synchronize_sched+0x5a/0x5c
[ 454.718972] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43
[ 454.721766] [<ffffffff81090d05>] cgroup_diput+0x37/0xe3
[ 454.724562] [<ffffffff81090cce>] ? parse_cgroupfs_options+0x353/0x353
[ 454.727328] [<ffffffff8112fc79>] dentry_kill+0xfa/0x121
[ 454.730103] [<ffffffff81130189>] dput+0xdd/0xea
[ 454.732866] [<ffffffff8112aa68>] do_rmdir+0xc6/0xfe
[ 454.735481] [<ffffffff8111dc78>] ? filp_close+0x6e/0x7a
[ 454.737990] [<ffffffff8112b32f>] sys_rmdir+0x16/0x18
[ 454.740469] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
[ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
[ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
[ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
[ 454.753048] Call Trace:
[ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
[ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
[ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
[ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
[ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
[ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
[ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
[ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
[ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
[ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
[ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
[ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
[ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
[ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
[ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
[ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
[ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
[ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
[ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
[ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
[ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
[ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
[ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
[ 454.839453] Call Trace:
[ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
[ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
[ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
[ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
[ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
[ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
[ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
[ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
[ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
[ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
[ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000
[ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282
[ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40
[ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3
[ 454.899530] Call Trace:
[ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
[ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
[ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
[ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
[ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
[ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
[ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
[ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e
[ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204
[ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a
[ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
[ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
[ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
[ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
[ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
[ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
[ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3
[ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e
[ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 454.964732] dhclient-script D 0000000000000000 0 2856 2855 0x00000000
[ 454.967360] ffff88006e1f5b18 0000000000000082 ffff8800378da880 0000000000000000
[ 454.970056] ffff88006deb1700 ffff88006e1f5fd8 ffff88006e1f5fd8 0000000000013b40
[ 454.972706] ffff880071589700 ffff88006deb1700 ffff88006e1f5ad8 000000016e1f5ad8
[ 454.975323] Call Trace:
[ 454.977882] [<ffffffff810d5916>] ? sleep_on_page+0x12/0x12
[ 454.980477] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 454.983063] [<ffffffff810d5924>] sleep_on_page_killable+0xe/0x3b
[ 454.985622] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
[ 454.988182] [<ffffffff810d5819>] __lock_page_killable+0x66/0x68
[ 454.990785] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.993436] [<ffffffff810d5859>] lock_page_killable+0x3e/0x43
[ 454.996099] [<ffffffff810d71ea>] generic_file_aio_read+0x463/0x640
[ 454.998730] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
[ 455.001383] [<ffffffff811ebc34>] ? avc_has_perm+0x51/0x63
[ 455.004012] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 455.006644] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 455.009289] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
[ 455.011917] [<ffffffff811233eb>] kernel_read+0x41/0x4f
[ 455.014511] [<ffffffff811234dd>] prepare_binprm+0xe4/0xe8
[ 455.017094] [<ffffffff81124d40>] do_execve+0x114/0x277
[ 455.019678] [<ffffffff8100ff91>] sys_execve+0x43/0x5a
[ 455.022281] [<ffffffff81482a9c>] stub_execve+0x6c/0xc0
[ 455.024888] Sched Debug Version: v0.10, 2.6.39-rc4+ #1
[ 455.027487] ktime : 455879.226285
[ 455.030142] sched_clk : 455024.886257
[ 455.032786] cpu_clk : 455024.886397
[ 455.035352] jiffies : 4295123185
[ 455.037904] sched_clock_stable : 1
[ 455.040413]
[ 455.042892] sysctl_sched
[ 455.045306] .sysctl_sched_latency : 18.000000
[ 455.047775] .sysctl_sched_min_granularity : 2.250000
[ 455.050206] .sysctl_sched_wakeup_granularity : 3.000000
[ 455.052643] .sysctl_sched_child_runs_first : 0
[ 455.055034] .sysctl_sched_features : 7279
[ 455.057423] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[ 455.059829]
[ 455.059830] cpu#0, 2491.994 MHz
[ 455.064443] .nr_running : 0
[ 455.066757] .load : 0
[ 455.069054] .nr_switches : 146510
[ 455.071353] .nr_load_updates : 233084
[ 455.073642] .nr_uninterruptible : 2
[ 455.075894] .next_balance : 4295.122831
[ 455.078152] .curr->pid : 0
[ 455.080396] .clock : 454680.481348
[ 455.082634] .cpu_load[0] : 0
[ 455.084867] .cpu_load[1] : 0
[ 455.087065] .cpu_load[2] : 0
[ 455.089233] .cpu_load[3] : 0
[ 455.091390] .cpu_load[4] : 0
[ 455.093499] .yld_count : 0
[ 455.095605] .sched_switch : 0
[ 455.097667] .sched_count : 149062
[ 455.099765] .sched_goidle : 62756
[ 455.101781] .avg_idle : 1000000
[ 455.103807] .ttwu_count : 77219
[ 455.105958] .ttwu_local : 74144
[ 455.107957] .bkl_count : 0
[ 455.109914]
[ 455.109915] cfs_rq[0]:/
[ 455.113642] .exec_clock : 20017.048374
[ 455.115515] .MIN_vruntime : 0.000001
[ 455.117353] .min_vruntime : 28900.800090
[ 455.119185] .max_vruntime : 0.000001
[ 455.121028] .spread : 0.000000
[ 455.122820] .spread0 : 0.000000
[ 455.124581] .nr_spread_over : 54
[ 455.126318] .nr_running : 0
[ 455.128045] .load : 0
[ 455.129743] .load_avg : 0.000000
[ 455.131414] .load_period : 0.000000
[ 455.133052] .load_contrib : 0
[ 455.134692] .load_tg : 0
[ 455.136305]
[ 455.136306] runnable tasks:
[ 455.136307] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.136310] ----------------------------------------------------------------------------------------------------------
[ 455.142856]
[ 455.142857] cpu#1, 2491.994 MHz
[ 455.146215] .nr_running : 0
[ 455.147931] .load : 0
[ 455.149648] .nr_switches : 62175
[ 455.151348] .nr_load_updates : 217324
[ 455.153045] .nr_uninterruptible : 2
[ 455.154747] .next_balance : 4295.123306
[ 455.156457] .curr->pid : 0
[ 455.158149] .clock : 455157.364642
[ 455.159867] .cpu_load[0] : 0
[ 455.161594] .cpu_load[1] : 0
[ 455.163283] .cpu_load[2] : 0
[ 455.164954] .cpu_load[3] : 0
[ 455.166575] .cpu_load[4] : 0
[ 455.168185] .yld_count : 60
[ 455.169791] .sched_switch : 0
[ 455.171401] .sched_count : 62899
[ 455.172984] .sched_goidle : 27394
[ 455.174580] .avg_idle : 1000000
[ 455.176171] .ttwu_count : 30510
[ 455.177739] .ttwu_local : 25277
[ 455.179292] .bkl_count : 0
[ 455.180882]
[ 455.180883] cfs_rq[1]:/
[ 455.183954] .exec_clock : 10655.021809
[ 455.185550] .MIN_vruntime : 0.000001
[ 455.187141] .min_vruntime : 19718.135550
[ 455.188771] .max_vruntime : 0.000001
[ 455.190407] .spread : 0.000000
[ 455.192016] .spread0 : -9182.664540
[ 455.193634] .nr_spread_over : 80
[ 455.195242] .nr_running : 0
[ 455.196848] .load : 0
[ 455.198441] .load_avg : 0.000000
[ 455.200059] .load_period : 0.000000
[ 455.201654] .load_contrib : 0
[ 455.203240] .load_tg : 0
[ 455.204836]
[ 455.204837] runnable tasks:
[ 455.204838] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.204841] ----------------------------------------------------------------------------------------------------------
[ 455.211403]
[ 455.211404] cpu#2, 2491.994 MHz
[ 455.214958] .nr_running : 0
[ 455.216771] .load : 0
[ 455.218564] .nr_switches : 37444
[ 455.220394] .nr_load_updates : 219042
[ 455.222202] .nr_uninterruptible : 0
[ 455.224039] .next_balance : 4295.123375
[ 455.225897] .curr->pid : 0
[ 455.227769] .clock : 455227.204963
[ 455.229681] .cpu_load[0] : 0
[ 455.231548] .cpu_load[1] : 0
[ 455.233353] .cpu_load[2] : 0
[ 455.235059] .cpu_load[3] : 0
[ 455.236676] .cpu_load[4] : 0
[ 455.238280] .yld_count : 0
[ 455.239879] .sched_switch : 0
[ 455.241471] .sched_count : 37815
[ 455.243075] .sched_goidle : 16831
[ 455.244674] .avg_idle : 1000000
[ 455.246270] .ttwu_count : 18348
[ 455.247849] .ttwu_local : 16899
[ 455.249430] .bkl_count : 0
[ 455.250992]
[ 455.250993] cfs_rq[2]:/
[ 455.254040] .exec_clock : 6758.351942
[ 455.255630] .MIN_vruntime : 0.000001
[ 455.257236] .min_vruntime : 13719.942866
[ 455.258861] .max_vruntime : 0.000001
[ 455.260497] .spread : 0.000000
[ 455.262122] .spread0 : -15180.857224
[ 455.263753] .nr_spread_over : 21
[ 455.265389] .nr_running : 0
[ 455.267018] .load : 0
[ 455.268637] .load_avg : 0.000000
[ 455.270292] .load_period : 0.000000
[ 455.271911] .load_contrib : 0
[ 455.273530] .load_tg : 0
[ 455.275163]
[ 455.275165] runnable tasks:
[ 455.275166] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.275169] ----------------------------------------------------------------------------------------------------------
[ 455.281810]
[ 455.281811] cpu#3, 2491.994 MHz
[ 455.285349] .nr_running : 4
[ 455.287149] .load : 2048
[ 455.288952] .nr_switches : 30358
[ 455.290787] .nr_load_updates : 223302
[ 455.292599] .nr_uninterruptible : 1
[ 455.294419] .next_balance : 4295.123506
[ 455.296250] .curr->pid : 46
[ 455.298090] .clock : 431792.394314
[ 455.299953] .cpu_load[0] : 2048
[ 455.301810] .cpu_load[1] : 2048
[ 455.303610] .cpu_load[2] : 2048
[ 455.305311] .cpu_load[3] : 2048
[ 455.306926] .cpu_load[4] : 2048
[ 455.308531] .yld_count : 63
[ 455.310130] .sched_switch : 0
[ 455.311723] .sched_count : 30824
[ 455.313315] .sched_goidle : 13491
[ 455.314904] .avg_idle : 1000000
[ 455.316481] .ttwu_count : 13858
[ 455.318057] .ttwu_local : 12507
[ 455.319629] .bkl_count : 0
[ 455.321187]
[ 455.321188] cfs_rq[3]:/
[ 455.324213] .exec_clock : 5334.144946
[ 455.325795] .MIN_vruntime : 13295.262803
[ 455.327401] .min_vruntime : 13302.523317
[ 455.329046] .max_vruntime : 13295.262803
[ 455.330656] .spread : 0.000000
[ 455.332297] .spread0 : -15598.276773
[ 455.333925] .nr_spread_over : 117
[ 455.335577] .nr_running : 2
[ 455.337204] .load : 2048
[ 455.338823] .load_avg : 0.000000
[ 455.340476] .load_period : 0.000000
[ 455.342094] .load_contrib : 0
[ 455.343702] .load_tg : 0
[ 455.345302]
[ 455.345303] rt_rq[3]:/
[ 455.348416] .rt_nr_running : 1
[ 455.350006] .rt_throttled : 0
[ 455.351587] .rt_time : 0.000000
[ 455.353173] .rt_runtime : 950.000000
[ 455.354759]
[ 455.354760] runnable tasks:
[ 455.354761] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.354764] ----------------------------------------------------------------------------------------------------------
[ 455.361357] migration/3 17 0.000000 1084 0 0.000000 0.000933 0.000000 /
[ 455.363278] watchdog/3 20 0.000000 31 0 0.000000 0.780405 0.002351 /
[ 455.365212] R kswapd0 46 13302.523317 714 120 13302.523317 1148.767369 245855.504389 /
[ 455.367220] kworker/3:1 74 13295.262803 10488 120 13295.262803 324.924994 315669.659686 /
[ 455.369317]
[ cc'd linux-ext4 ]
James is hitting softlockups in kswapd while doing writes to a large
ext4 file.
Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400:
> On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> > Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> > case better.
>
> Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
>
> This is the trace with sysrq-l and sysrq-w
>
> The repro this time doesn't have a soft lockup, just the tar is hung and
> one of my CPUs is in 99% system.
> [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> [ 454.753048] Call Trace:
> [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
So our flusher threads are stuck waiting in
jbd2_journal_get_write_access, which means they aren't cleaning dirty
pages.
In order to get write access, they probably need the transaction to
commit:
> [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> [ 454.839453] Call Trace:
> [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
But that seems to be waiting for a page lock. Probably the same page
lock held by the flusher thread? Looks like tar is stuck in the same
boat down below.
> [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000
> [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282
> [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40
> [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3
> [ 454.899530] Call Trace:
> [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
> [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e
> [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204
> [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a
> [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
> [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
> [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
> [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
> [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
> [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
> [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3
> [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e
> [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
-chris
On Wed, 2011-04-27 at 13:34 -0400, Chris Mason wrote:
> [ cc'd linux-ext4 ]
>
> James is hitting softlockups in kswapd while doing writes to a large
> ext4 file.
>
> Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400:
> > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> > > case better.
> >
> > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
> >
> > This is the trace with sysrq-l and sysrq-w
>
> >
> > The repro this time doesn't have a soft lockup, just the tar is hung and
> > one of my CPUs is in 99% system.
> > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> > [ 454.753048] Call Trace:
> > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
>
> So our flusher threads are stuck waiting in
> jbd2_journal_get_write_access, which means they aren't cleaning dirty
> pages.
>
> In order to get write access, they probably need the transaction to
> commit:
>
> > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> > [ 454.839453] Call Trace:
> > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
>
> But that seems to be waiting for a page lock. Probably the same page
> lock held by the flusher thread? Looks like tar is stuck in the same
> boat down below.
To test the theory, Chris asked me to try with data=ordered.
Unfortunately, the deadlock still shows up. This is what I get.
James
---
[ 263.749738] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
[ 263.751207] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 263.756081] CPU 2
[ 263.756091] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 263.763033]
[ 263.764761] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 263.766509] RIP: 0010:[<ffffffff810e1fa2>] [<ffffffff810e1fa2>] shrink_slab+0x86/0x166
[ 263.768256] RSP: 0018:ffff8800709ebda0 EFLAGS: 00000206
[ 263.769976] RAX: 0000000000000000 RBX: ffff8800709ebde0 RCX: 0000000000000002
[ 263.771683] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81a44e50
[ 263.773378] RBP: ffff8800709ebde0 R08: 0000000000000000 R09: 000000000000a2b2
[ 263.775053] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e
[ 263.776728] R13: ffff8800709ebe58 R14: 0000000000000004 R15: 00000000000178f9
[ 263.778408] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 263.780098] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 263.781777] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 263.783458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 263.785140] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 263.786811] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000)
[ 263.788491] Stack:
[ 263.790153] 000000000000003d 0000000000000080 ffff880000000000 ffff8801005e6700
[ 263.791810] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c
[ 263.793426] ffff8800709ebee0 ffffffff810e4bcc 0000000000000003 ffff88006dfa8000
[ 263.795038] Call Trace:
[ 263.796614] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 263.798183] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 263.799755] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 263.801328] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 263.802904] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 263.804434] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 263.805914] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7
[ 263.806080] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7
[ 263.809185] Call Trace:
[ 263.810748] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 263.812327] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 263.813913] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 263.815486] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 263.817053] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 263.818621] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 272.421176] SysRq : Show backtrace of all active CPUs
[ 272.422748] sending NMI to all CPUs:
[ 272.424304] NMI backtrace for cpu 2
[ 272.426364] CPU 2
[ 272.426380] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 272.435128]
[ 272.437299] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 272.439506] RIP: 0010:[<ffffffff810d9b6e>] [<ffffffff810d9b6e>] zone_watermark_ok_safe+0xa8/0xae
[ 272.441728] RSP: 0018:ffff8800709ebd80 EFLAGS: 00000282
[ 272.443916] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 272.446101] RDX: 0000000000000813 RSI: 0000000000000000 RDI: ffff8801005e6000
[ 272.448258] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 00000000000007f0
[ 272.450362] R10: 0000000000000002 R11: 00000000000000cd R12: ffff8801005e6000
[ 272.452399] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8801005e6000
[ 272.454367] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 272.456306] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 272.458188] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 272.460021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 272.461789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 272.463503] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000)
[ 272.465234] Stack:
[ 272.466929] 0000000000000000 0000000000000000 ffff8800709ebde0 ffffffff810e1e8a
[ 272.468675] ffff880000000002 ffffffff8106e701 ffff880000000002 ffff8801005e6000
[ 272.470406] ffff8801005f9e68 0000000000000002 0000000000000000 0000000000000000
[ 272.472144] Call Trace:
[ 272.473850] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2
[ 272.475582] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77
[ 272.477304] [<ffffffff810e47fe>] kswapd+0x165/0x798
[ 272.479015] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 272.480712] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 272.482414] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 272.484106] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 272.485809] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 272.487507] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 272.489175] Code: a6 00 48 8b 55 d8 8b 4d d0 44 8b 45 c8 7c c0 45 31 c9 4d 85 ed 4d 0f 49 cd 44 89 e6 48 89 df e8 5e f4 ff ff 48 83 c4 20 5b 41 5c
[ 272.489409] 5d 41 5e 5d c3 55 48 89 e5 0f 1f 44 00 00 bf da 00 02 00 e8
[ 272.492911] Call Trace:
[ 272.494651] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2
[ 272.496408] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77
[ 272.498173] [<ffffffff810e47fe>] kswapd+0x165/0x798
[ 272.499926] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 272.501678] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 272.503444] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 272.505198] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 272.506968] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 272.508691] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 272.510355] NMI backtrace for cpu 1
[ 272.511996] CPU 1
[ 272.512011] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 272.519303]
[ 272.521156] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 272.523190] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 272.525110] RSP: 0018:ffff880071587e68 EFLAGS: 00000046
[ 272.527018] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
[ 272.528960] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640
[ 272.530906] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5
[ 272.532800] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010
[ 272.534634] R13: 12187a34a107f726 R14: 0000000000000002 R15: 0000000000000001
[ 272.536399] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
[ 272.538145] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 272.539879] CR2: 00000037da2830e4 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 272.541614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 272.543329] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 272.545026] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700)
[ 272.546756] Stack:
[ 272.548457] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290
[ 272.550225] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170
[ 272.552009] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868
[ 272.553780] Call Trace:
[ 272.555514] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 272.557259] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.559005] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.560755] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 272.562485] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 272.566380] Call Trace:
[ 272.568193] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 272.570031] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.571855] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.573658] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 272.575463] NMI backtrace for cpu 0
[ 272.576865] CPU 0
[ 272.576875] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 272.582932]
[ 272.584474] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 272.586057] RIP: 0010:[<ffffffff81232e6e>] [<ffffffff81232e6e>] __const_udelay+0x23/0x2e
[ 272.587604] RSP: 0018:ffff880100203bf8 EFLAGS: 00000887
[ 272.589120] RAX: 00000000fa1c3100 RBX: 0000000000002710 RCX: 0000000000000040
[ 272.590635] RDX: 000000000026074e RSI: 0000000000000100 RDI: 0000000000418958
[ 272.592139] RBP: ffff880100203bf8 R08: 000000008b000052 R09: 0000000000000000
[ 272.593644] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000008
[ 272.595143] R13: 000000000000006c R14: 0000000000000002 R15: 0000000000000001
[ 272.596635] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000
[ 272.598139] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 272.599637] CR2: 00000037d909e270 CR3: 000000006f2d7000 CR4: 00000000000406f0
[ 272.601151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 272.602669] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 272.604187] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020)
[ 272.605715] Stack:
[ 272.607238] ffff880100203c18 ffffffff8102166e 0000000000000000 ffffffff81a5e690
[ 272.608756] ffff880100203c28 ffffffff812be0ad ffff880100203c68 ffffffff812be310
[ 272.610226] ffff880037d01900 0000000000000026 ffff880037d01900 0000000000000001
[ 272.611653] Call Trace:
[ 272.613026] <IRQ>
[ 272.614388] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 272.615761] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 272.617118] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 272.618465] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 272.619814] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 272.621158] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 272.622497] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 272.623844] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 272.625172] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 272.626489] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 272.627791] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 272.629085] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 272.630383] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 272.631675] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 272.632946] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 272.634205] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 272.635455] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 272.636699] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 272.637927] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 272.639155] <EOI>
[ 272.640374] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 272.641612] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 272.642840] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 272.644012] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.645115] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.646215] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 272.647313] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 272.648411] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 272.649514] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 272.650624] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 272.651725] Code: ff 15 37 2b 82 00 5d c3 55 48 89 e5 0f 1f 44 00 00 65 48 8b 14 25 98 3a 01 00 48 8d 04 bd 00 00 00 00 48 69 d2 fa 00 00 00 f7 e2
[ 272.651891] 8d 7a 01 e8 c3 ff ff ff 5d c3 55 48 89 e5 0f 1f 44 00 00 48
[ 272.654210] Call Trace:
[ 272.655298] <IRQ> [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 272.656372] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 272.657421] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 272.658456] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 272.659480] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 272.660486] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 272.661487] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 272.662497] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 272.663502] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 272.664519] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 272.665525] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 272.666522] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 272.667509] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 272.668492] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 272.669467] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 272.670445] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 272.671408] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 272.672357] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 272.673284] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 272.674225] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 272.675166] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 272.676098] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 272.677032] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.677951] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.678865] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 272.679770] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 272.680685] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 272.681603] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 272.682526] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 272.683442] NMI backtrace for cpu 3
[ 272.684688] CPU 3
[ 272.684705] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 272.690650]
[ 272.692143] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 272.693726] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 272.695312] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046
[ 272.696892] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
[ 272.698508] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640
[ 272.700117] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003e4
[ 272.701740] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030
[ 272.703369] R13: 12187a34a107edc5 R14: 0000000000000004 R15: 0000000000000003
[ 272.704985] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
[ 272.706626] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 272.708277] CR2: 00000037d904c480 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 272.709952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 272.711633] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 272.713299] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700)
[ 272.715001] Stack:
[ 272.716681] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1290 0000000000011290
[ 272.718411] ffff8800715dfeb8 000000038139c97a ffffe8ffffcc0170 ffffe8ffffcc0170
[ 272.720153] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b868
[ 272.721848] Call Trace:
[ 272.723465] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 272.725095] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.726712] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.728304] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 272.729891] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 272.733594] Call Trace:
[ 272.735317] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 272.737068] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.738815] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.740554] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 274.865072] SysRq : Show Blocked State
[ 274.866350] task PC stack pid father
[ 274.867634] jbd2/dm-1-8 D 0000000000000000 0 363 2 0x00000000
[ 274.868930] ffff880037d05ba0 0000000000000046 ffff880037d05b30 ffffffff00000000
[ 274.870242] ffff880037acc500 ffff880037d05fd8 ffff880037d05fd8 0000000000013b40
[ 274.871555] ffffffff81a0b020 ffff880037acc500 ffff8801002143c0 00000001005bd1d0
[ 274.872869] Call Trace:
[ 274.874168] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.875485] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.876791] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.878089] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
[ 274.879382] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
[ 274.880684] [<ffffffff8147af4e>] out_of_line_wait_on_bit+0x72/0x7d
[ 274.881986] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.883290] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 274.884601] [<ffffffff811421c6>] __wait_on_buffer+0x26/0x28
[ 274.885912] [<ffffffff811c8a3c>] wait_on_buffer+0x35/0x39
[ 274.887221] [<ffffffff811c9633>] jbd2_journal_commit_transaction+0xab3/0x1194
[ 274.888547] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 274.889864] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 274.891178] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 274.892500] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 274.893776] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
[ 274.895001] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.896179] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
[ 274.897328] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 274.898465] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 274.899591] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 274.900708] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 274.901813] flush-253:1 D 0000000000000000 0 419 2 0x00000000
[ 274.902938] ffff8800706bf760 0000000000000046 ffff8800706bf6f0 ffffffff00000000
[ 274.904067] ffff880037afc500 ffff8800706bffd8 ffff8800706bffd8 0000000000013b40
[ 274.905198] ffffffff81a0b020 ffff880037afc500 ffff8800706bf720 00000001706bf720
[ 274.906327] Call Trace:
[ 274.907430] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.908536] [<ffffffff81217b00>] get_request_wait+0x102/0x18b
[ 274.909633] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.910729] [<ffffffff81218841>] __make_request+0x18a/0x2b8
[ 274.911826] [<ffffffff81217538>] generic_make_request+0x2a9/0x323
[ 274.912908] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17
[ 274.913979] [<ffffffff81217690>] submit_bio+0xde/0xfd
[ 274.915040] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3
[ 274.916104] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29
[ 274.917161] [<ffffffff81141c42>] submit_bh+0xe6/0x105
[ 274.918208] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7
[ 274.919261] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
[ 274.920306] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
[ 274.921361] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
[ 274.922411] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
[ 274.923396] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97
[ 274.924324] [<ffffffff81144873>] block_write_full_page+0x15/0x17
[ 274.925248] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a
[ 274.926168] [<ffffffff810dd3c1>] __writepage+0x15/0x2e
[ 274.927083] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330
[ 274.928001] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33
[ 274.928923] [<ffffffff8104e4c1>] ? find_busiest_group+0x253/0x8b9
[ 274.929834] [<ffffffff81198ba9>] ? ext4_da_writepages+0x85/0x44d
[ 274.930744] [<ffffffff810dd363>] generic_writepages+0x40/0x56
[ 274.931646] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
[ 274.932536] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
[ 274.933430] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
[ 274.934317] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
[ 274.935194] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
[ 274.936061] [<ffffffff81045c65>] ? hrtick_update+0x32/0x34
[ 274.936941] [<ffffffff810dd79a>] ? global_dirty_limits+0x2b/0xd1
[ 274.936943] [<ffffffff8113d8cd>] wb_do_writeback+0x17f/0x19d
[ 274.936945] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
[ 274.936947] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
[ 274.936949] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 274.936951] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 274.936953] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 274.936956] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 274.936958] jbd2/dm-2-8 D ffff88006e17c700 0 787 2 0x00000000
[ 274.936960] ffff88006ffe5cd0 0000000000000046 0000000000000000 0000000000000000
[ 274.936962] ffff88006f0e0000 ffff88006ffe5fd8 ffff88006ffe5fd8 0000000000013b40
[ 274.936964] ffff8800715fc500 ffff88006f0e0000 ffff88006ffe5cd0 ffffffff8106e7c3
[ 274.936966] Call Trace:
[ 274.936968] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
[ 274.936970] [<ffffffff811c8d44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 274.936972] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 274.936974] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 274.936977] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 274.936979] [<ffffffff8100804e>] ? load_TLS+0x10/0x14
[ 274.936981] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 274.936983] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.936985] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 274.936987] [<ffffffff8147be8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 274.936989] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 274.936991] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
[ 274.936993] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.936995] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
[ 274.936997] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 274.936998] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 274.937001] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 274.937003] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 274.937004] flush-253:2 D ffff88006d5a0000 0 851 2 0x00000000
[ 274.937006] ffff88006e45d760 0000000000000046 ffff880037a30000 0000000000000246
[ 274.937008] ffff88006d5a0000 ffff88006e45dfd8 ffff88006e45dfd8 0000000000013b40
[ 274.937010] ffff880071588000 ffff88006d5a0000 ffff88006e45d720 ffff88006e45d720
[ 274.937011] Call Trace:
[ 274.937013] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.937015] [<ffffffff81217b00>] get_request_wait+0x102/0x18b
[ 274.937017] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.937019] [<ffffffff81218841>] __make_request+0x18a/0x2b8
[ 274.937020] [<ffffffff81217538>] generic_make_request+0x2a9/0x323
[ 274.937022] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17
[ 274.937024] [<ffffffff81217690>] submit_bio+0xde/0xfd
[ 274.937026] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3
[ 274.937028] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29
[ 274.937030] [<ffffffff81141c42>] submit_bh+0xe6/0x105
[ 274.937031] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7
[ 274.937033] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
[ 274.937035] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
[ 274.937037] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
[ 274.937039] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
[ 274.937040] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97
[ 274.937042] [<ffffffff81144873>] block_write_full_page+0x15/0x17
[ 274.937044] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a
[ 274.937046] [<ffffffff810dd3c1>] __writepage+0x15/0x2e
[ 274.937048] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330
[ 274.937049] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33
[ 274.937052] [<ffffffff810dd363>] generic_writepages+0x40/0x56
[ 274.937054] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
[ 274.937055] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
[ 274.937057] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
[ 274.937059] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
[ 274.937061] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
[ 274.937063] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 274.937065] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 274.937067] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
[ 274.937069] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
[ 274.937071] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
[ 274.937073] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
[ 274.937075] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 274.937077] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 274.937079] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 274.937081] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 274.937082] tar D 0000000000000000 0 980 826 0x00000000
[ 274.937084] ffff88006f19f9e8 0000000000000086 00000000ffff9fd7 0000000000000000
[ 274.937086] ffff88006d670000 ffff88006f19ffd8 ffff88006f19ffd8 0000000000013b40
[ 274.937088] ffffffff81a0b020 ffff88006d670000 ffff88006f19f9b8 0000000100000282
[ 274.937090] Call Trace:
[ 274.937091] [<ffffffff8147acbe>] schedule_timeout+0xa7/0xde
[ 274.937093] [<ffffffff8106015c>] ? del_timer+0x7a/0x7a
[ 274.937095] [<ffffffff8147abc2>] io_schedule_timeout+0x6f/0x98
[ 274.937097] [<ffffffff810ddc26>] balance_dirty_pages_ratelimited_nr+0x341/0x3b6
[ 274.937100] [<ffffffff810d5efb>] generic_file_buffered_write+0x1dc/0x23a
[ 274.937102] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
[ 274.937105] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
[ 274.937108] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
[ 274.937111] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
[ 274.937112] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
[ 274.937115] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
[ 274.937118] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 274.937120] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 274.937122] [<ffffffff8111f421>] vfs_write+0xac/0xf3
[ 274.937123] [<ffffffff8111f610>] sys_write+0x4a/0x6e
[ 274.937126] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937128] make D 0000000000000000 0 2252 2143 0x00000004
[ 274.937130] ffff88006d631948 0000000000000082 ffff88006d6318d8 ffffffff00000000
[ 274.937132] ffff880037d9dc00 ffff88006d631fd8 ffff88006d631fd8 0000000000013b40
[ 274.937133] ffffffff81a0b020 ffff880037d9dc00 ffff88006d631908 000000016d631908
[ 274.937135] Call Trace:
[ 274.937137] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937139] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.937140] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
[ 274.937142] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
[ 274.937144] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
[ 274.937146] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937148] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 274.937149] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
[ 274.937151] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
[ 274.937153] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
[ 274.937155] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d
[ 274.937156] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
[ 274.937158] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 274.937161] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 274.937163] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
[ 274.937165] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
[ 274.937167] [<ffffffff8147c515>] ? page_fault+0x25/0x30
[ 274.937169] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
[ 274.937171] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
[ 274.937173] [<ffffffff811322c7>] touch_atime+0x10e/0x131
[ 274.937176] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640
[ 274.937178] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
[ 274.937180] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 274.937181] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 274.937183] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
[ 274.937185] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e
[ 274.937187] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937188] make D 0000000000000000 0 2262 2254 0x00000004
[ 274.937190] ffff880070b2b948 0000000000000082 ffff880070b2b8d8 ffffffff00000000
[ 274.937192] ffff88006d428000 ffff880070b2bfd8 ffff880070b2bfd8 0000000000013b40
[ 274.937194] ffffffff81a0b020 ffff88006d428000 ffff880070b2b908 0000000170b2b908
[ 274.937195] Call Trace:
[ 274.937197] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937199] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.937200] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
[ 274.937202] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
[ 274.937204] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
[ 274.937205] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937208] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 274.937209] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
[ 274.937211] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
[ 274.937212] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
[ 274.937214] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d
[ 274.937216] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
[ 274.937217] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 274.937219] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 274.937221] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
[ 274.937223] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
[ 274.937225] [<ffffffff8147c515>] ? page_fault+0x25/0x30
[ 274.937227] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
[ 274.937228] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
[ 274.937230] [<ffffffff811322c7>] touch_atime+0x10e/0x131
[ 274.937232] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640
[ 274.937234] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
[ 274.937236] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 274.937238] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 274.937240] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
[ 274.937241] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e
[ 274.937243] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937244] sh D 0000000000000000 0 2270 2253 0x00000004
[ 274.937246] ffff8800709fd918 0000000000000086 ffff8800709fd8a8 ffffffff00000000
[ 274.937248] ffff880037f60000 ffff8800709fdfd8 ffff8800709fdfd8 0000000000013b40
[ 274.937250] ffff880071589700 ffff880037f60000 ffff8801002543c0 00000001005b89d0
[ 274.937252] Call Trace:
[ 274.937253] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937255] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.937256] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
[ 274.937258] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
[ 274.937260] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
[ 274.937261] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937264] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 274.937265] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
[ 274.937267] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
[ 274.937268] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
[ 274.937270] [<ffffffff81196fe4>] ? ext4_getblk+0x8e/0x153
[ 274.937272] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 274.937274] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 274.937276] [<ffffffff8119c0ce>] ext4_add_entry+0x11f/0x8cf
[ 274.937277] [<ffffffff811429f3>] ? __brelse+0x15/0x33
[ 274.937280] [<ffffffff81191bac>] ? ext4_new_inode+0xc63/0xd0a
[ 274.937282] [<ffffffff8119c89c>] ext4_add_nondir+0x1e/0x67
[ 274.937283] [<ffffffff8119cb1b>] ext4_create+0xf5/0x13e
[ 274.937286] [<ffffffff81129b12>] vfs_create+0x6c/0x8e
[ 274.937288] [<ffffffff81129dad>] do_last+0x279/0x5ab
[ 274.937289] [<ffffffff8112ac73>] path_openat+0xc8/0x31c
[ 274.937291] [<ffffffff810f2332>] ? handle_mm_fault+0x1ac/0x1bf
[ 274.937293] [<ffffffff8112aeff>] do_filp_open+0x38/0x86
[ 274.937295] [<ffffffff81233de1>] ? might_fault+0x21/0x23
[ 274.937298] [<ffffffff8113477b>] ? alloc_fd+0x72/0x11d
[ 274.937299] [<ffffffff8111e995>] do_sys_open+0x6e/0x100
[ 274.937301] [<ffffffff8111ea47>] sys_open+0x20/0x22
[ 274.937303] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937304] gcc D 0000000000000000 0 2272 2271 0x00000004
[ 274.937306] ffff88006f197c58 0000000000000086 ffff88006f197d14 0000000000000000
[ 274.937307] ffff880037f64500 ffff88006f197fd8 ffff88006f197fd8 0000000000013b40
[ 274.937309] ffffffff81a0b020 ffff880037f64500 ffff88006f197c58 000000018106e7c3
[ 274.937311] Call Trace:
[ 274.937313] [<ffffffff811c6fd5>] start_this_handle+0x2e8/0x465
[ 274.937315] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.937317] [<ffffffff811c7418>] jbd2__journal_start+0x94/0xda
[ 274.937318] [<ffffffff811c7471>] jbd2_journal_start+0x13/0x15
[ 274.937320] [<ffffffff811ab92a>] ext4_journal_start_sb+0x108/0x120
[ 274.937323] [<ffffffff8119cc24>] ext4_symlink+0xc0/0x212
[ 274.937325] [<ffffffff81129928>] vfs_symlink+0x54/0x74
[ 274.937326] [<ffffffff8112b40a>] sys_symlinkat+0x96/0xef
[ 274.937329] [<ffffffff81122d89>] ? sys_newstat+0x2a/0x33
[ 274.937330] [<ffffffff8112b479>] sys_symlink+0x16/0x18
[ 274.937332] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937335] Sched Debug Version: v0.10, 2.6.39-rc4+ #1
[ 274.937336] ktime : 275382.978028
[ 274.937337] sched_clk : 274937.334155
[ 274.937338] cpu_clk : 274937.334189
[ 274.937339] jiffies : 4294942678
[ 274.937340] sched_clock_stable : 1
[ 274.937341]
[ 274.937342] sysctl_sched
[ 274.937342] .sysctl_sched_latency : 18.000000
[ 274.937344] .sysctl_sched_min_granularity : 2.250000
[ 274.937345] .sysctl_sched_wakeup_granularity : 3.000000
[ 274.937346] .sysctl_sched_child_runs_first : 0
[ 274.937347] .sysctl_sched_features : 7279
[ 274.937348] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[ 274.937349]
[ 274.937350] cpu#0, 2492.220 MHz
[ 274.937350] .nr_running : 0
[ 274.937351] .load : 0
[ 274.937352] .nr_switches : 105039
[ 274.937353] .nr_load_updates : 121182
[ 274.937354] .nr_uninterruptible : 8
[ 274.937355] .next_balance : 4294.942607
[ 274.937356] .curr->pid : 0
[ 274.937357] .clock : 274864.523786
[ 274.937359] .cpu_load[0] : 0
[ 274.937360] .cpu_load[1] : 0
[ 274.937361] .cpu_load[2] : 0
[ 274.937362] .cpu_load[3] : 0
[ 274.937363] .cpu_load[4] : 0
[ 274.937364] .yld_count : 0
[ 274.937364] .sched_switch : 0
[ 274.937365] .sched_count : 107487
[ 274.937366] .sched_goidle : 43368
[ 274.937367] .avg_idle : 1000000
[ 274.937368] .ttwu_count : 56534
[ 274.937369] .ttwu_local : 53925
[ 274.937370] .bkl_count : 0
[ 274.937371]
[ 274.937372] cfs_rq[0]:/system
[ 274.937372] .exec_clock : 5906.036519
[ 274.937374] .MIN_vruntime : 0.000001
[ 274.937375] .min_vruntime : 7138.254657
[ 274.937376] .max_vruntime : 0.000001
[ 274.937377] .spread : 0.000000
[ 274.937378] .spread0 : -14302.149798
[ 274.937379] .nr_spread_over : 0
[ 274.937379] .nr_running : 0
[ 274.937380] .load : 0
[ 274.937381] .load_avg : 159.999997
[ 274.937382] .load_period : 5.496191
[ 274.937383] .load_contrib : 29
[ 274.937384] .load_tg : 29
[ 274.937385] .se->exec_start : 272446.423795
[ 274.937386] .se->vruntime : 21431.398168
[ 274.937387] .se->sum_exec_runtime : 5906.111805
[ 274.937388] .se->statistics.wait_start : 0.000000
[ 274.937389] .se->statistics.sleep_start : 0.000000
[ 274.937390] .se->statistics.block_start : 0.000000
[ 274.937391] .se->statistics.sleep_max : 0.000000
[ 274.937392] .se->statistics.block_max : 0.000000
[ 274.937393] .se->statistics.exec_max : 15.266617
[ 274.937394] .se->statistics.slice_max : 5.258344
[ 274.937395] .se->statistics.wait_max : 7.305679
[ 274.937396] .se->statistics.wait_sum : 190.506686
[ 274.937397] .se->statistics.wait_count : 18946
[ 274.937398] .se->load.weight : 2
[ 274.937399]
[ 274.937399] cfs_rq[0]:/
[ 274.937400] .exec_clock : 17409.691339
[ 274.937401] .MIN_vruntime : 0.000001
[ 274.937402] .min_vruntime : 21440.404455
[ 274.937403] .max_vruntime : 0.000001
[ 274.937403] .spread : 0.000000
[ 274.937404] .spread0 : 0.000000
[ 274.937405] .nr_spread_over : 58
[ 274.937406] .nr_running : 0
[ 274.937407] .load : 0
[ 274.937408] .load_avg : 0.000000
[ 274.937409] .load_period : 0.000000
[ 274.937409] .load_contrib : 0
[ 274.937410] .load_tg : 0
[ 274.937411]
[ 274.937412] cfs_rq[0]:/system/rescue.service
[ 274.937413] .exec_clock : 4319.868509
[ 274.937414] .MIN_vruntime : 0.000001
[ 274.937414] .min_vruntime : 4255.887524
[ 274.937415] .max_vruntime : 0.000001
[ 274.937416] .spread : 0.000000
[ 274.937417] .spread0 : -17184.516931
[ 274.937418] .nr_spread_over : 24
[ 274.937419] .nr_running : 0
[ 274.937420] .load : 0
[ 274.937421] .load_avg : 160.004112
[ 274.937421] .load_period : 5.496149
[ 274.937422] .load_contrib : 29
[ 274.937423] .load_tg : 29
[ 274.937424] .se->exec_start : 272446.423795
[ 274.937425] .se->vruntime : 7138.254657
[ 274.937426] .se->sum_exec_runtime : 4319.796162
[ 274.937427] .se->statistics.wait_start : 0.000000
[ 274.937428] .se->statistics.sleep_start : 0.000000
[ 274.937429] .se->statistics.block_start : 0.000000
[ 274.937430] .se->statistics.sleep_max : 0.000000
[ 274.937431] .se->statistics.block_max : 0.000000
[ 274.937432] .se->statistics.exec_max : 6.213083
[ 274.937432] .se->statistics.slice_max : 7.894226
[ 274.937433] .se->statistics.wait_max : 4.870567
[ 274.937434] .se->statistics.wait_sum : 91.737803
[ 274.937435] .se->statistics.wait_count : 15504
[ 274.937436] .se->load.weight : 2
[ 274.937437]
[ 274.937438] runnable tasks:
[ 274.937438] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 274.937439] ----------------------------------------------------------------------------------------------------------
[ 274.937444]
[ 274.937445] cpu#1, 2492.220 MHz
[ 274.937445] .nr_running : 0
[ 274.937446] .load : 0
[ 274.937447] .nr_switches : 35274
[ 274.937448] .nr_load_updates : 105259
[ 274.937449] .nr_uninterruptible : 1
[ 274.937449] .next_balance : 4294.942681
[ 274.937450] .curr->pid : 0
[ 274.937451] .clock : 274937.357922
[ 274.937452] .cpu_load[0] : 0
[ 274.937453] .cpu_load[1] : 0
[ 274.937454] .cpu_load[2] : 0
[ 274.937454] .cpu_load[3] : 0
[ 274.937455] .cpu_load[4] : 0
[ 274.937456] .yld_count : 0
[ 274.937457] .sched_switch : 0
[ 274.937458] .sched_count : 35560
[ 274.937458] .sched_goidle : 14561
[ 274.937459] .avg_idle : 1000000
[ 274.937460] .ttwu_count : 16938
[ 274.937461] .ttwu_local : 14491
[ 274.937462] .bkl_count : 0
[ 274.937463]
[ 274.937463] cfs_rq[1]:/
[ 274.937464] .exec_clock : 5221.725565
[ 274.937465] .MIN_vruntime : 0.000001
[ 274.937466] .min_vruntime : 9077.562937
[ 274.937466] .max_vruntime : 0.000001
[ 274.937467] .spread : 0.000000
[ 274.937468] .spread0 : -12362.841518
[ 274.937469] .nr_spread_over : 69
[ 274.937470] .nr_running : 0
[ 274.937471] .load : 0
[ 274.937472] .load_avg : 0.000000
[ 274.937473] .load_period : 0.000000
[ 274.937473] .load_contrib : 0
[ 274.937474] .load_tg : 0
[ 274.937475]
[ 274.937476] runnable tasks:
[ 274.937476] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 274.937477] ----------------------------------------------------------------------------------------------------------
[ 274.937481]
[ 274.937482] cpu#2, 2492.220 MHz
[ 274.937482] .nr_running : 4
[ 274.937483] .load : 2048
[ 274.937484] .nr_switches : 7092
[ 274.937485] .nr_load_updates : 100502
[ 274.937486] .nr_uninterruptible : 0
[ 274.937487] .next_balance : 4294.942806
[ 274.937487] .curr->pid : 46
[ 274.937488] .clock : 183206.600407
[ 274.937489] .cpu_load[0] : 2048
[ 274.937490] .cpu_load[1] : 2048
[ 274.937491] .cpu_load[2] : 2048
[ 274.937492] .cpu_load[3] : 2048
[ 274.937492] .cpu_load[4] : 2048
[ 274.937493] .yld_count : 0
[ 274.937494] .sched_switch : 0
[ 274.937495] .sched_count : 7323
[ 274.937496] .sched_goidle : 2831
[ 274.937496] .avg_idle : 1000000
[ 274.937497] .ttwu_count : 3275
[ 274.937498] .ttwu_local : 2028
[ 274.937499] .bkl_count : 0
[ 274.937500]
[ 274.937500] cfs_rq[2]:/
[ 274.937501] .exec_clock : 2187.192139
[ 274.937502] .MIN_vruntime : 5109.505143
[ 274.937503] .min_vruntime : 5118.505143
[ 274.937504] .max_vruntime : 5109.505143
[ 274.937505] .spread : 0.000000
[ 274.937506] .spread0 : -16321.899312
[ 274.937506] .nr_spread_over : 31
[ 274.937507] .nr_running : 2
[ 274.937508] .load : 2048
[ 274.937509] .load_avg : 0.000000
[ 274.937510] .load_period : 0.000000
[ 274.937511] .load_contrib : 0
[ 274.937511] .load_tg : 0
[ 274.937512]
[ 274.937513] rt_rq[2]:/
[ 274.937513] .rt_nr_running : 1
[ 274.937514] .rt_throttled : 0
[ 274.937515] .rt_time : 0.000000
[ 274.937516] .rt_runtime : 950.000000
[ 274.937517]
[ 274.937517] runnable tasks:
[ 274.937518] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 274.937519] ----------------------------------------------------------------------------------------------------------
[ 274.937520] migration/2 13 0.000000 113 0 0.000000 0.000815 0.000000 /
[ 274.937524] watchdog/2 16 0.000000 17 0 0.000000 1.785026 0.002124 /
[ 274.937527] R kswapd0 46 5118.505143 370 120 5118.505143 415.766870 181943.841718 /
[ 274.937531] kworker/2:1 76 5109.505143 1083 120 5109.505143 25.615305 181183.059001 /
[ 274.937535]
[ 274.937536] cpu#3, 2492.220 MHz
[ 274.937536] .nr_running : 1
[ 274.937537] .load : 1024
[ 274.937538] .nr_switches : 22324
[ 274.937539] .nr_load_updates : 59855
[ 274.937540] .nr_uninterruptible : 0
[ 274.937541] .next_balance : 4294.942679
[ 274.937542] .curr->pid : 73
[ 274.937543] .clock : 274936.938702
[ 274.937543] .cpu_load[0] : 0
[ 274.937544] .cpu_load[1] : 0
[ 274.937545] .cpu_load[2] : 0
[ 274.937546] .cpu_load[3] : 0
[ 274.937546] .cpu_load[4] : 0
[ 274.937547] .yld_count : 123
[ 274.937548] .sched_switch : 0
[ 274.937549] .sched_count : 22774
[ 274.937550] .sched_goidle : 9952
[ 274.937551] .avg_idle : 946660
[ 274.937551] .ttwu_count : 10075
[ 274.937552] .ttwu_local : 8964
[ 274.937553] .bkl_count : 0
[ 274.937554]
[ 274.937554] cfs_rq[3]:/
[ 274.937555] .exec_clock : 3244.912177
[ 274.937556] .MIN_vruntime : 0.000001
[ 274.937557] .min_vruntime : 7517.482154
[ 274.937558] .max_vruntime : 0.000001
[ 274.937559] .spread : 0.000000
[ 274.937559] .spread0 : -13922.922301
[ 274.937560] .nr_spread_over : 128
[ 274.937561] .nr_running : 1
[ 274.937562] .load : 1024
[ 274.937563] .load_avg : 0.000000
[ 274.937564] .load_period : 0.000000
[ 274.937565] .load_contrib : 0
[ 274.937565] .load_tg : 0
[ 274.937567]
[ 274.937567] runnable tasks:
[ 274.937567] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 274.937568] ----------------------------------------------------------------------------------------------------------
[ 274.937571] R kworker/3:1 73 7517.482154 7294 120 7517.482154 264.876541 272702.297840 /
[ 274.937575]
Hi,
Just like to add that I've seen almost identical issues with 2.6.38
copying large amounts of data to an ext4 filesystem with systems with
small amounts of memory.
I found that increasing /sys/fs/ext4/sdaX/max_writeback_mb_bump worked
around the issue.
Colin
On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote:
> On Wed, 2011-04-27 at 13:34 -0400, Chris Mason wrote:
> > [ cc'd linux-ext4 ]
> >
> > James is hitting softlockups in kswapd while doing writes to a large
> > ext4 file.
> >
> > Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400:
> > > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> > > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> > > > case better.
> > >
> > > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
> > >
> > > This is the trace with sysrq-l and sysrq-w
> >
> > >
> > > The repro this time doesn't have a soft lockup, just the tar is hung and
> > > one of my CPUs is in 99% system.
> > > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> > > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> > > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> > > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> > > [ 454.753048] Call Trace:
> > > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> > > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> > > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> > > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> > > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> > > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> > > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> > > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> > > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> > > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> > > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> > > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> > > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> > > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> >
> > So our flusher threads are stuck waiting in
> > jbd2_journal_get_write_access, which means they aren't cleaning dirty
> > pages.
> >
> > In order to get write access, they probably need the transaction to
> > commit:
> >
> > > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> > > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> > > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> > > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> > > [ 454.839453] Call Trace:
> > > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> > > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> > > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> > > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> > > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> > > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> > > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> > > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> > > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> > > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
> >
> > But that seems to be waiting for a page lock. Probably the same page
> > lock held by the flusher thread? Looks like tar is stuck in the same
> > boat down below.
>
> To test the theory, Chris asked me to try with data=ordered.
> Unfortunately, the deadlock still shows up. This is what I get.
>
> James
>
> ---
>
> [ 263.749738] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
> [ 263.751207] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 263.756081] CPU 2
> [ 263.756091] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 263.763033]
> [ 263.764761] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 263.766509] RIP: 0010:[<ffffffff810e1fa2>] [<ffffffff810e1fa2>] shrink_slab+0x86/0x166
> [ 263.768256] RSP: 0018:ffff8800709ebda0 EFLAGS: 00000206
> [ 263.769976] RAX: 0000000000000000 RBX: ffff8800709ebde0 RCX: 0000000000000002
> [ 263.771683] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81a44e50
> [ 263.773378] RBP: ffff8800709ebde0 R08: 0000000000000000 R09: 000000000000a2b2
> [ 263.775053] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e
> [ 263.776728] R13: ffff8800709ebe58 R14: 0000000000000004 R15: 00000000000178f9
> [ 263.778408] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
> [ 263.780098] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 263.781777] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 263.783458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 263.785140] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 263.786811] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000)
> [ 263.788491] Stack:
> [ 263.790153] 000000000000003d 0000000000000080 ffff880000000000 ffff8801005e6700
> [ 263.791810] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c
> [ 263.793426] ffff8800709ebee0 ffffffff810e4bcc 0000000000000003 ffff88006dfa8000
> [ 263.795038] Call Trace:
> [ 263.796614] [<ffffffff810e4bcc>] kswapd+0x533/0x798
> [ 263.798183] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 263.799755] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 263.801328] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 263.802904] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 263.804434] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 263.805914] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7
> [ 263.806080] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7
> [ 263.809185] Call Trace:
> [ 263.810748] [<ffffffff810e4bcc>] kswapd+0x533/0x798
> [ 263.812327] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 263.813913] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 263.815486] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 263.817053] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 263.818621] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 272.421176] SysRq : Show backtrace of all active CPUs
> [ 272.422748] sending NMI to all CPUs:
> [ 272.424304] NMI backtrace for cpu 2
> [ 272.426364] CPU 2
> [ 272.426380] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 272.435128]
> [ 272.437299] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 272.439506] RIP: 0010:[<ffffffff810d9b6e>] [<ffffffff810d9b6e>] zone_watermark_ok_safe+0xa8/0xae
> [ 272.441728] RSP: 0018:ffff8800709ebd80 EFLAGS: 00000282
> [ 272.443916] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> [ 272.446101] RDX: 0000000000000813 RSI: 0000000000000000 RDI: ffff8801005e6000
> [ 272.448258] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 00000000000007f0
> [ 272.450362] R10: 0000000000000002 R11: 00000000000000cd R12: ffff8801005e6000
> [ 272.452399] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8801005e6000
> [ 272.454367] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
> [ 272.456306] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 272.458188] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 272.460021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 272.461789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 272.463503] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000)
> [ 272.465234] Stack:
> [ 272.466929] 0000000000000000 0000000000000000 ffff8800709ebde0 ffffffff810e1e8a
> [ 272.468675] ffff880000000002 ffffffff8106e701 ffff880000000002 ffff8801005e6000
> [ 272.470406] ffff8801005f9e68 0000000000000002 0000000000000000 0000000000000000
> [ 272.472144] Call Trace:
> [ 272.473850] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2
> [ 272.475582] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77
> [ 272.477304] [<ffffffff810e47fe>] kswapd+0x165/0x798
> [ 272.479015] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 272.480712] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 272.482414] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 272.484106] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 272.485809] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 272.487507] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 272.489175] Code: a6 00 48 8b 55 d8 8b 4d d0 44 8b 45 c8 7c c0 45 31 c9 4d 85 ed 4d 0f 49 cd 44 89 e6 48 89 df e8 5e f4 ff ff 48 83 c4 20 5b 41 5c
> [ 272.489409] 5d 41 5e 5d c3 55 48 89 e5 0f 1f 44 00 00 bf da 00 02 00 e8
> [ 272.492911] Call Trace:
> [ 272.494651] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2
> [ 272.496408] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77
> [ 272.498173] [<ffffffff810e47fe>] kswapd+0x165/0x798
> [ 272.499926] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 272.501678] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 272.503444] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 272.505198] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 272.506968] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 272.508691] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 272.510355] NMI backtrace for cpu 1
> [ 272.511996] CPU 1
> [ 272.512011] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 272.519303]
> [ 272.521156] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 272.523190] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
> [ 272.525110] RSP: 0018:ffff880071587e68 EFLAGS: 00000046
> [ 272.527018] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
> [ 272.528960] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640
> [ 272.530906] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5
> [ 272.532800] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010
> [ 272.534634] R13: 12187a34a107f726 R14: 0000000000000002 R15: 0000000000000001
> [ 272.536399] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
> [ 272.538145] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 272.539879] CR2: 00000037da2830e4 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 272.541614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 272.543329] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 272.545026] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700)
> [ 272.546756] Stack:
> [ 272.548457] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290
> [ 272.550225] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170
> [ 272.552009] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868
> [ 272.553780] Call Trace:
> [ 272.555514] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 272.557259] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.559005] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.560755] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 272.562485] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
> [ 272.566380] Call Trace:
> [ 272.568193] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 272.570031] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.571855] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.573658] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 272.575463] NMI backtrace for cpu 0
> [ 272.576865] CPU 0
> [ 272.576875] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 272.582932]
> [ 272.584474] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 272.586057] RIP: 0010:[<ffffffff81232e6e>] [<ffffffff81232e6e>] __const_udelay+0x23/0x2e
> [ 272.587604] RSP: 0018:ffff880100203bf8 EFLAGS: 00000887
> [ 272.589120] RAX: 00000000fa1c3100 RBX: 0000000000002710 RCX: 0000000000000040
> [ 272.590635] RDX: 000000000026074e RSI: 0000000000000100 RDI: 0000000000418958
> [ 272.592139] RBP: ffff880100203bf8 R08: 000000008b000052 R09: 0000000000000000
> [ 272.593644] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000008
> [ 272.595143] R13: 000000000000006c R14: 0000000000000002 R15: 0000000000000001
> [ 272.596635] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000
> [ 272.598139] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 272.599637] CR2: 00000037d909e270 CR3: 000000006f2d7000 CR4: 00000000000406f0
> [ 272.601151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 272.602669] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 272.604187] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020)
> [ 272.605715] Stack:
> [ 272.607238] ffff880100203c18 ffffffff8102166e 0000000000000000 ffffffff81a5e690
> [ 272.608756] ffff880100203c28 ffffffff812be0ad ffff880100203c68 ffffffff812be310
> [ 272.610226] ffff880037d01900 0000000000000026 ffff880037d01900 0000000000000001
> [ 272.611653] Call Trace:
> [ 272.613026] <IRQ>
> [ 272.614388] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
> [ 272.615761] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
> [ 272.617118] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
> [ 272.618465] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
> [ 272.619814] [<ffffffff81365764>] input_pass_event+0x94/0xcc
> [ 272.621158] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
> [ 272.622497] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
> [ 272.623844] [<ffffffff81366cf2>] input_event+0x69/0x87
> [ 272.625172] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
> [ 272.626489] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
> [ 272.627791] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
> [ 272.629085] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
> [ 272.630383] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
> [ 272.631675] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
> [ 272.632946] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
> [ 272.634205] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
> [ 272.635455] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
> [ 272.636699] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
> [ 272.637927] [<ffffffff8147c253>] common_interrupt+0x13/0x13
> [ 272.639155] <EOI>
> [ 272.640374] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
> [ 272.641612] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
> [ 272.642840] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
> [ 272.644012] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.645115] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.646215] [<ffffffff8145a91e>] rest_init+0x72/0x74
> [ 272.647313] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
> [ 272.648411] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
> [ 272.649514] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
> [ 272.650624] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
> [ 272.651725] Code: ff 15 37 2b 82 00 5d c3 55 48 89 e5 0f 1f 44 00 00 65 48 8b 14 25 98 3a 01 00 48 8d 04 bd 00 00 00 00 48 69 d2 fa 00 00 00 f7 e2
> [ 272.651891] 8d 7a 01 e8 c3 ff ff ff 5d c3 55 48 89 e5 0f 1f 44 00 00 48
> [ 272.654210] Call Trace:
> [ 272.655298] <IRQ> [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
> [ 272.656372] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
> [ 272.657421] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
> [ 272.658456] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
> [ 272.659480] [<ffffffff81365764>] input_pass_event+0x94/0xcc
> [ 272.660486] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
> [ 272.661487] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
> [ 272.662497] [<ffffffff81366cf2>] input_event+0x69/0x87
> [ 272.663502] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
> [ 272.664519] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
> [ 272.665525] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
> [ 272.666522] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
> [ 272.667509] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
> [ 272.668492] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
> [ 272.669467] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
> [ 272.670445] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
> [ 272.671408] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
> [ 272.672357] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
> [ 272.673284] [<ffffffff8147c253>] common_interrupt+0x13/0x13
> [ 272.674225] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
> [ 272.675166] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
> [ 272.676098] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
> [ 272.677032] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.677951] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.678865] [<ffffffff8145a91e>] rest_init+0x72/0x74
> [ 272.679770] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
> [ 272.680685] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
> [ 272.681603] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
> [ 272.682526] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
> [ 272.683442] NMI backtrace for cpu 3
> [ 272.684688] CPU 3
> [ 272.684705] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 272.690650]
> [ 272.692143] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 272.693726] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
> [ 272.695312] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046
> [ 272.696892] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
> [ 272.698508] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640
> [ 272.700117] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003e4
> [ 272.701740] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030
> [ 272.703369] R13: 12187a34a107edc5 R14: 0000000000000004 R15: 0000000000000003
> [ 272.704985] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
> [ 272.706626] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 272.708277] CR2: 00000037d904c480 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 272.709952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 272.711633] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 272.713299] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700)
> [ 272.715001] Stack:
> [ 272.716681] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1290 0000000000011290
> [ 272.718411] ffff8800715dfeb8 000000038139c97a ffffe8ffffcc0170 ffffe8ffffcc0170
> [ 272.720153] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b868
> [ 272.721848] Call Trace:
> [ 272.723465] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 272.725095] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.726712] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.728304] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 272.729891] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
> [ 272.733594] Call Trace:
> [ 272.735317] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 272.737068] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.738815] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.740554] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 274.865072] SysRq : Show Blocked State
> [ 274.866350] task PC stack pid father
> [ 274.867634] jbd2/dm-1-8 D 0000000000000000 0 363 2 0x00000000
> [ 274.868930] ffff880037d05ba0 0000000000000046 ffff880037d05b30 ffffffff00000000
> [ 274.870242] ffff880037acc500 ffff880037d05fd8 ffff880037d05fd8 0000000000013b40
> [ 274.871555] ffffffff81a0b020 ffff880037acc500 ffff8801002143c0 00000001005bd1d0
> [ 274.872869] Call Trace:
> [ 274.874168] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.875485] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.876791] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.878089] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
> [ 274.879382] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> [ 274.880684] [<ffffffff8147af4e>] out_of_line_wait_on_bit+0x72/0x7d
> [ 274.881986] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.883290] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 274.884601] [<ffffffff811421c6>] __wait_on_buffer+0x26/0x28
> [ 274.885912] [<ffffffff811c8a3c>] wait_on_buffer+0x35/0x39
> [ 274.887221] [<ffffffff811c9633>] jbd2_journal_commit_transaction+0xab3/0x1194
> [ 274.888547] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 274.889864] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
> [ 274.891178] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
> [ 274.892500] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
> [ 274.893776] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 274.895001] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.896179] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 274.897328] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 274.898465] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 274.899591] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 274.900708] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 274.901813] flush-253:1 D 0000000000000000 0 419 2 0x00000000
> [ 274.902938] ffff8800706bf760 0000000000000046 ffff8800706bf6f0 ffffffff00000000
> [ 274.904067] ffff880037afc500 ffff8800706bffd8 ffff8800706bffd8 0000000000013b40
> [ 274.905198] ffffffff81a0b020 ffff880037afc500 ffff8800706bf720 00000001706bf720
> [ 274.906327] Call Trace:
> [ 274.907430] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.908536] [<ffffffff81217b00>] get_request_wait+0x102/0x18b
> [ 274.909633] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.910729] [<ffffffff81218841>] __make_request+0x18a/0x2b8
> [ 274.911826] [<ffffffff81217538>] generic_make_request+0x2a9/0x323
> [ 274.912908] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17
> [ 274.913979] [<ffffffff81217690>] submit_bio+0xde/0xfd
> [ 274.915040] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3
> [ 274.916104] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29
> [ 274.917161] [<ffffffff81141c42>] submit_bh+0xe6/0x105
> [ 274.918208] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7
> [ 274.919261] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
> [ 274.920306] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
> [ 274.921361] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
> [ 274.922411] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
> [ 274.923396] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97
> [ 274.924324] [<ffffffff81144873>] block_write_full_page+0x15/0x17
> [ 274.925248] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a
> [ 274.926168] [<ffffffff810dd3c1>] __writepage+0x15/0x2e
> [ 274.927083] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330
> [ 274.928001] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33
> [ 274.928923] [<ffffffff8104e4c1>] ? find_busiest_group+0x253/0x8b9
> [ 274.929834] [<ffffffff81198ba9>] ? ext4_da_writepages+0x85/0x44d
> [ 274.930744] [<ffffffff810dd363>] generic_writepages+0x40/0x56
> [ 274.931646] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> [ 274.932536] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
> [ 274.933430] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
> [ 274.934317] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
> [ 274.935194] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
> [ 274.936061] [<ffffffff81045c65>] ? hrtick_update+0x32/0x34
> [ 274.936941] [<ffffffff810dd79a>] ? global_dirty_limits+0x2b/0xd1
> [ 274.936943] [<ffffffff8113d8cd>] wb_do_writeback+0x17f/0x19d
> [ 274.936945] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
> [ 274.936947] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
> [ 274.936949] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 274.936951] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 274.936953] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 274.936956] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 274.936958] jbd2/dm-2-8 D ffff88006e17c700 0 787 2 0x00000000
> [ 274.936960] ffff88006ffe5cd0 0000000000000046 0000000000000000 0000000000000000
> [ 274.936962] ffff88006f0e0000 ffff88006ffe5fd8 ffff88006ffe5fd8 0000000000013b40
> [ 274.936964] ffff8800715fc500 ffff88006f0e0000 ffff88006ffe5cd0 ffffffff8106e7c3
> [ 274.936966] Call Trace:
> [ 274.936968] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
> [ 274.936970] [<ffffffff811c8d44>] jbd2_journal_commit_transaction+0x1c4/0x1194
> [ 274.936972] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
> [ 274.936974] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
> [ 274.936977] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
> [ 274.936979] [<ffffffff8100804e>] ? load_TLS+0x10/0x14
> [ 274.936981] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 274.936983] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.936985] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
> [ 274.936987] [<ffffffff8147be8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
> [ 274.936989] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
> [ 274.936991] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 274.936993] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.936995] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 274.936997] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 274.936998] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 274.937001] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 274.937003] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 274.937004] flush-253:2 D ffff88006d5a0000 0 851 2 0x00000000
> [ 274.937006] ffff88006e45d760 0000000000000046 ffff880037a30000 0000000000000246
> [ 274.937008] ffff88006d5a0000 ffff88006e45dfd8 ffff88006e45dfd8 0000000000013b40
> [ 274.937010] ffff880071588000 ffff88006d5a0000 ffff88006e45d720 ffff88006e45d720
> [ 274.937011] Call Trace:
> [ 274.937013] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.937015] [<ffffffff81217b00>] get_request_wait+0x102/0x18b
> [ 274.937017] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.937019] [<ffffffff81218841>] __make_request+0x18a/0x2b8
> [ 274.937020] [<ffffffff81217538>] generic_make_request+0x2a9/0x323
> [ 274.937022] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17
> [ 274.937024] [<ffffffff81217690>] submit_bio+0xde/0xfd
> [ 274.937026] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3
> [ 274.937028] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29
> [ 274.937030] [<ffffffff81141c42>] submit_bh+0xe6/0x105
> [ 274.937031] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7
> [ 274.937033] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
> [ 274.937035] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
> [ 274.937037] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
> [ 274.937039] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
> [ 274.937040] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97
> [ 274.937042] [<ffffffff81144873>] block_write_full_page+0x15/0x17
> [ 274.937044] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a
> [ 274.937046] [<ffffffff810dd3c1>] __writepage+0x15/0x2e
> [ 274.937048] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330
> [ 274.937049] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33
> [ 274.937052] [<ffffffff810dd363>] generic_writepages+0x40/0x56
> [ 274.937054] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> [ 274.937055] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
> [ 274.937057] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
> [ 274.937059] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
> [ 274.937061] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
> [ 274.937063] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
> [ 274.937065] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
> [ 274.937067] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
> [ 274.937069] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
> [ 274.937071] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
> [ 274.937073] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
> [ 274.937075] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 274.937077] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 274.937079] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 274.937081] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 274.937082] tar D 0000000000000000 0 980 826 0x00000000
> [ 274.937084] ffff88006f19f9e8 0000000000000086 00000000ffff9fd7 0000000000000000
> [ 274.937086] ffff88006d670000 ffff88006f19ffd8 ffff88006f19ffd8 0000000000013b40
> [ 274.937088] ffffffff81a0b020 ffff88006d670000 ffff88006f19f9b8 0000000100000282
> [ 274.937090] Call Trace:
> [ 274.937091] [<ffffffff8147acbe>] schedule_timeout+0xa7/0xde
> [ 274.937093] [<ffffffff8106015c>] ? del_timer+0x7a/0x7a
> [ 274.937095] [<ffffffff8147abc2>] io_schedule_timeout+0x6f/0x98
> [ 274.937097] [<ffffffff810ddc26>] balance_dirty_pages_ratelimited_nr+0x341/0x3b6
> [ 274.937100] [<ffffffff810d5efb>] generic_file_buffered_write+0x1dc/0x23a
> [ 274.937102] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
> [ 274.937105] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
> [ 274.937108] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
> [ 274.937111] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
> [ 274.937112] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
> [ 274.937115] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
> [ 274.937118] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 274.937120] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 274.937122] [<ffffffff8111f421>] vfs_write+0xac/0xf3
> [ 274.937123] [<ffffffff8111f610>] sys_write+0x4a/0x6e
> [ 274.937126] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937128] make D 0000000000000000 0 2252 2143 0x00000004
> [ 274.937130] ffff88006d631948 0000000000000082 ffff88006d6318d8 ffffffff00000000
> [ 274.937132] ffff880037d9dc00 ffff88006d631fd8 ffff88006d631fd8 0000000000013b40
> [ 274.937133] ffffffff81a0b020 ffff880037d9dc00 ffff88006d631908 000000016d631908
> [ 274.937135] Call Trace:
> [ 274.937137] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937139] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.937140] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
> [ 274.937142] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
> [ 274.937144] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
> [ 274.937146] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937148] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 274.937149] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
> [ 274.937151] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
> [ 274.937153] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
> [ 274.937155] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d
> [ 274.937156] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 274.937158] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 274.937161] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 274.937163] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 274.937165] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 274.937167] [<ffffffff8147c515>] ? page_fault+0x25/0x30
> [ 274.937169] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 274.937171] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 274.937173] [<ffffffff811322c7>] touch_atime+0x10e/0x131
> [ 274.937176] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640
> [ 274.937178] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
> [ 274.937180] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 274.937181] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 274.937183] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
> [ 274.937185] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e
> [ 274.937187] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937188] make D 0000000000000000 0 2262 2254 0x00000004
> [ 274.937190] ffff880070b2b948 0000000000000082 ffff880070b2b8d8 ffffffff00000000
> [ 274.937192] ffff88006d428000 ffff880070b2bfd8 ffff880070b2bfd8 0000000000013b40
> [ 274.937194] ffffffff81a0b020 ffff88006d428000 ffff880070b2b908 0000000170b2b908
> [ 274.937195] Call Trace:
> [ 274.937197] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937199] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.937200] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
> [ 274.937202] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
> [ 274.937204] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
> [ 274.937205] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937208] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 274.937209] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
> [ 274.937211] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
> [ 274.937212] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
> [ 274.937214] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d
> [ 274.937216] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 274.937217] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 274.937219] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 274.937221] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 274.937223] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 274.937225] [<ffffffff8147c515>] ? page_fault+0x25/0x30
> [ 274.937227] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 274.937228] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 274.937230] [<ffffffff811322c7>] touch_atime+0x10e/0x131
> [ 274.937232] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640
> [ 274.937234] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
> [ 274.937236] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 274.937238] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 274.937240] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
> [ 274.937241] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e
> [ 274.937243] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937244] sh D 0000000000000000 0 2270 2253 0x00000004
> [ 274.937246] ffff8800709fd918 0000000000000086 ffff8800709fd8a8 ffffffff00000000
> [ 274.937248] ffff880037f60000 ffff8800709fdfd8 ffff8800709fdfd8 0000000000013b40
> [ 274.937250] ffff880071589700 ffff880037f60000 ffff8801002543c0 00000001005b89d0
> [ 274.937252] Call Trace:
> [ 274.937253] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937255] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.937256] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
> [ 274.937258] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
> [ 274.937260] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
> [ 274.937261] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937264] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 274.937265] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
> [ 274.937267] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
> [ 274.937268] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
> [ 274.937270] [<ffffffff81196fe4>] ? ext4_getblk+0x8e/0x153
> [ 274.937272] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 274.937274] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 274.937276] [<ffffffff8119c0ce>] ext4_add_entry+0x11f/0x8cf
> [ 274.937277] [<ffffffff811429f3>] ? __brelse+0x15/0x33
> [ 274.937280] [<ffffffff81191bac>] ? ext4_new_inode+0xc63/0xd0a
> [ 274.937282] [<ffffffff8119c89c>] ext4_add_nondir+0x1e/0x67
> [ 274.937283] [<ffffffff8119cb1b>] ext4_create+0xf5/0x13e
> [ 274.937286] [<ffffffff81129b12>] vfs_create+0x6c/0x8e
> [ 274.937288] [<ffffffff81129dad>] do_last+0x279/0x5ab
> [ 274.937289] [<ffffffff8112ac73>] path_openat+0xc8/0x31c
> [ 274.937291] [<ffffffff810f2332>] ? handle_mm_fault+0x1ac/0x1bf
> [ 274.937293] [<ffffffff8112aeff>] do_filp_open+0x38/0x86
> [ 274.937295] [<ffffffff81233de1>] ? might_fault+0x21/0x23
> [ 274.937298] [<ffffffff8113477b>] ? alloc_fd+0x72/0x11d
> [ 274.937299] [<ffffffff8111e995>] do_sys_open+0x6e/0x100
> [ 274.937301] [<ffffffff8111ea47>] sys_open+0x20/0x22
> [ 274.937303] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937304] gcc D 0000000000000000 0 2272 2271 0x00000004
> [ 274.937306] ffff88006f197c58 0000000000000086 ffff88006f197d14 0000000000000000
> [ 274.937307] ffff880037f64500 ffff88006f197fd8 ffff88006f197fd8 0000000000013b40
> [ 274.937309] ffffffff81a0b020 ffff880037f64500 ffff88006f197c58 000000018106e7c3
> [ 274.937311] Call Trace:
> [ 274.937313] [<ffffffff811c6fd5>] start_this_handle+0x2e8/0x465
> [ 274.937315] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.937317] [<ffffffff811c7418>] jbd2__journal_start+0x94/0xda
> [ 274.937318] [<ffffffff811c7471>] jbd2_journal_start+0x13/0x15
> [ 274.937320] [<ffffffff811ab92a>] ext4_journal_start_sb+0x108/0x120
> [ 274.937323] [<ffffffff8119cc24>] ext4_symlink+0xc0/0x212
> [ 274.937325] [<ffffffff81129928>] vfs_symlink+0x54/0x74
> [ 274.937326] [<ffffffff8112b40a>] sys_symlinkat+0x96/0xef
> [ 274.937329] [<ffffffff81122d89>] ? sys_newstat+0x2a/0x33
> [ 274.937330] [<ffffffff8112b479>] sys_symlink+0x16/0x18
> [ 274.937332] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937335] Sched Debug Version: v0.10, 2.6.39-rc4+ #1
> [ 274.937336] ktime : 275382.978028
> [ 274.937337] sched_clk : 274937.334155
> [ 274.937338] cpu_clk : 274937.334189
> [ 274.937339] jiffies : 4294942678
> [ 274.937340] sched_clock_stable : 1
> [ 274.937341]
> [ 274.937342] sysctl_sched
> [ 274.937342] .sysctl_sched_latency : 18.000000
> [ 274.937344] .sysctl_sched_min_granularity : 2.250000
> [ 274.937345] .sysctl_sched_wakeup_granularity : 3.000000
> [ 274.937346] .sysctl_sched_child_runs_first : 0
> [ 274.937347] .sysctl_sched_features : 7279
> [ 274.937348] .sysctl_sched_tunable_scaling : 1 (logaritmic)
> [ 274.937349]
> [ 274.937350] cpu#0, 2492.220 MHz
> [ 274.937350] .nr_running : 0
> [ 274.937351] .load : 0
> [ 274.937352] .nr_switches : 105039
> [ 274.937353] .nr_load_updates : 121182
> [ 274.937354] .nr_uninterruptible : 8
> [ 274.937355] .next_balance : 4294.942607
> [ 274.937356] .curr->pid : 0
> [ 274.937357] .clock : 274864.523786
> [ 274.937359] .cpu_load[0] : 0
> [ 274.937360] .cpu_load[1] : 0
> [ 274.937361] .cpu_load[2] : 0
> [ 274.937362] .cpu_load[3] : 0
> [ 274.937363] .cpu_load[4] : 0
> [ 274.937364] .yld_count : 0
> [ 274.937364] .sched_switch : 0
> [ 274.937365] .sched_count : 107487
> [ 274.937366] .sched_goidle : 43368
> [ 274.937367] .avg_idle : 1000000
> [ 274.937368] .ttwu_count : 56534
> [ 274.937369] .ttwu_local : 53925
> [ 274.937370] .bkl_count : 0
> [ 274.937371]
> [ 274.937372] cfs_rq[0]:/system
> [ 274.937372] .exec_clock : 5906.036519
> [ 274.937374] .MIN_vruntime : 0.000001
> [ 274.937375] .min_vruntime : 7138.254657
> [ 274.937376] .max_vruntime : 0.000001
> [ 274.937377] .spread : 0.000000
> [ 274.937378] .spread0 : -14302.149798
> [ 274.937379] .nr_spread_over : 0
> [ 274.937379] .nr_running : 0
> [ 274.937380] .load : 0
> [ 274.937381] .load_avg : 159.999997
> [ 274.937382] .load_period : 5.496191
> [ 274.937383] .load_contrib : 29
> [ 274.937384] .load_tg : 29
> [ 274.937385] .se->exec_start : 272446.423795
> [ 274.937386] .se->vruntime : 21431.398168
> [ 274.937387] .se->sum_exec_runtime : 5906.111805
> [ 274.937388] .se->statistics.wait_start : 0.000000
> [ 274.937389] .se->statistics.sleep_start : 0.000000
> [ 274.937390] .se->statistics.block_start : 0.000000
> [ 274.937391] .se->statistics.sleep_max : 0.000000
> [ 274.937392] .se->statistics.block_max : 0.000000
> [ 274.937393] .se->statistics.exec_max : 15.266617
> [ 274.937394] .se->statistics.slice_max : 5.258344
> [ 274.937395] .se->statistics.wait_max : 7.305679
> [ 274.937396] .se->statistics.wait_sum : 190.506686
> [ 274.937397] .se->statistics.wait_count : 18946
> [ 274.937398] .se->load.weight : 2
> [ 274.937399]
> [ 274.937399] cfs_rq[0]:/
> [ 274.937400] .exec_clock : 17409.691339
> [ 274.937401] .MIN_vruntime : 0.000001
> [ 274.937402] .min_vruntime : 21440.404455
> [ 274.937403] .max_vruntime : 0.000001
> [ 274.937403] .spread : 0.000000
> [ 274.937404] .spread0 : 0.000000
> [ 274.937405] .nr_spread_over : 58
> [ 274.937406] .nr_running : 0
> [ 274.937407] .load : 0
> [ 274.937408] .load_avg : 0.000000
> [ 274.937409] .load_period : 0.000000
> [ 274.937409] .load_contrib : 0
> [ 274.937410] .load_tg : 0
> [ 274.937411]
> [ 274.937412] cfs_rq[0]:/system/rescue.service
> [ 274.937413] .exec_clock : 4319.868509
> [ 274.937414] .MIN_vruntime : 0.000001
> [ 274.937414] .min_vruntime : 4255.887524
> [ 274.937415] .max_vruntime : 0.000001
> [ 274.937416] .spread : 0.000000
> [ 274.937417] .spread0 : -17184.516931
> [ 274.937418] .nr_spread_over : 24
> [ 274.937419] .nr_running : 0
> [ 274.937420] .load : 0
> [ 274.937421] .load_avg : 160.004112
> [ 274.937421] .load_period : 5.496149
> [ 274.937422] .load_contrib : 29
> [ 274.937423] .load_tg : 29
> [ 274.937424] .se->exec_start : 272446.423795
> [ 274.937425] .se->vruntime : 7138.254657
> [ 274.937426] .se->sum_exec_runtime : 4319.796162
> [ 274.937427] .se->statistics.wait_start : 0.000000
> [ 274.937428] .se->statistics.sleep_start : 0.000000
> [ 274.937429] .se->statistics.block_start : 0.000000
> [ 274.937430] .se->statistics.sleep_max : 0.000000
> [ 274.937431] .se->statistics.block_max : 0.000000
> [ 274.937432] .se->statistics.exec_max : 6.213083
> [ 274.937432] .se->statistics.slice_max : 7.894226
> [ 274.937433] .se->statistics.wait_max : 4.870567
> [ 274.937434] .se->statistics.wait_sum : 91.737803
> [ 274.937435] .se->statistics.wait_count : 15504
> [ 274.937436] .se->load.weight : 2
> [ 274.937437]
> [ 274.937438] runnable tasks:
> [ 274.937438] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 274.937439] ----------------------------------------------------------------------------------------------------------
> [ 274.937444]
> [ 274.937445] cpu#1, 2492.220 MHz
> [ 274.937445] .nr_running : 0
> [ 274.937446] .load : 0
> [ 274.937447] .nr_switches : 35274
> [ 274.937448] .nr_load_updates : 105259
> [ 274.937449] .nr_uninterruptible : 1
> [ 274.937449] .next_balance : 4294.942681
> [ 274.937450] .curr->pid : 0
> [ 274.937451] .clock : 274937.357922
> [ 274.937452] .cpu_load[0] : 0
> [ 274.937453] .cpu_load[1] : 0
> [ 274.937454] .cpu_load[2] : 0
> [ 274.937454] .cpu_load[3] : 0
> [ 274.937455] .cpu_load[4] : 0
> [ 274.937456] .yld_count : 0
> [ 274.937457] .sched_switch : 0
> [ 274.937458] .sched_count : 35560
> [ 274.937458] .sched_goidle : 14561
> [ 274.937459] .avg_idle : 1000000
> [ 274.937460] .ttwu_count : 16938
> [ 274.937461] .ttwu_local : 14491
> [ 274.937462] .bkl_count : 0
> [ 274.937463]
> [ 274.937463] cfs_rq[1]:/
> [ 274.937464] .exec_clock : 5221.725565
> [ 274.937465] .MIN_vruntime : 0.000001
> [ 274.937466] .min_vruntime : 9077.562937
> [ 274.937466] .max_vruntime : 0.000001
> [ 274.937467] .spread : 0.000000
> [ 274.937468] .spread0 : -12362.841518
> [ 274.937469] .nr_spread_over : 69
> [ 274.937470] .nr_running : 0
> [ 274.937471] .load : 0
> [ 274.937472] .load_avg : 0.000000
> [ 274.937473] .load_period : 0.000000
> [ 274.937473] .load_contrib : 0
> [ 274.937474] .load_tg : 0
> [ 274.937475]
> [ 274.937476] runnable tasks:
> [ 274.937476] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 274.937477] ----------------------------------------------------------------------------------------------------------
> [ 274.937481]
> [ 274.937482] cpu#2, 2492.220 MHz
> [ 274.937482] .nr_running : 4
> [ 274.937483] .load : 2048
> [ 274.937484] .nr_switches : 7092
> [ 274.937485] .nr_load_updates : 100502
> [ 274.937486] .nr_uninterruptible : 0
> [ 274.937487] .next_balance : 4294.942806
> [ 274.937487] .curr->pid : 46
> [ 274.937488] .clock : 183206.600407
> [ 274.937489] .cpu_load[0] : 2048
> [ 274.937490] .cpu_load[1] : 2048
> [ 274.937491] .cpu_load[2] : 2048
> [ 274.937492] .cpu_load[3] : 2048
> [ 274.937492] .cpu_load[4] : 2048
> [ 274.937493] .yld_count : 0
> [ 274.937494] .sched_switch : 0
> [ 274.937495] .sched_count : 7323
> [ 274.937496] .sched_goidle : 2831
> [ 274.937496] .avg_idle : 1000000
> [ 274.937497] .ttwu_count : 3275
> [ 274.937498] .ttwu_local : 2028
> [ 274.937499] .bkl_count : 0
> [ 274.937500]
> [ 274.937500] cfs_rq[2]:/
> [ 274.937501] .exec_clock : 2187.192139
> [ 274.937502] .MIN_vruntime : 5109.505143
> [ 274.937503] .min_vruntime : 5118.505143
> [ 274.937504] .max_vruntime : 5109.505143
> [ 274.937505] .spread : 0.000000
> [ 274.937506] .spread0 : -16321.899312
> [ 274.937506] .nr_spread_over : 31
> [ 274.937507] .nr_running : 2
> [ 274.937508] .load : 2048
> [ 274.937509] .load_avg : 0.000000
> [ 274.937510] .load_period : 0.000000
> [ 274.937511] .load_contrib : 0
> [ 274.937511] .load_tg : 0
> [ 274.937512]
> [ 274.937513] rt_rq[2]:/
> [ 274.937513] .rt_nr_running : 1
> [ 274.937514] .rt_throttled : 0
> [ 274.937515] .rt_time : 0.000000
> [ 274.937516] .rt_runtime : 950.000000
> [ 274.937517]
> [ 274.937517] runnable tasks:
> [ 274.937518] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 274.937519] ----------------------------------------------------------------------------------------------------------
> [ 274.937520] migration/2 13 0.000000 113 0 0.000000 0.000815 0.000000 /
> [ 274.937524] watchdog/2 16 0.000000 17 0 0.000000 1.785026 0.002124 /
> [ 274.937527] R kswapd0 46 5118.505143 370 120 5118.505143 415.766870 181943.841718 /
> [ 274.937531] kworker/2:1 76 5109.505143 1083 120 5109.505143 25.615305 181183.059001 /
> [ 274.937535]
> [ 274.937536] cpu#3, 2492.220 MHz
> [ 274.937536] .nr_running : 1
> [ 274.937537] .load : 1024
> [ 274.937538] .nr_switches : 22324
> [ 274.937539] .nr_load_updates : 59855
> [ 274.937540] .nr_uninterruptible : 0
> [ 274.937541] .next_balance : 4294.942679
> [ 274.937542] .curr->pid : 73
> [ 274.937543] .clock : 274936.938702
> [ 274.937543] .cpu_load[0] : 0
> [ 274.937544] .cpu_load[1] : 0
> [ 274.937545] .cpu_load[2] : 0
> [ 274.937546] .cpu_load[3] : 0
> [ 274.937546] .cpu_load[4] : 0
> [ 274.937547] .yld_count : 123
> [ 274.937548] .sched_switch : 0
> [ 274.937549] .sched_count : 22774
> [ 274.937550] .sched_goidle : 9952
> [ 274.937551] .avg_idle : 946660
> [ 274.937551] .ttwu_count : 10075
> [ 274.937552] .ttwu_local : 8964
> [ 274.937553] .bkl_count : 0
> [ 274.937554]
> [ 274.937554] cfs_rq[3]:/
> [ 274.937555] .exec_clock : 3244.912177
> [ 274.937556] .MIN_vruntime : 0.000001
> [ 274.937557] .min_vruntime : 7517.482154
> [ 274.937558] .max_vruntime : 0.000001
> [ 274.937559] .spread : 0.000000
> [ 274.937559] .spread0 : -13922.922301
> [ 274.937560] .nr_spread_over : 128
> [ 274.937561] .nr_running : 1
> [ 274.937562] .load : 1024
> [ 274.937563] .load_avg : 0.000000
> [ 274.937564] .load_period : 0.000000
> [ 274.937565] .load_contrib : 0
> [ 274.937565] .load_tg : 0
> [ 274.937567]
> [ 274.937567] runnable tasks:
> [ 274.937567] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 274.937568] ----------------------------------------------------------------------------------------------------------
> [ 274.937571] R kworker/3:1 73 7517.482154 7294 120 7517.482154 264.876541 272702.297840 /
> [ 274.937575]
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote:
> To test the theory, Chris asked me to try with data=ordered.
> Unfortunately, the deadlock still shows up. This is what I get.
As another data point: I'm trying the same kernel with CONFIG_PREEMPT
enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets
pegged at 99% CPU for much of the untar, but it does eventually
complete.
James
On Wed, Apr 27, 2011 at 12:21:42PM -0500, James Bottomley wrote:
> On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> > Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> > case better.
>
> Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
>
> This is the trace with sysrq-l and sysrq-w
>
> The repro this time doesn't have a soft lockup, just the tar is hung and
> one of my CPUs is in 99% system.
>
> James
>
> ---
> [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> [ 454.753048] Call Trace:
> [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> [ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
> [ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
> [ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
> [ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
> [ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
> [ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
> [ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
> [ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
> [ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
> [ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
> [ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13
Looks like it is blocked waiting for the journal.
> [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> [ 454.839453] Call Trace:
> [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
> [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13
which is blocked waiting for _data_ IO completion.
> [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000
> [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282
> [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40
> [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3
> [ 454.899530] Call Trace:
> [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
> [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e
> [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204
> [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a
> [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
> [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
> [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
> [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
> [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
> [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
> [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3
> [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e
> [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
And tar is blocked waiting for the journal, too.
Looks like a case of a ordered mode data writeback journal stall
(the same basic problem as the ext3 fsync sync-the-world issue).
Cheers,
Dave.
--
Dave Chinner
[email protected]
One more data point to add, I've been looking at an identical issue when
copying large amounts of data. I bisected this - and the lockups occur
with commit
3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
issue. With this commit, my file copy test locks up after ~8-10
iterations, before this commit I can copy > 100 times and don't see the
lockup.
On Wed, 2011-04-27 at 15:05 -0500, James Bottomley wrote:
> On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote:
> > To test the theory, Chris asked me to try with data=ordered.
> > Unfortunately, the deadlock still shows up. This is what I get.
>
> As another data point: I'm trying the same kernel with CONFIG_PREEMPT
> enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets
> pegged at 99% CPU for much of the untar, but it does eventually
> complete.
>
> James
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400:
> One more data point to add, I've been looking at an identical issue when
> copying large amounts of data. I bisected this - and the lockups occur
> with commit
> 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> issue. With this commit, my file copy test locks up after ~8-10
> iterations, before this commit I can copy > 100 times and don't see the
> lockup.
Well, that's really interesting. I tried with compaction on here and
couldn't trigger it, but this (very very lightly) tested patch might
help.
It moves the writeout throttle before the goto restart, and also makes
sure we do at least one cond_resched before we loop.
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 6771ea7..cb08b41 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1934,12 +1934,14 @@ restart:
if (inactive_anon_is_low(zone, sc))
shrink_active_list(SWAP_CLUSTER_MAX, zone, sc, priority, 0);
+ throttle_vm_writeout(sc->gfp_mask);
+
/* reclaim/compaction might need reclaim to continue */
if (should_continue_reclaim(zone, nr_reclaimed,
- sc->nr_scanned - nr_scanned, sc))
+ sc->nr_scanned - nr_scanned, sc)) {
+ cond_resched();
goto restart;
-
- throttle_vm_writeout(sc->gfp_mask);
+ }
}
/*
On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote:
> Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400:
> > One more data point to add, I've been looking at an identical issue when
> > copying large amounts of data. I bisected this - and the lockups occur
> > with commit
> > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > issue. With this commit, my file copy test locks up after ~8-10
> > iterations, before this commit I can copy > 100 times and don't see the
> > lockup.
>
> Well, that's really interesting. I tried with compaction on here and
> couldn't trigger it, but this (very very lightly) tested patch might
> help.
>
Thanks Chris,
I've given this a soak test but I still see the same lockup.
> It moves the writeout throttle before the goto restart, and also makes
> sure we do at least one cond_resched before we loop.
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 6771ea7..cb08b41 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -1934,12 +1934,14 @@ restart:
> if (inactive_anon_is_low(zone, sc))
> shrink_active_list(SWAP_CLUSTER_MAX, zone, sc, priority, 0);
>
> + throttle_vm_writeout(sc->gfp_mask);
> +
> /* reclaim/compaction might need reclaim to continue */
> if (should_continue_reclaim(zone, nr_reclaimed,
> - sc->nr_scanned - nr_scanned, sc))
> + sc->nr_scanned - nr_scanned, sc)) {
> + cond_resched();
> goto restart;
> -
> - throttle_vm_writeout(sc->gfp_mask);
> + }
> }
>
> /*
Excerpts from Colin Ian King's message of 2011-04-28 09:42:20 -0400:
>
> On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote:
> > Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400:
> > > One more data point to add, I've been looking at an identical issue when
> > > copying large amounts of data. I bisected this - and the lockups occur
> > > with commit
> > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > > issue. With this commit, my file copy test locks up after ~8-10
> > > iterations, before this commit I can copy > 100 times and don't see the
> > > lockup.
> >
> > Well, that's really interesting. I tried with compaction on here and
> > couldn't trigger it, but this (very very lightly) tested patch might
> > help.
> >
> Thanks Chris,
>
> I've given this a soak test but I still see the same lockup.
Could you post the soft lockups you're seeing?
-chris
On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> One more data point to add, I've been looking at an identical issue when
> copying large amounts of data. I bisected this - and the lockups occur
> with commit
> 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> issue. With this commit, my file copy test locks up after ~8-10
> iterations, before this commit I can copy > 100 times and don't see the
> lockup.
Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
of yours causes kswapd on non-preempt kernels spin for a *long* time...
Honza
> On Wed, 2011-04-27 at 15:05 -0500, James Bottomley wrote:
> > On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote:
> > > To test the theory, Chris asked me to try with data=ordered.
> > > Unfortunately, the deadlock still shows up. This is what I get.
> >
> > As another data point: I'm trying the same kernel with CONFIG_PREEMPT
> > enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets
> > pegged at 99% CPU for much of the untar, but it does eventually
> > complete.
> >
> > James
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to [email protected]
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at http://www.tux.org/lkml/
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Jan Kara <[email protected]>
SUSE Labs, CR
> Could you post the soft lockups you're seeing?
As requested, attached
Colin
>
> -chris
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
Excerpts from Colin Ian King's message of 2011-04-28 10:01:22 -0400:
>
> > Could you post the soft lockups you're seeing?
>
> As requested, attached
These are not good, but they aren't the lockup James was seeing. Were
these messages with my patch? If yes, please post the messages from
without my patch.
-chris
On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote:
> On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> > One more data point to add, I've been looking at an identical issue when
> > copying large amounts of data. I bisected this - and the lockups occur
> > with commit
> > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > issue. With this commit, my file copy test locks up after ~8-10
> > iterations, before this commit I can copy > 100 times and don't see the
> > lockup.
> Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
> of yours causes kswapd on non-preempt kernels spin for a *long* time...
>
I'm still thinking about the traces which do not point the finger
directly at compaction per-se but it's possible that the change means
kswapd is not reclaiming like it should be.
To test this theory, does applying
[d527caf2: mm: compaction: prevent kswapd compacting memory to reduce
CPU usage] help?
--
Mel Gorman
SUSE Labs
On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote:
> > On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> > > One more data point to add, I've been looking at an identical issue when
> > > copying large amounts of data. I bisected this - and the lockups occur
> > > with commit
> > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > > issue. With this commit, my file copy test locks up after ~8-10
> > > iterations, before this commit I can copy > 100 times and don't see the
> > > lockup.
> > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
> > of yours causes kswapd on non-preempt kernels spin for a *long* time...
> >
>
> I'm still thinking about the traces which do not point the finger
> directly at compaction per-se but it's possible that the change means
> kswapd is not reclaiming like it should be.
>
> To test this theory, does applying
> [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce
> CPU usage] help?
I can answer definitively no to this. The upstream kernel I reproduced
this on has that patch included.
James
On Thu 28-04-11 15:01:22, Colin Ian King wrote:
>
> > Could you post the soft lockups you're seeing?
>
> As requested, attached
Hum, what keeps puzzling me is that in all the cases of hangs I've seen
so far, we are stuck waiting for IO to finish for a long time - e.g. in the
traces below kjournald waits for PageWriteback bit to get cleared. Also we
are stuck waiting for page locks which might be because those pages are
being read in? All in all it seems that the IO is just incredibly slow.
But it's not clear to me what pushes us into that situation (especially
since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
underlying blocks are not already allocated.
Honza
[ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
[ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
[ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
[ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
[ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
[ 287.088400] Call Trace:
[ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
[ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
[ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
[ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
[ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
[ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
[ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
[ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
[ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
[ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
[ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
[ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
[ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
[ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
[ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
[ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
[ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
[ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
[ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
[ 287.088484] Call Trace:
[ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
[ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
[ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
[ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
[ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
[ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
[ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
[ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
[ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
[ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
[ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
[ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
[ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
[ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
[ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
[ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
[ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
[ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
[ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
[ 287.088577] Call Trace:
[ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
[ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
[ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
[ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
[ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
[ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
[ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
[ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
[ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
[ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
[ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
[ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
[ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
[ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
[ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
[ 287.088647] Call Trace:
[ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
[ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
[ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
[ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
[ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
[ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
[ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
[ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
[ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
[ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
[ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
[ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
[ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
[ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
[ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
[ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
[ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
[ 287.088723] Call Trace:
[ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
[ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
[ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
[ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
[ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
[ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
[ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
[ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
[ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
[ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
[ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
[ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
[ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
[ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
[ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
[ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
[ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
[ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
[ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
[ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
[ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
[ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
--
Jan Kara <[email protected]>
SUSE Labs, CR
On Thu 28-04-11 16:25:51, Jan Kara wrote:
> On Thu 28-04-11 15:01:22, Colin Ian King wrote:
> >
> > > Could you post the soft lockups you're seeing?
> >
> > As requested, attached
> Hum, what keeps puzzling me is that in all the cases of hangs I've seen
> so far, we are stuck waiting for IO to finish for a long time - e.g. in the
> traces below kjournald waits for PageWriteback bit to get cleared. Also we
> are stuck waiting for page locks which might be because those pages are
> being read in? All in all it seems that the IO is just incredibly slow.
>
> But it's not clear to me what pushes us into that situation (especially
> since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
> underlying blocks are not already allocated.
Hmm, maybe because the system is under memory pressure (and kswapd is not
able to get rid of dirty pages), we page out clean pages. Thus also pages
of executables which need to be paged in soon anyway thus putting heavy
read load on the system which makes writes crawl? I'm not sure why
compaction should make this any worse but maybe it can.
James, Colin, can you capture output of 'vmstat 1' while you do the
copying? Thanks.
Honza
> [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
> [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
> [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
> [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
> [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
> [ 287.088400] Call Trace:
> [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
> [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
> [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
> [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
> [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
> [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
> [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
> [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
> [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
> [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
> [ 287.088484] Call Trace:
> [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
> [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
> [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
> [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
> [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
> [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
> [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
> [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
> [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
> [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
> [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
> [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
> [ 287.088577] Call Trace:
> [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
> [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
> [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
> [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
> [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
> [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
> [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
> [ 287.088647] Call Trace:
> [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
> [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
> [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
> [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
> [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
> [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
> [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
> [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
> [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
> [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
> [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
> [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
> [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
> [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
> [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
> [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
> [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
> [ 287.088723] Call Trace:
> [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
> [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
> [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
> [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
> [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
> [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
> [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
> [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
> [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
> [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
> [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
> [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
> [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
> [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
> --
> Jan Kara <[email protected]>
> SUSE Labs, CR
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Jan Kara <[email protected]>
SUSE Labs, CR
On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote:
> Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400:
> > One more data point to add, I've been looking at an identical issue when
> > copying large amounts of data. I bisected this - and the lockups occur
> > with commit
> > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > issue. With this commit, my file copy test locks up after ~8-10
> > iterations, before this commit I can copy > 100 times and don't see the
> > lockup.
>
> Well, that's really interesting. I tried with compaction on here and
> couldn't trigger it, but this (very very lightly) tested patch might
> help.
>
> It moves the writeout throttle before the goto restart, and also makes
> sure we do at least one cond_resched before we loop.
It seems to take longer, but with a PREEMPT kernel, kswapd eventually
shoots up to 99% during the tar.
James
On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote:
> On Thu 28-04-11 16:25:51, Jan Kara wrote:
> > On Thu 28-04-11 15:01:22, Colin Ian King wrote:
> > >
> > > > Could you post the soft lockups you're seeing?
> > >
> > > As requested, attached
> > Hum, what keeps puzzling me is that in all the cases of hangs I've seen
> > so far, we are stuck waiting for IO to finish for a long time - e.g. in the
> > traces below kjournald waits for PageWriteback bit to get cleared. Also we
> > are stuck waiting for page locks which might be because those pages are
> > being read in? All in all it seems that the IO is just incredibly slow.
> >
> > But it's not clear to me what pushes us into that situation (especially
> > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
> > underlying blocks are not already allocated.
> Hmm, maybe because the system is under memory pressure (and kswapd is not
> able to get rid of dirty pages), we page out clean pages. Thus also pages
> of executables which need to be paged in soon anyway thus putting heavy
> read load on the system which makes writes crawl? I'm not sure why
> compaction should make this any worse but maybe it can.
>
> James, Colin, can you capture output of 'vmstat 1' while you do the
> copying? Thanks.
Attached.
>
> Honza
>
> > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
> > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
> > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
> > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
> > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
> > [ 287.088400] Call Trace:
> > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
> > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
> > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
> > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
> > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
> > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
> > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
> > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
> > [ 287.088484] Call Trace:
> > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
> > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
> > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
> > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
> > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
> > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
> > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
> > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
> > [ 287.088577] Call Trace:
> > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
> > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
> > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
> > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
> > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
> > [ 287.088647] Call Trace:
> > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
> > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
> > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
> > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
> > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
> > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
> > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
> > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
> > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
> > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
> > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
> > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
> > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
> > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
> > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
> > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
> > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
> > [ 287.088723] Call Trace:
> > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
> > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
> > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
> > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
> > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
> > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
> > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
> > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
> > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
> > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
> > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
> > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
> > --
> > Jan Kara <[email protected]>
> > SUSE Labs, CR
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> > the body of a message to [email protected]
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Apr 28, 2011 at 09:25:14AM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote:
> > On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote:
> > > On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> > > > One more data point to add, I've been looking at an identical issue when
> > > > copying large amounts of data. I bisected this - and the lockups occur
> > > > with commit
> > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > > > issue. With this commit, my file copy test locks up after ~8-10
> > > > iterations, before this commit I can copy > 100 times and don't see the
> > > > lockup.
> > > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
> > > of yours causes kswapd on non-preempt kernels spin for a *long* time...
> > >
> >
> > I'm still thinking about the traces which do not point the finger
> > directly at compaction per-se but it's possible that the change means
> > kswapd is not reclaiming like it should be.
> >
> > To test this theory, does applying
> > [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce
> > CPU usage] help?
>
> I can answer definitively no to this. The upstream kernel I reproduced
> this on has that patch included.
>
So it is.
Another consequence of this patch is that when high order allocations
are in progress (is the test case fork heavy in any way for
example? alternatively, it might be something in the storage stack
that requires high-order allocs) we are no longer necessarily going
to sleep because of should_reclaim_continue() check. This could
explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is
set (does unsetting CONFIG_COMPACTION help). If the bug only triggers
for CONFIG_COMPACTION, does the following *untested* patch help any?
(as a warning, I'm offline Friday until Tuesday morning. I'll try
check mail over the weekend but it's unlikely I'll find a terminal
or be allowed to use it without an ass kicking)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 148c6e6..c74a501 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1842,15 +1842,22 @@ static inline bool should_continue_reclaim(struct zone *zone,
return false;
/*
- * If we failed to reclaim and have scanned the full list, stop.
- * NOTE: Checking just nr_reclaimed would exit reclaim/compaction far
- * faster but obviously would be less likely to succeed
- * allocation. If this is desirable, use GFP_REPEAT to decide
- * if both reclaimed and scanned should be checked or just
- * reclaimed
+ * For direct reclaimers
+ * If we failed to reclaim and have scanned the full list, stop.
+ * The caller will check congestion and sleep if necessary until
+ * some IO completes.
+ * For kswapd
+ * Check just nr_reclaimed. If we are failing to reclaim, we
+ * want to stop this reclaim loop, increase the priority and
+ * go to sleep if necessary to allow IO a change to complete.
+ * This avoids kswapd going into a busy loop in shrink_zone()
*/
- if (!nr_reclaimed && !nr_scanned)
- return false;
+ if (!nr_reclaimed) {
+ if (current_is_kswapd())
+ return false;
+ else if (!nr_scanned)
+ return false;
+ }
/*
* If we have not reclaimed enough pages for compaction and the
@@ -1924,8 +1931,13 @@ restart:
/* reclaim/compaction might need reclaim to continue */
if (should_continue_reclaim(zone, nr_reclaimed,
- sc->nr_scanned - nr_scanned, sc))
+ sc->nr_scanned - nr_scanned, sc)) {
+ /* Throttle direct reclaimers if congested */
+ if (!current_is_kswapd())
+ wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
+
goto restart;
+ }
throttle_vm_writeout(sc->gfp_mask);
}
On Thu, 2011-04-28 at 10:04 -0400, Chris Mason wrote:
> Excerpts from Colin Ian King's message of 2011-04-28 10:01:22 -0400:
> >
> > > Could you post the soft lockups you're seeing?
> >
> > As requested, attached
>
> These are not good, but they aren't the lockup James was seeing. Were
> these messages with my patch? If yes, please post the messages from
> without my patch.
Attached are the messages without your patch.
Colin
>
> -chris
On Wed, 2011-04-27 at 19:25 +0100, Colin Ian King wrote:
> Just like to add that I've seen almost identical issues with 2.6.38
> copying large amounts of data to an ext4 filesystem with systems with
> small amounts of memory.
>
> I found that increasing /sys/fs/ext4/sdaX/max_writeback_mb_bump worked
> around the issue.
With the PREEMPT kernel, values of 256 and 512 don't prevent kswapd
spinning up to 99% and staying there.
James
On Thu, 2011-04-28 at 16:08 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 09:25:14AM -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote:
> > > On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote:
> > > > On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> > > > > One more data point to add, I've been looking at an identical issue when
> > > > > copying large amounts of data. I bisected this - and the lockups occur
> > > > > with commit
> > > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > > > > issue. With this commit, my file copy test locks up after ~8-10
> > > > > iterations, before this commit I can copy > 100 times and don't see the
> > > > > lockup.
> > > > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
> > > > of yours causes kswapd on non-preempt kernels spin for a *long* time...
> > > >
> > >
> > > I'm still thinking about the traces which do not point the finger
> > > directly at compaction per-se but it's possible that the change means
> > > kswapd is not reclaiming like it should be.
> > >
> > > To test this theory, does applying
> > > [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce
> > > CPU usage] help?
> >
> > I can answer definitively no to this. The upstream kernel I reproduced
> > this on has that patch included.
> >
>
> So it is.
>
> Another consequence of this patch is that when high order allocations
> are in progress (is the test case fork heavy in any way for
> example?
It's a simple huge untar, so it shouldn't fork.
> alternatively, it might be something in the storage stack
> that requires high-order allocs)
I've tried switching from CFQ to deadline with no apparent changes
> we are no longer necessarily going
> to sleep because of should_reclaim_continue() check. This could
> explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is
> set (does unsetting CONFIG_COMPACTION help). If the bug only triggers
> for CONFIG_COMPACTION, does the following *untested* patch help any?
Turning off COMPACTION and HUGEPAGES doesn't help ... kswapd still goes
to 99% on the PREEMPT kernel, so it doesn't seem to be related
James
On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote:
> James, Colin, can you capture output of 'vmstat 1' while you do the
> copying? Thanks.
Sure; this is with a PREEMPT kernel so the tar goes to completion
without locking the system.
I started using a memory balloon to clear out the page cache.
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 98708 1676100 4112 39556 0 45 1566 1879 392 168 14 17 59 10 0
0 0 98708 1676100 4112 39604 0 0 0 0 37 35 0 0 100 0 0
0 0 98708 1676100 4112 39652 0 0 0 0 31 35 0 0 100 0 0
0 0 98708 1676100 4112 39652 0 0 0 0 26 27 0 0 100 0 0
0 0 98708 1676100 4112 39652 32 0 32 0 50 48 0 0 100 0 0
1 1 98708 1673984 4616 40576 192 0 1812 0 286 117 0 0 92 8 0
0 1 98708 1590540 4792 124236 0 0 41972 0 901 822 0 4 67 29 0
0 1 98708 1499524 4808 214820 0 0 45328 0 917 890 0 4 77 20 0
0 1 98708 1404532 4808 308316 64 0 46784 0 965 1033 0 3 77 20 0
0 1 98708 1312020 5612 399008 0 0 44836 0 1274 1075 0 6 77 18 0
0 1 98708 1304540 5628 406324 0 0 3600 47260 289 115 0 2 68 30 0
0 1 98708 1230760 5672 479020 0 0 36260 712 914 854 0 3 79 18 0
0 1 98708 1208696 5676 500016 0 0 10500 93888 421 251 0 3 67 31 0
0 1 98708 1157632 5676 551264 0 0 25600 0 658 518 0 2 75 23 0
0 1 98708 1052356 5680 655344 0 0 51972 0 1112 1112 0 4 78 19 0
0 2 98708 1024960 5680 679080 0 0 11904 87304 477 277 0 3 54 43 0
1 1 98708 1027288 5684 679432 0 0 128 2896 195 42 0 0 85 14 0
0 1 98708 947124 5688 758528 0 0 39552 4 932 899 0 4 68 28 0
0 1 98708 849792 5692 854272 0 0 47876 4468 1045 936 0 4 77 20 0
0 1 98708 813000 5820 890492 0 0 18176 18300 512 386 0 3 66 32 0
1 2 98708 794304 5820 908132 0 0 8832 95640 379 324 0 2 77 21 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 1 98708 794272 5824 908128 0 0 0 39456 212 58 0 1 73 26 0
0 1 98708 792344 5832 910452 0 0 1156 44440 193 62 0 1 74 25 0
0 1 98708 764392 5836 938288 0 0 13956 0 441 374 0 1 79 19 0
0 1 98708 660828 5840 1040232 0 0 50948 0 1141 992 0 6 65 29 0
0 1 98708 555156 5840 1144216 0 0 51968 0 1072 1008 0 4 76 20 0
0 1 98708 458692 5856 1239212 0 0 47496 120 995 1046 0 3 78 19 0
0 1 98708 411336 5856 1285896 0 0 23296 17764 536 463 0 3 71 26 0
1 1 98708 367716 5856 1328904 0 0 21504 22272 541 463 0 3 78 20 0
0 1 98708 330048 5864 1366044 0 0 18568 17524 529 515 0 2 78 20 0
0 1 98708 301620 5864 1393912 0 0 13952 28148 382 309 0 2 68 30 0
1 1 98708 245140 5872 1449576 0 0 27776 13224 610 567 0 3 73 24 0
0 2 98708 226868 5872 1467000 0 0 8704 114804 375 322 0 2 77 21 0
0 2 98708 226268 5876 1467552 0 0 260 51720 204 40 0 1 75 24 0
0 1 98708 226548 5880 1467536 0 0 4 13180 149 29 0 1 70 30 0
2 0 98708 159792 5880 1533844 0 0 33152 0 721 747 0 2 81 17 0
0 1 98708 55252 5888 1636692 0 0 51456 104 1021 977 0 5 70 25 0
1 1 98708 22804 2680 1671344 0 0 53632 0 1806 1188 0 20 59 21 0
2 1 98708 22832 2672 1671372 0 0 40704 11196 1967 1126 0 22 60 17 0
1 1 98708 22908 2676 1671196 0 0 24580 23040 1621 705 0 35 36 30 0
1 1 98708 22968 2676 1670796 0 0 21504 22212 1576 674 0 22 60 18 0
1 1 98708 22672 2684 1670996 0 0 17920 16968 1450 687 0 27 51 23 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 1 98708 22904 2452 1671344 0 0 24832 25780 1632 714 0 27 50 23 0
1 2 98708 22764 2452 1670724 0 0 15488 96304 1459 531 0 37 33 30 0
2 2 98708 22640 2452 1670724 0 0 0 60300 1219 356 0 20 45 35 0
1 1 98708 22744 2452 1670688 0 0 768 29312 1200 253 0 29 36 34 0
2 0 98708 22964 2460 1671040 0 0 10752 68 1345 445 0 23 56 21 0
1 1 98708 22652 2536 1671560 0 0 51540 0 2113 1246 0 28 49 23 0
1 1 98708 22676 2528 1671768 0 0 118276 14604 4875 2681 0 27 51 22 0
1 1 98708 22644 2528 1671804 0 0 23680 18172 1539 774 0 29 44 27 0
1 1 98708 22720 2536 1671464 0 0 21888 24664 1494 651 0 24 55 22 0
2 1 98708 22672 2496 1671592 0 0 22528 18144 1493 643 0 30 42 28 0
1 1 98708 22732 2488 1671508 0 0 22528 25192 1537 726 0 23 55 21 0
1 2 98708 22884 2488 1670504 0 0 18816 89440 1464 581 0 31 37 32 0
1 2 98708 22740 2488 1670764 0 0 768 75012 1196 254 0 22 55 23 0
1 2 98708 22848 2492 1670600 0 0 256 15512 1225 358 0 29 27 44 0
1 1 98708 22620 2500 1671628 0 0 19076 11408 1490 575 0 23 54 23 0
1 1 98708 22988 2500 1671108 0 0 55040 0 2118 1203 0 27 51 22 0
1 1 98708 22876 2488 1671044 0 0 53760 0 2138 1323 0 28 49 23 0
1 1 98708 22904 2500 1671012 0 0 48012 5528 2049 1097 0 32 43 25 0
1 1 98708 22856 2508 1670876 0 0 21504 18536 1500 646 0 23 55 22 0
2 0 98708 22684 2504 1670972 0 0 23680 23552 1537 784 0 29 44 26 0
1 1 98708 22960 2500 1670520 0 0 23808 25552 1529 677 0 24 55 21 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 1 98708 22820 2504 1670804 0 0 21764 25104 1507 647 0 30 43 26 0
1 1 98708 22880 2504 1670812 0 0 24192 18900 1541 768 0 23 55 22 0
2 3 98708 22716 2512 1670068 0 0 2816 132592 1257 303 0 30 30 40 0
1 1 98708 22944 2512 1669940 0 0 512 40400 1165 247 0 23 57 20 0
1 1 98708 22672 2512 1670624 0 0 512 0 1186 349 0 28 44 29 0
1 1 98708 22788 2504 1670416 0 0 51200 0 2130 1158 0 25 53 22 0
1 1 98708 22660 2508 1670688 0 0 50564 0 1972 1161 0 31 43 26 0
1 1 98708 22780 2516 1670088 0 0 50176 64 2028 1204 0 24 56 20 0
1 1 98708 22856 2508 1669884 0 0 32640 19040 1723 819 0 31 43 26 0
3 0 98708 22808 2512 1669812 0 0 23812 19176 1535 670 0 23 55 22 0
1 1 98708 22744 2504 1669728 0 0 21888 24872 1535 653 0 30 43 27 0
1 1 98708 22696 2504 1669636 0 0 22784 25032 1529 647 0 23 55 21 0
1 1 98708 22632 2512 1669596 0 0 21896 19228 1515 647 0 30 43 27 0
1 2 98708 22676 2512 1668992 0 0 9600 74484 1366 418 0 24 43 33 0
1 2 98708 22780 2512 1668772 0 0 256 107120 1211 245 0 29 42 30 0
1 1 98708 23196 2512 1668920 0 0 256 40 1158 228 0 23 57 20 0
2 1 98708 22968 2512 1669092 0 0 47360 0 2023 1097 0 26 49 25 0
1 1 98708 22864 2704 1668784 0 0 39992 116 1878 994 0 27 49 24 0
1 1 98708 22704 2708 1668872 0 0 53388 0 2166 1206 0 28 49 23 0
1 1 98708 22796 2712 1668472 0 0 39940 12784 1873 966 0 27 52 21 0
2 1 98708 22624 2712 1668508 0 0 21888 21904 1498 645 0 30 43 27 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 1 98708 22700 2704 1668688 0 0 24320 26240 1548 685 0 23 55 22 0
1 1 98708 22776 2712 1668284 0 0 18048 17532 1438 586 0 30 43 27 0
1 1 98708 22868 2712 1668336 0 0 24192 21764 1552 661 0 23 55 23 0
2 2 98708 22772 2716 1667696 0 0 12420 88772 1433 494 0 30 41 28 0
1 2 98708 23424 2720 1667584 0 0 4 9092 1181 246 0 23 40 37 0
1 1 98708 23120 2724 1667584 0 0 4 86380 1262 269 0 29 42 29 0
1 0 98708 23408 2984 1667708 0 0 364 0 1187 249 0 22 60 18 0
2 0 98708 23416 2992 1667772 0 0 0 120 1045 246 0 25 74 1 0
1 0 98708 23408 2992 1667720 0 0 0 0 1025 221 0 26 74 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1030 225 0 28 72 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1024 224 0 23 77 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1034 226 0 28 72 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1022 221 0 23 77 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1032 227 0 28 72 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1024 221 0 23 77 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1034 228 0 28 72 0 0
kswapd goes up to 99% at nearly the end of the second page. I ^C on the
tar but kswapd stays at 99% for the end.
James
This is the output of perf record -g -a -f sleep 5
(hopefully the list won't choke)
James
---
# Events: 1K cycles
#
# Overhead Command Shared Object Symbol
# ........ ........... ................. .......................................
#
9.03% tar [kernel.kallsyms] [k] copy_user_generic_string
|
--- copy_user_generic_string
|
|--60.55%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--39.45%-- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
3.45% swapper [kernel.kallsyms] [k] intel_idle
|
--- intel_idle
cpuidle_idle_call
cpu_idle
|
|--76.47%-- rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--23.53%-- start_secondary
3.39% tar [kernel.kallsyms] [k] zero_user_segments
|
--- zero_user_segments
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.78% tar [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--16.72%-- __percpu_counter_add
| |
| |--59.73%-- __prop_inc_single
| | task_dirty_inc
| | account_page_dirtied
| | __set_page_dirty
| | mark_buffer_dirty
| | __block_commit_write
| | block_write_end
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --40.27%-- ext4_claim_free_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--14.83%-- _raw_spin_lock_irq
| |
| |--63.66%-- add_to_page_cache_locked
| | add_to_page_cache_lru
| | |
| | |--51.51%-- mpage_readpages
| | | ext4_readpages
| | | __do_page_cache_readahead
| | | ra_submit
| | | ondemand_readahead
| | | page_cache_async_readahead
| | | generic_file_aio_read
| | | do_sync_read
| | | vfs_read
| | | sys_read
| | | system_call_fastpath
| | | __GI___libc_read
| | |
| | --48.49%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --36.34%-- blk_throtl_bio
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--13.85%-- bit_spin_lock
| jbd2_journal_add_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--12.19%-- __lru_cache_add
| add_to_page_cache_lru
| |
| |--70.86%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --29.14%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--8.30%-- bit_spin_lock
| lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--58.88%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --41.12%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--7.17%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.90%-- __srcu_read_lock
| fsnotify
| fsnotify_modify
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.61%-- __srcu_read_unlock
| fsnotify
| fsnotify_modify
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.14%-- _raw_read_lock_irqsave
| dm_get_live_table
| dm_merge_bvec
| __bio_add_page.part.2
| bio_add_page
| do_mpage_readpage
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.81%-- _raw_read_lock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.77%-- balance_dirty_pages_ratelimited_nr
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.55%-- __add_bdi_stat
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--2.18%-- _raw_spin_lock
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.70% tar [kernel.kallsyms] [k] __list_del_entry
|
--- __list_del_entry
|
|--72.43%-- list_del
| |
| |--76.01%-- __rmqueue
| | get_page_from_freelist
| | __alloc_pages_nodemask
| | alloc_pages_current
| | |
| | |--92.23%-- __page_cache_alloc
| | | |
| | | |--60.54%-- __do_page_cache_readahead
| | | | ra_submit
| | | | ondemand_readahead
| | | | page_cache_async_readahead
| | | | generic_file_aio_read
| | | | do_sync_read
| | | | vfs_read
| | | | sys_read
| | | | system_call_fastpath
| | | | __GI___libc_read
| | | |
| | | --39.46%-- grab_cache_page_write_begin
| | | ext4_da_write_begin
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | --7.77%-- alloc_slab_page
| | new_slab
| | __slab_alloc
| | kmem_cache_alloc
| | alloc_buffer_head
| | alloc_page_buffers
| | create_empty_buffers
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--18.55%-- get_page_from_freelist
| | __alloc_pages_nodemask
| | alloc_pages_current
| | __page_cache_alloc
| | |
| | |--60.32%-- __do_page_cache_readahead
| | | ra_submit
| | | ondemand_readahead
| | | page_cache_async_readahead
| | | generic_file_aio_read
| | | do_sync_read
| | | vfs_read
| | | sys_read
| | | system_call_fastpath
| | | __GI___libc_read
| | |
| | --39.68%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --5.44%-- activate_page
| mark_page_accessed
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.37%-- __rmqueue
| get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--63.38%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --36.62%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--10.20%-- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--65.70%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--34.30%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.53% tar tar [.] 0x75ff
|
--- 0x407546
|
--- 0x42a9c2
|
--- 0x40753f
|
--- 0x40e790
|
--- 0x407586
|
--- 0x40fe1a
|
--- 0x424b50
|
--- 0x412972
|
--- 0x407501
|
--- 0x433336
|
--- 0x4075ff
|
--- 0x403b40
|
--- 0x4055c7
|
--- 0x4332d9
|
--- 0x405efa
|
--- 0x42a9fe
|
--- 0x405a43
|
--- 0x4332d6
1.44% tar [kernel.kallsyms] [k] __ext4_get_inode_loc
|
--- __ext4_get_inode_loc
ext4_get_inode_loc
|
|--62.07%-- ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| |
| |--94.67%-- generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --5.33%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--37.93%-- ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.43% tar [kernel.kallsyms] [k] kmem_cache_alloc
|
--- kmem_cache_alloc
|
|--66.77%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--22.22%-- jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| |
| |--65.27%-- ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --34.73%-- ext4_dirty_inode
| __mark_inode_dirty
| file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--6.29%-- radix_tree_preload
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--4.72%-- scsi_pool_alloc_command
scsi_host_alloc_command
__scsi_get_command
scsi_get_command
scsi_get_cmd_from_req
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
1.25% tar [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--24.47%-- _raw_spin_unlock_irqrestore
| |
| |--30.14%-- scsi_get_command
| | scsi_get_cmd_from_req
| | scsi_setup_fs_cmnd
| | sd_prep_fn
| | blk_peek_request
| | scsi_request_fn
| | __blk_run_queue
| | queue_unplugged
| | blk_flush_plug_list
| | io_schedule
| | sleep_on_page_killable
| | __wait_on_bit_lock
| | __lock_page_killable
| | lock_page_killable
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| |--28.74%-- prepare_to_wait_exclusive
| | __wait_on_bit_lock
| | __lock_page_killable
| | lock_page_killable
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| |--23.48%-- __wake_up
| | jbd2_journal_stop
| | __ext4_journal_stop
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --17.65%-- blkiocg_update_dispatch_stats
| throtl_charge_bio
| blk_throtl_bio
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--12.68%-- radix_tree_preload_end
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--12.51%-- pagefault_enable
| iov_iter_copy_from_user_atomic
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.90%-- _raw_spin_unlock_irq
| |
| |--52.31%-- add_to_page_cache_locked
| | add_to_page_cache_lru
| | mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --47.69%-- __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.37%-- mem_cgroup_charge_statistics
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--9.99%-- _raw_spin_unlock
| |
| |--60.73%-- inode_add_rsv_space
| | __dquot_alloc_space
| | ext4_da_get_block_prep
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --39.27%-- ext4_ext_map_blocks
| ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--5.89%-- __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.86%-- balance_dirty_pages_ratelimited_nr
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.52%-- add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--3.81%-- avc_has_perm_noaudit
task_has_capability
selinux_capable
security_capable
ns_capable
generic_permission
inode_permission
may_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
1.24% tar [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
|
|--94.43%-- lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--56.27%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --43.73%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--5.57%-- __mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.13% swapper [kernel.kallsyms] [k] poll_idle
|
--- poll_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
1.00% tar [kernel.kallsyms] [k] ext4_da_get_block_prep
|
--- ext4_da_get_block_prep
|
|--82.10%-- __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--17.90%-- ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.95% tar [kernel.kallsyms] [k] __find_get_block
|
--- __find_get_block
|
|--85.78%-- __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| |
| |--84.94%-- ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --15.06%-- ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--7.86%-- ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--6.36%-- ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.92% tar [kernel.kallsyms] [k] radix_tree_lookup_element
|
--- radix_tree_lookup_element
|
|--79.19%-- radix_tree_lookup_slot
| find_get_page
| |
| |--39.67%-- __find_get_block_slow
| | unmap_underlying_metadata
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--37.78%-- generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --22.55%-- find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--13.79%-- radix_tree_lookup
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--7.02%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.86% tar [kernel.kallsyms] [k] ext4_mark_iloc_dirty
|
--- ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.84% tar [kernel.kallsyms] [k] __alloc_pages_nodemask
|
--- __alloc_pages_nodemask
|
|--88.77%-- alloc_pages_current
| __page_cache_alloc
| |
| |--53.12%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --46.88%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--11.23%-- __page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.80% tar [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
|
|--27.09%-- add_preempt_count
| |
| |--44.58%-- jbd_lock_bh_state
| | do_get_write_access
| | jbd2_journal_get_write_access
| | __ext4_journal_get_write_access
| | ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--32.43%-- bit_spin_lock
| | lock_page_cgroup
| | __mem_cgroup_commit_charge.constprop.28
| | __mem_cgroup_commit_charge_lrucare
| | mem_cgroup_cache_charge
| | add_to_page_cache_locked
| | add_to_page_cache_lru
| | grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --22.99%-- _raw_spin_lock_irqsave
| __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--26.54%-- sub_preempt_count
| |
| |--43.60%-- __percpu_counter_add
| | ext4_claim_free_blocks
| | ext4_da_get_block_prep
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--29.86%-- bit_spin_unlock
| | unlock_page_cgroup
| | __mem_cgroup_commit_charge.constprop.28
| | __mem_cgroup_commit_charge_lrucare
| | mem_cgroup_cache_charge
| | add_to_page_cache_locked
| | add_to_page_cache_lru
| | mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --26.53%-- __lru_cache_add
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--13.83%-- radix_tree_preload
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--9.64%-- jbd_lock_bh_state
| jbd2_journal_dirty_metadata
| __ext4_handle_dirty_metadata
| ext4_mark_iloc_dirty
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--8.02%-- bit_spin_unlock
| jbd2_journal_add_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--7.60%-- _raw_spin_lock
| inode_add_rsv_space
| __dquot_alloc_space
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--7.27%-- balance_dirty_pages_ratelimited_nr
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.74% tar [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
|
|--55.40%-- _raw_spin_lock_irq
| |
| |--86.22%-- __set_page_dirty
| | mark_buffer_dirty
| | __block_commit_write
| | block_write_end
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --13.78%-- blk_throtl_bio
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--44.60%-- _raw_spin_lock
|
|--76.80%-- inode_add_rsv_space
| __dquot_alloc_space
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--23.20%-- ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.71% tar [kernel.kallsyms] [k] put_page_testzero
|
--- put_page_testzero
put_page
|
|--49.42%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--26.92%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--23.66%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.69% tar [kernel.kallsyms] [k] get_page_from_freelist
|
--- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--63.44%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--36.56%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.67% tar [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
|
|--90.71%-- unlock_page
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--9.29%-- wake_up_bit
unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.65% tar [kernel.kallsyms] [k] put_mems_allowed
|
--- put_mems_allowed
|
|--92.20%-- alloc_pages_current
| __page_cache_alloc
| |
| |--54.97%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --45.03%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--7.80%-- __page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.65% tar [kernel.kallsyms] [k] __rmqueue
|
--- __rmqueue
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--70.26%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--29.74%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.62% tar [kernel.kallsyms] [k] start_this_handle
|
--- start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--71.12%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--28.88%-- ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.58% tar [kernel.kallsyms] [k] __rcu_read_unlock
|
--- __rcu_read_unlock
|
|--35.45%-- __prop_inc_single
| task_dirty_inc
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--14.77%-- find_get_page
| find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--11.81%-- __find_get_block_slow
| __find_get_block
| ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--11.28%-- avc_has_perm_noaudit
| avc_has_perm
| inode_has_perm
| selinux_inode_permission
| security_inode_permission
| inode_permission
| may_delete
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--10.21%-- __mem_cgroup_try_charge.constprop.27
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--8.34%-- generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--8.15%-- dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.58% tar [kernel.kallsyms] [k] __mark_inode_dirty
|
--- __mark_inode_dirty
|
|--54.06%-- __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--33.43%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--12.51%-- ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.55% tar [kernel.kallsyms] [k] _raw_spin_unlock
|
--- _raw_spin_unlock
|
|--55.58%-- ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.42%-- create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.53% tar [kernel.kallsyms] [k] next_zones_zonelist
|
--- next_zones_zonelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--55.47%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--44.53%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.52% tar [kernel.kallsyms] [k] bit_spin_unlock
|
--- bit_spin_unlock
|
|--55.87%-- jbd2_journal_put_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| |
| |--71.83%-- ext4_dirty_inode
| | __mark_inode_dirty
| | |
| | |--57.59%-- generic_write_end
| | | ext4_da_write_end
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | --42.41%-- file_update_time
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --28.17%-- add_dirent_to_buf
| ext4_add_entry
| ext4_add_nondir
| ext4_create
| vfs_create
| do_last
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
--44.13%-- jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.51% tar [kernel.kallsyms] [k] jbd_unlock_bh_state
|
--- jbd_unlock_bh_state
|
|--56.56%-- do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| |
| |--85.42%-- ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | |
| | |--77.84%-- generic_write_end
| | | ext4_da_write_end
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | --22.16%-- file_update_time
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --14.58%-- ext4_free_inode
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--43.44%-- jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
|
|--53.02%-- ext4_free_inode
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--46.98%-- ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.50% tar [kernel.kallsyms] [k] __block_write_begin
|
--- __block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.49% tar [kernel.kallsyms] [k] __might_sleep
|
--- __might_sleep
|
|--30.54%-- jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.09%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--14.20%-- __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.60%-- mutex_lock
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.58%-- kmem_cache_alloc
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--9.67%-- lock_buffer
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--7.31%-- ext4_map_blocks
ext4_getblk
ext4_bread
dx_probe
ext4_add_entry
ext4_add_nondir
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.49% tar [kernel.kallsyms] [k] jbd2_journal_add_journal_head
|
--- jbd2_journal_add_journal_head
|
|--69.06%-- __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| |
| |--77.62%-- generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --22.38%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--30.94%-- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.48% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge.constprop.28
|
--- __mem_cgroup_commit_charge.constprop.28
|
|--83.14%-- __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--56.52%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --43.48%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--16.86%-- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.46% tar [kernel.kallsyms] [k] __rcu_read_lock
|
--- __rcu_read_lock
|
|--30.47%-- avc_has_perm_noaudit
| |
| |--54.86%-- task_has_capability
| | selinux_capable
| | security_capable
| | ns_capable
| | exec_permission
| | link_path_walk
| | path_lookupat
| | do_path_lookup
| | user_path_parent
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --45.14%-- avc_has_perm
| may_create
| selinux_inode_create
| security_inode_create
| vfs_create
| do_last
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
|--20.91%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--18.25%-- find_get_page
| find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.06%-- __account_system_time
| irqtime_account_process_tick
| account_process_tick
| update_process_times
| tick_sched_timer
| __run_hrtimer
| hrtimer_interrupt
| smp_apic_timer_interrupt
| apic_timer_interrupt
| rw_verify_area
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--13.31%-- __prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.44% tar [kernel.kallsyms] [k] jbd2_journal_cancel_revoke
|
--- jbd2_journal_cancel_revoke
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.44% tar [kernel.kallsyms] [k] mark_page_accessed
|
--- mark_page_accessed
|
|--56.95%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--15.53%-- do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--14.54%-- __find_get_block
| __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--12.98%-- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.44% tar [kernel.kallsyms] [k] put_mems_allowed
|
--- put_mems_allowed
|
|--55.41%-- alloc_pages_current
| __page_cache_alloc
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.59%-- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--68.42%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--31.58%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.43% tar [kernel.kallsyms] [k] in_lock_functions
|
--- in_lock_functions
|
|--76.43%-- get_parent_ip
| |
| |--80.00%-- add_preempt_count
| | |
| | |--39.87%-- balance_dirty_pages_ratelimited_nr
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | |--30.07%-- radix_tree_preload
| | | add_to_page_cache_locked
| | | add_to_page_cache_lru
| | | grab_cache_page_write_begin
| | | ext4_da_write_begin
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | --30.07%-- _raw_read_lock
| | start_this_handle
| | jbd2__journal_start
| | jbd2_journal_start
| | ext4_journal_start_sb
| | ext4_dirty_inode
| | __mark_inode_dirty
| | ext4_setattr
| | notify_change
| | chown_common
| | sys_fchown
| | system_call_fastpath
| | __GI___fchown
| |
| --20.00%-- sub_preempt_count
| avc_has_perm_noaudit
| avc_has_perm
| inode_has_perm
| dentry_has_perm
| selinux_inode_setattr
| security_inode_setattr
| notify_change
| chown_common
| sys_fchown
| system_call_fastpath
| __GI___fchown
|
|--12.38%-- add_preempt_count
| jbd_lock_bh_state
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--11.19%-- sub_preempt_count
_raw_spin_unlock_irqrestore
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.43% swapper [kernel.kallsyms] [k] ext4_end_bio
|
--- ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.42% tar [kernel.kallsyms] [k] ext4_get_group_desc
|
--- ext4_get_group_desc
|
|--88.05%-- __ext4_get_inode_loc
| ext4_get_inode_loc
| |
| |--64.96%-- ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --35.04%-- ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--11.95%-- ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.42% tar [kernel.kallsyms] [k] __percpu_counter_add
|
--- __percpu_counter_add
|
|--39.75%-- ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--25.32%-- ext4_claim_free_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--19.28%-- ext4_free_inode
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--15.64%-- __add_bdi_stat
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.42% tar [kernel.kallsyms] [k] do_get_write_access
|
--- do_get_write_access
|
|--80.57%-- jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--19.43%-- __ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.42% tar [kernel.kallsyms] [k] generic_file_buffered_write
|
--- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.41% tar [kernel.kallsyms] [k] jbd2_journal_stop
|
--- jbd2_journal_stop
__ext4_journal_stop
|
|--79.71%-- ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--20.29%-- ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.41% tar [kernel.kallsyms] [k] jbd2__journal_start
|
--- jbd2__journal_start
|
|--82.99%-- jbd2_journal_start
| ext4_journal_start_sb
| |
| |--83.87%-- ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --16.13%-- ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--17.01%-- ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.40% tar [kernel.kallsyms] [k] do_mpage_readpage
|
--- do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.39% tar [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
__might_sleep
|
|--55.35%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--61.18%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --38.82%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--28.25%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--16.40%-- __getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.38% tar [kernel.kallsyms] [k] alloc_pages_current
|
--- alloc_pages_current
|
|--82.52%-- __page_cache_alloc
| |
| |--74.42%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --25.58%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--17.48%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.37% tar [kernel.kallsyms] [k] __nr_to_section
|
--- __nr_to_section
lookup_page_cgroup
|
|--70.03%-- __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.38%-- mem_cgroup_get_reclaim_stat_from_page
| update_page_reclaim_stat
| ____pagevec_lru_add_fn
| pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--12.60%-- mem_cgroup_del_lru_list
activate_page
mark_page_accessed
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.37% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
|
|--52.73%-- _raw_spin_unlock_irqrestore
| |
| |--51.73%-- __wake_up
| | jbd2_journal_stop
| | __ext4_journal_stop
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --48.27%-- pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.27%-- __wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.36% tar [kernel.kallsyms] [k] __mem_cgroup_try_charge.constprop.27
|
--- __mem_cgroup_try_charge.constprop.27
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--67.66%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--32.34%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.36% swapper [kernel.kallsyms] [k] unlock_page
|
--- unlock_page
|
|--85.03%-- mpage_end_io
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--14.97%-- bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.35% tar [kernel.kallsyms] [k] ext4_xattr_get
|
--- ext4_xattr_get
|
|--68.95%-- ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--31.05%-- generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.35% perf [kernel.kallsyms] [k] seq_puts
|
--- seq_puts
render_sigset_t
proc_pid_status
proc_single_show
seq_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.35% tar [kernel.kallsyms] [k] grab_cache_page_write_begin
|
--- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.34% tar [kernel.kallsyms] [k] alloc_page_buffers
|
--- alloc_page_buffers
|
|--72.20%-- create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--27.80%-- __block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.34% tar [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
|
|--58.75%-- get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--70.45%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --29.55%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--41.25%-- __lru_cache_add
add_to_page_cache_lru
|
|--61.47%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--38.53%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.34% perf [kernel.kallsyms] [k] clear_page_c
|
--- clear_page_c
__alloc_pages_nodemask
perf_mmap_alloc_page
perf_mmap
mmap_region
do_mmap_pgoff
sys_mmap_pgoff
sys_mmap
system_call_fastpath
__mmap
0x418523
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.34% tar [kernel.kallsyms] [k] iov_iter_copy_from_user_atomic
|
--- iov_iter_copy_from_user_atomic
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.33% tar [kernel.kallsyms] [k] page_cache_get_speculative
|
--- page_cache_get_speculative
find_get_page
|
|--84.05%-- find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--15.95%-- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.33% top [kernel.kallsyms] [k] follow_managed
|
--- follow_managed
walk_component
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.33% flush-253:2 [kernel.kallsyms] [k] ext4_num_dirty_pages
|
--- ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.32% swapper [kernel.kallsyms] [k] do_timer
|
--- do_timer
tick_do_update_jiffies64
|
|--61.57%-- tick_sched_timer
| __run_hrtimer
| hrtimer_interrupt
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--38.43%-- tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.32% tar [kernel.kallsyms] [k] __generic_file_aio_write
|
--- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.32% tar [kernel.kallsyms] [k] mem_cgroup_add_lru_list
|
--- mem_cgroup_add_lru_list
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
|
|--87.33%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--12.67%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.32% tar [kernel.kallsyms] [k] ext4_da_write_end
|
--- ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.32% flush-253:2 [kernel.kallsyms] [k] ext4_bio_write_page
|
--- ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.32% swapper [kernel.kallsyms] [k] tick_nohz_stop_sched_tick
|
--- tick_nohz_stop_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.31% tar [kernel.kallsyms] [k] down_read
|
--- down_read
|
|--78.35%-- ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--21.65%-- ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.31% tar [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--83.21%-- __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--16.79%-- force_qs_rnp
force_quiescent_state
__rcu_process_callbacks
rcu_process_callbacks
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.30% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--59.37%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--40.63%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.30% perf [kernel.kallsyms] [k] file_update_time
|
--- file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0.30% tar [kernel.kallsyms] [k] mark_buffer_dirty
|
--- mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.29% swapper [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--54.58%-- _raw_spin_lock_irqsave
| |
| |--57.58%-- scsi_device_unbusy
| | scsi_finish_command
| | scsi_softirq_done
| | blk_done_softirq
| | __do_softirq
| | call_softirq
| | do_softirq
| | irq_exit
| | do_IRQ
| | common_interrupt
| | cpuidle_idle_call
| | cpu_idle
| | rest_init
| | start_kernel
| | x86_64_start_reservations
| | x86_64_start_kernel
| |
| --42.42%-- test_clear_page_writeback
| end_page_writeback
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--20.90%-- __percpu_counter_add
| __add_bdi_stat
| test_clear_page_writeback
| end_page_writeback
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--15.95%-- irq_enter
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--8.56%-- scsi_device_unbusy
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.29% tar [kernel.kallsyms] [k] mnt_want_write_file
|
--- mnt_want_write_file
|
|--76.04%-- __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--23.96%-- system_call_fastpath
__GI___fchown
0.27% tar [kernel.kallsyms] [k] find_busiest_group
|
--- find_busiest_group
load_balance
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.26% perf [kernel.kallsyms] [k] perf_event_mmap_output
|
--- perf_event_mmap_output
perf_event_mmap_ctx
perf_event_mmap
mmap_region
do_mmap_pgoff
sys_mmap_pgoff
sys_mmap
system_call_fastpath
__mmap
0.26% swapper [kernel.kallsyms] [k] kmem_cache_free
|
--- kmem_cache_free
|
|--59.24%-- mempool_free
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--40.76%-- scsi_pool_free_command
__scsi_put_command
scsi_put_command
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.26% perf [kernel.kallsyms] [k] fsnotify
|
--- fsnotify
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4293b8
0x429c0a
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.26% perf [kernel.kallsyms] [k] put_bh
|
--- put_bh
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4293b8
0x429c0a
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.26% perf [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
prepend_path
path_with_deleted
d_path
seq_path
show_map_vma
show_map
seq_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.26% tar [kernel.kallsyms] [k] get_page
|
--- get_page
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.26% tar [kernel.kallsyms] [k] selinux_file_permission
|
--- selinux_file_permission
security_file_permission
rw_verify_area
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.26% tar [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
need_resched
should_resched
_cond_resched
|
|--64.24%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--35.76%-- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.26% perf [kernel.kallsyms] [k] number
|
--- number
vsnprintf
seq_printf
show_map_vma
show_map
seq_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.25% tar libc-2.13.90.so [.] __GI___libc_write
|
--- __GI___libc_write
0.25% tar [kernel.kallsyms] [k] test_and_set_bit
|
--- test_and_set_bit
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.25% perf [kernel.kallsyms] [k] vsnprintf
|
--- vsnprintf
seq_printf
show_map_vma
show_map
seq_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.25% tar [kernel.kallsyms] [k] run_timer_softirq
|
--- run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.25% perf [kernel.kallsyms] [k] setup_object
|
--- setup_object
new_slab
__slab_alloc
kmem_cache_alloc
d_alloc
proc_fill_cache
proc_task_readdir
vfs_readdir
sys_getdents
system_call_fastpath
__getdents64
0x429beb
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.24% swapper [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--40.64%-- __bit_spin_unlock.constprop.24
| __slab_free
| kmem_cache_free
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--33.53%-- _raw_spin_unlock_irqrestore
| rcu_report_qs_rnp
| __rcu_process_callbacks
| rcu_process_callbacks
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--25.82%-- __slab_free
kmem_cache_free
mempool_free_slab
mempool_free
bio_free
bio_fs_destructor
bio_put
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.24% tar [kernel.kallsyms] [k] add_to_page_cache_locked
|
--- add_to_page_cache_locked
|
|--64.11%-- add_to_page_cache_lru
| |
| |--63.51%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --36.49%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--35.89%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.24% tar [kernel.kallsyms] [k] file_remove_suid
|
--- file_remove_suid
|
|--81.19%-- __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--18.81%-- generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.24% tar [kernel.kallsyms] [k] clear_bit_unlock
|
--- clear_bit_unlock
|
|--71.97%-- unlock_buffer
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| |
| |--58.50%-- file_update_time
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --41.50%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--28.03%-- do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.23% tar [kernel.kallsyms] [k] sys_write
|
--- sys_write
system_call_fastpath
__GI___libc_write
0.23% swapper [kernel.kallsyms] [k] SetPageUptodate
|
--- SetPageUptodate
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.23% tar [kernel.kallsyms] [k] __bio_add_page.part.2
|
--- __bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.23% swapper [kernel.kallsyms] [k] rcu_irq_enter
|
--- rcu_irq_enter
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.23% tar [kernel.kallsyms] [k] _raw_spin_unlock_irq
|
--- _raw_spin_unlock_irq
|
|--45.40%-- generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--29.98%-- mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--24.62%-- add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.23% tar [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
|
|--51.02%-- jbd2_journal_put_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.98%-- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.22% swapper [kernel.kallsyms] [k] nr_iowait_cpu
|
--- nr_iowait_cpu
tick_nohz_stop_idle
tick_check_idle
irq_enter
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.22% flush-253:2 [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
|
|--51.36%-- unlock_page
| ext4_bio_write_page
| mpage_da_submit_io
| mpage_da_map_and_submit
| ext4_da_writepages
| do_writepages
| writeback_single_inode
| writeback_sb_inodes
| writeback_inodes_wb
| wb_writeback
| wb_do_writeback
| bdi_writeback_thread
| kthread
| kernel_thread_helper
|
--48.64%-- ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.22% tar [kernel.kallsyms] [k] __jbd2_log_space_left
|
--- __jbd2_log_space_left
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--50.77%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--49.23%-- ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.22% tar [kernel.kallsyms] [k] mb_test_bit
|
--- mb_test_bit
mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.22% tar [kernel.kallsyms] [k] __blk_recalc_rq_segments
|
--- __blk_recalc_rq_segments
blk_recount_segments
bio_phys_segments
blk_rq_bio_prep
init_request_from_bio
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.22% tar [kernel.kallsyms] [k] generic_write_end
|
--- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.22% tar [kernel.kallsyms] [k] jbd2_journal_dirty_metadata
|
--- jbd2_journal_dirty_metadata
|
|--62.74%-- __ext4_handle_dirty_metadata
| ext4_mark_iloc_dirty
| ext4_mark_inode_dirty
| |
| |--59.79%-- ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --40.21%-- ext4_unlink
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--37.26%-- ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.22% tar [kernel.kallsyms] [k] __mem_cgroup_threshold
|
--- __mem_cgroup_threshold
memcg_check_events
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.21% tar [kernel.kallsyms] [k] kfree
|
--- kfree
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--79.14%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--20.86%-- ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.21% tar [kernel.kallsyms] [k] ext4_get_inode_flags
|
--- ext4_get_inode_flags
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
|
|--63.71%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--36.29%-- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.21% flush-253:2 [kernel.kallsyms] [k] put_page_testzero
|
--- put_page_testzero
release_pages
__pagevec_release
pagevec_release
|
|--57.93%-- ext4_num_dirty_pages
| ext4_da_writepages
| do_writepages
| writeback_single_inode
| writeback_sb_inodes
| writeback_inodes_wb
| wb_writeback
| wb_do_writeback
| bdi_writeback_thread
| kthread
| kernel_thread_helper
|
--42.07%-- write_cache_pages_da
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.21% flush-253:2 [kernel.kallsyms] [k] __rcu_read_unlock
|
--- __rcu_read_unlock
__find_get_block_slow
unmap_underlying_metadata
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.21% tar [kernel.kallsyms] [k] avtab_search
|
--- avtab_search
security_compute_sid
security_transition_sid
may_create
selinux_inode_create
security_inode_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.21% tar [kernel.kallsyms] [k] search_dirblock
|
--- search_dirblock
ext4_dx_find_entry
ext4_find_entry
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.21% tar [kernel.kallsyms] [k] __sg_alloc_table
|
--- __sg_alloc_table
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.21% tar [kernel.kallsyms] [k] ext4_journal_start_sb
|
--- ext4_journal_start_sb
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.21% top libc-2.13.90.so [.] __GI_____strtoll_l_internal
|
--- __GI_____strtoll_l_internal
0.20% swapper [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock
|
|--57.86%-- handle_edge_irq
| handle_irq
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--42.14%-- tick_do_update_jiffies64
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.20% tar [kernel.kallsyms] [k] generic_write_sync
|
--- generic_write_sync
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.20% tar [kernel.kallsyms] [k] put_page
|
--- put_page
|
|--64.65%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--35.35%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.20% flush-253:2 [kernel.kallsyms] [k] tag_get
|
--- tag_get
|
|--58.46%-- __lookup_tag
| radix_tree_gang_lookup_tag_slot
| find_get_pages_tag
| pagevec_lookup_tag
| ext4_num_dirty_pages
| ext4_da_writepages
| do_writepages
| writeback_single_inode
| writeback_sb_inodes
| writeback_inodes_wb
| wb_writeback
| wb_do_writeback
| bdi_writeback_thread
| kthread
| kernel_thread_helper
|
--41.54%-- test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.20% tar [kernel.kallsyms] [k] dm_merge_bvec
|
--- dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.19% tar [kernel.kallsyms] [k] __ext4_handle_dirty_metadata
|
--- __ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.19% tar [kernel.kallsyms] [k] __block_commit_write
|
--- __block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.19% tar [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore
|
--- _raw_spin_unlock_irqrestore
|
|--54.81%-- __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--45.19%-- __delayacct_blkio_end
delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.19% tar [kernel.kallsyms] [k] ext4_writepage_trans_blocks
|
--- ext4_writepage_trans_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.19% tar [kernel.kallsyms] [k] jbd_lock_bh_state
|
--- jbd_lock_bh_state
|
|--62.13%-- do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--37.87%-- jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.19% tar [kernel.kallsyms] [k] __list_add
|
--- __list_add
|
|--70.74%-- __make_request
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--29.26%-- expand
__rmqueue
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.19% tar [kernel.kallsyms] [k] fsnotify_modify
|
--- fsnotify_modify
|
|--74.13%-- vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--25.87%-- sys_write
system_call_fastpath
__GI___libc_write
0.19% flush-253:2 [kernel.kallsyms] [k] lock_page
|
--- lock_page
write_cache_pages_da
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.19% flush-253:2 [kernel.kallsyms] [k] rb_next
|
--- rb_next
set_next_entity
pick_next_task_fair
pick_next_task
schedule
preempt_schedule_irq
retint_kernel
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.19% tar [kernel.kallsyms] [k] ext4_ext_calc_metadata_amount
|
--- ext4_ext_calc_metadata_amount
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.18% swapper [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
|
|--61.24%-- unlock_page
| mpage_end_io
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--38.76%-- end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.18% swapper [kernel.kallsyms] [k] select_nohz_load_balancer
|
--- select_nohz_load_balancer
|
|--75.59%-- tick_nohz_restart_sched_tick
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--24.41%-- tick_nohz_stop_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.18% swapper [kernel.kallsyms] [k] menu_select
|
--- menu_select
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.18% flush-253:2 [kernel.kallsyms] [k] page_cache_get_speculative
|
--- page_cache_get_speculative
find_get_pages
pagevec_lookup
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.18% tar [kernel.kallsyms] [k] generic_file_aio_read
|
--- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.18% tar [kernel.kallsyms] [k] __prop_inc_single
|
--- __prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.18% swapper [kernel.kallsyms] [k] page_index
|
--- page_index
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.17% tar [kernel.kallsyms] [k] arch_read_lock
|
--- arch_read_lock
_raw_read_lock
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] mutex_unlock
|
--- mutex_unlock
|
|--73.48%-- generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--26.52%-- do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.17% tar [kernel.kallsyms] [k] blk_rq_map_sg
|
--- blk_rq_map_sg
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.17% tar [kernel.kallsyms] [k] bit_waitqueue
|
--- bit_waitqueue
|
|--51.20%-- wake_up_bit
| unlock_buffer
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.80%-- unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] SetPageLRU
|
--- SetPageLRU
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
|
|--71.47%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--28.53%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6
|
--- test_ti_thread_flag.constprop.6
|
|--64.90%-- _raw_spin_unlock_irq
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--35.10%-- inode_add_rsv_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] lookup_page_cgroup
|
--- lookup_page_cgroup
|
|--58.52%-- __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--41.48%-- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] generic_make_request
|
--- generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.17% tar [kernel.kallsyms] [k] lock_page_cgroup
|
--- lock_page_cgroup
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% swapper [kernel.kallsyms] [k] mempool_free
|
--- mempool_free
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.17% tar [kernel.kallsyms] [k] put_prev_task_fair
|
--- put_prev_task_fair
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.17% tar [kernel.kallsyms] [k] perf_event_task_tick
|
--- perf_event_task_tick
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
|
|--51.07%-- scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.93%-- _ext4_get_block
ext4_get_block
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.17% tar [kernel.kallsyms] [k] arch_read_unlock
|
--- arch_read_unlock
|
|--66.07%-- _raw_read_unlock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--33.93%-- _raw_read_unlock_irqrestore
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.16% swapper [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
|
|--38.89%-- __cycles_2_ns
| native_sched_clock
| sched_clock
| sched_clock_cpu
| account_system_vtime
| irq_exit
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--32.56%-- tick_check_idle
| irq_enter
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--14.27%-- account_idle_time
| irqtime_account_process_tick
| account_idle_ticks
| tick_nohz_restart_sched_tick
| cpu_idle
| start_secondary
|
--14.27%-- irqtime_account_process_tick
account_idle_ticks
tick_nohz_restart_sched_tick
cpu_idle
start_secondary
0.16% tar [kernel.kallsyms] [k] fget_light
|
--- fget_light
|
|--51.07%-- sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.93%-- sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] SetPageUptodate
|
--- SetPageUptodate
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] __ext4_journal_stop
|
--- __ext4_journal_stop
|
|--50.22%-- ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--49.78%-- ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] ext4_clear_inode_state
|
--- ext4_clear_inode_state
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
|
|--51.89%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.11%-- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] mem_cgroup_cache_charge
|
--- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--65.55%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--34.45%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] ext4_map_blocks
|
--- ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] ata_dev_enabled
|
--- ata_dev_enabled
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.16% tar [kernel.kallsyms] [k] new_slab
|
--- new_slab
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% swapper [kernel.kallsyms] [k] hrtimer_cancel
|
--- hrtimer_cancel
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.15% tar [kernel.kallsyms] [k] lock_buffer
|
--- lock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% tar [kernel.kallsyms] [k] bit_spin_unlock
|
--- bit_spin_unlock
unlock_page_cgroup
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--72.67%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--27.33%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% tar [kernel.kallsyms] [k] up_read
|
--- up_read
|
|--62.56%-- ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--37.44%-- ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% tar [kernel.kallsyms] [k] preempt_schedule
|
--- preempt_schedule
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_read_lock
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% tar [kernel.kallsyms] [k] fsnotify_access
|
--- fsnotify_access
sys_read
system_call_fastpath
__GI___libc_read
0.15% swapper [kernel.kallsyms] [k] native_read_tsc
|
--- native_read_tsc
paravirt_read_tsc
native_sched_clock
sched_clock
sched_clock_cpu
account_system_vtime
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.14% tar [kernel.kallsyms] [k] dequeue_task_fair
|
--- dequeue_task_fair
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.14% tar [kernel.kallsyms] [k] ata_scsi_queuecmd
|
--- ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.14% tar [kernel.kallsyms] [k] file_read_actor
|
--- file_read_actor
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.14% tar [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
|
|--52.50%-- ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.50%-- ext4_ext_map_blocks
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.14% top [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
_raw_spin_lock
nameidata_drop_rcu
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.14% tar [kernel.kallsyms] [k] generic_file_aio_write
|
--- generic_file_aio_write
|
|--52.11%-- do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.89%-- ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.14% tar [kernel.kallsyms] [k] block_write_end
|
--- block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.14% tar [kernel.kallsyms] [k] hrtimer_interrupt
|
--- hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
|
|--52.67%-- jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.33%-- string_to_context_struct
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.14% flush-253:2 [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
__percpu_counter_add
__add_bdi_stat
clear_page_dirty_for_io
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.14% swapper [kernel.kallsyms] [k] __rcu_read_lock
|
--- __rcu_read_lock
|
|--50.04%-- __prop_inc_percpu_max
| test_clear_page_writeback
| end_page_writeback
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--49.96%-- __atomic_notifier_call_chain
atomic_notifier_call_chain
exit_idle
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.13% tar [kernel.kallsyms] [k] ktime_get_ts
|
--- ktime_get_ts
__delayacct_blkio_end
delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.13% tar [kernel.kallsyms] [k] page_zone
|
--- page_zone
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] __read_lock_failed
|
--- __read_lock_failed
_raw_read_lock
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] prop_norm_single
|
--- prop_norm_single
__prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] put_bh
|
--- put_bh
__brelse
brelse
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] radix_tree_insert
|
--- radix_tree_insert
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] policy_zonelist
|
--- policy_zonelist
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.13% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% swapper [kernel.kallsyms] [k] irq_entries_start
|
--- irq_entries_start
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.13% top libproc-3.2.8.so [.] 0x44c2
|
--- 0x37d94044c2
0.13% flush-253:2 [kernel.kallsyms] [k] attempt_back_merge
|
--- attempt_back_merge
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.13% tar [kernel.kallsyms] [k] virt_to_head_page
|
--- virt_to_head_page
kmem_cache_free
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] mnt_clone_write
|
--- mnt_clone_write
mnt_want_write_file
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] rb_erase
|
--- rb_erase
deadline_remove_request
deadline_move_request
deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.13% tar [kernel.kallsyms] [k] __set_page_dirty
|
--- __set_page_dirty
|
|--51.11%-- __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.89%-- mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] do_sync_write
|
--- do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
__find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] arch_local_irq_disable
|
--- arch_local_irq_disable
_raw_spin_lock_irq
|
|--55.62%-- add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.38%-- __set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] kmem_cache_free
|
--- kmem_cache_free
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] ext4_da_write_begin
|
--- ext4_da_write_begin
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% swapper [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
|
|--69.63%-- account_system_vtime
| irq_exit
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--30.37%-- irqtime_account_process_tick
account_idle_ticks
tick_nohz_restart_sched_tick
cpu_idle
start_secondary
0.12% flush-253:2 [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.12% tar [kernel.kallsyms] [k] inode_reserved_space
|
--- inode_reserved_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% swapper [kernel.kallsyms] [k] update_cfs_shares
|
--- update_cfs_shares
enqueue_task_fair
enqueue_task
activate_task
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.12% tar [kernel.kallsyms] [k] schedule
|
--- schedule
|
|--59.95%-- io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--40.05%-- __GI___libc_read
0.12% tar [kernel.kallsyms] [k] security_sid_to_context_force
|
--- security_sid_to_context_force
security_inode_init_security
ext4_init_security
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.12% tar [kernel.kallsyms] [k] radix_tree_preload
|
--- radix_tree_preload
add_to_page_cache_locked
add_to_page_cache_lru
|
|--54.07%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--45.93%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.12% swapper [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
_local_bh_enable
irq_enter
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.12% flush-253:2 [kernel.kallsyms] [k] release_pages
|
--- release_pages
__pagevec_release
pagevec_release
write_cache_pages_da
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.12% flush-253:2 [kernel.kallsyms] [k] page_mkclean
|
--- page_mkclean
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.12% tar [kernel.kallsyms] [k] set_bh_page
|
--- set_bh_page
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% tar [kernel.kallsyms] [k] account_entity_dequeue
|
--- account_entity_dequeue
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.12% tar [kernel.kallsyms] [k] ata_qc_issue
|
--- ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.12% tar libc-2.13.90.so [.] __cfree
|
--- __cfree
0.12% tar [kernel.kallsyms] [k] radix_tree_lookup_slot
|
--- radix_tree_lookup_slot
|
|--62.63%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--37.37%-- find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% top libc-2.13.90.so [.] _IO_vfscanf_internal
|
--- _IO_vfscanf_internal
_IO_vsscanf
0x7fffef292b20
0.12% tar [kernel.kallsyms] [k] ext4_inode_table
|
--- ext4_inode_table
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% tar [kernel.kallsyms] [k] xattr_resolve_name
|
--- xattr_resolve_name
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% tar [kernel.kallsyms] [k] add_to_page_cache_lru
|
--- add_to_page_cache_lru
|
|--67.10%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--32.90%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.12% migration/0 [kernel.kallsyms] [k] resched_task
|
--- resched_task
check_preempt_curr
__migrate_task
migration_cpu_stop
cpu_stopper_thread
kthread
kernel_thread_helper
0.12% tar [kernel.kallsyms] [k] __count_vm_events
|
--- __count_vm_events
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--50.90%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--49.10%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% tar [kernel.kallsyms] [k] system_call_fastpath
|
--- system_call_fastpath
__GI___libc_write
0.11% flush-253:2 [kernel.kallsyms] [k] __percpu_counter_add
|
--- __percpu_counter_add
__add_bdi_stat
clear_page_dirty_for_io
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.11% swapper [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.11% kworker/0:2 [kernel.kallsyms] [k] worker_thread
|
--- worker_thread
kthread
kernel_thread_helper
0.11% tar [kernel.kallsyms] [k] fsnotify
|
--- fsnotify
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% tar [kernel.kallsyms] [k] is_handle_aborted
|
--- is_handle_aborted
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] blk_finish_plug
|
--- blk_finish_plug
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] jbd2_journal_get_write_access
|
--- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] vfs_write
|
--- vfs_write
system_call_fastpath
__GI___libc_write
0.11% swapper [kernel.kallsyms] [k] irqtime_account_process_tick
|
--- irqtime_account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.11% tar [kernel.kallsyms] [k] blk_flush_plug_list
|
--- blk_flush_plug_list
blk_finish_plug
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% tar [kernel.kallsyms] [k] deadline_remove_request
|
--- deadline_remove_request
deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% tar [kernel.kallsyms] [k] mb_find_buddy
|
--- mb_find_buddy
mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.11% tar [kernel.kallsyms] [k] sys_read
|
--- sys_read
|
|--58.45%-- system_call_fastpath
| __GI___libc_read
|
--41.55%-- __GI___libc_read
0.11% flush-253:2 [kernel.kallsyms] [k] arch_read_unlock
|
--- arch_read_unlock
_raw_read_unlock_irqrestore
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.11% swapper [kernel.kallsyms] [k] __slab_free
|
--- __slab_free
kmem_cache_free
mempool_free_slab
mempool_free
scsi_sg_free
__sg_free_table
__scsi_release_buffers
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.11% tar [kernel.kallsyms] [k] sidtab_search_core
|
--- sidtab_search_core
sidtab_search
security_compute_sid
security_transition_sid
may_create
selinux_inode_create
security_inode_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.11% swapper [kernel.kallsyms] [k] switch_mm
|
--- switch_mm
schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.11% flush-253:2 [kernel.kallsyms] [k] mpage_da_submit_io
|
--- mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.11% tar [kernel.kallsyms] [k] mem_cgroup_charge_statistics
|
--- mem_cgroup_charge_statistics
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--56.66%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.34%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% tar [kernel.kallsyms] [k] zone_watermark_ok
|
--- zone_watermark_ok
|
|--50.80%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--49.20%-- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] ext4_free_blocks
|
--- ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.11% tar [kernel.kallsyms] [k] ext4_ext_map_blocks
|
--- ext4_ext_map_blocks
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] add_dirent_to_buf
|
--- add_dirent_to_buf
ext4_add_entry
ext4_add_nondir
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.11% tar [kernel.kallsyms] [k] strcmp
|
--- strcmp
symcmp
hashtab_search
string_to_context_struct
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.11% tar [kernel.kallsyms] [k] select_idle_sibling
|
--- select_idle_sibling
select_task_rq_fair
select_task_rq
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% flush-253:2 [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
unlock_page
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.11% tar [kernel.kallsyms] [k] radix_tree_node_alloc
|
--- radix_tree_node_alloc
radix_tree_insert
add_to_page_cache_locked
add_to_page_cache_lru
|
|--60.14%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--39.86%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% swapper [kernel.kallsyms] [k] rcu_irq_exit
|
--- rcu_irq_exit
|
|--54.97%-- smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--45.03%-- irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.10% tar [kernel.kallsyms] [k] radix_tree_tag_set
|
--- radix_tree_tag_set
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% tar [kernel.kallsyms] [k] check_irqs_on
|
--- check_irqs_on
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% flush-253:2 [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
__add_bdi_stat
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% flush-253:2 [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
__might_sleep
unmap_underlying_metadata
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] irqtime_account_process_tick
|
--- irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% tar [kernel.kallsyms] [k] alloc_buffer_head
|
--- alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% tar [kernel.kallsyms] [k] jiffies_to_timeval
|
--- jiffies_to_timeval
__account_system_time
irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
__find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.10% tar [kernel.kallsyms] [k] ext4_get_reserved_space
|
--- ext4_get_reserved_space
inode_add_rsv_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% swapper [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.10% tar [kernel.kallsyms] [k] blk_start_plug
|
--- blk_start_plug
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% flush-253:2 [kernel.kallsyms] [k] sg_init_table
|
--- sg_init_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
__make_request
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] ata_qc_new_init
|
--- ata_qc_new_init
|
|--51.58%-- ata_scsi_queuecmd
| scsi_dispatch_cmd
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.42%-- __ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% tar [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_unlock_irq
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% flush-253:2 [kernel.kallsyms] [k] inc_zone_page_state
|
--- inc_zone_page_state
account_page_writeback
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] system_call
|
--- system_call
__GI___libc_write
0.10% flush-253:2 [kernel.kallsyms] [k] __bio_clone
|
--- __bio_clone
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
ext4_io_submit
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% flush-253:2 [kernel.kallsyms] [k] mod_state
|
--- mod_state
inc_zone_page_state
account_page_writeback
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
_raw_spin_lock_irqsave
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% tar [kernel.kallsyms] [k] bio_get_nr_vecs
|
--- bio_get_nr_vecs
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_set
|
--- radix_tree_tag_set
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
__mem_cgroup_try_charge.constprop.27
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% flush-253:2 [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.09% tar [kernel.kallsyms] [k] inode_init_once
|
--- inode_init_once
init_once
setup_object
new_slab
__slab_alloc
kmem_cache_alloc
ext4_alloc_inode
alloc_inode
new_inode
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.09% jbd2/dm-2-8 [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--61.31%-- _raw_spin_lock
| jbd2_journal_file_buffer
| jbd2_journal_write_metadata_buffer
| jbd2_journal_commit_transaction
| kjournald2
| kthread
| kernel_thread_helper
|
--38.69%-- jbd_trylock_bh_state
journal_clean_one_cp_list
__jbd2_journal_clean_checkpoint_list
jbd2_journal_commit_transaction
kjournald2
kthread
kernel_thread_helper
0.09% swapper [kernel.kallsyms] [k] kobject_put
|
--- kobject_put
__scsi_put_command
scsi_put_command
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% flush-253:2 [kernel.kallsyms] [k] __elv_add_request
|
--- __elv_add_request
__make_request
generic_make_request
submit_bio
ext4_io_submit
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.09% swapper [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
rcu_process_gp_end
__rcu_process_callbacks
rcu_process_callbacks
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% tar [kernel.kallsyms] [k] round_jiffies_up
|
--- round_jiffies_up
blk_start_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] attach_page_buffers
|
--- attach_page_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.09% tar [kernel.kallsyms] [k] hrtick_update
|
--- hrtick_update
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] ext4_clear_inode_flag
|
--- ext4_clear_inode_flag
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.09% jbd2/dm-2-8 [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock_irq
blk_throtl_bio
generic_make_request
submit_bio
submit_bh
jbd2_journal_commit_transaction
kjournald2
kthread
kernel_thread_helper
0.09% top [kernel.kallsyms] [k] __rcu_read_lock
|
--- __rcu_read_lock
__d_lookup
walk_component
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.09% tar [kernel.kallsyms] [k] __brelse
|
--- __brelse
brelse
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.09% flush-253:2 [kernel.kallsyms] [k] page_zone
|
--- page_zone
inc_zone_page_state
account_page_writeback
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.09% tar [kernel.kallsyms] [k] bio_integrity_enabled
|
--- bio_integrity_enabled
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] dm_table_find_target
|
--- dm_table_find_target
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] __mod_zone_page_state
|
--- __mod_zone_page_state
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] ahci_fill_cmd_slot
|
--- ahci_fill_cmd_slot
ahci_qc_prep
ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% swapper [kernel.kallsyms] [k] blk_unprep_request
|
--- blk_unprep_request
blk_finish_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% top libc-2.13.90.so [.] __GI___close
|
--- __GI___close
0.09% tar [kernel.kallsyms] [k] put_bh
|
--- put_bh
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.09% swapper [kernel.kallsyms] [k] finish_task_switch
|
--- finish_task_switch
schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% flush-253:2 [kernel.kallsyms] [k] page_mapping
|
--- page_mapping
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.09% tar [kernel.kallsyms] [k] alloc_tio
|
--- alloc_tio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_clear
|
--- radix_tree_tag_clear
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.08% tar [kernel.kallsyms] [k] timespec_trunc
|
--- timespec_trunc
touch_atime
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% tar [kernel.kallsyms] [k] ext4_claim_free_blocks
|
--- ext4_claim_free_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% tar libc-2.13.90.so [.] __GI___errno_location
|
--- __GI___errno_location
0.08% tar [kernel.kallsyms] [k] path_to_nameidata
|
--- path_to_nameidata
walk_component
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.08% tar [kernel.kallsyms] [k] current_sid
|
--- current_sid
selinux_file_permission
security_file_permission
rw_verify_area
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% top [kernel.kallsyms] [k] path_openat
|
--- path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.08% tar [kernel.kallsyms] [k] __do_page_cache_readahead
|
--- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% tar [kernel.kallsyms] [k] _cond_resched
|
--- _cond_resched
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% swapper [kernel.kallsyms] [k] run_timer_softirq
|
--- run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.08% swapper [kernel.kallsyms] [k] note_interrupt
|
--- note_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.08% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% swapper [kernel.kallsyms] [k] native_apic_mem_write
|
--- native_apic_mem_write
apic_write
ack_APIC_irq
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.08% tar [kernel.kallsyms] [k] bio_data
|
--- bio_data
init_request_from_bio
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% tar [kernel.kallsyms] [k] string_to_context_struct
|
--- string_to_context_struct
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.08% tar [kernel.kallsyms] [k] mb_clear_bit
|
--- mb_clear_bit
mb_clear_bits
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.08% tar [kernel.kallsyms] [k] ext4_mb_load_buddy
|
--- ext4_mb_load_buddy
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.08% top [kernel.kallsyms] [k] __fsnotify_parent
|
--- __fsnotify_parent
fsnotify_perm
security_file_permission
rw_verify_area
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% sleep [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock
|
|--60.83%-- __do_fault
| handle_pte_fault
| handle_mm_fault
| do_page_fault
| page_fault
| 0x37d9082b80
|
--39.17%-- deny_write_access
mmap_region
do_mmap_pgoff
do_mmap
elf_map
load_elf_binary
search_binary_handler
do_execve
sys_execve
stub_execve
0x37d90ae097
0.08% swapper [kernel.kallsyms] [k] ahci_interrupt
|
--- ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.08% tar [kernel.kallsyms] [k] __srcu_read_unlock
|
--- __srcu_read_unlock
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% tar [kernel.kallsyms] [k] signal_pending
|
--- signal_pending
fatal_signal_pending
__mem_cgroup_try_charge.constprop.27
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% tar [kernel.kallsyms] [k] __read_seqcount_begin
|
--- __read_seqcount_begin
path_init
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.08% tar [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
mnt_clone_write
mnt_want_write_file
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% tar [kernel.kallsyms] [k] jbd2_journal_start
|
--- jbd2_journal_start
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] __srcu_read_lock
|
--- __srcu_read_lock
fsnotify
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% swapper [kernel.kallsyms] [k] handle_irq
|
--- handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% swapper [kernel.kallsyms] [k] __ata_scsi_queuecmd
|
--- __ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
drive_stat_acct
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] __switch_to
|
--- __switch_to
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] ext4_file_write
|
--- ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% swapper [kernel.kallsyms] [k] wake_bit_function
|
--- wake_bit_function
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] _raw_spin_lock_irq
|
--- _raw_spin_lock_irq
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] drive_stat_acct
|
--- drive_stat_acct
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.22
|
--- test_ti_thread_flag.constprop.22
zero_user_segments
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] sd_prep_fn
|
--- sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% top libc-2.13.90.so [.] __strchrnul
|
--- __strchrnul
___vsnprintf_chk
0x39335b1b6d5b1b42
0.07% swapper [kernel.kallsyms] [k] sd_prep_fn
|
--- sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] try_to_wake_up
|
--- try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_unlock_irqrestore
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] __bio_clone
|
--- __bio_clone
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] __zone_watermark_ok
|
--- __zone_watermark_ok
zone_watermark_ok
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% swapper [kernel.kallsyms] [k] __remove_hrtimer
|
--- __remove_hrtimer
remove_hrtimer.part.4
hrtimer_try_to_cancel
hrtimer_cancel
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% swapper [kernel.kallsyms] [k] scsi_device_unbusy
|
--- scsi_device_unbusy
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] scsi_prep_return
|
--- scsi_prep_return
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] prepare_to_wait_exclusive
|
--- prepare_to_wait_exclusive
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] set_buffer_uptodate
|
--- set_buffer_uptodate
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] ata_get_xlat_func
|
--- ata_get_xlat_func
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] __find_get_block_slow
|
--- __find_get_block_slow
__find_get_block
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.07% tar [kernel.kallsyms] [k] generic_getxattr
|
--- generic_getxattr
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% swapper [kernel.kallsyms] [k] bit_spin_lock.constprop.22
|
--- bit_spin_lock.constprop.22
__slab_free
kmem_cache_free
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% swapper [kernel.kallsyms] [k] is_swiotlb_buffer
|
--- is_swiotlb_buffer
unmap_single
swiotlb_unmap_sg_attrs
ata_sg_clean
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] native_read_tsc
|
--- native_read_tsc
paravirt_read_tsc
native_sched_clock
sched_clock
blk_rq_init
get_request
get_request_wait
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] ext4_has_free_blocks
|
--- ext4_has_free_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar libc-2.13.90.so [.] __GI___fxstat64
|
--- __GI___fxstat64
0.07% swapper [kernel.kallsyms] [k] perf_pmu_disable
|
--- perf_pmu_disable
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] sidtab_context_to_sid
|
--- sidtab_context_to_sid
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.07% tar [kernel.kallsyms] [k] scsi_pool_alloc_command
|
--- scsi_pool_alloc_command
scsi_host_alloc_command
__scsi_get_command
scsi_get_command
scsi_get_cmd_from_req
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] list_del
|
--- list_del
activate_page
mark_page_accessed
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] pagefault_enable
|
--- pagefault_enable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] security_file_permission
|
--- security_file_permission
rw_verify_area
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% swapper [kernel.kallsyms] [k] scsi_next_command
|
--- scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] __phys_addr
|
--- __phys_addr
virt_to_head_page
kmem_cache_free
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] page_cache_async_readahead
|
--- page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] truncate_inode_pages_range
|
--- truncate_inode_pages_range
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.07% tar [kernel.kallsyms] [k] mem_cgroup_del_lru_list
|
--- mem_cgroup_del_lru_list
activate_page
mark_page_accessed
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] ext4_xattr_check_names
|
--- ext4_xattr_check_names
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] arch_local_irq_enable
|
--- arch_local_irq_enable
__find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] get_request
|
--- get_request
get_request_wait
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] swiotlb_map_sg_attrs
|
--- swiotlb_map_sg_attrs
ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] dm_get_live_table
|
--- dm_get_live_table
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] mem_cgroup_get_reclaim_stat_from_page
|
--- mem_cgroup_get_reclaim_stat_from_page
update_page_reclaim_stat
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] check_preempt_curr
|
--- check_preempt_curr
ttwu_post_activation
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] atomic_long_add
|
--- atomic_long_add
zone_page_state_add
__mod_zone_page_state
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] pm_qos_request
|
--- pm_qos_request
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] __kmalloc
|
--- __kmalloc
kzalloc.constprop.9
ext4_ext_find_extent
ext4_ext_map_blocks
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] rcu_bh_qs
|
--- rcu_bh_qs
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% swapper [kernel.kallsyms] [k] __blk_complete_request
|
--- __blk_complete_request
scsi_done
ata_scsi_qc_complete
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] __d_lookup
|
--- __d_lookup
walk_component
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.06% tar [kernel.kallsyms] [k] context_struct_to_string
|
--- context_struct_to_string
security_sid_to_context_core
security_sid_to_context_force
selinux_inode_init_security
security_inode_init_security
ext4_init_security
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.06% swapper [kernel.kallsyms] [k] tick_nohz_restart_sched_tick
|
--- tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] max_sane_readahead
|
--- max_sane_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] rw_verify_area
|
--- rw_verify_area
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] ahci_qc_prep
|
--- ahci_qc_prep
ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] linear_map
|
--- linear_map
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% swapper [kernel.kallsyms] [k] sched_clock
|
--- sched_clock
account_system_vtime
irq_enter
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] fput_light
|
--- fput_light
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] iov_iter_advance
|
--- iov_iter_advance
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] __queue_work
|
--- __queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
lock_page_cgroup
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% swapper [kernel.kallsyms] [k] scsi_io_completion
|
--- scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] activate_page
|
--- activate_page
mark_page_accessed
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] touch_atime
|
--- touch_atime
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] atomic_sub
|
--- atomic_sub
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] __inc_zone_page_state
|
--- __inc_zone_page_state
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] ext4_test_inode_state
|
--- ext4_test_inode_state
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] ext4_xattr_find_entry
|
--- ext4_xattr_find_entry
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] deadline_dispatch_requests
|
--- deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% swapper [kernel.kallsyms] [k] hrtick_update
|
--- hrtick_update
enqueue_task_fair
enqueue_task
activate_task
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% swapper [kernel.kallsyms] [k] try_to_wake_up
|
--- try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% swapper [kernel.kallsyms] [k] pick_next_task_fair
|
--- pick_next_task_fair
schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] pick_next_task_idle
|
--- pick_next_task_idle
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% swapper [kernel.kallsyms] [k] ktime_get
|
--- ktime_get
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] balance_dirty_pages_ratelimited_nr
|
--- balance_dirty_pages_ratelimited_nr
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] mpage_end_io
|
--- mpage_end_io
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] radix_tree_preload_end
|
--- radix_tree_preload_end
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] submit_bio
|
--- submit_bio
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] __ata_scsi_queuecmd
|
--- __ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] dm_table_put
|
--- dm_table_put
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% sleep [kernel.kallsyms] [k] __ext4_journal_get_write_access
|
--- __ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
touch_atime
ext4_file_mmap
mmap_region
do_mmap_pgoff
sys_mmap_pgoff
sys_mmap
system_call_fastpath
__mmap
_nl_find_locale
__GI_setlocale
0x40124c
0.05% tar [kernel.kallsyms] [k] sysret_check
|
--- sysret_check
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] clockevents_program_event
|
--- clockevents_program_event
tick_dev_program_event
tick_program_event
__hrtimer_start_range_ns
hrtimer_start_range_ns
hrtimer_start_expires.constprop.1
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] pick_next_task_rt
|
--- pick_next_task_rt
pick_next_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] account_entity_enqueue
|
--- account_entity_enqueue
enqueue_task
activate_task
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] need_resched
|
--- need_resched
_cond_resched
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] find_lock_page
|
--- find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] memcmp
|
--- memcmp
search_dirblock
ext4_dx_find_entry
ext4_find_entry
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.05% tar [kernel.kallsyms] [k] generic_segment_checks
|
--- generic_segment_checks
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% tar [kernel.kallsyms] [k] update_vsyscall
|
--- update_vsyscall
do_timer
tick_do_update_jiffies64
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_lock
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% swapper [kernel.kallsyms] [k] hrtimer_start_expires.constprop.1
|
--- hrtimer_start_expires.constprop.1
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] __page_cache_alloc
|
--- __page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] avc_has_perm_noaudit
|
--- avc_has_perm_noaudit
avc_has_perm
inode_has_perm
selinux_inode_permission
security_inode_exec_permission
exec_permission
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.05% tar [kernel.kallsyms] [k] deactivate_slab
|
--- deactivate_slab
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% swapper [kernel.kallsyms] [k] bio_free
|
--- bio_free
bio_fs_destructor
bio_put
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar libc-2.13.90.so [.] __GI___libc_read
|
--- __GI___libc_read
0.05% tar [kernel.kallsyms] [k] selinux_capable
|
--- selinux_capable
ns_capable
capable
inode_change_ok
ext4_setattr
notify_change
chown_common
sys_fchown
system_call_fastpath
__GI___fchown
0.05% tar [kernel.kallsyms] [k] _raw_read_lock_irqsave
|
--- _raw_read_lock_irqsave
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] scsi_handle_queue_ramp_up
|
--- scsi_handle_queue_ramp_up
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] dm_table_any_congested
|
--- dm_table_any_congested
dm_any_congested
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% tar [kernel.kallsyms] [k] __wake_up_common
|
--- __wake_up_common
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] acct_update_integrals
|
--- acct_update_integrals
__account_system_time
irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
blk_finish_plug
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] ns_to_timespec
|
--- ns_to_timespec
intel_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] _raw_read_unlock_irqrestore
|
--- _raw_read_unlock_irqrestore
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% tar [kernel.kallsyms] [k] __add_bdi_stat
|
--- __add_bdi_stat
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] sg_init_table
|
--- sg_init_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] getnstimeofday
|
--- getnstimeofday
ktime_get_real
intel_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.04% tar [kernel.kallsyms] [k] get_page
|
--- get_page
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] timekeeping_get_ns
|
--- timekeeping_get_ns
delayacct_end
__delayacct_blkio_end
delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] trylock_page
|
--- trylock_page
lock_page
find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% swapper [kernel.kallsyms] [k] native_sched_clock
|
--- native_sched_clock
sched_clock_cpu
account_system_vtime
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.04% tar [kernel.kallsyms] [k] native_load_tls
|
--- native_load_tls
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] files_lglock_local_unlock_cpu
|
--- files_lglock_local_unlock_cpu
file_sb_list_del
fput
filp_close
sys_close
system_call_fastpath
__GI___close
0.04% tar [kernel.kallsyms] [k] release_pages
|
--- release_pages
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% jbd2/dm-2-8 [kernel.kallsyms] [k] bit_spin_lock.constprop.22
|
--- bit_spin_lock.constprop.22
__slab_free
kmem_cache_free
__journal_remove_journal_head
jbd2_journal_remove_journal_head
journal_clean_one_cp_list
__jbd2_journal_clean_checkpoint_list
jbd2_journal_commit_transaction
kjournald2
kthread
kernel_thread_helper
0.04% tar [kernel.kallsyms] [k] pick_next_task_fair
|
--- pick_next_task_fair
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% swapper [kernel.kallsyms] [k] ktime_get_real
|
--- ktime_get_real
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.04% tar [kernel.kallsyms] [k] tick_do_update_jiffies64
|
--- tick_do_update_jiffies64
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% swapper [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
arch_local_irq_save
irqtime_account_process_tick
account_idle_ticks
tick_nohz_restart_sched_tick
cpu_idle
start_secondary
0.03% sleep [kernel.kallsyms] [k] mem_cgroup_uncharge_start
|
--- mem_cgroup_uncharge_start
exit_mmap
mmput
flush_old_exec
load_elf_binary
search_binary_handler
do_execve
sys_execve
stub_execve
__execve
0.02% swapper [kernel.kallsyms] [k] irq_enter
|
--- irq_enter
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
#
# (For a higher level overview, try: perf report --sort comm,dso)
#
On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote:
> This is the output of perf record -g -a -f sleep 5
>
> (hopefully the list won't choke)
Um, this one actually shows kswapd
James
---
# Events: 6K cycles
#
# Overhead Command Shared Object Symbol
# ........ ........... ................... .......................................
#
20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
|
--- shrink_slab
|
|--99.91%-- kswapd
| kthread
| kernel_thread_helper
--0.09%-- [...]
9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone
|
--- shrink_zone
|
|--99.46%-- kswapd
| kthread
| kernel_thread_helper
|
--0.54%-- kthread
kernel_thread_helper
7.70% kswapd0 [kernel.kallsyms] [k] kswapd
|
--- kswapd
kthread
kernel_thread_helper
5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe
|
--- zone_watermark_ok_safe
|
|--72.66%-- kswapd
| kthread
| kernel_thread_helper
|
|--20.88%-- sleeping_prematurely.part.12
| kswapd
| kthread
| kernel_thread_helper
|
--6.46%-- kthread
kernel_thread_helper
4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
|
|--77.49%-- _raw_spin_lock
| |
| |--51.85%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --48.15%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--19.47%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--1.73%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--0.88%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
--0.44%-- [...]
3.34% kswapd0 [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--48.58%-- _raw_spin_unlock
| |
| |--80.77%-- mem_cgroup_soft_limit_reclaim
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --19.23%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--17.63%-- _raw_spin_unlock_irqrestore
| |
| |--52.96%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --47.04%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--13.83%-- _raw_spin_unlock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--6.67%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--5.54%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--2.76%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--2.24%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--1.68%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--0.55%-- irq_exit
| smp_apic_timer_interrupt
| apic_timer_interrupt
| mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
--0.54%-- mem_cgroup_charge_statistics
__mem_cgroup_uncharge_common
mem_cgroup_uncharge_cache_page
__remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
2.96% kswapd0 [kernel.kallsyms] [k] __zone_watermark_ok
|
--- __zone_watermark_ok
|
|--85.77%-- zone_watermark_ok_safe
| |
| |--74.67%-- kswapd
| | kthread
| | kernel_thread_helper
| |
| --25.33%-- sleeping_prematurely.part.12
| kswapd
| kthread
| kernel_thread_helper
|
|--11.78%-- kswapd
| kthread
| kernel_thread_helper
|
--2.45%-- sleeping_prematurely.part.12
kswapd
kthread
kernel_thread_helper
2.67% kswapd0 [kernel.kallsyms] [k] global_dirty_limits
|
--- global_dirty_limits
|
|--97.28%-- throttle_vm_writeout
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--2.72%-- shrink_zone
kswapd
kthread
kernel_thread_helper
2.30% kswapd0 [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--30.33%-- _raw_spin_lock
| |
| |--52.29%-- mem_cgroup_soft_limit_reclaim
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --47.71%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--28.79%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--24.01%-- _raw_spin_lock_irqsave
| |
| |--63.28%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --36.72%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--6.31%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--4.82%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--3.28%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--1.64%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--0.82%-- finish_wait
kswapd
kthread
kernel_thread_helper
2.02% kswapd0 [kernel.kallsyms] [k] up_read
|
--- up_read
|
|--95.48%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--4.52%-- kswapd
kthread
kernel_thread_helper
2.01% kswapd0 [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
|
|--37.59%-- sub_preempt_count
| |
| |--66.19%-- _raw_spin_unlock
| | |
| | |--55.77%-- mem_cgroup_soft_limit_reclaim
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --44.23%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| |--17.13%-- _raw_spin_unlock_irqrestore
| | |
| | |--57.38%-- finish_wait
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --42.62%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --16.68%-- _raw_spin_unlock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--32.16%-- add_preempt_count
| |
| |--71.32%-- _raw_spin_lock
| | |
| | |--56.02%-- mem_cgroup_soft_limit_reclaim
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --43.98%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| |--17.25%-- _raw_spin_lock_irqsave
| | |
| | |--83.09%-- finish_wait
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --16.91%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --11.43%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--8.20%-- _raw_spin_unlock
| |
| |--66.09%-- mem_cgroup_soft_limit_reclaim
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --33.91%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--7.42%-- _raw_spin_lock
| |
| |--62.14%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --37.86%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--5.55%-- _raw_spin_lock_irqsave
| |
| |--66.67%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --33.33%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--4.53%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--3.62%-- _raw_spin_unlock_irqrestore
| |
| |--51.20%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --48.80%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--0.94%-- _raw_spin_unlock_irq
shrink_zone
kswapd
kthread
kernel_thread_helper
1.88% kswapd0 [kernel.kallsyms] [k] zone_nr_lru_pages
|
--- zone_nr_lru_pages
|
|--85.21%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--14.79%-- kswapd
kthread
kernel_thread_helper
1.79% kswapd0 [kernel.kallsyms] [k] down_read_trylock
|
--- down_read_trylock
|
|--92.80%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--7.20%-- kswapd
kthread
kernel_thread_helper
1.69% kswapd0 [kernel.kallsyms] [k] mutex_trylock
|
--- mutex_trylock
i915_gem_inactive_shrink
shrink_slab
kswapd
kthread
kernel_thread_helper
1.66% kswapd0 [kernel.kallsyms] [k] sleeping_prematurely.part.12
|
--- sleeping_prematurely.part.12
|
|--93.41%-- kswapd
| kthread
| kernel_thread_helper
|
--6.59%-- kthread
kernel_thread_helper
1.54% kswapd0 [kernel.kallsyms] [k] find_next_bit
|
--- find_next_bit
|
|--84.45%-- cpumask_next
| zone_watermark_ok_safe
| kswapd
| kthread
| kernel_thread_helper
|
--15.55%-- zone_watermark_ok_safe
kswapd
kthread
kernel_thread_helper
1.51% kswapd0 [kernel.kallsyms] [k] throttle_vm_writeout
|
--- throttle_vm_writeout
shrink_zone
kswapd
kthread
kernel_thread_helper
1.48% kswapd0 [kernel.kallsyms] [k] mb_cache_shrink_fn
|
--- mb_cache_shrink_fn
|
|--96.31%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--3.69%-- kswapd
kthread
kernel_thread_helper
1.31% kswapd0 [kernel.kallsyms] [k] mem_cgroup_soft_limit_reclaim
|
--- mem_cgroup_soft_limit_reclaim
|
|--97.19%-- kswapd
| kthread
| kernel_thread_helper
|
--2.81%-- kthread
kernel_thread_helper
1.26% kswapd0 [kernel.kallsyms] [k] mutex_unlock
|
--- mutex_unlock
|
|--98.57%-- i915_gem_inactive_shrink
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--1.43%-- shrink_slab
kswapd
kthread
kernel_thread_helper
1.19% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--47.70%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--47.65%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--4.65%-- kswapd
kthread
kernel_thread_helper
1.14% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock
|
--- _raw_spin_unlock
|
|--59.71%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--38.63%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--1.66%-- kswapd
kthread
kernel_thread_helper
0.96% kswapd0 [kernel.kallsyms] [k] __mem_cgroup_largest_soft_limit_node
|
--- __mem_cgroup_largest_soft_limit_node
|
|--59.47%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
--40.53%-- kswapd
kthread
kernel_thread_helper
0.88% kswapd0 [kernel.kallsyms] [k] in_lock_functions
|
--- in_lock_functions
|
|--43.80%-- get_parent_ip
| |
| |--52.47%-- add_preempt_count
| | |
| | |--72.99%-- _raw_spin_lock
| | | |
| | | |--62.71%-- mem_cgroup_soft_limit_reclaim
| | | | kswapd
| | | | kthread
| | | | kernel_thread_helper
| | | |
| | | --37.29%-- mb_cache_shrink_fn
| | | shrink_slab
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | |--18.30%-- _raw_spin_lock_irq
| | | shrink_zone
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --8.71%-- _raw_spin_lock_irqsave
| | prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --47.53%-- sub_preempt_count
| |
| |--49.81%-- _raw_spin_unlock
| | mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| |--30.32%-- _raw_spin_unlock_irqrestore
| | finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --19.86%-- _raw_spin_unlock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--29.12%-- add_preempt_count
| |
| |--49.50%-- _raw_spin_lock
| | |
| | |--56.31%-- mem_cgroup_soft_limit_reclaim
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --43.69%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| |--29.19%-- _raw_spin_lock_irq
| | shrink_zone
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --21.32%-- _raw_spin_lock_irqsave
| |
| |--65.66%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --34.34%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--27.07%-- sub_preempt_count
|
|--62.26%-- _raw_spin_unlock
| |
| |--50.27%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --49.73%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--22.85%-- _raw_spin_unlock_irqrestore
| |
| |--67.34%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --32.66%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--14.89%-- _raw_spin_unlock_irq
shrink_zone
kswapd
kthread
kernel_thread_helper
0.81% kswapd0 [kernel.kallsyms] [k] cpumask_next
|
--- cpumask_next
|
|--79.81%-- zone_watermark_ok_safe
| kswapd
| kthread
| kernel_thread_helper
|
--20.19%-- kswapd
kthread
kernel_thread_helper
0.79% kswapd0 [kernel.kallsyms] [k] zone_reclaimable_pages
|
--- zone_reclaimable_pages
|
|--93.08%-- kswapd
| kthread
| kernel_thread_helper
|
--6.92%-- kthread
kernel_thread_helper
0.77% kswapd0 [i915] [k] i915_gem_inactive_shrink
|
--- i915_gem_inactive_shrink
|
|--97.55%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--2.45%-- kswapd
kthread
kernel_thread_helper
0.76% kswapd0 [kernel.kallsyms] [k] prepare_to_wait
|
--- prepare_to_wait
|
|--97.54%-- kswapd
| kthread
| kernel_thread_helper
|
--2.46%-- kthread
kernel_thread_helper
0.74% kswapd0 [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
|
|--79.88%-- _raw_spin_unlock_irqrestore
| |
| |--62.49%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --37.51%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--15.16%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--4.96%-- finish_wait
kswapd
kthread
kernel_thread_helper
0.65% kswapd0 [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6
|
--- test_ti_thread_flag.constprop.6
|
|--20.12%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--20.09%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--14.28%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--14.16%-- _raw_spin_unlock
| |
| |--80.68%-- mem_cgroup_soft_limit_reclaim
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --19.32%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--11.40%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--8.63%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--8.58%-- _raw_spin_unlock_irqrestore
| |
| |--66.13%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --33.87%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
--2.74%-- _raw_spin_unlock_irq
shrink_zone
kswapd
kthread
kernel_thread_helper
0.63% tar [kernel.kallsyms] [k] copy_user_generic_string
|
--- copy_user_generic_string
|
|--56.11%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--43.89%-- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.59% kswapd0 [kernel.kallsyms] [k] test_tsk_thread_flag
|
--- test_tsk_thread_flag
|
|--50.16%-- kswapd
| kthread
| kernel_thread_helper
|
|--46.73%-- try_to_freeze
| kswapd
| kthread
| kernel_thread_helper
|
--3.11%-- kthread
kernel_thread_helper
0.59% kswapd0 [kernel.kallsyms] [k] shrink_icache_memory
|
--- shrink_icache_memory
|
|--81.19%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--18.81%-- kswapd
kthread
kernel_thread_helper
0.52% kswapd0 [kernel.kallsyms] [k] shrink_dqcache_memory
|
--- shrink_dqcache_memory
|
|--89.35%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--10.65%-- kswapd
kthread
kernel_thread_helper
0.50% kswapd0 [kernel.kallsyms] [k] zone_clear_flag
|
--- zone_clear_flag
|
|--77.78%-- kswapd
| kthread
| kernel_thread_helper
|
--22.22%-- kthread
kernel_thread_helper
0.45% kswapd0 [kernel.kallsyms] [k] kthread_should_stop
|
--- kthread_should_stop
|
|--54.28%-- kswapd
| kthread
| kernel_thread_helper
|
--45.72%-- kthread
kernel_thread_helper
0.44% kswapd0 [kernel.kallsyms] [k] global_page_state
|
--- global_page_state
|
|--33.60%-- throttle_vm_writeout
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--29.47%-- determine_dirtyable_memory
| global_dirty_limits
| throttle_vm_writeout
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--20.52%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--16.41%-- global_dirty_limits
throttle_vm_writeout
shrink_zone
kswapd
kthread
kernel_thread_helper
0.44% kswapd0 [kernel.kallsyms] [k] shrink_dcache_memory
|
--- shrink_dcache_memory
shrink_slab
kswapd
kthread
kernel_thread_helper
0.41% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore
|
--- _raw_spin_unlock_irqrestore
|
|--59.01%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--40.99%-- finish_wait
kswapd
kthread
kernel_thread_helper
0.39% kswapd0 [kernel.kallsyms] [k] global_reclaimable_pages
|
--- global_reclaimable_pages
determine_dirtyable_memory
global_dirty_limits
throttle_vm_writeout
shrink_zone
kswapd
kthread
kernel_thread_helper
0.37% kswapd0 [kernel.kallsyms] [k] finish_wait
|
--- finish_wait
|
|--95.03%-- kswapd
| kthread
| kernel_thread_helper
|
--4.97%-- kthread
kernel_thread_helper
0.33% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
|
|--60.70%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
--39.30%-- mb_cache_shrink_fn
shrink_slab
kswapd
kthread
kernel_thread_helper
0.31% kswapd0 [kernel.kallsyms] [k] arch_local_irq_enable
|
--- arch_local_irq_enable
|
|--76.94%-- _raw_spin_unlock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--23.06%-- shrink_zone
kswapd
kthread
kernel_thread_helper
0.28% kswapd0 [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
|
|--93.18%-- _raw_spin_lock_irqsave
| |
| |--64.21%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --35.79%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
--6.82%-- finish_wait
kswapd
kthread
kernel_thread_helper
0.22% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock_irq
|
--- _raw_spin_unlock_irq
|
|--92.00%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--8.00%-- __remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.22% kswapd0 [kernel.kallsyms] [k] arch_local_irq_disable
|
--- arch_local_irq_disable
|
|--58.81%-- arch_local_irq_save
| _raw_spin_lock_irqsave
| |
| |--70.97%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --29.03%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--33.06%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--8.13%-- shrink_zone
kswapd
kthread
kernel_thread_helper
0.22% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock_irq
|
--- _raw_spin_lock_irq
|
|--58.62%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--33.19%-- kswapd
| kthread
| kernel_thread_helper
|
--8.19%-- __remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.22% swapper [kernel.kallsyms] [k] poll_idle
|
--- poll_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.20% kswapd0 [kernel.kallsyms] [k] __list_add
|
--- __list_add
|
|--90.97%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--9.03%-- free_hot_cold_page
__pagevec_free
free_page_list
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.15% tar [kernel.kallsyms] [k] zero_user_segments
|
--- zero_user_segments
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% kswapd0 [kernel.kallsyms] [k] rb_last
|
--- rb_last
mem_cgroup_soft_limit_reclaim
kswapd
kthread
kernel_thread_helper
0.14% swapper [kernel.kallsyms] [k] intel_idle
|
--- intel_idle
cpuidle_idle_call
cpu_idle
|
|--65.05%-- rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--34.95%-- start_secondary
0.13% kswapd0 [kernel.kallsyms] [k] get_reclaim_stat
|
--- get_reclaim_stat
|
|--57.55%-- kswapd
| kthread
| kernel_thread_helper
|
--42.45%-- shrink_zone
kswapd
kthread
kernel_thread_helper
0.12% perf [kernel.kallsyms] [k] selinux_file_alloc_security
|
--- selinux_file_alloc_security
security_file_alloc
get_empty_filp
path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0x429beb
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.12% tar [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--18.90%-- _raw_spin_lock
| |
| |--63.01%-- ext4_da_get_block_prep
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--20.46%-- get_page_from_freelist
| | __alloc_pages_nodemask
| | alloc_pages_current
| | __page_cache_alloc
| | grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --16.53%-- ext4_ext_map_blocks
| ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--13.68%-- __lru_cache_add
| add_to_page_cache_lru
| |
| |--51.23%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --48.77%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--13.32%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--12.87%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--9.79%-- _raw_spin_lock_irq
| |
| |--62.37%-- add_to_page_cache_locked
| | add_to_page_cache_lru
| | grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --37.63%-- __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--7.52%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.32%-- lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.69%-- __percpu_counter_add
| __add_bdi_stat
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.54%-- bit_spin_lock
| lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.41%-- create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.32%-- _raw_read_lock_irqsave
| dm_get_live_table
| __split_and_process_bio
| dm_request
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.29%-- _raw_read_lock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--2.33%-- dput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.11% kswapd0 [kernel.kallsyms] [k] try_to_freeze
|
--- try_to_freeze
|
|--83.76%-- kswapd
| kthread
| kernel_thread_helper
|
--16.24%-- kthread
kernel_thread_helper
0.10% swapper [kernel.kallsyms] [k] ahci_interrupt
|
--- ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% kswapd0 [kernel.kallsyms] [k] determine_dirtyable_memory
|
--- determine_dirtyable_memory
|
|--80.46%-- global_dirty_limits
| throttle_vm_writeout
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--19.54%-- throttle_vm_writeout
shrink_zone
kswapd
kthread
kernel_thread_helper
0.08% tar [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--27.37%-- _raw_spin_unlock
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--12.90%-- blk_dequeue_request
| blk_start_request
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--9.76%-- bit_spin_unlock
| unlock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--6.85%-- _raw_spin_unlock_irq
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--6.81%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--5.86%-- _raw_spin_unlock_irqrestore
| pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--5.51%-- _raw_read_unlock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--5.37%-- jbd_unlock_bh_state
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_free_inode
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--4.68%-- deactivate_slab
| __slab_alloc
| kmem_cache_alloc
| alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.15%-- __srcu_read_unlock
| fsnotify
| fsnotify_modify
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.72%-- radix_tree_preload_end
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.62%-- bit_spin_unlock
| jbd2_journal_add_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--3.41%-- pagefault_enable
iov_iter_copy_from_user_atomic
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% swapper [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--52.45%-- _raw_spin_lock
| get_next_timer_interrupt
| tick_nohz_stop_sched_tick
| irq_exit
| smp_call_function_single_interrupt
| call_function_single_interrupt
| cpuidle_idle_call
| cpu_idle
| start_secondary
|
|--26.09%-- bit_spin_lock.constprop.22
| __slab_free
| kmem_cache_free
| ext4_free_io_end
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--21.46%-- __percpu_counter_add
__prop_inc_percpu_max
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.08% tar [kernel.kallsyms] [k] kmem_cache_alloc
|
--- kmem_cache_alloc
|
|--74.26%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--12.98%-- jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--6.88%-- mempool_alloc_slab
| mempool_alloc
| scsi_sg_alloc
| __sg_alloc_table
| scsi_alloc_sgtable
| scsi_init_sgtable
| scsi_init_io
| scsi_setup_fs_cmnd
| sd_prep_fn
| blk_peek_request
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--5.87%-- selinux_inode_alloc_security
security_inode_alloc
inode_init_always
alloc_inode
new_inode
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.07% kswapd0 [kernel.kallsyms] [k] __free_one_page
|
--- __free_one_page
free_pcppages_bulk
free_hot_cold_page
__pagevec_free
free_page_list
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.07% swapper [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--56.43%-- __percpu_counter_add
| __add_bdi_stat
| test_clear_page_writeback
| end_page_writeback
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--17.99%-- _raw_spin_unlock_irqrestore
| __hrtimer_start_range_ns
| hrtimer_start_range_ns
| hrtimer_start_expires.constprop.1
| tick_nohz_restart_sched_tick
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--15.61%-- _raw_spin_unlock
| |
| |--55.21%-- handle_irq_event
| | handle_edge_irq
| | handle_irq
| | do_IRQ
| | common_interrupt
| | cpuidle_idle_call
| | cpu_idle
| | rest_init
| | start_kernel
| | x86_64_start_reservations
| | x86_64_start_kernel
| |
| --44.79%-- scheduler_tick
| update_process_times
| tick_sched_timer
| __run_hrtimer
| hrtimer_interrupt
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--5.91%-- try_to_wake_up
| default_wake_function
| autoremove_wake_function
| wake_bit_function
| __wake_up_common
| __wake_up
| __wake_up_bit
| unlock_page
| mpage_end_io
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--4.06%-- default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] mark_page_accessed
|
--- mark_page_accessed
|
|--87.24%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--6.42%-- __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--6.35%-- __find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] get_page_from_freelist
|
--- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--54.79%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--45.21%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] __ext4_get_inode_loc
|
--- __ext4_get_inode_loc
ext4_get_inode_loc
|
|--55.97%-- ext4_reserve_inode_write
| ext4_mark_inode_dirty
| |
| |--76.52%-- ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --23.48%-- ext4_ext_dirty
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--44.03%-- ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar tar [.] 0x2aa04
|
--- 0x40753f
|
--- 0x4055c7
|
--- 0x42aa04
|
--- 0x4166d8
0x417081
|
--- 0x40fdfb
|
--- 0x405f26
|
--- 0x4332d6
|
--- 0x40fe1a
|
--- 0x408ab1
|
--- 0x40fdb8
|
--- 0x408a87
|
--- 0x4100c5
|
--- 0x407564
0.06% perf [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4293b8
0x429c0a
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.05% tar [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
|
|--79.42%-- _raw_spin_lock_irq
| |
| |--57.06%-- __set_page_dirty
| | mark_buffer_dirty
| | __block_commit_write
| | block_write_end
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --42.94%-- add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--52.51%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --47.49%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--15.99%-- _raw_spin_lock
| |
| |--60.63%-- ext4_ext_map_blocks
| | ext4_map_blocks
| | ext4_da_get_block_prep
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --39.37%-- inode_add_rsv_space
| __dquot_alloc_space
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--4.59%-- inode_add_rsv_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge.constprop.28
|
--- __mem_cgroup_commit_charge.constprop.28
|
|--91.58%-- __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--51.67%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --48.33%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--8.42%-- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] radix_tree_lookup_element
|
--- radix_tree_lookup_element
|
|--69.74%-- radix_tree_lookup_slot
| find_get_page
| |
| |--58.70%-- generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --41.30%-- find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.45%-- radix_tree_lookup
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--12.81%-- find_get_page
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] __alloc_pages_nodemask
|
--- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--71.39%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--28.61%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] put_mems_allowed
|
--- put_mems_allowed
alloc_pages_current
__page_cache_alloc
|
|--70.79%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--29.21%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% swapper [kernel.kallsyms] [k] load_balance
|
--- load_balance
rebalance_domains
run_rebalance_domains
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.04% flush-253:2 [kernel.kallsyms] [k] __bio_clone
|
--- __bio_clone
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.04% tar [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
|
|--89.40%-- unlock_page
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--10.60%-- wake_up_bit
unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% tar [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
|
|--29.28%-- sub_preempt_count
| |
| |--59.65%-- pagefault_enable
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --40.35%-- bit_spin_unlock
| jbd2_journal_put_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--19.16%-- avc_has_perm_noaudit
| avc_has_perm
| inode_has_perm
| selinux_inode_permission
| security_inode_exec_permission
| exec_permission
| link_path_walk
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
|--19.01%-- _raw_spin_lock
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.20%-- bit_spin_lock
| lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--15.36%-- add_preempt_count
file_read_actor
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] ext4_mark_iloc_dirty
|
--- ext4_mark_iloc_dirty
ext4_mark_inode_dirty
|
|--77.33%-- ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--13.35%-- ext4_new_inode
| ext4_create
| vfs_create
| do_last
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
--9.32%-- ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.04% kswapd0 [kernel.kallsyms] [k] native_write_msr_safe
|
--- native_write_msr_safe
paravirt_write_msr
intel_pmu_disable_all
x86_pmu_disable
perf_pmu_disable
perf_event_task_tick
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
|
|--50.07%-- zone_watermark_ok_safe
| sleeping_prematurely.part.12
| kswapd
| kthread
| kernel_thread_helper
|
--49.93%-- kswapd
kthread
kernel_thread_helper
0.04% swapper [kernel.kallsyms] [k] ext4_end_bio
|
--- ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.04% kswapd0 [kernel.kallsyms] [k] __list_del_entry
|
--- __list_del_entry
finish_wait
kswapd
kthread
kernel_thread_helper
0.04% kswapd0 [kernel.kallsyms] [k] mcount
|
--- mcount
_raw_spin_lock_irq
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% tar [kernel.kallsyms] [k] __percpu_counter_add
|
--- __percpu_counter_add
|
|--87.00%-- ext4_claim_free_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--13.00%-- __prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% tar [kernel.kallsyms] [k] arch_read_unlock
|
--- arch_read_unlock
|
|--57.64%-- _raw_read_unlock_irqrestore
| dm_get_live_table
| dm_merge_bvec
| __bio_add_page.part.2
| bio_add_page
| do_mpage_readpage
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--31.54%-- _raw_read_unlock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--10.82%-- start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% kswapd0 [kernel.kallsyms] [k] mem_cgroup_del_lru_list
|
--- mem_cgroup_del_lru_list
mem_cgroup_del_lru
isolate_lru_pages
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% kswapd0 [kernel.kallsyms] [k] test_and_set_bit
|
--- test_and_set_bit
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% tar [kernel.kallsyms] [k] __might_sleep
|
--- __might_sleep
|
|--22.45%-- kmem_cache_alloc
| |
| |--59.27%-- getname_flags
| | getname
| | do_sys_open
| | sys_openat
| | system_call_fastpath
| | __GI___openat
| |
| --40.73%-- jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--22.30%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--54.37%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --45.63%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--15.00%-- lock_buffer
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--11.33%-- __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--11.31%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.36%-- unmap_underlying_metadata
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--7.25%-- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% kswapd0 [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
|
|--50.69%-- __pagevec_free
| free_page_list
| shrink_page_list
| shrink_inactive_list
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--49.31%-- free_hot_cold_page
__pagevec_free
free_page_list
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% kswapd0 [kernel.kallsyms] [k] __mem_cgroup_uncharge_common
|
--- __mem_cgroup_uncharge_common
mem_cgroup_uncharge_cache_page
__remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% kswapd0 [kernel.kallsyms] [k] tag_get
|
--- tag_get
|
|--50.59%-- radix_tree_delete
| __delete_from_page_cache
| __remove_mapping
| shrink_page_list
| shrink_inactive_list
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--49.41%-- __delete_from_page_cache
__remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.03% flush-253:2 [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--51.77%-- _raw_read_lock_irqsave
| dm_get_live_table
| dm_merge_bvec
| __bio_add_page.part.2
| bio_add_page
| ext4_bio_write_page
| mpage_da_submit_io
| mpage_da_map_and_submit
| ext4_da_writepages
| do_writepages
| writeback_single_inode
| writeback_sb_inodes
| writeback_inodes_wb
| wb_writeback
| wb_do_writeback
| bdi_writeback_thread
| kthread
| kernel_thread_helper
|
--48.23%-- blk_throtl_bio
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] __inc_zone_state
|
--- __inc_zone_state
|
|--49.98%-- __inc_zone_page_state
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--79.48%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --20.52%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--32.04%-- zone_statistics
| get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--17.98%-- account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% tar [kernel.kallsyms] [k] __list_del_entry
|
--- __list_del_entry
|
|--56.86%-- list_del
| __rmqueue
| get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--78.10%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --21.90%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.14%-- __rmqueue
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% flush-253:2 [kernel.kallsyms] [k] dm_table_find_target
|
--- dm_table_find_target
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] jbd_lock_bh_state
|
--- jbd_lock_bh_state
|
|--61.84%-- do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| |
| |--72.93%-- generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --27.07%-- ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--38.16%-- jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% flush-253:2 [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% flush-253:2 [kernel.kallsyms] [k] radix_tree_lookup_element
|
--- radix_tree_lookup_element
radix_tree_lookup_slot
find_get_page
__find_get_block_slow
unmap_underlying_metadata
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] fcheck_files
|
--- fcheck_files
fget_light
|
|--71.87%-- sys_read
| system_call_fastpath
| __GI___libc_read
|
--28.13%-- sys_write
system_call_fastpath
__GI___libc_write
0.03% flush-253:2 [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% flush-253:2 [kernel.kallsyms] [k] lock_page
|
--- lock_page
ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] put_page_testzero
|
--- put_page_testzero
|
|--70.70%-- release_pages
| pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| |
| |--77.62%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --22.38%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--29.30%-- put_page
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% tar [kernel.kallsyms] [k] zone_watermark_ok
|
--- zone_watermark_ok
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--84.29%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--15.71%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% tar [kernel.kallsyms] [k] start_this_handle
|
--- start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--86.30%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--13.70%-- ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% flush-253:2 [kernel.kallsyms] [k] ext4_num_dirty_pages
|
--- ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] put_mems_allowed
|
--- put_mems_allowed
|
|--74.79%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--65.17%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --34.83%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--25.21%-- alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% tar [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
|
|--57.28%-- load_balance
| schedule
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--25.57%-- radix_tree_preload
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--17.15%-- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% swapper [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.03% tar [kernel.kallsyms] [k] avc_has_perm_noaudit
|
--- avc_has_perm_noaudit
avc_has_perm
|
|--82.82%-- inode_has_perm
| |
| |--71.72%-- dentry_has_perm
| | selinux_inode_getattr
| | security_inode_getattr
| | vfs_getattr
| | vfs_fstat
| | sys_newfstat
| | system_call_fastpath
| | __GI___fxstat64
| |
| --28.28%-- selinux_inode_permission
| security_inode_exec_permission
| exec_permission
| link_path_walk
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
--17.18%-- may_create
selinux_inode_create
security_inode_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.03% tar [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% tar [kernel.kallsyms] [k] do_mpage_readpage
|
--- do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% tar [kernel.kallsyms] [k] list_del
|
--- list_del
|
|--55.44%-- get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--67.14%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --32.86%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--28.53%-- ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--16.02%-- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] __rmqueue
|
--- __rmqueue
|
|--77.01%-- get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--22.99%-- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge_lrucare
|
--- __mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--83.13%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--16.87%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] update_page_reclaim_stat
|
--- update_page_reclaim_stat
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
|
|--75.26%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--24.74%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% tar [kernel.kallsyms] [k] radix_tree_insert
|
--- radix_tree_insert
add_to_page_cache_locked
add_to_page_cache_lru
|
|--81.25%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--18.75%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] add_to_page_cache_locked
|
--- add_to_page_cache_locked
add_to_page_cache_lru
|
|--62.06%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--37.94%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
|
|--58.60%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--41.40%-- unlock_page
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] up_read
|
--- up_read
|
|--52.38%-- ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.62%-- ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% swapper [kernel.kallsyms] [k] elv_completed_request
|
--- elv_completed_request
blk_finish_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% tar [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
|
|--84.09%-- task_dirty_inc
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--15.91%-- account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] ext4_map_blocks
|
--- ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% flush-253:2 [kernel.kallsyms] [k] mcount
|
--- mcount
_raw_spin_unlock_irqrestore
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] clear_buffer_dirty
|
--- clear_buffer_dirty
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] mem_cgroup_add_lru_list
|
--- mem_cgroup_add_lru_list
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% swapper [kernel.kallsyms] [k] blk_complete_request
|
--- blk_complete_request
scsi_done
ata_scsi_qc_complete
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% swapper [kernel.kallsyms] [k] sg_init_table
|
--- sg_init_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% tar [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] touch_atime
|
--- touch_atime
|
|--90.35%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--9.65%-- do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% perf ld-2.13.90.so [.] _dl_fixup
|
--- _dl_fixup
0.02% tar [kernel.kallsyms] [k] __block_write_begin
|
--- __block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% kswapd0 [kernel.kallsyms] [k] irqtime_account_process_tick
|
--- irqtime_account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
i915_gem_inactive_shrink
shrink_slab
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] apic_timer_interrupt
|
--- apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% kworker/2:1 [kernel.kallsyms] [k] ns_to_timeval
|
--- ns_to_timeval
get_cpu_iowait_time_us
do_dbs_timer
process_one_work
worker_thread
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] task_rq_lock
|
--- task_rq_lock
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
zone_watermark_ok_safe
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] hrtimer_run_queues
|
--- hrtimer_run_queues
run_local_timers
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] next_zones_zonelist
|
--- next_zones_zonelist
|
|--71.41%-- get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--28.59%-- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% kswapd0 [kernel.kallsyms] [k] arch_local_irq_disable
|
--- arch_local_irq_disable
run_local_timers
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] __rcu_pending
|
--- __rcu_pending
rcu_check_callbacks
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] write_cache_pages_da
|
--- write_cache_pages_da
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] intel_pmu_disable_all
|
--- intel_pmu_disable_all
x86_pmu_disable
perf_pmu_disable
perf_event_task_tick
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
sleeping_prematurely.part.12
kswapd
kthread
kernel_thread_helper
0.02% swapper [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
arch_local_irq_save
irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% kswapd0 [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
call_rcu
d_free
dentry_kill
shrink_dentry_list
__shrink_dcache_sb
shrink_dcache_memory
shrink_slab
kswapd
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] SetPageLRU
|
--- SetPageLRU
|
|--66.95%-- ____pagevec_lru_add_fn
| pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| |
| |--56.88%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --43.12%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--33.05%-- pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% kswapd0 [kernel.kallsyms] [k] run_timer_softirq
|
--- run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
i915_gem_inactive_shrink
shrink_slab
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] __isolate_lru_page
|
--- __isolate_lru_page
isolate_lru_pages
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
__might_sleep
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] radix_tree_delete
|
--- radix_tree_delete
__delete_from_page_cache
__remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_clear
|
--- radix_tree_tag_clear
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] ext4_map_blocks
|
--- ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
__might_sleep
|
|--51.15%-- __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| |
| |--50.51%-- ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_ext_truncate
| | ext4_truncate
| | ext4_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --49.49%-- ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.85%-- kmem_cache_alloc
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% kswapd0 [kernel.kallsyms] [k] mem_cgroup_uncharge_cache_page
|
--- mem_cgroup_uncharge_cache_page
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] update_curr
|
--- update_curr
task_tick_fair
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] timerqueue_del
|
--- timerqueue_del
__remove_hrtimer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_unlock_irqrestore
prepare_to_wait
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] ack_APIC_irq
|
--- ack_APIC_irq
smp_apic_timer_interrupt
apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% kworker/2:1 [kernel.kallsyms] [k] too_many_workers
|
--- too_many_workers
worker_enter_idle
worker_thread
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] update_isolated_counts
|
--- update_isolated_counts
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6
|
--- test_ti_thread_flag.constprop.6
_raw_spin_unlock_irqrestore
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] jbd2_journal_get_write_access
|
--- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_ext_dirty
ext4_ext_insert_extent
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_set
|
--- radix_tree_tag_set
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] __lookup
|
--- __lookup
radix_tree_gang_lookup_slot
find_get_pages
pagevec_lookup
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] page_cache_get_speculative
|
--- page_cache_get_speculative
find_get_pages_tag
pagevec_lookup_tag
ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
_raw_spin_unlock_irqrestore
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] ext4_ext_map_blocks
|
--- ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] jbd_lock_bh_state
|
--- jbd_lock_bh_state
jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mb_mark_diskspace_used
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
jbd_unlock_bh_state
jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mb_mark_diskspace_used
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] alloc_pages_current
|
--- alloc_pages_current
__page_cache_alloc
|
|--72.21%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--27.79%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% flush-253:2 [kernel.kallsyms] [k] ext4_bio_write_page
|
--- ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] ext4_mb_complex_scan_group
|
--- ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% top [kernel.kallsyms] [k] __d_lookup
|
--- __d_lookup
d_lookup
proc_fill_cache
proc_pid_readdir
proc_root_readdir
vfs_readdir
sys_getdents
system_call_fastpath
__getdents64
0.02% flush-253:2 [kernel.kallsyms] [k] kmem_cache_alloc
|
--- kmem_cache_alloc
ext4_init_io_end
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] page_cache_get_speculative
|
--- page_cache_get_speculative
find_get_page
|
|--51.46%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.54%-- find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% flush-253:2 [kernel.kallsyms] [k] mb_find_extent.constprop.12
|
--- mb_find_extent.constprop.12
ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] generic_file_buffered_write
|
--- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% top libc-2.13.90.so [.] __GI_vfprintf
|
--- __GI_vfprintf
___vsprintf_chk
0.02% tar [kernel.kallsyms] [k] ext4_da_get_block_prep
|
--- ext4_da_get_block_prep
|
|--75.27%-- __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--24.73%-- ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% top [kernel.kallsyms] [k] getname_flags
|
--- getname_flags
getname
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.02% swapper [kernel.kallsyms] [k] __count_vm_events
|
--- __count_vm_events
__free_pages
__free_slab
discard_slab
__slab_free
kmem_cache_free
ext4_free_io_end
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% swapper [kernel.kallsyms] [k] tick_check_idle
|
--- tick_check_idle
irq_enter
|
|--73.47%-- smp_call_function_single_interrupt
| call_function_single_interrupt
| cpuidle_idle_call
| cpu_idle
| start_secondary
|
--26.53%-- smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% flush-253:2 [kernel.kallsyms] [k] find_next_zero_bit
|
--- find_next_zero_bit
mb_find_next_zero_bit
ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] arch_read_lock
|
--- arch_read_lock
_raw_read_lock
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% flush-253:2 [kernel.kallsyms] [k] mb_find_buddy
|
--- mb_find_buddy
mb_mark_used
ext4_mb_use_best_found
ext4_mb_check_limits
ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% swapper [kernel.kallsyms] [k] __rcu_read_unlock
|
--- __rcu_read_unlock
__prop_inc_percpu_max
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% tar [kernel.kallsyms] [k] mark_buffer_dirty
|
--- mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] new_slab
|
--- new_slab
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% swapper [kernel.kallsyms] [k] __do_softirq
|
--- __do_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% swapper [kernel.kallsyms] [k] select_task_rq_fair
|
--- select_task_rq_fair
select_task_rq
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% tar [kernel.kallsyms] [k] do_get_write_access
|
--- do_get_write_access
|
|--70.88%-- jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| |
| |--54.37%-- ext4_new_inode
| | ext4_create
| | vfs_create
| | do_last
| | path_openat
| | do_filp_open
| | do_sys_open
| | sys_openat
| | system_call_fastpath
| | __GI___openat
| |
| --45.63%-- ext4_reserve_inode_write
| ext4_orphan_add
| ext4_unlink
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--29.12%-- __ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] in_lock_functions
|
--- in_lock_functions
|
|--58.27%-- get_parent_ip
| add_preempt_count
| |
| |--56.37%-- jbd_lock_bh_state
| | jbd2_journal_dirty_metadata
| | __ext4_handle_dirty_metadata
| | ext4_mark_iloc_dirty
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --43.63%-- radix_tree_preload
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--41.73%-- sub_preempt_count
bit_spin_unlock
jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] jbd2_journal_dirty_metadata
|
--- jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] ext4_ext_map_blocks
|
--- ext4_ext_map_blocks
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] block_write_end
|
--- block_write_end
|
|--67.22%-- ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--32.78%-- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% flush-253:2 [kernel.kallsyms] [k] mb_test_bit
|
--- mb_test_bit
mb_find_order_for_block
mb_find_extent.constprop.12
ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] attach_page_buffers
|
--- attach_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __phys_addr
|
--- __phys_addr
virt_to_head_page
kmem_cache_free
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] do_sync_read
|
--- do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] swiotlb_map_sg_attrs
|
--- swiotlb_map_sg_attrs
ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] jbd2__journal_start
|
--- jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--73.97%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--26.03%-- ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] iov_iter_advance
|
--- iov_iter_advance
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] generic_getxattr
|
--- generic_getxattr
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% jbd2/dm-2-8 [kernel.kallsyms] [k] elv_rq_merge_ok
|
--- elv_rq_merge_ok
elv_try_merge
__make_request
generic_make_request
submit_bio
submit_bh
jbd2_journal_commit_transaction
kjournald2
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] clear_bit_unlock
|
--- clear_bit_unlock
unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
|
|--52.26%-- unlock_page
| mpage_end_io
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--47.74%-- mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] ext4_xattr_find_entry
|
--- ext4_xattr_find_entry
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
lock_page_cgroup
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--76.11%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--23.89%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% flush-253:2 [kernel.kallsyms] [k] writeback_single_inode
|
--- writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] find_get_page
|
--- find_get_page
|
|--70.62%-- __find_get_block_slow
| __find_get_block
| ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--29.38%-- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% flush-253:2 [kernel.kallsyms] [k] __ext4_get_inode_loc
|
--- __ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] fsnotify_modify
|
--- fsnotify_modify
|
|--73.71%-- vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--26.29%-- sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] blk_rq_map_sg
|
--- blk_rq_map_sg
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] get_page
|
--- get_page
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--44.23%-- blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--32.84%-- blk_run_queue
| scsi_run_queue
| scsi_next_command
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--22.93%-- pm_qos_request
menu_select
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] __bio_clone
|
--- __bio_clone
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] sg_next
|
--- sg_next
|
|--36.30%-- ata_qc_issue
| __ata_scsi_queuecmd
| ata_scsi_queuecmd
| scsi_dispatch_cmd
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--32.06%-- swiotlb_map_sg_attrs
| ata_qc_issue
| __ata_scsi_queuecmd
| ata_scsi_queuecmd
| scsi_dispatch_cmd
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--31.64%-- blk_rq_map_sg
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] __rcu_read_unlock
|
--- __rcu_read_unlock
|
|--70.72%-- __prop_inc_single
| task_dirty_inc
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--29.28%-- avc_has_perm_noaudit
task_has_capability
selinux_capable
security_capable
ns_capable
capable
setattr_copy
ext4_setattr
notify_change
sys_fchmod
system_call_fastpath
__GI___fchmod
0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock
|
--- _raw_spin_unlock
|
|--36.11%-- smp_apic_timer_interrupt
| apic_timer_interrupt
| __find_get_block_slow
| unmap_underlying_metadata
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--34.75%-- scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--29.14%-- __block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] generic_file_aio_read
|
--- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] map_bh
|
--- map_bh
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] memcmp
|
--- memcmp
|
|--67.73%-- ext4_dx_find_entry
| ext4_find_entry
| ext4_lookup
| d_alloc_and_lookup
| __lookup_hash.part.3
| lookup_hash
| do_last
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
--32.27%-- search_dirblock
ext4_dx_find_entry
ext4_find_entry
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% swapper [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] clone_endio
|
--- clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] __slab_free
|
--- __slab_free
kmem_cache_free
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] nr_iowait_cpu
|
--- nr_iowait_cpu
tick_nohz_stop_idle
tick_check_idle
irq_enter
smp_call_function_single_interrupt
call_function_single_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.01% swapper [kernel.kallsyms] [k] timekeeping_get_ns
|
--- timekeeping_get_ns
getnstimeofday
ktime_get_real
intel_idle
cpuidle_idle_call
cpu_idle
start_secondary
0.01% flush-253:2 [kernel.kallsyms] [k] crc16
|
--- crc16
ext4_group_desc_csum
ext4_mb_mark_diskspace_used
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] ata_scsi_queuecmd
|
--- ata_scsi_queuecmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] jbd2_journal_put_journal_head
|
--- jbd2_journal_put_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
|
|--60.89%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--39.11%-- ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] ext4_da_write_end
|
--- ext4_da_write_end
|
|--68.79%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--31.21%-- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] acct_update_integrals
|
--- acct_update_integrals
__account_system_time
irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
|
|--51.45%-- get_parent_ip
| add_preempt_count
| __lru_cache_add
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.55%-- find_get_page
__find_get_block_slow
__find_get_block
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] __inc_zone_page_state
|
--- __inc_zone_page_state
|
|--61.16%-- account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--38.84%-- add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] account_idle_time
|
--- account_idle_time
account_idle_ticks
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] recalc_bh_state
|
--- recalc_bh_state
|
|--62.12%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--37.88%-- alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] put_page_testzero
|
--- put_page_testzero
put_page
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] generic_segment_checks
|
--- generic_segment_checks
|
|--56.32%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--43.68%-- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
__free_pages
__free_slab
discard_slab
__slab_free
kmem_cache_free
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] rw_verify_area
|
--- rw_verify_area
|
|--65.20%-- vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--34.80%-- vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] file_update_time
|
--- file_update_time
|
|--70.97%-- __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--29.03%-- generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% flush-253:2 [kernel.kallsyms] [k] alloc_tio
|
--- alloc_tio
dm_request
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __mark_inode_dirty
|
--- __mark_inode_dirty
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore
|
--- _raw_spin_unlock_irqrestore
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] jbd2_journal_cancel_revoke
|
--- jbd2_journal_cancel_revoke
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] lookup_page_cgroup
|
--- lookup_page_cgroup
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--53.32%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--46.68%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock_irq
|
--- _raw_spin_unlock_irq
add_to_page_cache_locked
add_to_page_cache_lru
|
|--69.09%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--30.91%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __do_page_cache_readahead
|
--- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] ext4_acl_chmod
|
--- ext4_acl_chmod
notify_change
sys_fchmod
system_call_fastpath
__GI___fchmod
0.01% flush-253:2 [kernel.kallsyms] [k] start_this_handle
|
--- start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] _raw_read_lock
|
--- _raw_read_lock
|
|--56.63%-- start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.37%-- jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] mix_pool_bytes_extract
|
--- mix_pool_bytes_extract
add_timer_randomness
add_disk_randomness
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] system_call_after_swapgs
|
--- system_call_after_swapgs
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] ext4_claim_free_blocks
|
--- ext4_claim_free_blocks
|
|--53.08%-- __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--46.92%-- ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] sys_write
|
--- sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] mcount
|
--- mcount
|
|--52.17%-- radix_tree_preload_end
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.83%-- _raw_spin_unlock
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] pagevec_lru_move_fn
|
--- pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] jbd2_journal_add_journal_head
|
--- jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __block_commit_write
|
--- __block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] virt_to_head_page
|
--- virt_to_head_page
|
|--57.98%-- kmem_cache_free
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--42.02%-- jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] mb_test_bit
|
--- mb_test_bit
|
|--53.79%-- ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--46.21%-- mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
|
|--57.00%-- ext4_mark_iloc_dirty
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.00%-- ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ahci_qc_prep
|
--- ahci_qc_prep
|
|--56.69%-- __ata_scsi_queuecmd
| ata_scsi_queuecmd
| scsi_dispatch_cmd
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--43.31%-- ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] generic_write_end
|
--- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] iov_iter_fault_in_readable
|
--- iov_iter_fault_in_readable
|
|--51.45%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.55%-- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] inode_reserved_space
|
--- inode_reserved_space
|
|--54.74%-- inode_add_rsv_space
| __dquot_alloc_space
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--45.26%-- __dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] update_curr
|
--- update_curr
dequeue_task_fair
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] atomic_sub
|
--- atomic_sub
|
|--56.22%-- __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.78%-- jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_get_inode_flags
|
--- ext4_get_inode_flags
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] prop_norm_single
|
--- prop_norm_single
|
|--55.85%-- task_dirty_inc
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.15%-- __prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] jbd_unlock_bh_state
|
--- jbd_unlock_bh_state
|
|--50.48%-- jbd2_journal_dirty_metadata
| __ext4_handle_dirty_metadata
| ext4_mark_iloc_dirty
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--49.52%-- do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] arch_local_irq_disable
|
--- arch_local_irq_disable
arch_local_irq_save
_raw_spin_lock_irqsave
|
|--65.53%-- __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--34.47%-- pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __generic_file_aio_write
|
--- __generic_file_aio_write
|
|--53.33%-- generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--46.67%-- ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] test_and_set_bit
|
--- test_and_set_bit
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% flush-253:2 [kernel.kallsyms] [k] _raw_read_lock_irqsave
|
--- _raw_read_lock_irqsave
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __find_get_block
|
--- __find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
|
|--64.27%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--35.73%-- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __split_and_process_bio
|
--- __split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] memcg_check_events
|
--- memcg_check_events
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--52.26%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.74%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] xattr_resolve_name
|
--- xattr_resolve_name
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% kworker/0:2 [kernel.kallsyms] [k] bit_cursor
|
--- bit_cursor
fb_flashcursor
process_one_work
worker_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __set_page_dirty
|
--- __set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_file_write
|
--- ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] bit_spin_unlock
|
--- bit_spin_unlock
jbd2_journal_put_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] fget_light
|
--- fget_light
sys_read
system_call_fastpath
__GI___libc_read
0.01% flush-253:2 [kernel.kallsyms] [k] ext4_clear_inode_state
|
--- ext4_clear_inode_state
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% swapper [kernel.kallsyms] [k] ahci_scr_read
|
--- ahci_scr_read
sata_scr_read
sata_async_notification
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] add_to_page_cache_lru
|
--- add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
_local_bh_enable
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] current_fs_time
|
--- current_fs_time
|
|--50.34%-- touch_atime
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--49.66%-- file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] need_resched
|
--- need_resched
_cond_resched
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] mb_clear_bit
|
--- mb_clear_bit
|
|--57.54%-- ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--42.46%-- mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] __list_add
|
--- __list_add
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] task_dirty_inc
|
--- task_dirty_inc
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] vfs_read
|
--- vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
|
|--55.69%-- jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.31%-- jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] paravirt_read_tsc
|
--- paravirt_read_tsc
read_tsc
timekeeping_get_ns
ktime_get
tick_nohz_stop_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] irq_entries_start
|
--- irq_entries_start
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] iov_iter_copy_from_user_atomic
|
--- iov_iter_copy_from_user_atomic
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] clear_buffer_new
|
--- clear_buffer_new
|
|--57.80%-- __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--42.20%-- block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] _raw_read_lock_irqsave
|
--- _raw_read_lock_irqsave
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] __prop_inc_single
|
--- __prop_inc_single
|
|--52.30%-- account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.70%-- task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] dm_merge_bvec
|
--- dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] menu_select
|
--- menu_select
|
|--59.21%-- cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--40.79%-- cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] scsi_decide_disposition
|
--- scsi_decide_disposition
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] bit_spin_trylock.constprop.23
|
--- bit_spin_trylock.constprop.23
get_partial_node
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] exit_idle
|
--- exit_idle
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% flush-253:2 [kernel.kallsyms] [k] mpage_da_submit_io
|
--- mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% swapper [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock_irq
schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] get_cwq
|
--- get_cwq
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_lock
inode_add_rsv_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ____pagevec_lru_add_fn
|
--- ____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
|
|--57.19%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--42.81%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] alloc_buffer_head
|
--- alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] generic_make_request
|
--- generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] rcu_irq_enter
|
--- rcu_irq_enter
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] leave_mm
|
--- leave_mm
intel_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] __lru_cache_add
|
--- __lru_cache_add
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar libc-2.13.90.so [.] __GI___libc_write
|
--- __GI___libc_write
0.01% swapper [kernel.kallsyms] [k] task_rq_lock
|
--- task_rq_lock
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] __elv_add_request
|
--- __elv_add_request
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] may_link
|
--- may_link
selinux_inode_unlink
security_inode_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] __find_get_block_slow
|
--- __find_get_block_slow
unmap_underlying_metadata
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] radix_tree_tag_set
|
--- radix_tree_tag_set
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% flush-253:2 [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] delayacct_blkio_start
|
--- delayacct_blkio_start
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] current_umask
|
--- current_umask
ext4_init_acl
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.01% tar [kernel.kallsyms] [k] linear_merge
|
--- linear_merge
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% flush-253:2 [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
_raw_spin_lock
ext4_discard_preallocations
ext4_da_update_reserve_space
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __blk_run_queue
|
--- __blk_run_queue
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] __page_cache_alloc
|
--- __page_cache_alloc
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
account_system_vtime
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] ext4_new_inode
|
--- ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.01% tar [kernel.kallsyms] [k] __slab_alloc
|
--- __slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_ext_calc_metadata_amount
|
--- ext4_ext_calc_metadata_amount
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] task_of
|
--- task_of
dequeue_task_fair
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] tick_program_event
|
--- tick_program_event
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% perf [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
unlock_page
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.01% perf [kernel.kallsyms] [k] generic_write_end
|
--- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.01% kworker/0:2 [kernel.kallsyms] [k] native_read_msr_safe
|
--- native_read_msr_safe
acpi_cpufreq_init
do_drv_write
acpi_cpufreq_target
__cpufreq_driver_target
do_dbs_timer
process_one_work
worker_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] kmem_cache_free
|
--- kmem_cache_free
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_journal_start_sb
|
--- ext4_journal_start_sb
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_get_inode_loc
|
--- ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __mem_cgroup_try_charge.constprop.27
|
--- __mem_cgroup_try_charge.constprop.27
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] hrtick_update
|
--- hrtick_update
enqueue_task
activate_task
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] mem_cgroup_get_reclaim_stat_from_page
|
--- mem_cgroup_get_reclaim_stat_from_page
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] bio_endio
|
--- bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] kfree
|
--- kfree
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_free_blocks
|
--- ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] ext4_test_inode_flag
|
--- ext4_test_inode_flag
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] scsi_run_queue
|
--- scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% kworker/0:2 [cpufreq_ondemand] [k] do_dbs_timer
|
--- do_dbs_timer
worker_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] alloc_page_buffers
|
--- alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] linear_map
|
--- linear_map
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] test_ti_thread_flag.constprop.7
|
--- test_ti_thread_flag.constprop.7
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] delayacct_blkio_end
|
--- delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] inode_permission
|
--- inode_permission
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.01% kworker/0:2 [kernel.kallsyms] [k] kobject_get
|
--- kobject_get
__cpufreq_driver_getavg
do_dbs_timer
process_one_work
worker_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __wake_up_common
|
--- __wake_up_common
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_read_inode_bitmap
|
--- ext4_read_inode_bitmap
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] mnt_add_count
|
--- mnt_add_count
mntget
path_get
nameidata_to_filp
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.01% tar [kernel.kallsyms] [k] schedule
|
--- schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] try_to_wake_up
|
--- try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] inode_doinit_with_dentry
|
--- inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
ext4_add_nondir
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% tar [kernel.kallsyms] [k] radix_tree_preload
|
--- radix_tree_preload
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __mod_zone_page_state
|
--- __mod_zone_page_state
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] unmap_single
|
--- unmap_single
swiotlb_unmap_sg_attrs
ata_sg_clean
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar libc-2.13.90.so [.] _int_malloc
|
--- _int_malloc
0.00% tar [kernel.kallsyms] [k] ext4_test_inode_state
|
--- ext4_test_inode_state
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6
|
--- test_ti_thread_flag.constprop.6
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __ext4_handle_dirty_metadata
|
--- __ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] queue_unplugged
|
--- queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] mutex_lock
|
--- mutex_lock
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] crc16
|
--- crc16
ext4_group_desc_csum
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% tar [kernel.kallsyms] [k] lock_page_killable
|
--- lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] lock_page
|
--- lock_page
find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] balance_dirty_pages_ratelimited_nr
|
--- balance_dirty_pages_ratelimited_nr
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] setup_object
|
--- setup_object
new_slab
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] sd_prep_fn
|
--- sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] system_call_fastpath
|
--- system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] unlock_page
|
--- unlock_page
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] blk_throtl_bio
|
--- blk_throtl_bio
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] ihold
|
--- ihold
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% tar [kernel.kallsyms] [k] mb_find_buddy
|
--- mb_find_buddy
mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% tar [kernel.kallsyms] [k] jbd2_journal_get_write_access
|
--- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] _raw_spin_unlock
|
--- _raw_spin_unlock
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] bit_spin_unlock
|
--- bit_spin_unlock
unlock_page_cgroup
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] fput_light
|
--- fput_light
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] audit_inode.constprop.6
|
--- audit_inode.constprop.6
system_call_fastpath
__GI___fchown
0.00% tar [kernel.kallsyms] [k] ext4_test_inode_state
|
--- ext4_test_inode_state
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] system_call
|
--- system_call
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] ktime_get
|
--- ktime_get
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] radix_tree_lookup_slot
|
--- radix_tree_lookup_slot
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] apic_timer_interrupt
|
--- apic_timer_interrupt
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] __rcu_read_lock
|
--- __rcu_read_lock
__prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] arch_local_irq_enable
|
--- arch_local_irq_enable
_raw_spin_unlock_irq
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] file_read_actor
|
--- file_read_actor
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] dquot_active
|
--- dquot_active
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% sleep [kernel.kallsyms] [k] tlb_gather_mmu
|
--- tlb_gather_mmu
mmput
exit_mm
do_exit
do_group_exit
__wake_up_parent
system_call_fastpath
0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] bvec_free_bs
|
--- bvec_free_bs
dm_bio_destructor
bio_put
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] ktime_get_real
|
--- ktime_get_real
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] generic_write_sync
|
--- generic_write_sync
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] _raw_spin_lock_irq
|
--- _raw_spin_lock_irq
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] posix_acl_release
|
--- posix_acl_release
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% tar [kernel.kallsyms] [k] ext4_dirty_inode
|
--- ext4_dirty_inode
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] test_clear_page_writeback
|
--- test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] deadline_dispatch_requests
|
--- deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] deadline_move_request
|
--- deadline_move_request
deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% kworker/0:2 [kernel.kallsyms] [k] schedule
|
--- schedule
worker_thread
kthread
kernel_thread_helper
0.00% tar [kernel.kallsyms] [k] blk_start_plug
|
--- blk_start_plug
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] set_buffer_uptodate
|
--- set_buffer_uptodate
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] strncpy_from_user
|
--- strncpy_from_user
getname
user_path_parent
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
drive_stat_acct
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] update_curr
|
--- update_curr
enqueue_task_fair
enqueue_task
activate_task
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
rcu_process_callbacks
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] page_cache_async_readahead
|
--- page_cache_async_readahead
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] bvec_alloc_bs
|
--- bvec_alloc_bs
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] page_mapping
|
--- page_mapping
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] kmem_cache_free
|
--- kmem_cache_free
mempool_free_slab
mempool_free
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] prop_norm_percpu
|
--- prop_norm_percpu
__prop_inc_percpu_max
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] sg_init_table
|
--- sg_init_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] bit_waitqueue
|
--- bit_waitqueue
wake_up_bit
unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] apic_write
|
--- apic_write
ack_APIC_irq
ack_apic_edge
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] native_write_msr_safe
|
--- native_write_msr_safe
0.00% swapper [kernel.kallsyms] [k] req_bio_endio
|
--- req_bio_endio
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] __wake_up
|
--- __wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] mem_cgroup_disabled
|
--- mem_cgroup_disabled
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __make_request
|
--- __make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] tick_check_oneshot_broadcast
|
--- tick_check_oneshot_broadcast
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] raise_softirq
|
--- raise_softirq
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
radix_tree_preload_end
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] ext4_setattr
|
--- ext4_setattr
notify_change
utimes_common
do_utimes
sys_utimensat
system_call_fastpath
futimens
0.00% swapper [kernel.kallsyms] [k] native_sched_clock
|
--- native_sched_clock
sched_clock_cpu
account_system_vtime
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] read_tsc
|
--- read_tsc
timekeeping_get_ns
ktime_get
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] cap_inode_need_killpriv
|
--- cap_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __cycles_2_ns
|
--- __cycles_2_ns
native_sched_clock
sched_clock
blk_rq_init
get_request
get_request_wait
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] load_balance
|
--- load_balance
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] read_tsc
|
--- read_tsc
ktime_get_ts
delayacct_end
__delayacct_blkio_end
delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] scsi_sg_alloc
|
--- scsi_sg_alloc
__sg_alloc_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] cpuacct_charge
|
--- cpuacct_charge
update_curr
dequeue_task_fair
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] elv_rqhash_add
|
--- elv_rqhash_add
__elv_add_request
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.22
|
--- test_ti_thread_flag.constprop.22
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] ata_find_dev.part.3
|
--- ata_find_dev.part.3
__ata_scsi_find_dev
ata_scsi_find_dev
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] fsnotify
|
--- fsnotify
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% kworker/0:2 [kernel.kallsyms] [k] ns_to_timeval
|
--- ns_to_timeval
get_cpu_iowait_time_us
do_dbs_timer
process_one_work
worker_thread
kthread
kernel_thread_helper
0.00% tar [kernel.kallsyms] [k] expand
|
--- expand
__rmqueue
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] preempt_schedule
|
--- preempt_schedule
blk_finish_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] max_io_len
|
--- max_io_len
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] release_pages
|
--- release_pages
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] irqtime_account_process_tick
|
--- irqtime_account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] atomic_long_add
|
--- atomic_long_add
zone_page_state_add
__inc_zone_state
__inc_zone_page_state
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] zone_statistics
|
--- zone_statistics
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] str2hashbuf_signed
|
--- str2hashbuf_signed
ext4fs_dirhash
dx_probe
ext4_dx_find_entry
ext4_find_entry
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% swapper [kernel.kallsyms] [k] ata_scsi_qc_complete
|
--- ata_scsi_qc_complete
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] ext4_has_free_blocks
|
--- ext4_has_free_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] deadline_add_request
|
--- deadline_add_request
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] pagefault_enable
|
--- pagefault_enable
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] deadline_merge
|
--- deadline_merge
elv_merge
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] rb_next
|
--- rb_next
timerqueue_del
__remove_hrtimer
remove_hrtimer.part.4
hrtimer_try_to_cancel
hrtimer_cancel
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] mem_cgroup_cache_charge
|
--- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] sched_clock_tick
|
--- sched_clock_tick
sched_clock_idle_wakeup_event
tick_nohz_stop_idle
tick_check_idle
irq_enter
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] put_bh
|
--- put_bh
__brelse
brelse
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] get_page
|
--- get_page
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] hashtab_search
|
--- hashtab_search
string_to_context_struct
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% tar [kernel.kallsyms] [k] ata_qc_new_init
|
--- ata_qc_new_init
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] sd_done
|
--- sd_done
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% sleep [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
add_preempt_count
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
pte_alloc_one
__pte_alloc
handle_mm_fault
do_page_fault
page_fault
_dl_vdso_vsym
0x7fff16d617f5
0x535345535f474458
0.00% sleep [kernel.kallsyms] [k] policy_zonelist
|
--- policy_zonelist
pte_alloc_one
__pte_alloc
handle_mm_fault
do_page_fault
page_fault
_dl_init_paths
_dl_sysdep_start
0.00% tar [kernel.kallsyms] [k] find_lock_page
|
--- find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] lock_buffer
|
--- lock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] _cond_resched
|
--- _cond_resched
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] ext4_da_write_begin
|
--- ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] dquot_free_inode
|
--- dquot_free_inode
ext4_free_inode
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% tar [kernel.kallsyms] [k] get_request_wait
|
--- get_request_wait
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] __local_bh_enable
|
--- __local_bh_enable
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] mutex_unlock
|
--- mutex_unlock
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] hrtimer_interrupt
|
--- hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] create_empty_buffers
|
--- create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] radix_tree_preload_end
|
--- radix_tree_preload_end
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] ext4_xattr_set_entry
|
--- ext4_xattr_set_entry
ext4_xattr_ibody_set
ext4_xattr_set_handle
ext4_init_security
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% swapper [kernel.kallsyms] [k] SetPageUptodate
|
--- SetPageUptodate
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] mem_cgroup_charge_statistics
|
--- mem_cgroup_charge_statistics
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar libc-2.13.90.so [.] _int_free
|
--- _int_free
0.00% swapper [kernel.kallsyms] [k] schedule
|
--- schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] sysret_check
|
--- sysret_check
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] blk_peek_request
|
--- blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] ata_scsi_rw_xlat
|
--- ata_scsi_rw_xlat
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] blk_flush_plug_list
|
--- blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] jbd2_journal_stop
|
--- jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __srcu_read_lock
|
--- __srcu_read_lock
fsnotify
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
#
# (For a higher level overview, try: perf report --sort comm,dso)
#
On Thu, 2011-04-28 at 16:08 +0100, Mel Gorman wrote:
[ text deleted ]
> Another consequence of this patch is that when high order allocations
> are in progress (is the test case fork heavy in any way for
> example? alternatively, it might be something in the storage stack
> that requires high-order allocs) we are no longer necessarily going
> to sleep because of should_reclaim_continue() check. This could
> explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is
> set (does unsetting CONFIG_COMPACTION help). If the bug only triggers
> for CONFIG_COMPACTION, does the following *untested* patch help any?
Afraid to report this patch didn't help either.
>
> (as a warning, I'm offline Friday until Tuesday morning. I'll try
> check mail over the weekend but it's unlikely I'll find a terminal
> or be allowed to use it without an ass kicking)
Ditto, me, to, I will pick this up Tuesday.
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 148c6e6..c74a501 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -1842,15 +1842,22 @@ static inline bool should_continue_reclaim(struct zone *zone,
> return false;
>
> /*
> - * If we failed to reclaim and have scanned the full list, stop.
> - * NOTE: Checking just nr_reclaimed would exit reclaim/compaction far
> - * faster but obviously would be less likely to succeed
> - * allocation. If this is desirable, use GFP_REPEAT to decide
> - * if both reclaimed and scanned should be checked or just
> - * reclaimed
> + * For direct reclaimers
> + * If we failed to reclaim and have scanned the full list, stop.
> + * The caller will check congestion and sleep if necessary until
> + * some IO completes.
> + * For kswapd
> + * Check just nr_reclaimed. If we are failing to reclaim, we
> + * want to stop this reclaim loop, increase the priority and
> + * go to sleep if necessary to allow IO a change to complete.
> + * This avoids kswapd going into a busy loop in shrink_zone()
> */
> - if (!nr_reclaimed && !nr_scanned)
> - return false;
> + if (!nr_reclaimed) {
> + if (current_is_kswapd())
> + return false;
> + else if (!nr_scanned)
> + return false;
> + }
>
> /*
> * If we have not reclaimed enough pages for compaction and the
> @@ -1924,8 +1931,13 @@ restart:
>
> /* reclaim/compaction might need reclaim to continue */
> if (should_continue_reclaim(zone, nr_reclaimed,
> - sc->nr_scanned - nr_scanned, sc))
> + sc->nr_scanned - nr_scanned, sc)) {
> + /* Throttle direct reclaimers if congested */
> + if (!current_is_kswapd())
> + wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> +
> goto restart;
> + }
>
> throttle_vm_writeout(sc->gfp_mask);
> }
On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote:
> > This is the output of perf record -g -a -f sleep 5
> >
> > (hopefully the list won't choke)
>
> Um, this one actually shows kswapd
>
> James
>
> ---
>
> # Events: 6K cycles
> #
> # Overhead Command Shared Object Symbol
> # ........ ........... ................... .......................................
> #
> 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
> |
> --- shrink_slab
> |
> |--99.91%-- kswapd
> | kthread
> | kernel_thread_helper
> --0.09%-- [...]
>
Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use
compaction instead of lumpy reclaim" is related unless we are seeing
two problems that happen to manifest in a similar manner.
However, there were a number of changes made to dcache in particular
for 2.6.38. Specifically thinks like dentry_kill use trylock and is
happy to loop around if it fails to acquire anything. See things like
this for example;
static void try_prune_one_dentry(struct dentry *dentry)
__releases(dentry->d_lock)
{
struct dentry *parent;
parent = dentry_kill(dentry, 0);
/*
* If dentry_kill returns NULL, we have nothing more to do.
* if it returns the same dentry, trylocks failed. In either
* case, just loop again.
If this in combination with many inodes being locked for whatever
reason (writeback locking them maybe?) is causing the shrinker to
return after zero progress, it could in turn cause kswapd to enter
into a loop for longish periods of time in shrink_slab here;
while (total_scan >= SHRINK_BATCH) {
long this_scan = SHRINK_BATCH;
int shrink_ret;
int nr_before;
nr_before = (*shrinker->shrink)(shrinker, 0, gfp_mask);
shrink_ret = (*shrinker->shrink)(shrinker, this_scan,
gfp_mask);
if (shrink_ret == -1)
break;
if (shrink_ret < nr_before)
ret += nr_before - shrink_ret;
count_vm_events(SLABS_SCANNED, this_scan);
total_scan -= this_scan;
cond_resched();
}
That would explain this trace.
> 9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone
> |
> --- shrink_zone
> |
> |--99.46%-- kswapd
> | kthread
> | kernel_thread_helper
> |
> --0.54%-- kthread
> kernel_thread_helper
>
> 7.70% kswapd0 [kernel.kallsyms] [k] kswapd
> |
> --- kswapd
> kthread
> kernel_thread_helper
>
> 5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe
> |
> --- zone_watermark_ok_safe
> |
> |--72.66%-- kswapd
> | kthread
> | kernel_thread_helper
> |
> |--20.88%-- sleeping_prematurely.part.12
> | kswapd
> | kthread
> | kernel_thread_helper
> |
> --6.46%-- kthread
> kernel_thread_helper
>
We are also spending an astonishing amount of time in
sleeping_prematurely leading me to believe we are failing to balance the
zones and are continually under the min watermark for one of the zones.
We are never going to sleep because of this check;
if (total_scanned && (priority < DEF_PRIORITY - 2)) {
if (has_under_min_watermark_zone)
count_vm_event(KSWAPD_SKIP_CONGESTION_WAIT);
else
congestion_wait(BLK_RW_ASYNC, HZ/10);
}
However, I think this is a secondary effect to the failure of shrinkers
to do their work. If slabs were being shrunk, one would expect us to
be getting over the min watermark.
> 4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock
> |
> --- do_raw_spin_lock
> |
> |--77.49%-- _raw_spin_lock
> | |
> | |--51.85%-- mb_cache_shrink_fn
> | | shrink_slab
> | | kswapd
> | | kthread
> | | kernel_thread_helper
> | |
> | --48.15%-- mem_cgroup_soft_limit_reclaim
> | kswapd
> | kthread
> | kernel_thread_helper
> |
Way hey, cgroups are also in the mix. How jolly.
Is systemd a common element of the machines hitting this bug by any
chance?
The remaining traces seem to be follow-on damage related to the three
issues of "shrinkers are bust in some manner" causing "we are not
getting over the min watermark" and as a side-show "we are spending lots
of time doing something unspecified but unhelpful in cgroups".
> <SNIP>
--
Mel Gorman
SUSE Labs
On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote:
> > # Events: 6K cycles
> > #
> > # Overhead Command Shared Object Symbol
> > # ........ ........... ................... .......................................
> > #
> > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
> > |
> > --- shrink_slab
> > |
> > |--99.91%-- kswapd
> > | kthread
> > | kernel_thread_helper
> > --0.09%-- [...]
> >
>
> Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use
> compaction instead of lumpy reclaim" is related unless we are seeing
> two problems that happen to manifest in a similar manner.
>
> However, there were a number of changes made to dcache in particular
> for 2.6.38. Specifically thinks like dentry_kill use trylock and is
> happy to loop around if it fails to acquire anything. See things like
> this for example;
OK, so for this, I tried a 2.6.37 kernel. It doesn't work very well,
networking is hosed for no reason I can see (probably systemd / cgroups
problems).
However, it runs enough for me to say that the tar proceeds to
completion in a non-PREEMPT kernel. (I tried several times for good
measure). That makes this definitely a regression of some sort, but it
doesn't definitively identify the dcache code ... it could be an ext4
bug that got introduced in 2.6.38 either.
> static void try_prune_one_dentry(struct dentry *dentry)
> __releases(dentry->d_lock)
> {
> struct dentry *parent;
>
> parent = dentry_kill(dentry, 0);
> /*
> * If dentry_kill returns NULL, we have nothing more to do.
> * if it returns the same dentry, trylocks failed. In either
> * case, just loop again.
>
>
> If this in combination with many inodes being locked for whatever
> reason (writeback locking them maybe?) is causing the shrinker to
> return after zero progress, it could in turn cause kswapd to enter
> into a loop for longish periods of time in shrink_slab here;
>
> while (total_scan >= SHRINK_BATCH) {
> long this_scan = SHRINK_BATCH;
> int shrink_ret;
> int nr_before;
>
> nr_before = (*shrinker->shrink)(shrinker, 0, gfp_mask);
> shrink_ret = (*shrinker->shrink)(shrinker, this_scan,
> gfp_mask);
> if (shrink_ret == -1)
> break;
> if (shrink_ret < nr_before)
> ret += nr_before - shrink_ret;
> count_vm_events(SLABS_SCANNED, this_scan);
> total_scan -= this_scan;
>
> cond_resched();
> }
>
> That would explain this trace.
>
> > 9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone
> > |
> > --- shrink_zone
> > |
> > |--99.46%-- kswapd
> > | kthread
> > | kernel_thread_helper
> > |
> > --0.54%-- kthread
> > kernel_thread_helper
> >
> > 7.70% kswapd0 [kernel.kallsyms] [k] kswapd
> > |
> > --- kswapd
> > kthread
> > kernel_thread_helper
> >
> > 5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe
> > |
> > --- zone_watermark_ok_safe
> > |
> > |--72.66%-- kswapd
> > | kthread
> > | kernel_thread_helper
> > |
> > |--20.88%-- sleeping_prematurely.part.12
> > | kswapd
> > | kthread
> > | kernel_thread_helper
> > |
> > --6.46%-- kthread
> > kernel_thread_helper
> >
>
> We are also spending an astonishing amount of time in
> sleeping_prematurely leading me to believe we are failing to balance the
> zones and are continually under the min watermark for one of the zones.
> We are never going to sleep because of this check;
>
> if (total_scanned && (priority < DEF_PRIORITY - 2)) {
> if (has_under_min_watermark_zone)
> count_vm_event(KSWAPD_SKIP_CONGESTION_WAIT);
> else
> congestion_wait(BLK_RW_ASYNC, HZ/10);
> }
>
> However, I think this is a secondary effect to the failure of shrinkers
> to do their work. If slabs were being shrunk, one would expect us to
> be getting over the min watermark.
>
> > 4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock
> > |
> > --- do_raw_spin_lock
> > |
> > |--77.49%-- _raw_spin_lock
> > | |
> > | |--51.85%-- mb_cache_shrink_fn
> > | | shrink_slab
> > | | kswapd
> > | | kthread
> > | | kernel_thread_helper
> > | |
> > | --48.15%-- mem_cgroup_soft_limit_reclaim
> > | kswapd
> > | kthread
> > | kernel_thread_helper
> > |
>
> Way hey, cgroups are also in the mix. How jolly.
>
> Is systemd a common element of the machines hitting this bug by any
> chance?
Well, yes, the bug report is against FC15, which needs cgroups for
systemd.
> The remaining traces seem to be follow-on damage related to the three
> issues of "shrinkers are bust in some manner" causing "we are not
> getting over the min watermark" and as a side-show "we are spending lots
> of time doing something unspecified but unhelpful in cgroups".
Heh, well find a way for me to verify this: I can't turn off cgroups
because systemd then won't work and the machine won't boot ...
James
On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote:
> > On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote:
> > > # Events: 6K cycles
> > > #
> > > # Overhead Command Shared Object Symbol
> > > # ........ ........... ................... .......................................
> > > #
> > > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
> > > |
> > > --- shrink_slab
> > > |
> > > |--99.91%-- kswapd
> > > | kthread
> > > | kernel_thread_helper
> > > --0.09%-- [...]
> > >
> >
> > Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use
> > compaction instead of lumpy reclaim" is related unless we are seeing
> > two problems that happen to manifest in a similar manner.
> >
> > However, there were a number of changes made to dcache in particular
> > for 2.6.38. Specifically thinks like dentry_kill use trylock and is
> > happy to loop around if it fails to acquire anything. See things like
> > this for example;
>
> OK, so for this, I tried a 2.6.37 kernel. It doesn't work very well,
> networking is hosed for no reason I can see (probably systemd / cgroups
> problems).
>
> However, it runs enough for me to say that the tar proceeds to
> completion in a non-PREEMPT kernel. (I tried several times for good
> measure). That makes this definitely a regression of some sort, but it
> doesn't definitively identify the dcache code ... it could be an ext4
> bug that got introduced in 2.6.38 either.
>
True, it could be any shrinker and dcache is just a guess.
> > <SNIP>
> >
> > Way hey, cgroups are also in the mix. How jolly.
> >
> > Is systemd a common element of the machines hitting this bug by any
> > chance?
>
> Well, yes, the bug report is against FC15, which needs cgroups for
> systemd.
>
Ok although we do not have direct evidence that it's the problem yet. A
broken shrinker could just mean we are also trying to aggressively
reclaim in cgroups.
> > The remaining traces seem to be follow-on damage related to the three
> > issues of "shrinkers are bust in some manner" causing "we are not
> > getting over the min watermark" and as a side-show "we are spending lots
> > of time doing something unspecified but unhelpful in cgroups".
>
> Heh, well find a way for me to verify this: I can't turn off cgroups
> because systemd then won't work and the machine won't boot ...
>
Same testcase, same kernel but a distro that is not using systemd to
verify if cgroups are the problem. Not ideal I know. When I'm back
online Tuesday, I'll try reproducing this on a !Fedora distribution. In
the meantime, the following untested hatchet job might spit out
which shrinker we are getting stuck in. It is also breaking out of
the shrink_slab loop so it'd even be interesting to see if the bug
is mitigated in any way.
diff --git a/mm/vmscan.c b/mm/vmscan.c
index c74a501..ed99104 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -225,6 +225,7 @@ unsigned long shrink_slab(unsigned long scanned, gfp_t gfp_mask,
{
struct shrinker *shrinker;
unsigned long ret = 0;
+ unsigned long shrink_expired = jiffies + HZ;
if (scanned == 0)
scanned = SWAP_CLUSTER_MAX;
@@ -270,6 +271,14 @@ unsigned long shrink_slab(unsigned long scanned, gfp_t gfp_mask,
gfp_mask);
if (shrink_ret == -1)
break;
+ if (time_after(jiffies, shrink_expired)) {
+ printk(KERN_WARNING "Slab shrinker %p gone mental"
+ " comm=%s nr=%ld\n",
+ shrinker->shrink,
+ current->comm,
+ shrinker->nr);
+ break;
+ }
if (shrink_ret < nr_before)
ret += nr_before - shrink_ret;
count_vm_events(SLABS_SCANNED, this_scan);
On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote:
> > > Way hey, cgroups are also in the mix. How jolly.
> > >
> > > Is systemd a common element of the machines hitting this bug by any
> > > chance?
> >
> > Well, yes, the bug report is against FC15, which needs cgroups for
> > systemd.
> >
>
> Ok although we do not have direct evidence that it's the problem yet. A
> broken shrinker could just mean we are also trying to aggressively
> reclaim in cgroups.
>
> > > The remaining traces seem to be follow-on damage related to the three
> > > issues of "shrinkers are bust in some manner" causing "we are not
> > > getting over the min watermark" and as a side-show "we are spending lots
> > > of time doing something unspecified but unhelpful in cgroups".
> >
> > Heh, well find a way for me to verify this: I can't turn off cgroups
> > because systemd then won't work and the machine won't boot ...
> >
>
> Same testcase, same kernel but a distro that is not using systemd to
> verify if cgroups are the problem. Not ideal I know. When I'm back
> online Tuesday, I'll try reproducing this on a !Fedora distribution. In
> the meantime, the following untested hatchet job might spit out
> which shrinker we are getting stuck in. It is also breaking out of
> the shrink_slab loop so it'd even be interesting to see if the bug
> is mitigated in any way.
Actually, talking to Chris, I think I can get the system up using
init=/bin/bash without systemd, so I can try the no cgroup config.
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index c74a501..ed99104 100644
In the mean time, this patch produces:
(that's nothing ... apparently the trace doesn't activate when kswapd
goes mad).
James
On Thu, Apr 28, 2011 at 02:59:27PM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote:
> > On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote:
> > > > Way hey, cgroups are also in the mix. How jolly.
> > > >
> > > > Is systemd a common element of the machines hitting this bug by any
> > > > chance?
> > >
> > > Well, yes, the bug report is against FC15, which needs cgroups for
> > > systemd.
> > >
> >
> > Ok although we do not have direct evidence that it's the problem yet. A
> > broken shrinker could just mean we are also trying to aggressively
> > reclaim in cgroups.
> >
> > > > The remaining traces seem to be follow-on damage related to the three
> > > > issues of "shrinkers are bust in some manner" causing "we are not
> > > > getting over the min watermark" and as a side-show "we are spending lots
> > > > of time doing something unspecified but unhelpful in cgroups".
> > >
> > > Heh, well find a way for me to verify this: I can't turn off cgroups
> > > because systemd then won't work and the machine won't boot ...
> > >
> >
> > Same testcase, same kernel but a distro that is not using systemd to
> > verify if cgroups are the problem. Not ideal I know. When I'm back
> > online Tuesday, I'll try reproducing this on a !Fedora distribution. In
> > the meantime, the following untested hatchet job might spit out
> > which shrinker we are getting stuck in. It is also breaking out of
> > the shrink_slab loop so it'd even be interesting to see if the bug
> > is mitigated in any way.
>
> Actually, talking to Chris, I think I can get the system up using
> init=/bin/bash without systemd, so I can try the no cgroup config.
>
> > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > index c74a501..ed99104 100644
>
> In the mean time, this patch produces:
>
> (that's nothing ... apparently the trace doesn't activate when kswapd
> goes mad).
>
Or is looping there for shorter than we expect. HZ/10?
--
Mel Gorman
SUSE Labs
On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote:
> Actually, talking to Chris, I think I can get the system up using
> init=/bin/bash without systemd, so I can try the no cgroup config.
OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back
runs of untar without locking or getting kswapd pegged, so I'm pretty
certain this is cgroups related. The next steps are to turn cgroups
back on but try disabling the memory and IO controllers.
James
On Thu 28-04-11 15:58:21, Colin Ian King wrote:
> On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote:
> > On Thu 28-04-11 16:25:51, Jan Kara wrote:
> > > On Thu 28-04-11 15:01:22, Colin Ian King wrote:
> > > >
> > > > > Could you post the soft lockups you're seeing?
> > > >
> > > > As requested, attached
> > > Hum, what keeps puzzling me is that in all the cases of hangs I've seen
> > > so far, we are stuck waiting for IO to finish for a long time - e.g. in the
> > > traces below kjournald waits for PageWriteback bit to get cleared. Also we
> > > are stuck waiting for page locks which might be because those pages are
> > > being read in? All in all it seems that the IO is just incredibly slow.
> > >
> > > But it's not clear to me what pushes us into that situation (especially
> > > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
> > > underlying blocks are not already allocated.
> > Hmm, maybe because the system is under memory pressure (and kswapd is not
> > able to get rid of dirty pages), we page out clean pages. Thus also pages
> > of executables which need to be paged in soon anyway thus putting heavy
> > read load on the system which makes writes crawl? I'm not sure why
> > compaction should make this any worse but maybe it can.
> >
> > James, Colin, can you capture output of 'vmstat 1' while you do the
> > copying? Thanks.
>
> Attached.
Thanks. So I there are a few interesting points in the vmstat output:
For first 30 seconds, we are happily copying data - relatively steady read
throughput (about 20-40 MB/s) and occasional peak from flusher thread
flushing dirty data. During this time free memory drops from about 1.4 GB
to about 22!!! MB - mm seems to like to really use the machine ;). Then
things get interesting:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 1 0 83372 5228 1450776 0 0 39684 90132 450 918 0 4 74 22
0 1 0 22416 5228 1509864 0 0 29452 48492 403 869 1 2 80 18
2 0 0 20056 5384 1513996 0 0 2248 2116 434 1191 4 4 71 21
0 1 0 19800 5932 1514600 0 0 644 104 454 1166 8 3 64 24
1 0 0 21848 5940 1515244 0 0 292 380 468 1775 15 6 75 3
1 0 0 20936 5940 1515876 0 0 296 296 496 1846 18 8 74 0
1 0 0 17792 5940 1516580 0 0 356 356 484 1862 23 8 69 0
1 0 0 17544 5940 1517364 0 0 412 412 482 1812 16 7 77 0
4 0 0 18148 5948 1517968 0 0 288 344 436 1749 19 9 69 3
0 2 220 137528 1616 1402468 0 220 74708 2164 849 1806 3 6 63 28
0 3 224 36184 1628 1499648 0 4 50820 86204 532 1272 0 4 64 32
0 2 19680 53688 1628 1484388 32 19456 6080 62972 242 287 0 2 63 34
0 2 36928 1407432 1356 150980 0 17252 1564 17276 368 764 1 3 73 22
So when free memory reached about 20 MB, both read and write activity
almost stalled for 7 s (probably everybody waits for free memory). Then
mm manages to free 100 MB from page cache, things move on for two seconds,
then we swap out! about 36 MB and page reclaim also finally decides it
maybe has too much of page cache and reaps most of it (1.3 GB in one go).
Then things get going again, although there are still occasional stalls
such as this (about 30s later):
1 3 36688 753192 1208 792344 0 0 35948 32768 435 6625 0 6 61 33
0 2 36668 754996 1344 792760 0 0 252 58736 2832 16239 0 1 60 39
0 2 36668 750132 1388 796688 0 0 2508 1524 325 959 1 3 68 28
1 0 36668 751160 1400 797968 0 0 620 620 460 1470 6 6 50 38
1 0 36668 750516 1400 798520 0 0 300 300 412 1764 17 8 75 1
1 0 36668 750648 1408 799108 0 0 280 340 423 1816 18 6 73 3
1 0 36668 748856 1408 799752 0 0 336 328 409 1788 18 8 75 0
1 0 36668 748120 1416 800604 0 0 428 452 407 1723 14 10 75 2
1 0 36668 750048 1416 801176 0 0 296 296 405 1779 18 7 75 1
0 1 36668 650428 1420 897252 0 0 48100 556 658 1718 10 3 71 15
0 2 36668 505444 1424 1037012 0 0 69888 90272 686 1491 1 4 68 27
0 1 36668 479264 1428 1063372 0 0 12984 40896 324 674 1 1 76 23
...
I'm not sure what we were blocked on here since there is still plenty of
free memory (750 MB). These stalls repeat once in a while but things go on.
Then at about 350s, things just stop:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 1 75280 73564 1844 1503848 0 0 43396 81976 627 1061 0 25 42 32
3 3 75280 73344 1852 1504256 0 0 256 20 240 149 0 26 25 49
3 3 75280 73344 1852 1504268 0 0 0 0 265 140 0 29 13 58
3 3 75280 73468 1852 1504232 0 0 0 0 265 132 0 22 34 44
3 3 75280 73468 1852 1504232 0 0 0 0 339 283 0 25 26 49
3 3 75280 73468 1852 1504232 0 0 0 0 362 327 0 25 25 50
3 3 75280 73468 1852 1504232 0 0 0 0 317 320 0 26 25 49
3 3 75280 73468 1852 1504232 0 0 0 0 361 343 0 26 25 50
and nothing really happens for 150 s, except more and more tasks blocking
in D state (second column).
3 6 75272 73416 1872 1503872 0 0 0 0 445 700 0 25 25 50
0 7 75264 67940 1884 1509008 64 0 5056 16 481 876 0 22 23 55
Then suddently things get going again:
0 2 75104 76808 1892 1502552 352 0 14292 40456 459 14865 0 2 39 59
0 1 75104 75704 1900 1503412 0 0 820 32 405 788 1 1 72 27
1 0 75104 76512 1904 1505576 0 0 1068 1072 454 1586 8 7 74 11
I guess this 150 s stall is when kernel barfs the "task blocked for more
than 30 seconds" messages. And from the traces we know that everyone is
waiting for PageWriteback or page lock during this time. Also James's vmstat
report shows that IO is stalled when kswapd is spinning. Really strange.
James in the meantime identified that cgroups are somehow involved. Are you
using systemd by any chance? Maybe cgroup IO throttling screws us?
Honza
> > > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
> > > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
> > > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
> > > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
> > > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
> > > [ 287.088400] Call Trace:
> > > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
> > > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
> > > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
> > > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
> > > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
> > > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
> > > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
> > > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
> > > [ 287.088484] Call Trace:
> > > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
> > > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
> > > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
> > > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
> > > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
> > > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
> > > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
> > > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
> > > [ 287.088577] Call Trace:
> > > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
> > > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
> > > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
> > > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
> > > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
> > > [ 287.088647] Call Trace:
> > > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
> > > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
> > > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
> > > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
> > > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
> > > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
> > > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
> > > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
> > > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
> > > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
> > > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
> > > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
> > > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
> > > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
> > > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
> > > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
> > > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
> > > [ 287.088723] Call Trace:
> > > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
> > > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
> > > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
> > > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
> > > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
> > > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
> > > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
> > > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
> > > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
> > > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
> > > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
> > > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
> > > --
> > > Jan Kara <[email protected]>
> > > SUSE Labs, CR
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> > > the body of a message to [email protected]
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Jan Kara <[email protected]>
SUSE Labs, CR
On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote:
> > Actually, talking to Chris, I think I can get the system up using
> > init=/bin/bash without systemd, so I can try the no cgroup config.
>
> OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back
> runs of untar without locking or getting kswapd pegged, so I'm pretty
> certain this is cgroups related. The next steps are to turn cgroups
> back on but try disabling the memory and IO controllers.
I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR.
The results are curious: the tar does complete (I've done three back to
back). However, I did get one soft lockup in kswapd (below). But the
system recovers instead of halting I/O and hanging like it did
previously.
The soft lockup is in shrink_slab, so perhaps it's a combination of slab
shrinker and cgroup memory controller issues?
James
---
[ 670.823843] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
[ 670.825472] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel btusb snd_hda_codec snd_hwdep iwlagn snd_seq mac80211 bluetooth snd_seq_device uvcvideo snd_pcm cfg80211 wmi microcode e1000e videodev xhci_hcd rfkill snd_timer iTCO_wdt v4l2_compat_ioctl32 iTCO_vendor_support pcspkr i2c_i801 snd soundcore snd_page_alloc joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 670.830864] CPU 2
[ 670.830881] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel btusb snd_hda_codec snd_hwdep iwlagn snd_seq mac80211 bluetooth snd_seq_device uvcvideo snd_pcm cfg80211 wmi microcode e1000e videodev xhci_hcd rfkill snd_timer iTCO_wdt v4l2_compat_ioctl32 iTCO_vendor_support pcspkr i2c_i801 snd soundcore snd_page_alloc joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 670.838385]
[ 670.840289] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #3 LENOVO 4170CTO/4170CTO
[ 670.842193] RIP: 0010:[<ffffffff810e07cb>] [<ffffffff810e07cb>] shrink_slab+0x86/0x166
[ 670.844063] RSP: 0018:ffff88006eea5da0 EFLAGS: 00000206
[ 670.845881] RAX: 0000000000000000 RBX: ffff88006eea5de0 RCX: 0000000000000002
[ 670.847652] RDX: 0000000000000000 RSI: ffff88006eea5d60 RDI: ffff88006eea5d60
[ 670.849394] RBP: ffff88006eea5de0 R08: 000000000000000c R09: 0000000000000000
[ 670.851091] R10: 0000000000000001 R11: 000000000000005f R12: ffffffff8147b50e
[ 670.852733] R13: ffff8801005e6e00 R14: 0000000000000010 R15: 0000000000017fb6
[ 670.854351] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 670.855968] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 670.857555] CR2: 00000037d90ae040 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 670.859138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 670.860720] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 670.862320] Process kswapd0 (pid: 46, threadinfo ffff88006eea4000, task ffff88006eeb0000)
[ 670.863932] Stack:
[ 670.865477] 0000000000000001 0000000000000080 ffff880000000002 ffff8801005e6e00
[ 670.867023] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c
[ 670.868558] ffff88006eea5ee0 ffffffff810e308c 0000000000000003 ffff88006eeb0000
[ 670.870120] Call Trace:
[ 670.871652] [<ffffffff810e308c>] kswapd+0x4f0/0x774
[ 670.873218] [<ffffffff810e2b9c>] ? try_to_free_pages+0xe5/0xe5
[ 670.874786] [<ffffffff8106ce57>] kthread+0x84/0x8c
[ 670.876327] [<ffffffff8147bc64>] kernel_thread_helper+0x4/0x10
[ 670.877871] [<ffffffff8106cdd3>] ? kthread_worker_fn+0x148/0x148
[ 670.879403] [<ffffffff8147bc60>] ? gs_change+0x13/0x13
[ 670.880932] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7
[ 670.881086] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7
[ 670.884285] Call Trace:
[ 670.885884] [<ffffffff810e308c>] kswapd+0x4f0/0x774
[ 670.887462] [<ffffffff810e2b9c>] ? try_to_free_pages+0xe5/0xe5
[ 670.889031] [<ffffffff8106ce57>] kthread+0x84/0x8c
[ 670.890578] [<ffffffff8147bc64>] kernel_thread_helper+0x4/0x10
[ 670.892130] [<ffffffff8106cdd3>] ? kthread_worker_fn+0x148/0x148
[ 670.893653] [<ffffffff8147bc60>] ? gs_change+0x13/0x13
On Fri, 2011-04-29 at 00:40 +0200, Jan Kara wrote:
> James in the meantime identified that cgroups are somehow involved.
> Are you
> using systemd by any chance? Maybe cgroup IO throttling screws us?
Yes, this is a FC15 system with systemd as the default.
See related post: it looks like disabling the memory controller is what
makes this go away.
James
Sorry for the top-posting!
But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
First, I have seen negative effects on my UP-system was when playing
with linux-next [2].
It was not clear what the origin was and the the side-effects were
somehow "bizarre".
The issue could be easily reproduced by tar-ing the kernel build-dir
to an external USB-hdd.
The issue kept RCU and TIP folks really busy.
Before stepping 4 weeks in the dark, give it a try and let me know in
case of success.
For the systemd/cgroups part of the discussion (yes, I followed this
thread in parallel):
The patch from [4] might be interesting (untested here).
Hope this helps you.
Have fun!
- Sedat -
[1] http://git.us.kernel.org/?p=linux/kernel/git/tip/linux-2.6-tip.git;a=commit;h=ce31332d3c77532d6ea97ddcb475a2b02dd358b4
[2] http://lkml.org/lkml/2011/3/25/97
[3] http://lkml.org/lkml/2011/4/28/444
[4] https://patchwork.kernel.org/patch/738921/
On Wed, Apr 27, 2011 at 7:21 PM, James Bottomley
<[email protected]> wrote:
> On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
>> Ok, I'd try turning it on so we catch the sleeping with a spinlock held
>> case better.
>
> Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
>
> This is the trace with sysrq-l and sysrq-w
>
> The repro this time doesn't have a soft lockup, just the tar is hung and
> one of my CPUs is in 99% system.
>
> James
>
> ---
>
>
> [ Â 453.351255] SysRq : Show backtrace of all active CPUs
> [ Â 453.352601] sending NMI to all CPUs:
> [ Â 453.353849] NMI backtrace for cpu 3
> [ Â 453.355545] CPU 3
> [ Â 453.355560] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ Â 453.363188]
> [ Â 453.365162] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ Â 453.367133] RIP: 0010:[<ffffffff8147af8c>] Â [<ffffffff8147af8c>] mutex_trylock+0x16/0x38
> [ Â 453.369122] RSP: 0018:ffff88006dfc1d40 Â EFLAGS: 00000246
> [ Â 453.371098] RAX: 0000000000000001 RBX: ffff880037de15f0 RCX: 0000000000000001
> [ Â 453.373099] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037875820
> [ Â 453.375097] RBP: ffff88006dfc1d40 R08: 0000000000000000 R09: 00000000000074ad
> [ Â 453.377079] R10: 0000000000000002 R11: ffffffff81a44e50 R12: 0000000000000000
> [ Â 453.379052] R13: 0000000000000000 R14: ffff880037875800 R15: ffff880037875820
> [ Â 453.381015] FS: Â 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
> [ Â 453.382985] CS: Â 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ Â 453.384944] CR2: 00007fbf8ea8d090 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ Â 453.386920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ Â 453.388887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ Â 453.390789] Process kswapd0 (pid: 46, threadinfo ffff88006dfc0000, task ffff88006dfb8000)
> [ Â 453.392649] Stack:
> [ Â 453.394473] Â ffff88006dfc1d90 ffffffffa007ff52 ffff88006dfc1d90 ffffffff811613b5
> [ Â 453.396337] Â ffff88006dfc1d60 ffff880037de15f0 0000000000000000 0000000000000000
> [ Â 453.398174] Â 00000000000000d0 000000000006366c ffff88006dfc1de0 ffffffff810e1f89
> [ Â 453.400015] Call Trace:
> [ Â 453.401845] Â [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
> [ Â 453.403702] Â [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
> [ Â 453.405499] Â [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
> [ Â 453.407234] Â [<ffffffff810e4bcc>] kswapd+0x533/0x798
> [ Â 453.408952] Â [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ Â 453.410690] Â [<ffffffff8106e157>] kthread+0x84/0x8c
> [ Â 453.412432] Â [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ Â 453.414187] Â [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ Â 453.415945] Â [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ Â 453.417671] Code: 00 48 c7 47 18 00 00 00 00 f0 ff 07 7f 05 e8 ed 02 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 b9 01 00 00 00 31 d2 89 c8 f0 0f b1 17
> [ Â 453.417905] Â c1 31 c0 ff c9 75 18 65 48 8b 04 25 c8 cc 00 00 48 2d d8 1f
> [ Â 453.421534] Call Trace:
> [ Â 453.423337] Â [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
> [ Â 453.425172] Â [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
> [ Â 453.426997] Â [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
> [ Â 453.428818] Â [<ffffffff810e4bcc>] kswapd+0x533/0x798
> [ Â 453.430639] Â [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ Â 453.432485] Â [<ffffffff8106e157>] kthread+0x84/0x8c
> [ Â 453.434298] Â [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ Â 453.436112] Â [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ Â 453.437894] Â [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ Â 453.439654] NMI backtrace for cpu 2
> [ Â 453.441508] CPU 2
> [ Â 453.441525] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ Â 453.449517]
> [ Â 453.451516] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ Â 453.453704] RIP: 0010:[<ffffffff81275d36>] Â [<ffffffff81275d36>] intel_idle+0xaa/0x100
> [ Â 453.455772] RSP: 0018:ffff8800715c9e68 Â EFLAGS: 00000046
> [ Â 453.457827] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
> [ Â 453.459903] RDX: 0000000000000000 RSI: ffff8800715c9fd8 RDI: ffffffff81a0e640
> [ Â 453.461999] RBP: ffff8800715c9eb8 R08: 000000000000006d R09: 00000000000003e5
> [ Â 453.464039] R10: ffffffff00000002 R11: ffff880100293b40 R12: 0000000000000030
> [ Â 453.466013] R13: 12187898d4512537 R14: 0000000000000004 R15: 0000000000000002
> [ Â 453.467918] FS: Â 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
> [ Â 453.469797] CS: Â 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ Â 453.471644] CR2: 0000000000452630 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ Â 453.473512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ Â 453.475356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ Â 453.477185] Process kworker/0:1 (pid: 0, threadinfo ffff8800715c8000, task ffff8800715a1700)
> [ Â 453.479024] Stack:
> [ Â 453.480855] Â ffff8800715c9e88 ffffffff810731c0 ffff880100291290 0000000000011290
> [ Â 453.482739] Â ffff8800715c9eb8 000000028139c97a ffffe8ffffc80170 ffffe8ffffc80170
> [ Â 453.484631] Â ffffe8ffffc80300 0000000000000000 ffff8800715c9ef8 ffffffff8139b868
> [ Â 453.486518] Call Trace:
> [ Â 453.488365] Â [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ Â 453.490226] Â [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ Â 453.492096] Â [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ Â 453.493947] Â [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ Â 453.495791] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
> [ Â 453.499958] Call Trace:
> [ Â 453.501895] Â [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ Â 453.503837] Â [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ Â 453.505775] Â [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ Â 453.507687] Â [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ Â 453.509598] NMI backtrace for cpu 1
> [ Â 453.511390] CPU 1
> [ Â 453.511405] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ Â 453.519288]
> [ Â 453.521289] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ Â 453.523353] RIP: 0010:[<ffffffff81275d36>] Â [<ffffffff81275d36>] intel_idle+0xaa/0x100
> [ Â 453.525377] RSP: 0018:ffff880071587e68 Â EFLAGS: 00000046
> [ Â 453.527353] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
> [ Â 453.529332] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640
> [ Â 453.531303] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5
> [ Â 453.533276] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010
> [ Â 453.535249] R13: 12187898d4512ee3 R14: 0000000000000002 R15: 0000000000000001
> [ Â 453.537229] FS: Â 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
> [ Â 453.539224] CS: Â 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ Â 453.541210] CR2: 00000037d9071fa0 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ Â 453.543220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ Â 453.545233] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ Â 453.547239] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700)
> [ Â 453.549274] Stack:
> [ Â 453.551289] Â ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290
> [ Â 453.553301] Â ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170
> [ Â 453.555257] Â ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868
> [ Â 453.557156] Call Trace:
> [ Â 453.558980] Â [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ Â 453.560817] Â [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ Â 453.562635] Â [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ Â 453.564438] Â [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ Â 453.566227] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
> [ Â 453.570292] Call Trace:
> [ Â 453.572196] Â [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ Â 453.574117] Â [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ Â 453.576031] Â [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ Â 453.577924] Â [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ Â 453.579811] NMI backtrace for cpu 0
> [ Â 453.581279] CPU 0
> [ Â 453.581289] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ Â 453.587576]
> [ Â 453.589160] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ Â 453.590777] RIP: 0010:[<ffffffff8100f0fa>] Â [<ffffffff8100f0fa>] native_read_tsc+0x1/0x14
> [ Â 453.592390] RSP: 0018:ffff880100203b98 Â EFLAGS: 00000883
> [ Â 453.594001] RAX: 00000000f9ab6980 RBX: 0000000000002710 RCX: 0000000000000040
> [ Â 453.595624] RDX: 000000000026066c RSI: 0000000000000100 RDI: 000000000026066d
> [ Â 453.597250] RBP: ffff880100203ba8 R08: 000000008b000052 R09: 0000000000000000
> [ Â 453.598872] R10: 0000000000000000 R11: 0000000000000003 R12: 000000000026066d
> [ Â 453.600437] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000001
> [ Â 453.601924] FS: Â 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000
> [ Â 453.603434] CS: Â 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ Â 453.604942] CR2: 0000000000440360 CR3: 0000000001a03000 CR4: 00000000000406f0
> [ Â 453.606460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ Â 453.607981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ Â 453.609597] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020)
> [ Â 453.611134] Stack:
> [ Â 453.612655] Â ffff880100203ba8 ffffffff81232dfe ffff880100203bd8 ffffffff81232ecb
> [ Â 453.614176] Â 0000000000002710 0000000000000008 000000000000006c 0000000000000002
> [ Â 453.615656] Â ffff880100203be8 ffffffff81232e49 ffff880100203bf8 ffffffff81232e77
> [ Â 453.617101] Call Trace:
> [ Â 453.618492] Â <IRQ>
> [ Â 453.619874] Â [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
> [ Â 453.621251] Â [<ffffffff81232ecb>] delay_tsc+0x27/0x74
> [ Â 453.622613] Â [<ffffffff81232e49>] __delay+0xf/0x11
> [ Â 453.623973] Â [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
> [ Â 453.625322] Â [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
> [ Â 453.626673] Â [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
> [ Â 453.628027] Â [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
> [ Â 453.629372] Â [<ffffffff812be514>] sysrq_filter+0x112/0x16e
> [ Â 453.630706] Â [<ffffffff81365764>] input_pass_event+0x94/0xcc
> [ Â 453.632028] Â [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
> [ Â 453.633342] Â [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
> [ Â 453.634655] Â [<ffffffff81366cf2>] input_event+0x69/0x87
> [ Â 453.635978] Â [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
> [ Â 453.637283] Â [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
> [ Â 453.638563] Â [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
> [ Â 453.639830] Â [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
> [ Â 453.641092] Â [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
> [ Â 453.642354] Â [<ffffffff810a9746>] handle_irq_event+0x38/0x56
> [ Â 453.643598] Â [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
> [ Â 453.644832] Â [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
> [ Â 453.646069] Â [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
> [ Â 453.647299] Â [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
> [ Â 453.648527] Â [<ffffffff8147c253>] common_interrupt+0x13/0x13
> [ Â 453.649695] Â <EOI>
> [ Â 453.650794] Â [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
> [ Â 453.651911] Â [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
> [ Â 453.653025] Â [<ffffffff81275d46>] ? intel_idle+0xba/0x100
> [ Â 453.654129] Â [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ Â 453.655231] Â [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ Â 453.656329] Â [<ffffffff8145a91e>] rest_init+0x72/0x74
> [ Â 453.657417] Â [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
> [ Â 453.658514] Â [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
> [ Â 453.659576] Â [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
> [ Â 453.660589] Â [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
> [ Â 453.661559] Code: 21 00 00 e8 74 3d 22 00 5d c3 90 90 90 55 40 88 f8 48 89 e5 e6 70 e4 71 5d c3 55 40 88 f0 48 89 e5 e6 70 40 88 f8 e6 71 5d c3 55
> [ Â 453.661721] Â 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 55 b2
> [ Â 453.663728] Call Trace:
> [ Â 453.664713] Â <IRQ> Â [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
> [ Â 453.665710] Â [<ffffffff81232ecb>] delay_tsc+0x27/0x74
> [ Â 453.666698] Â [<ffffffff81232e49>] __delay+0xf/0x11
> [ Â 453.667686] Â [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
> [ Â 453.668675] Â [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
> [ Â 453.669676] Â [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
> [ Â 453.670681] Â [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
> [ Â 453.671676] Â [<ffffffff812be514>] sysrq_filter+0x112/0x16e
> [ Â 453.672662] Â [<ffffffff81365764>] input_pass_event+0x94/0xcc
> [ Â 453.673646] Â [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
> [ Â 453.674627] Â [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
> [ Â 453.675614] Â [<ffffffff81366cf2>] input_event+0x69/0x87
> [ Â 453.676587] Â [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
> [ Â 453.677550] Â [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
> [ Â 453.678498] Â [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
> [ Â 453.679431] Â [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
> [ Â 453.680351] Â [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
> [ Â 453.681270] Â [<ffffffff810a9746>] handle_irq_event+0x38/0x56
> [ Â 453.682186] Â [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
> [ Â 453.683100] Â [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
> [ Â 453.684014] Â [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
> [ Â 453.684924] Â [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
> [ Â 453.685831] Â [<ffffffff8147c253>] common_interrupt+0x13/0x13
> [ Â 453.686740] Â <EOI> Â [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
> [ Â 453.687655] Â [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
> [ Â 453.688573] Â [<ffffffff81275d46>] ? intel_idle+0xba/0x100
> [ Â 453.689480] Â [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ Â 453.690387] Â [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ Â 453.691293] Â [<ffffffff8145a91e>] rest_init+0x72/0x74
> [ Â 453.692195] Â [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
> [ Â 453.693095] Â [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
> [ Â 453.693996] Â [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
> [ Â 453.694905] Â [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
> [ Â 454.680802] SysRq : Show Blocked State
> [  454.683427]  task             PC stack  pid father
> [  454.686058] systemd     D 0000000000000000   0   1    0 0x00000000
> [ Â 454.688752] Â ffff8801003bdcd8 0000000000000082 ffff8801003bdc88 ffffffff00000000
> [ Â 454.691491] Â ffff880100370000 ffff8801003bdfd8 ffff8801003bdfd8 0000000000013b40
> [ Â 454.694228] Â ffff8800715e1700 ffff880100370000 ffff8801003bdcb8 00000001811329db
> [ Â 454.696969] Call Trace:
> [ Â 454.699683] Â [<ffffffff8147ac4b>] schedule_timeout+0x34/0xde
> [ Â 454.702447] Â [<ffffffff810ad618>] ? __call_rcu+0x123/0x12c
> [ Â 454.705184] Â [<ffffffff810ad64d>] ? call_rcu_sched+0x15/0x17
> [ Â 454.707906] Â [<ffffffff8147aa14>] wait_for_common+0xac/0x101
> [ Â 454.710656] Â [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226
> [ Â 454.713445] Â [<ffffffff8147ab1d>] wait_for_completion+0x1d/0x1f
> [ Â 454.716199] Â [<ffffffff810ada5a>] synchronize_sched+0x5a/0x5c
> [ Â 454.718972] Â [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43
> [ Â 454.721766] Â [<ffffffff81090d05>] cgroup_diput+0x37/0xe3
> [ Â 454.724562] Â [<ffffffff81090cce>] ? parse_cgroupfs_options+0x353/0x353
> [ Â 454.727328] Â [<ffffffff8112fc79>] dentry_kill+0xfa/0x121
> [ Â 454.730103] Â [<ffffffff81130189>] dput+0xdd/0xea
> [ Â 454.732866] Â [<ffffffff8112aa68>] do_rmdir+0xc6/0xfe
> [ Â 454.735481] Â [<ffffffff8111dc78>] ? filp_close+0x6e/0x7a
> [ Â 454.737990] Â [<ffffffff8112b32f>] sys_rmdir+0x16/0x18
> [ Â 454.740469] Â [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ Â 454.742935] flush-253:2 Â Â D 0000000000000000 Â Â 0 Â 793 Â Â Â 2 0x00000000
> [ Â 454.745425] Â ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> [ Â 454.747955] Â ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> [ Â 454.750506] Â ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> [ Â 454.753048] Call Trace:
> [ Â 454.755537] Â [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ Â 454.758071] Â [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ Â 454.760644] Â [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ Â 454.763206] Â [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> [ Â 454.765770] Â [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ Â 454.768353] Â [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> [ Â 454.770942] Â [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> [ Â 454.773501] Â [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> [ Â 454.776082] Â [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> [ Â 454.778711] Â [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> [ Â 454.781323] Â [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> [ Â 454.783894] Â [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> [ Â 454.786491] Â [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> [ Â 454.789090] Â [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> [ Â 454.791703] Â [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
> [ Â 454.794334] Â [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
> [ Â 454.796962] Â [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
> [ Â 454.799582] Â [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
> [ Â 454.802204] Â [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
> [ Â 454.804803] Â [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
> [ Â 454.807427] Â [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
> [ Â 454.810077] Â [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
> [ Â 454.812776] Â [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
> [ Â 454.815464] Â [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
> [ Â 454.818129] Â [<ffffffff8106e157>] kthread+0x84/0x8c
> [ Â 454.820808] Â [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ Â 454.823452] Â [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ Â 454.826103] Â [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ Â 454.828711] jbd2/dm-2-8 Â Â D 0000000000000000 Â Â 0 Â 799 Â Â Â 2 0x00000000
> [ Â 454.831390] Â ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> [ Â 454.834094] Â ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> [ Â 454.836788] Â ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> [ Â 454.839453] Call Trace:
> [ Â 454.842098] Â [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ Â 454.844738] Â [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ Â 454.847303] Â [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ Â 454.849877] Â [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> [ Â 454.852469] Â [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> [ Â 454.855021] Â [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> [ Â 454.857583] Â [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ Â 454.860171] Â [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> [ Â 454.862744] Â [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> [ Â 454.865299] Â [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
> [ Â 454.867892] Â [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ Â 454.870496] Â [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ Â 454.873103] Â [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ Â 454.875690] Â [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ Â 454.878327] Â [<ffffffff8106e157>] kthread+0x84/0x8c
> [ Â 454.880961] Â [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ Â 454.883604] Â [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ Â 454.886262] Â [<ffffffff81483760>] ? gs_change+0x13/0x13
> [  454.888875] tar       D ffff88006e573af8   0  991   838 0x00000000
> [ Â 454.891546] Â ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282
> [ Â 454.894213] Â ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40
> [ Â 454.896889] Â ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3
> [ Â 454.899530] Call Trace:
> [ Â 454.902118] Â [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
> [ Â 454.904724] Â [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ Â 454.907344] Â [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ Â 454.909967] Â [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ Â 454.912574] Â [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ Â 454.915192] Â [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ Â 454.917819] Â [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ Â 454.920459] Â [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ Â 454.923070] Â [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ Â 454.925660] Â [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ Â 454.928247] Â [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e
> [ Â 454.930865] Â [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204
> [ Â 454.933454] Â [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a
> [ Â 454.936062] Â [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
> [ Â 454.938648] Â [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
> [ Â 454.941288] Â [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
> [ Â 454.943909] Â [<ffffffff8111edab>] do_sync_write+0xbf/0xff
> [ Â 454.946501] Â [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
> [ Â 454.949114] Â [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
> [ Â 454.951736] Â [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ Â 454.954349] Â [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ Â 454.956943] Â [<ffffffff8111f421>] vfs_write+0xac/0xf3
> [ Â 454.959530] Â [<ffffffff8111f610>] sys_write+0x4a/0x6e
> [ Â 454.962129] Â [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ Â 454.964732] dhclient-script D 0000000000000000 Â Â 0 Â 2856 Â 2855 0x00000000
> [ Â 454.967360] Â ffff88006e1f5b18 0000000000000082 ffff8800378da880 0000000000000000
> [ Â 454.970056] Â ffff88006deb1700 ffff88006e1f5fd8 ffff88006e1f5fd8 0000000000013b40
> [ Â 454.972706] Â ffff880071589700 ffff88006deb1700 ffff88006e1f5ad8 000000016e1f5ad8
> [ Â 454.975323] Call Trace:
> [ Â 454.977882] Â [<ffffffff810d5916>] ? sleep_on_page+0x12/0x12
> [ Â 454.980477] Â [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ Â 454.983063] Â [<ffffffff810d5924>] sleep_on_page_killable+0xe/0x3b
> [ Â 454.985622] Â [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
> [ Â 454.988182] Â [<ffffffff810d5819>] __lock_page_killable+0x66/0x68
> [ Â 454.990785] Â [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ Â 454.993436] Â [<ffffffff810d5859>] lock_page_killable+0x3e/0x43
> [ Â 454.996099] Â [<ffffffff810d71ea>] generic_file_aio_read+0x463/0x640
> [ Â 454.998730] Â [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
> [ Â 455.001383] Â [<ffffffff811ebc34>] ? avc_has_perm+0x51/0x63
> [ Â 455.004012] Â [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ Â 455.006644] Â [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ Â 455.009289] Â [<ffffffff8111f511>] vfs_read+0xa9/0xf0
> [ Â 455.011917] Â [<ffffffff811233eb>] kernel_read+0x41/0x4f
> [ Â 455.014511] Â [<ffffffff811234dd>] prepare_binprm+0xe4/0xe8
> [ Â 455.017094] Â [<ffffffff81124d40>] do_execve+0x114/0x277
> [ Â 455.019678] Â [<ffffffff8100ff91>] sys_execve+0x43/0x5a
> [ Â 455.022281] Â [<ffffffff81482a9c>] stub_execve+0x6c/0xc0
> [ Â 455.024888] Sched Debug Version: v0.10, 2.6.39-rc4+ #1
> [  455.027487] ktime                  : 455879.226285
> [  455.030142] sched_clk                : 455024.886257
> [  455.032786] cpu_clk                 : 455024.886397
> [  455.035352] jiffies                 : 4295123185
> [  455.037904] sched_clock_stable            : 1
> [ Â 455.040413]
> [ Â 455.042892] sysctl_sched
> [  455.045306]  .sysctl_sched_latency           : 18.000000
> [  455.047775]  .sysctl_sched_min_granularity       : 2.250000
> [  455.050206]  .sysctl_sched_wakeup_granularity     : 3.000000
> [  455.052643]  .sysctl_sched_child_runs_first      : 0
> [  455.055034]  .sysctl_sched_features          : 7279
> [  455.057423]  .sysctl_sched_tunable_scaling       : 1 (logaritmic)
> [ Â 455.059829]
> [ Â 455.059830] cpu#0, 2491.994 MHz
> [  455.064443]  .nr_running           : 0
> [  455.066757]  .load              : 0
> [  455.069054]  .nr_switches          : 146510
> [  455.071353]  .nr_load_updates        : 233084
> [  455.073642]  .nr_uninterruptible       : 2
> [  455.075894]  .next_balance          : 4295.122831
> [  455.078152]  .curr->pid           : 0
> [  455.080396]  .clock             : 454680.481348
> [ Â 455.082634] Â .cpu_load[0] Â Â Â Â Â Â Â Â Â : 0
> [ Â 455.084867] Â .cpu_load[1] Â Â Â Â Â Â Â Â Â : 0
> [ Â 455.087065] Â .cpu_load[2] Â Â Â Â Â Â Â Â Â : 0
> [ Â 455.089233] Â .cpu_load[3] Â Â Â Â Â Â Â Â Â : 0
> [ Â 455.091390] Â .cpu_load[4] Â Â Â Â Â Â Â Â Â : 0
> [  455.093499]  .yld_count           : 0
> [  455.095605]  .sched_switch          : 0
> [  455.097667]  .sched_count          : 149062
> [  455.099765]  .sched_goidle          : 62756
> [  455.101781]  .avg_idle            : 1000000
> [  455.103807]  .ttwu_count           : 77219
> [  455.105958]  .ttwu_local           : 74144
> [  455.107957]  .bkl_count           : 0
> [ Â 455.109914]
> [ Â 455.109915] cfs_rq[0]:/
> [  455.113642]  .exec_clock           : 20017.048374
> [  455.115515]  .MIN_vruntime          : 0.000001
> [  455.117353]  .min_vruntime          : 28900.800090
> [  455.119185]  .max_vruntime          : 0.000001
> [  455.121028]  .spread             : 0.000000
> [ Â 455.122820] Â .spread0 Â Â Â Â Â Â Â Â Â Â Â : 0.000000
> [  455.124581]  .nr_spread_over         : 54
> [  455.126318]  .nr_running           : 0
> [  455.128045]  .load              : 0
> [  455.129743]  .load_avg            : 0.000000
> [  455.131414]  .load_period          : 0.000000
> [  455.133052]  .load_contrib          : 0
> [  455.134692]  .load_tg            : 0
> [ Â 455.136305]
> [ Â 455.136306] runnable tasks:
> [  455.136307]       task  PID     tree-key  switches  prio   exec-runtime     sum-exec     sum-sleep
> [ Â 455.136310] ----------------------------------------------------------------------------------------------------------
> [ Â 455.142856]
> [ Â 455.142857] cpu#1, 2491.994 MHz
> [  455.146215]  .nr_running           : 0
> [  455.147931]  .load              : 0
> [  455.149648]  .nr_switches          : 62175
> [  455.151348]  .nr_load_updates        : 217324
> [  455.153045]  .nr_uninterruptible       : 2
> [  455.154747]  .next_balance          : 4295.123306
> [  455.156457]  .curr->pid           : 0
> [  455.158149]  .clock             : 455157.364642
> [ Â 455.159867] Â .cpu_load[0] Â Â Â Â Â Â Â Â Â : 0
> [ Â 455.161594] Â .cpu_load[1] Â Â Â Â Â Â Â Â Â : 0
> [ Â 455.163283] Â .cpu_load[2] Â Â Â Â Â Â Â Â Â : 0
> [ Â 455.164954] Â .cpu_load[3] Â Â Â Â Â Â Â Â Â : 0
> [ Â 455.166575] Â .cpu_load[4] Â Â Â Â Â Â Â Â Â : 0
> [  455.168185]  .yld_count           : 60
> [  455.169791]  .sched_switch          : 0
> [  455.171401]  .sched_count          : 62899
> [  455.172984]  .sched_goidle          : 27394
> [  455.174580]  .avg_idle            : 1000000
> [  455.176171]  .ttwu_count           : 30510
> [  455.177739]  .ttwu_local           : 25277
> [  455.179292]  .bkl_count           : 0
> [ Â 455.180882]
> [ Â 455.180883] cfs_rq[1]:/
> [  455.183954]  .exec_clock           : 10655.021809
> [  455.185550]  .MIN_vruntime          : 0.000001
> [  455.187141]  .min_vruntime          : 19718.135550
> [  455.188771]  .max_vruntime          : 0.000001
> [  455.190407]  .spread             : 0.000000
> [ Â 455.192016] Â .spread0 Â Â Â Â Â Â Â Â Â Â Â : -9182.664540
> [  455.193634]  .nr_spread_over         : 80
> [  455.195242]  .nr_running           : 0
> [  455.196848]  .load              : 0
> [  455.198441]  .load_avg            : 0.000000
> [  455.200059]  .load_period          : 0.000000
> [  455.201654]  .load_contrib          : 0
> [  455.203240]  .load_tg            : 0
> [ Â 455.204836]
> [ Â 455.204837] runnable tasks:
> [  455.204838]       task  PID     tree-key  switches  prio   exec-runtime     sum-exec     sum-sleep
> [ Â 455.204841] ----------------------------------------------------------------------------------------------------------
> [ Â 455.211403]
> [ Â 455.211404] cpu#2, 2491.994 MHz
> [  455.214958]  .nr_running           : 0
> [  455.216771]  .load              : 0
> [  455.218564]  .nr_switches          : 37444
> [  455.220394]  .nr_load_updates        : 219042
> [  455.222202]  .nr_uninterruptible       : 0
> [  455.224039]  .next_balance          : 4295.123375
> [  455.225897]  .curr->pid           : 0
> [  455.227769]  .clock             : 455227.204963
> [ Â 455.229681] Â .cpu_load[0] Â Â Â Â Â Â Â Â Â : 0
> [ Â 455.231548] Â .cpu_load[1] Â Â Â Â Â Â Â Â Â : 0
> [ Â 455.233353] Â .cpu_load[2] Â Â Â Â Â Â Â Â Â : 0
> [ Â 455.235059] Â .cpu_load[3] Â Â Â Â Â Â Â Â Â : 0
> [ Â 455.236676] Â .cpu_load[4] Â Â Â Â Â Â Â Â Â : 0
> [  455.238280]  .yld_count           : 0
> [  455.239879]  .sched_switch          : 0
> [  455.241471]  .sched_count          : 37815
> [  455.243075]  .sched_goidle          : 16831
> [  455.244674]  .avg_idle            : 1000000
> [  455.246270]  .ttwu_count           : 18348
> [  455.247849]  .ttwu_local           : 16899
> [  455.249430]  .bkl_count           : 0
> [ Â 455.250992]
> [ Â 455.250993] cfs_rq[2]:/
> [  455.254040]  .exec_clock           : 6758.351942
> [  455.255630]  .MIN_vruntime          : 0.000001
> [  455.257236]  .min_vruntime          : 13719.942866
> [  455.258861]  .max_vruntime          : 0.000001
> [  455.260497]  .spread             : 0.000000
> [ Â 455.262122] Â .spread0 Â Â Â Â Â Â Â Â Â Â Â : -15180.857224
> [  455.263753]  .nr_spread_over         : 21
> [  455.265389]  .nr_running           : 0
> [  455.267018]  .load              : 0
> [  455.268637]  .load_avg            : 0.000000
> [  455.270292]  .load_period          : 0.000000
> [  455.271911]  .load_contrib          : 0
> [  455.273530]  .load_tg            : 0
> [ Â 455.275163]
> [ Â 455.275165] runnable tasks:
> [  455.275166]       task  PID     tree-key  switches  prio   exec-runtime     sum-exec     sum-sleep
> [ Â 455.275169] ----------------------------------------------------------------------------------------------------------
> [ Â 455.281810]
> [ Â 455.281811] cpu#3, 2491.994 MHz
> [  455.285349]  .nr_running           : 4
> [  455.287149]  .load              : 2048
> [  455.288952]  .nr_switches          : 30358
> [  455.290787]  .nr_load_updates        : 223302
> [  455.292599]  .nr_uninterruptible       : 1
> [  455.294419]  .next_balance          : 4295.123506
> [  455.296250]  .curr->pid           : 46
> [  455.298090]  .clock             : 431792.394314
> [ Â 455.299953] Â .cpu_load[0] Â Â Â Â Â Â Â Â Â : 2048
> [ Â 455.301810] Â .cpu_load[1] Â Â Â Â Â Â Â Â Â : 2048
> [ Â 455.303610] Â .cpu_load[2] Â Â Â Â Â Â Â Â Â : 2048
> [ Â 455.305311] Â .cpu_load[3] Â Â Â Â Â Â Â Â Â : 2048
> [ Â 455.306926] Â .cpu_load[4] Â Â Â Â Â Â Â Â Â : 2048
> [  455.308531]  .yld_count           : 63
> [  455.310130]  .sched_switch          : 0
> [  455.311723]  .sched_count          : 30824
> [  455.313315]  .sched_goidle          : 13491
> [  455.314904]  .avg_idle            : 1000000
> [  455.316481]  .ttwu_count           : 13858
> [  455.318057]  .ttwu_local           : 12507
> [  455.319629]  .bkl_count           : 0
> [ Â 455.321187]
> [ Â 455.321188] cfs_rq[3]:/
> [  455.324213]  .exec_clock           : 5334.144946
> [  455.325795]  .MIN_vruntime          : 13295.262803
> [  455.327401]  .min_vruntime          : 13302.523317
> [  455.329046]  .max_vruntime          : 13295.262803
> [  455.330656]  .spread             : 0.000000
> [ Â 455.332297] Â .spread0 Â Â Â Â Â Â Â Â Â Â Â : -15598.276773
> [  455.333925]  .nr_spread_over         : 117
> [  455.335577]  .nr_running           : 2
> [  455.337204]  .load              : 2048
> [  455.338823]  .load_avg            : 0.000000
> [  455.340476]  .load_period          : 0.000000
> [  455.342094]  .load_contrib          : 0
> [  455.343702]  .load_tg            : 0
> [ Â 455.345302]
> [ Â 455.345303] rt_rq[3]:/
> [  455.348416]  .rt_nr_running         : 1
> [  455.350006]  .rt_throttled          : 0
> [  455.351587]  .rt_time            : 0.000000
> [  455.353173]  .rt_runtime           : 950.000000
> [ Â 455.354759]
> [ Â 455.354760] runnable tasks:
> [  455.354761]       task  PID     tree-key  switches  prio   exec-runtime     sum-exec     sum-sleep
> [ Â 455.354764] ----------------------------------------------------------------------------------------------------------
> [ Â 455.361357] Â Â Â migration/3 Â Â 17 Â Â Â Â 0.000000 Â Â Â 1084 Â Â 0 Â Â Â Â 0.000000 Â Â Â Â 0.000933 Â Â Â Â 0.000000 /
> [ Â 455.363278] Â Â Â watchdog/3 Â Â 20 Â Â Â Â 0.000000 Â Â Â Â 31 Â Â 0 Â Â Â Â 0.000000 Â Â Â Â 0.780405 Â Â Â Â 0.002351 /
> [ Â 455.365212] R Â Â Â Â kswapd0 Â Â 46 Â Â 13302.523317 Â Â Â 714 Â 120 Â Â 13302.523317 Â Â Â 1148.767369 Â Â 245855.504389 /
> [ Â 455.367220] Â Â Â kworker/3:1 Â Â 74 Â Â 13295.262803 Â Â 10488 Â 120 Â Â 13295.262803 Â Â Â 324.924994 Â Â 315669.659686 /
> [ Â 455.369317]
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to [email protected]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
On Thu, 2011-04-28 at 21:27 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 02:59:27PM -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote:
> > > On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote:
> > > > > Way hey, cgroups are also in the mix. How jolly.
> > > > >
> > > > > Is systemd a common element of the machines hitting this bug by any
> > > > > chance?
> > > >
> > > > Well, yes, the bug report is against FC15, which needs cgroups for
> > > > systemd.
> > > >
> > >
> > > Ok although we do not have direct evidence that it's the problem yet. A
> > > broken shrinker could just mean we are also trying to aggressively
> > > reclaim in cgroups.
> > >
> > > > > The remaining traces seem to be follow-on damage related to the three
> > > > > issues of "shrinkers are bust in some manner" causing "we are not
> > > > > getting over the min watermark" and as a side-show "we are spending lots
> > > > > of time doing something unspecified but unhelpful in cgroups".
> > > >
> > > > Heh, well find a way for me to verify this: I can't turn off cgroups
> > > > because systemd then won't work and the machine won't boot ...
> > > >
> > >
> > > Same testcase, same kernel but a distro that is not using systemd to
> > > verify if cgroups are the problem. Not ideal I know. When I'm back
> > > online Tuesday, I'll try reproducing this on a !Fedora distribution. In
> > > the meantime, the following untested hatchet job might spit out
> > > which shrinker we are getting stuck in. It is also breaking out of
> > > the shrink_slab loop so it'd even be interesting to see if the bug
> > > is mitigated in any way.
> >
> > Actually, talking to Chris, I think I can get the system up using
> > init=/bin/bash without systemd, so I can try the no cgroup config.
> >
> > > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > > index c74a501..ed99104 100644
> >
> > In the mean time, this patch produces:
> >
> > (that's nothing ... apparently the trace doesn't activate when kswapd
> > goes mad).
> >
>
> Or is looping there for shorter than we expect. HZ/10?
Still doesn't print anything, even with HZ/10.
James
On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote:
> But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
> First, I have seen negative effects on my UP-system was when playing
> with linux-next [2].
> It was not clear what the origin was and the the side-effects were
> somehow "bizarre".
> The issue could be easily reproduced by tar-ing the kernel build-dir
> to an external USB-hdd.
> The issue kept RCU and TIP folks really busy.
> Before stepping 4 weeks in the dark, give it a try and let me know in
> case of success.
Well, it's highly unlikely because that's a 2.6.39 artifact and the bug
showed up in 2.6.38 ... I tried it just in case with no effect, so we
know it isn't the cause.
> For the systemd/cgroups part of the discussion (yes, I followed this
> thread in parallel):
> The patch from [4] might be interesting (untested here).
What makes you think that? This patch is about rcu problems in cgroup
manipulation through the mounted cgroup control filesystems. The bug in
question manifests when all of that is quiescent, so it's not likely to
be related.
James
On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote:
> On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote:
> > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
> > First, I have seen negative effects on my UP-system was when playing
> > with linux-next [2].
> > It was not clear what the origin was and the the side-effects were
> > somehow "bizarre".
> > The issue could be easily reproduced by tar-ing the kernel build-dir
> > to an external USB-hdd.
> > The issue kept RCU and TIP folks really busy.
> > Before stepping 4 weeks in the dark, give it a try and let me know in
> > case of success.
>
> Well, it's highly unlikely because that's a 2.6.39 artifact and the bug
> showed up in 2.6.38 ... I tried it just in case with no effect, so we
> know it isn't the cause.
Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but
it does seem to prevent non-PREEMPT from locking up totally (at least it
survives three back to back untar runs).
It's probable it alters the memory pin conditions that cause the spin,
so it's masking the problem rather than fixing it.
The question now is, what causes it in 2.6.38 stable?
James
On Fri, 2011-04-29 at 11:31 -0500, James Bottomley wrote:
> On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote:
> > On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote:
> > > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
> > > First, I have seen negative effects on my UP-system was when playing
> > > with linux-next [2].
> > > It was not clear what the origin was and the the side-effects were
> > > somehow "bizarre".
> > > The issue could be easily reproduced by tar-ing the kernel build-dir
> > > to an external USB-hdd.
> > > The issue kept RCU and TIP folks really busy.
> > > Before stepping 4 weeks in the dark, give it a try and let me know in
> > > case of success.
> >
> > Well, it's highly unlikely because that's a 2.6.39 artifact and the bug
> > showed up in 2.6.38 ... I tried it just in case with no effect, so we
> > know it isn't the cause.
>
> Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but
> it does seem to prevent non-PREEMPT from locking up totally (at least it
> survives three back to back untar runs).
>
> It's probable it alters the memory pin conditions that cause the spin,
> so it's masking the problem rather than fixing it.
Confirmed ... it's just harder to reproduce with the hrtimers init fix.
The problem definitely still exists (I had to load up the system more
before doing the tar).
This time I've caught kswapd in mem_cgroup_shrink_node_zone. sysrq-w
doesn't complete for an unknown reason
James
---
[ 575.083025] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
[ 575.083043] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 575.083251] CPU 2
[ 575.083255] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 575.083430]
[ 575.083437] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO
[ 575.083455] RIP: 0010:[<ffffffffa007ff8f>] [<ffffffffa007ff8f>] i915_gem_inactive_shrink+0x6c/0x194 [i915]
[ 575.083497] RSP: 0018:ffff88006dfb7d50 EFLAGS: 00000206
[ 575.083504] RAX: ffff88005f018200 RBX: 00000000000000c0 RCX: 0000000000000000
[ 575.083512] RDX: ffff88005f01bab0 RSI: 0000000000000000 RDI: ffff880037885820
[ 575.083529] RBP: ffff88006dfb7d90 R08: 0000000000000000 R09: 0000000000017131
[ 575.083535] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff81482e0e
[ 575.083541] R13: ffff88006dfb7cf0 R14: ffff88006dfb7cf8 R15: ffffffff810dd44d
[ 575.083548] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 575.083555] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 575.083572] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 575.083579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 575.083587] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 575.083596] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000)
[ 575.083602] Stack:
[ 575.083608] ffff88006dfb7d90 ffff880037709638 ffff88006dfb7d60 ffff8800377095f0
[ 575.083626] 0000000000000000 0000000000000000 00000000000000d0 000000000005103b
[ 575.083643] ffff88006dfb7de0 ffffffff810e1d89 000000000000003d 0000000000000080
[ 575.083660] Call Trace:
[ 575.083676] [<ffffffff810e1d89>] shrink_slab+0x6d/0x166
[ 575.083686] [<ffffffff810e49cc>] kswapd+0x533/0x798
[ 575.083698] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 575.083712] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 575.083726] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 575.083738] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 575.083750] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 575.083756] Code: e4 48 89 45 c8 75 37 48 8b 43 48 45 31 ed 48 83 c3 48 48 2d b0 00 00 00 eb 0a 48 8d 82 50 ff ff ff 41 ff c5 48 8b 90 b0 00 00 00
[ 575.083887] 05 b0 00 00 00 48 39 d8 0f 18 0a 75 e1 e9 da 00 00 00 4c 89
[ 575.083949] Call Trace:
[ 575.083957] [<ffffffff810e1d89>] shrink_slab+0x6d/0x166
[ 575.083966] [<ffffffff810e49cc>] kswapd+0x533/0x798
[ 575.083976] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 575.083986] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 575.083998] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 575.084010] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 575.084030] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 658.883363] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
[ 658.883382] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 658.883574] CPU 2
[ 658.883579] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 658.883749]
[ 658.883757] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO
[ 658.883776] RIP: 0010:[<ffffffff81080bc0>] [<ffffffff81080bc0>] arch_local_irq_restore+0xc/0xd
[ 658.883797] RSP: 0018:ffff88006dfb7d98 EFLAGS: 00000246
[ 658.883805] RAX: ffff8801005f9e68 RBX: ffff8801005e6e00 RCX: 0000000000004ae4
[ 658.883813] RDX: ffff88006dfb8000 RSI: 0000000000000246 RDI: 0000000000000246
[ 658.883821] RBP: ffff88006dfb7da0 R08: ffff8801005f9e70 R09: 00000000000007eb
[ 658.883828] R10: 0000000000000000 R11: 00000000000000cd R12: ffffffff81482e0e
[ 658.883835] R13: 0000000000016fd3 R14: 000000000000a731 R15: ffffffff810ddb31
[ 658.883844] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 658.883853] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 658.883860] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 658.883867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 658.883875] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 658.883883] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000)
[ 658.883890] Stack:
[ 658.883895] ffffffff8147bc8c ffff88006dfb7de0 ffffffff8106e7c3 ffff880000000002
[ 658.883913] 0000000000000000 ffff8801005e6000 ffff8801005f9e68 0000000000000002
[ 658.883929] 0000000000000000 ffff88006dfb7ee0 ffffffff810e45f0 0000000000000003
[ 658.883947] Call Trace:
[ 658.883961] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 658.883976] [<ffffffff8106e7c3>] prepare_to_wait+0x6c/0x78
[ 658.883988] [<ffffffff810e45f0>] kswapd+0x157/0x798
[ 658.884000] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 658.884009] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 658.884020] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 658.884033] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 658.884045] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 658.884057] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 658.884062] Code: 1f 44 00 00 66 ff 05 30 16 98 00 fb 66 0f 1f 44 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 5d c3 55 48 89 e5 57 9d 0f 1f 44 00 00 5d <c3> 55 48 89 e5 fa 66 0f 1f 44 00 00 5d c3 55 48 89 e5 50 9c 58
[ 658.884238] Call Trace:
[ 658.884248] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 658.884260] [<ffffffff8106e7c3>] prepare_to_wait+0x6c/0x78
[ 658.884269] [<ffffffff810e45f0>] kswapd+0x157/0x798
[ 658.884280] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 658.884289] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 658.884299] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 658.884311] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 658.884322] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 658.884333] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 690.767137] SysRq : Show backtrace of all active CPUs
[ 690.767158] sending NMI to all CPUs:
[ 690.767171] NMI backtrace for cpu 2
[ 690.767188] CPU 2
[ 690.767196] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 690.767457]
[ 690.767468] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO
[ 690.767493] RIP: 0010:[<ffffffff810e3980>] [<ffffffff810e3980>] shrink_zone+0x221/0x4b3
[ 690.767519] RSP: 0018:ffff88006dfb7d30 EFLAGS: 00000206
[ 690.767529] RAX: 0000000000000000 RBX: ffff88006dfb7e58 RCX: 000000000000000c
[ 690.767539] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 690.767550] RBP: ffff88006dfb7de0 R08: 0000000000000001 R09: 00000000000007eb
[ 690.767561] R10: 0000000000000002 R11: 00000000000000f5 R12: ffff8801005e6000
[ 690.767572] R13: ffff8801005e64d8 R14: 0000000000001572 R15: 00000000ffffffff
[ 690.767584] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 690.767597] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 690.767607] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 690.767618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 690.767628] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 690.767640] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000)
[ 690.767650] Stack:
[ 690.767657] 0000000000000001 000000000000008c 000000000000003c 0000000000000000
[ 690.767683] ffffffffffffffff ffff8801005e6480 000000000000000c 0000000c81118591
[ 690.767717] 0000000000000000 0000000000000000 ffff880100530380 0000000000000000
[ 690.767742] Call Trace:
[ 690.767764] [<ffffffff810d9967>] ? zone_watermark_ok_safe+0xa1/0xae
[ 690.767779] [<ffffffff810e49a6>] kswapd+0x50d/0x798
[ 690.767794] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 690.767811] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 690.767830] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 690.767846] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 690.767861] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 690.767868] Code: e2 ff ff 83 7d 80 00 44 8b 85 50 ff ff ff 74 1b 8a 4d 8c 31 f6 41 83 ff 01 40 0f 96 c6 31 d2 48 d3 e8 48 0f af 44 f5 c0 49 f7 f6
[ 690.768028] 89 c1 48 8d 51 04 49 03 44 d5 00 48 83 f8 1f 49 89 44 d5 00
[ 690.768113] Call Trace:
[ 690.768126] [<ffffffff810d9967>] ? zone_watermark_ok_safe+0xa1/0xae
[ 690.768139] [<ffffffff810e49a6>] kswapd+0x50d/0x798
[ 690.768154] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 690.768168] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 690.768183] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 690.768199] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 690.768213] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 690.768223] NMI backtrace for cpu 3
[ 690.768236] CPU 3
[ 690.768244] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 690.768832]
[ 690.768843] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO
[ 690.768867] RIP: 0010:[<ffffffff81275b36>] [<ffffffff81275b36>] intel_idle+0xaa/0x100
[ 690.768890] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046
[ 690.768900] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
[ 690.768912] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640
[ 690.768923] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003c2
[ 690.768934] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030
[ 690.768946] R13: 12191803fa451798 R14: 0000000000000004 R15: 0000000000000003
[ 690.768959] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
[ 690.768971] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 690.768982] CR2: 0000000000429bf0 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 690.768993] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 690.769005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 690.769017] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700)
[ 690.769027] Stack:
[ 690.769034] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1280 0000000000011280
[ 690.769063] ffff8800715dfeb8 000000038139c77a ffffe8ffffcc0170 ffffe8ffffcc0170
[ 690.769092] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b668
[ 690.769121] Call Trace:
[ 690.769137] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 690.769153] [<ffffffff8139b668>] cpuidle_idle_call+0xe7/0x166
[ 690.769173] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 690.769191] [<ffffffff8146ac57>] start_secondary+0x223/0x225
[ 690.769203] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 0b e0 ff 4c 29 e8 48 89 c7 e8 ab 2b de ff 4c 69 e0 40 42
[ 690.769477] Call Trace:
[ 690.769491] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 690.769506] [<ffffffff8139b668>] cpuidle_idle_call+0xe7/0x166
[ 690.769522] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 690.770451] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 690.771380] [<ffffffff81483f73>] ? smp_apic_timer_interrupt+0x7e/0x8c
[ 692.887516] SysRq : Show Blocked State
[ 692.887534] task PC stack pid father
[ 692.887551] fsnotify_mark D 0000000000000000 0 49 2 0x00000000
[ 692.887567] ffff88006dfb1cc0 0000000000000046 0000000000000000 0000000000000000
[ 692.887585] ffff88006dfbc500 ffff88006dfb1fd8 ffff88006dfb1fd8 0000000000013b40
[ 692.887601] ffff880071589700 ffff88006dfbc500 ffff88006dfb1cb0 000000010024f4c0
[ 692.887618] Call Trace:
[ 692.887635] [<ffffffff8147aa4b>] schedule_timeout+0x34/0xde
[ 692.887650] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 692.887662] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 692.887673] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 692.887683] [<ffffffff8147a814>] wait_for_common+0xac/0x101
[ 692.887692] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226
[ 692.887702] [<ffffffff810ada0d>] ? synchronize_rcu_bh+0x5c/0x5c
[ 692.887711] [<ffffffff8147a91d>] wait_for_completion+0x1d/0x1f
[ 692.887720] [<ffffffff810ada67>] synchronize_sched+0x5a/0x5c
[ 692.887730] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43
[ 692.887740] [<ffffffff810726fc>] __synchronize_srcu+0x31/0x89
[ 692.887751] [<ffffffff81072780>] synchronize_srcu+0x15/0x17
[ 692.887762] [<ffffffff8114c8be>] fsnotify_mark_destroy+0x90/0x165
[ 692.887774] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.887783] [<ffffffff8114c82e>] ? fsnotify_put_mark+0x1c/0x1c
[ 692.887793] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.887806] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.887818] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.887828] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.887840] jbd2/dm-1-8 D 0000000000000000 0 377 2 0x00000000
[ 692.887854] ffff88006d765cd0 0000000000000046 0000000000000000 0000000000000000
[ 692.887871] ffff88006d64dc00 ffff88006d765fd8 ffff88006d765fd8 0000000000013b40
[ 692.887888] ffff880071589700 ffff88006d64dc00 ffff88006d765cd0 000000018106e7c3
[ 692.887905] Call Trace:
[ 692.887917] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 692.887928] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 692.887939] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 692.887949] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 692.887960] [<ffffffff8100804e>] ? load_TLS+0x10/0x14
[ 692.887971] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 692.887982] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.887992] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 692.888003] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 692.888012] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 692.888023] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a
[ 692.888033] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888043] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10
[ 692.888053] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.888065] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.888076] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.888087] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.888095] flush-253:1 D 0000000000000000 0 439 2 0x00000000
[ 692.888117] ffff88006d70f730 0000000000000046 ffff88006d70f6c0 ffffffff00000000
[ 692.888135] ffff880037b7ae00 ffff88006d70ffd8 ffff88006d70ffd8 0000000000013b40
[ 692.888153] ffff880071589700 ffff880037b7ae00 ffff8801002543c0 000000016d70f798
[ 692.888170] Call Trace:
[ 692.888180] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e
[ 692.888190] [<ffffffff81217900>] get_request_wait+0x102/0x18b
[ 692.888200] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888209] [<ffffffff81218641>] __make_request+0x18a/0x2b8
[ 692.888219] [<ffffffff81217338>] generic_make_request+0x2a9/0x323
[ 692.888227] [<ffffffff81217490>] submit_bio+0xde/0xfd
[ 692.888238] [<ffffffff81199bd9>] ? ext4_bio_write_page+0x30d/0x320
[ 692.888248] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58
[ 692.888257] [<ffffffff81194b65>] mpage_da_submit_io+0x371/0x389
[ 692.888267] [<ffffffff8119527d>] ? ext4_mark_iloc_dirty+0x4db/0x543
[ 692.888276] [<ffffffff8119552d>] ? ext4_mark_inode_dirty+0x1c5/0x1f0
[ 692.888287] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd
[ 692.888298] [<ffffffff811ab72a>] ? ext4_journal_start_sb+0x108/0x120
[ 692.888308] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d
[ 692.888319] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a
[ 692.888331] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc
[ 692.888341] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161
[ 692.888351] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b
[ 692.888361] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335
[ 692.888371] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 692.888381] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1
[ 692.888392] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d
[ 692.888402] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5
[ 692.888413] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d
[ 692.888423] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.888434] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.888445] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.888455] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.888472] jbd2/dm-2-8 D ffff880071617380 0 791 2 0x00000000
[ 692.888485] ffff88006cfb1cd0 0000000000000046 0000000000000000 ffff880000000001
[ 692.888501] ffff88003769ae00 ffff88006cfb1fd8 ffff88006cfb1fd8 0000000000013b40
[ 692.888517] ffff880058cdc500 ffff88003769ae00 ffff88006cfb1cd0 000000018106e7c3
[ 692.888533] Call Trace:
[ 692.888547] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 692.888559] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888568] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 692.888578] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 692.888587] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 692.888597] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a
[ 692.888607] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888617] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10
[ 692.888627] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.888637] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.888648] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.888659] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.888665] NetworkManager D 0000000000000000 0 809 1 0x00000000
[ 692.888682] ffff88006ff23ad8 0000000000000082 ffff880100293b40 0000000000000000
[ 692.888697] ffff88006cfe2e00 ffff88006ff23fd8 ffff88006ff23fd8 0000000000013b40
[ 692.888713] ffffffff81a0b020 ffff88006cfe2e00 ffff88006ff23ad8 000000018106e7c3
[ 692.888729] Call Trace:
[ 692.888738] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 692.888753] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888761] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 692.888770] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 692.888780] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 692.888790] [<ffffffff81044796>] ? delayacct_blkio_end+0x21/0x41
[ 692.888799] [<ffffffff8147a5df>] ? io_schedule+0x79/0x7e
[ 692.888808] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c
[ 692.888818] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175
[ 692.888827] [<ffffffff811320c7>] touch_atime+0x10e/0x131
[ 692.888840] [<ffffffff810d7160>] generic_file_aio_read+0x5d9/0x640
[ 692.888851] [<ffffffff8111ecaa>] do_sync_read+0xbf/0xff
[ 692.888860] [<ffffffff8103fd1f>] ? should_resched+0xe/0x2d
[ 692.888872] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33
[ 692.888884] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 692.888894] [<ffffffff8111f311>] vfs_read+0xa9/0xf0
[ 692.888902] [<ffffffff8111f3a2>] sys_read+0x4a/0x6e
[ 692.888912] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 692.888923] rs:main Q:Reg D 0000000000000000 0 886 1 0x00000000
[ 692.888935] ffff88006ee35a38 0000000000000082 ffff88010028fa00 ffff880100000000
[ 692.888950] ffff88006ed5dc00 ffff88006ee35fd8 ffff88006ee35fd8 0000000000013b40
[ 692.888966] ffff880071589700 ffff88006ed5dc00 ffff88006ee35a38 000000018106e7c3
[ 692.888986] Call Trace:
[ 692.888994] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 692.889005] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.889013] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 692.889022] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 692.889031] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 692.889041] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c
[ 692.889050] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175
[ 692.889059] [<ffffffff81131f95>] file_update_time+0xef/0x113
[ 692.889070] [<ffffffff810d69b8>] __generic_file_aio_write+0x15d/0x272
[ 692.889086] [<ffffffff8107cd0e>] ? futex_wait_queue_me+0xc4/0xe0
[ 692.889099] [<ffffffff810d6b2e>] generic_file_aio_write+0x61/0xba
[ 692.889115] [<ffffffff8118fc00>] ext4_file_write+0x1dc/0x234
[ 692.889125] [<ffffffff8111ebab>] do_sync_write+0xbf/0xff
[ 692.889133] [<ffffffff8114b810>] ? fsnotify+0x1ff/0x217
[ 692.889144] [<ffffffff811f1666>] ? selinux_file_permission+0x58/0xb4
[ 692.889155] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33
[ 692.889164] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 692.889172] [<ffffffff8111f221>] vfs_write+0xac/0xf3
[ 692.889181] [<ffffffff8111f410>] sys_write+0x4a/0x6e
[ 692.889191] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 692.889199] flush-253:2 D 0000000000000000 0 883 2 0x00000000
[ 692.889211] ffff88006ee736b0 0000000000000046 ffff88006ee73640 ffffffff00000000
[ 692.889446] ffff88006d5d9700 ffff88006ee73fd8 ffff88006ee73fd8 0000000000013b40
[ 692.889462] ffff880071589700 ffff88006d5d9700 ffff8801002543c0 000000016ee73718
[ 692.889478] Call Trace:
[ 692.889488] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e
[ 692.889496] [<ffffffff81217900>] get_request_wait+0x102/0x18b
[ 692.889507] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.889516] [<ffffffff81218641>] __make_request+0x18a/0x2b8
[ 692.889525] [<ffffffff81217338>] generic_make_request+0x2a9/0x323
[ 692.889536] [<ffffffff8147bd19>] ? _raw_read_unlock_irqrestore+0x17/0x19
[ 692.889547] [<ffffffff8138d520>] ? dm_target_exit+0x19/0x19
[ 692.889555] [<ffffffff81217490>] submit_bio+0xde/0xfd
[ 692.889566] [<ffffffff810ea4e1>] ? inc_zone_page_state+0x27/0x29
[ 692.889576] [<ffffffff810dc9fd>] ? account_page_writeback+0x25/0x29
[ 692.889587] [<ffffffff8122e200>] ? radix_tree_gang_lookup_slot+0x66/0x87
[ 692.889597] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58
[ 692.889606] [<ffffffff81199a48>] ext4_bio_write_page+0x17c/0x320
[ 692.889616] [<ffffffff810ea4a1>] ? mod_state+0x76/0x7d
[ 692.889625] [<ffffffff81194afa>] mpage_da_submit_io+0x306/0x389
[ 692.889636] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd
[ 692.889645] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d
[ 692.889656] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a
[ 692.889666] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc
[ 692.889677] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161
[ 692.889687] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b
[ 692.889697] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335
[ 692.889706] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 692.889717] [<ffffffff8147bc3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 692.889727] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1
[ 692.889736] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d
[ 692.889746] [<ffffffff8147aac7>] ? schedule_timeout+0xb0/0xde
[ 692.889756] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5
[ 692.889766] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d
[ 692.889776] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.889787] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.889798] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.889809] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.889815] abrt-dump-oops D 0000000000000000 0 903 817 0x00000000
[ 692.889827] ffff88006eda1c78 0000000000000082 0000000000000005 0000000000000000
[ 692.889843] ffff880037698000 ffff88006eda1fd8 ffff88006eda1fd8 0000000000013b40
[ 692.889860] ffffffff81a0b020 ffff880037698000 ffff88006eda1c78 000000018106e7c3
[ 692.889876] Call Trace:
[ 692.889884] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 692.889895] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.889904] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 692.889913] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 692.889923] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 692.889934] [<ffffffff8119ccaa>] ext4_mkdir+0xc9/0x33e
[ 692.889944] [<ffffffff811297a7>] vfs_mkdir+0x5f/0x9b
[ 692.889954] [<ffffffff8112b0ae>] sys_mkdirat+0x97/0xe8
[ 692.889965] [<ffffffff81122b89>] ? sys_newstat+0x2a/0x33
[ 692.889975] [<ffffffff8112b117>] sys_mkdir+0x18/0x1a
[ 692.889984] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 692.889998] Xorg D 0000000000000000 0 1187 1183 0x00400084
[ 692.890012] ffff88007171ba38 0000000000000082 ffff8800177db2d8 ffff880000000000
[ 692.890029] ffff88006d581700 ffff88007171bfd8 ffff88007171bfd8 0000000000013b40
[ 692.890046] ffffffff81a0b020 ffff88006d581700 ffff88007171ba38
nc: Write error: No route to host
[ 709.230547] SysRq : Show Blocked State
[ 709.230567] task PC stack pid father
[ 709.230584] fsnotify_mark D 0000000000000000 0 49 2 0x00000000
[ 709.230600] ffff88006dfb1cc0 0000000000000046 0000000000000000 0000000000000000
[ 709.230621] ffff88006dfbc500 ffff88006dfb1fd8 ffff88006dfb1fd8 0000000000013b40
[ 709.230638] ffff880071589700 ffff88006dfbc500 ffff88006dfb1cb0 000000010024f4c0
[ 709.230664] Call Trace:
[ 709.230683] [<ffffffff8147aa4b>] schedule_timeout+0x34/0xde
[ 709.230697] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 709.230710] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 709.230721] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 709.230731] [<ffffffff8147a814>] wait_for_common+0xac/0x101
[ 709.230740] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226
[ 709.230751] [<ffffffff810ada0d>] ? synchronize_rcu_bh+0x5c/0x5c
[ 709.230761] [<ffffffff8147a91d>] wait_for_completion+0x1d/0x1f
[ 709.230769] [<ffffffff810ada67>] synchronize_sched+0x5a/0x5c
[ 709.230780] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43
[ 709.230790] [<ffffffff810726fc>] __synchronize_srcu+0x31/0x89
[ 709.230800] [<ffffffff81072780>] synchronize_srcu+0x15/0x17
[ 709.230810] [<ffffffff8114c8be>] fsnotify_mark_destroy+0x90/0x165
[ 709.230822] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.230831] [<ffffffff8114c82e>] ? fsnotify_put_mark+0x1c/0x1c
[ 709.230842] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.230854] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.230866] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.230876] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.230889] jbd2/dm-1-8 D 0000000000000000 0 377 2 0x00000000
[ 709.230902] ffff88006d765cd0 0000000000000046 0000000000000000 0000000000000000
[ 709.230920] ffff88006d64dc00 ffff88006d765fd8 ffff88006d765fd8 0000000000013b40
[ 709.230937] ffff880071589700 ffff88006d64dc00 ffff88006d765cd0 000000018106e7c3
[ 709.230955] Call Trace:
[ 709.230967] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 709.230978] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 709.230989] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 709.230999] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 709.231011] [<ffffffff8100804e>] ? load_TLS+0x10/0x14
[ 709.231021] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 709.231033] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231043] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 709.231055] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 709.231064] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 709.231075] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a
[ 709.231086] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231096] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10
[ 709.231106] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.231117] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.231129] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.231139] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.231147] flush-253:1 D 0000000000000000 0 439 2 0x00000000
[ 709.231161] ffff88006d70f730 0000000000000046 ffff88006d70f6c0 ffffffff00000000
[ 709.231178] ffff880037b7ae00 ffff88006d70ffd8 ffff88006d70ffd8 0000000000013b40
[ 709.231195] ffff880071589700 ffff880037b7ae00 ffff8801002543c0 000000016d70f798
[ 709.231220] Call Trace:
[ 709.231230] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e
[ 709.231240] [<ffffffff81217900>] get_request_wait+0x102/0x18b
[ 709.231251] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231260] [<ffffffff81218641>] __make_request+0x18a/0x2b8
[ 709.231270] [<ffffffff81217338>] generic_make_request+0x2a9/0x323
[ 709.231279] [<ffffffff81217490>] submit_bio+0xde/0xfd
[ 709.231290] [<ffffffff81199bd9>] ? ext4_bio_write_page+0x30d/0x320
[ 709.231300] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58
[ 709.231309] [<ffffffff81194b65>] mpage_da_submit_io+0x371/0x389
[ 709.231318] [<ffffffff8119527d>] ? ext4_mark_iloc_dirty+0x4db/0x543
[ 709.231328] [<ffffffff8119552d>] ? ext4_mark_inode_dirty+0x1c5/0x1f0
[ 709.231338] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd
[ 709.231350] [<ffffffff811ab72a>] ? ext4_journal_start_sb+0x108/0x120
[ 709.231360] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d
[ 709.231372] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a
[ 709.231383] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc
[ 709.231393] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161
[ 709.231404] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b
[ 709.231414] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335
[ 709.231424] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 709.231434] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1
[ 709.231445] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d
[ 709.231455] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5
[ 709.231465] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d
[ 709.231475] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.231486] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.231497] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.231513] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.231521] jbd2/dm-2-8 D ffff880071617380 0 791 2 0x00000000
[ 709.231534] ffff88006cfb1cd0 0000000000000046 0000000000000000 ffff880000000001
[ 709.231550] ffff88003769ae00 ffff88006cfb1fd8 ffff88006cfb1fd8 0000000000013b40
[ 709.231567] ffff880058cdc500 ffff88003769ae00 ffff88006cfb1cd0 000000018106e7c3
[ 709.231587] Call Trace:
[ 709.231597] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 709.231609] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231618] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 709.231628] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 709.231638] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 709.231652] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a
[ 709.231664] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231674] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10
[ 709.231684] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.231695] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.231706] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.231716] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.231727] NetworkManager D 0000000000000000 0 809 1 0x00000000
[ 709.231739] ffff88006ff23ad8 0000000000000082 ffff880100293b40 0000000000000000
[ 709.231755] ffff88006cfe2e00 ffff88006ff23fd8 ffff88006ff23fd8 0000000000013b40
[ 709.231772] ffffffff81a0b020 ffff88006cfe2e00 ffff88006ff23ad8 000000018106e7c3
[ 709.231787] Call Trace:
[ 709.231801] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 709.231812] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231821] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 709.231829] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 709.231839] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 709.231850] [<ffffffff81044796>] ? delayacct_blkio_end+0x21/0x41
[ 709.231859] [<ffffffff8147a5df>] ? io_schedule+0x79/0x7e
[ 709.231868] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c
[ 709.231878] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175
[ 709.231887] [<ffffffff811320c7>] touch_atime+0x10e/0x131
[ 709.231899] [<ffffffff810d7160>] generic_file_aio_read+0x5d9/0x640
[ 709.231911] [<ffffffff8111ecaa>] do_sync_read+0xbf/0xff
[ 709.231919] [<ffffffff8103fd1f>] ? should_resched+0xe/0x2d
[ 709.231935] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33
[ 709.231944] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 709.231953] [<ffffffff8111f311>] vfs_read+0xa9/0xf0
[ 709.231962] [<ffffffff8111f3a2>] sys_read+0x4a/0x6e
[ 709.231972] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 709.231983] rs:main Q:Reg D 0000000000000000 0 886 1 0x00000000
[ 709.231995] ffff88006ee35a38 0000000000000082 ffff88010028fa00 ffff880100000000
[ 709.232011] ffff88006ed5dc00 ffff88006ee35fd8 ffff88006ee35fd8 0000000000013b40
[ 709.232027] ffff880071589700 ffff88006ed5dc00 ffff88006ee35a38 000000018106e7c3
[ 709.232047] Call Trace:
[ 709.232055] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 709.232066] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.232075] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 709.232083] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 709.232093] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 709.232102] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c
[ 709.232112] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175
[ 709.232121] [<ffffffff81131f95>] file_update_time+0xef/0x113
[ 709.232136] [<ffffffff810d69b8>] __generic_file_aio_write+0x15d/0x272
[ 709.232148] [<ffffffff8107cd0e>] ? futex_wait_queue_me+0xc4/0xe0
[ 709.232159] [<ffffffff810d6b2e>] generic_file_aio_write+0x61/0xba
[ 709.232171] [<ffffffff8118fc00>] ext4_file_write+0x1dc/0x234
[ 709.232181] [<ffffffff8111ebab>] do_sync_write+0xbf/0xff
[ 709.232189] [<ffffffff8114b810>] ? fsnotify+0x1ff/0x217
[ 709.232200] [<ffffffff811f1666>] ? selinux_file_permission+0x58/0xb4
[ 709.232210] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33
[ 709.232219] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 709.232228] [<ffffffff8111f221>] vfs_write+0xac/0xf3
[ 709.232237] [<ffffffff8111f410>] sys_write+0x4a/0x6e
[ 709.232246] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 709.232255] flush-253:2 D 0000000000000000 0 883 2 0x00000000
[ 709.232267] ffff88006ee736b0 0000000000000046 ffff88006ee73640 ffffffff00000000
[ 709.232500] ffff88006d5d9700 ffff88006ee73fd8 ffff88006ee73fd8 0000000000013b40
[ 709.232516] ffff880071589700 ffff88006d5d9700 ffff8801002543c0 000000016ee73718
[ 709.232532] Call Trace:
[ 709.232542] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e
[ 709.232550] [<ffffffff81217900>] get_request_wait+0x102/0x18b
[ 709.232561] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.232570] [<ffffffff81218641>] __make_request+0x18a/0x2b8
[ 709.232579] [<ffffffff81217338>] generic_make_request+0x2a9/0x323
[ 709.232589] [<ffffffff8147bd19>] ? _raw_read_unlock_irqrestore+0x17/0x19
[ 709.232601] [<ffffffff8138d520>] ? dm_target_exit+0x19/0x19
[ 709.232609] [<ffffffff81217490>] submit_bio+0xde/0xfd
[ 709.232620] [<ffffffff810ea4e1>] ? inc_zone_page_state+0x27/0x29
[ 709.232630] [<ffffffff810dc9fd>] ? account_page_writeback+0x25/0x29
[ 709.232641] [<ffffffff8122e200>] ? radix_tree_gang_lookup_slot+0x66/0x87
[ 709.232656] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58
[ 709.232666] [<ffffffff81199a48>] ext4_bio_write_page+0x17c/0x320
[ 709.232676] [<ffffffff810ea4a1>] ? mod_state+0x76/0x7d
[ 709.232685] [<ffffffff81194afa>] mpage_da_submit_io+0x306/0x389
[ 709.232696] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd
[ 709.232706] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d
[ 709.232716] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a
[ 709.232728] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc
[ 709.232738] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161
[ 709.232748] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b
[ 709.232758] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335
[ 709.232768] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 709.232779] [<ffffffff8147bc3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 709.232789] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1
[ 709.232799] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d
[ 709.232808] [<ffffffff8147aac7>] ? schedule_timeout+0xb0/0xde
[ 709.232819] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5
[ 709.232829] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d
[ 709.232839] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.232850] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.232861] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.232872] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.232878] abrt-dump-oops D 0000000000000000 0 903 817 0x00000000
[ 709.232891] ffff88006eda1c78 0000000000000082 0000000000000005 0000000000000000
[ 709.232908] ffff880037698000 ffff88006eda1fd8 ffff88006eda1fd8 0000000000013b40
[ 709.232925] ffffffff81a0b020 ffff880037698000 ffff88006eda1c78 000000018106e7c3
[ 709.232942] Call Trace:
[ 709.232951] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 709.232962] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.232971] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 709.232979] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 709.232989] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 709.233000] [<ffffffff8119ccaa>] ext4_mkdir+0xc9/0x33e
[ 709.233010] [<ffffffff811297a7>] vfs_mkdir+0x5f/0x9b
[ 709.233021] [<ffffffff8112b0ae>] sys_mkdirat+0x97/0xe8
[ 709.233032] [<ffffffff81122b89>] ? sys_newstat+0x2a/0x33
[ 709.233042] [<ffffffff8112b117>] sys_mkdir+0x18/0x1a
[ 709.233051] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 709.233066] Xorg D 0000000000000000 0 1187 1183 0x00400084
[ 709.233080] ffff88007171ba38 0000000000000082 ffff8800177db2d8 ffff880000000000
[ 709.233097] ffff88006d581700 ffff88007171bfd8 ffff88007171bfd8 0000000000013b40
[ 709.233114] ffffffff81a0b020 ffff88006d581700[ 709.235257] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 709.235532] .MIN_vruntime : 0.000001
[ 709.235656]
[ 709.236129] .se->load.weight : 2
[ 709.236374] .se->load.weight : 2
[ 709.236431]
On Fri, 2011-04-29 at 13:02 -0500, James Bottomley wrote:
> On Fri, 2011-04-29 at 11:31 -0500, James Bottomley wrote:
> > On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote:
> > > On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote:
> > > > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
> > > > First, I have seen negative effects on my UP-system was when playing
> > > > with linux-next [2].
> > > > It was not clear what the origin was and the the side-effects were
> > > > somehow "bizarre".
> > > > The issue could be easily reproduced by tar-ing the kernel build-dir
> > > > to an external USB-hdd.
> > > > The issue kept RCU and TIP folks really busy.
> > > > Before stepping 4 weeks in the dark, give it a try and let me know in
> > > > case of success.
> > >
> > > Well, it's highly unlikely because that's a 2.6.39 artifact and the bug
> > > showed up in 2.6.38 ... I tried it just in case with no effect, so we
> > > know it isn't the cause.
> >
> > Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but
> > it does seem to prevent non-PREEMPT from locking up totally (at least it
> > survives three back to back untar runs).
> >
> > It's probable it alters the memory pin conditions that cause the spin,
> > so it's masking the problem rather than fixing it.
>
> Confirmed ... it's just harder to reproduce with the hrtimers init fix.
> The problem definitely still exists (I had to load up the system more
> before doing the tar).
>
> This time I've caught kswapd in mem_cgroup_shrink_node_zone. sysrq-w
> doesn't complete for an unknown reason
As a follow on to this, there's a shrink_zone(0, ...) in the cgroup
path. This causes it to scan all memory exhaustively (generating quite
a lot of work). The comment above it implies it's some type of hack for
cgroup accounting, but reducing it to DEF_PRIORITY makes the hang go
away (verified on both 2.6.39-rc4 and 2.6.38.4). Note that I still get
soft lockups in kswapd0, but they no longer hang the box. I'll prepare
a patch for the next round of debate.
James
On Thu, Apr 28, 2011 at 05:43:48PM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote:
> > > Actually, talking to Chris, I think I can get the system up using
> > > init=/bin/bash without systemd, so I can try the no cgroup config.
> >
> > OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back
> > runs of untar without locking or getting kswapd pegged, so I'm pretty
> > certain this is cgroups related. The next steps are to turn cgroups
> > back on but try disabling the memory and IO controllers.
>
> I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR.
>
> The results are curious: the tar does complete (I've done three back to
> back). However, I did get one soft lockup in kswapd (below). But the
> system recovers instead of halting I/O and hanging like it did
> previously.
>
> The soft lockup is in shrink_slab, so perhaps it's a combination of slab
> shrinker and cgroup memory controller issues?
>
So, kswapd is still looping in reclaim and spending a lot of time in
shrink_slab but it must not be the shrinker itself or that debug patch
would have triggered. It's curious that cgroups are involved with
systemd considering that one would expect those groups to be fairly
small. I still don't have a new theory but will get hold of a Fedora 15
install CD and see can I reproduce it locally.
One last thing, what is the value of /proc/sys/vm/zone_reclaim_mode? Two
of the reporting machines could be NUMA and if that proc file reads as
1, I'd be interested in hearing the results of a test with it set to 0.
Thanks.
On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote:
> > > This is the output of perf record -g -a -f sleep 5
> > >
> > > (hopefully the list won't choke)
> >
> > Um, this one actually shows kswapd
> >
> > James
> >
> > ---
> >
> > # Events: 6K cycles
> > #
> > # Overhead Command Shared Object Symbol
> > # ........ ........... ................... .......................................
> > #
> > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
> > |
> > --- shrink_slab
> > |
> > |--99.91%-- kswapd
> > | kthread
> > | kernel_thread_helper
> > --0.09%-- [...]
> >
>
> Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use
> compaction instead of lumpy reclaim" is related unless we are seeing
> two problems that happen to manifest in a similar manner.
That is a distinct possibility.
>
> However, there were a number of changes made to dcache in particular
> for 2.6.38. Specifically thinks like dentry_kill use trylock and is
> happy to loop around if it fails to acquire anything. See things like
> this for example;
>
[ text deleted ]
> Way hey, cgroups are also in the mix. How jolly.
>
> Is systemd a common element of the machines hitting this bug by any
> chance?
Not in my case, using upstart on my machine.
>
Colin
On Tue, 2011-05-03 at 10:13 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 05:43:48PM -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote:
> > > On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote:
> > > > Actually, talking to Chris, I think I can get the system up using
> > > > init=/bin/bash without systemd, so I can try the no cgroup config.
> > >
> > > OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back
> > > runs of untar without locking or getting kswapd pegged, so I'm pretty
> > > certain this is cgroups related. The next steps are to turn cgroups
> > > back on but try disabling the memory and IO controllers.
> >
> > I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR.
> >
> > The results are curious: the tar does complete (I've done three back to
> > back). However, I did get one soft lockup in kswapd (below). But the
> > system recovers instead of halting I/O and hanging like it did
> > previously.
> >
> > The soft lockup is in shrink_slab, so perhaps it's a combination of slab
> > shrinker and cgroup memory controller issues?
> >
>
> So, kswapd is still looping in reclaim and spending a lot of time in
> shrink_slab but it must not be the shrinker itself or that debug patch
> would have triggered. It's curious that cgroups are involved with
> systemd considering that one would expect those groups to be fairly
> small. I still don't have a new theory but will get hold of a Fedora 15
> install CD and see can I reproduce it locally.
I've got a ftrace output of kswapd ... it's 500k compressed, so I'll
send under separate cover.
> One last thing, what is the value of /proc/sys/vm/zone_reclaim_mode? Two
> of the reporting machines could be NUMA and if that proc file reads as
> 1, I'd be interested in hearing the results of a test with it set to 0.
> Thanks.
It's zero, I'm afraid
James
On Fri, 2011-04-29 at 00:40 +0200, Jan Kara wrote:
> On Thu 28-04-11 15:58:21, Colin Ian King wrote:
> > On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote:
> > > On Thu 28-04-11 16:25:51, Jan Kara wrote:
> > > > On Thu 28-04-11 15:01:22, Colin Ian King wrote:
> > > > >
> > > > > > Could you post the soft lockups you're seeing?
> > > > >
> > > > > As requested, attached
> > > > Hum, what keeps puzzling me is that in all the cases of hangs I've seen
> > > > so far, we are stuck waiting for IO to finish for a long time - e.g. in the
> > > > traces below kjournald waits for PageWriteback bit to get cleared. Also we
> > > > are stuck waiting for page locks which might be because those pages are
> > > > being read in? All in all it seems that the IO is just incredibly slow.
> > > >
> > > > But it's not clear to me what pushes us into that situation (especially
> > > > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
> > > > underlying blocks are not already allocated.
> > > Hmm, maybe because the system is under memory pressure (and kswapd is not
> > > able to get rid of dirty pages), we page out clean pages. Thus also pages
> > > of executables which need to be paged in soon anyway thus putting heavy
> > > read load on the system which makes writes crawl? I'm not sure why
> > > compaction should make this any worse but maybe it can.
> > >
> > > James, Colin, can you capture output of 'vmstat 1' while you do the
> > > copying? Thanks.
> >
> > Attached.
> Thanks. So I there are a few interesting points in the vmstat output:
> For first 30 seconds, we are happily copying data - relatively steady read
> throughput (about 20-40 MB/s) and occasional peak from flusher thread
> flushing dirty data. During this time free memory drops from about 1.4 GB
> to about 22!!! MB - mm seems to like to really use the machine ;). Then
> things get interesting:
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 0 1 0 83372 5228 1450776 0 0 39684 90132 450 918 0 4 74 22
> 0 1 0 22416 5228 1509864 0 0 29452 48492 403 869 1 2 80 18
> 2 0 0 20056 5384 1513996 0 0 2248 2116 434 1191 4 4 71 21
> 0 1 0 19800 5932 1514600 0 0 644 104 454 1166 8 3 64 24
> 1 0 0 21848 5940 1515244 0 0 292 380 468 1775 15 6 75 3
> 1 0 0 20936 5940 1515876 0 0 296 296 496 1846 18 8 74 0
> 1 0 0 17792 5940 1516580 0 0 356 356 484 1862 23 8 69 0
> 1 0 0 17544 5940 1517364 0 0 412 412 482 1812 16 7 77 0
> 4 0 0 18148 5948 1517968 0 0 288 344 436 1749 19 9 69 3
> 0 2 220 137528 1616 1402468 0 220 74708 2164 849 1806 3 6 63 28
> 0 3 224 36184 1628 1499648 0 4 50820 86204 532 1272 0 4 64 32
> 0 2 19680 53688 1628 1484388 32 19456 6080 62972 242 287 0 2 63 34
> 0 2 36928 1407432 1356 150980 0 17252 1564 17276 368 764 1 3 73 22
>
> So when free memory reached about 20 MB, both read and write activity
> almost stalled for 7 s (probably everybody waits for free memory). Then
> mm manages to free 100 MB from page cache, things move on for two seconds,
> then we swap out! about 36 MB and page reclaim also finally decides it
> maybe has too much of page cache and reaps most of it (1.3 GB in one go).
> Then things get going again, although there are still occasional stalls
> such as this (about 30s later):
> 1 3 36688 753192 1208 792344 0 0 35948 32768 435 6625 0 6 61 33
> 0 2 36668 754996 1344 792760 0 0 252 58736 2832 16239 0 1 60 39
> 0 2 36668 750132 1388 796688 0 0 2508 1524 325 959 1 3 68 28
> 1 0 36668 751160 1400 797968 0 0 620 620 460 1470 6 6 50 38
> 1 0 36668 750516 1400 798520 0 0 300 300 412 1764 17 8 75 1
> 1 0 36668 750648 1408 799108 0 0 280 340 423 1816 18 6 73 3
> 1 0 36668 748856 1408 799752 0 0 336 328 409 1788 18 8 75 0
> 1 0 36668 748120 1416 800604 0 0 428 452 407 1723 14 10 75 2
> 1 0 36668 750048 1416 801176 0 0 296 296 405 1779 18 7 75 1
> 0 1 36668 650428 1420 897252 0 0 48100 556 658 1718 10 3 71 15
> 0 2 36668 505444 1424 1037012 0 0 69888 90272 686 1491 1 4 68 27
> 0 1 36668 479264 1428 1063372 0 0 12984 40896 324 674 1 1 76 23
> ...
> I'm not sure what we were blocked on here since there is still plenty of
> free memory (750 MB). These stalls repeat once in a while but things go on.
> Then at about 350s, things just stop:
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 3 1 75280 73564 1844 1503848 0 0 43396 81976 627 1061 0 25 42 32
> 3 3 75280 73344 1852 1504256 0 0 256 20 240 149 0 26 25 49
> 3 3 75280 73344 1852 1504268 0 0 0 0 265 140 0 29 13 58
> 3 3 75280 73468 1852 1504232 0 0 0 0 265 132 0 22 34 44
> 3 3 75280 73468 1852 1504232 0 0 0 0 339 283 0 25 26 49
> 3 3 75280 73468 1852 1504232 0 0 0 0 362 327 0 25 25 50
> 3 3 75280 73468 1852 1504232 0 0 0 0 317 320 0 26 25 49
> 3 3 75280 73468 1852 1504232 0 0 0 0 361 343 0 26 25 50
>
> and nothing really happens for 150 s, except more and more tasks blocking
> in D state (second column).
> 3 6 75272 73416 1872 1503872 0 0 0 0 445 700 0 25 25 50
> 0 7 75264 67940 1884 1509008 64 0 5056 16 481 876 0 22 23 55
> Then suddently things get going again:
> 0 2 75104 76808 1892 1502552 352 0 14292 40456 459 14865 0 2 39 59
> 0 1 75104 75704 1900 1503412 0 0 820 32 405 788 1 1 72 27
> 1 0 75104 76512 1904 1505576 0 0 1068 1072 454 1586 8 7 74 11
>
> I guess this 150 s stall is when kernel barfs the "task blocked for more
> than 30 seconds" messages. And from the traces we know that everyone is
> waiting for PageWriteback or page lock during this time. Also James's vmstat
> report shows that IO is stalled when kswapd is spinning. Really strange.
Just to add, this machine has relatively low amount of memory (1GB).
I've re-run the tests today with cgroups disabled and it ran for 47
'copy' cycles, 27 'copy' cycles and then 35 'copy' cycles.
One extra data point, with maxcpus=1 I get a lockup after 2 'copy'
cycles every time, so it's more predictable than the default 4 processor
configuration.
>
> James in the meantime identified that cgroups are somehow involved. Are you
> using systemd by any chance?
No, I'm using upstart.
> Maybe cgroup IO throttling screws us?
>
> Honza
>
> > > > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
> > > > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
> > > > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
> > > > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
> > > > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
> > > > [ 287.088400] Call Trace:
> > > > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
> > > > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
> > > > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > > > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > > > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > > > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > > > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
> > > > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
> > > > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
> > > > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
> > > > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
> > > > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
> > > > [ 287.088484] Call Trace:
> > > > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > > > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > > > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > > > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > > > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
> > > > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
> > > > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
> > > > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
> > > > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
> > > > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
> > > > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
> > > > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
> > > > [ 287.088577] Call Trace:
> > > > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > > > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > > > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > > > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
> > > > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
> > > > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
> > > > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
> > > > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
> > > > [ 287.088647] Call Trace:
> > > > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
> > > > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
> > > > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
> > > > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
> > > > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
> > > > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
> > > > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
> > > > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
> > > > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
> > > > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
> > > > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
> > > > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
> > > > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
> > > > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
> > > > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
> > > > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
> > > > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
> > > > [ 287.088723] Call Trace:
> > > > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > > > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > > > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > > > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
> > > > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
> > > > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
> > > > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
> > > > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
> > > > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
> > > > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
> > > > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
> > > > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
> > > > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
> > > > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
> > > > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
> > > > --
> > > > Jan Kara <[email protected]>
> > > > SUSE Labs, CR
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> > > > the body of a message to [email protected]
> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
>
On Tue, May 03, 2011 at 09:22:33AM -0500, James Bottomley wrote:
> On Tue, 2011-05-03 at 09:13 -0500, James Bottomley wrote:
> > I've got a ftrace output of kswapd ... it's 500k compressed, so I'll
> > send under separate cover.
>
> Here it is ... it's under 2.6.38.4 vanilla, but the code is similar.
>
I was quiet because I was off trying to reproduce this but not having
much luck. It doesn't seem directly related to filesystems or
cgroups. For example, here is what I see with ext4 without cgroups
2.6.34-vanilla 2.6.37-vanilla 2.6.38-vanilla rc6-vanilla
download tar 70 ( 0.00%) 68 ( 2.94%) 69 ( 1.45%) 70 ( 0.00%)
unpack tar 601 ( 0.00%) 605 (-0.66%) 604 (-0.50%) 605 (-0.66%)
copy source files 319 ( 0.00%) 321 (-0.62%) 320 (-0.31%) 332 (-3.92%)
create tarfile 1368 ( 0.00%) 1372 (-0.29%) 1371 (-0.22%) 1363 ( 0.37%)
delete source dirs 21 ( 0.00%) 21 ( 0.00%) 23 (-8.70%) 22 (-4.55%)
expand tar 263 ( 0.00%) 261 ( 0.77%) 257 ( 2.33%) 259 ( 1.54%)
(all results are in seconds)
When running in cgroups, the results are similar - bit slower but
not remarkably so. ext3 is slower but not enough to count as the bug.
The trace you posted is very short but kswapd is not going to sleep
in it. It's less than a seconds worth on different cpus so it's hard
to draw any conclusion from it other than sleeping_prematurely()
is often deciding that kswapd should not sleep.
So lets consider what keeps it awake.
1. High-order allocations? You machine is using i915 and RPC, something
neither of my test machine uses. i915 is potentially a source for
high-order allocations. I'm attaching a perl script. Please run it as
./watch-highorder.pl --output /tmp/highorders.txt
while you are running tar. When kswapd is running for about 30
seconds, interrupt it with ctrl+c twice in quick succession and
post /tmp/highorders.txt
2. All unreclaimable is not being set or we are not balancing at all.
Can you post the output of sysrq+m while the machine is struggling
please?
3. Slab may not be shrinking for some reason. Can you run a shell
script like this during the whole test and record its output please?
#!/bin/bash
while [ 1 ]; do
echo time: `date +%s`
cat /proc/vmstat
sleep 2
done
Similarly if this is a slab issue, it'd be nice to know who it is so
#!/bin/bash
while [ 1 ]; do
echo time: `date +%s`
cat /proc/slabinfo
sleep $MONITOR_UPDATE_FREQUENCY
done
4. Lets get a better look at what is going on in kswapd
echo 1 > /sys/kernel/debug/tracing/events/vmscan/enable
cat /sys/kernel/debug/tracing/trace_pipe > vmscan-ftrace.txt
Out of curiousity, what's the exact machine you are testing on
because I want to see what sort of hardware combination triggers
this problem? Is the tar local or is it copied over NFS? What is the
configuration of the disk or partition you are copying to?
Thanks
--
Mel Gorman
SUSE Labs
On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> 1. High-order allocations? You machine is using i915 and RPC, something
> neither of my test machine uses. i915 is potentially a source for
> high-order allocations. I'm attaching a perl script. Please run it as
> ./watch-highorder.pl --output /tmp/highorders.txt
> while you are running tar. When kswapd is running for about 30
> seconds, interrupt it with ctrl+c twice in quick succession and
> post /tmp/highorders.txt
>
Attached this time :/
--
Mel Gorman
SUSE Labs
On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> On Tue, May 03, 2011 at 09:22:33AM -0500, James Bottomley wrote:
> > On Tue, 2011-05-03 at 09:13 -0500, James Bottomley wrote:
> > > I've got a ftrace output of kswapd ... it's 500k compressed, so I'll
> > > send under separate cover.
> >
> > Here it is ... it's under 2.6.38.4 vanilla, but the code is similar.
> >
>
> I was quiet because I was off trying to reproduce this but not having
> much luck. It doesn't seem directly related to filesystems or
> cgroups. For example, here is what I see with ext4 without cgroups
>
> 2.6.34-vanilla 2.6.37-vanilla 2.6.38-vanilla rc6-vanilla
> download tar 70 ( 0.00%) 68 ( 2.94%) 69 ( 1.45%) 70 ( 0.00%)
> unpack tar 601 ( 0.00%) 605 (-0.66%) 604 (-0.50%) 605 (-0.66%)
> copy source files 319 ( 0.00%) 321 (-0.62%) 320 (-0.31%) 332 (-3.92%)
> create tarfile 1368 ( 0.00%) 1372 (-0.29%) 1371 (-0.22%) 1363 ( 0.37%)
> delete source dirs 21 ( 0.00%) 21 ( 0.00%) 23 (-8.70%) 22 (-4.55%)
> expand tar 263 ( 0.00%) 261 ( 0.77%) 257 ( 2.33%) 259 ( 1.54%)
>
> (all results are in seconds)
>
> When running in cgroups, the results are similar - bit slower but
> not remarkably so. ext3 is slower but not enough to count as the bug.
>
> The trace you posted is very short but kswapd is not going to sleep
> in it. It's less than a seconds worth on different cpus so it's hard
> to draw any conclusion from it other than sleeping_prematurely()
> is often deciding that kswapd should not sleep.
>
> So lets consider what keeps it awake.
>
> 1. High-order allocations? You machine is using i915 and RPC, something
> neither of my test machine uses. i915 is potentially a source for
> high-order allocations. I'm attaching a perl script. Please run it as
> ./watch-highorder.pl --output /tmp/highorders.txt
> while you are running tar. When kswapd is running for about 30
> seconds, interrupt it with ctrl+c twice in quick succession and
> post /tmp/highorders.txt
>
> 2. All unreclaimable is not being set or we are not balancing at all.
> Can you post the output of sysrq+m while the machine is struggling
> please?
>
> 3. Slab may not be shrinking for some reason. Can you run a shell
> script like this during the whole test and record its output please?
>
> #!/bin/bash
> while [ 1 ]; do
> echo time: `date +%s`
> cat /proc/vmstat
> sleep 2
> done
>
> Similarly if this is a slab issue, it'd be nice to know who it is so
>
> #!/bin/bash
> while [ 1 ]; do
> echo time: `date +%s`
> cat /proc/slabinfo
> sleep $MONITOR_UPDATE_FREQUENCY
> done
>
> 4. Lets get a better look at what is going on in kswapd
>
> echo 1 > /sys/kernel/debug/tracing/events/vmscan/enable
> cat /sys/kernel/debug/tracing/trace_pipe > vmscan-ftrace.txt
>
Also, could you test the patch at https://lkml.org/lkml/2011/3/5/121
please?
--
Mel Gorman
SUSE Labs
On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
>
> 1. High-order allocations? You machine is using i915 and RPC, something
> neither of my test machine uses. i915 is potentially a source for
> high-order allocations. I'm attaching a perl script. Please run it as
> ./watch-highorder.pl --output /tmp/highorders.txt
> while you are running tar. When kswapd is running for about 30
> seconds, interrupt it with ctrl+c twice in quick succession and
> post /tmp/highorders.txt
>
Colin send me this information for his test case at least and I see
11932 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC
=> alloc_pages_current+0xa5/0x110 <ffffffff81149ef5>
=> new_slab+0x1f5/0x290 <ffffffff81153645>
=> __slab_alloc+0x262/0x390 <ffffffff81155192>
=> kmem_cache_alloc+0x115/0x120 <ffffffff81155ab5>
=> mempool_alloc_slab+0x15/0x20 <ffffffff8110e705>
=> mempool_alloc+0x59/0x140 <ffffffff8110ea49>
=> bio_alloc_bioset+0x3e/0xf0 <ffffffff811976ae>
=> bio_alloc+0x15/0x30 <ffffffff81197805>
Colin and James: Did you happen to switch from SLAB to SLUB between
2.6.37 and 2.6.38? My own tests were against SLAB which might be why I
didn't see the problem. Am restarting the tests with SLUB.
--
Mel Gorman
SUSE Labs
On Fri, 2011-05-06 at 08:42 +0100, Mel Gorman wrote:
[...]
Will try tests and post results
> Out of curiousity, what's the exact machine you are testing on
> because I want to see what sort of hardware combination triggers
> this problem? Is the tar local or is it copied over NFS? What is the
> configuration of the disk or partition you are copying to?
It's a Lenovo T420s with 2GB of memory. It's core i5, so it has two
cores each with two threads, so it emulates a 4 cpu system. It's
running Fedora 15 Beta, so the 2.6.38.4 kernel (although I've tried the
default 2.6.38.5 as well, since the fedora i915 updates are a bit
unstable). I'm also booting 2.6.39-rc on it (with and without PREEMPT).
The standard Fedora config is SLUB. Because it's FC15, it has all of
the cgroups stuff enabled for systemd.
Just in case, I've attached the FC15 config ... I modify it slightly for
2.6.39-rc but since the bug shows up in standard FC15, that should be
enough.
James
On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote:
> Colin and James: Did you happen to switch from SLAB to SLUB between
> 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I
> didn't see the problem. Am restarting the tests with SLUB.
Aargh ... I'm an idiot. I should have thought of SLUB immediately ...
it's been causing oopses since debian switched to it.
So I recompiled the 2.6.38.4 stable kernel with SLAB instead of SLUB and
the problem goes away ... at least from three untar runs on a loaded
box ... of course it could manifest a few ms after I send this email ...
There are material differences, as well: SLAB isn't taking my system
down to very low memory on the untar ... it's keeping about 0.5Gb listed
as free. SLUB took that to under 100kb, so it could just be that SLAB
isn't wandering as close to the cliff edge?
James
On Fri, May 06, 2011 at 02:14:37PM -0500, James Bottomley wrote:
> On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote:
> > Colin and James: Did you happen to switch from SLAB to SLUB between
> > 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I
> > didn't see the problem. Am restarting the tests with SLUB.
>
> Aargh ... I'm an idiot. I should have thought of SLUB immediately ...
> it's been causing oopses since debian switched to it.
>
> So I recompiled the 2.6.38.4 stable kernel with SLAB instead of SLUB and
> the problem goes away ... at least from three untar runs on a loaded
> box ... of course it could manifest a few ms after I send this email ...
>
> There are material differences, as well: SLAB isn't taking my system
> down to very low memory on the untar ... it's keeping about 0.5Gb listed
> as free. SLUB took that to under 100kb, so it could just be that SLAB
> isn't wandering as close to the cliff edge?
>
A comparison of watch-highorder.pl with SLAB and SLUB may be
enlightening as well as testing SLUB altering allocate_slab() to read
alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & ~__GFP_NOFAIL;
i.e. try adding the __GFP_NO_KSWAPD. My own tests are still in progress
but I'm still not seeing the problem. I'm installing Fedora on another
test machine at the moment to see if X and other applications have to be
running to pressure high-order allocations properly.
--
Mel Gorman
SUSE Labs
On Fri, 2011-05-06 at 09:07 +0100, Mel Gorman wrote:
> On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> > 1. High-order allocations? You machine is using i915 and RPC, something
> > neither of my test machine uses. i915 is potentially a source for
> > high-order allocations. I'm attaching a perl script. Please run it as
> > ./watch-highorder.pl --output /tmp/highorders.txt
> > while you are running tar. When kswapd is running for about 30
> > seconds, interrupt it with ctrl+c twice in quick succession and
> > post /tmp/highorders.txt
> >
>
> Attached this time :/
Here's the output (loaded with tar, evolution and firefox). The top
trace is different this time because your perl script perturbs the
system quite a bit. This was with your slub allocation fix applied.
James
---
1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> sg_kmalloc+0x24/0x26 <ffffffff81230eb9>
=> __sg_alloc_table+0x63/0x11c <ffffffff81230dbf>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd74b>
=> ra_submit+0x21/0x25 <ffffffff810dda8d>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc5a>
=> page_cache_sync_readahead+0x3d/0x40 <ffffffff810ddd49>
=> generic_file_aio_read+0x27d/0x5e0 <ffffffff810d64d0>
177 instances order=2 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> radix_tree_preload+0x31/0x81 <ffffffff81229399>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> do_wp_page+0x348/0x560 <ffffffff810ef711>
=> handle_pte_fault+0x73b/0x7a5 <ffffffff810f13b7>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
46 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> prepare_creds+0x26/0xae <ffffffff81074d4b>
=> sys_faccessat+0x37/0x162 <ffffffff8111d255>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199>
=> __alloc_skb+0x40/0x133 <ffffffff813b5df7>
=> __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f4a>
1 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> alloc_inode+0x30/0x78 <ffffffff811317f8>
=> new_inode+0x1b/0x4b <ffffffff8113185b>
16 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b6a>
=> handle_mm_fault+0x113/0x1ce <ffffffff810f16f0>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
5 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f4a>
252 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> radix_tree_preload+0x31/0x81 <ffffffff81229399>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> skb_clone+0x50/0x70 <ffffffff813b5d97>
=> packet_rcv+0x101/0x2b2 <ffffffff81441e80>
1 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> alloc_inode+0x30/0x78 <ffffffff811317f8>
=> iget_locked+0x61/0xdd <ffffffff8113239a>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> sg_kmalloc+0x24/0x26 <ffffffff81230eb9>
=> __sg_alloc_table+0x63/0x11c <ffffffff81230dbf>
1 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> copy_process+0x66e/0x10c5 <ffffffff81053961>
=> do_fork+0x104/0x286 <ffffffff810544f7>
1 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> wait_for_helper+0x6b/0xa3 <ffffffff81068fca>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
133 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> khugepaged+0x543/0xf2d <ffffffff81113cc1>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
593 instances order=3 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f498>
=> alloc_inode+0x1d/0x78 <ffffffff811317e5>
2 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_execbuffer2+0x4f/0x12e [i915] <ffffffffa0080f81>
14 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> d_alloc+0x26/0x18d <ffffffff8112e4c5>
=> shmem_file_setup+0xb6/0x160 <ffffffff810e70ad>
38 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mmap_region+0x1b7/0x446 <ffffffff810f6555>
=> do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6a7c>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> idr_pre_get+0x2d/0x6f <ffffffff812266c4>
=> drm_gem_handle_create+0x2f/0x82 [drm] <ffffffffa0024f75>
400 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
9 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199>
=> __alloc_skb+0x40/0x133 <ffffffff813b5df7>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> handle_pte_fault+0x16f/0x7a5 <ffffffff810f0deb>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
=> unix_stream_recvmsg+0x40f/0x536 <ffffffff8143dbe0>
=> sock_aio_read.part.7+0x10d/0x121 <ffffffff813afa54>
2 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> __call_usermodehelper+0x43/0x76 <ffffffff81069132>
=> process_one_work+0x186/0x298 <ffffffff8106b3ca>
14 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
781 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> kmalloc_large_node+0x56/0x95 <ffffffff8146a55d>
=> __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff08>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25c2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d217>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af778>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> __pollwait+0x5e/0xcc <ffffffff8112c9e7>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc50>
=> unix_poll+0x1c/0x9a <ffffffff8143cc71>
=> sock_poll+0x1a/0x1c <ffffffff813ad41a>
=> do_sys_poll+0x1fa/0x386 <ffffffff8112d75d>
48 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> __split_vma+0x6c/0x21b <ffffffff810f5099>
=> split_vma+0x20/0x22 <ffffffff810f59a6>
3 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> sk_prot_alloc+0x37/0x13a <ffffffff813b27d4>
=> sk_alloc+0x2c/0x8e <ffffffff813b2935>
24 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> proc_alloc_inode+0x20/0x91 <ffffffff8116829c>
=> alloc_inode+0x1d/0x78 <ffffffff811317e5>
1 instances order=1 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> ext4_mb_new_blocks+0x118/0x3c2 <ffffffff811b73a7>
=> ext4_ext_map_blocks+0x192b/0x1b5b <ffffffff811b0a5d>
5 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> sock_alloc_inode+0x1d/0xaa <ffffffff813ad605>
=> alloc_inode+0x1d/0x78 <ffffffff811317e5>
3 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> dst_alloc+0x48/0xaa <ffffffff813c6652>
=> __ip_route_output_key+0x561/0x764 <ffffffff813ee4f3>
3 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> __pollwait+0x5e/0xcc <ffffffff8112c9e7>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc50>
=> unix_poll+0x1c/0x9a <ffffffff8143cc71>
=> sock_poll+0x1a/0x1c <ffffffff813ad41a>
=> do_select+0x2fb/0x4f5 <ffffffff8112cf13>
4 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> __split_vma+0x6c/0x21b <ffffffff810f5099>
=> do_munmap+0x194/0x30b <ffffffff810f57ad>
229 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> dup_mm+0x1f8/0x486 <ffffffff81053039>
=> copy_process+0x917/0x10c5 <ffffffff81053c0a>
1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> bprm_mm_init+0x70/0x1a0 <ffffffff811239aa>
=> do_execve+0xd6/0x277 <ffffffff81123dd4>
2 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> find_skb+0x3a/0x82 <ffffffff813d1996>
2 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> __split_vma+0x6c/0x21b <ffffffff810f5099>
=> do_munmap+0x15f/0x30b <ffffffff810f5778>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> handle_pte_fault+0x16f/0x7a5 <ffffffff810f0deb>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
501 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2d1>
=> do_filp_open+0xe7/0x60a <ffffffff81129bcf>
18 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d61>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144df4>
1370 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> d_alloc+0x26/0x18d <ffffffff8112e4c5>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d0e>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> kzalloc.constprop.1+0x13/0x15 [i915] <ffffffffa007fbfd>
=> i915_gem_do_execbuffer+0x306/0x1116 [i915] <ffffffffa007ff05>
1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> taskstats_exit+0x5b/0x2e3 <ffffffff810b0810>
=> do_exit+0x22a/0x6ef <ffffffff8105841f>
17 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> d_alloc+0x26/0x18d <ffffffff8112e4c5>
=> proc_fill_cache+0x82/0x135 <ffffffff8116903b>
99 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> inet_twsk_alloc+0x31/0x11e <ffffffff813f8a16>
=> tcp_time_wait+0xb9/0x29a <ffffffff8140e24f>
26 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff81196376>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d5408>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d617d>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6205>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249>
6 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> kthreadd+0xe7/0x124 <ffffffff8106f61f>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
2 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199>
=> __alloc_skb+0x40/0x133 <ffffffff813b5df7>
=> find_skb+0x3a/0x82 <ffffffff813d1996>
131 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3047>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30cd>
6 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> d_alloc+0x26/0x18d <ffffffff8112e4c5>
=> d_alloc_pseudo+0x18/0x46 <ffffffff8112e6a7>
6 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d61>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144df4>
11 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> shmem_alloc_inode+0x1a/0x2f <ffffffff810e58a0>
=> alloc_inode+0x1d/0x78 <ffffffff811317e5>
4 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249>
680 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
8 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> copy_process+0xc6/0x10c5 <ffffffff810533b9>
=> do_fork+0x104/0x286 <ffffffff810544f7>
4 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> do_huge_pmd_wp_page+0x15a/0x637 <ffffffff81114a01>
=> handle_mm_fault+0x169/0x1ce <ffffffff810f1746>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
193 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
140358 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
3 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
17 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd74b>
=> ra_submit+0x21/0x25 <ffffffff810dda8d>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc5a>
=> page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddce4>
=> generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6510>
9 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> kzalloc.constprop.15+0x13/0x15 [i915] <ffffffffa007a956>
=> i915_gem_alloc_object+0x27/0x111 [i915] <ffffffffa007e4b3>
14 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2d1>
=> alloc_file+0x1e/0xbf <ffffffff8111f3b6>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> kmalloc_order_trace+0x2c/0x5e <ffffffff8110c718>
=> __kmalloc+0x37/0x10d <ffffffff8110df41>
=> kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c>
=> tracing_open_pipe+0x40/0x147 <ffffffff810b907a>
=> __dentry_open+0x161/0x283 <ffffffff8111c95c>
High-order normal allocations: 145450
High-order atomic allocations: 927
On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote:
> On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> >
> > 1. High-order allocations? You machine is using i915 and RPC, something
> > neither of my test machine uses. i915 is potentially a source for
> > high-order allocations. I'm attaching a perl script. Please run it as
> > ./watch-highorder.pl --output /tmp/highorders.txt
> > while you are running tar. When kswapd is running for about 30
> > seconds, interrupt it with ctrl+c twice in quick succession and
> > post /tmp/highorders.txt
> >
>
> Colin send me this information for his test case at least and I see
>
> 11932 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC
> => alloc_pages_current+0xa5/0x110 <ffffffff81149ef5>
> => new_slab+0x1f5/0x290 <ffffffff81153645>
> => __slab_alloc+0x262/0x390 <ffffffff81155192>
> => kmem_cache_alloc+0x115/0x120 <ffffffff81155ab5>
> => mempool_alloc_slab+0x15/0x20 <ffffffff8110e705>
> => mempool_alloc+0x59/0x140 <ffffffff8110ea49>
> => bio_alloc_bioset+0x3e/0xf0 <ffffffff811976ae>
> => bio_alloc+0x15/0x30 <ffffffff81197805>
>
> Colin and James: Did you happen to switch from SLAB to SLUB between
> 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I
> didn't see the problem. Am restarting the tests with SLUB.
So I tested with SLAB instead of SLUB and I reliably ran my copy test
for 4+ hours with several hundred iterations of the test. (Apologies
for taking time to respond, but I was travelling).
>
On Mon, May 09, 2011 at 01:16:20PM -0500, James Bottomley wrote:
> On Fri, 2011-05-06 at 09:07 +0100, Mel Gorman wrote:
> > On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> > > 1. High-order allocations? You machine is using i915 and RPC, something
> > > neither of my test machine uses. i915 is potentially a source for
> > > high-order allocations. I'm attaching a perl script. Please run it as
> > > ./watch-highorder.pl --output /tmp/highorders.txt
> > > while you are running tar. When kswapd is running for about 30
> > > seconds, interrupt it with ctrl+c twice in quick succession and
> > > post /tmp/highorders.txt
> > >
> >
> > Attached this time :/
>
> Here's the output (loaded with tar, evolution and firefox). The top
> trace is different this time because your perl script perturbs the
> system quite a bit. This was with your slub allocation fix applied.
>
I note that certain flags like __GFP_NO_KSWAPD are not recognised by
tracing which might explain why they are missing from the script output.
I regret the script perturbs the system quite a bit. It's possible it
can be made better by filtering events but it's not high on the list of
things to do.
How does the output compare without the fix? I can't find a similar
report in my inbox.
Does the fix help the system when the perl script is not running?
> 177 instances order=2 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => radix_tree_preload+0x31/0x81 <ffffffff81229399>
> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
>
Ouch.
> 46 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => prepare_creds+0x26/0xae <ffffffff81074d4b>
> => sys_faccessat+0x37/0x162 <ffffffff8111d255>
>
Less ouch, but still.
> 252 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => radix_tree_preload+0x31/0x81 <ffffffff81229399>
> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
>
Ouch again.
> 593 instances order=3 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f498>
> => alloc_inode+0x1d/0x78 <ffffffff811317e5>
>
Again, filesystem-related calls are hitting high-order paths quite a
bit.
> 781 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => kmalloc_large_node+0x56/0x95 <ffffffff8146a55d>
> => __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff08>
> => __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25c2>
> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d217>
> => __sock_sendmsg+0x69/0x76 <ffffffff813af778>
>
A number of network paths are also being hit although this is the worst.
> 501 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => get_empty_filp+0x7a/0x141 <ffffffff8111f2d1>
> => do_filp_open+0xe7/0x60a <ffffffff81129bcf>
>
More filesystem impairment.
> 1370 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => d_alloc+0x26/0x18d <ffffffff8112e4c5>
> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d0e>
>
*cries*
> 140358 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
> => mempool_alloc+0x68/0x116 <ffffffff810d70f6>
Wonder which pool this is!
It goes on. A number of filesystem and network paths are being hit
with high-order allocs. i915 was a red herring, it's present but not
in massive numbers. The filesystem, network and mempool allocations
are likely to be kicking kswapd awake frequently and hurting overall
system performance as a result.
I really would like to hear if the fix makes a big difference or
if we need to consider forcing SLUB high-order allocations bailing
at the first sign of trouble (e.g. by masking out __GFP_WAIT in
allocate_slab). Even with the fix applied, kswapd might be waking up
less but processes will still be getting stalled in direct compaction
and direct reclaim so it would still be jittery.
> High-order normal allocations: 145450
> High-order atomic allocations: 927
>
I bet a shiny penny that the high-order allocations for SLAB are lower
than this
--
Mel Gorman
SUSE Labs
On Tue, May 10, 2011 at 1:21 PM, Mel Gorman <[email protected]> wrote:
> It goes on. A number of filesystem and network paths are being hit
> with high-order allocs. i915 was a red herring, it's present but not
> in massive numbers. The filesystem, network and mempool allocations
> are likely to be kicking kswapd awake frequently and hurting overall
> system performance as a result.
>
> I really would like to hear if the fix makes a big difference or
> if we need to consider forcing SLUB high-order allocations bailing
> at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> allocate_slab). Even with the fix applied, kswapd might be waking up
> less but processes will still be getting stalled in direct compaction
> and direct reclaim so it would still be jittery.
Yes, sounds reasonable to me.
Pekka
On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> I really would like to hear if the fix makes a big difference or
> if we need to consider forcing SLUB high-order allocations bailing
> at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> allocate_slab). Even with the fix applied, kswapd might be waking up
> less but processes will still be getting stalled in direct compaction
> and direct reclaim so it would still be jittery.
"the fix" being this
https://lkml.org/lkml/2011/3/5/121
In addition to your GFP_KSWAPD one?
OK, will retry with that.
James
On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > I really would like to hear if the fix makes a big difference or
> > if we need to consider forcing SLUB high-order allocations bailing
> > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > allocate_slab). Even with the fix applied, kswapd might be waking up
> > less but processes will still be getting stalled in direct compaction
> > and direct reclaim so it would still be jittery.
>
> "the fix" being this
>
> https://lkml.org/lkml/2011/3/5/121
>
Drop this for the moment. It was a long shot at best and there is little
evidence the problem is in this area.
I'm attaching two patches. The first is the NO_KSWAPD one to stop
kswapd being woken up by SLUB using speculative high-orders. The second
one is more drastic and prevents slub entering direct reclaim or
compaction. It applies on top of patch 1. These are both untested and
afraid are a bit rushed as well :(
--
Mel Gorman
SUSE Labs
On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > I really would like to hear if the fix makes a big difference or
> > > if we need to consider forcing SLUB high-order allocations bailing
> > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > less but processes will still be getting stalled in direct compaction
> > > and direct reclaim so it would still be jittery.
> >
> > "the fix" being this
> >
> > https://lkml.org/lkml/2011/3/5/121
> >
>
> Drop this for the moment. It was a long shot at best and there is little
> evidence the problem is in this area.
>
> I'm attaching two patches. The first is the NO_KSWAPD one to stop
> kswapd being woken up by SLUB using speculative high-orders. The second
> one is more drastic and prevents slub entering direct reclaim or
> compaction. It applies on top of patch 1. These are both untested and
> afraid are a bit rushed as well :(
Preliminary results with both patches applied still show kswapd
periodically going up to 99% but it doesn't stay there, it comes back
down again (and, obviously, the system doesn't hang).
This is sysrq-M from a couple of times when it went up there:
[ 426.736958] SysRq : Show Memory
[ 426.736974] Mem-Info:
[ 426.736977] Node 0 DMA per-cpu:
[ 426.736983] CPU 0: hi: 0, btch: 1 usd: 0
[ 426.736986] CPU 1: hi: 0, btch: 1 usd: 0
[ 426.736989] CPU 2: hi: 0, btch: 1 usd: 0
[ 426.736993] CPU 3: hi: 0, btch: 1 usd: 0
[ 426.736996] Node 0 DMA32 per-cpu:
[ 426.737002] CPU 0: hi: 186, btch: 31 usd: 169
[ 426.737005] CPU 1: hi: 186, btch: 31 usd: 40
[ 426.737009] CPU 2: hi: 186, btch: 31 usd: 166
[ 426.737012] CPU 3: hi: 186, btch: 31 usd: 168
[ 426.737015] Node 0 Normal per-cpu:
[ 426.737020] CPU 0: hi: 0, btch: 1 usd: 0
[ 426.737024] CPU 1: hi: 0, btch: 1 usd: 0
[ 426.737027] CPU 2: hi: 0, btch: 1 usd: 0
[ 426.737030] CPU 3: hi: 0, btch: 1 usd: 0
[ 426.737036] active_anon:108658 inactive_anon:37031 isolated_anon:0
[ 426.737037] active_file:32006 inactive_file:41051 isolated_file:32
[ 426.737038] unevictable:8 dirty:41202 writeback:204 unstable:0
[ 426.737039] free:191520 slab_reclaimable:8490 slab_unreclaimable:27477
[ 426.737039] mapped:9176 shmem:26412 pagetables:5427 bounce:0
[ 426.737051] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 426.737064] lowmem_reserve[]: 0 1856 1862 1862
[ 426.737078] Node 0 DMA32 free:757892kB min:66820kB low:83524kB high:100228kB active_anon:434632kB inactive_anon:148112kB active_file:127784kB inactive_file:156924kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:1901408kB mlocked:32kB dirty:164808kB writeback:816kB mapped:36704kB shmem:105648kB slab_reclaimable:33676kB slab_unreclaimable:108372kB kernel_stack:2304kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
[ 426.737092] lowmem_reserve[]: 0 0 5 5
[ 426.737106] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 426.737119] lowmem_reserve[]: 0 0 0 0
[ 426.737132] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
[ 426.737174] Node 0 DMA32: 945*4kB 585*8kB 6469*16kB 4871*32kB 3189*64kB 1338*128kB 228*256kB 58*512kB 24*1024kB 1*2048kB 0*4096kB = 757884kB
[ 426.737227] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
[ 426.737257] 99595 total pagecache pages
[ 426.737260] 0 pages in swap cache
[ 426.737263] Swap cache stats: add 0, delete 0, find 0/0
[ 426.737266] Free swap = 3768316kB
[ 426.737268] Total swap = 3768316kB
[ 426.744603] 525808 pages RAM
[ 426.744612] 57618 pages reserved
[ 426.744614] 141551 pages shared
[ 426.744617] 186065 pages non-shared
[ 472.301810] SysRq : Show Memory
[ 472.301826] Mem-Info:
[ 472.301829] Node 0 DMA per-cpu:
[ 472.301835] CPU 0: hi: 0, btch: 1 usd: 0
[ 472.301839] CPU 1: hi: 0, btch: 1 usd: 0
[ 472.301842] CPU 2: hi: 0, btch: 1 usd: 0
[ 472.301845] CPU 3: hi: 0, btch: 1 usd: 0
[ 472.301848] Node 0 DMA32 per-cpu:
[ 472.301854] CPU 0: hi: 186, btch: 31 usd: 184
[ 472.301857] CPU 1: hi: 186, btch: 31 usd: 46
[ 472.301860] CPU 2: hi: 186, btch: 31 usd: 158
[ 472.301863] CPU 3: hi: 186, btch: 31 usd: 163
[ 472.301866] Node 0 Normal per-cpu:
[ 472.301871] CPU 0: hi: 0, btch: 1 usd: 0
[ 472.301874] CPU 1: hi: 0, btch: 1 usd: 0
[ 472.301878] CPU 2: hi: 0, btch: 1 usd: 0
[ 472.301881] CPU 3: hi: 0, btch: 1 usd: 0
[ 472.301886] active_anon:107673 inactive_anon:37031 isolated_anon:0
[ 472.301887] active_file:31533 inactive_file:33323 isolated_file:32
[ 472.301888] unevictable:8 dirty:26256 writeback:6475 unstable:0
[ 472.301889] free:198742 slab_reclaimable:9347 slab_unreclaimable:28647
[ 472.301889] mapped:8307 shmem:26412 pagetables:5427 bounce:0
[ 472.301901] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 472.301915] lowmem_reserve[]: 0 1856 1862 1862
[ 472.301928] Node 0 DMA32 free:786780kB min:66820kB low:83524kB high:100228kB active_anon:430692kB inactive_anon:148112kB active_file:125892kB inactive_file:126012kB unevictable:32kB isolated(anon):0kB isolated(file):128kB present:1901408kB mlocked:32kB dirty:105024kB writeback:25900kB mapped:33228kB shmem:105648kB slab_reclaimable:37104kB slab_unreclaimable:113052kB kernel_stack:2288kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3196 all_unreclaimable? no
[ 472.301943] lowmem_reserve[]: 0 0 5 5
[ 472.301956] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 472.301968] lowmem_reserve[]: 0 0 0 0
[ 472.301982] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
[ 472.302015] Node 0 DMA32: 6121*4kB 1912*8kB 5094*16kB 4920*32kB 3168*64kB 1381*128kB 262*256kB 68*512kB 24*1024kB 1*2048kB 0*4096kB = 786756kB
[ 472.302048] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
[ 472.302090] 91312 total pagecache pages
[ 472.302093] 0 pages in swap cache
[ 472.302096] Swap cache stats: add 0, delete 0, find 0/0
[ 472.302099] Free swap = 3768316kB
[ 472.302102] Total swap = 3768316kB
[ 472.309521] 525808 pages RAM
[ 472.309529] 57618 pages reserved
[ 472.309548] 142496 pages shared
[ 472.309551] 177182 pages non-shared
I'll finish off this verification, and then re-run with the watch-highorder script running.
James
On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote:
> On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > > I really would like to hear if the fix makes a big difference or
> > > > if we need to consider forcing SLUB high-order allocations bailing
> > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > > less but processes will still be getting stalled in direct compaction
> > > > and direct reclaim so it would still be jittery.
> > >
> > > "the fix" being this
> > >
> > > https://lkml.org/lkml/2011/3/5/121
> > >
> >
> > Drop this for the moment. It was a long shot at best and there is little
> > evidence the problem is in this area.
> >
> > I'm attaching two patches. The first is the NO_KSWAPD one to stop
> > kswapd being woken up by SLUB using speculative high-orders. The second
> > one is more drastic and prevents slub entering direct reclaim or
> > compaction. It applies on top of patch 1. These are both untested and
> > afraid are a bit rushed as well :(
>
> Preliminary results with both patches applied still show kswapd
> periodically going up to 99% but it doesn't stay there, it comes back
> down again (and, obviously, the system doesn't hang).
This is a second run with the watch highorders.
At the end of the run, the system hung temporarily and now comes back
with CPU3 spinning in all system time at kswapd shrink_slab
James
---
97 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
127 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
2 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d81>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e14>
47 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> radix_tree_preload+0x31/0x81 <ffffffff812293b9>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc_trace+0x7b/0xd8 <ffffffff8110e2ce>
=> inode_doinit_with_dentry+0x124/0x45c <ffffffff811e96db>
=> selinux_d_instantiate+0x1c/0x1e <ffffffff811e9a2f>
3 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdb>
=> kzalloc.constprop.15+0x13/0x15 [i915] <ffffffffa007a956>
=> i915_gem_alloc_object+0x27/0x111 [i915] <ffffffffa007e4b3>
4 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mmap_region+0x1b7/0x446 <ffffffff810f6575>
=> do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6a9c>
27 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76b>
=> ra_submit+0x21/0x25 <ffffffff810ddaad>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7a>
=> page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd04>
=> generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6510>
159 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
446 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f1>
=> do_filp_open+0xe7/0x60a <ffffffff81129bef>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> arch_dup_task_struct+0x42/0x8b <ffffffff81010f47>
=> copy_process+0xf9/0x10c5 <ffffffff810533ec>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sk_stream_alloc_skb+0x38/0xd3 <ffffffff813fc5a9>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff81196396>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d5408>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d617d>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6205>
25 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f1>
=> alloc_file+0x1e/0xbf <ffffffff8111f3d6>
115 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d81>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e14>
56 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> d_alloc+0x26/0x18d <ffffffff8112e4e5>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d2e>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> alloc_buffer_head+0x21/0x49 <ffffffff81141405>
=> alloc_page_buffers+0x32/0xce <ffffffff81141533>
9 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107021>
=> khugepaged+0x543/0xf2d <ffffffff81113ce1>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
347 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4b8>
=> alloc_inode+0x1d/0x78 <ffffffff81131805>
56 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff81196396>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d5408>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d617d>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6205>
60 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> dup_mm+0x1f8/0x486 <ffffffff81053039>
=> copy_process+0x917/0x10c5 <ffffffff81053c0a>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> bprm_mm_init+0x70/0x1a0 <ffffffff811239ca>
=> do_execve+0xd6/0x277 <ffffffff81123df4>
6 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107021>
=> do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b8a>
=> handle_mm_fault+0x113/0x1ce <ffffffff810f1710>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
6 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> do_brk+0x1cd/0x293 <ffffffff810f5fd8>
=> load_elf_binary+0xd2d/0x175f <ffffffff8115ccf2>
4 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> kthreadd+0xe7/0x124 <ffffffff8106f61f>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
4 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1b9>
=> __alloc_skb+0x40/0x133 <ffffffff813b5e17>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
534 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> kmalloc_large_node+0x56/0x95 <ffffffff8146a57d>
=> __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff28>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> kmalloc_order_trace+0x2c/0x5e <ffffffff8110c738>
=> __kmalloc+0x37/0x10d <ffffffff8110df61>
=> kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c>
=> tracing_open_pipe+0x40/0x147 <ffffffff810b907a>
=> __dentry_open+0x161/0x283 <ffffffff8111c97c>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76b>
=> ra_submit+0x21/0x25 <ffffffff810ddaad>
=> filemap_fault+0x17b/0x36c <ffffffff810d6ada>
=> __do_fault+0x55/0x377 <ffffffff810ee887>
=> handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f43>
55 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
8 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> shmem_alloc_inode+0x1a/0x2f <ffffffff810e58c0>
=> alloc_inode+0x1d/0x78 <ffffffff81131805>
563 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
12 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> __split_vma+0x6c/0x21b <ffffffff810f50b9>
=> split_vma+0x20/0x22 <ffffffff810f59c6>
5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> __pollwait+0x5e/0xcc <ffffffff8112ca07>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc70>
=> unix_poll+0x1c/0x9a <ffffffff8143cc91>
=> sock_poll+0x1a/0x1c <ffffffff813ad43a>
=> do_select+0x2fb/0x4f5 <ffffffff8112cf33>
145216 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
48 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> proc_alloc_inode+0x20/0x91 <ffffffff811682bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131805>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> copy_process+0xc6/0x10c5 <ffffffff810533b9>
=> do_fork+0x104/0x286 <ffffffff810544f7>
31 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> prepare_creds+0x26/0xae <ffffffff81074d4b>
=> sys_faccessat+0x37/0x162 <ffffffff8111d275>
1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f6a>
High-order normal allocations: 147471
High-order atomic allocations: 720
On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote:
> On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote:
> > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > > > I really would like to hear if the fix makes a big difference or
> > > > > if we need to consider forcing SLUB high-order allocations bailing
> > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > > > less but processes will still be getting stalled in direct compaction
> > > > > and direct reclaim so it would still be jittery.
> > > >
> > > > "the fix" being this
> > > >
> > > > https://lkml.org/lkml/2011/3/5/121
> > > >
> > >
> > > Drop this for the moment. It was a long shot at best and there is little
> > > evidence the problem is in this area.
> > >
> > > I'm attaching two patches. The first is the NO_KSWAPD one to stop
> > > kswapd being woken up by SLUB using speculative high-orders. The second
> > > one is more drastic and prevents slub entering direct reclaim or
> > > compaction. It applies on top of patch 1. These are both untested and
> > > afraid are a bit rushed as well :(
> >
> > Preliminary results with both patches applied still show kswapd
> > periodically going up to 99% but it doesn't stay there, it comes back
> > down again (and, obviously, the system doesn't hang).
>
> This is a second run with the watch highorders.
>
> At the end of the run, the system hung temporarily and now comes back
> with CPU3 spinning in all system time at kswapd shrink_slab
Here's a trace in the same situation with the ftrace stack entries
bumped to 16 as requested on IRC. There was no hang for this one.
James
---
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> shmem_alloc_inode+0x1a/0x2f <ffffffff810e58c4>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> new_inode+0x1b/0x4b <ffffffff8113187f>
=> shmem_get_inode+0x45/0x1f6 <ffffffff810e49b3>
=> shmem_file_setup+0xe4/0x160 <ffffffff810e70ff>
=> drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc>
=> i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd>
=> i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
701 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> scsi_sg_alloc+0x2d/0x2f <ffffffff812f9e90>
=> __sg_alloc_table+0x63/0x11c <ffffffff81230ddf>
=> scsi_alloc_sgtable+0x2a/0x4f <ffffffff812f9ebc>
=> scsi_init_sgtable+0x21/0x61 <ffffffff812f9f02>
=> scsi_init_io+0x32/0x13b <ffffffff812f9f74>
=> scsi_setup_fs_cmnd+0xb9/0xbe <ffffffff812fa1b0>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
189 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> get_request+0x180/0x262 <ffffffff812124f6>
=> get_request_wait+0x35/0x193 <ffffffff81212f71>
=> __make_request+0x2b4/0x400 <ffffffff81213898>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> submit_bh+0xe6/0x105 <ffffffff81140a6f>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
=> block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> inet_twsk_alloc+0x31/0x11e <ffffffff813f8a36>
=> tcp_time_wait+0xb9/0x29a <ffffffff8140e26f>
=> tcp_fin+0x7b/0x10f <ffffffff8140191a>
=> tcp_data_queue+0x1e4/0x876 <ffffffff8140207f>
=> tcp_rcv_state_process+0x880/0x8cf <ffffffff81404f67>
=> tcp_v4_do_rcv+0x312/0x363 <ffffffff8140b698>
=> tcp_v4_rcv+0x434/0x69b <ffffffff8140d209>
=> ip_local_deliver_finish+0x13d/0x1ae <ffffffff813f104d>
=> NF_HOOK.constprop.3+0x51/0x58 <ffffffff813f0eb2>
=> ip_local_deliver+0x4f/0x54 <ffffffff813f1205>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333>
=> ext4_iget+0x58/0x64c <ffffffff81192a48>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> __lookup_hash+0x7f/0x88 <ffffffff81127eaa>
9 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1bd>
=> __alloc_skb+0x40/0x133 <ffffffff813b5e17>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d>
=> do_readv_writev+0xa7/0x11e <ffffffff8111ea03>
=> vfs_writev+0x45/0x47 <ffffffff8111eabf>
=> sys_writev+0x4a/0x93 <ffffffff8111ebe2>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> sk_prot_alloc+0x37/0x13a <ffffffff813b27f4>
=> sk_alloc+0x2c/0x8e <ffffffff813b2955>
=> unix_create1+0x4e/0x14d <ffffffff8143cae7>
=> unix_stream_connect+0x98/0x431 <ffffffff8143ecc6>
=> sys_connect+0x98/0xd6 <ffffffff813b0437>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
5 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> alloc_file+0x1e/0xbf <ffffffff8111f3da>
=> create_read_pipe+0x2b/0x5b <ffffffff81125ffe>
=> do_pipe_flags+0x4e/0xf4 <ffffffff8112607c>
=> sys_pipe2+0x1c/0x5c <ffffffff8112613e>
=> sys_pipe+0x10/0x12 <ffffffff8112618e>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
6 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f6a>
=> e1000_alloc_rx_buffers+0x8b/0x14e [e1000e] <ffffffffa01bfb58>
=> e1000_clean_rx_irq+0x2a6/0x2df [e1000e] <ffffffffa01c05a9>
=> e1000_clean+0x73/0x228 [e1000e] <ffffffffa01c0a07>
=> net_rx_action+0xa9/0x1ac <ffffffff813bfea2>
=> __do_softirq+0xd2/0x19d <ffffffff8105abe0>
=> call_softirq+0x1c/0x30 <ffffffff8100aa9c>
=> do_softirq+0x46/0x81 <ffffffff8100c0c1>
=> irq_exit+0x49/0x8b <ffffffff8105ad64>
48 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d540c>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d6181>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6209>
=> ext4_file_write+0x1dc/0x234 <ffffffff8118e898>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
=> vfs_write+0xac/0xf3 <ffffffff8111e3e5>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76f>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> filemap_fault+0x17b/0x36c <ffffffff810d6ade>
=> __do_fault+0x55/0x377 <ffffffff810ee88b>
=> handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
16 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> do_read_cache_page+0x63/0x127 <ffffffff810d59f4>
=> read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6>
=> i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad>
=> i915_gem_object_pin+0xc7/0x13c [i915] <ffffffffa007d609>
=> i915_gem_execbuffer_reserve+0x1b1/0x2fc [i915] <ffffffffa007f76d>
=> i915_gem_do_execbuffer+0x45b/0x1116 [i915] <ffffffffa008005a>
=> i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe>
409 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> jbd2_journal_commit_transaction+0x6e4/0x1052 <ffffffff811c56a6>
=> kjournald2+0xc9/0x20a <ffffffff811c9764>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> mpage_readpages+0xf0/0x104 <ffffffff81148e05>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> copy_vma+0xe3/0x1e9 <ffffffff810f5cd3>
=> move_vma+0x8a/0x1fb <ffffffff810f7edb>
=> do_mremap+0x40e/0x46a <ffffffff810f845a>
=> sys_mremap+0x54/0x76 <ffffffff810f850a>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> do_read_cache_page+0x63/0x127 <ffffffff810d59f4>
=> read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6>
=> i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad>
=> i915_gem_fault+0x85/0x18e [i915] <ffffffffa007d287>
=> __do_fault+0x55/0x377 <ffffffff810ee88b>
=> handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc>
1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdf>
=> kzalloc.constprop.1+0x13/0x15 [i915] <ffffffffa007fbfd>
=> i915_gem_do_execbuffer+0x306/0x1116 [i915] <ffffffffa007ff05>
=> i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
5 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> copy_process+0xc6/0x10c5 <ffffffff810533b9>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ext4_mb_init_cache+0x1e6/0x52b <ffffffff811b45c8>
=> ext4_mb_load_buddy+0x134/0x30f <ffffffff811b4b1c>
=> ext4_mb_regular_allocator+0x13c/0x275 <ffffffff811b6637>
=> ext4_mb_new_blocks+0x17f/0x3c2 <ffffffff811b7432>
=> ext4_ext_map_blocks+0x192b/0x1b5b <ffffffff811b0a81>
102 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
=> __breadahead+0x2e/0x39 <ffffffff81141c86>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_iget+0x58/0x64c <ffffffff81192a48>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
1 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> get_request+0x180/0x262 <ffffffff812124f6>
=> get_request_wait+0x35/0x193 <ffffffff81212f71>
=> __make_request+0x2b4/0x400 <ffffffff81213898>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> mpage_readpages+0xf0/0x104 <ffffffff81148e05>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
3 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> __split_vma+0x6c/0x21b <ffffffff810f50bd>
=> split_vma+0x20/0x22 <ffffffff810f59ca>
=> mprotect_fixup+0x105/0x65b <ffffffff810f70ad>
=> sys_mprotect+0x197/0x209 <ffffffff810f779a>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
14 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
=> __scsi_get_command+0x1b/0x96 <ffffffff812f3150>
=> scsi_get_command+0x3d/0xa5 <ffffffff812f3283>
=> scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
=> scsi_request_fn+0x82/0x421 <ffffffff812f9a5f>
=> __generic_unplug_device+0x34/0x38 <ffffffff81212f38>
=> generic_unplug_device+0x2d/0x3d <ffffffff812131f0>
7 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> __split_vma+0x6c/0x21b <ffffffff810f50bd>
=> split_vma+0x20/0x22 <ffffffff810f59ca>
=> mprotect_fixup+0x125/0x65b <ffffffff810f70cd>
=> sys_mprotect+0x197/0x209 <ffffffff810f779a>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> find_or_create_page+0x5a/0x8a <ffffffff810d5b6d>
=> __getblk+0x137/0x23a <ffffffff81141b55>
=> ext4_getblk+0x8e/0x153 <ffffffff811957c0>
=> ext4_bread+0x16/0x66 <ffffffff8119589b>
=> ext4_mkdir+0x159/0x33e <ffffffff8119b786>
=> vfs_mkdir+0x5f/0x9b <ffffffff81128b01>
=> sys_mkdirat+0x97/0xe8 <ffffffff8112a3dd>
4 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_sendmsg+0xa1/0xb6 <ffffffff813af90f>
=> sys_sendmsg+0x237/0x2b5 <ffffffff813b0cd9>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
2 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d>
=> do_readv_writev+0xa7/0x11e <ffffffff8111ea03>
=> vfs_writev+0x45/0x47 <ffffffff8111eabf>
=> sys_writev+0x4a/0x93 <ffffffff8111ebe2>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ext4_mb_init_cache+0x1e6/0x52b <ffffffff811b45c8>
=> ext4_mb_init_group+0x100/0x281 <ffffffff8146b9f8>
=> ext4_mb_good_group+0x58/0xdb <ffffffff811b4965>
=> ext4_mb_regular_allocator+0x125/0x275 <ffffffff811b6620>
=> ext4_mb_new_blocks+0x17f/0x3c2 <ffffffff811b7432>
39 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
=> __breadahead+0x2e/0x39 <ffffffff81141c86>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_reserve_inode_write+0x26/0x83 <ffffffff81193de1>
66 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> scsi_sg_alloc+0x2d/0x2f <ffffffff812f9e90>
=> __sg_alloc_table+0x63/0x11c <ffffffff81230ddf>
=> scsi_alloc_sgtable+0x2a/0x4f <ffffffff812f9ebc>
=> scsi_init_sgtable+0x21/0x61 <ffffffff812f9f02>
=> scsi_init_io+0x32/0x13b <ffffffff812f9f74>
=> scsi_setup_fs_cmnd+0xb9/0xbe <ffffffff812fa1b0>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
2 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> do_read_cache_page+0x48/0x127 <ffffffff810d59d9>
=> read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6>
=> i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad>
=> i915_gem_object_pin+0xc7/0x13c [i915] <ffffffffa007d609>
=> i915_gem_execbuffer_reserve+0x1b1/0x2fc [i915] <ffffffffa007f76d>
=> i915_gem_do_execbuffer+0x45b/0x1116 [i915] <ffffffffa008005a>
=> i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
148 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107025>
=> khugepaged+0x543/0xf2d <ffffffff81113ce5>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
526 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> new_inode+0x1b/0x4b <ffffffff8113187f>
=> ext4_new_inode+0xa1/0xcf2 <ffffffff8118f9cb>
=> ext4_create+0xbc/0x13e <ffffffff8119b32e>
=> vfs_create+0x6c/0x8e <ffffffff81128c64>
=> do_last+0x166/0x2df <ffffffff81129993>
=> do_filp_open+0x2ac/0x60a <ffffffff81129db8>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_openat+0x11/0x13 <ffffffff8111da20>
17 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
=> __scsi_get_command+0x1b/0x96 <ffffffff812f3150>
=> scsi_get_command+0x3d/0xa5 <ffffffff812f3283>
=> scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
=> scsi_request_fn+0x82/0x421 <ffffffff812f9a5f>
=> __generic_unplug_device+0x34/0x38 <ffffffff81212f38>
=> __make_request+0x3a3/0x400 <ffffffff81213987>
43 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
=> __breadahead+0x2e/0x39 <ffffffff81141c86>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_get+0x57/0x213 <ffffffff811baf2b>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdf>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_execbuffer2+0x4f/0x12e [i915] <ffffffffa0080f81>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
6 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mmap_region+0x1b7/0x446 <ffffffff810f6579>
=> do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6aa0>
=> sys_mmap_pgoff+0x115/0x164 <ffffffff810f6c0f>
=> sys_mmap+0x22/0x24 <ffffffff8100da68>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
763 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> do_filp_open+0xe7/0x60a <ffffffff81129bf3>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_open+0x20/0x22 <ffffffff8111da0d>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
59 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> new_inode+0x1b/0x4b <ffffffff8113187f>
=> ext4_new_inode+0xa1/0xcf2 <ffffffff8118f9cb>
=> ext4_mkdir+0x10b/0x33e <ffffffff8119b738>
=> vfs_mkdir+0x5f/0x9b <ffffffff81128b01>
=> sys_mkdirat+0x97/0xe8 <ffffffff8112a3dd>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
83 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> dup_mm+0x1f8/0x486 <ffffffff81053039>
=> copy_process+0x917/0x10c5 <ffffffff81053c0a>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
4 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d85>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> mpage_alloc+0x2a/0x84 <ffffffff8114866b>
=> do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e>
33 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
=> __scsi_get_command+0x1b/0x96 <ffffffff812f3150>
=> scsi_get_command+0x3d/0xa5 <ffffffff812f3283>
=> scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
=> scsi_request_fn+0x82/0x421 <ffffffff812f9a5f>
=> __generic_unplug_device+0x34/0x38 <ffffffff81212f38>
=> elv_insert+0x172/0x17b <ffffffff8120dae8>
1 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107025>
=> do_wp_page+0x348/0x560 <ffffffff810ef735>
=> handle_pte_fault+0x73b/0x7a5 <ffffffff810f13db>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x239/0x3b8 <ffffffff81128469>
=> do_path_lookup+0x4d/0xf6 <ffffffff8112881b>
=> do_filp_open+0x129/0x60a <ffffffff81129c35>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_open+0x20/0x22 <ffffffff8111da0d>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_get+0x57/0x213 <ffffffff811baf2b>
=> ext4_xattr_security_get+0x2a/0x2c <ffffffff811bc92d>
=> generic_getxattr+0x57/0x60 <ffffffff81137e27>
1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdf>
=> kcalloc+0x38/0x3a <ffffffff811f5ff2>
=> security_get_user_sids+0x1c2/0x369 <ffffffff811f8503>
=> sel_write_user+0xd6/0x19d <ffffffff811ef5b3>
=> selinux_transaction_write+0x5b/0x77 <ffffffff811edbfc>
=> vfs_write+0xac/0xf3 <ffffffff8111e3e5>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
15 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> proc_alloc_inode+0x20/0x91 <ffffffff811682c0>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> new_inode+0x1b/0x4b <ffffffff8113187f>
=> proc_pid_make_inode+0x14/0xb7 <ffffffff811697c0>
=> proc_pident_instantiate+0x22/0xb1 <ffffffff8116c202>
=> proc_pident_lookup+0x84/0xa3 <ffffffff8116c3ce>
=> proc_tgid_base_lookup+0x1a/0x1c <ffffffff8116c43f>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x239/0x3b8 <ffffffff81128469>
213 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
=> block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
=> block_write_full_page+0x15/0x17 <ffffffff81143693>
=> mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8>
=> mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32>
=> __lookup_hash+0x7f/0x88 <ffffffff81127eaa>
=> lookup_hash+0x19/0x1b <ffffffff81127ecc>
=> lookup_create+0x4e/0x94 <ffffffff81127f1c>
=> sys_mkdirat+0x43/0xe8 <ffffffff8112a389>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
53 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> submit_bh+0xe6/0x105 <ffffffff81140a6f>
=> jbd2_journal_commit_transaction+0x6e4/0x1052 <ffffffff811c56a6>
31 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
=> __scsi_get_command+0x1b/0x96 <ffffffff812f3150>
=> scsi_get_command+0x3d/0xa5 <ffffffff812f3283>
=> scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
=> scsi_request_fn+0x82/0x421 <ffffffff812f9a5f>
=> __blk_run_queue+0x4a/0x7a <ffffffff81212b10>
=> blk_run_queue+0x28/0x3c <ffffffff81212bf9>
1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d540c>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d6181>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6209>
=> ext4_file_write+0x1dc/0x234 <ffffffff8118e898>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
=> vfs_write+0xac/0xf3 <ffffffff8111e3e5>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
8 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d>
=> do_readv_writev+0xa7/0x11e <ffffffff8111ea03>
=> vfs_writev+0x45/0x47 <ffffffff8111eabf>
=> sys_writev+0x4a/0x93 <ffffffff8111ebe2>
2 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> __pollwait+0x5e/0xcc <ffffffff8112ca0b>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc70>
=> unix_poll+0x1c/0x9a <ffffffff8143cc91>
=> sock_poll+0x1a/0x1c <ffffffff813ad43a>
=> do_sys_poll+0x1fa/0x386 <ffffffff8112d781>
=> sys_poll+0x51/0xbb <ffffffff8112d9b0>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1bd>
=> __alloc_skb+0x40/0x133 <ffffffff813b5e17>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_sendmsg+0xa1/0xb6 <ffffffff813af90f>
=> sys_sendmsg+0x237/0x2b5 <ffffffff813b0cd9>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
3 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> alloc_file+0x1e/0xbf <ffffffff8111f3da>
=> create_write_pipe+0x151/0x1bf <ffffffff81125f35>
=> do_pipe_flags+0x32/0xf4 <ffffffff81126060>
=> sys_pipe2+0x1c/0x5c <ffffffff8112613e>
=> sys_pipe+0x10/0x12 <ffffffff8112618e>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> dup_fd+0x33/0x25f <ffffffff81133573>
=> copy_process+0x5b8/0x10c5 <ffffffff810538ab>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
2 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1>
=> ext4_init_security+0x61/0x82 <ffffffff811bc9d3>
=> ext4_new_inode+0xb69/0xcf2 <ffffffff81190493>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> skb_clone+0x50/0x70 <ffffffff813b5db7>
=> dev_hard_start_xmit+0xd1/0x535 <ffffffff813c1660>
=> sch_direct_xmit+0x6b/0x147 <ffffffff813d7173>
=> dev_queue_xmit+0x2ac/0x461 <ffffffff813c1d70>
=> neigh_resolve_output+0x205/0x236 <ffffffff813ca2be>
=> ip_finish_output2+0x1cc/0x20f <ffffffff813f3979>
=> ip_finish_output+0x65/0x6a <ffffffff813f42a9>
=> ip_output+0x8c/0x91 <ffffffff813f49f2>
=> dst_output+0x14/0x16 <ffffffff813f34d7>
=> ip_local_out+0x20/0x24 <ffffffff813f435f>
3 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> __pollwait+0x5e/0xcc <ffffffff8112ca0b>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc70>
=> unix_poll+0x1c/0x9a <ffffffff8143cc91>
=> sock_poll+0x1a/0x1c <ffffffff813ad43a>
=> do_select+0x2fb/0x4f5 <ffffffff8112cf37>
=> core_sys_select+0x163/0x202 <ffffffff8112d294>
=> sys_select+0x91/0xb9 <ffffffff8112d3c4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
2 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d85>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> mpage_alloc+0x2a/0x84 <ffffffff8114866b>
=> do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> filemap_fault+0x17b/0x36c <ffffffff810d6ade>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> do_filp_open+0xe7/0x60a <ffffffff81129bf3>
=> open_exec+0x26/0x82 <ffffffff8112265e>
=> load_elf_binary+0x1df/0x175f <ffffffff8115c1a8>
=> search_binary_handler+0xb6/0x22e <ffffffff81122ce3>
=> do_execve+0x17e/0x277 <ffffffff81123ea0>
=> sys_execve+0x43/0x5a <ffffffff81011553>
=> stub_execve+0x6c/0xc0 <ffffffff81009fdc>
210 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> grab_cache_page_write_begin+0x6d/0x9e <ffffffff810d5960>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d540c>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d6181>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6209>
=> ext4_file_write+0x1dc/0x234 <ffffffff8118e898>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
6 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> do_filp_open+0xe7/0x60a <ffffffff81129bf3>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_openat+0x11/0x13 <ffffffff8111da20>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
8 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107025>
=> do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b8e>
=> handle_mm_fault+0x113/0x1ce <ffffffff810f1714>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> submit_bh+0xe6/0x105 <ffffffff81140a6f>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
5 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333>
=> ext4_iget+0x58/0x64c <ffffffff81192a48>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
257 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
=> block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
=> block_write_full_page+0x15/0x17 <ffffffff81143693>
=> ext4_writepage+0x201/0x210 <ffffffff81193854>
=> shrink_page_list+0x3c7/0x6de <ffffffff810e22a1>
809 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> kmalloc_large_node+0x56/0x95 <ffffffff8146a57d>
=> __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff2c>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d>
=> do_readv_writev+0xa7/0x11e <ffffffff8111ea03>
=> vfs_writev+0x45/0x47 <ffffffff8111eabf>
=> sys_writev+0x4a/0x93 <ffffffff8111ebe2>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
2 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> iget_locked+0x61/0xdd <ffffffff811323be>
=> ext4_iget+0x2c/0x64c <ffffffff81192a1c>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x239/0x3b8 <ffffffff81128469>
=> do_path_lookup+0x4d/0xf6 <ffffffff8112881b>
=> do_filp_open+0x129/0x60a <ffffffff81129c35>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> alloc_file+0x1e/0xbf <ffffffff8111f3da>
=> shmem_file_setup+0x125/0x160 <ffffffff810e7140>
=> drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc>
=> i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd>
=> i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
99 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> mpage_alloc+0x2a/0x84 <ffffffff8114866b>
=> do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
24 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76f>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e>
=> page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd08>
=> generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6514>
=> do_sync_read+0xbf/0xff <ffffffff8111de6e>
=> vfs_read+0xa9/0xf0 <ffffffff8111e4d5>
=> sys_read+0x4a/0x6e <ffffffff8111e566>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
21 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> prepare_creds+0x26/0xae <ffffffff81074d4b>
=> sys_faccessat+0x37/0x162 <ffffffff8111d279>
=> sys_access+0x18/0x1a <ffffffff8111d3bc>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> kmalloc_order_trace+0x2c/0x5e <ffffffff8110c73c>
=> __kmalloc+0x37/0x10d <ffffffff8110df65>
=> kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c>
=> tracing_open_pipe+0x40/0x147 <ffffffff810b907a>
=> __dentry_open+0x161/0x283 <ffffffff8111c980>
=> nameidata_to_filp+0x60/0x67 <ffffffff8111d8f4>
=> finish_open+0xa1/0x17f <ffffffff8112974f>
=> do_filp_open+0x186/0x60a <ffffffff81129c92>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_open+0x20/0x22 <ffffffff8111da0d>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> do_mpage_readpage+0x34a/0x468 <ffffffff81148b8d>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
2 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> find_or_create_page+0x5a/0x8a <ffffffff810d5b6d>
=> __getblk+0x137/0x23a <ffffffff81141b55>
=> __ext4_get_inode_loc+0x118/0x344 <ffffffff8119115f>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1>
=> ext4_init_security+0x61/0x82 <ffffffff811bc9d3>
=> ext4_new_inode+0xb69/0xcf2 <ffffffff81190493>
8 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
=> __breadahead+0x2e/0x39 <ffffffff81141c86>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1>
58 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32>
=> __lookup_hash+0x7f/0x88 <ffffffff81127eaa>
=> lookup_hash+0x19/0x1b <ffffffff81127ecc>
=> do_last+0xcf/0x2df <ffffffff811298fc>
=> do_filp_open+0x2ac/0x60a <ffffffff81129db8>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_openat+0x11/0x13 <ffffffff8111da20>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> mpage_readpages+0x99/0x104 <ffffffff81148dae>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ondemand_readahead+0x15a/0x1d8 <ffffffff810ddc0f>
=> page_cache_sync_readahead+0x3d/0x40 <ffffffff810ddd6d>
=> generic_file_aio_read+0x27d/0x5e0 <ffffffff810d64d4>
=> do_sync_read+0xbf/0xff <ffffffff8111de6e>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> alloc_file+0x1e/0xbf <ffffffff8111f3da>
=> create_read_pipe+0x2b/0x5b <ffffffff81125ffe>
=> do_pipe_flags+0x4e/0xf4 <ffffffff8112607c>
=> sys_pipe2+0x1c/0x5c <ffffffff8112613e>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
9 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> kthreadd+0xe7/0x124 <ffffffff8106f61f>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d540c>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d6181>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6209>
=> ext4_file_write+0x1dc/0x234 <ffffffff8118e898>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
=> vfs_write+0xac/0xf3 <ffffffff8111e3e5>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdf>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249>
=> i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> dst_alloc+0x48/0xaa <ffffffff813c6672>
=> __ip_route_output_key+0x561/0x764 <ffffffff813ee513>
=> tcp_v4_connect+0x121/0x47d <ffffffff8140becf>
=> inet_stream_connect+0xa8/0x25f <ffffffff81418e24>
=> sys_connect+0x98/0xd6 <ffffffff813b0437>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> kmalloc_large_node+0x56/0x95 <ffffffff8146a57d>
=> __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff2c>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
=> vfs_write+0xb3/0xf3 <ffffffff8111e3ec>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> __nf_conntrack_alloc+0xc9/0x1af <ffffffff813e4533>
=> nf_conntrack_in+0x2f6/0x77a <ffffffff813e490f>
=> ipv4_conntrack_local+0x46/0x4a <ffffffff8142864c>
=> nf_iterate+0x48/0x83 <ffffffff813e1cb8>
=> nf_hook_slow+0x6a/0xe9 <ffffffff813e1d5d>
=> nf_hook_thresh.constprop.8+0x42/0x44 <ffffffff813f42f0>
=> __ip_local_out+0x49/0x4d <ffffffff813f433b>
=> ip_local_out+0x13/0x24 <ffffffff813f4352>
=> ip_queue_xmit+0x2b3/0x30e <ffffffff813f4616>
=> tcp_transmit_skb+0x708/0x739 <ffffffff81407217>
1 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> dst_alloc+0x48/0xaa <ffffffff813c6672>
=> __ip_route_output_key+0x561/0x764 <ffffffff813ee513>
=> ip_route_output_flow+0x2c/0x1ea <ffffffff813ee742>
=> ip4_datagram_connect+0x18d/0x298 <ffffffff8140ede5>
=> inet_dgram_connect+0x65/0x6c <ffffffff81418b9d>
=> sys_connect+0x98/0xd6 <ffffffff813b0437>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107025>
=> do_huge_pmd_wp_page+0x15a/0x637 <ffffffff81114a25>
=> handle_mm_fault+0x169/0x1ce <ffffffff810f176a>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> find_or_create_page+0x41/0x8a <ffffffff810d5b54>
=> __getblk+0x137/0x23a <ffffffff81141b55>
=> __breadahead+0x12/0x39 <ffffffff81141c6a>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_iget+0x58/0x64c <ffffffff81192a48>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x239/0x3b8 <ffffffff81128469>
=> do_path_lookup+0x4d/0xf6 <ffffffff8112881b>
=> user_path_at+0x57/0x94 <ffffffff811294e0>
=> sys_faccessat+0xa0/0x162 <ffffffff8111d2e2>
=> sys_access+0x18/0x1a <ffffffff8111d3bc>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> shmem_file_setup+0xb6/0x160 <ffffffff810e70d1>
=> drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc>
=> i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd>
=> i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
416 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
=> block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
=> block_write_full_page+0x15/0x17 <ffffffff81143693>
=> blkdev_writepage+0x18/0x1a <ffffffff811460b8>
=> __writepage+0x15/0x2e <ffffffff810dca43>
13 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> mpage_readpages+0x99/0x104 <ffffffff81148dae>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> filemap_fault+0x17b/0x36c <ffffffff810d6ade>
=> __do_fault+0x55/0x377 <ffffffff810ee88b>
=> handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47>
58 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d85>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> do_mpage_readpage+0x34a/0x468 <ffffffff81148b8d>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32>
=> __lookup_hash+0x7f/0x88 <ffffffff81127eaa>
=> lookup_hash+0x19/0x1b <ffffffff81127ecc>
=> do_last+0xcf/0x2df <ffffffff811298fc>
=> do_filp_open+0x2ac/0x60a <ffffffff81129db8>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_open+0x20/0x22 <ffffffff8111da0d>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
7727 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> submit_bh+0xe6/0x105 <ffffffff81140a6f>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> __split_vma+0x6c/0x21b <ffffffff810f50bd>
=> do_munmap+0x194/0x30b <ffffffff810f57d1>
=> mmap_region+0x9b/0x446 <ffffffff810f645d>
=> do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6aa0>
=> sys_mmap_pgoff+0x115/0x164 <ffffffff810f6c0f>
=> sys_mmap+0x22/0x24 <ffffffff8100da68>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> __call_usermodehelper+0x43/0x76 <ffffffff81069132>
=> process_one_work+0x186/0x298 <ffffffff8106b3ca>
=> worker_thread+0xda/0x15d <ffffffff8106bf29>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
18 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d85>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> mpage_readpages+0xf0/0x104 <ffffffff81148e05>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> iget_locked+0x61/0xdd <ffffffff811323be>
=> ext4_iget+0x2c/0x64c <ffffffff81192a1c>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x14b/0x3b8 <ffffffff8112837b>
=> do_filp_open+0x1eb/0x60a <ffffffff81129cf7>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> find_or_create_page+0x5a/0x8a <ffffffff810d5b6d>
=> ext4_mb_load_buddy+0x1f7/0x30f <ffffffff811b4bdf>
=> ext4_free_blocks+0x41d/0x6ba <ffffffff811b7a92>
=> ext4_ext_truncate+0x361/0x78a <ffffffff811b1012>
=> ext4_truncate+0xa1/0x4d9 <ffffffff8119598c>
=> ext4_evict_inode+0x13f/0x228 <ffffffff811978e9>
=> evict+0x21/0x89 <ffffffff81131b92>
149 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> mpage_readpages+0x99/0x104 <ffffffff81148dae>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e>
=> page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd08>
=> generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6514>
High-order normal allocations: 152966
High-order atomic allocations: 875
On Tue, May 10, 2011 at 12:05:54PM -0500, James Bottomley wrote:
> On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote:
> > On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote:
> > > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> > > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > > > > I really would like to hear if the fix makes a big difference or
> > > > > > if we need to consider forcing SLUB high-order allocations bailing
> > > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > > > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > > > > less but processes will still be getting stalled in direct compaction
> > > > > > and direct reclaim so it would still be jittery.
> > > > >
> > > > > "the fix" being this
> > > > >
> > > > > https://lkml.org/lkml/2011/3/5/121
> > > > >
> > > >
> > > > Drop this for the moment. It was a long shot at best and there is little
> > > > evidence the problem is in this area.
> > > >
> > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop
> > > > kswapd being woken up by SLUB using speculative high-orders. The second
> > > > one is more drastic and prevents slub entering direct reclaim or
> > > > compaction. It applies on top of patch 1. These are both untested and
> > > > afraid are a bit rushed as well :(
> > >
> > > Preliminary results with both patches applied still show kswapd
> > > periodically going up to 99% but it doesn't stay there, it comes back
> > > down again (and, obviously, the system doesn't hang).
> >
> > This is a second run with the watch highorders.
> >
> > At the end of the run, the system hung temporarily and now comes back
> > with CPU3 spinning in all system time at kswapd shrink_slab
>
> Here's a trace in the same situation with the ftrace stack entries
> bumped to 16 as requested on IRC. There was no hang for this one.
>
Ok, so the bulk of the high-order allocs are coming from
> 140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
> => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
> => new_slab+0x50/0x199 <ffffffff8110dc48>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
> => mempool_alloc+0x68/0x116 <ffffffff810d70fa>
> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
> => bio_alloc+0x15/0x24 <ffffffff81144ef5>
> => submit_bh+0x6d/0x105 <ffffffff811409f6>
> => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
> => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
> => block_write_full_page+0x15/0x17 <ffffffff81143693>
> => mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8>
> => mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88>
>
That at least is in line with the large untar and absolves i915
from being the main cause of trouble. The lack of the hang implies
that SLUB doing high order allocations is stressing the system too
much and needs to be more willing to fall back to order-0 although
it does not adequately explain why it hung as opposed to just being
incredible slow.
I'm also still concerned with the reports of getting stuck in a heavy
loop on the i915 shrinker so will try again reproducing this locally
with a greater focus on something X related happening at the same time.
One thing at a time though, SLUB needs to be less aggressive so I'll
prepare a series in the morning, have another go at generating data
and see what shakes out.
Thanks.
--
Mel Gorman
SUSE Labs
On Tue, 2011-05-10 at 18:17 +0100, Mel Gorman wrote:
> On Tue, May 10, 2011 at 12:05:54PM -0500, James Bottomley wrote:
> > On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote:
> > > On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote:
> > > > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> > > > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > > > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > > > > > I really would like to hear if the fix makes a big difference or
> > > > > > > if we need to consider forcing SLUB high-order allocations bailing
> > > > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > > > > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > > > > > less but processes will still be getting stalled in direct compaction
> > > > > > > and direct reclaim so it would still be jittery.
> > > > > >
> > > > > > "the fix" being this
> > > > > >
> > > > > > https://lkml.org/lkml/2011/3/5/121
> > > > > >
> > > > >
> > > > > Drop this for the moment. It was a long shot at best and there is little
> > > > > evidence the problem is in this area.
> > > > >
> > > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop
> > > > > kswapd being woken up by SLUB using speculative high-orders. The second
> > > > > one is more drastic and prevents slub entering direct reclaim or
> > > > > compaction. It applies on top of patch 1. These are both untested and
> > > > > afraid are a bit rushed as well :(
> > > >
> > > > Preliminary results with both patches applied still show kswapd
> > > > periodically going up to 99% but it doesn't stay there, it comes back
> > > > down again (and, obviously, the system doesn't hang).
> > >
> > > This is a second run with the watch highorders.
> > >
> > > At the end of the run, the system hung temporarily and now comes back
> > > with CPU3 spinning in all system time at kswapd shrink_slab
> >
> > Here's a trace in the same situation with the ftrace stack entries
> > bumped to 16 as requested on IRC. There was no hang for this one.
> >
>
> Ok, so the bulk of the high-order allocs are coming from
>
> > 140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
> > => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
> > => new_slab+0x50/0x199 <ffffffff8110dc48>
> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
> > => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
> > => mempool_alloc+0x68/0x116 <ffffffff810d70fa>
> > => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
> > => bio_alloc+0x15/0x24 <ffffffff81144ef5>
> > => submit_bh+0x6d/0x105 <ffffffff811409f6>
> > => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
> > => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
> > => block_write_full_page+0x15/0x17 <ffffffff81143693>
> > => mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8>
> > => mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88>
> >
>
> That at least is in line with the large untar and absolves i915
> from being the main cause of trouble. The lack of the hang implies
> that SLUB doing high order allocations is stressing the system too
> much and needs to be more willing to fall back to order-0 although
> it does not adequately explain why it hung as opposed to just being
> incredible slow.
>
> I'm also still concerned with the reports of getting stuck in a heavy
> loop on the i915 shrinker so will try again reproducing this locally
> with a greater focus on something X related happening at the same time.
>
> One thing at a time though, SLUB needs to be less aggressive so I'll
> prepare a series in the morning, have another go at generating data
> and see what shakes out.
As a final data point, I tried slub_max_order=0. kswapd is much more
quiescent (it calms down to nothing). It shows spikes of up to 70% and
once up to 99% though, but it doesn't go up to 99% and stay there like
it did previously. This is just with top, so it's a bit inaccurate.
James
* On Tue, May 10, 2011 at 10:29:57AM -0500, James Bottomley <[email protected]> wrote:
>On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
>> On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
>> > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
>> > > I really would like to hear if the fix makes a big difference or
>> > > if we need to consider forcing SLUB high-order allocations bailing
>> > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
>> > > allocate_slab). Even with the fix applied, kswapd might be waking up
>> > > less but processes will still be getting stalled in direct compaction
>> > > and direct reclaim so it would still be jittery.
>> > "the fix" being this
>> > https://lkml.org/lkml/2011/3/5/121
>> Drop this for the moment. It was a long shot at best and there is little
>> evidence the problem is in this area.
>> I'm attaching two patches. The first is the NO_KSWAPD one to stop
>> kswapd being woken up by SLUB using speculative high-orders. The second
>> one is more drastic and prevents slub entering direct reclaim or
>> compaction. It applies on top of patch 1. These are both untested and
>> afraid are a bit rushed as well :(
>
>Preliminary results with both patches applied still show kswapd
>periodically going up to 99% but it doesn't stay there, it comes back
>down again (and, obviously, the system doesn't hang).
>
>This is sysrq-M from a couple of times when it went up there:
>
>[ 426.736958] SysRq : Show Memory
>[ 426.736974] Mem-Info:
>[ 426.736977] Node 0 DMA per-cpu:
>[ 426.736983] CPU 0: hi: 0, btch: 1 usd: 0
>[ 426.736986] CPU 1: hi: 0, btch: 1 usd: 0
>[ 426.736989] CPU 2: hi: 0, btch: 1 usd: 0
>[ 426.736993] CPU 3: hi: 0, btch: 1 usd: 0
>[ 426.736996] Node 0 DMA32 per-cpu:
>[ 426.737002] CPU 0: hi: 186, btch: 31 usd: 169
>[ 426.737005] CPU 1: hi: 186, btch: 31 usd: 40
>[ 426.737009] CPU 2: hi: 186, btch: 31 usd: 166
>[ 426.737012] CPU 3: hi: 186, btch: 31 usd: 168
>[ 426.737015] Node 0 Normal per-cpu:
>[ 426.737020] CPU 0: hi: 0, btch: 1 usd: 0
>[ 426.737024] CPU 1: hi: 0, btch: 1 usd: 0
>[ 426.737027] CPU 2: hi: 0, btch: 1 usd: 0
>[ 426.737030] CPU 3: hi: 0, btch: 1 usd: 0
>[ 426.737036] active_anon:108658 inactive_anon:37031 isolated_anon:0
>[ 426.737037] active_file:32006 inactive_file:41051 isolated_file:32
>[ 426.737038] unevictable:8 dirty:41202 writeback:204 unstable:0
>[ 426.737039] free:191520 slab_reclaimable:8490 slab_unreclaimable:27477
>[ 426.737039] mapped:9176 shmem:26412 pagetables:5427 bounce:0
>[ 426.737051] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
>[ 426.737064] lowmem_reserve[]: 0 1856 1862 1862
>[ 426.737078] Node 0 DMA32 free:757892kB min:66820kB low:83524kB high:100228kB active_anon:434632kB inactive_anon:148112kB active_file:127784kB inactive_file:156924kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:1901408kB mlocked:32kB dirty:164808kB writeback:816kB mapped:36704kB shmem:105648kB slab_reclaimable:33676kB slab_unreclaimable:108372kB kernel_stack:2304kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
>[ 426.737092] lowmem_reserve[]: 0 0 5 5
>[ 426.737106] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
>[ 426.737119] lowmem_reserve[]: 0 0 0 0
>[ 426.737132] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
>[ 426.737174] Node 0 DMA32: 945*4kB 585*8kB 6469*16kB 4871*32kB 3189*64kB 1338*128kB 228*256kB 58*512kB 24*1024kB 1*2048kB 0*4096kB = 757884kB
>[ 426.737227] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
>[ 426.737257] 99595 total pagecache pages
>[ 426.737260] 0 pages in swap cache
>[ 426.737263] Swap cache stats: add 0, delete 0, find 0/0
>[ 426.737266] Free swap = 3768316kB
>[ 426.737268] Total swap = 3768316kB
>[ 426.744603] 525808 pages RAM
>[ 426.744612] 57618 pages reserved
>[ 426.744614] 141551 pages shared
>[ 426.744617] 186065 pages non-shared
>[ 472.301810] SysRq : Show Memory
>[ 472.301826] Mem-Info:
>[ 472.301829] Node 0 DMA per-cpu:
>[ 472.301835] CPU 0: hi: 0, btch: 1 usd: 0
>[ 472.301839] CPU 1: hi: 0, btch: 1 usd: 0
>[ 472.301842] CPU 2: hi: 0, btch: 1 usd: 0
>[ 472.301845] CPU 3: hi: 0, btch: 1 usd: 0
>[ 472.301848] Node 0 DMA32 per-cpu:
>[ 472.301854] CPU 0: hi: 186, btch: 31 usd: 184
>[ 472.301857] CPU 1: hi: 186, btch: 31 usd: 46
>[ 472.301860] CPU 2: hi: 186, btch: 31 usd: 158
>[ 472.301863] CPU 3: hi: 186, btch: 31 usd: 163
>[ 472.301866] Node 0 Normal per-cpu:
>[ 472.301871] CPU 0: hi: 0, btch: 1 usd: 0
>[ 472.301874] CPU 1: hi: 0, btch: 1 usd: 0
>[ 472.301878] CPU 2: hi: 0, btch: 1 usd: 0
>[ 472.301881] CPU 3: hi: 0, btch: 1 usd: 0
>[ 472.301886] active_anon:107673 inactive_anon:37031 isolated_anon:0
>[ 472.301887] active_file:31533 inactive_file:33323 isolated_file:32
>[ 472.301888] unevictable:8 dirty:26256 writeback:6475 unstable:0
>[ 472.301889] free:198742 slab_reclaimable:9347 slab_unreclaimable:28647
>[ 472.301889] mapped:8307 shmem:26412 pagetables:5427 bounce:0
>[ 472.301901] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
>[ 472.301915] lowmem_reserve[]: 0 1856 1862 1862
>[ 472.301928] Node 0 DMA32 free:786780kB min:66820kB low:83524kB high:100228kB active_anon:430692kB inactive_anon:148112kB active_file:125892kB inactive_file:126012kB unevictable:32kB isolated(anon):0kB isolated(file):128kB present:1901408kB mlocked:32kB dirty:105024kB writeback:25900kB mapped:33228kB shmem:105648kB slab_reclaimable:37104kB slab_unreclaimable:113052kB kernel_stack:2288kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3196 all_unreclaimable? no
>[ 472.301943] lowmem_reserve[]: 0 0 5 5
>[ 472.301956] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
>[ 472.301968] lowmem_reserve[]: 0 0 0 0
>[ 472.301982] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
>[ 472.302015] Node 0 DMA32: 6121*4kB 1912*8kB 5094*16kB 4920*32kB 3168*64kB 1381*128kB 262*256kB 68*512kB 24*1024kB 1*2048kB 0*4096kB = 786756kB
>[ 472.302048] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
>[ 472.302090] 91312 total pagecache pages
>[ 472.302093] 0 pages in swap cache
>[ 472.302096] Swap cache stats: add 0, delete 0, find 0/0
>[ 472.302099] Free swap = 3768316kB
>[ 472.302102] Total swap = 3768316kB
>[ 472.309521] 525808 pages RAM
>[ 472.309529] 57618 pages reserved
>[ 472.309548] 142496 pages shared
>[ 472.309551] 177182 pages non-shared
>
>I'll finish off this verification, and then re-run with the watch-highorder script running.
>
>James
Hi,
I saw similar lockups while doing a heavy copy routine (basically
copying an entire partition). I don't think this may be related to
ext4, since the source and destination filesystems were btrfs and
xfs.
===============================================================================================================
9 00:04:58 localhost kernel: [39824.230679] INFO: task kswapd0:528 blocked for more than 120 seconds.
May 9 00:04:58 localhost kernel: [39824.230684] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 9 00:04:58 localhost kernel: [39824.230690] kswapd0 D ffff8800bb90d740 0 528 2 0x00000000
May 9 00:04:58 localhost kernel: [39824.230699] ffff88013f6a3750 0000000000000046 0000000000000246 ffff88013b290040
May 9 00:04:58 localhost kernel: [39824.230708] ffff88013b290000 0000000000000001 ffff88013f6ea210 ffff88013f6a3fd8
May 9 00:04:58 localhost kernel: [39824.230716] ffff88013f6a3fd8 ffff88013f6a3fd8 0000000000000004 ffff88013f6ea470
May 9 00:04:58 localhost kernel: [39824.230724] Call Trace:
May 9 00:04:58 localhost kernel: [39824.230738] [<ffffffff81548aa3>] io_schedule+0x73/0xc0
May 9 00:04:58 localhost kernel: [39824.230747] [<ffffffff813343bb>] get_request_wait+0xdb/0x1e0
May 9 00:04:58 localhost kernel: [39824.230757] [<ffffffff810608f0>] ? autoremove_wake_function+0x0/0x40
May 9 00:05:22 localhost kernel: [39824.230766] [<ffffffff81334cb6>] __make_request+0x76/0x560
May 9 00:05:22 localhost kernel: [39824.230774] [<ffffffff810e1518>] ? account_page_writeback+0x28/0x30
May 9 00:05:22 localhost kernel: [39824.230782] [<ffffffff81332bce>] generic_make_request+0x28e/0x560
May 9 00:05:22 localhost kernel: [39824.230790] [<ffffffff81332f25>] submit_bio+0x85/0x110
May 9 00:05:22 localhost kernel: [39824.230797] [<ffffffff811552b2>] ? bio_alloc_bioset+0xb2/0xf0
May 9 00:05:22 localhost kernel: [39824.230806] [<ffffffff8127da17>] xfs_submit_ioend_bio.isra.11+0x57/0x80
May 9 00:05:22 localhost kernel: [39824.230816] [<ffffffff8127dafb>] xfs_submit_ioend+0xbb/0x110
May 9 00:05:22 localhost kernel: [39824.230818] [<ffffffff8127ed47>] xfs_vm_writepage+0x3f7/0x540
May 9 00:05:22 localhost kernel: [39824.230821] [<ffffffff810e90b0>] shrink_page_list+0x650/0x8f0
May 9 00:05:22 localhost kernel: [39824.230823] [<ffffffff810e9758>] shrink_inactive_list+0x178/0x410
May 9 00:05:22 localhost kernel: [39824.230826] [<ffffffff8104e89a>] ? del_timer_sync+0x3a/0x60
May 9 00:05:22 localhost kernel: [39824.230828] [<ffffffff810e9e6e>] shrink_zone+0x47e/0x540
May 9 00:05:22 localhost kernel: [39824.230831] [<ffffffff810e7a25>] ? shrink_slab+0x145/0x180
*May 9 00:05:22 localhost kernel: [39824.230833] [<ffffffff810ea5bd>] kswapd+0x68d/0xa20
*May 9 00:05:22 localhost kernel: [39824.230835] [<ffffffff810e9f30>] ? kswapd+0x0/0xa20
May 9 00:05:22 localhost kernel: [39824.230837] [<ffffffff8105fffc>] kthread+0x8c/0xa0
May 9 00:05:22 localhost kernel: [39824.230841] [<ffffffff81039a7c>] ? schedule_tail+0x4c/0xf0
May 9 00:05:22 localhost kernel: [39824.230843] [<ffffffff81003c94>] kernel_thread_helper+0x4/0x10
May 9 00:05:22 localhost kernel: [39824.230846] [<ffffffff8105ff70>] ? kthread+0x0/0xa0
May 9 00:05:22 localhost kernel: [39824.230848] [<ffffffff81003c90>] ? kernel_thread_helper+0x0/0x10
May 9 00:05:22 localhost kernel: [39824.230853] INFO: task xfsbufd/sda9:3757 blocked for more than 120 seconds.
May 9 00:05:22 localhost kernel: [39824.230854] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 9 00:05:22 localhost kernel: [39824.230856] xfsbufd/sda9 D ffff8800bb84d740 0 3757 2 0x00000000
May 9 00:05:22 localhost kernel: [39824.230858] ffff88013b12fb10 0000000000000046 0000000000000246 ffff88013f6736c0
May 9 00:05:22 localhost kernel: [39824.230861] ffff88013f673680 0000000000000001 ffff88013cec2210 ffff88013b12ffd8
May 9 00:05:22 localhost kernel: [39824.230863] ffff88013b12ffd8 ffff88013b12ffd8 0000000000000001 ffff88013cec2470
May 9 00:05:22 localhost kernel: [39824.230866] Call Trace:
May 9 00:05:22 localhost kernel: [39824.230868] [<ffffffff81548aa3>] io_schedule+0x73/0xc0
May 9 00:05:22 localhost kernel: [39824.230870] [<ffffffff813343bb>] get_request_wait+0xdb/0x1e0
May 9 00:05:22 localhost kernel: [39824.230873] [<ffffffff810608f0>] ? autoremove_wake_function+0x0/0x40
May 9 00:05:22 localhost kernel: [39824.230875] [<ffffffff81334cb6>] __make_request+0x76/0x560
May 9 00:05:22 localhost kernel: [39824.230878] [<ffffffff81037434>] ? resched_best_mask.isra.60+0x114/0x130
May 9 00:05:22 localhost kernel: [39824.230880] [<ffffffff81332bce>] generic_make_request+0x28e/0x560
May 9 00:05:22 localhost kernel: [39824.230883] [<ffffffff8103640c>] ? update_cpu_clock+0x19c/0x3d0
May 9 00:05:22 localhost kernel: [39824.230885] [<ffffffff81332f25>] submit_bio+0x85/0x110
May 9 00:05:22 localhost kernel: [39824.230887] [<ffffffff8115525a>] ? bio_alloc_bioset+0x5a/0xf0
May 9 00:05:22 localhost kernel: [39824.230889] [<ffffffff8128159c>] _xfs_buf_ioapply+0x19c/0x300
May 9 00:05:22 localhost kernel: [39824.230892] [<ffffffff812825cd>] ? xfs_bdstrat_cb+0x5d/0x100
May 9 00:05:22 localhost kernel: [39824.230894] [<ffffffff8128194e>] xfs_buf_iorequest+0x5e/0x140
May 9 00:05:22 localhost kernel: [39824.230896] [<ffffffff812825cd>] xfs_bdstrat_cb+0x5d/0x100
May 9 00:05:22 localhost kernel: [39824.230899] [<ffffffff81282776>] xfsbufd+0x106/0x180
May 9 00:05:22 localhost kernel: [39824.230901] [<ffffffff81282670>] ? xfsbufd+0x0/0x180
May 9 00:05:22 localhost kernel: [39824.230903] [<ffffffff8105fffc>] kthread+0x8c/0xa0
May 9 00:05:22 localhost kernel: [39824.230905] [<ffffffff81039a7c>] ? schedule_tail+0x4c/0xf0
May 9 00:05:22 localhost kernel: [39824.230907] [<ffffffff81003c94>] kernel_thread_helper+0x4/0x10
May 9 00:05:22 localhost kernel: [39824.230909] [<ffffffff8105ff70>] ? kthread+0x0/0xa0
May 9 00:05:22 localhost kernel: [39824.230911] [<ffffffff81003c90>] ? kernel_thread_helper+0x0/0x10
May 9 00:05:22 localhost kernel: [39824.230913] INFO: task xfsbufd/sda11:3769 blocked for more than 120 seconds.
May 9 00:05:22 localhost kernel: [39824.230914] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 9 00:05:22 localhost kernel: [39824.230915] xfsbufd/sda11 D ffff8800bb84d740 0 3769 2 0x00000000
May 9 00:05:22 localhost kernel: [39824.230918] ffff88013b183b10 0000000000000046 0000000000000246 ffff88013f6736c0
May 9 00:05:22 localhost kernel: [39824.230920] ffff88013f673680 0000000000000001 ffff88013b299b40 ffff88013b183fd8
May 9 00:05:22 localhost kernel: [39824.230923] ffff88013b183fd8 ffff88013b183fd8 0000000000000001 ffff88013b299da0
May 9 00:05:22 localhost kernel: [39824.230925] Call Trace:
May 9 00:05:22 localhost kernel: [39824.230928] [<ffffffff81548aa3>] io_schedule+0x73/0xc0
May 9 00:05:22 localhost kernel: [39824.230930] [<ffffffff813343bb>] get_request_wait+0xdb/0x1e0
May 9 00:05:22 localhost kernel: [39824.230932] [<ffffffff810608f0>] ? autoremove_wake_function+0x0/0x40
May 9 00:05:22 localhost kernel: [39824.230935] [<ffffffff81334cb6>] __make_request+0x76/0x560
May 9 00:05:22 localhost kernel: [39824.230938] [<ffffffff8135d7af>] ? __sg_alloc_table+0x7f/0x140
May 9 00:05:22 localhost kernel: [39824.230941] [<ffffffff81332bce>] generic_make_request+0x28e/0x560
May 9 00:05:22 localhost kernel: [39824.230943] [<ffffffff8103640c>] ? update_cpu_clock+0x19c/0x3d0
May 9 00:05:22 localhost kernel: [39824.230946] [<ffffffff81431a60>] ? ata_scsi_rw_xlat+0x0/0x210
May 9 00:05:22 localhost kernel: [39824.230948] [<ffffffff81332f25>] submit_bio+0x85/0x110
May 9 00:05:22 localhost kernel: [39824.230950] [<ffffffff8115525a>] ? bio_alloc_bioset+0x5a/0xf0
May 9 00:05:22 localhost kernel: [39824.230953] [<ffffffff8128159c>] _xfs_buf_ioapply+0x19c/0x300
May 9 00:05:22 localhost kernel: [39824.230955] [<ffffffff812825cd>] ? xfs_bdstrat_cb+0x5d/0x100
May 9 00:05:22 localhost kernel: [39824.230957] [<ffffffff8128194e>] xfs_buf_iorequest+0x5e/0x140
May 9 00:05:22 localhost kernel: [39824.230959] [<ffffffff812825cd>] xfs_bdstrat_cb+0x5d/0x100
May 9 00:05:22 localhost kernel: [39824.230962] [<ffffffff81282776>] xfsbufd+0x106/0x180
May 9 00:05:22 localhost kernel: [39824.230964] [<ffffffff81282670>] ? xfsbufd+0x0/0x180
May 9 00:05:22 localhost kernel: [39824.230966] [<ffffffff8105fffc>] kthread+0x8c/0xa0
May 9 00:05:22 localhost kernel: [39824.230968] [<ffffffff81039a7c>] ? schedule_tail+0x4c/0xf0
May 9 00:05:22 localhost kernel: [39824.230970] [<ffffffff81003c94>] kernel_thread_helper+0x4/0x10
May 9 00:05:22 localhost kernel: [39824.230972] [<ffffffff8105ff70>] ? kthread+0x0/0xa0
May 9 00:05:22 localhost kernel: [39824.230974] [<ffffffff81003c90>] ? kernel_thread_helper+0x0/0x10
May 9 00:05:22 localhost kernel: [39824.230989] INFO: task logrotate:1749 blocked for more than 120 seconds.
May 9 00:05:22 localhost kernel: [39824.230990] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 9 00:05:22 localhost kernel: [39824.230992] logrotate D ffff8800bb90d740 0 1749 1748 0x00000000
May 9 00:05:22 localhost kernel: [39824.230994] ffff88000c6f3788 0000000000000082 ffff88000c6f3738 ffff88013f475890
May 9 00:05:22 localhost kernel: [39824.230997] 0000000000000001 0000000000000001 0000000000000001 ffff88000c6f3fd8
May 9 00:05:22 localhost kernel: [39824.230999] ffff88000c6f3fd8 ffff88000c6f3fd8 0000000000000004 ffff88013f475af0
May 9 00:05:22 localhost kernel: [39824.231002] Call Trace:
May 9 00:05:22 localhost kernel: [39824.231004] [<ffffffff81548aa3>] io_schedule+0x73/0xc0
May 9 00:05:22 localhost kernel: [39824.231006] [<ffffffff813343bb>] get_request_wait+0xdb/0x1e0
May 9 00:05:22 localhost kernel: [39824.231008] [<ffffffff810608f0>] ? autoremove_wake_function+0x0/0x40
May 9 00:05:22 localhost kernel: [39824.231011] [<ffffffff81334cb6>] __make_request+0x76/0x560
May 9 00:05:22 localhost kernel: [39824.231014] [<ffffffff81150af4>] ? __getblk+0x24/0x280
May 9 00:05:22 localhost kernel: [39824.231016] [<ffffffff81332bce>] generic_make_request+0x28e/0x560
May 9 00:05:22 localhost kernel: [39824.231020] [<ffffffff811ae4fc>] ? ext4_mark_iloc_dirty+0x39c/0x5a0
May 9 00:05:22 localhost kernel: [39824.231022] [<ffffffff81332f25>] submit_bio+0x85/0x110
May 9 00:05:22 localhost kernel: [39824.231024] [<ffffffff8115525a>] ? bio_alloc_bioset+0x5a/0xf0
May 9 00:05:22 localhost kernel: [39824.231027] [<ffffffff8114f66b>] submit_bh+0xeb/0x120
May 9 00:05:22 localhost kernel: [39824.231029] [<ffffffff81151720>] __block_write_full_page+0x210/0x390
May 9 00:05:22 localhost kernel: [39824.231031] [<ffffffff810605de>] ? wake_up_bit+0x2e/0x40
May 9 00:05:22 localhost kernel: [39824.231034] [<ffffffff811518a0>] ? end_buffer_async_write+0x0/0x200
May 9 00:05:22 localhost kernel: [39824.231036] [<ffffffff811b00c0>] ? noalloc_get_block_write+0x0/0x30
May 9 00:05:22 localhost kernel: [39824.231038] [<ffffffff811518a0>] ? end_buffer_async_write+0x0/0x200
May 9 00:05:22 localhost kernel: [39824.231041] [<ffffffff81151f65>] block_write_full_page_endio+0xf5/0x140
May 9 00:05:22 localhost kernel: [39824.231043] [<ffffffff811b00c0>] ? noalloc_get_block_write+0x0/0x30
May 9 00:05:22 localhost kernel: [39824.231045] [<ffffffff81151fc5>] block_write_full_page+0x15/0x20
May 9 00:05:22 localhost kernel: [39824.231048] [<ffffffff811add81>] mpage_da_submit_io+0x451/0x510
May 9 00:05:22 localhost kernel: [39824.231050] [<ffffffff811ae4fc>] ? ext4_mark_iloc_dirty+0x39c/0x5a0
May 9 00:05:22 localhost kernel: [39824.231053] [<ffffffff811ae822>] ? ext4_mark_inode_dirty+0x82/0x230
May 9 00:05:22 localhost kernel: [39824.231055] [<ffffffff811b26a6>] mpage_da_map_and_submit+0x1d6/0x410
May 9 00:05:22 localhost kernel: [39824.231059] [<ffffffff811e8983>] ? jbd2_journal_start+0x13/0x20
May 9 00:05:22 localhost kernel: [39824.231061] [<ffffffff811b31b1>] ext4_da_writepages+0x3e1/0x750
May 9 00:05:22 localhost kernel: [39824.231064] [<ffffffff810e2e91>] do_writepages+0x21/0x40
May 9 00:05:22 localhost kernel: [39824.231066] [<ffffffff810da00b>] __filemap_fdatawrite_range+0x5b/0x60
May 9 00:05:22 localhost kernel: [39824.231068] [<ffffffff810da74c>] filemap_flush+0x1c/0x20
May 9 00:05:22 localhost kernel: [39824.231070] [<ffffffff811acafc>] ext4_alloc_da_blocks+0x4c/0xe0
May 9 00:05:22 localhost kernel: [39824.231073] [<ffffffff811a78e9>] ext4_release_file+0x79/0xc0
May 9 00:05:22 localhost kernel: [39824.231075] [<ffffffff81124d7a>] fput+0xda/0x200
May 9 00:05:22 localhost kernel: [39824.231077] [<ffffffff811214e5>] filp_close+0x55/0x80
May 9 00:05:22 localhost kernel: [39824.231079] [<ffffffff811215c6>] sys_close+0xb6/0x120
May 9 00:05:22 localhost kernel: [39824.231082] [<ffffffff81002eeb>] system_call_fastpath+0x16/0x1b
===========================================================================================================================================================
The mv was basically frozen though after sometime it resumed. I also noticed that levels of dirty data were really high (=~ 1G) which I presumed may have caused this.
The kernel is
========================================================
Linux Xye 2.6.39-rc6 #78 SMP PREEMPT Mon May 9 14:38:17 IST 2011 x86_64 Intel(R) Core(TM) i7 CPU Q 740 @ 1.73GHz GenuineIntel GNU/Linux
======================================================
Also, I am not using cgroups or systemd at the moment. Also, this is not a Fedora system. (Arch linux).
This happened only once. After which it has been fine, though I have not done such high throughput copying after that.
---------------------
Raghavendra D Prabhu
On Wed, May 11, 2011 at 02:38:09AM +0530, Raghavendra D Prabhu wrote:
> * On Tue, May 10, 2011 at 10:29:57AM -0500, James Bottomley <[email protected]> wrote:
> >On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> >>On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> >>> On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> >>> > I really would like to hear if the fix makes a big difference or
> >>> > if we need to consider forcing SLUB high-order allocations bailing
> >>> > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> >>> > allocate_slab). Even with the fix applied, kswapd might be waking up
> >>> > less but processes will still be getting stalled in direct compaction
> >>> > and direct reclaim so it would still be jittery.
>
> >>> "the fix" being this
>
> >>> https://lkml.org/lkml/2011/3/5/121
>
>
> >>Drop this for the moment. It was a long shot at best and there is little
> >>evidence the problem is in this area.
>
> >>I'm attaching two patches. The first is the NO_KSWAPD one to stop
> >>kswapd being woken up by SLUB using speculative high-orders. The second
> >>one is more drastic and prevents slub entering direct reclaim or
> >>compaction. It applies on top of patch 1. These are both untested and
> >>afraid are a bit rushed as well :(
> >
> >Preliminary results with both patches applied still show kswapd
> >periodically going up to 99% but it doesn't stay there, it comes back
> >down again (and, obviously, the system doesn't hang).
> >
> >This is sysrq-M from a couple of times when it went up there:
> >
> >[ 426.736958] SysRq : Show Memory
> >[ 426.736974] Mem-Info:
> >[ 426.736977] Node 0 DMA per-cpu:
> >[ 426.736983] CPU 0: hi: 0, btch: 1 usd: 0
> >[ 426.736986] CPU 1: hi: 0, btch: 1 usd: 0
> >[ 426.736989] CPU 2: hi: 0, btch: 1 usd: 0
> >[ 426.736993] CPU 3: hi: 0, btch: 1 usd: 0
> >[ 426.736996] Node 0 DMA32 per-cpu:
> >[ 426.737002] CPU 0: hi: 186, btch: 31 usd: 169
> >[ 426.737005] CPU 1: hi: 186, btch: 31 usd: 40
> >[ 426.737009] CPU 2: hi: 186, btch: 31 usd: 166
> >[ 426.737012] CPU 3: hi: 186, btch: 31 usd: 168
> >[ 426.737015] Node 0 Normal per-cpu:
> >[ 426.737020] CPU 0: hi: 0, btch: 1 usd: 0
> >[ 426.737024] CPU 1: hi: 0, btch: 1 usd: 0
> >[ 426.737027] CPU 2: hi: 0, btch: 1 usd: 0
> >[ 426.737030] CPU 3: hi: 0, btch: 1 usd: 0
> >[ 426.737036] active_anon:108658 inactive_anon:37031 isolated_anon:0
> >[ 426.737037] active_file:32006 inactive_file:41051 isolated_file:32
> >[ 426.737038] unevictable:8 dirty:41202 writeback:204 unstable:0
> >[ 426.737039] free:191520 slab_reclaimable:8490 slab_unreclaimable:27477
> >[ 426.737039] mapped:9176 shmem:26412 pagetables:5427 bounce:0
> >[ 426.737051] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> >[ 426.737064] lowmem_reserve[]: 0 1856 1862 1862
> >[ 426.737078] Node 0 DMA32 free:757892kB min:66820kB low:83524kB high:100228kB active_anon:434632kB inactive_anon:148112kB active_file:127784kB inactive_file:156924kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:1901408kB mlocked:32kB dirty:164808kB writeback:816kB mapped:36704kB shmem:105648kB slab_reclaimable:33676kB slab_unreclaimable:108372kB kernel_stack:2304kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
> >[ 426.737092] lowmem_reserve[]: 0 0 5 5
> >[ 426.737106] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> >[ 426.737119] lowmem_reserve[]: 0 0 0 0
> >[ 426.737132] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
> >[ 426.737174] Node 0 DMA32: 945*4kB 585*8kB 6469*16kB 4871*32kB 3189*64kB 1338*128kB 228*256kB 58*512kB 24*1024kB 1*2048kB 0*4096kB = 757884kB
> >[ 426.737227] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
> >[ 426.737257] 99595 total pagecache pages
> >[ 426.737260] 0 pages in swap cache
> >[ 426.737263] Swap cache stats: add 0, delete 0, find 0/0
> >[ 426.737266] Free swap = 3768316kB
> >[ 426.737268] Total swap = 3768316kB
> >[ 426.744603] 525808 pages RAM
> >[ 426.744612] 57618 pages reserved
> >[ 426.744614] 141551 pages shared
> >[ 426.744617] 186065 pages non-shared
> >[ 472.301810] SysRq : Show Memory
> >[ 472.301826] Mem-Info:
> >[ 472.301829] Node 0 DMA per-cpu:
> >[ 472.301835] CPU 0: hi: 0, btch: 1 usd: 0
> >[ 472.301839] CPU 1: hi: 0, btch: 1 usd: 0
> >[ 472.301842] CPU 2: hi: 0, btch: 1 usd: 0
> >[ 472.301845] CPU 3: hi: 0, btch: 1 usd: 0
> >[ 472.301848] Node 0 DMA32 per-cpu:
> >[ 472.301854] CPU 0: hi: 186, btch: 31 usd: 184
> >[ 472.301857] CPU 1: hi: 186, btch: 31 usd: 46
> >[ 472.301860] CPU 2: hi: 186, btch: 31 usd: 158
> >[ 472.301863] CPU 3: hi: 186, btch: 31 usd: 163
> >[ 472.301866] Node 0 Normal per-cpu:
> >[ 472.301871] CPU 0: hi: 0, btch: 1 usd: 0
> >[ 472.301874] CPU 1: hi: 0, btch: 1 usd: 0
> >[ 472.301878] CPU 2: hi: 0, btch: 1 usd: 0
> >[ 472.301881] CPU 3: hi: 0, btch: 1 usd: 0
> >[ 472.301886] active_anon:107673 inactive_anon:37031 isolated_anon:0
> >[ 472.301887] active_file:31533 inactive_file:33323 isolated_file:32
> >[ 472.301888] unevictable:8 dirty:26256 writeback:6475 unstable:0
> >[ 472.301889] free:198742 slab_reclaimable:9347 slab_unreclaimable:28647
> >[ 472.301889] mapped:8307 shmem:26412 pagetables:5427 bounce:0
> >[ 472.301901] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> >[ 472.301915] lowmem_reserve[]: 0 1856 1862 1862
> >[ 472.301928] Node 0 DMA32 free:786780kB min:66820kB low:83524kB high:100228kB active_anon:430692kB inactive_anon:148112kB active_file:125892kB inactive_file:126012kB unevictable:32kB isolated(anon):0kB isolated(file):128kB present:1901408kB mlocked:32kB dirty:105024kB writeback:25900kB mapped:33228kB shmem:105648kB slab_reclaimable:37104kB slab_unreclaimable:113052kB kernel_stack:2288kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3196 all_unreclaimable? no
> >[ 472.301943] lowmem_reserve[]: 0 0 5 5
> >[ 472.301956] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> >[ 472.301968] lowmem_reserve[]: 0 0 0 0
> >[ 472.301982] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
> >[ 472.302015] Node 0 DMA32: 6121*4kB 1912*8kB 5094*16kB 4920*32kB 3168*64kB 1381*128kB 262*256kB 68*512kB 24*1024kB 1*2048kB 0*4096kB = 786756kB
> >[ 472.302048] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
> >[ 472.302090] 91312 total pagecache pages
> >[ 472.302093] 0 pages in swap cache
> >[ 472.302096] Swap cache stats: add 0, delete 0, find 0/0
> >[ 472.302099] Free swap = 3768316kB
> >[ 472.302102] Total swap = 3768316kB
> >[ 472.309521] 525808 pages RAM
> >[ 472.309529] 57618 pages reserved
> >[ 472.309548] 142496 pages shared
> >[ 472.309551] 177182 pages non-shared
> >
> >I'll finish off this verification, and then re-run with the watch-highorder script running.
> >
> >James
> Hi,
>
> I saw similar lockups while doing a heavy copy routine (basically
> copying an entire partition). I don't think this may be related to
> ext4, since the source and destination filesystems were btrfs and
> xfs.
>
If you have SLUB configured, can you try again with the same two patches
I attached earlier in the thread please?
> <SNIP>
>
> The mv was basically frozen though after sometime it resumed. I also noticed that levels of dirty data were really high (=~ 1G) which I presumed may have caused this.
>
Is 1G higher than what /proc/sys/vm/dirty_ratio should allow?
--
Mel Gorman
SUSE Labs