LinuxLists.cc - Performance of ext4

[permalink] [raw]

Subject: Re: Performance of ext4

On Wed, Jun 11, 2008 at 08:02:32AM +0000, Holger Kiehl wrote:
> Hello
>
> Doing some performance test between ext3 and ext4 I noticed that ext4
> is not much faster or in some cases slower then ext3. Two years ago when
> I tested ext4 it was a lot faster then ext3 (see my mail:
> http://lkml.org/lkml/2006/6/6/65). Doing some simple tests with bonnie++
> I got the following results:
>
> Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> ext3(2 years ago)16G 38621 98 194816 94 87776 49 37921 92 239128 54 1402 5
> 16G 47000 99 194276 94 89232 49 38628 92 240539 55 1399 5
> 16G 45873 98 178195 90 89726 50 38482 92 240490 55 1381 4
>
> ext3 (now) 16G 51501 97 210601 91 100479 32 55528 98 301589 44 1198 5
> 16G 52702 98 215540 94 99339 32 55376 97 300933 44 1159 4
> 16G 52426 99 212584 94 99091 31 55656 98 301669 44 1160 4
>
> ext4(2 years ago)16G 59223 91 264155 45 111459 36 57313 99 317944 63 1478 7
> 16G 58814 92 276803 47 110418 36 57105 99 317534 65 1525 5
> 16G 58299 92 274523 48 110290 36 56723 99 318839 65 1502 4
>
> ext4 (now) 16G 52965 98 224199 89 108440 32 56389 99 303792 42 1526 4
> 16G 52792 98 223980 92 107685 32 56350 98 303066 42 1532 4
> 16G 52994 98 222354 92 107802 32 56386 99 303727 41 1455 4
>
> For this system the write performance is the most important factor and one
> can see today ext4 is marginally faster then ext3. But 2 years ago ext4 was
> a lot faster (~270MB against ~223MB).
>
> Using my own benchmark afdbench where many process copy thousands of small
> files around the results are as follows:
>
> For ext3: 5449.76 files per second 15.58 MiB/s
> For ext4: 5162.16 files per second 15.48 MiB/s
>
> So in this test ext4 is a bit slower then ext3. Since afdbench has seen
> considerable changes, one cannot compare these results with those 2 years
> ago. But 2 years ago ext4 was 12% faster then ext3.
>
> Test where done with kernel 2.6.25.4 and file system where created as follows:
>
> ext3: mke2fs -b 4096 -m 0 -O dir_index,large_file,filetype,has_journal,sparse_super -j /dev/md7
> ext4: mke2fs -b 4096 -E test_fs -m 0 -O dir_index,large_file,filetype,has_journal,sparse_super -j /dev/md7
>
> And both where mounted with the following options:
>
> noatime,nodiratime,commit=15
>
> 2 years ago I used 2.6.16.8 but the hardware is still the same. So what has
> happened with the performance of ext4? I noticed that 2 years ago I could
> use extents+mballoc+delalloc, now there is only extents+mballoc in the
> current kernels. Could delalloc make the big difference? I saw that
> in Andrew Morton mm tree delalloc is included. Unfortunately when I tried using
> 2.6.26-rc2-mm1 a sync would never return and there where lot of other
> odd things, so I could not do any tests with delalloc.

The sync and other related hangs should be fix with the latest patches
vailable at http://repo.or.cz/w/ext4-patch-queue.git. Using mballoc have
an impact on CPU utilization because we try to build an in-memory extent
map of free blocks available in the group. The cold cache run (the first
run) would take more time because of the time needed to build the extent
map. So repeating the same test and looking at the numbers would help us
understand the impact of in-memory extent building code.

>
> So any idea what I am doing wrong or what I could do to improve those numbers?
> Please CC me since I am not subscribed to the list.
>

You should be able to apply the patches in the patchqueue mentioned above to 2.6.26-rc5
Can you test with the same and get the numbers. ? Also delalloc enabled
by default with changes from patchqueue force writeback mode for
journal. So you may want to enable writeback for ext3.

-aneesh

2008-06-11 13:54:43

[permalink] [raw]

Subject: Re: Performance of ext4

On Wed, Jun 11, 2008 at 08:02:32AM +0000, Holger Kiehl wrote:
>
> Doing some performance test between ext3 and ext4 I noticed that ext4
> is not much faster or in some cases slower then ext3. Two years ago when
> I tested ext4 it was a lot faster then ext3 (see my mail:
> http://lkml.org/lkml/2006/6/6/65). Doing some simple tests with bonnie++
> I got the following results:

Hi Holger,

You didn't say exactly which version of the kernel/ext4 you were
testing, but a recent change which we made to ext4, but which hasn't
been made to ext3 yet is that barrier support has been enabled to
improve filesystem safety; unfortunately this does imply with it a
slight performance slowdown, which would be more pronounced on
benchmarks with small filesystems. So when you mount the filesystem
for ext3 and ext4 for benchmarking purposes, you should consistently
use a mount options of barrier=1 or barrier=0. With ext4, you can use
the mount option "barrier=1,journal_async_commit" which should
ameliorate part of the performance decrease. The reason why it is not
yet the default is it requires support from e2fsprogs that has not
been released except in development git branches; but as long as you
don't require running e2fsck on uncleanly shutdown systems (probably
not necessary if you are just benchmarking), you can use
journal_async_commit in good health.

Another change which might help out the bonnie benchmark, but which
again requires the latest version of e2fsprogs is to create the
filesystem with flex_bg filesystem feature. In fact, for
convenience's sake, if you are using the latest development version of
e2fsprogs, you can just use the command "mke2fs -t ext4dev /dev/hdXX"
and it will set up the filesystem with the correct filesystem features
for ext4. (The "ext4dev" sets the test_fs feature, and is basically
there so it's clear we are still trying to finish up ext4 support.)

> 2 years ago I used 2.6.16.8 but the hardware is still the same. So what has
> happened with the performance of ext4? I noticed that 2 years ago I could
> use extents+mballoc+delalloc, now there is only extents+mballoc in the
> current kernels. Could delalloc make the big difference? I saw that
> in Andrew Morton mm tree delalloc is included. Unfortunately when I tried
> using
> 2.6.26-rc2-mm1 a sync would never return and there where lot of other
> odd things, so I could not do any tests with delalloc.

As Aneesh has mentioned, there were some bugs in version of ext4's
delalloc caused by insufficient testing of the ext4 patch queue some
changes to our locking strategy went in. That's been fixed in the
latest patch queue, and we're in the process of cleaning up delalloc
before merging it into the mainline kernel. (When you tested ext4 two
years ago, none of this was yet in mainline, so it's not a matter of
things delalloc "disappearing", but rather that we've been slow
getting to the point where it was ready for merging.

- Ted

2008-06-11 19:58:55

[permalink] [raw]

Subject: Re: Performance of ext4

On Wed, 11 Jun 2008, Aneesh Kumar K.V wrote:

> On Wed, Jun 11, 2008 at 08:02:32AM +0000, Holger Kiehl wrote:
>>
>> 2 years ago I used 2.6.16.8 but the hardware is still the same. So what has
>> happened with the performance of ext4? I noticed that 2 years ago I could
>> use extents+mballoc+delalloc, now there is only extents+mballoc in the
>> current kernels. Could delalloc make the big difference? I saw that
>> in Andrew Morton mm tree delalloc is included. Unfortunately when I tried using
>> 2.6.26-rc2-mm1 a sync would never return and there where lot of other
>> odd things, so I could not do any tests with delalloc.
>
>
> The sync and other related hangs should be fix with the latest patches
> vailable at http://repo.or.cz/w/ext4-patch-queue.git. Using mballoc have
> an impact on CPU utilization because we try to build an in-memory extent
> map of free blocks available in the group. The cold cache run (the first
> run) would take more time because of the time needed to build the extent
> map. So repeating the same test and looking at the numbers would help us
> understand the impact of in-memory extent building code.
>
Not sure if I understand this. I first formated the file system and run the
test three times.

>>
>> So any idea what I am doing wrong or what I could do to improve those numbers?
>> Please CC me since I am not subscribed to the list.
>>
>
> You should be able to apply the patches in the patchqueue mentioned above to 2.6.26-rc5
> Can you test with the same and get the numbers. ? Also delalloc enabled
> by default with changes from patchqueue force writeback mode for
> journal. So you may want to enable writeback for ext3.
>
Against a clean 2.6.26-rc5 the patchqueue does not apply. Lots of rejects.
The patchqueue also contains changes for defrag that are not in 2.6.26-rc5.
Or did you mean to apply those against the mm tree?

Thanks,
Holger

2008-06-11 20:17:39

by Nick Dokos

[permalink] [raw]

Subject: Re: Performance of ext4

Holger Kiehl <[email protected]> wrote:

> Against a clean 2.6.26-rc5 the patchqueue does not apply. Lots of rejects.
> The patchqueue also contains changes for defrag that are not in 2.6.26-rc5.
> Or did you mean to apply those against the mm tree?
>

Just an FYI and a data point: I was able to apply the patch queue
against 2.6.26-rc5 (after I set up quilt correctly) without any
problems.

Regards,
Nick

2008-06-11 20:21:54

[permalink] [raw]

Subject: Re: Performance of ext4

On Wed, 11 Jun 2008, Theodore Tso wrote:

> On Wed, Jun 11, 2008 at 08:02:32AM +0000, Holger Kiehl wrote:
>>
>> Doing some performance test between ext3 and ext4 I noticed that ext4
>> is not much faster or in some cases slower then ext3. Two years ago when
>> I tested ext4 it was a lot faster then ext3 (see my mail:
>> http://lkml.org/lkml/2006/6/6/65). Doing some simple tests with bonnie++
>> I got the following results:
>
> Hi Holger,
>
> You didn't say exactly which version of the kernel/ext4 you were
> testing, but a recent change which we made to ext4, but which hasn't
> been made to ext3 yet is that barrier support has been enabled to
> improve filesystem safety; unfortunately this does imply with it a
> slight performance slowdown, which would be more pronounced on
> benchmarks with small filesystems.
>
I think you mean 'small files'? The test where done with a plain 2.6.25.4,
does that already have barriers enabled? While I did some test I hit this bug:

Jun 11 19:27:39 helena kernel: EXT4-fs: mballoc: 62053 blocks 62053 reqs (0 success)
Jun 11 19:27:39 helena kernel: EXT4-fs: mballoc: 62053 extents scanned, 14086 goal hits, 47967 2^N hits, 0 breaks, 0 lost
Jun 11 19:27:39 helena kernel: EXT4-fs: mballoc: 561 generated and it took 4080175
Jun 11 19:27:39 helena kernel: EXT4-fs: mballoc: 20889593 preallocated, 2761463 discarded
Jun 11 19:28:05 helena kernel: kjournald2 starting. Commit interval 15 seconds
Jun 11 19:28:05 helena kernel: EXT4 FS on md7, internal journal
Jun 11 19:28:05 helena kernel: EXT4-fs: mounted filesystem with ordered data mode.
Jun 11 19:28:05 helena kernel: EXT4-fs: file extents enabled
Jun 11 19:28:05 helena kernel: EXT4-fs: mballoc enabled
Jun 11 19:28:50 helena kernel: JBD: barrier-based sync failed on md7 - disabling barriers
Jun 11 19:28:50 helena kernel: ------------[ cut here ]------------
Jun 11 19:28:50 helena kernel: kernel BUG at fs/buffer.c:2879!
Jun 11 19:28:50 helena kernel: invalid opcode: 0000 [1] SMP
Jun 11 19:28:50 helena kernel: CPU 0
Jun 11 19:28:50 helena kernel: Modules linked in: w83627hf lm85 hwmon_vid bonding ipt_REJECT xt_tcpudp nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables binfmt_misc floppy i2c_amd756 k8temp i2c_core sg ohci_hcd button usbcore
Jun 11 19:28:50 helena kernel: Pid: 26731, comm: kjournald2 Not tainted 2.6.25.4 #12
Jun 11 19:28:50 helena kernel: RIP: 0010:[submit_bh+16/253] [submit_bh+16/253] submit_bh+0x10/0xfd
Jun 11 19:28:50 helena kernel: RSP: 0018:ffff8101775fdce0 EFLAGS: 00010246
Jun 11 19:28:50 helena kernel: RAX: 000000000000a02b RBX: ffff81005a182d90 RCX: ffff81027d7c7e60
Jun 11 19:28:50 helena kernel: RDX: 0000000100000000 RSI: ffff81005a182d90 RDI: 0000000000000001
Jun 11 19:28:50 helena kernel: RBP: 0000000000000001 R08: ffffffff8054cdd0 R09: 00000000fffff7e8
Jun 11 19:28:50 helena kernel: R10: ffff810001009940 R11: 0000000000000046 R12: ffff8101972ee000
Jun 11 19:28:50 helena kernel: R13: ffff81007ee79d00 R14: ffff8101775fde48 R15: 00000000fffffffb
Jun 11 19:28:50 helena kernel: FS: 00007f31d2ed67a0(0000) GS:ffffffff8057c000(0000) knlGS:0000000000000000
Jun 11 19:28:50 helena kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Jun 11 19:28:50 helena kernel: CR2: 00007f13a584a000 CR3: 000000027ee35000 CR4: 00000000000006e0
Jun 11 19:28:50 helena kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 11 19:28:50 helena kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jun 11 19:28:50 helena kernel: Process kjournald2 (pid: 26731, threadinfo ffff8101775fc000, task ffff81013773d680)
Jun 11 19:28:50 helena kernel: Stack: ffff8101972ee024 ffff81005a182d90 ffff8101972ee000 ffffffff802f5f96
Jun 11 19:28:50 helena kernel: ffff81000037646d ffffffff8023c8cf 0000000000000003 0000000000000287
Jun 11 19:28:50 helena kernel: ffff81005a182850 ffff810075df7240 0000000000000000 ffff8101972ee000
Jun 11 19:28:50 helena kernel: Call Trace:
Jun 11 19:28:50 helena kernel: [journal_submit_commit_record+315/331] ? journal_submit_commit_record+0x13b/0x14b
Jun 11 19:28:50 helena kernel: [bit_waitqueue+16/163] ? bit_waitqueue+0x10/0xa3
Jun 11 19:28:50 helena kernel: [jbd2_journal_commit_transaction+2811/3994] ? jbd2_journal_commit_transaction+0xafb/0xf9a
Jun 11 19:28:50 helena kernel: [lock_timer_base+38/76] ? lock_timer_base+0x26/0x4c
Jun 11 19:28:50 helena kernel: [kjournald2+223/514] ? kjournald2+0xdf/0x202
Jun 11 19:28:50 helena kernel: [autoremove_wake_function+0/46] ? autoremove_wake_function+0x0/0x2e
Jun 11 19:28:50 helena kernel: [kjournald2+0/514] ? kjournald2+0x0/0x202
Jun 11 19:28:50 helena kernel: [kthread+71/115] ? kthread+0x47/0x73
Jun 11 19:28:50 helena kernel: [child_rip+10/18] ? child_rip+0xa/0x12
Jun 11 19:28:50 helena kernel: [kthread+0/115] ? kthread+0x0/0x73
Jun 11 19:28:50 helena kernel: [child_rip+0/18] ? child_rip+0x0/0x12
Jun 11 19:28:50 helena kernel:
Jun 11 19:28:50 helena kernel:
Jun 11 19:28:50 helena kernel: Code: 5f 08 e8 14 fb ff ff 48 3b 5c 24 08 48 89 df eb eb 41 5b 5b 5b 89 e8 5d 41 5c c3 41 54 55 89 fd 53 48 8b 06 48 89 f3 a8 04 75 04 <0f> 0b eb fe a8 20 75 04 0f 0b eb fe 48 83 7e 38 00 75 04 0f 0b
Jun 11 19:28:50 helena kernel: RIP [submit_bh+16/253] submit_bh+0x10/0xfd
Jun 11 19:28:50 helena kernel: RSP <ffff8101775fdce0>
Jun 11 19:28:50 helena kernel: ---[ end trace 5b73613cd770052b ]---

That happened when the filesystem (mounted with barrier=0) was unmounted
and then mounted with barrier=1.

> So when you mount the filesystem
> for ext3 and ext4 for benchmarking purposes, you should consistently
> use a mount options of barrier=1 or barrier=0. With ext4, you can use
> the mount option "barrier=1,journal_async_commit" which should
> ameliorate part of the performance decrease. The reason why it is not
> yet the default is it requires support from e2fsprogs that has not
> been released except in development git branches; but as long as you
> don't require running e2fsck on uncleanly shutdown systems (probably
> not necessary if you are just benchmarking), you can use
> journal_async_commit in good health.
>
> Another change which might help out the bonnie benchmark, but which
> again requires the latest version of e2fsprogs is to create the
> filesystem with flex_bg filesystem feature. In fact, for
> convenience's sake, if you are using the latest development version of
> e2fsprogs, you can just use the command "mke2fs -t ext4dev /dev/hdXX"
> and it will set up the filesystem with the correct filesystem features
> for ext4. (The "ext4dev" sets the test_fs feature, and is basically
> there so it's clear we are still trying to finish up ext4 support.)
>
Thank you for the detailed suggestions. I will try those and post the
results as soon as I have them.

Holger

2008-06-12 01:35:13

[permalink] [raw]

Subject: Re: Performance of ext4

On Wed, Jun 11, 2008 at 08:21:50PM +0000, Holger Kiehl wrote:
> I think you mean 'small files'?

Yes, sorry, I meant "small files".

> The test where done with a plain 2.6.25.4,
> does that already have barriers enabled? While I did some test I hit this
> bug:

No, 2.6.25.4 does not have barriers enabled. It's in the latest
post-2.6.25.5 git repository (which also has the fix so that
barriers=1 doesn't cause an OOPS if you are doing the test on a LVM
device which doesn't support barriers). Sorry, I wasnt sure if you
were using any of the ext4 patches, and I didn't know whether you were
testing on a raw device or an LVM device.

- Ted

2008-06-12 09:02:39

[permalink] [raw]

Subject: Re: Performance of ext4

On Wed, 11 Jun 2008, Nick Dokos wrote:

> Holger Kiehl <[email protected]> wrote:
>
>> Against a clean 2.6.26-rc5 the patchqueue does not apply. Lots of rejects.
>> The patchqueue also contains changes for defrag that are not in 2.6.26-rc5.
>> Or did you mean to apply those against the mm tree?
>>
>
> Just an FYI and a data point: I was able to apply the patch queue
> against 2.6.26-rc5 (after I set up quilt correctly) without any
> problems.
>
I have never used quilt, so how do I set it up correctly? I am using
Fedora 9 and have copied all patches from ext4-patch-queue into
2.6.26-rc5/patches. Then I always issue the following command in
2.6.26-rc5 directory:

quilt push

That works until it comes to patch patches/ext4_ialloc-flexbg.patch when
it gets lots of rejects. So how do I set this up correctly?

Thanks,
Holger

2008-06-12 10:58:07

by Solofo.Ramangalahy

[permalink] [raw]

Subject: Re: Performance of ext4

Hi Holger,

Holger Kiehl writes:
> That works until it comes to patch patches/ext4_ialloc-flexbg.patch when
> it gets lots of rejects. So how do I set this up correctly?

I think your quilt is set up correctly.

The rejects probably come from the patch queue being based on a
different version than 2.6.26-rc5.

Which patchset did you use?

What I do when wanting a patch queue for a specific kernel is to
search for commit message indicating the rebase for this version.

There is a "Rebase to linux-2.6.26-rc5" on
http://repo.or.cz/w/ext4-patch-queue.git
So the corresponding snapshot
http://repo.or.cz/w/ext4-patch-queue.git?a=snapshot;h=ed6aaac3b0cc0ba47dbb2af0884306584271e5a2;sf=tgz
patches 2.6.26-rc5 smoothly.
(the last patchset also applies, with offset and fuzz, though)

--
solofo

2008-06-12 12:01:02

[permalink] [raw]

Subject: Re: Performance of ext4

Hello Solofo

On Thu, 12 Jun 2008, [email protected] wrote:

> Hi Holger,
>
> Holger Kiehl writes:
> > That works until it comes to patch patches/ext4_ialloc-flexbg.patch when
> > it gets lots of rejects. So how do I set this up correctly?
>
> I think your quilt is set up correctly.
>
> The rejects probably come from the patch queue being based on a
> different version than 2.6.26-rc5.
>
> Which patchset did you use?
>
> What I do when wanting a patch queue for a specific kernel is to
> search for commit message indicating the rebase for this version.
>
> There is a "Rebase to linux-2.6.26-rc5" on
> http://repo.or.cz/w/ext4-patch-queue.git
> So the corresponding snapshot
> http://repo.or.cz/w/ext4-patch-queue.git?a=snapshot;h=ed6aaac3b0cc0ba47dbb2af0884306584271e5a2;sf=tgz
> patches 2.6.26-rc5 smoothly.
> (the last patchset also applies, with offset and fuzz, though)
>
Yes, that is the one I took. Just to make sure, I downloaded it again (and
the linux-2.6.26-rc5 tree) and now it works. When I compare the two patchsets
the one I pulled this morning had patches from 7 June and this one has more
and they are from 11th June.

Thanks a lot for your help, at last I now have all the patches and can go
on with testing.

Thanks,
Holger

2008-06-12 13:20:17

[permalink] [raw]

Subject: Re: Performance of ext4

On Thu, Jun 12, 2008 at 12:00:54PM +0000, Holger Kiehl wrote:
> Yes, that is the one I took. Just to make sure, I downloaded it
> again (and the linux-2.6.26-rc5 tree) and now it works. When I
> compare the two patchsets the one I pulled this morning had patches
> from 7 June and this one has more and they are from 11th June.

Note that you don't have to download things from scratch, necessarily;
the git command "git pull" will (if you havent made any changes in the
git repository; if you have there are a few more steps you might have
to take) pull down any newer changes and update your patch directory
directly.

Note that after you type "git pull", you may find that the
ext4-patch-queue has been rebased to a newer version of the kernel.
For example, the patch queue is currently against 2.6.26-rc5. The
latest bleedig edge kernel as released by Linus in his git tree of the
kernel already has these patches:

ext4-fix-use-of-uninitalized-data.patch
ext4-fix-uninit-block-group-initialization-with-FLEX_BG.patch
ext4-fix-onine-resize-bug.patch
fix-journal-checksum-mem-leak
jbd2-if-a-journal-checksum-error-is-detected-propa.patch
show-journal-async-commit-option
jbd2-Fix-barrier-fallback-code-to-re-lock-the-buffe.patch
ext4-enable-barriers-by-default.patch

from the patch series. But, he hasn't released an official kernel
release for 2.6.26-rc5 yet. So in the near future, the ext4 patch
queue will probably get rebased against one of the "daily snapshots",
as found in /pub/linux/kernel/v2.6/snapshots on ftp.kernel.org. So
for example, 2.6.26-rc5-git6 is the sixth snapshot since -rc5 was
released. There is a patch against -rc5 named
patch-2.6.26-rc5-git6.bz2 as located in the aforementioned directory
on ftp.kernel.org, or if you are tracking Linus's kernel git tree, the
file patch-2.6.26-rc5-git6.id proclaims that -rc5-git6 is git id
631025b. You will see that when you look at the first line of the
series file in the ext4 patch queue, where it currently states:

# BASE 2.6.26-rc5

and where it will in the future say something like this:

# BASE 2.6.26-rc5-git6

That can be used by automated scripts to automatically determine which
version of the Linux kernel should be grabbed before trying to apply
the latest version of the ext4 patch queue. In general we try to use
mainly official -rc1, -rc2, etc. release points, but after Linus
pulled down some stable bug fixes, we will sometimes use a daily git
snapshot release such as -rc5-git6 as described above.

Hope this helps,

- Ted

2008-06-12 14:07:37

[permalink] [raw]

Subject: Re: Performance of ext4

Hello Ted

On Thu, 12 Jun 2008, Theodore Tso wrote:

> On Thu, Jun 12, 2008 at 12:00:54PM +0000, Holger Kiehl wrote:
>> Yes, that is the one I took. Just to make sure, I downloaded it
>> again (and the linux-2.6.26-rc5 tree) and now it works. When I
>> compare the two patchsets the one I pulled this morning had patches
>> from 7 June and this one has more and they are from 11th June.
>
> Note that you don't have to download things from scratch, necessarily;
> the git command "git pull" will (if you havent made any changes in the
> git repository; if you have there are a few more steps you might have
> to take) pull down any newer changes and update your patch directory
> directly.
>
> Note that after you type "git pull", you may find that the
> ext4-patch-queue has been rebased to a newer version of the kernel.
> For example, the patch queue is currently against 2.6.26-rc5. The
> latest bleedig edge kernel as released by Linus in his git tree of the
> kernel already has these patches:
>
> ext4-fix-use-of-uninitalized-data.patch
> ext4-fix-uninit-block-group-initialization-with-FLEX_BG.patch
> ext4-fix-onine-resize-bug.patch
> fix-journal-checksum-mem-leak
> jbd2-if-a-journal-checksum-error-is-detected-propa.patch
> show-journal-async-commit-option
> jbd2-Fix-barrier-fallback-code-to-re-lock-the-buffe.patch
> ext4-enable-barriers-by-default.patch
>
> from the patch series. But, he hasn't released an official kernel
> release for 2.6.26-rc5 yet. So in the near future, the ext4 patch
>
I think this should be 2.6.26-rc6?

> queue will probably get rebased against one of the "daily snapshots",
> as found in /pub/linux/kernel/v2.6/snapshots on ftp.kernel.org. So
> for example, 2.6.26-rc5-git6 is the sixth snapshot since -rc5 was
> released. There is a patch against -rc5 named
> patch-2.6.26-rc5-git6.bz2 as located in the aforementioned directory
> on ftp.kernel.org, or if you are tracking Linus's kernel git tree, the
> file patch-2.6.26-rc5-git6.id proclaims that -rc5-git6 is git id
> 631025b. You will see that when you look at the first line of the
> series file in the ext4 patch queue, where it currently states:
>
> # BASE 2.6.26-rc5
>
> and where it will in the future say something like this:
>
> # BASE 2.6.26-rc5-git6
>
> That can be used by automated scripts to automatically determine which
> version of the Linux kernel should be grabbed before trying to apply
> the latest version of the ext4 patch queue. In general we try to use
> mainly official -rc1, -rc2, etc. release points, but after Linus
> pulled down some stable bug fixes, we will sometimes use a daily git
> snapshot release such as -rc5-git6 as described above.
>
> Hope this helps,
>
Yes, thanks a lot. I still have not yet managed to get this to work
with git. But downloading linux-2.6.26-rc5.tar.bz2 and then copying
the ext4-patch-queue into linux-2.6.26-rc5/patches and then using
quilt pull did get all the patches applied.

Here are the first test results:

Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ext4(patchqueue)16G 59727 98 252733 52 110177 25 55821 98 296739 42 1553 5
16G 61047 99 239242 48 111664 25 55706 98 297151 42 1545 4
16G 60503 99 241532 47 109655 25 55671 98 297648 42 1552 3

That is with 2.6.26-rc5 + ext4-patch-queue and filesystem was created with
latest e2fsprogs snapshot as you suggested and formated with the following
command:

mke2fs -t ext4dev /dev/md7

Besides, I am doing those tests on a software raid 1+0. I think that is also
the reason why barriers are disabled by defaults:

Jun 12 12:17:48 helena kernel: kjournald2 starting. Commit interval 15 seconds
Jun 12 12:17:48 helena kernel: EXT4 FS on md7, internal journal
Jun 12 12:17:48 helena kernel: EXT4-fs: mounted filesystem with writeback data mode.
Jun 12 12:17:48 helena kernel: EXT4-fs: file extents enabled
Jun 12 12:17:48 helena kernel: EXT4-fs: mballoc enabled
Jun 12 12:18:26 helena kernel: JBD: barrier-based sync failed on md7 - disabling barriers

If one compares the results with those of 2.6.25.4 (without ext4-patch-queue):

Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ext4 (2.6.25.4) 16G 52965 98 224199 89 108440 32 56389 99 303792 42 1526 4
16G 52792 98 223980 92 107685 32 56350 98 303066 42 1532 4
16G 52994 98 222354 92 107802 32 56386 99 303727 41 1455 4

This is quit an improvement but still does not match those numbers two years
ago.

However there must be still some bug, I am unable to run my own afdbench
benchmark on this (2.6.26-rc5 + ext4-patch-queue). It starts fine but then
about 3 minutes into the test (the test runs ~60 minutes) all process start
hanging in D-state (here a list):

afdbench 16381 0.0 0.0 4064 760 ? D 13:08 0:00 sf_ftp /home/afdbench/afd3 0 0 1 22765b95/0/48511fd7_2db_0
afdbench 16388 0.0 0.0 4064 760 ? D 13:08 0:00 sf_ftp /home/afdbench/afd3 3 0 1 22765b95/0/48511fd7_2dc_0
nobody 16391 0.0 0.0 44304 1444 ? Ss 13:08 0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
afdbench 16395 0.0 0.0 4064 760 ? D 13:08 0:00 sf_ftp /home/afdbench/afd3 5 0 1 22765b95/0/48511fd7_2dd_0
nobody 16397 0.0 0.0 44304 1448 ? Ss 13:08 0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
nobody 16400 0.0 0.0 44304 1448 ? Ss 13:08 0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
afdbench 16404 0.0 0.0 44328 992 ? D 13:08 0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
afdbench 16409 0.0 0.0 44328 996 ? D 13:08 0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
afdbench 16411 0.0 0.0 44328 996 ? D 13:08 0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
afdbench 16848 0.0 0.0 4064 716 ? D 13:08 0:00 sf_ftp /home/afdbench/afd1 2 0 0 820dc3d0/0/48511fdd_2f3_0
afdbench 16855 0.0 0.0 4064 712 ? D 13:08 0:00 sf_ftp /home/afdbench/afd1 1 0 7 cd52409d/0/48511fdd_2f3_0
afdbench 16857 0.0 0.0 4064 708 ? D 13:08 0:00 sf_ftp /home/afdbench/afd1 0 0 8 e1d94fe0/0/48511fdd_2f3_0
afdbench 16861 0.0 0.0 4064 708 ? D 13:08 0:00 sf_ftp /home/afdbench/afd1 2 0 9 26686393/0/48511fdd_2f3_0
afdbench 16865 0.0 0.0 4064 704 ? D 13:08 0:00 sf_ftp /home/afdbench/afd1 0 0 10 ae36cee/0/48511fdd_2f3_0
afdbench 16871 0.0 0.0 4064 716 ? D 13:08 0:00 sf_ftp /home/afdbench/afd2 1 0 0 bca9d45f/0/48511fdd_2d8_0
afdbench 16876 0.0 0.0 4064 716 ? D 13:08 0:00 sf_ftp /home/afdbench/afd1 1 0 0 820dc3d0/0/48511fdd_2f4_0
afdbench 16878 0.0 0.0 4064 772 ? D 13:08 0:00 sf_ftp /home/afdbench/afd1 0 0 6 b8cf511a/0/48511fdd_2f4_0
nobody 16879 0.0 0.0 44304 1448 ? Ss 13:08 0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
afdbench 16881 0.0 0.0 4064 704 ? D 13:08 0:00 sf_ftp /home/afdbench/afd1 0 0 7 cd52409d/0/48511fdd_2f4_0
afdbench 16885 0.0 0.0 44404 1024 ? D 13:08 0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
afdbench 16886 0.0 0.0 4064 712 ? D 13:08 0:00 sf_ftp /home/afdbench/afd1 2 0 8 e1d94fe0/0/48511fdd_2f4_0
afdbench 16890 0.0 0.0 4064 716 ? D 13:08 0:00 sf_ftp /home/afdbench/afd2 2 0 0 bca9d45f/0/48511fdd_2d9_0
afdbench 16891 0.0 0.0 4064 712 ? D 13:08 0:00 sf_ftp /home/afdbench/afd1 0 0 9 26686393/0/48511fdd_2f4_0
afdbench 16892 0.0 0.0 4064 712 ? D 13:08 0:00 sf_ftp /home/afdbench/afd1 1 0 10 ae36cee/0/48511fdd_2f4_0

This time there is no OOPS and system is still up running without any
problem (except any process wanting to write something to this filesystem
gets stuck forever).

What can I do to help find the problem? The system is still up with all those
process hanging in D-state.

Thanks,
Holger

2008-06-12 18:06:27

[permalink] [raw]

Subject: Re: Performance of ext4

On Thu, Jun 12, 2008 at 02:07:30PM +0000, Holger Kiehl wrote:
> This time there is no OOPS and system is still up running without any
> problem (except any process wanting to write something to this filesystem
> gets stuck forever).
>
> What can I do to help find the problem? The system is still up with all those
> process hanging in D-state.
>

if you can login to the system get the dmesg output after

echo t > /proc/sysrq-trigger

-aneesh

2008-06-12 19:50:43

[permalink] [raw]

Subject: Re: Performance of ext4

On Thu, 12 Jun 2008, Aneesh Kumar K.V wrote:

> On Thu, Jun 12, 2008 at 02:07:30PM +0000, Holger Kiehl wrote:
>> This time there is no OOPS and system is still up running without any
>> problem (except any process wanting to write something to this filesystem
>> gets stuck forever).
>>
>> What can I do to help find the problem? The system is still up with all those
>> process hanging in D-state.
>>
>
> if you can login to the system get the dmesg output after
>
> echo t > /proc/sysrq-trigger
>
Unfortunately I have not set CONFIG_MAGIC_SYSRQ. Tomorrow I will try to
reproduce this with a kernel that has CONFIG_MAGIC_SYSRQ set.

Holger

2008-06-13 08:05:22

[permalink] [raw]

Subject: Re: Performance of ext4

On Thu, 12 Jun 2008, Holger Kiehl wrote:

> On Thu, 12 Jun 2008, Aneesh Kumar K.V wrote:
>
>> On Thu, Jun 12, 2008 at 02:07:30PM +0000, Holger Kiehl wrote:
>>> This time there is no OOPS and system is still up running without any
>>> problem (except any process wanting to write something to this filesystem
>>> gets stuck forever).
>>>
>>> What can I do to help find the problem? The system is still up with all
>>> those
>>> process hanging in D-state.
>>>
>>
>> if you can login to the system get the dmesg output after
>>
>> echo t > /proc/sysrq-trigger
>>
> Unfortunately I have not set CONFIG_MAGIC_SYSRQ. Tomorrow I will try to
> reproduce this with a kernel that has CONFIG_MAGIC_SYSRQ set.
>
After recompiling, rebooting and run afdbench first I got an OOPS and the
system hanged up solid. The only thing I was able to catch is this:

RIP [<ffffffff803019f9>] jbd2_journal_release_jbd_inode+0xcb/0x100
RSP <ffff8101fe259c18>

This was copied by hand. And this I cut 'cut and past' from my terminal:

kernel: Code: c3 e8 31 ce f3 ff 41 fe 04 24 e8 fe 3f 16 00 4c 89 fe 48 89 df e8 5f cd f3 ff eb 82 48 83 7d 00 00 74 27 48 8b 55 10 48 8b 45 18 <48> 89 42 08 48 89 10 48 c7 45 18 00 02 20 00 48 c7 45 10 00 01

So I rebooted run bonnie and then afdbench and I get the same problem as
yesterday. All process trying to write something to this filesystem hang
in D-state. But now I was able to do the echo t > /proc/sysrq-trigger.
Here the dmesg output:

node_dirty+0x134/0x147
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff802959f7>] file_update_time+0xaf/0xf4
[<ffffffff8025cfb5>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80295a4e>] touch_atime+0x12/0xfa
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d799>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9ed8>] ext4_file_write+0x96/0x11a
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff8022a715>] hrtick_set+0xa1/0x10a
[<ffffffff802846ac>] vfs_write+0xad/0x136
[<ffffffff802847f1>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

trans_db_log S 00000000ffffffff 0 8010 8005
ffff81016f58d9e8 0000000000000082 0000000000000002 0000000000000000
ffffffff80600cc0 ffff810180001600 0000000000000000 ffff81017f6c1640
ffff81007fb43210 ffff81017f6c1878 0000000280266f65 ffff81017f6c1878
Call Trace:
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffff80235889>] __mod_timer+0xaa/0xb8
[<ffffffff80466249>] schedule_timeout+0xa6/0xc9
[<ffffffff80235392>] process_timeout+0x0/0x5
[<ffffffff80290728>] do_select+0x414/0x482
[<ffffffff80290dd5>] __pollwait+0x0/0xe3
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff802dca1c>] ext4_mark_inode_dirty+0x134/0x147
[<ffffffff802ec6aa>] __ext4_journal_dirty_metadata+0x1e/0x46
[<ffffffff8023e617>] bit_waitqueue+0x10/0xa0
[<ffffffff802fe4d7>] do_get_write_access+0x3ab/0x3ed
[<ffffffff802a3c0f>] __getblk+0x1e/0x28d
[<ffffffff8025ba69>] find_lock_page+0x29/0x87
[<ffffffff8025c094>] filemap_fault+0x1cb/0x329
[<ffffffff80268f59>] __do_fault+0x321/0x369
[<ffffffff80290941>] core_sys_select+0x1ab/0x253
[<ffffffff8026a7dc>] handle_mm_fault+0x348/0x6a3
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8028e58a>] do_filp_open+0x3f2/0x832
[<ffffffff80290d25>] sys_select+0xb7/0x167
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

archive_watch S 00000000ffffffff 0 8011 8005
ffff81017f79b9e8 0000000000000086 ffff810000002618 0000000100000000
ffffffff80600cc0 0000000000000000 ffff810000002608 ffff81017f6c10b0
ffffffff8054d350 ffff81017f6c12e8 0000000000001580 ffff81017f6c12e8
Call Trace:
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffff80235889>] __mod_timer+0xaa/0xb8
[<ffffffff80466249>] schedule_timeout+0xa6/0xc9
[<ffffffff80235392>] process_timeout+0x0/0x5
[<ffffffff80290728>] do_select+0x414/0x482
[<ffffffff80290dd5>] __pollwait+0x0/0xe3
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff802a3be1>] __find_get_block+0x14c/0x15c
[<ffffffff802a3c0f>] __getblk+0x1e/0x28d
[<ffffffff802de7f2>] ext4_getblk+0xb4/0x170
[<ffffffff80293dad>] dput+0x21/0x10a
[<ffffffff80294aed>] __d_lookup+0xbd/0x107
[<ffffffff802e6fbf>] ext4fs_dirhash+0x10d/0x1d2
[<ffffffff802d9d81>] ext4_htree_store_dirent+0xe6/0xf1
[<ffffffff802e3745>] htree_dirblock_to_tree+0x12c/0x13e
[<ffffffff80290941>] core_sys_select+0x1ab/0x253
[<ffffffff8028f91c>] filldir+0x68/0xb7
[<ffffffff8028f8b4>] filldir+0x0/0xb7
[<ffffffff802d9938>] ext4_readdir+0x1c1/0x524
[<ffffffff8028f8b4>] filldir+0x0/0xb7
[<ffffffff8028e572>] do_filp_open+0x3da/0x832
[<ffffffff80286f15>] cp_new_stat+0xe9/0xfc
[<ffffffff802985f0>] mnt_want_write+0x25/0x77
[<ffffffff80290d25>] sys_select+0xb7/0x167
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

input_log D 00000000ffffffff 0 8012 8005
ffff81017f79db58 0000000000000086 0000000000000008 0000000000000000
ffffffff80600cc0 0000000000000008 0000000000000000 ffff81017f6c5370
ffff81007fb442c0 ffff81017f6c55a8 0000000380290dd5 ffff81017f6c55a8
Call Trace:
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802dca1c>] ext4_mark_inode_dirty+0x134/0x147
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff802959f7>] file_update_time+0xaf/0xf4
[<ffffffff8025cfb5>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80295a4e>] touch_atime+0x12/0xfa
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d799>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9ed8>] ext4_file_write+0x96/0x11a
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff8022a715>] hrtick_set+0xa1/0x10a
[<ffffffff802846ac>] vfs_write+0xad/0x136
[<ffffffff802847f1>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

output_log D 00000000ffffffff 0 8013 8005
ffff81016f7b9b58 0000000000000086 0000000000000008 0000000000000000
ffffffff80600cc0 0000000000000008 0000000000000000 ffff81017f6c4de0
ffff81007fb42160 ffff81017f6c5018 0000000180290dd5 ffff81017f6c5018
Call Trace:
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802dca1c>] ext4_mark_inode_dirty+0x134/0x147
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff802959f7>] file_update_time+0xaf/0xf4
[<ffffffff8025cfb5>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80295a4e>] touch_atime+0x12/0xfa
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d799>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9ed8>] ext4_file_write+0x96/0x11a
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff8022a715>] hrtick_set+0xa1/0x10a
[<ffffffff802846ac>] vfs_write+0xad/0x136
[<ffffffff802847f1>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

delete_log S 0000000000000000 0 8014 8005
ffff81017f7099e8 0000000000000086 0000000000000002 0000000000000000
ffffffff80600cc0 ffff810000002600 0000000000000000 ffff81017f6c26f0
ffff81017f784850 ffff81017f6c2928 0000000080266f65 ffff81017f6c2928
Call Trace:
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffff80235889>] __mod_timer+0xaa/0xb8
[<ffffffff80466249>] schedule_timeout+0xa6/0xc9
[<ffffffff80235392>] process_timeout+0x0/0x5
[<ffffffff80290728>] do_select+0x414/0x482
[<ffffffff80290dd5>] __pollwait+0x0/0xe3
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff802dca1c>] ext4_mark_inode_dirty+0x134/0x147
[<ffffffff802ec6aa>] __ext4_journal_dirty_metadata+0x1e/0x46
[<ffffffff80266f65>] zone_statistics+0x3c/0x90
[<ffffffff8026112c>] get_page_from_freelist+0x461/0x5d5
[<ffffffff80266f65>] zone_statistics+0x3c/0x90
[<ffffffff8026112c>] get_page_from_freelist+0x461/0x5d5
[<ffffffff80261447>] __alloc_pages_internal+0xd2/0x40c
[<ffffffff80290941>] core_sys_select+0x1ab/0x253
[<ffffffff8026a75d>] handle_mm_fault+0x2c9/0x6a3
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8026f46b>] mmap_region+0x41a/0x505
[<ffffffff80290d25>] sys_select+0xb7/0x167
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

production_lo S 00000000ffffffff 0 8015 8005
ffff81017f70b9e8 0000000000000082 0000000000000002 0000000000000000
ffffffff80600cc0 ffff810200001600 0000000000000000 ffff81017f6c42c0
ffff81007fb43210 ffff81017f6c44f8 00000002794b5d20 ffff81017f6c44f8
Call Trace:
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffff80235889>] __mod_timer+0xaa/0xb8
[<ffffffff80466249>] schedule_timeout+0xa6/0xc9
[<ffffffff80235392>] process_timeout+0x0/0x5
[<ffffffff80290728>] do_select+0x414/0x482
[<ffffffff80290dd5>] __pollwait+0x0/0xe3
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff802e6dc1>] __ext4_journal_stop+0x1f/0x3d
[<ffffffff802a0156>] __mark_inode_dirty+0xed/0x187
[<ffffffff8025c732>] generic_file_buffered_write+0x22c/0x658
[<ffffffff802a0156>] __mark_inode_dirty+0xed/0x187
[<ffffffff8025d09a>] __generic_file_aio_write_nolock+0x34a/0x37e
[<ffffffff80268f59>] __do_fault+0x321/0x369
[<ffffffff80290941>] core_sys_select+0x1ab/0x253
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8026f46b>] mmap_region+0x41a/0x505
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80290d25>] sys_select+0xb7/0x167
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

distribution_ S 00000000ffffffff 0 8016 8005
ffff81016f5959e8 0000000000000086 0000000000080780 ffffe20008ba3f98
ffffffff80600cc0 ffffffff802a3869 0000fee20000011d ffff81017f6c3d30
ffff81007fb442c0 ffff81017f6c3f68 00000003794b5d20 ffff81017f6c3f68
Call Trace:
[<ffffffff802a3869>] __find_get_block_slow+0xcf/0xdb
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffff80235889>] __mod_timer+0xaa/0xb8
[<ffffffff80466249>] schedule_timeout+0xa6/0xc9
[<ffffffff80235392>] process_timeout+0x0/0x5
[<ffffffff80290728>] do_select+0x414/0x482
[<ffffffff80290dd5>] __pollwait+0x0/0xe3
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff802e6dc1>] __ext4_journal_stop+0x1f/0x3d
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff8025c732>] generic_file_buffered_write+0x22c/0x658
[<ffffffff802dca1c>] ext4_mark_inode_dirty+0x134/0x147
[<ffffffff8023136c>] current_fs_time+0x1e/0x24
[<ffffffff8025b358>] remove_suid+0x18/0x4b
[<ffffffff8025d09a>] __generic_file_aio_write_nolock+0x34a/0x37e
[<ffffffff80295a4e>] touch_atime+0x12/0xfa
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff80290941>] core_sys_select+0x1ab/0x253
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff8022a715>] hrtick_set+0xa1/0x10a
[<ffffffff80290d25>] sys_select+0xb7/0x167
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

amg S 00000000ffffffff 0 8017 8005
ffff81016f5e59e8 0000000000000082 ffff81023ec9a070 0000000000000282
ffffffff80600cc0 ffff81007f2e4ac0 0000000000011210 ffff81017f6c37a0
ffffffff8054d350 ffff81017f6c39d8 0000000000000040 ffff81017f6c39d8
Call Trace:
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffff80235889>] __mod_timer+0xaa/0xb8
[<ffffffff80466249>] schedule_timeout+0xa6/0xc9
[<ffffffff80235392>] process_timeout+0x0/0x5
[<ffffffff80290728>] do_select+0x414/0x482
[<ffffffff80290dd5>] __pollwait+0x0/0xe3
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff802a3c0f>] __getblk+0x1e/0x28d
[<ffffffff80266f65>] zone_statistics+0x3c/0x90
[<ffffffff80294aed>] __d_lookup+0xbd/0x107
[<ffffffff8028ad6a>] do_lookup+0x63/0x1a1
[<ffffffff80293dad>] dput+0x21/0x10a
[<ffffffff80294aed>] __d_lookup+0xbd/0x107
[<ffffffff8028ad6a>] do_lookup+0x63/0x1a1
[<ffffffff80293dad>] dput+0x21/0x10a
[<ffffffff8028cebc>] __link_path_walk+0xcb1/0xdd9
[<ffffffff80290941>] core_sys_select+0x1ab/0x253
[<ffffffff8028c1a9>] getname+0x13e/0x1a0
[<ffffffff80299236>] mntput_no_expire+0x27/0x11e
[<ffffffff8023e876>] remove_wait_queue+0x12/0x41
[<ffffffff80230135>] do_wait+0xae4/0xb88
[<ffffffff80286f15>] cp_new_stat+0xe9/0xfc
[<ffffffff80290d25>] sys_select+0xb7/0x167
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

afdd S 00000000ffffffff 0 8018 8005
ffff81016f5e79e8 0000000000000086 ffff810200001618 0000000200000000
ffffffff80600cc0 0000000000000000 ffff810200001608 ffff81017f6c0000
ffff81007fb442c0 ffff81017f6c0238 0000000300000b00 ffff81017f6c0238
Call Trace:
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffff80235889>] __mod_timer+0xaa/0xb8
[<ffffffff80466249>] schedule_timeout+0xa6/0xc9
[<ffffffff80235392>] process_timeout+0x0/0x5
[<ffffffff80290728>] do_select+0x414/0x482
[<ffffffff80290dd5>] __pollwait+0x0/0xe3
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff8036750a>] extract_buf+0x84/0xfc
[<ffffffff80367166>] mix_pool_bytes_extract+0x58/0x148
[<ffffffff8036750a>] extract_buf+0x84/0xfc
[<ffffffff80290941>] core_sys_select+0x1ab/0x253
[<ffffffff80367476>] account+0xd6/0xe6
[<ffffffff80224a67>] __wake_up+0x38/0x4f
[<ffffffff80290d25>] sys_select+0xb7/0x167
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check D ffff810027160690 0 8019 8017
ffff81007e8b1d70 0000000000000086 ffff81007e8b1d38 ffff81007e8a0e00
ffffffff80600cc0 0000000000000000 ffffffff80293dad ffff8100707910b0
ffff8100469a2160 ffff8100707912e8 000000028028c8f9 ffff8100707912e8
Call Trace:
[<ffffffff80293dad>] dput+0x21/0x10a
[<ffffffff8028cebc>] __link_path_walk+0xcb1/0xdd9
[<ffffffff80299236>] mntput_no_expire+0x27/0x11e
[<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466484>] mutex_lock+0xa/0xb
[<ffffffff8028ac27>] lock_rename+0x35/0xc5
[<ffffffff8028d51e>] do_rename+0x98/0x1b2
[<ffffffff80286f15>] cp_new_stat+0xe9/0xfc
[<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d684>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

afd_stat S 00000000ffffffff 0 8020 8005
ffff81016f7fd9e8 0000000000000086 0000000000000000 000000000009829c
ffffffff80600cc0 ffffffff80323d50 ffff810201e124a0 ffff81017f6c0590
ffff81007fb442c0 ffff81017f6c07c8 0000000300000000 ffff81017f6c07c8
Call Trace:
[<ffffffff80323d50>] rb_insert_color+0x66/0xe2
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffff80235889>] __mod_timer+0xaa/0xb8
[<ffffffff80466249>] schedule_timeout+0xa6/0xc9
[<ffffffff80235392>] process_timeout+0x0/0x5
[<ffffffff80290728>] do_select+0x414/0x482
[<ffffffff8023e617>] bit_waitqueue+0x10/0xa0
[<ffffffff80290dd5>] __pollwait+0x0/0xe3
[<ffffffff802fe4d7>] do_get_write_access+0x3ab/0x3ed
[<ffffffff802a3c0f>] __getblk+0x1e/0x28d
[<ffffffff8023e724>] wake_bit_function+0x0/0x23
[<ffffffff802ec6aa>] __ext4_journal_dirty_metadata+0x1e/0x46
[<ffffffff802dbbee>] ext4_mark_iloc_dirty+0x415/0x47e
[<ffffffff802dca1c>] ext4_mark_inode_dirty+0x134/0x147
[<ffffffff802db3e9>] walk_page_buffers+0x65/0x8b
[<ffffffff802db568>] ext4_bh_unmapped+0x0/0xe
[<ffffffff80324411>] __up_read+0x13/0x8a
[<ffffffff80322db6>] __prop_inc_single+0x1d/0x36
[<ffffffff8026941e>] do_wp_page+0x47d/0x4ba
[<ffffffff80290941>] core_sys_select+0x1ab/0x253
[<ffffffff8026aac1>] handle_mm_fault+0x62d/0x6a3
[<ffffffff8022a715>] hrtick_set+0xa1/0x10a
[<ffffffff80324411>] __up_read+0x13/0x8a
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
[<ffffffff80286f15>] cp_new_stat+0xe9/0xfc
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80290d25>] sys_select+0xb7/0x167
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

fd S 00000000ffffffff 0 8021 8005
ffff81016f5139e8 0000000000000082 0000000000000000 0000000000000000
ffffffff80600cc0 0000000000000001 0000000180001608 ffff81017f6c3210
ffff81007fb442c0 ffff81017f6c3448 0000000300000b00 ffff81017f6c3448
Call Trace:
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffff80235889>] __mod_timer+0xaa/0xb8
[<ffffffff80466249>] schedule_timeout+0xa6/0xc9
[<ffffffff80235392>] process_timeout+0x0/0x5
[<ffffffff80290728>] do_select+0x414/0x482
[<ffffffff80290dd5>] __pollwait+0x0/0xe3
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff80293dad>] dput+0x21/0x10a
[<ffffffff80294aed>] __d_lookup+0xbd/0x107
[<ffffffff80299236>] mntput_no_expire+0x27/0x11e
[<ffffffff8028cfc6>] __link_path_walk+0xdbb/0xdd9
[<ffffffff80290941>] core_sys_select+0x1ab/0x253
[<ffffffff8028c1a9>] getname+0x13e/0x1a0
[<ffffffff8028de4c>] __user_walk_fd+0x41/0x4c
[<ffffffff80286ded>] vfs_stat_fd+0x1b/0x4a
[<ffffffff80290d25>] sys_select+0xb7/0x167
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

show_bench_st S 00000000ffffffff 0 8043 2879
ffff810023a77ea8 0000000000000082 00007fff24bbfe80 00007f631cabd000
ffffffff80600cc0 ffffffff80242f85 ffff810023a77ed8 ffff810070793d30
ffff81007fb43210 ffff810070793f68 0000000200000000 ffff810070793f68
Call Trace:
[<ffffffff80242f85>] getnstimeofday+0x38/0x84
[<ffffffff80241154>] hrtimer_start+0x100/0x122
[<ffffffff80241035>] hrtimer_try_to_cancel+0x61/0x6a
[<ffffffff80466799>] do_nanosleep+0x64/0xa6
[<ffffffff802411ca>] hrtimer_nanosleep+0x54/0xbf
[<ffffffff80240ab2>] hrtimer_wakeup+0x0/0x22
[<ffffffff80241281>] sys_nanosleep+0x4c/0x62
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

tee D 0000000000000000 0 8044 2879
ffff810046939b58 0000000000000086 ffff81001af65240 ffff810049075630
ffffffff80600cc0 ffffffff80324411 ffff810049075630 ffff8100707942c0
ffff81007e84de90 ffff8100707944f8 00000000ff8254a0 ffff8100707944f8
Call Trace:
[<ffffffff80324411>] __up_read+0x13/0x8a
[<ffffffff80268f59>] __do_fault+0x321/0x369
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff802959f7>] file_update_time+0xaf/0xf4
[<ffffffff8025cfb5>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff8025d799>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9ed8>] ext4_file_write+0x96/0x11a
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff8036a969>] tty_write+0x200/0x21b
[<ffffffff802846ac>] vfs_write+0xad/0x136
[<ffffffff802847f1>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check ? ffff810070796518 0 8413 8019
ffff81003453bef8 0000000000000046 ffff810070462908 0000000000000056
ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff810070796420
ffff81017f6c42c0 ffff810070796658 0000000200000000 ffff810070796658
Call Trace:
[<ffffffff80237be4>] do_notify_parent+0x18c/0x199
[<ffffffff8023096d>] do_exit+0x661/0x665
[<ffffffff802309d7>] do_group_exit+0x66/0x93
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check ? ffff810070796aa8 0 8415 8019
ffff8100492c9ef8 0000000000000046 ffff810070462908 0000000000000056
ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff8100707969b0
ffff8101ff6fbd30 ffff810070796be8 0000000000000000 ffff810070796be8
Call Trace:
[<ffffffff80237be4>] do_notify_parent+0x18c/0x199
[<ffffffff8023096d>] do_exit+0x661/0x665
[<ffffffff802309d7>] do_group_exit+0x66/0x93
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check ? ffff81016f416aa8 0 8460 7996
ffff81016f6c1ef8 0000000000000046 ffff810070461048 0000000000000056
ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff81016f4169b0
ffff81017f7826f0 ffff81016f416be8 0000000100000000 ffff81016f416be8
Call Trace:
[<ffffffff80237be4>] do_notify_parent+0x18c/0x199
[<ffffffff8023096d>] do_exit+0x661/0x665
[<ffffffff802309d7>] do_group_exit+0x66/0x93
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check ? ffff81016f417038 0 8463 7996
ffff81016f429ef8 0000000000000046 ffff810070461048 0000000000000056
ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff81016f416f40
ffff81027759d370 ffff81016f417178 0000000100000003 ffff81016f417178
Call Trace:
[<ffffffff80237be4>] do_notify_parent+0x18c/0x199
[<ffffffff8023096d>] do_exit+0x661/0x665
[<ffffffff802309d7>] do_group_exit+0x66/0x93
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check ? ffff810070790c18 0 8492 7973
ffff81002382def8 0000000000000046 ffff8101ff70fb88 0000000000000056
ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff810070790b20
ffff81027f148000 ffff810070790d58 0000000100000000 ffff810070790d58
Call Trace:
[<ffffffff80237be4>] do_notify_parent+0x18c/0x199
[<ffffffff8023096d>] do_exit+0x661/0x665
[<ffffffff802309d7>] do_group_exit+0x66/0x93
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check ? 00000000ffffffff 0 8515 7973
ffff81006e4a7ef8 0000000000000046 ffff8101ff70fb88 0000000000000056
ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff810070791640
ffff81007fb43210 ffff810070791878 0000000200000000 ffff810070791878
Call Trace:
[<ffffffff80237be4>] do_notify_parent+0x18c/0x199
[<ffffffff8023096d>] do_exit+0x661/0x665
[<ffffffff802309d7>] do_group_exit+0x66/0x93
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D 0000000000000000 0 9928 8021
ffff81027851fbe8 0000000000000086 ffff81017f72de90 0000000300000000
ffffffff80600cc0 0000000000000000 0000000000000000 ffff81027f2bcde0
ffff81017f6c1bd0 ffff81027f2bd018 0000000100000000 ffff81027f2bd018
Call Trace:
[<ffffffffa0068801>] :nf_conntrack:nf_conntrack_in+0x455/0x508
[<ffffffff804674e1>] _read_lock_bh+0x9/0x19
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff802959f7>] file_update_time+0xaf/0xf4
[<ffffffff8028a261>] pipe_write+0x4bc/0x4ce
[<ffffffff8028d093>] path_walk+0xaf/0xb9
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff80282ad0>] __dentry_open+0x141/0x226
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846ac>] vfs_write+0xad/0x136
[<ffffffff802847f1>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S ffff810052a38000 0 9929 2487
ffff810046a57bc8 0000000000000086 0000000000000096 ffffffff80224edc
ffffffff80600cc0 0000000000000001 0000000000000096 ffff8100469a37a0
ffff8101b97f8590 ffff8100469a39d8 0000000100000001 ffff8100469a39d8
Call Trace:
[<ffffffff80224edc>] try_to_wake_up+0x155/0x166
[<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
[<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
[<ffffffff80409098>] sock_def_readable+0x10/0x5d
[<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
[<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80405f35>] sock_aio_read+0x107/0x112
[<ffffffff80283f9a>] do_sync_read+0xce/0x113
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802848d7>] vfs_read+0xbd/0x132
[<ffffffff80284a08>] sys_read+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D 00000000ffffffff 0 9937 9929
ffff8101f8e99dd0 0000000000000086 ffffffff8040820c ffff8101f8e99d34
ffffffff80600cc0 0000000ed348224f ffff81007e8a0ec8 ffff8101b97f8590
ffff81007fb42160 ffff8101b97f87c8 0000000100000000 ffff8101b97f87c8
Call Trace:
[<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
[<ffffffff8028d093>] path_walk+0xaf/0xb9
[<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466484>] mutex_lock+0xa/0xb
[<ffffffff8028e2a4>] do_filp_open+0x10c/0x832
[<ffffffff8028280b>] get_unused_fd_flags+0x7f/0x10e
[<ffffffff802828e0>] do_sys_open+0x46/0xc3
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D 0000000000000000 0 9945 8021
ffff81027f295be8 0000000000000082 ffff81007e856290 ffffffff8042e6b5
ffffffff80600cc0 ffff81027f295b90 ffff8100370343f0 ffff81027f2bd900
ffff81027f2ba160 ffff81027f2bdb38 0000000037008438 ffff81027f2bdb38
Call Trace:
[<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
[<ffffffff802a3c0f>] __getblk+0x1e/0x28d
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff8022595b>] check_preempt_wakeup+0x7f/0xbc
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff802959f7>] file_update_time+0xaf/0xf4
[<ffffffff8028a261>] pipe_write+0x4bc/0x4ce
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802736e8>] free_pages_and_swap_cache+0x70/0x8b
[<ffffffff8026d509>] unmap_region+0x10f/0x125
[<ffffffff802846ac>] vfs_write+0xad/0x136
[<ffffffff802847f1>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S ffff81016f680000 0 9946 2487
ffff8100706ddbc8 0000000000000082 0000000000000096 ffffffff80224edc
ffffffff80600cc0 0000000000000001 0000000000000096 ffff8100469a5900
ffff81016f4137a0 ffff8100469a5b38 0000000000000001 ffff8100469a5b38
Call Trace:
[<ffffffff80224edc>] try_to_wake_up+0x155/0x166
[<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
[<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
[<ffffffff80409098>] sock_def_readable+0x10/0x5d
[<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
[<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80405f35>] sock_aio_read+0x107/0x112
[<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
[<ffffffff80283f9a>] do_sync_read+0xce/0x113
[<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802848d7>] vfs_read+0xbd/0x132
[<ffffffff80284a08>] sys_read+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D 00000000ffffffff 0 9949 9946
ffff810147d79dd0 0000000000000082 ffffffff8040820c ffff810147d79d34
ffffffff80600cc0 0000000ed34681af ffff81007e8a0ec8 ffff81016f4137a0
ffffffff8054d350 ffff81016f4139d8 0000000000000000 ffff81016f4139d8
Call Trace:
[<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
[<ffffffff8028d093>] path_walk+0xaf/0xb9
[<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466484>] mutex_lock+0xa/0xb
[<ffffffff8028e2a4>] do_filp_open+0x10c/0x832
[<ffffffff8028280b>] get_unused_fd_flags+0x7f/0x10e
[<ffffffff802828e0>] do_sys_open+0x46/0xc3
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S ffff8101ff703c80 0 9959 2487
ffff8101865c7bc8 0000000000000086 0000000000000096 ffffffff80224edc
ffffffff80600cc0 0000000000000001 0000000000000096 ffff8101b97ff4d0
ffff81016f4137a0 ffff8101b97ff708 0000000000000001 ffff8101b97ff708
Call Trace:
[<ffffffff80224edc>] try_to_wake_up+0x155/0x166
[<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
[<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
[<ffffffff80409098>] sock_def_readable+0x10/0x5d
[<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
[<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80405f35>] sock_aio_read+0x107/0x112
[<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
[<ffffffff80283f9a>] do_sync_read+0xce/0x113
[<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802848d7>] vfs_read+0xbd/0x132
[<ffffffff80284a08>] sys_read+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D ffff81007e85a400 0 9961 9959
ffff8101b9793c08 0000000000000082 000000000002842a ffff8101b9793d84
ffffffff80600cc0 0000000000000000 ffff81027a138ea0 ffff8101b97f90b0
ffff8101b97ff4d0 ffff8101b97f92e8 0000000000000001 ffff8101b97f92e8
Call Trace:
[<ffffffff802e6fbf>] ext4fs_dirhash+0x10d/0x1d2
[<ffffffff803019dd>] jbd2_journal_release_jbd_inode+0xaf/0x100
[<ffffffff8023e724>] wake_bit_function+0x0/0x23
[<ffffffff802d7d9b>] ext4_discard_reservation+0x1e/0x5a
[<ffffffff80296488>] clear_inode+0x69/0xc0
[<ffffffff80296afd>] generic_drop_inode+0x13a/0x157
[<ffffffff802db069>] ext4_new_inode+0xc46/0xcac
[<ffffffff802feabf>] start_this_handle+0x2c7/0x370
[<ffffffff80299236>] mntput_no_expire+0x27/0x11e
[<ffffffff802e251d>] ext4_create+0x88/0x105
[<ffffffff802e27e7>] ext4_lookup+0x97/0xc1
[<ffffffff8028b7d9>] vfs_create+0x7d/0xed
[<ffffffff8028e385>] do_filp_open+0x1ed/0x832
[<ffffffff802828e0>] do_sys_open+0x46/0xc3
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D 0000000000000000 0 9969 8021
ffff8102785bfbe8 0000000000000082 ffff81017f72dcd0 ffffffff8042e6b5
ffffffff80600cc0 0000000000000000 0000000000000000 ffff81027f2bbd30
ffff81017f6c1bd0 ffff81027f2bbf68 000000019740516a ffff81027f2bbf68
Call Trace:
[<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff8022595b>] check_preempt_wakeup+0x7f/0xbc
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff802959f7>] file_update_time+0xaf/0xf4
[<ffffffff8028a261>] pipe_write+0x4bc/0x4ce
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802736e8>] free_pages_and_swap_cache+0x70/0x8b
[<ffffffff8026d509>] unmap_region+0x10f/0x125
[<ffffffff802846ac>] vfs_write+0xad/0x136
[<ffffffff802847f1>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S 00000000ffffffff 0 9970 2487
ffff810224881bc8 0000000000000082 00000000008db1fc ffffffff80241310
ffffffff80600cc0 0000000000000001 ffffffff80266f65 ffff81027f2bb7a0
ffff81007fb442c0 ffff81027f2bb9d8 000000037f2bc888 ffff81027f2bb9d8
Call Trace:
[<ffffffff80241310>] ktime_get_ts+0x21/0x4a
[<ffffffff80266f65>] zone_statistics+0x3c/0x90
[<ffffffff80266f65>] zone_statistics+0x3c/0x90
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
[<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff8021eee3>] do_page_fault+0x60c/0x825
[<ffffffff80405f35>] sock_aio_read+0x107/0x112
[<ffffffff80283f9a>] do_sync_read+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff80467699>] error_exit+0x0/0x51
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802848d7>] vfs_read+0xbd/0x132
[<ffffffff80284a08>] sys_read+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D ffff81027a138f60 0 9973 9970
ffff81022cadfdd0 0000000000000086 0000000000000000 ffff810200000000
ffffffff80600cc0 0000000f13fc4c86 ffff81007e8a0ec8 ffff81027f2bf4d0
ffff81027f2bb7a0 ffff81027f2bf708 0000000300000000 ffff81027f2bf708
Call Trace:
[<ffffffff8028d093>] path_walk+0xaf/0xb9
[<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466484>] mutex_lock+0xa/0xb
[<ffffffff8028e2a4>] do_filp_open+0x10c/0x832
[<ffffffff8028280b>] get_unused_fd_flags+0x7f/0x10e
[<ffffffff802828e0>] do_sys_open+0x46/0xc3
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D ffff810027160690 0 9979 8021
ffff81022e071d70 0000000000000086 ffff81022e071d38 ffff81007e8a0e00
ffffffff80600cc0 0000000000000000 ffffffff80293dad ffff81027f2bc850
ffff81016f416420 ffff81027f2bca88 000000018028c8f9 ffff81027f2bca88
Call Trace:
[<ffffffff80293dad>] dput+0x21/0x10a
[<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
[<ffffffff80299236>] mntput_no_expire+0x27/0x11e
[<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466484>] mutex_lock+0xa/0xb
[<ffffffff8028ac27>] lock_rename+0x35/0xc5
[<ffffffff8028d51e>] do_rename+0x98/0x1b2
[<ffffffff802921b3>] __posix_lock_file+0x43d/0x44e
[<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d684>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S ffff81016f683c80 0 9981 2487
ffff81016f4f9bc8 0000000000000082 0000000000000096 ffffffff80224edc
ffffffff80600cc0 0000000000000001 0000000000000096 ffff81016f416420
ffff81017f6c4de0 ffff81016f416658 0000000100000001 ffff81016f416658
Call Trace:
[<ffffffff80224edc>] try_to_wake_up+0x155/0x166
[<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
[<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
[<ffffffff80409098>] sock_def_readable+0x10/0x5d
[<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
[<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80405f35>] sock_aio_read+0x107/0x112
[<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
[<ffffffff80283f9a>] do_sync_read+0xce/0x113
[<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802848d7>] vfs_read+0xbd/0x132
[<ffffffff80284a08>] sys_read+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S 00007f17790fc000 0 9984 9981
ffff81016f555b68 0000000000000086 ffff81017f72de90 ffffffff802414e9
ffffffff80600cc0 000000000f4c63de 0000000000000046 ffff81016f417a60
ffff81027f2bc850 ffff81016f417c98 000000017f72de90 ffff81016f417c98
Call Trace:
[<ffffffff802414e9>] ktime_get_real+0xc/0x43
[<ffffffff80231c26>] local_bh_enable+0x6a/0x7c
[<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
[<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff802a3c0f>] __getblk+0x1e/0x28d
[<ffffffff804674fa>] _spin_lock_bh+0x9/0x1f
[<ffffffff80408cba>] release_sock+0x13/0x95
[<ffffffff80409265>] sk_wait_data+0x83/0xc6
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80434547>] tcp_recvmsg+0x3b3/0x7d2
[<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
[<ffffffff80406a49>] sock_recvmsg+0xd5/0xed
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80242f85>] getnstimeofday+0x38/0x84
[<ffffffff80406509>] sockfd_lookup_light+0x1a/0x52
[<ffffffff80407a7d>] sys_recvfrom+0xc3/0x122
[<ffffffff80241154>] hrtimer_start+0x100/0x122
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D ffff810027160690 0 10194 7998
ffff810186669d70 0000000000000086 ffff810186669d38 ffff81007e8a0e00
ffffffff80600cc0 0000000000000000 ffffffff80293dad ffff8101b97fb210
ffff81027f149bd0 ffff8101b97fb448 000000028028c8f9 ffff8101b97fb448
Call Trace:
[<ffffffff80293dad>] dput+0x21/0x10a
[<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
[<ffffffff80299236>] mntput_no_expire+0x27/0x11e
[<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466484>] mutex_lock+0xa/0xb
[<ffffffff8028ac27>] lock_rename+0x35/0xc5
[<ffffffff8028d51e>] do_rename+0x98/0x1b2
[<ffffffff802921b3>] __posix_lock_file+0x43d/0x44e
[<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d684>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S ffff8101fff33700 0 10195 2487
ffff81016f625bc8 0000000000000086 0000000000000096 ffffffff80224edc
ffffffff80600cc0 0000000000000001 0000000000000096 ffff81016f414850
ffff8101b97fef40 ffff81016f414a88 0000000200000001 ffff81016f414a88
Call Trace:
[<ffffffff80224edc>] try_to_wake_up+0x155/0x166
[<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
[<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
[<ffffffff80409098>] sock_def_readable+0x10/0x5d
[<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
[<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80405f35>] sock_aio_read+0x107/0x112
[<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
[<ffffffff80283f9a>] do_sync_read+0xce/0x113
[<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802848d7>] vfs_read+0xbd/0x132
[<ffffffff80284a08>] sys_read+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S 00007f17790fc000 0 10197 10195
ffff8101ff7f1b68 0000000000000086 ffff8101867b4450 ffffffff802414e9
ffffffff80600cc0 000000000f5245fe 0000000000000046 ffff8101b97fd900
ffff8101b97fb210 ffff8101b97fdb38 00000002867b4450 ffff8101b97fdb38
Call Trace:
[<ffffffff802414e9>] ktime_get_real+0xc/0x43
[<ffffffff80231c26>] local_bh_enable+0x6a/0x7c
[<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
[<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff802a3c0f>] __getblk+0x1e/0x28d
[<ffffffff804674fa>] _spin_lock_bh+0x9/0x1f
[<ffffffff80408cba>] release_sock+0x13/0x95
[<ffffffff80409265>] sk_wait_data+0x83/0xc6
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80434547>] tcp_recvmsg+0x3b3/0x7d2
[<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
[<ffffffff80406a49>] sock_recvmsg+0xd5/0xed
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80242f85>] getnstimeofday+0x38/0x84
[<ffffffff80406509>] sockfd_lookup_light+0x1a/0x52
[<ffffffff80407a7d>] sys_recvfrom+0xc3/0x122
[<ffffffff80241154>] hrtimer_start+0x100/0x122
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp S 0000000000000000 0 10198 7975
ffff81007054f9e8 0000000000000086 ffff8101f8f1e310 9c9ba13000000834
ffffffff80600cc0 000000038023e617 ffff8101db55d930 ffff8100469a5370
ffff8100469a10b0 ffff8100469a55a8 00000002867b5cd0 ffff8100469a55a8
Call Trace:
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffff80235889>] __mod_timer+0xaa/0xb8
[<ffffffff80466249>] schedule_timeout+0xa6/0xc9
[<ffffffff80235392>] process_timeout+0x0/0x5
[<ffffffff80290728>] do_select+0x414/0x482
[<ffffffff80290dd5>] __pollwait+0x0/0xe3
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffffa0068801>] :nf_conntrack:nf_conntrack_in+0x455/0x508
[<ffffffff804674e1>] _read_lock_bh+0x9/0x19
[<ffffffffa00634f5>] :ip_tables:ipt_do_table+0x2e9/0x30c
[<ffffffffa006bd34>] :nf_conntrack:nf_ct_deliver_cached_events+0x4d/0x7d
[<ffffffffa007937e>] :nf_conntrack_ipv4:ipv4_confirm+0xcc/0xd7
[<ffffffff80425272>] nf_iterate+0x41/0x7d
[<ffffffff80242f85>] getnstimeofday+0x38/0x84
[<ffffffff8042ea4b>] ip_finish_output+0x0/0x239
[<ffffffff80290941>] core_sys_select+0x1ab/0x253
[<ffffffff80231c26>] local_bh_enable+0x6a/0x7c
[<ffffffff802985f0>] mnt_want_write+0x25/0x77
[<ffffffff80295a4e>] touch_atime+0x12/0xfa
[<ffffffff80408dd6>] lock_sock_nested+0x9a/0xa5
[<ffffffff804674fa>] _spin_lock_bh+0x9/0x1f
[<ffffffff80408cba>] release_sock+0x13/0x95
[<ffffffff8023e617>] bit_waitqueue+0x10/0xa0
[<ffffffff8023e6e5>] wake_up_bit+0x11/0x22
[<ffffffff80290d25>] sys_select+0xb7/0x167
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S ffff810052a3b700 0 10199 2487
ffff810147c4fbc8 0000000000000082 0000000000000096 ffffffff80224edc
ffffffff80600cc0 0000000000000001 ffff810181e11cc0 ffff81016f412160
ffff8100469a10b0 ffff81016f412398 0000000200000001 ffff81016f412398
Call Trace:
[<ffffffff80224edc>] try_to_wake_up+0x155/0x166
[<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
[<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
[<ffffffff80409098>] sock_def_readable+0x10/0x5d
[<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
[<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80405f35>] sock_aio_read+0x107/0x112
[<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
[<ffffffff80283f9a>] do_sync_read+0xce/0x113
[<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802848d7>] vfs_read+0xbd/0x132
[<ffffffff80284a08>] sys_read+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D 00000000ffffffff 0 10201 10199
ffff810046851b58 0000000000000082 ffff8101fff7b200 0000000000000292
ffffffff80600cc0 ffffffff80411a78 ffff81018670386c ffff8100469a10b0
ffff81007fb43210 ffff8100469a12e8 0000000200000246 ffff8100469a12e8
Call Trace:
[<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80409110>] sk_reset_timer+0xf/0x19
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff802959f7>] file_update_time+0xaf/0xf4
[<ffffffff8025cfb5>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff8025d799>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9ed8>] ext4_file_write+0x96/0x11a
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846ac>] vfs_write+0xad/0x136
[<ffffffff802847f1>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D ffff810027160690 0 10202 7998
ffff8101b9727d70 0000000000000082 ffff8101b9727d38 ffff81007e8a0e00
ffffffff80600cc0 0000000000000000 ffffffff80293dad ffff8101b97fc2c0
ffff81016f4126f0 ffff8101b97fc4f8 000000038028c8f9 ffff8101b97fc4f8
Call Trace:
[<ffffffff80293dad>] dput+0x21/0x10a
[<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
[<ffffffff80299236>] mntput_no_expire+0x27/0x11e
[<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466484>] mutex_lock+0xa/0xb
[<ffffffff8028ac27>] lock_rename+0x35/0xc5
[<ffffffff8028d51e>] do_rename+0x98/0x1b2
[<ffffffff802921b3>] __posix_lock_file+0x43d/0x44e
[<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d684>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S ffff810052a3a3c0 0 10203 2487
ffff810147c47bc8 0000000000000082 0000000000000096 ffffffff80224edc
ffffffff80600cc0 0000000000000001 0000000000000096 ffff81016f4126f0
ffff81027f2b8b20 ffff81016f412928 0000000300000001 ffff81016f412928
Call Trace:
[<ffffffff80224edc>] try_to_wake_up+0x155/0x166
[<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
[<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
[<ffffffff80409098>] sock_def_readable+0x10/0x5d
[<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
[<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80405f35>] sock_aio_read+0x107/0x112
[<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
[<ffffffff80283f9a>] do_sync_read+0xce/0x113
[<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802848d7>] vfs_read+0xbd/0x132
[<ffffffff80284a08>] sys_read+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D 00000000ffffffff 0 10205 7998
ffff8101f8f71d70 0000000000000082 ffff8101f8f71d38 ffff81007e8a0e00
ffffffff80600cc0 0000000000000000 ffffffff80293dad ffff8101b97f8000
ffffffff8054d350 ffff8101b97f8238 000000008028c8f9 ffff8101b97f8238
Call Trace:
[<ffffffff80293dad>] dput+0x21/0x10a
[<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
[<ffffffff80299236>] mntput_no_expire+0x27/0x11e
[<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466484>] mutex_lock+0xa/0xb
[<ffffffff8028ac27>] lock_rename+0x35/0xc5
[<ffffffff8028d51e>] do_rename+0x98/0x1b2
[<ffffffff802921b3>] __posix_lock_file+0x43d/0x44e
[<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d684>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S ffff81016f681b80 0 10206 2487
ffff81017f769bc8 0000000000000082 0000000000000096 ffffffff80224edc
ffffffff80600cc0 0000000000000001 ffff810001006cc0 ffff81016f410b20
ffff8101b97fd370 ffff81016f410d58 0000000300000001 ffff81016f410d58
Call Trace:
[<ffffffff80224edc>] try_to_wake_up+0x155/0x166
[<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
[<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
[<ffffffff80409098>] sock_def_readable+0x10/0x5d
[<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
[<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80405f35>] sock_aio_read+0x107/0x112
[<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
[<ffffffff80283f9a>] do_sync_read+0xce/0x113
[<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802848d7>] vfs_read+0xbd/0x132
[<ffffffff80284a08>] sys_read+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S 00007f17790fc000 0 10208 10203
ffff810034705b68 0000000000000086 ffff810246fa5410 ffffffff802414e9
ffffffff80600cc0 000000000f5342a7 0000000000000046 ffff8100469a5e90
ffff8101b97fc2c0 ffff8100469a60c8 0000000346fa5410 ffff8100469a60c8
Call Trace:
[<ffffffff802414e9>] ktime_get_real+0xc/0x43
[<ffffffff80231c26>] local_bh_enable+0x6a/0x7c
[<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
[<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff802a3c0f>] __getblk+0x1e/0x28d
[<ffffffff804674fa>] _spin_lock_bh+0x9/0x1f
[<ffffffff80408cba>] release_sock+0x13/0x95
[<ffffffff80409265>] sk_wait_data+0x83/0xc6
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80434547>] tcp_recvmsg+0x3b3/0x7d2
[<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
[<ffffffff80406a49>] sock_recvmsg+0xd5/0xed
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80242f85>] getnstimeofday+0x38/0x84
[<ffffffff80406509>] sockfd_lookup_light+0x1a/0x52
[<ffffffff80407a7d>] sys_recvfrom+0xc3/0x122
[<ffffffff80241154>] hrtimer_start+0x100/0x122
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S 00007f17790fc000 0 10209 10206
ffff810147d37b68 0000000000000086 ffff81007e857b10 ffffffff802414e9
ffffffff80600cc0 000000000f1a1bb0 0000000000000046 ffff81016f413210
ffff8101b97f8000 ffff81016f413448 000000007e857b10 ffff81016f413448
Call Trace:
[<ffffffff802414e9>] ktime_get_real+0xc/0x43
[<ffffffff80231c26>] local_bh_enable+0x6a/0x7c
[<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
[<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff802a3c0f>] __getblk+0x1e/0x28d
[<ffffffff804674fa>] _spin_lock_bh+0x9/0x1f
[<ffffffff80408cba>] release_sock+0x13/0x95
[<ffffffff80409265>] sk_wait_data+0x83/0xc6
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80434547>] tcp_recvmsg+0x3b3/0x7d2
[<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
[<ffffffff80406a49>] sock_recvmsg+0xd5/0xed
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80242f85>] getnstimeofday+0x38/0x84
[<ffffffff80406509>] sockfd_lookup_light+0x1a/0x52
[<ffffffff80407a7d>] sys_recvfrom+0xc3/0x122
[<ffffffff80241154>] hrtimer_start+0x100/0x122
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D 00000000ffffffff 0 10210 7975
ffff81001ac6fbe8 0000000000000086 ffff8101867b4d10 000000038042e6b5
ffffffff80600cc0 0000000000000246 ffff81007fbbb000 ffff8100469a2160
ffff81007fb43210 ffff8100469a2398 0000000200000000 ffff8100469a2398
Call Trace:
[<ffffffffa0068801>] :nf_conntrack:nf_conntrack_in+0x455/0x508
[<ffffffff804674e1>] _read_lock_bh+0x9/0x19
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff802959f7>] file_update_time+0xaf/0xf4
[<ffffffff8028a261>] pipe_write+0x4bc/0x4ce
[<ffffffff8028d093>] path_walk+0xaf/0xb9
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff80282ad0>] __dentry_open+0x141/0x226
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846ac>] vfs_write+0xad/0x136
[<ffffffff802847f1>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S ffff8101fff32100 0 10211 2487
ffff8101b960bbc8 0000000000000086 0000000000000096 ffffffff80224edc
ffffffff80600cc0 0000000000000001 0000000000000096 ffff8101b97fa160
ffff8101b97fb210 ffff8101b97fa398 0000000200000001 ffff8101b97fa398
Call Trace:
[<ffffffff80224edc>] try_to_wake_up+0x155/0x166
[<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
[<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
[<ffffffff80409098>] sock_def_readable+0x10/0x5d
[<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
[<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80405f35>] sock_aio_read+0x107/0x112
[<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
[<ffffffff80283f9a>] do_sync_read+0xce/0x113
[<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802848d7>] vfs_read+0xbd/0x132
[<ffffffff80284a08>] sys_read+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D ffff8101dbe48f60 0 10213 10211
ffff8101f8ff3d70 0000000000000082 ffff8101f8ff3d38 ffff81007e8a0e00
ffffffff80600cc0 0000000000000000 ffffffff80293dad ffff8101b97fef40
ffff8100469a10b0 ffff8101b97ff178 000000028028c8f9 ffff8101b97ff178
Call Trace:
[<ffffffff80293dad>] dput+0x21/0x10a
[<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
[<ffffffff80299236>] mntput_no_expire+0x27/0x11e
[<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466484>] mutex_lock+0xa/0xb
[<ffffffff8028acaf>] lock_rename+0xbd/0xc5
[<ffffffff8028d51e>] do_rename+0x98/0x1b2
[<ffffffff80261447>] __alloc_pages_internal+0xd2/0x40c
[<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d684>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D 0000000000000000 0 10214 7975
ffff810046addbe8 0000000000000082 ffff81017f72cb50 ffffffff8042e6b5
ffffffff80600cc0 0000000000000246 ffff81007fbbb000 ffff8100469a3d30
ffff81027f14de90 ffff8100469a3f68 000000017f6a2a6c ffff8100469a3f68
Call Trace:
[<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
[<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff8022595b>] check_preempt_wakeup+0x7f/0xbc
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff802959f7>] file_update_time+0xaf/0xf4
[<ffffffff8028a261>] pipe_write+0x4bc/0x4ce
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802736e8>] free_pages_and_swap_cache+0x70/0x8b
[<ffffffff8026d509>] unmap_region+0x10f/0x125
[<ffffffff802846ac>] vfs_write+0xad/0x136
[<ffffffff802847f1>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S ffff810220725e40 0 10215 2487
ffff8101866ffbc8 0000000000000086 0000000000000096 ffffffff80224edc
ffffffff80600cc0 0000000000000001 0000000000000096 ffff8101b97fe420
ffff81027f2b90b0 ffff8101b97fe658 0000000100000001 ffff8101b97fe658
Call Trace:
[<ffffffff80224edc>] try_to_wake_up+0x155/0x166
[<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
[<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
[<ffffffff80409098>] sock_def_readable+0x10/0x5d
[<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
[<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80405f35>] sock_aio_read+0x107/0x112
[<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
[<ffffffff80283f9a>] do_sync_read+0xce/0x113
[<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802848d7>] vfs_read+0xbd/0x132
[<ffffffff80284a08>] sys_read+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D 0000000000000000 0 10217 10215
ffff810220501bf8 0000000000000086 0000000000000001 0000000000000000
ffffffff80600cc0 0000000000000000 ffff810220501ba8 ffff81027f2b90b0
ffff81017f781bd0 ffff81027f2b92e8 0000000100000000 ffff81027f2b92e8
Call Trace:
[<ffffffff80224a67>] __wake_up+0x38/0x4f
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802e2b2c>] ext4_rename+0x4d/0x64a
[<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
[<ffffffff8028bc30>] vfs_rename+0x1fd/0x376
[<ffffffff8028d5d7>] do_rename+0x151/0x1b2
[<ffffffff80261447>] __alloc_pages_internal+0xd2/0x40c
[<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d684>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D 00000000ffffffff 0 10218 7998
ffff8101866c1be8 0000000000000082 ffff810246fa4610 ffffffff8042e6b5
ffffffff80600cc0 ffff8101866c1b90 ffff8101db68f7e0 ffff8101b97fc850
ffff81007fb442c0 ffff8101b97fca88 000000034f9ed590 ffff8101b97fca88
Call Trace:
[<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
[<ffffffff802a3c0f>] __getblk+0x1e/0x28d
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff8025ba69>] find_lock_page+0x29/0x87
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff802959f7>] file_update_time+0xaf/0xf4
[<ffffffff8028a261>] pipe_write+0x4bc/0x4ce
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802736e8>] free_pages_and_swap_cache+0x70/0x8b
[<ffffffff8026d509>] unmap_region+0x10f/0x125
[<ffffffff802846ac>] vfs_write+0xad/0x136
[<ffffffff802847f1>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S ffff81016f682100 0 10219 2487
ffff8101f8ea7bc8 0000000000000082 0000000000000096 ffffffff80224edc
ffffffff80600cc0 0000000000000001 ffff810201e11cc0 ffff8101b97fde90
ffff81027f2b90b0 ffff8101b97fe0c8 0000000100000001 ffff8101b97fe0c8
Call Trace:
[<ffffffff80224edc>] try_to_wake_up+0x155/0x166
[<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
[<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
[<ffffffff80409098>] sock_def_readable+0x10/0x5d
[<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
[<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80405f35>] sock_aio_read+0x107/0x112
[<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
[<ffffffff80283f9a>] do_sync_read+0xce/0x113
[<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802848d7>] vfs_read+0xbd/0x132
[<ffffffff80284a08>] sys_read+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp S 0000000000000000 0 10220 7975
ffff81016f4859e8 0000000000000082 ffff81023b55c8b0 9c34adbf00000834
ffffffff80600cc0 000000038023e617 ffff8101db55d930 ffff81016f415900
ffff81027f2b8b20 ffff81016f415b38 0000000346fa5250 ffff81016f415b38
Call Trace:
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffff80235889>] __mod_timer+0xaa/0xb8
[<ffffffff80466249>] schedule_timeout+0xa6/0xc9
[<ffffffff80235392>] process_timeout+0x0/0x5
[<ffffffff80290728>] do_select+0x414/0x482
[<ffffffff80290dd5>] __pollwait+0x0/0xe3
[<ffffffff80224eed>] default_wake_function+0x0/0xe
[<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
[<ffffffff802356ef>] lock_timer_base+0x26/0x4b
[<ffffffffa0068801>] :nf_conntrack:nf_conntrack_in+0x455/0x508
[<ffffffff804674e1>] _read_lock_bh+0x9/0x19
[<ffffffffa00634f5>] :ip_tables:ipt_do_table+0x2e9/0x30c
[<ffffffffa006bd34>] :nf_conntrack:nf_ct_deliver_cached_events+0x4d/0x7d
[<ffffffffa007937e>] :nf_conntrack_ipv4:ipv4_confirm+0xcc/0xd7
[<ffffffff80425272>] nf_iterate+0x41/0x7d
[<ffffffff80242f85>] getnstimeofday+0x38/0x84
[<ffffffff8042ea4b>] ip_finish_output+0x0/0x239
[<ffffffff80290941>] core_sys_select+0x1ab/0x253
[<ffffffff80231c26>] local_bh_enable+0x6a/0x7c
[<ffffffff802985f0>] mnt_want_write+0x25/0x77
[<ffffffff80295a4e>] touch_atime+0x12/0xfa
[<ffffffff80408dd6>] lock_sock_nested+0x9a/0xa5
[<ffffffff804674fa>] _spin_lock_bh+0x9/0x1f
[<ffffffff80408cba>] release_sock+0x13/0x95
[<ffffffff8023e617>] bit_waitqueue+0x10/0xa0
[<ffffffff8023e6e5>] wake_up_bit+0x11/0x22
[<ffffffff80290d25>] sys_select+0xb7/0x167
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd S ffff810220726680 0 10221 2487
ffff810195039bc8 0000000000000082 0000000000000096 ffffffff80224edc
ffffffff80600cc0 0000000000000001 0000000000000096 ffff8101b97fd370
ffff81016f410590 ffff8101b97fd5a8 0000000300000001 ffff8101b97fd5a8
Call Trace:
[<ffffffff80224edc>] try_to_wake_up+0x155/0x166
[<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
[<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
[<ffffffff80409098>] sock_def_readable+0x10/0x5d
[<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
[<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
[<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
[<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
[<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff80405f35>] sock_aio_read+0x107/0x112
[<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
[<ffffffff80283f9a>] do_sync_read+0xce/0x113
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802848d7>] vfs_read+0xbd/0x132
[<ffffffff80284a08>] sys_read+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D 0000000000000000 0 10224 10221
ffff810246ef1a68 0000000000000086 000000003b55c7e0 ffffffff80233c4d
ffffffff80600cc0 ffffffff802d7ba2 0000000000000003 ffff81027f2b8b20
ffff8101b97fd370 ffff81027f2b8d58 0000000334b600c0 ffff81027f2b8d58
Call Trace:
[<ffffffff80233c4d>] __capable+0x9/0x1f
[<ffffffff802d7ba2>] ext4_has_free_blocks+0x2b/0xad
[<ffffffff802a3869>] __find_get_block_slow+0xcf/0xdb
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff802a4264>] generic_write_end+0x5d/0x68
[<ffffffff8025c6bc>] generic_file_buffered_write+0x1b6/0x658
[<ffffffff80409110>] sk_reset_timer+0xf/0x19
[<ffffffff8043cd07>] tcp_rcv_established+0x908/0xbc5
[<ffffffff8023136c>] current_fs_time+0x1e/0x24
[<ffffffff8025b358>] remove_suid+0x18/0x4b
[<ffffffff8025d09a>] __generic_file_aio_write_nolock+0x34a/0x37e
[<ffffffff8025d799>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9ed8>] ext4_file_write+0x96/0x11a
[<ffffffff80283e87>] do_sync_write+0xce/0x113
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846ac>] vfs_write+0xad/0x136
[<ffffffff802847f1>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D 00000000ffffffff 0 10225 10219
ffff810147d55ce8 0000000000000086 ffff8101a2dcadd8 0000000000000006
ffffffff80600cc0 ffff8102796cabd0 ffff810274479000 ffff81016f410590
ffff81007fb442c0 ffff81016f4107c8 000000030000000f ffff81016f4107c8
Call Trace:
[<ffffffff8028ad6a>] do_lookup+0x63/0x1a1
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802e24e1>] ext4_create+0x4c/0x105
[<ffffffff802e27e7>] ext4_lookup+0x97/0xc1
[<ffffffff8028b7d9>] vfs_create+0x7d/0xed
[<ffffffff8028e385>] do_filp_open+0x1ed/0x832
[<ffffffff802828e0>] do_sys_open+0x46/0xc3
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D 00000000ffffffff 0 10226 8021
ffff81027f0afbb8 0000000000000082 0000003df881a530 ffff81003726f1c0
ffffffff80600cc0 ffffffff80260923 0000000000000002 ffff81027f2bfa60
ffff81007fb42160 ffff81027f2bfc98 000000010000001e ffff81027f2bfc98
Call Trace:
[<ffffffff80260923>] __rmqueue_smallest+0x9d/0x124
[<ffffffff802609c7>] __rmqueue+0x1d/0x1fb
[<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
[<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
[<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
[<ffffffff802959f7>] file_update_time+0xaf/0xf4
[<ffffffff802693fc>] do_wp_page+0x45b/0x4ba
[<ffffffff8026aac1>] handle_mm_fault+0x62d/0x6a3
[<ffffffff8021ed0c>] do_page_fault+0x435/0x825
[<ffffffff80299236>] mntput_no_expire+0x27/0x11e
[<ffffffff8028f8b4>] filldir+0x0/0xb7
[<ffffffff802921b3>] __posix_lock_file+0x43d/0x44e
[<ffffffff8029260e>] fcntl_setlk+0x285/0x296
[<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff80467699>] error_exit+0x0/0x51

2008-06-16 17:54:10

by Jan Kara

[permalink] [raw]

Subject: Re: Performance of ext4

> On Thu, 12 Jun 2008, Holger Kiehl wrote:
>
> >On Thu, 12 Jun 2008, Aneesh Kumar K.V wrote:
> >
> >>On Thu, Jun 12, 2008 at 02:07:30PM +0000, Holger Kiehl wrote:
> >>>This time there is no OOPS and system is still up running without any
> >>>problem (except any process wanting to write something to this filesystem
> >>>gets stuck forever).
> >>>
> >>>What can I do to help find the problem? The system is still up with all
> >>>those
> >>>process hanging in D-state.
> >>>
> >>
> >>if you can login to the system get the dmesg output after
> >>
> >>echo t > /proc/sysrq-trigger
> >>
> >Unfortunately I have not set CONFIG_MAGIC_SYSRQ. Tomorrow I will try to
> >reproduce this with a kernel that has CONFIG_MAGIC_SYSRQ set.
> >
> After recompiling, rebooting and run afdbench first I got an OOPS and the
> system hanged up solid. The only thing I was able to catch is this:
>
> RIP [<ffffffff803019f9>] jbd2_journal_release_jbd_inode+0xcb/0x100
> RSP <ffff8101fe259c18>
>
> This was copied by hand. And this I cut 'cut and past' from my terminal:
>
> kernel: Code: c3 e8 31 ce f3 ff 41 fe 04 24 e8 fe 3f 16 00 4c 89 fe 48 89
> df e8 5f cd f3 ff eb 82 48 83 7d 00 00 74 27 48 8b 55 10 48 8b 45 18 <48>
> 89 42 08 48 89 10 48 c7 45 18 00 02 20 00 48 c7 45 10 00 01
Aneesh found cause of this oops I think... Aneesh, would you send the
fix to Holger? Thanks.

> So I rebooted run bonnie and then afdbench and I get the same problem as
> yesterday. All process trying to write something to this filesystem hang
> in D-state. But now I was able to do the echo t > /proc/sysrq-trigger.
> Here the dmesg output:
Sadly, the output seems to be truncated (mainly, the kjournald process
is missing, which is probably the root cause of the hang). So could you
have a look whether /var/log/messages doesn't contain the dump of all
processes? And if no, then could you do "echo w >/proc/sysrq-trigger"
please? That will dump only blocked processes which should fit in the log
buffer. You can also increase log buffer size in kernel config but that
shouldn't be needed. Thanks.

Honza
--
Jan Kara <[email protected]>
SuSE CR Labs

2008-06-16 18:13:53

[permalink] [raw]

Subject: Re: Performance of ext4

On Mon, Jun 16, 2008 at 07:54:08PM +0200, Jan Kara wrote:
> > On Thu, 12 Jun 2008, Holger Kiehl wrote:
> >
> > >On Thu, 12 Jun 2008, Aneesh Kumar K.V wrote:
> > >
> > >>On Thu, Jun 12, 2008 at 02:07:30PM +0000, Holger Kiehl wrote:
> > >>>This time there is no OOPS and system is still up running without any
> > >>>problem (except any process wanting to write something to this filesystem
> > >>>gets stuck forever).
> > >>>
> > >>>What can I do to help find the problem? The system is still up with all
> > >>>those
> > >>>process hanging in D-state.
> > >>>
> > >>
> > >>if you can login to the system get the dmesg output after
> > >>
> > >>echo t > /proc/sysrq-trigger
> > >>
> > >Unfortunately I have not set CONFIG_MAGIC_SYSRQ. Tomorrow I will try to
> > >reproduce this with a kernel that has CONFIG_MAGIC_SYSRQ set.
> > >
> > After recompiling, rebooting and run afdbench first I got an OOPS and the
> > system hanged up solid. The only thing I was able to catch is this:
> >
> > RIP [<ffffffff803019f9>] jbd2_journal_release_jbd_inode+0xcb/0x100
> > RSP <ffff8101fe259c18>
> >
> > This was copied by hand. And this I cut 'cut and past' from my terminal:
> >
> > kernel: Code: c3 e8 31 ce f3 ff 41 fe 04 24 e8 fe 3f 16 00 4c 89 fe 48 89
> > df e8 5f cd f3 ff eb 82 48 83 7d 00 00 74 27 48 8b 55 10 48 8b 45 18 <48>
> > 89 42 08 48 89 10 48 c7 45 18 00 02 20 00 48 c7 45 10 00 01
> Aneesh found cause of this oops I think... Aneesh, would you send the
> fix to Holger? Thanks.
>

That fix was mainly done with the help of Holger. Many thanks to him for
doing multiple test during weekend with different combination. He had
already confirmed that the fix worked for me.

There is another issue that he is hitting when running the test with
mke2fs -m 0. But I think that is not related to lock inversion.

-aneesh

2008-06-17 11:42:53

[permalink] [raw]

Subject: Re: Performance of ext4

On Mon, 16 Jun 2008, Aneesh Kumar K.V wrote:

> On Mon, Jun 16, 2008 at 07:54:08PM +0200, Jan Kara wrote:
>>> On Thu, 12 Jun 2008, Holger Kiehl wrote:
>>>
>>>> On Thu, 12 Jun 2008, Aneesh Kumar K.V wrote:
>>>>
>>>>> On Thu, Jun 12, 2008 at 02:07:30PM +0000, Holger Kiehl wrote:
>>>>>> This time there is no OOPS and system is still up running without any
>>>>>> problem (except any process wanting to write something to this filesystem
>>>>>> gets stuck forever).
>>>>>>
>>>>>> What can I do to help find the problem? The system is still up with all
>>>>>> those
>>>>>> process hanging in D-state.
>>>>>>
>>>>>
>>>>> if you can login to the system get the dmesg output after
>>>>>
>>>>> echo t > /proc/sysrq-trigger
>>>>>
>>>> Unfortunately I have not set CONFIG_MAGIC_SYSRQ. Tomorrow I will try to
>>>> reproduce this with a kernel that has CONFIG_MAGIC_SYSRQ set.
>>>>
>>> After recompiling, rebooting and run afdbench first I got an OOPS and the
>>> system hanged up solid. The only thing I was able to catch is this:
>>>
>>> RIP [<ffffffff803019f9>] jbd2_journal_release_jbd_inode+0xcb/0x100
>>> RSP <ffff8101fe259c18>
>>>
>>> This was copied by hand. And this I cut 'cut and past' from my terminal:
>>>
>>> kernel: Code: c3 e8 31 ce f3 ff 41 fe 04 24 e8 fe 3f 16 00 4c 89 fe 48 89
>>> df e8 5f cd f3 ff eb 82 48 83 7d 00 00 74 27 48 8b 55 10 48 8b 45 18 <48>
>>> 89 42 08 48 89 10 48 c7 45 18 00 02 20 00 48 c7 45 10 00 01
>> Aneesh found cause of this oops I think... Aneesh, would you send the
>> fix to Holger? Thanks.
>>
>
> That fix was mainly done with the help of Holger. Many thanks to him for
> doing multiple test during weekend with different combination. He had
> already confirmed that the fix worked for me.
>
> There is another issue that he is hitting when running the test with
> mke2fs -m 0. But I think that is not related to lock inversion.
>
Doing several test with '-m 0' I was unable to reproduce this and I could
now do several runs with afdbench. However the results do show that with
ext4-patch-queue it actually slower:

For ext3: 5449.76 files per second 15.58 MiB/s
For ext4: 5162.16 files per second 15.48 MiB/s
For ext4+patch-queue: 4963.6975 files per second 14.73 MiB/s

On the positive side the bonnie++ numbers got much better:

Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ext3 16G 51501 97 210601 91 100479 32 55528 98 301589 44 1198 5
16G 52702 98 215540 94 99339 32 55376 97 300933 44 1159 4
16G 52426 99 212584 94 99091 31 55656 98 301669 44 1160 4
ext4 16G 52965 98 224199 89 108440 32 56389 99 303792 42 1526 4
16G 52792 98 223980 92 107685 32 56350 98 303066 42 1532 4
16G 52994 98 222354 92 107802 32 56386 99 303727 41 1455 4
ext4(patchqueue)16G 59727 98 252733 52 110177 25 55821 98 296739 42 1553 5
16G 61047 99 239242 48 111664 25 55706 98 297151 42 1545 4
16G 60503 99 241532 47 109655 25 55671 98 297648 42 1552 3

ext3 and ext4 tests where done with 2.6.25.4 and those with patch-queue was
2.6.26-rc5. I will do another test run with 2.6.26-rc5 without patch-queue
just to make sure that the slowdown does not happen due to changes in the
2.6.26 branch.

Another important observation is that ext4+patch-queue does still have the
same error I reported 2 years ago, see:
http://marc.info/?l=linux-ext4&m=115192918624449&w=2

During afdbench the results are written to $HOME/results, see the following
listing before running afdbench:

afdbench@helena:~/results$ ls -ltr
total 260
-rw-r--r-- 1 afdbench afdbench 1274 2008-05-26 09:40 results.10740.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1320 2008-05-26 09:47 results.15132.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1593 2008-05-26 10:03 results.20899.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1499 2008-05-26 10:20 results.30577.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1319 2008-05-26 10:33 results.9819.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1273 2008-05-26 10:40 results.16657.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1274 2008-05-26 10:48 results.21319.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1274 2008-05-26 10:59 results.25168.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1274 2008-05-26 11:05 results.29113.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1274 2008-05-26 11:15 results.1626.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1273 2008-05-26 11:47 results.6247.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1273 2008-05-26 11:53 results.10889.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1274 2008-05-26 12:09 results.8970.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 838 2008-05-26 12:22 results.2635.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1281 2008-05-26 12:35 results.2686.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1327 2008-05-26 12:43 results.7525.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1693 2008-05-26 12:56 results.13665.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1600 2008-05-26 13:15 results.23610.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1600 2008-05-26 15:10 results.13041.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1604 2008-05-26 15:23 results.22862.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1604 2008-05-26 15:43 results.32516.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 14582 2008-05-28 17:16 results.2585.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 14583 2008-05-28 20:17 results.32490.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 14628 2008-05-28 23:19 results.11812.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 14845 2008-05-29 02:22 results.31472.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 21134 2008-05-29 05:30 results.27636.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 21970 2008-05-29 08:40 results.25944.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 3309 2008-05-29 11:11 results.14789.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 3310 2008-05-29 11:43 results.20842.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 4107 2008-06-16 21:45 results.2190.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 4108 2008-06-16 23:04 results.8794.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 4108 2008-06-17 00:22 results.27273.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 4109 2008-06-17 01:41 results.4505.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 8450 2008-06-17 03:05 results.19995.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 9230 2008-06-17 04:35 results.24033.helena.dwd.de

And here is the listing after running the benchmark:

afdbench@helena:~/results$ ls -ltr
total 284
-rw-r--r-- 1 afdbench afdbench 1274 2008-05-26 09:40 results.10740.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1320 2008-05-26 09:47 results.15132.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1593 2008-05-26 10:03 results.20899.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1499 2008-05-26 10:20 results.30577.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1319 2008-05-26 10:33 results.9819.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1273 2008-05-26 10:40 results.16657.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1274 2008-05-26 10:48 results.21319.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1274 2008-05-26 10:59 results.25168.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1274 2008-05-26 11:05 results.29113.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1274 2008-05-26 11:15 results.1626.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1273 2008-05-26 11:47 results.6247.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1273 2008-05-26 11:53 results.10889.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1274 2008-05-26 12:09 results.8970.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 838 2008-05-26 12:22 results.2635.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1281 2008-05-26 12:35 results.2686.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1327 2008-05-26 12:43 results.7525.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1693 2008-05-26 12:56 results.13665.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1600 2008-05-26 13:15 results.23610.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1600 2008-05-26 15:10 results.13041.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1604 2008-05-26 15:23 results.22862.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 1604 2008-05-26 15:43 results.32516.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 14582 2008-05-28 17:16 results.2585.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 14583 2008-05-28 20:17 results.32490.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 14628 2008-05-28 23:19 results.11812.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 14845 2008-05-29 02:22 results.31472.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 21134 2008-05-29 05:30 results.27636.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 21970 2008-05-29 08:40 results.25944.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 3309 2008-05-29 11:11 results.14789.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 3310 2008-05-29 11:43 results.20842.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 4107 2008-06-16 21:45 results.2190.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 4108 2008-06-16 23:04 results.8794.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 4108 2008-06-17 00:22 results.27273.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 4109 2008-06-17 01:41 results.4505.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 8450 2008-06-17 03:05 results.19995.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 8208 2008-06-17 04:35 results.24033.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 8361 2008-06-17 08:19 results.9628.helena.dwd.de
-rw-r--r-- 1 afdbench afdbench 8858 2008-06-17 09:49 results.8134.helena.dwd.de

Note how the size of file results.24033.helena.dwd.de changes from 9230 before
the test to 8208 bytes after the test. Also note the date both have the same
timestamp "2008-06-17 04:35". I have made a copy of results.24033.helena.dwd.de
before the test and compared it with that after the test. The file is just
truncated by 1022 bytes and there is no garbage.

This is reproducible but not always, sometimes only a small part gets truncated
and sometimes several kilobytes get truncated.

Holger

2008-06-18 05:58:09

[permalink] [raw]

Subject: Re: Performance of ext4

On Tue, 17 Jun 2008, Holger Kiehl wrote:

> Doing several test with '-m 0' I was unable to reproduce this and I could
> now do several runs with afdbench. However the results do show that with
> ext4-patch-queue it actually slower:
>
> For ext3: 5449.76 files per second 15.58 MiB/s
> For ext4: 5162.16 files per second 15.48 MiB/s
> For ext4+patch-queue: 4963.6975 files per second 14.73 MiB/s
>
> On the positive side the bonnie++ numbers got much better:
>
> Version 1.03 ------Sequential Output------ --Sequential Input-
> --Random-
> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
> %CP
> ext3 16G 51501 97 210601 91 100479 32 55528 98 301589 44 1198
> 5
> 16G 52702 98 215540 94 99339 32 55376 97 300933 44 1159
> 4
> 16G 52426 99 212584 94 99091 31 55656 98 301669 44 1160
> 4
> ext4 16G 52965 98 224199 89 108440 32 56389 99 303792 42 1526
> 4
> 16G 52792 98 223980 92 107685 32 56350 98 303066 42 1532
> 4
> 16G 52994 98 222354 92 107802 32 56386 99 303727 41 1455
> 4
> ext4(patchqueue)16G 59727 98 252733 52 110177 25 55821 98 296739 42 1553
> 5
> 16G 61047 99 239242 48 111664 25 55706 98 297151 42 1545
> 4
> 16G 60503 99 241532 47 109655 25 55671 98 297648 42 1552
> 3
>
> ext3 and ext4 tests where done with 2.6.25.4 and those with patch-queue was
> 2.6.26-rc5. I will do another test run with 2.6.26-rc5 without patch-queue
> just to make sure that the slowdown does not happen due to changes in the
> 2.6.26 branch.
>
Here the results without patch queue:

Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
16G 52133 98 221378 95 106873 32 55707 99 297065 42 1546 4
16G 52042 98 220931 93 107715 32 55939 98 298810 42 1543 3
16G 52975 98 220976 93 108060 31 56426 98 298906 42 1452 4

For afdbench: 5336.41 files per second 15.63 MiB/s

So it seems that for afdbench the ext4-patch-queue is a slowdown.

I forgot to mention that for bonnie ext4-patch-queue reduces CPU-load
a lot. For block writting it is nearly halved.

Holger

2008-06-19 06:58:49

by Andreas Dilger

[permalink] [raw]

Subject: Re: Performance of ext4

On Jun 18, 2008 05:58 +0000, Holger Kiehl wrote:
> 1.03 ------Sequential Output------ --Sequential Input- --Random-
> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> ext4 K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> 16G 52133 98 221378 95 106873 32 55707 99 297065 42 1546 4
> 16G 52042 98 220931 93 107715 32 55939 98 298810 42 1543 3
> 16G 52975 98 220976 93 108060 31 56426 98 298906 42 1452 4
>
>ext4(patchqueue)16G 59727 98 252733 52 110177 25 55821 98 296739 42 1553 5
>> 16G 61047 99 239242 48 111664 25 55706 98 297151 42 1545 4
>> 16G 60503 99 241532 47 109655 25 55671 98 297648 42 1552 3
>
> I forgot to mention that for bonnie ext4-patch-queue reduces CPU-load
> a lot. For block writting it is nearly halved.

That was the main reason for developing mballoc. With this patch it
would be able to drive almost 500MB/s write, 600MB/s read on the above
system instead of being CPU limited at 250/300MB/s.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

2008-06-19 11:11:14

[permalink] [raw]

Subject: Re: Performance of ext4

On Wed, Jun 18, 2008 at 05:58:00AM +0000, Holger Kiehl wrote:
> For afdbench: 5336.41 files per second 15.63 MiB/s
>
> So it seems that for afdbench the ext4-patch-queue is a slowdown.

Can you remind me where afdbench can be downloaded? And if I remember
correctly, it creates and deletes large numbers of small files,
correct?

It would be interesting to see which new feature introduced by the
ext4 patch queue --- probably dellayed allocation or mballoc --- is
responsible for the slowdown. One or the other (or both) can be
disabled by mounting the filesystem (using a kernel with the ext4
patch queue) with the mount options -O nomballoc or -O nodelalloc.

If it turns out that nomballoc restores the speed for afdbench, for
example, then it will tell us where we need to look more closely.
Ideally we would not want to have one mount option needed to optimize
filesystem operations for large amoutns of modifications to small
files, and another mode of operation when mostly writing to large
files. So if you could do a round of tests using the ext4 patch queue
kernel, with -O nomballoc and -O nodelalloc (and if both seem to
improve things, try "-O nomballoc,nodelalloc" and see if you get back
to the pre-ext4 patch queue speed), it would be very much appreciated.

Thanks, regards,

- Ted

2008-06-19 15:04:39

[permalink] [raw]

Subject: Re: Performance of ext4

On Thu, 19 Jun 2008, Theodore Tso wrote:

> On Wed, Jun 18, 2008 at 05:58:00AM +0000, Holger Kiehl wrote:
>> For afdbench: 5336.41 files per second 15.63 MiB/s
>>
>> So it seems that for afdbench the ext4-patch-queue is a slowdown.
>
> Can you remind me where afdbench can be downloaded? And if I remember
> correctly, it creates and deletes large numbers of small files,
> correct?
>
Yes and yes.

You can download the benchmark but it is complicated to setup. You can
download it from ftp://ftp.dwd.de/pub/afd/afdbench-2.0.0.tar.bz2 and
you will also need ftp://ftp.dwd.de/pub/afd/development/afd-1.4.0pre1.tar.bz2

Here a short guide how you need to set this up (there is also a SETUP
file in afdbench-2.0.0.tar.bz2):

- create a new user for example afdbench
- untar afdbench-2.0.0.tar.bz2 where ever your test filesystem is mounted
eg /home
- ln -s /home/afdbench-2.0.0 /home/afdbench
- ensure that in .bash_profile of user afdbench you have:
PATH=$PATH:$HOME/bin
- login as afdbench
- Untar afd-1.4.0pre1.tar.bz2
- cd afd-1.4.0pre1
- ./configure --prefix=$HOME --enable-ftp_reuse_data_port --enable-passwd_in_msg --enable-expand_path_in_message --enable-compiler-optimizations --enable-with_afdbench_settings --enable-splice_support --enable-sendfile_support
- make
- make install-strip
- In afdbench script change BENCH_PASSWD to whatever you have set the
password of user afdbench.

If you have problems because you do not have openmotif or lesstif, just use
the configure switch --with-gui=none. Also make sure you have an FTP-server
running, I always used vsftpd. To run the test I just called tiny-bench,
in it you will find how you can start it. You can also run without FTP-server
but I do not know if the problems are reproduceable.

> It would be interesting to see which new feature introduced by the
> ext4 patch queue --- probably dellayed allocation or mballoc --- is
> responsible for the slowdown. One or the other (or both) can be
> disabled by mounting the filesystem (using a kernel with the ext4
> patch queue) with the mount options -O nomballoc or -O nodelalloc.
>
> If it turns out that nomballoc restores the speed for afdbench, for
> example, then it will tell us where we need to look more closely.
> Ideally we would not want to have one mount option needed to optimize
> filesystem operations for large amoutns of modifications to small
> files, and another mode of operation when mostly writing to large
> files. So if you could do a round of tests using the ext4 patch queue
> kernel, with -O nomballoc and -O nodelalloc (and if both seem to
> improve things, try "-O nomballoc,nodelalloc" and see if you get back
> to the pre-ext4 patch queue speed), it would be very much appreciated.
>
Yes, I will try and redo the test as suggested, it might just take a while
since I just made my testing system operational.

What worries me more is the truncation of files, it makes the filesystem
unusable since you loose data. I hope there will be a solution for this.

Holger

2008-06-19 15:58:03

[permalink] [raw]

Subject: Re: Performance of ext4

On Tue, Jun 17, 2008 at 11:42:36AM +0000, Holger Kiehl wrote:
> Note how the size of file results.24033.helena.dwd.de changes from
> 9230 before the test to 8208 bytes after the test. Also note the
> date both have the same timestamp "2008-06-17 04:35". I have made a
> copy of results.24033.helena.dwd.de before the test and compared it
> with that after the test. The file is just truncated by 1022 bytes
> and there is no garbage.

So the corruption is always a truncation, correct?

Did you notice the problem with ext4 w/o the patch queue? I have a
suspicion that the problem may have been introduced by the delayed
allocation code, but I don't have hard evidence. When you rerun your
benchmark (which seems to be the closest thing we have to a
reproduction case), it would be interesting to know if the problem
goes away with -o nodelalloc (again, it would localize where we need
to look).

Thanks, regards,

- Ted

2008-06-19 16:52:04

by Eric Sandeen

[permalink] [raw]

Subject: Re: Performance of ext4

Theodore Tso wrote:
> On Tue, Jun 17, 2008 at 11:42:36AM +0000, Holger Kiehl wrote:
>> Note how the size of file results.24033.helena.dwd.de changes from
>> 9230 before the test to 8208 bytes after the test. Also note the
>> date both have the same timestamp "2008-06-17 04:35". I have made a
>> copy of results.24033.helena.dwd.de before the test and compared it
>> with that after the test. The file is just truncated by 1022 bytes
>> and there is no garbage.
>
> So the corruption is always a truncation, correct?
>
> Did you notice the problem with ext4 w/o the patch queue? I have a
> suspicion that the problem may have been introduced by the delayed
> allocation code, but I don't have hard evidence. When you rerun your
> benchmark (which seems to be the closest thing we have to a
> reproduction case), it would be interesting to know if the problem
> goes away with -o nodelalloc (again, it would localize where we need
> to look).
>
> Thanks, regards,

It might be worth runninga "simple" fsx under your kernel too; last time
I tested fsx it was still happy and it exercises fs ops (including
truncate) at random...

-Eric

2008-06-19 17:43:46

[permalink] [raw]

Subject: Re: Performance of ext4

On Thu, Jun 19, 2008 at 11:41:17AM -0500, Eric Sandeen wrote:
>
> It might be worth runninga "simple" fsx under your kernel too; last time
> I tested fsx it was still happy and it exercises fs ops (including
> truncate) at random...
>

>From what Holger described, it's doubtful that the bug is in the
truncate operation. It sounds like i_size is actually dropping in
size at some pointer long after the file was written. If I had to
guess the value in the inode cache is correct; and perhaps so is the
value on the journal. But somehow, the wrong value is getting written
to disk (remember the jbd layer can keep up to three different
versions of filesystem metadata in memory, because most of the time we
don't block modifications to the filesystem while we are in the middle
of writing a previous commit to disk). So depending on whether the
inode gets redirtied or not, the inconsistency could self-heal, and if
the inode never gets pushed out of memory due to memory pressure, the
problem might not be noticed until the system reboots or the
filesystem is unmounted.

This is one of the reasons why I'm a bit suspicious that the problem
may lie in the delayed allocation code; changing i_size without first
starting a transaction could lead to this sort of problem, for
example, and the delayed allocation could represent a different code
path where file blocks get allocated and i_size gets changed.

- Ted

2008-06-19 19:51:27

[permalink] [raw]

Subject: Re: Performance of ext4

On Thu, 2008-06-19 at 13:42 -0400, Theodore Tso wrote:
> On Thu, Jun 19, 2008 at 11:41:17AM -0500, Eric Sandeen wrote:
> >
> > It might be worth runninga "simple" fsx under your kernel too; last time
> > I tested fsx it was still happy and it exercises fs ops (including
> > truncate) at random...
> >
>
> From what Holger described, it's doubtful that the bug is in the
> truncate operation. It sounds like i_size is actually dropping in
> size at some pointer long after the file was written. If I had to
> guess the value in the inode cache is correct; and perhaps so is the
> value on the journal. But somehow, the wrong value is getting written
> to disk (remember the jbd layer can keep up to three different
> versions of filesystem metadata in memory, because most of the time we
> don't block modifications to the filesystem while we are in the middle
> of writing a previous commit to disk). So depending on whether the
> inode gets redirtied or not, the inconsistency could self-heal, and if
> the inode never gets pushed out of memory due to memory pressure, the
> problem might not be noticed until the system reboots or the
> filesystem is unmounted.
>
> This is one of the reasons why I'm a bit suspicious that the problem
> may lie in the delayed allocation code; changing i_size without first
> starting a transaction could lead to this sort of problem, for
> example, and the delayed allocation could represent a different code
> path where file blocks get allocated and i_size gets changed.
>

I tend to agree. Without delayed allocation, the in-memory i_size and
on-disk i_disksize normally match each other, since we do block
allocation at prepare_write/write_begin time, and the i_size update just
immedietly around that time. However, with delayed allocation, the in
memory i_size is being update around prepare_write/commit_write, but the
i_disksize won't updated until later writepage/writepages() time. The
window now gets much larger.

With writeback mode, since there is no ordering there, I think it's
possible the the inode dirty pages have been sync to disk and the inode
structure being pushed out of the memory due to memory pressure, before
the i_disksize update cached in jbd2 reach to disk. Perhaps that explain
the "truncation"?

Not sure if this still a issue with the delalloc on new ordered mode, I
guess as long as the inode is on jinode list, and that inode can't push
out of memeory due to memory pressure since jbd is referencing it, then
this seems couldn't happen...

> - Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2008-06-20 08:10:01

[permalink] [raw]

Subject: Re: Performance of ext4

On Thu, 19 Jun 2008, Theodore Tso wrote:

> On Tue, Jun 17, 2008 at 11:42:36AM +0000, Holger Kiehl wrote:
>> Note how the size of file results.24033.helena.dwd.de changes from
>> 9230 before the test to 8208 bytes after the test. Also note the
>> date both have the same timestamp "2008-06-17 04:35". I have made a
>> copy of results.24033.helena.dwd.de before the test and compared it
>> with that after the test. The file is just truncated by 1022 bytes
>> and there is no garbage.
>
> So the corruption is always a truncation, correct?
>
Correct.

> Did you notice the problem with ext4 w/o the patch queue?
>
No, without patch queue it I did not not notice the problem.

> I have a
> suspicion that the problem may have been introduced by the delayed
> allocation code, but I don't have hard evidence. When you rerun your
> benchmark (which seems to be the closest thing we have to a
> reproduction case), it would be interesting to know if the problem
> goes away with -o nodelalloc (again, it would localize where we need
> to look).
>
Ok I will do thats soon as I have a system available.

Holger

2008-06-20 08:32:52

[permalink] [raw]

Subject: Re: Performance of ext4

On Thu, 19 Jun 2008, Theodore Tso wrote:

> On Thu, Jun 19, 2008 at 11:41:17AM -0500, Eric Sandeen wrote:
>>
>> It might be worth runninga "simple" fsx under your kernel too; last time
>> I tested fsx it was still happy and it exercises fs ops (including
>> truncate) at random...
>>
>
> From what Holger described, it's doubtful that the bug is in the
> truncate operation.
>
Correct, the benchmark just copies, moves, hardlinks and deletes a lot
of small files. It also overwrites existing files but not at the same
scale it does the other operations.

> It sounds like i_size is actually dropping in
> size at some pointer long after the file was written. If I had to
> guess the value in the inode cache is correct; and perhaps so is the
> value on the journal. But somehow, the wrong value is getting written
> to disk (remember the jbd layer can keep up to three different
> versions of filesystem metadata in memory, because most of the time we
> don't block modifications to the filesystem while we are in the middle
> of writing a previous commit to disk). So depending on whether the
> inode gets redirtied or not, the inconsistency could self-heal, and if
> the inode never gets pushed out of memory due to memory pressure, the
> problem might not be noticed until the system reboots or the
> filesystem is unmounted.
>
I always had the feeling that waiting a day or unmounting caused a lot
more truncation. On my system at home for example I mounted the test
filesystem again and saw that files where truncated and I am pretty sure
that when I looked at those files during and shortly after the test they
where still complete. But I will recheck and do test as you suggested.

What I find strange is that the missing parts of the file are not for
example exactly 512 or 1024 or 4096 bytes it is mostly some odd number
of bytes.

Holger

2008-06-20 09:00:42

[permalink] [raw]

Subject: Re: Performance of ext4

On Fri, Jun 20, 2008 at 08:32:52AM +0000, Holger Kiehl wrote:
>> It sounds like i_size is actually dropping in
>> size at some pointer long after the file was written. If I had to

sorry, "at some point"...

>> guess the value in the inode cache is correct; and perhaps so is the
>> value on the journal. But somehow, the wrong value is getting written
>> to disk

Or, "the right value is never getting written to disk". (Which as I
think about it is more likely; it's likely that an update to i_size is
getting *lost*, perhaps because the delalloc code is possibly
modifying i_size without starting a transaction first. Again this is
just a guess.)

> What I find strange is that the missing parts of the file are not for
> example exactly 512 or 1024 or 4096 bytes it is mostly some odd number
> of bytes.

Is there any chance the truncation point is related to how the program
is writing its output file? i.e., if it is a text file, is the
truncation happening after a new-line or when the stdio library might
have done an explicit or implicit fflush()?

- Ted

2008-06-20 09:21:53

[permalink] [raw]

Subject: Re: Performance of ext4

On Fri, 20 Jun 2008, Theodore Tso wrote:

> On Fri, Jun 20, 2008 at 08:32:52AM +0000, Holger Kiehl wrote:
>>> It sounds like i_size is actually dropping in
>>> size at some pointer long after the file was written. If I had to
>
> sorry, "at some point"...
>
>>> guess the value in the inode cache is correct; and perhaps so is the
>>> value on the journal. But somehow, the wrong value is getting written
>>> to disk
>
> Or, "the right value is never getting written to disk". (Which as I
> think about it is more likely; it's likely that an update to i_size is
> getting *lost*, perhaps because the delalloc code is possibly
> modifying i_size without starting a transaction first. Again this is
> just a guess.)
>
>> What I find strange is that the missing parts of the file are not for
>> example exactly 512 or 1024 or 4096 bytes it is mostly some odd number
>> of bytes.
>
> Is there any chance the truncation point is related to how the program
> is writing its output file? i.e., if it is a text file, is the
> truncation happening after a new-line or when the stdio library might
> have done an explicit or implicit fflush()?
>
When the benchmark runs it writes to stdout and with tee to the result
file. It first writes some information about the system, prepares the
test files (creates lots of small files), calls sync and then starts
the test. Then every minute one line gets written to the result file.
Often I have seen that everything after the sync was missing. But
sometimes it happened that some parts at the end are missing. But it
was always a clean cut, that is there where no lines that where cut
partially. The lines where always complete.

Holger

2008-06-21 15:02:54

[permalink] [raw]

Subject: Re: Performance of ext4

On Fri, 20 Jun 2008, Holger Kiehl wrote:

> On Thu, 19 Jun 2008, Theodore Tso wrote:
>
>> I have a
>> suspicion that the problem may have been introduced by the delayed
>> allocation code, but I don't have hard evidence. When you rerun your
>> benchmark (which seems to be the closest thing we have to a
>> reproduction case), it would be interesting to know if the problem
>> goes away with -o nodelalloc (again, it would localize where we need
>> to look).
>>
> Ok I will do thats soon as I have a system available.
>
No, with nodelalloc the problem does not occur. I can now very reliably
repoduce the problem if I run the benchmark for 5 minutes and then unmount
the filesystem and remount it. Anything else I can do to find the problem?

During testing on a newly setup system (used the same kernel 2.6.26-rc5+ext4-
patch-queue) I have hit two oopses:

------------[ cut here ]------------
kernel BUG at fs/ext4/inode.c:1931!
invalid opcode: 0000 [1] SMP
CPU 3
Modules linked in: w83627hf lm85 hwmon_vid bonding ipt_REJECT xt_tcpudp nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables binfmt_misc sg floppy ohci_hcd i2c_amd756 k8temp i2c_core button usbcore
Pid: 242, comm: pdflush Not tainted 2.6.26-rc5 #4
RIP: 0010:[<ffffffff802dd1c5>] [<ffffffff802dd1c5>] ext4_normal_writepage+0x27/0xd8
RSP: 0018:ffff81007f329c20 EFLAGS: 00010246
RAX: 0a0000000001002d RBX: ffffe20006f658c8 RCX: 00000000ffffffe8
RDX: 0000000000004278 RSI: ffff81007f329e50 RDI: ffffe20006f658c8
RBP: ffff81007f329e50 R08: ffff81027ff054c0 R09: ffff81007f329b60
R10: 0000000000000040 R11: 0000000000000040 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
FS: 00007fbc99d8b6f0(0000) GS:ffff81027ff0e000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007f64c0d43488 CR3: 00000001efa65000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process pdflush (pid: 242, threadinfo ffff81007f328000, task ffff81007fbc3d30)
Stack: 0000000000000000 ffff810149bc41d0 ffff81007f329e50 ffffffff8026179a
ffffe20006f658c8 ffffffff80261f57 ffff810149bc41d0 ffffffff80261790
ffff810149bc41d0 0000000180241154 ffff8101ff6caea0 ffffffffffffffff
Call Trace:
[<ffffffff8026179a>] ? __writepage+0xa/0x27
[<ffffffff80261f57>] ? write_cache_pages+0x1bb/0x30d
[<ffffffff80261790>] ? __writepage+0x0/0x27
[<ffffffff802620ec>] ? do_writepages+0x27/0x2d
[<ffffffff8029f66e>] ? __writeback_single_inode+0x16f/0x35e
[<ffffffff8029fc8b>] ? sync_sb_inodes+0x283/0x3b8
[<ffffffff802a0014>] ? writeback_inodes+0x89/0xde
[<ffffffff80262620>] ? background_writeout+0x8f/0xc9
[<ffffffff80262c74>] ? pdflush+0x122/0x1c9
[<ffffffff80262591>] ? background_writeout+0x0/0xc9
[<ffffffff80262b52>] ? pdflush+0x0/0x1c9
[<ffffffff8023e3e7>] ? kthread+0x47/0x73
[<ffffffff8022ae3a>] ? schedule_tail+0x27/0x5c
[<ffffffff8020bf78>] ? child_rip+0xa/0x12
[<ffffffff8023e3a0>] ? kthread+0x0/0x73
[<ffffffff8020bf6e>] ? child_rip+0x0/0x12

Code: 3e 8c fc ff 55 48 89 f5 53 48 89 fb 48 83 ec 08 48 8b 47 18 48 8b 00 48 8b 50 68 48 8b 07 a8 01 75 04 0f 0b eb fe f6 c4 08 75 04 <0f> 0b eb fe 48 89 d0 81 e2 ff 0f 00 00 48 8b 77 10 48 c1 f8 0c
RIP [<ffffffff802dd1c5>] ext4_normal_writepage+0x27/0xd8
RSP <ffff81007f329c20>
---[ end trace 82357a26b61bf3d2 ]---
------------[ cut here ]------------
kernel BUG at fs/ext4/inode.c:1931!
invalid opcode: 0000 [2] SMP
CPU 3
Modules linked in: w83627hf lm85 hwmon_vid bonding ipt_REJECT xt_tcpudp nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables binfmt_misc sg floppy ohci_hcd i2c_amd756 k8temp i2c_core button usbcore
Pid: 7896, comm: dir_check Tainted: G D 2.6.26-rc5 #4
RIP: 0010:[<ffffffff802dd1c5>] [<ffffffff802dd1c5>] ext4_normal_writepage+0x27/0xd8
RSP: 0018:ffff810172ee1d78 EFLAGS: 00010246
RAX: 0a0000000001002d RBX: ffffe200063cd890 RCX: 00000000ffffffe8
RDX: 0000000000004278 RSI: ffff810172ee1eb8 RDI: ffffe200063cd890
RBP: ffff810172ee1eb8 R08: ffff81027ff054c0 R09: ffff810172ee1cb8
R10: ffff810250e6ac80 R11: ffff810250e6a080 R12: 0000000000000000
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
FS: 00007f3778d0e6f0(0000) GS:ffff81027ff0e000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f3778d25000 CR3: 000000027ecd8000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process dir_check (pid: 7896, threadinfo ffff810172ee0000, task ffff81017f0fa6f0)
Stack: 0000000000000000 ffff81024ed0c8c0 ffff810172ee1eb8 ffffffff8026179a
ffffe200063cd890 ffffffff80261f57 ffff81024ed0c8c0 ffffffff80261790
ffff81024ed0c8c0 0000000500000000 ffff8101ff6caea0 0007ffffffffffff
Call Trace:
[<ffffffff8026179a>] ? __writepage+0xa/0x27
[<ffffffff80261f57>] ? write_cache_pages+0x1bb/0x30d
[<ffffffff80261790>] ? __writepage+0x0/0x27
[<ffffffff8028ea23>] ? fasync_helper+0x47/0xc8
[<ffffffff80286f15>] ? cp_new_stat+0xe9/0xfc
[<ffffffff802921b3>] ? __posix_lock_file+0x43d/0x44e
[<ffffffff802620ec>] ? do_writepages+0x27/0x2d
[<ffffffff8025c44f>] ? __filemap_fdatawrite_range+0x4a/0x55
[<ffffffff802568c8>] ? audit_syscall_exit+0x2ee/0x30c
[<ffffffff802a28e1>] ? do_fsync+0x2f/0x87
[<ffffffff80270af7>] ? sys_msync+0x107/0x178
[<ffffffff8020b2d2>] ? tracesys+0xd5/0xda

Code: 3e 8c fc ff 55 48 89 f5 53 48 89 fb 48 83 ec 08 48 8b 47 18 48 8b 00 48 8b 50 68 48 8b 07 a8 01 75 04 0f 0b eb fe f6 c4 08 75 04 <0f> 0b eb fe 48 89 d0 81 e2 ff 0f 00 00 48 8b 77 10 48 c1 f8 0c
RIP [<ffffffff802dd1c5>] ext4_normal_writepage+0x27/0xd8
RSP <ffff810172ee1d78>
---[ end trace 82357a26b61bf3d2 ]---
SysRq : Show Blocked State
task PC stack pid father
dir_check D ffff810181e03eb8 0 7873 1
ffff8101eea3dc48 0000000000000082 0000000000000096 ffff81007ec39ec0
ffffffff80600cc0 0000000000000000 ffff81027ff8a658 ffff8101eea21bd0
ffff81007f395370 ffff8101eea21e08 000000017ff8a640 ffff8101eea21e08
Call Trace:
[<ffffffff80224a67>] __wake_up+0x38/0x4f
[<ffffffff8025b87d>] sync_page+0x0/0x41
[<ffffffff804660f0>] io_schedule+0x2d/0x39
[<ffffffff8025b8b9>] sync_page+0x3c/0x41
[<ffffffff804663d4>] __wait_on_bit+0x41/0x70
[<ffffffff8025bb32>] wait_on_page_bit+0x6b/0x71
[<ffffffff8023e724>] wake_bit_function+0x0/0x23
[<ffffffff8026940b>] do_wp_page+0x46a/0x4ba
[<ffffffff8026aac1>] handle_mm_fault+0x62d/0x6a3
[<ffffffff80299236>] mntput_no_expire+0x27/0x11e
[<ffffffff8021ed0c>] do_page_fault+0x435/0x825
[<ffffffff802921b3>] __posix_lock_file+0x43d/0x44e
[<ffffffff8029260e>] fcntl_setlk+0x285/0x296
[<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff80467699>] error_exit+0x0/0x51

Holger

2008-06-23 17:45:52

[permalink] [raw]

Subject: Re: Performance of ext4

On Fri, Jun 20, 2008 at 09:21:48AM +0000, Holger Kiehl wrote:
> On Fri, 20 Jun 2008, Theodore Tso wrote:
>
>> On Fri, Jun 20, 2008 at 08:32:52AM +0000, Holger Kiehl wrote:
>>>> It sounds like i_size is actually dropping in
>>>> size at some pointer long after the file was written. If I had to
>>
>> sorry, "at some point"...
>>
>>>> guess the value in the inode cache is correct; and perhaps so is the
>>>> value on the journal. But somehow, the wrong value is getting written
>>>> to disk
>>
>> Or, "the right value is never getting written to disk". (Which as I
>> think about it is more likely; it's likely that an update to i_size is
>> getting *lost*, perhaps because the delalloc code is possibly
>> modifying i_size without starting a transaction first. Again this is
>> just a guess.)
>>
>>> What I find strange is that the missing parts of the file are not for
>>> example exactly 512 or 1024 or 4096 bytes it is mostly some odd number
>>> of bytes.
>>
>> Is there any chance the truncation point is related to how the program
>> is writing its output file? i.e., if it is a text file, is the
>> truncation happening after a new-line or when the stdio library might
>> have done an explicit or implicit fflush()?
>>
> When the benchmark runs it writes to stdout and with tee to the result
> file. It first writes some information about the system, prepares the
> test files (creates lots of small files), calls sync and then starts
> the test. Then every minute one line gets written to the result file.
> Often I have seen that everything after the sync was missing. But
> sometimes it happened that some parts at the end are missing. But it
> was always a clean cut, that is there where no lines that where cut
> partially. The lines where always complete.
>

I found one place where we fail to update i_disksize. Can you try this
patch ?

diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 33f940b..9fa737f 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
loff_t size;
unsigned long len;
handle_t *handle = NULL;
+ ext4_lblk_t block;
+ loff_t disksize;
struct buffer_head *page_bufs;
+ struct buffer_head *bh, *head;
struct inode *inode = page->mapping->host;

handle = ext4_journal_current_handle();
@@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
else
ret = block_write_full_page(page, ext4_da_get_block_write, wbc);

+ if (ret)
+ return ret;
+ /*
+ * When called via shrink_page_list and if we don't have any unmapped
+ * buffer_head we still could have written some new content in an
+ * already mapped buffer. That means we need to extent i_disksize here
+ */
+ /* Find the last logical block number in the page. */
+ block = (sector_t)page->index << (PAGE_CACHE_SHIFT - inode->i_blkbits);
+ bh = head = page_buffers(page);
+ do {
+ bh = bh->b_this_page;
+ block++;
+ } while (bh != head);
+
+ disksize = ((loff_t) block) << inode->i_blkbits;
+ if (disksize > i_size_read(inode))
+ disksize = i_size_read(inode);
+ if (disksize > EXT4_I(inode)->i_disksize) {
+ /*
+ * XXX: replace with spinlock if seen contended -bzzz
+ */
+ down_write(&EXT4_I(inode)->i_data_sem);
+ if (disksize > EXT4_I(inode)->i_disksize)
+ EXT4_I(inode)->i_disksize = disksize;
+ up_write(&EXT4_I(inode)->i_data_sem);
+
+ if (EXT4_I(inode)->i_disksize == disksize) {
+ ret = ext4_mark_inode_dirty(handle, inode);
+ return ret;
+ }
+ }
return ret;
}

2008-06-23 20:55:12

by Andreas Dilger

[permalink] [raw]

Subject: Re: Performance of ext4

On Jun 20, 2008 09:21 +0000, Holger Kiehl wrote:
> When the benchmark runs it writes to stdout and with tee to the result
> file. It first writes some information about the system, prepares the
> test files (creates lots of small files), calls sync and then starts
> the test. Then every minute one line gets written to the result file.
> Often I have seen that everything after the sync was missing. But
> sometimes it happened that some parts at the end are missing. But it
> was always a clean cut, that is there where no lines that where cut
> partially. The lines where always complete.

Could you enhance your test to record the file size from "stat -c %s {file}"
at the end of the test, and also "dumpe2fs -c -R 'stat <inum>' {dev}"
where <inum> is from "stat -c %i {file}". If these two don't match after
60s, or after a "sync; sync" then there will likely be a data loss.

With delalloc it is expected that the "debugfs" output will not match up
to 30s+ after the last modification, because the write is only in memory.
With ext3 this window would be much smaller, and in fact not visible from
userspace because "debugfs" would be accessing the same (in memory) buffer
as the kernel, so it can't even access the stale data on disk.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

2008-06-24 00:31:11

[permalink] [raw]

Subject: Re: Performance of ext4

On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:

> I found one place where we fail to update i_disksize. Can you try this
> patch ?
>
> diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> index 33f940b..9fa737f 100644
> --- a/fs/ext4/inode.c
> +++ b/fs/ext4/inode.c
> @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
> loff_t size;
> unsigned long len;
> handle_t *handle = NULL;
> + ext4_lblk_t block;
> + loff_t disksize;
> struct buffer_head *page_bufs;
> + struct buffer_head *bh, *head;
> struct inode *inode = page->mapping->host;
>
> handle = ext4_journal_current_handle();
> @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
> else
> ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
>
> + if (ret)
> + return ret;
> + /*
> + * When called via shrink_page_list and if we don't have any unmapped
> + * buffer_head we still could have written some new content in an
> + * already mapped buffer. That means we need to extent i_disksize here
> + */

In this case(when extend the file without need block allocation),
wouldn't make sense to update the i_disksize at write_end() time? So
that the window of i_size different from i_disksize could be much
smaller in this case.

Something like below? (untested)

Signed-off-by: Mingming Cao <[email protected]>
---
fs/ext4/inode.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++------
1 file changed, 60 insertions(+), 6 deletions(-)

Index: linux-2.6.26-rc6/fs/ext4/inode.c
===================================================================
--- linux-2.6.26-rc6.orig/fs/ext4/inode.c 2008-06-23 17:28:01.000000000 -0700
+++ linux-2.6.26-rc6/fs/ext4/inode.c 2008-06-23 17:28:10.000000000 -0700
@@ -1573,6 +1573,65 @@ static int ext4_da_write_begin(struct fi
return ret;
}

+static int ext4_bh_unmapped_or_delay(handle_t *handle, struct buffer_head *bh)
+{
+ return !buffer_mapped(bh) || buffer_delay(bh);
+}
+
+static int ext4_da_write_end(struct file *file,
+ struct address_space *mapping,
+ loff_t pos, unsigned len, unsigned copied,
+ struct page *page, void *fsdata)
+{
+ handle_t *handle = NULL;
+ struct inode *inode = mapping->host;
+ int needed_blocks = ext4_writepage_trans_blocks(inode);
+ unsigned from, to;
+ int ret = 0, ret2;
+
+ from = pos & (PAGE_CACHE_SIZE - 1);
+ to = from + len;
+
+ if (ret == 0) {
+ /*
+ * generic_write_end() will run mark_inode_dirty() if i_size
+ * changes. So let's piggyback the i_disksize mark_inode_dirty
+ * into that.
+ */
+ loff_t new_i_size;
+
+ new_i_size = pos + copied;
+ if (new_i_size > EXT4_I(inode)->i_disksize)
+ if (!walk_page_buffers(NULL, page_buffers(page),
+ 0, len, NULL, ext4_bh_unmapped_or_delay)){
+ /*
+ * Updating i_disksize when extending file without
+ * need block allocation
+ */
+ handle = ext4_journal_start(inode, needed_blocks);
+ if (IS_ERR(handle)) {
+ ret = PTR_ERR(handle);
+ return ret;
+ }
+ if (ext4_should_order_data(inode))
+ ret = ext4_jbd2_file_inode(handle, inode);
+
+ EXT4_I(inode)->i_disksize = new_i_size;
+ }
+ ret2 = generic_write_end(file, mapping, pos, len, copied,
+ page, fsdata);
+ copied = ret2;
+ if (ret2 < 0)
+ ret = ret2;
+ }
+ if (handle)
+ ret2 = ext4_journal_stop(handle);
+ if (!ret)
+ ret = ret2;
+
+ return ret ? ret : copied;
+}
+
static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
{
struct buffer_head *head, *bh;
@@ -1682,11 +1741,6 @@ static int bput_one(handle_t *handle, st
return 0;
}

-static int ext4_bh_unmapped_or_delay(handle_t *handle, struct buffer_head *bh)
-{
- return !buffer_mapped(bh) || buffer_delay(bh);
-}

2008-06-24 03:07:49

[permalink] [raw]

Subject: Re: Performance of ext4

On Mon, Jun 23, 2008 at 05:31:32PM -0700, Mingming wrote:
>
> On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:
>
> > I found one place where we fail to update i_disksize. Can you try this
> > patch ?
> >
> > diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> > index 33f940b..9fa737f 100644
> > --- a/fs/ext4/inode.c
> > +++ b/fs/ext4/inode.c
> > @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
> > loff_t size;
> > unsigned long len;
> > handle_t *handle = NULL;
> > + ext4_lblk_t block;
> > + loff_t disksize;
> > struct buffer_head *page_bufs;
> > + struct buffer_head *bh, *head;
> > struct inode *inode = page->mapping->host;
> >
> > handle = ext4_journal_current_handle();
> > @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
> > else
> > ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
> >
> > + if (ret)
> > + return ret;
> > + /*
> > + * When called via shrink_page_list and if we don't have any unmapped
> > + * buffer_head we still could have written some new content in an
> > + * already mapped buffer. That means we need to extent i_disksize here
> > + */
>
> In this case(when extend the file without need block allocation),
> wouldn't make sense to update the i_disksize at write_end() time? So
> that the window of i_size different from i_disksize could be much
> smaller in this case.
>
>
> Something like below? (untested)

In this case you will have to start a transaction in write_begin . With
the below code transaction is started inside page_lock. Also I don't
think we need needed_blocks credit just 1 should be enough because we
are not doing any block allocation here. We just need to update the
inode block.

-aneesh

>
> Signed-off-by: Mingming Cao <[email protected]>
> ---
> fs/ext4/inode.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++------
> 1 file changed, 60 insertions(+), 6 deletions(-)
>
> Index: linux-2.6.26-rc6/fs/ext4/inode.c
> ===================================================================
> --- linux-2.6.26-rc6.orig/fs/ext4/inode.c 2008-06-23 17:28:01.000000000 -0700
> +++ linux-2.6.26-rc6/fs/ext4/inode.c 2008-06-23 17:28:10.000000000 -0700
> @@ -1573,6 +1573,65 @@ static int ext4_da_write_begin(struct fi
> return ret;
> }
>
> +static int ext4_bh_unmapped_or_delay(handle_t *handle, struct buffer_head *bh)
> +{
> + return !buffer_mapped(bh) || buffer_delay(bh);
> +}
> +
> +static int ext4_da_write_end(struct file *file,
> + struct address_space *mapping,
> + loff_t pos, unsigned len, unsigned copied,
> + struct page *page, void *fsdata)
> +{
> + handle_t *handle = NULL;
> + struct inode *inode = mapping->host;
> + int needed_blocks = ext4_writepage_trans_blocks(inode);
> + unsigned from, to;
> + int ret = 0, ret2;
> +
> + from = pos & (PAGE_CACHE_SIZE - 1);
> + to = from + len;
> +
> + if (ret == 0) {
> + /*
> + * generic_write_end() will run mark_inode_dirty() if i_size
> + * changes. So let's piggyback the i_disksize mark_inode_dirty
> + * into that.
> + */
> + loff_t new_i_size;
> +
> + new_i_size = pos + copied;
> + if (new_i_size > EXT4_I(inode)->i_disksize)
> + if (!walk_page_buffers(NULL, page_buffers(page),
> + 0, len, NULL, ext4_bh_unmapped_or_delay)){
> + /*
> + * Updating i_disksize when extending file without
> + * need block allocation
> + */
> + handle = ext4_journal_start(inode, needed_blocks);
> + if (IS_ERR(handle)) {
> + ret = PTR_ERR(handle);
> + return ret;
> + }
> + if (ext4_should_order_data(inode))
> + ret = ext4_jbd2_file_inode(handle, inode);
> +
> + EXT4_I(inode)->i_disksize = new_i_size;
> + }
> + ret2 = generic_write_end(file, mapping, pos, len, copied,
> + page, fsdata);
> + copied = ret2;
> + if (ret2 < 0)
> + ret = ret2;
> + }
> + if (handle)
> + ret2 = ext4_journal_stop(handle);
> + if (!ret)
> + ret = ret2;
> +
> + return ret ? ret : copied;
> +}
> +
> static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
> {
> struct buffer_head *head, *bh;
> @@ -1682,11 +1741,6 @@ static int bput_one(handle_t *handle, st
> return 0;
> }
>
> -static int ext4_bh_unmapped_or_delay(handle_t *handle, struct buffer_head *bh)
> -{
> - return !buffer_mapped(bh) || buffer_delay(bh);
> -}
> -
> /*
> * Note that we don't need to start a transaction unless we're journaling data
> * because we should have holes filled from ext4_page_mkwrite(). We even don't
> @@ -2050,7 +2104,7 @@ static const struct address_space_operat
> .writepages = ext4_da_writepages,
> .sync_page = block_sync_page,
> .write_begin = ext4_da_write_begin,
> - .write_end = generic_write_end,
> + .write_end = ext4_da_write_end,
> .bmap = ext4_bmap,
> .invalidatepage = ext4_da_invalidatepage,
> .releasepage = ext4_releasepage,
>
>

2008-06-24 03:28:36

[permalink] [raw]

Subject: Re: Performance of ext4

On Tue, Jun 24, 2008 at 08:37:21AM +0530, Aneesh Kumar K.V wrote:
> On Mon, Jun 23, 2008 at 05:31:32PM -0700, Mingming wrote:
> >
> > On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:
> >
> > > I found one place where we fail to update i_disksize. Can you try this
> > > patch ?
> > >
> > > diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> > > index 33f940b..9fa737f 100644
> > > --- a/fs/ext4/inode.c
> > > +++ b/fs/ext4/inode.c
> > > @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
> > > loff_t size;
> > > unsigned long len;
> > > handle_t *handle = NULL;
> > > + ext4_lblk_t block;
> > > + loff_t disksize;
> > > struct buffer_head *page_bufs;
> > > + struct buffer_head *bh, *head;
> > > struct inode *inode = page->mapping->host;
> > >
> > > handle = ext4_journal_current_handle();
> > > @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
> > > else
> > > ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
> > >
> > > + if (ret)
> > > + return ret;
> > > + /*
> > > + * When called via shrink_page_list and if we don't have any unmapped
> > > + * buffer_head we still could have written some new content in an
> > > + * already mapped buffer. That means we need to extent i_disksize here
> > > + */
> >
> > In this case(when extend the file without need block allocation),
> > wouldn't make sense to update the i_disksize at write_end() time? So
> > that the window of i_size different from i_disksize could be much
> > smaller in this case.
> >
> >
> > Something like below? (untested)
>
> In this case you will have to start a transaction in write_begin . With
> the below code transaction is started inside page_lock. Also I don't
> think we need needed_blocks credit just 1 should be enough because we
> are not doing any block allocation here. We just need to update the
> inode block.
>
>

diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 33f940b..0ccf7b9 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1770,6 +1770,7 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
struct page *page;
pgoff_t index;
unsigned from, to;
+ handle_t *handle;
struct inode *inode = mapping->host;

index = pos >> PAGE_CACHE_SHIFT;
@@ -1777,6 +1778,17 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
to = from + len;

retry:
+ /*
+ * If we are writing towards the end of an already mapped
+ * buffer_head, we don't do any block allocation. But we
+ * need to update i_disksize.
+ */
+ handle = ext4_journal_start(inode, 1);
+ if (IS_ERR(handle)) {
+ ret = PTR_ERR(handle);
+ goto out;
+ }
+
page = __grab_cache_page(mapping, index);
if (!page)
return -ENOMEM;
@@ -1786,15 +1798,65 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
ext4_da_get_block_prep);
if (ret < 0) {
unlock_page(page);
+ ext4_journal_stop(handle);
page_cache_release(page);
}

if (ret == -ENOSPC && ext4_should_retry_alloc(inode->i_sb, &retries))
goto retry;

+out:
return ret;
}

+static int ext4_da_write_end(struct file *file,
+ struct address_space *mapping,
+ loff_t pos, unsigned len, unsigned copied,
+ struct page *page, void *fsdata)
+{
+ handle_t *handle = ext4_journal_current_handle();
+ struct inode *inode = mapping->host;
+ unsigned from, to;
+ int ret = 0, ret2;
+
+ from = pos & (PAGE_CACHE_SIZE - 1);
+ to = from + len;
+
+ if (ret == 0) {
+ /*
+ * generic_write_end() will run mark_inode_dirty() if i_size
+ * changes. So let's piggyback the i_disksize mark_inode_dirty
+ * into that.
+ */
+ loff_t new_i_size;
+
+ new_i_size = pos + copied;
+ if (new_i_size > EXT4_I(inode)->i_disksize)
+ if (!walk_page_buffers(NULL, page_buffers(page),
+ 0, len, NULL,
+ ext4_bh_unmapped_or_delay)) {
+ /*
+ * Updating i_disksize when extending file without
+ * need block allocation
+ */
+ if (ext4_should_order_data(inode))
+ ret = ext4_jbd2_file_inode(handle, inode);
+
+ EXT4_I(inode)->i_disksize = new_i_size;
+ }
+ ret2 = generic_write_end(file, mapping, pos,
+ len, copied, page, fsdata);
+ copied = ret2;
+ if (ret2 < 0)
+ ret = ret2;
+ }
+ ret2 = ext4_journal_stop(handle);
+ if (!ret)
+ ret = ret2;
+
+ return ret ? ret : copied;
+}
+
static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
{
/*
@@ -2250,7 +2312,7 @@ static int ext4_journalled_set_page_dirty(struct page *page)
.writepages = ext4_da_writepages,
.sync_page = block_sync_page,
.write_begin = ext4_da_write_begin,
- .write_end = generic_write_end,
+ .write_end = ext4_da_write_end,
.bmap = ext4_bmap,
.invalidatepage = ext4_da_invalidatepage,
.releasepage = ext4_releasepage,

2008-06-24 03:34:11

[permalink] [raw]

Subject: Re: Performance of ext4

On Tue, Jun 24, 2008 at 08:37:21AM +0530, Aneesh Kumar K.V wrote:
> On Mon, Jun 23, 2008 at 05:31:32PM -0700, Mingming wrote:
> >
> > On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:
> >
> > > I found one place where we fail to update i_disksize. Can you try this
> > > patch ?
> > >
> > > diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> > > index 33f940b..9fa737f 100644
> > > --- a/fs/ext4/inode.c
> > > +++ b/fs/ext4/inode.c
> > > @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
> > > loff_t size;
> > > unsigned long len;
> > > handle_t *handle = NULL;
> > > + ext4_lblk_t block;
> > > + loff_t disksize;
> > > struct buffer_head *page_bufs;
> > > + struct buffer_head *bh, *head;
> > > struct inode *inode = page->mapping->host;
> > >
> > > handle = ext4_journal_current_handle();
> > > @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
> > > else
> > > ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
> > >
> > > + if (ret)
> > > + return ret;
> > > + /*
> > > + * When called via shrink_page_list and if we don't have any unmapped
> > > + * buffer_head we still could have written some new content in an
> > > + * already mapped buffer. That means we need to extent i_disksize here
> > > + */
> >
> > In this case(when extend the file without need block allocation),
> > wouldn't make sense to update the i_disksize at write_end() time? So
> > that the window of i_size different from i_disksize could be much
> > smaller in this case.
> >
> >
> > Something like below? (untested)
>
> In this case you will have to start a transaction in write_begin . With
> the below code transaction is started inside page_lock. Also I don't
> think we need needed_blocks credit just 1 should be enough because we
> are not doing any block allocation here. We just need to update the
> inode block.
>
>

diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 33f940b..bc925c5 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1770,6 +1770,7 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
struct page *page;
pgoff_t index;
unsigned from, to;
+ handle_t *handle;
struct inode *inode = mapping->host;

index = pos >> PAGE_CACHE_SHIFT;
@@ -1777,6 +1778,17 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
to = from + len;

retry:
+ /*
+ * If we are writing towards the end of an already mapped
+ * buffer_head, we don't do any block allocation. But we
+ * need to update i_disksize.
+ */
+ handle = ext4_journal_start(inode, 1);
+ if (IS_ERR(handle)) {
+ ret = PTR_ERR(handle);
+ goto out;
+ }
+
page = __grab_cache_page(mapping, index);
if (!page)
return -ENOMEM;
@@ -1786,15 +1798,63 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
ext4_da_get_block_prep);
if (ret < 0) {
unlock_page(page);
+ ext4_journal_stop(handle);
page_cache_release(page);
}

if (ret == -ENOSPC && ext4_should_retry_alloc(inode->i_sb, &retries))
goto retry;

+out:
return ret;
}

+static int ext4_da_write_end(struct file *file,
+ struct address_space *mapping,
+ loff_t pos, unsigned len, unsigned copied,
+ struct page *page, void *fsdata)
+{
+ loff_t new_i_size;
+ unsigned from, to;
+ int ret = 0, ret2;
+ struct inode *inode = mapping->host;
+ handle_t *handle = ext4_journal_current_handle();
+
+ from = pos & (PAGE_CACHE_SIZE - 1);
+ to = from + len;
+
+ /*
+ * generic_write_end() will run mark_inode_dirty() if i_size
+ * changes. So let's piggyback the i_disksize mark_inode_dirty
+ * into that.
+ */
+
+ new_i_size = pos + copied;
+ if (new_i_size > EXT4_I(inode)->i_disksize)
+ if (!walk_page_buffers(NULL, page_buffers(page),
+ 0, len, NULL,
+ ext4_bh_unmapped_or_delay)) {
+ /*
+ * Updating i_disksize when extending file without
+ * need block allocation
+ */
+ if (ext4_should_order_data(inode))
+ ret = ext4_jbd2_file_inode(handle, inode);
+
+ EXT4_I(inode)->i_disksize = new_i_size;
+ }
+ ret2 = generic_write_end(file, mapping, pos,
+ len, copied, page, fsdata);
+ copied = ret2;
+ if (ret2 < 0)
+ ret = ret2;
+ ret2 = ext4_journal_stop(handle);
+ if (!ret)
+ ret = ret2;
+
+ return ret ? ret : copied;
+}
+
static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
{
/*
@@ -2250,7 +2310,7 @@ static int ext4_journalled_set_page_dirty(struct page *page)
.writepages = ext4_da_writepages,
.sync_page = block_sync_page,
.write_begin = ext4_da_write_begin,
- .write_end = generic_write_end,
+ .write_end = ext4_da_write_end,
.bmap = ext4_bmap,
.invalidatepage = ext4_da_invalidatepage,
.releasepage = ext4_releasepage,

2008-06-24 12:57:26

[permalink] [raw]

Subject: Re: Performance of ext4

On Mon, 23 Jun 2008, Aneesh Kumar K.V wrote:

> On Fri, Jun 20, 2008 at 09:21:48AM +0000, Holger Kiehl wrote:
>> On Fri, 20 Jun 2008, Theodore Tso wrote:
>>
>>> On Fri, Jun 20, 2008 at 08:32:52AM +0000, Holger Kiehl wrote:
>>>>> It sounds like i_size is actually dropping in
>>>>> size at some pointer long after the file was written. If I had to
>>>
>>> sorry, "at some point"...
>>>
>>>>> guess the value in the inode cache is correct; and perhaps so is the
>>>>> value on the journal. But somehow, the wrong value is getting written
>>>>> to disk
>>>
>>> Or, "the right value is never getting written to disk". (Which as I
>>> think about it is more likely; it's likely that an update to i_size is
>>> getting *lost*, perhaps because the delalloc code is possibly
>>> modifying i_size without starting a transaction first. Again this is
>>> just a guess.)
>>>
>>>> What I find strange is that the missing parts of the file are not for
>>>> example exactly 512 or 1024 or 4096 bytes it is mostly some odd number
>>>> of bytes.
>>>
>>> Is there any chance the truncation point is related to how the program
>>> is writing its output file? i.e., if it is a text file, is the
>>> truncation happening after a new-line or when the stdio library might
>>> have done an explicit or implicit fflush()?
>>>
>> When the benchmark runs it writes to stdout and with tee to the result
>> file. It first writes some information about the system, prepares the
>> test files (creates lots of small files), calls sync and then starts
>> the test. Then every minute one line gets written to the result file.
>> Often I have seen that everything after the sync was missing. But
>> sometimes it happened that some parts at the end are missing. But it
>> was always a clean cut, that is there where no lines that where cut
>> partially. The lines where always complete.
>>
>
> I found one place where we fail to update i_disksize. Can you try this
> patch ?
>
Yes, I would like to however when I take
ext4-patch-queue-70acdb9605784bd5c4b06e1a19761828a494a337.tar.gz (which is
the current ext4-patch-queue from http://repo.or.cz/w/ext4-patch-queue.git)
and apply those to linux-2.6.26-rc6 I get the following reject:

***************
*** 574,579 ****
INIT_LIST_HEAD(&ei->i_prealloc_list);
spin_lock_init(&ei->i_prealloc_lock);
jbd2_journal_init_jbd_inode(&ei->jinode, &ei->vfs_inode);
return &ei->vfs_inode;
}

--- 574,584 ----
INIT_LIST_HEAD(&ei->i_prealloc_list);
spin_lock_init(&ei->i_prealloc_lock);
jbd2_journal_init_jbd_inode(&ei->jinode, &ei->vfs_inode);
+ ei->i_reserved_data_blocks = 0;
+ ei->i_reserved_meta_blocks = 0;
+ ei->i_allocated_meta_blocks = 0;
+ ei->i_delalloc_reserved_flag = 0;
+ spin_lock_init(&(ei->i_block_reservation_lock));
return &ei->vfs_inode;
}

Which is from delalloc-ext4-ENOSPC-handling.patch. What am I doing wrong?
I could apply this by hand but I do not know if this would be correct.
Please can anyone advice what I need to do?

Thanks,
Holger

2008-06-24 17:58:27

[permalink] [raw]

Subject: Re: Performance of ext4

On Tue, 2008-06-24 at 08:37 +0530, Aneesh Kumar K.V wrote:
> On Mon, Jun 23, 2008 at 05:31:32PM -0700, Mingming wrote:
> >
> > On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:
> >
> > > I found one place where we fail to update i_disksize. Can you try this
> > > patch ?
> > >
> > > diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> > > index 33f940b..9fa737f 100644
> > > --- a/fs/ext4/inode.c
> > > +++ b/fs/ext4/inode.c
> > > @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
> > > loff_t size;
> > > unsigned long len;
> > > handle_t *handle = NULL;
> > > + ext4_lblk_t block;
> > > + loff_t disksize;
> > > struct buffer_head *page_bufs;
> > > + struct buffer_head *bh, *head;
> > > struct inode *inode = page->mapping->host;
> > >
> > > handle = ext4_journal_current_handle();
> > > @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
> > > else
> > > ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
> > >
> > > + if (ret)
> > > + return ret;
> > > + /*
> > > + * When called via shrink_page_list and if we don't have any unmapped
> > > + * buffer_head we still could have written some new content in an
> > > + * already mapped buffer. That means we need to extent i_disksize here
> > > + */
> >
> > In this case(when extend the file without need block allocation),
> > wouldn't make sense to update the i_disksize at write_end() time? So
> > that the window of i_size different from i_disksize could be much
> > smaller in this case.
> >
> >
> > Something like below? (untested)
>
> In this case you will have to start a transaction in write_begin . With
> the below code transaction is started inside page_lock. Also I don't
> think we need needed_blocks credit just 1 should be enough because we
> are not doing any block allocation here. We just need to update the
> inode block.
>
>

Right! I will update the patch and merge it to the patch queue.

Mingming
> -aneesh
>
> >
> > Signed-off-by: Mingming Cao <[email protected]>
> > ---
> > fs/ext4/inode.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++------
> > 1 file changed, 60 insertions(+), 6 deletions(-)
> >
> > Index: linux-2.6.26-rc6/fs/ext4/inode.c
> > ===================================================================
> > --- linux-2.6.26-rc6.orig/fs/ext4/inode.c 2008-06-23 17:28:01.000000000 -0700
> > +++ linux-2.6.26-rc6/fs/ext4/inode.c 2008-06-23 17:28:10.000000000 -0700
> > @@ -1573,6 +1573,65 @@ static int ext4_da_write_begin(struct fi
> > return ret;
> > }
> >
> > +static int ext4_bh_unmapped_or_delay(handle_t *handle, struct buffer_head *bh)
> > +{
> > + return !buffer_mapped(bh) || buffer_delay(bh);
> > +}
> > +
> > +static int ext4_da_write_end(struct file *file,
> > + struct address_space *mapping,
> > + loff_t pos, unsigned len, unsigned copied,
> > + struct page *page, void *fsdata)
> > +{
> > + handle_t *handle = NULL;
> > + struct inode *inode = mapping->host;
> > + int needed_blocks = ext4_writepage_trans_blocks(inode);
> > + unsigned from, to;
> > + int ret = 0, ret2;
> > +
> > + from = pos & (PAGE_CACHE_SIZE - 1);
> > + to = from + len;
> > +
> > + if (ret == 0) {
> > + /*
> > + * generic_write_end() will run mark_inode_dirty() if i_size
> > + * changes. So let's piggyback the i_disksize mark_inode_dirty
> > + * into that.
> > + */
> > + loff_t new_i_size;
> > +
> > + new_i_size = pos + copied;
> > + if (new_i_size > EXT4_I(inode)->i_disksize)
> > + if (!walk_page_buffers(NULL, page_buffers(page),
> > + 0, len, NULL, ext4_bh_unmapped_or_delay)){
> > + /*
> > + * Updating i_disksize when extending file without
> > + * need block allocation
> > + */
> > + handle = ext4_journal_start(inode, needed_blocks);
> > + if (IS_ERR(handle)) {
> > + ret = PTR_ERR(handle);
> > + return ret;
> > + }
> > + if (ext4_should_order_data(inode))
> > + ret = ext4_jbd2_file_inode(handle, inode);
> > +
> > + EXT4_I(inode)->i_disksize = new_i_size;
> > + }
> > + ret2 = generic_write_end(file, mapping, pos, len, copied,
> > + page, fsdata);
> > + copied = ret2;
> > + if (ret2 < 0)
> > + ret = ret2;
> > + }
> > + if (handle)
> > + ret2 = ext4_journal_stop(handle);
> > + if (!ret)
> > + ret = ret2;
> > +
> > + return ret ? ret : copied;
> > +}
> > +
> > static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
> > {
> > struct buffer_head *head, *bh;
> > @@ -1682,11 +1741,6 @@ static int bput_one(handle_t *handle, st
> > return 0;
> > }
> >
> > -static int ext4_bh_unmapped_or_delay(handle_t *handle, struct buffer_head *bh)
> > -{
> > - return !buffer_mapped(bh) || buffer_delay(bh);
> > -}
> > -
> > /*
> > * Note that we don't need to start a transaction unless we're journaling data
> > * because we should have holes filled from ext4_page_mkwrite(). We even don't
> > @@ -2050,7 +2104,7 @@ static const struct address_space_operat
> > .writepages = ext4_da_writepages,
> > .sync_page = block_sync_page,
> > .write_begin = ext4_da_write_begin,
> > - .write_end = generic_write_end,
> > + .write_end = ext4_da_write_end,
> > .bmap = ext4_bmap,
> > .invalidatepage = ext4_da_invalidatepage,
> > .releasepage = ext4_releasepage,
> >
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2008-06-24 21:12:19

[permalink] [raw]

Subject: Re: Performance of ext4

On Tue, 24 Jun 2008, Aneesh Kumar K.V wrote:

> On Tue, Jun 24, 2008 at 08:37:21AM +0530, Aneesh Kumar K.V wrote:
>> On Mon, Jun 23, 2008 at 05:31:32PM -0700, Mingming wrote:
>>>
>>> On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:
>>>
>>>> I found one place where we fail to update i_disksize. Can you try this
>>>> patch ?
>>>>
>>>> diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
>>>> index 33f940b..9fa737f 100644
>>>> --- a/fs/ext4/inode.c
>>>> +++ b/fs/ext4/inode.c
>>>> @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
>>>> loff_t size;
>>>> unsigned long len;
>>>> handle_t *handle = NULL;
>>>> + ext4_lblk_t block;
>>>> + loff_t disksize;
>>>> struct buffer_head *page_bufs;
>>>> + struct buffer_head *bh, *head;
>>>> struct inode *inode = page->mapping->host;
>>>>
>>>> handle = ext4_journal_current_handle();
>>>> @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
>>>> else
>>>> ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
>>>>
>>>> + if (ret)
>>>> + return ret;
>>>> + /*
>>>> + * When called via shrink_page_list and if we don't have any unmapped
>>>> + * buffer_head we still could have written some new content in an
>>>> + * already mapped buffer. That means we need to extent i_disksize here
>>>> + */
>>>
>>> In this case(when extend the file without need block allocation),
>>> wouldn't make sense to update the i_disksize at write_end() time? So
>>> that the window of i_size different from i_disksize could be much
>>> smaller in this case.
>>>
>>>
>>> Something like below? (untested)
>>
>> In this case you will have to start a transaction in write_begin . With
>> the below code transaction is started inside page_lock. Also I don't
>> think we need needed_blocks credit just 1 should be enough because we
>> are not doing any block allocation here. We just need to update the
>> inode block.
>>
>>
>
> diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> index 33f940b..bc925c5 100644
> --- a/fs/ext4/inode.c
> +++ b/fs/ext4/inode.c
> @@ -1770,6 +1770,7 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
> struct page *page;
> pgoff_t index;
> unsigned from, to;
> + handle_t *handle;
> struct inode *inode = mapping->host;
>
> index = pos >> PAGE_CACHE_SHIFT;
> @@ -1777,6 +1778,17 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
> to = from + len;
>
> retry:
> + /*
> + * If we are writing towards the end of an already mapped
> + * buffer_head, we don't do any block allocation. But we
> + * need to update i_disksize.
> + */
> + handle = ext4_journal_start(inode, 1);
> + if (IS_ERR(handle)) {
> + ret = PTR_ERR(handle);
> + goto out;
> + }
> +
> page = __grab_cache_page(mapping, index);
> if (!page)
> return -ENOMEM;
> @@ -1786,15 +1798,63 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
> ext4_da_get_block_prep);
> if (ret < 0) {
> unlock_page(page);
> + ext4_journal_stop(handle);
> page_cache_release(page);
> }
>
> if (ret == -ENOSPC && ext4_should_retry_alloc(inode->i_sb, &retries))
> goto retry;
>
> +out:
> return ret;
> }
>
> +static int ext4_da_write_end(struct file *file,
> + struct address_space *mapping,
> + loff_t pos, unsigned len, unsigned copied,
> + struct page *page, void *fsdata)
> +{
> + loff_t new_i_size;
> + unsigned from, to;
> + int ret = 0, ret2;
> + struct inode *inode = mapping->host;
> + handle_t *handle = ext4_journal_current_handle();
> +
> + from = pos & (PAGE_CACHE_SIZE - 1);
> + to = from + len;
> +
> + /*
> + * generic_write_end() will run mark_inode_dirty() if i_size
> + * changes. So let's piggyback the i_disksize mark_inode_dirty
> + * into that.
> + */
> +
> + new_i_size = pos + copied;
> + if (new_i_size > EXT4_I(inode)->i_disksize)
> + if (!walk_page_buffers(NULL, page_buffers(page),
> + 0, len, NULL,
> + ext4_bh_unmapped_or_delay)) {
> + /*
> + * Updating i_disksize when extending file without
> + * need block allocation
> + */
> + if (ext4_should_order_data(inode))
> + ret = ext4_jbd2_file_inode(handle, inode);
> +
> + EXT4_I(inode)->i_disksize = new_i_size;
> + }
> + ret2 = generic_write_end(file, mapping, pos,
> + len, copied, page, fsdata);
> + copied = ret2;
> + if (ret2 < 0)
> + ret = ret2;
> + ret2 = ext4_journal_stop(handle);
> + if (!ret)
> + ret = ret2;
> +
> + return ret ? ret : copied;
> +}
> +
> static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
> {
> /*
> @@ -2250,7 +2310,7 @@ static int ext4_journalled_set_page_dirty(struct page *page)
> .writepages = ext4_da_writepages,
> .sync_page = block_sync_page,
> .write_begin = ext4_da_write_begin,
> - .write_end = generic_write_end,
> + .write_end = ext4_da_write_end,
> .bmap = ext4_bmap,
> .invalidatepage = ext4_da_invalidatepage,
> .releasepage = ext4_releasepage,
>
Yes, with this patch applied on top of latest patch queue I no longer
get truncated files, after running a short test. Tomorrow I will do some
more thorough testing and use the patch you have send to me in a separate
mail. The above patch did not apply but it was easy to apply by hand.

Thanks a lot for the patch!

Holger

2008-06-24 22:58:05

[permalink] [raw]

Subject: Re: Performance of ext4

On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
> On Tue, 24 Jun 2008, Aneesh Kumar K.V wrote:
>
> > On Tue, Jun 24, 2008 at 08:37:21AM +0530, Aneesh Kumar K.V wrote:
> >> On Mon, Jun 23, 2008 at 05:31:32PM -0700, Mingming wrote:
> >>>
> >>> On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:
> >>>
> >>>> I found one place where we fail to update i_disksize. Can you try this
> >>>> patch ?
> >>>>
> >>>> diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> >>>> index 33f940b..9fa737f 100644
> >>>> --- a/fs/ext4/inode.c
> >>>> +++ b/fs/ext4/inode.c
> >>>> @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
> >>>> loff_t size;
> >>>> unsigned long len;
> >>>> handle_t *handle = NULL;
> >>>> + ext4_lblk_t block;
> >>>> + loff_t disksize;
> >>>> struct buffer_head *page_bufs;
> >>>> + struct buffer_head *bh, *head;
> >>>> struct inode *inode = page->mapping->host;
> >>>>
> >>>> handle = ext4_journal_current_handle();
> >>>> @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
> >>>> else
> >>>> ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
> >>>>
> >>>> + if (ret)
> >>>> + return ret;
> >>>> + /*
> >>>> + * When called via shrink_page_list and if we don't have any unmapped
> >>>> + * buffer_head we still could have written some new content in an
> >>>> + * already mapped buffer. That means we need to extent i_disksize here
> >>>> + */
> >>>
> >>> In this case(when extend the file without need block allocation),
> >>> wouldn't make sense to update the i_disksize at write_end() time? So
> >>> that the window of i_size different from i_disksize could be much
> >>> smaller in this case.
> >>>
> >>>
> >>> Something like below? (untested)
> >>
> >> In this case you will have to start a transaction in write_begin . With
> >> the below code transaction is started inside page_lock. Also I don't
> >> think we need needed_blocks credit just 1 should be enough because we
> >> are not doing any block allocation here. We just need to update the
> >> inode block.
> >>
> >>
> >
> > diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> > index 33f940b..bc925c5 100644
> > --- a/fs/ext4/inode.c
> > +++ b/fs/ext4/inode.c
> > @@ -1770,6 +1770,7 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
> > struct page *page;
> > pgoff_t index;
> > unsigned from, to;
> > + handle_t *handle;
> > struct inode *inode = mapping->host;
> >
> > index = pos >> PAGE_CACHE_SHIFT;
> > @@ -1777,6 +1778,17 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
> > to = from + len;
> >
> > retry:
> > + /*
> > + * If we are writing towards the end of an already mapped
> > + * buffer_head, we don't do any block allocation. But we
> > + * need to update i_disksize.
> > + */
> > + handle = ext4_journal_start(inode, 1);
> > + if (IS_ERR(handle)) {
> > + ret = PTR_ERR(handle);
> > + goto out;
> > + }
> > +
> > page = __grab_cache_page(mapping, index);
> > if (!page)
> > return -ENOMEM;
> > @@ -1786,15 +1798,63 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
> > ext4_da_get_block_prep);
> > if (ret < 0) {
> > unlock_page(page);
> > + ext4_journal_stop(handle);
> > page_cache_release(page);
> > }
> >
> > if (ret == -ENOSPC && ext4_should_retry_alloc(inode->i_sb, &retries))
> > goto retry;
> >
> > +out:
> > return ret;
> > }
> >
> > +static int ext4_da_write_end(struct file *file,
> > + struct address_space *mapping,
> > + loff_t pos, unsigned len, unsigned copied,
> > + struct page *page, void *fsdata)
> > +{
> > + loff_t new_i_size;
> > + unsigned from, to;
> > + int ret = 0, ret2;
> > + struct inode *inode = mapping->host;
> > + handle_t *handle = ext4_journal_current_handle();
> > +
> > + from = pos & (PAGE_CACHE_SIZE - 1);
> > + to = from + len;
> > +
> > + /*
> > + * generic_write_end() will run mark_inode_dirty() if i_size
> > + * changes. So let's piggyback the i_disksize mark_inode_dirty
> > + * into that.
> > + */
> > +
> > + new_i_size = pos + copied;
> > + if (new_i_size > EXT4_I(inode)->i_disksize)
> > + if (!walk_page_buffers(NULL, page_buffers(page),
> > + 0, len, NULL,
> > + ext4_bh_unmapped_or_delay)) {
> > + /*
> > + * Updating i_disksize when extending file without
> > + * need block allocation
> > + */
> > + if (ext4_should_order_data(inode))
> > + ret = ext4_jbd2_file_inode(handle, inode);
> > +
> > + EXT4_I(inode)->i_disksize = new_i_size;
> > + }
> > + ret2 = generic_write_end(file, mapping, pos,
> > + len, copied, page, fsdata);
> > + copied = ret2;
> > + if (ret2 < 0)
> > + ret = ret2;
> > + ret2 = ext4_journal_stop(handle);
> > + if (!ret)
> > + ret = ret2;
> > +
> > + return ret ? ret : copied;
> > +}
> > +
> > static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
> > {
> > /*
> > @@ -2250,7 +2310,7 @@ static int ext4_journalled_set_page_dirty(struct page *page)
> > .writepages = ext4_da_writepages,
> > .sync_page = block_sync_page,
> > .write_begin = ext4_da_write_begin,
> > - .write_end = generic_write_end,
> > + .write_end = ext4_da_write_end,
> > .bmap = ext4_bmap,
> > .invalidatepage = ext4_da_invalidatepage,
> > .releasepage = ext4_releasepage,
> >
> Yes, with this patch applied on top of latest patch queue I no longer
> get truncated files, after running a short test. Tomorrow I will do some
> more thorough testing and use the patch you have send to me in a separate
> mail. The above patch did not apply but it was easy to apply by hand.

Thanks for quick response and test. I have updated the patch queue with
above patch merged. Please let me know if you still see apply issue and
file size update issue with current patch queue.

Regards,
Mingming

2008-06-25 09:09:29

[permalink] [raw]

Subject: Re: Performance of ext4

On Tue, 24 Jun 2008, Mingming wrote:

>
> On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
>> Yes, with this patch applied on top of latest patch queue I no longer
>> get truncated files, after running a short test. Tomorrow I will do some
>> more thorough testing and use the patch you have send to me in a separate
>> mail. The above patch did not apply but it was easy to apply by hand.
>
>
> Thanks for quick response and test. I have updated the patch queue with
> above patch merged. Please let me know if you still see apply issue and
> file size update issue with current patch queue.
>
Thanks, it applies without any problems. However I still hit an oops. What
I find strange is that I got the oops just as the benchmark is done and
all process where shutting down. The same behaviour I reported here:
http://www.ussg.iu.edu/hypermail/linux/kernel/0806.2/2113.html
Only this time I got just one oops. This is on x86_64 system (4 Opteron CPU's
and SW Raid 1+0). I have not seen this on my home system x86 (1 Dual Core
and HW Raid). Anyway, here the dmesg output:

kjournald2 starting. Commit interval 15 seconds
EXT4 FS on md7, internal journal
EXT4-fs: mounted filesystem with ordered data mode.
EXT4-fs: file extents enabled
EXT4-fs: mballoc enabled
JBD: barrier-based sync failed on md7 - disabling barriers
------------[ cut here ]------------
kernel BUG at fs/ext4/inode.c:1667!
invalid opcode: 0000 [1] SMP
CPU 3
Modules linked in: w83627hf lm85 hwmon_vid bonding nf_conntrack_ftp ipt_REJECT xt_tcpudp nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables binfmt_misc sg floppy k8temp ohci_hcd i2c_amd756 i2c_core button usbcore
Pid: 2661, comm: kjournald2 Not tainted 2.6.26-rc6 #1
RIP: 0010:[<ffffffff802dd314>] [<ffffffff802dd314>] ext4_da_writepage+0x30/0xf4
RSP: 0018:ffff81027d737c10 EFLAGS: 00010246
RAX: 010000000001002d RBX: ffffe200017e9218 RCX: 00000000ffffffe8
RDX: 0000000000004278 RSI: ffff81027d737e00 RDI: ffffe200017e9218
RBP: ffff81021dad66e8 R08: ffff81027ff05440 R09: ffff81027d737b50
R10: ffff81020fe54480 R11: ffff81020fe54400 R12: ffff81027d737e00
R13: 0000000000000001 R14: 0000000000000000 R15: ffff8101ff0b8120
FS: 00007f53ed10b6f0(0000) GS:ffff81027ff13000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 000000000081a000 CR3: 0000000166f03000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kjournald2 (pid: 2661, threadinfo ffff81027d736000, task ffff81027ed93210)
Stack: ffff81021dad67f8 ffff81027d737e00 0000000000000000 ffffffff80261816
ffffe200017e9218 ffffffff80261f8c ffff81021dad67f8 ffffffff8026180c
ffff81021dad67f8 0000000280224f0c 0000000000000004 000000007d737fd8
Call Trace:
[<ffffffff80261816>] ? __writepage+0xa/0x27
[<ffffffff80261f8c>] ? write_cache_pages+0x174/0x2be
[<ffffffff8026180c>] ? __writepage+0x0/0x27
[<ffffffff802ff1ed>] ? jbd2_journal_commit_transaction+0x343/0xe6a
[<ffffffff802411a8>] ? hrtimer_start+0x100/0x122
[<ffffffff802357ba>] ? try_to_del_timer_sync+0x52/0x5b
[<ffffffff80302972>] ? kjournald2+0xdf/0x202
[<ffffffff8023e74a>] ? autoremove_wake_function+0x0/0x2e
[<ffffffff80302893>] ? kjournald2+0x0/0x202
[<ffffffff8023e43b>] ? kthread+0x47/0x73
[<ffffffff8020bf78>] ? child_rip+0xa/0x12
[<ffffffff8023e3f4>] ? kthread+0x0/0x73
[<ffffffff8020bf6e>] ? child_rip+0x0/0x12

Code: 55 53 48 8b 47 18 48 89 fb 48 8b 28 65 48 8b 04 25 00 00 00 00 48 83 b8 d0 04 00 00 00 75 5f 48 8b 07 48 8b 55 68 f6 c4 08 75 04 <0f> 0b eb fe 48 89 d0 81 e2 ff 0f 00 00 48 8b 77 10 48 c1 f8 0c
RIP [<ffffffff802dd314>] ext4_da_writepage+0x30/0xf4
RSP <ffff81027d737c10>
---[ end trace efaf3891d582feb1 ]---
SysRq : Show Blocked State
task PC stack pid father
pdflush D 00000000ffffffff 0 242 2
ffff81007f319bb8 0000000000000046 ffffffff8031682e 000000000000000e
ffffffff80600cc0 000000000000000e ffff81007f319bf0 ffff81007fbbbd30
ffff8101fff33210 ffff81007fbbbf68 00000002802aa2a6 ffff81007fbbbf68
Call Trace:
[<ffffffff8031682e>] submit_bio+0xc4/0xcb
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff802fde7f>] jbd2_journal_stop+0x191/0x19d
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dd7e8>] ext4_da_writepages+0x75/0x173
[<ffffffff80262112>] do_writepages+0x20/0x2d
[<ffffffff8029f6da>] __writeback_single_inode+0x16f/0x35e
[<ffffffff8029fcf7>] sync_sb_inodes+0x283/0x3b8
[<ffffffff802a0080>] writeback_inodes+0x89/0xde
[<ffffffff80262252>] wb_kupdate+0x9f/0x116
[<ffffffff80262ca0>] pdflush+0x122/0x1c9
[<ffffffff802621b3>] wb_kupdate+0x0/0x116
[<ffffffff80262b7e>] pdflush+0x0/0x1c9
[<ffffffff8023e43b>] kthread+0x47/0x73
[<ffffffff8022ae6a>] schedule_tail+0x27/0x5c
[<ffffffff8020bf78>] child_rip+0xa/0x12
[<ffffffff8023e3f4>] kthread+0x0/0x73
[<ffffffff8020bf6e>] child_rip+0x0/0x12

init_afd D 00000000ffffffff 0 7974 1
ffff8101050d3b30 0000000000000082 ffff810055f08230 0000000000080768
ffffffff80600cc0 ffff8101ff8509c0 ffffffff802a38d5 ffff8101050aac80
ffff8101fff342c0 ffff8101050aaeb8 000000038023e66b ffff8101050aaeb8
Call Trace:
[<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
[<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff80224a97>] __wake_up+0x38/0x4f
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff80269428>] do_wp_page+0x45b/0x4ba
[<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
[<ffffffff8022a72f>] hrtick_set+0x8b/0x10a
[<ffffffff8021ed0c>] do_page_fault+0x435/0x825
[<ffffffff80242fdc>] getnstimeofday+0x38/0x92
[<ffffffff80240fcb>] lock_hrtimer_base+0x1b/0x3c
[<ffffffff80241089>] hrtimer_try_to_cancel+0x61/0x6a
[<ffffffff8024109e>] hrtimer_cancel+0xc/0x16
[<ffffffff8046687e>] do_nanosleep+0x6c/0xa6
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff80467779>] error_exit+0x0/0x51

system_log D 00000000ffffffff 0 7975 7974
ffff81002d533ad0 0000000000000086 ffffffff8028fce8 ffff81007ed55e90
ffffffff80600cc0 0000000000000100 ffff81027d7cae40 ffff81007ed55e90
ffff8101fff342c0 ffff81007ed560c8 000000032d533f50 ffff81007ed560c8
Call Trace:
[<ffffffff8028fce8>] poll_freewait+0x36/0x87
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
[<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80295a76>] touch_atime+0x12/0xfa
[<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8026f499>] mmap_region+0x41a/0x505
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

receive_log D 00000000ffffffff 0 7977 7974
ffff81002d4ffad0 0000000000000082 ffffffff8028fce8 ffff81007ed55900
ffffffff80600cc0 0000000000000100 ffff81007e86ed80 ffff81007ed55900
ffffffff8054d350 ffff81007ed55b38 000000002d4fff50 ffff81007ed55b38
Call Trace:
[<ffffffff8028fce8>] poll_freewait+0x36/0x87
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff802ec6a2>] __ext4_journal_dirty_metadata+0x1e/0x46
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff8028ad96>] do_lookup+0x63/0x1a1
[<ffffffff80293dd5>] dput+0x21/0x10a
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

archive_watch D 00000000ffffffff 0 7980 7974
ffff81007ecabce0 0000000000000082 ffffffff802de7cd ffff810000000001
ffffffff80600cc0 ffff810000000001 ffffffff8027ce42 ffff81007ed537a0
ffff8101fff33210 ffff81007ed539d8 00000002802d9b7a ffff81007ed539d8
Call Trace:
[<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
[<ffffffff8027ce42>] get_partial_node+0x15/0x78
[<ffffffff80294b15>] __d_lookup+0xbd/0x107
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8028c917>] __link_path_walk+0x6e0/0xdd4
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802e3350>] ext4_unlink+0x3f/0x1fe
[<ffffffff8028b364>] may_delete+0x5e/0x135
[<ffffffff8028b9af>] vfs_unlink+0x5e/0xa3
[<ffffffff8028d985>] do_unlinkat+0xc1/0x142
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8020b26e>] tracesys+0x71/0xda
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

amg D ffff81007e8b3cf8 0 7986 7974
ffff81002d50bb60 0000000000000086 0000000000000000 0000000000000100
ffffffff80600cc0 0000000000000000 0000000000000000 ffff81007ed50000
ffff8101050acde0 ffff81007ed50238 0000000100000000 ffff81007ed50238
Call Trace:
[<ffffffff80224f1d>] default_wake_function+0x0/0xe
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
[<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
[<ffffffff8028d0ba>] path_walk+0xaf/0xb9
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check D ffff81007e8b1690 0 7988 7986
ffff81002d625d70 0000000000000082 ffff81002d625d38 ffff81007edfcb00
ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81007ed542c0
ffff81007ed574d0 ffff81007ed544f8 000000028028c917 ffff81007ed544f8
Call Trace:
[<ffffffff80293dd5>] dput+0x21/0x10a
[<ffffffff8028cee0>] __link_path_walk+0xca9/0xdd4
[<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff8028ac53>] lock_rename+0x35/0xc5
[<ffffffff8028d545>] do_rename+0x98/0x1b2
[<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

afd_stat D 00000000ffffffff 0 7989 7974
ffff81002d461b30 0000000000000082 ffff81002d461da8 ffff81002d461da8
ffffffff80600cc0 ffff81002d461da8 ffffffff802a38d5 ffff81007ed57a60
ffffffff8054d350 ffff81007ed57c98 000000008023e66b ffff81007ed57c98
Call Trace:
[<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
[<ffffffff80290dfd>] __pollwait+0x0/0xe3
[<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
[<ffffffff80242fdc>] getnstimeofday+0x38/0x92
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff80262dda>] pdflush_operation+0x93/0x9d
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff80269428>] do_wp_page+0x45b/0x4ba
[<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
[<ffffffff8022a745>] hrtick_set+0xa1/0x10a
[<ffffffff8021ed0c>] do_page_fault+0x435/0x825
[<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
[<ffffffff802411a8>] hrtimer_start+0x100/0x122
[<ffffffff802258bb>] hrtick_start_fair+0x139/0x18a
[<ffffffff8022a745>] hrtick_set+0xa1/0x10a
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff80467779>] error_exit+0x0/0x51

init_afd D ffff81007e8b3cf8 0 7997 1
ffff81027ee41b30 0000000000000086 ffff81027ee41dd0 ffff81027ee41da8
ffffffff80600cc0 ffff81027ee41db8 0000000000000010 ffff81027ecca160
ffff8101bbec1640 ffff81027ecca398 0000000000000000 ffff81027ecca398
Call Trace:
[<ffffffff80290dfd>] __pollwait+0x0/0xe3
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff80269428>] do_wp_page+0x45b/0x4ba
[<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
[<ffffffff8021ed0c>] do_page_fault+0x435/0x825
[<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
[<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff80467779>] error_exit+0x0/0x51

transfer_log D 00000000ffffffff 0 8001 7997
ffff81002d681ad0 0000000000000082 ffffffff8028fce8 ffff81007fb8f4d0
ffffffff80600cc0 0000000000000100 ffff8101b1362540 ffff81007fb8f4d0
ffff8101fff33210 ffff81007fb8f708 000000022d681f50 ffff81007fb8f708
Call Trace:
[<ffffffff8028fce8>] poll_freewait+0x36/0x87
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80295a76>] touch_atime+0x12/0xfa
[<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

archive_watch D 00000000ffffffff 0 8003 7997
ffff81002d49dce0 0000000000000086 ffffffff802de7cd ffff810000000001
ffffffff80600cc0 ffff810000000001 ffffffff8027ce42 ffff81007fb8ef40
ffff8101fff32160 ffff81007fb8f178 00000001802d9b7a ffff81007fb8f178
Call Trace:
[<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
[<ffffffff8027ce42>] get_partial_node+0x15/0x78
[<ffffffff80294b15>] __d_lookup+0xbd/0x107
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8028c917>] __link_path_walk+0x6e0/0xdd4
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802e3350>] ext4_unlink+0x3f/0x1fe
[<ffffffff8028b364>] may_delete+0x5e/0x135
[<ffffffff8028b9af>] vfs_unlink+0x5e/0xa3
[<ffffffff8028d985>] do_unlinkat+0xc1/0x142
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8020b26e>] tracesys+0x71/0xda
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

input_log D 00000000ffffffff 0 8004 7997
ffff81002d419ad0 0000000000000086 ffffffff8028fce8 ffff81007fb8de90
ffffffff80600cc0 0000000000000100 ffff81027f76aa80 ffff81007fb8de90
ffff8101fff32160 ffff81007fb8e0c8 000000012d419f50 ffff81007fb8e0c8
Call Trace:
[<ffffffff8028fce8>] poll_freewait+0x36/0x87
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
[<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80295a76>] touch_atime+0x12/0xfa
[<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

output_log D ffff81007e8b3cf8 0 8005 7997
ffff81002d4fdad0 0000000000000086 ffffffff8028fce8 ffff81007fbbc2c0
ffffffff80600cc0 0000000000000100 ffff81017ff649c0 ffff81007fbbc2c0
ffff81001ce46420 ffff81007fbbc4f8 000000012d4fdf50 ffff81007fbbc4f8
Call Trace:
[<ffffffff8028fce8>] poll_freewait+0x36/0x87
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
[<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80295a76>] touch_atime+0x12/0xfa
[<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

production_lo D ffff81007e8b3cf8 0 8007 7997
ffff81002d615ad0 0000000000000082 ffffffff8028fce8 ffff81007d939bd0
ffffffff80600cc0 0000000000000100 ffff81027f76a3c0 ffff81007d939bd0
ffff81007ed50590 ffff81007d939e08 000000012d615f50 ffff81007d939e08
Call Trace:
[<ffffffff8028fce8>] poll_freewait+0x36/0x87
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
[<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80295a76>] touch_atime+0x12/0xfa
[<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8026f499>] mmap_region+0x41a/0x505
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

distribution_ D 00000000ffffffff 0 8008 7997
ffff81002d53fad0 0000000000000086 ffffffff8028fce8 ffff81007d93ac80
ffffffff80600cc0 0000000000000100 ffff81007f300600 ffff81007d93ac80
ffff8101fff33210 ffff81007d93aeb8 000000022d53ff50 ffff81007d93aeb8
Call Trace:
[<ffffffff8028fce8>] poll_freewait+0x36/0x87
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80295a76>] touch_atime+0x12/0xfa
[<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff8022a745>] hrtick_set+0xa1/0x10a
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check D ffff81007e8b1690 0 8011 8009
ffff8101e1359d70 0000000000000082 ffff8101e1359d38 ffff81007edfcb00
ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff8101bbec10b0
ffff81007fb8de90 ffff8101bbec12e8 000000018028c917 ffff8101bbec12e8
Call Trace:
[<ffffffff80293dd5>] dput+0x21/0x10a
[<ffffffff8028cee0>] __link_path_walk+0xca9/0xdd4
[<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff8028ac53>] lock_rename+0x35/0xc5
[<ffffffff8028d545>] do_rename+0x98/0x1b2
[<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

afd_stat D ffff81007e8b3cf8 0 8012 7997
ffff81002d69bb30 0000000000000082 ffff81002d69bda8 ffff81002d69bda8
ffffffff80600cc0 ffff81002d69bda8 ffffffff802a38d5 ffff81007d93c2c0
ffff81027eccd900 ffff81007d93c4f8 000000008023e66b ffff81007d93c4f8
Call Trace:
[<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
[<ffffffff80290dfd>] __pollwait+0x0/0xe3
[<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff80224a97>] __wake_up+0x38/0x4f
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff80269428>] do_wp_page+0x45b/0x4ba
[<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
[<ffffffff8022a745>] hrtick_set+0xa1/0x10a
[<ffffffff8021ed0c>] do_page_fault+0x435/0x825
[<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
[<ffffffff802411a8>] hrtimer_start+0x100/0x122
[<ffffffff802258bb>] hrtick_start_fair+0x139/0x18a
[<ffffffff8022a745>] hrtick_set+0xa1/0x10a
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff80467779>] error_exit+0x0/0x51

init_afd D 00000000ffffffff 0 8020 1
ffff8101bbdd5b30 0000000000000086 ffff8101bbdd5dd0 ffff8101bbdd5da8
ffffffff80600cc0 ffff8101bbdd5db8 0000000000000010 ffff8101bbec1640
ffffffff8054d350 ffff8101bbec1878 0000000000000000 ffff8101bbec1878
Call Trace:
[<ffffffff80290dfd>] __pollwait+0x0/0xe3
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff80269428>] do_wp_page+0x45b/0x4ba
[<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
[<ffffffff8021ed0c>] do_page_fault+0x435/0x825
[<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
[<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff80467779>] error_exit+0x0/0x51

receive_log D 00000000ffffffff 0 8023 8020
ffff81027ee1bad0 0000000000000086 ffffffff8028fce8 ffff81027eccc2c0
ffffffff80600cc0 0000000000000100 ffff81007f300000 ffff81027eccc2c0
ffff8101fff33210 ffff81027eccc4f8 000000027ee1bf50 ffff81027eccc4f8
Call Trace:
[<ffffffff8028fce8>] poll_freewait+0x36/0x87
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff802ec6a2>] __ext4_journal_dirty_metadata+0x1e/0x46
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff8028ad96>] do_lookup+0x63/0x1a1
[<ffffffff80295a76>] touch_atime+0x12/0xfa
[<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

transfer_log D 00000000ffffffff 0 8024 8020
ffff81027f799ad0 0000000000000086 ffffffff8028fce8 ffff81027eccc850
ffffffff80600cc0 0000000000000100 ffff81027d7443c0 ffff81027eccc850
ffffffff8054d350 ffff81027eccca88 000000007f799f50 ffff81027eccca88
Call Trace:
[<ffffffff8028fce8>] poll_freewait+0x36/0x87
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
[<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80295a76>] touch_atime+0x12/0xfa
[<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff8022a745>] hrtick_set+0xa1/0x10a
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

archive_watch D 00000000ffffffff 0 8026 8020
ffff81027f7ddce0 0000000000000082 ffffffff802de7cd ffff810200000001
ffffffff80600cc0 ffff810000000001 ffffffff8027ce42 ffff81027ecc90b0
ffff8101fff33210 ffff81027ecc92e8 00000002802d9b7a ffff81027ecc92e8
Call Trace:
[<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
[<ffffffff8027ce42>] get_partial_node+0x15/0x78
[<ffffffff80294b15>] __d_lookup+0xbd/0x107
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8028c917>] __link_path_walk+0x6e0/0xdd4
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802e3350>] ext4_unlink+0x3f/0x1fe
[<ffffffff8028b364>] may_delete+0x5e/0x135
[<ffffffff8028b9af>] vfs_unlink+0x5e/0xa3
[<ffffffff8028d985>] do_unlinkat+0xc1/0x142
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8020b26e>] tracesys+0x71/0xda
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

input_log D ffff81007e8b3cf8 0 8027 8020
ffff81027eed1ad0 0000000000000082 ffffffff8028fce8 ffff81027ecca6f0
ffffffff80600cc0 0000000000000100 ffff81027d744e40 ffff81027ecca6f0
ffff81007fb8de90 ffff81027ecca928 000000017eed1f50 ffff81027ecca928
Call Trace:
[<ffffffff8028fce8>] poll_freewait+0x36/0x87
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80295a76>] touch_atime+0x12/0xfa
[<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff8022a745>] hrtick_set+0xa1/0x10a
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

output_log D ffff81007e8b3cf8 0 8028 8020
ffff81027d64dad0 0000000000000086 ffffffff8028fce8 ffff81027eccac80
ffffffff80600cc0 0000000000000100 ffff81010508c840 ffff81027eccac80
ffff81007ed574d0 ffff81027eccaeb8 000000027d64df50 ffff81027eccaeb8
Call Trace:
[<ffffffff8028fce8>] poll_freewait+0x36/0x87
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80295a76>] touch_atime+0x12/0xfa
[<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff8022a745>] hrtick_set+0xa1/0x10a
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

distribution_ D 00000000ffffffff 0 8031 8020
ffff81027ee1dad0 0000000000000086 ffffffff8028fce8 ffff81027ecc8590
ffffffff80600cc0 0000000000000100 ffff810105128a80 ffff81027ecc8590
ffff8101fff33210 ffff81027ecc87c8 000000027ee1df50 ffff81027ecc87c8
Call Trace:
[<ffffffff8028fce8>] poll_freewait+0x36/0x87
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
[<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80295a76>] touch_atime+0x12/0xfa
[<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check D 00000000ffffffff 0 8034 8032
ffff81027ec01d70 0000000000000082 ffff81027ec01d38 ffff81007edfcb00
ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81027eccb210
ffff8101fff33210 ffff81027eccb448 000000028028c917 ffff81027eccb448
Call Trace:
[<ffffffff80293dd5>] dput+0x21/0x10a
[<ffffffff8028cee0>] __link_path_walk+0xca9/0xdd4
[<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff8028ac53>] lock_rename+0x35/0xc5
[<ffffffff8028d545>] do_rename+0x98/0x1b2
[<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

afd_stat D ffff81007e8b3cf8 0 8035 8020
ffff81027edebb30 0000000000000082 ffff81027edebda8 ffff81027edebda8
ffffffff80600cc0 ffff81027edebda8 ffffffff802a38d5 ffff81027eccd900
ffff81007ed57a60 ffff81027eccdb38 000000008023e66b ffff81027eccdb38
Call Trace:
[<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
[<ffffffff80290dfd>] __pollwait+0x0/0xe3
[<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff80224a97>] __wake_up+0x38/0x4f
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff80269428>] do_wp_page+0x45b/0x4ba
[<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
[<ffffffff8022a745>] hrtick_set+0xa1/0x10a
[<ffffffff8021ed0c>] do_page_fault+0x435/0x825
[<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
[<ffffffff802411a8>] hrtimer_start+0x100/0x122
[<ffffffff802258bb>] hrtick_start_fair+0x139/0x18a
[<ffffffff8022a745>] hrtick_set+0xa1/0x10a
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff80467779>] error_exit+0x0/0x51

fd D ffff81007e8b3cf8 0 8037 8020
ffff81027ededb30 0000000000000086 ffff81027ededdd0 ffff81027ededda8
ffffffff80600cc0 ffff81027ededdb8 00000000000007e0 ffff81027eccde90
ffff81027ed937a0 ffff81027ecce0c8 0000000200000000 ffff81027ecce0c8
Call Trace:
[<ffffffff80290dfd>] __pollwait+0x0/0xe3
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff80269428>] do_wp_page+0x45b/0x4ba
[<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
[<ffffffff8021ed0c>] do_page_fault+0x435/0x825
[<ffffffff8022f67f>] release_task+0x2cb/0x2dd
[<ffffffff80230175>] do_wait+0xae4/0xb88
[<ffffffff80286e19>] vfs_stat_fd+0x1b/0x4a
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff80467779>] error_exit+0x0/0x51

afd D 00000000ffffffff 0 25970 2891
ffff81017fae7ad0 0000000000000086 ffffffff8028fce8 0000000000000246
ffffffff80600cc0 0000000000000100 ffff81020fe5dc00 ffff8101050ab210
ffff8101fff342c0 ffff8101050ab448 000000037fae7f50 ffff8101050ab448
Call Trace:
[<ffffffff8028fce8>] poll_freewait+0x36/0x87
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80268f85>] __do_fault+0x321/0x369
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff80237d9a>] group_send_sig_info+0x62/0x6f
[<ffffffff80237dd8>] kill_pid_info+0x31/0x3b
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D ffff81007e8b3cf8 0 26537 8037
ffff8101ff7b1b70 0000000000000086 ffffffff802411a8 000000000000148d
ffffffff80600cc0 0000000000000286 00000000ffffffff ffff8101bbec26f0
ffff81027eccb210 ffff8101bbec2928 00000002ffffffff ffff8101bbec2928
Call Trace:
[<ffffffff802411a8>] hrtimer_start+0x100/0x122
[<ffffffff8022a745>] hrtick_set+0xa1/0x10a
[<ffffffff80466148>] thread_return+0x63/0xbb
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802e2b08>] ext4_rename+0x4d/0x64a
[<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
[<ffffffff8028bc5c>] vfs_rename+0x1fd/0x376
[<ffffffff8028d5fe>] do_rename+0x151/0x1b2
[<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D ffff81007e8b1690 0 26543 8037
ffff8101ddec3d70 0000000000000086 ffff8101ddec3d38 ffff81007edfcb00
ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff8101bbec3210
ffff81007ed542c0 ffff8101bbec3448 000000028028c917 ffff8101bbec3448
Call Trace:
[<ffffffff80293dd5>] dput+0x21/0x10a
[<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
[<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff8028ac53>] lock_rename+0x35/0xc5
[<ffffffff8028d545>] do_rename+0x98/0x1b2
[<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D ffff81007e8b3cf8 0 26559 8037
ffff8101b11ddb60 0000000000000086 ffff8101bfc0b4d0 0000000000000202
ffffffff80600cc0 ffff8101bfc0b4d0 00007f1ea612f000 ffff8101bbec74d0
ffff81007ed574d0 ffff8101bbec7708 0000000200000000 ffff8101bbec7708
Call Trace:
[<ffffffff80224f1d>] default_wake_function+0x0/0xe
[<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff80467779>] error_exit+0x0/0x51
[<ffffffff8025bae5>] find_lock_page+0x29/0x87
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
[<ffffffff8026d535>] unmap_region+0x10f/0x125
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D 00000000ffffffff 0 26569 8013
ffff8101f40c1b60 0000000000000082 ffff81013b896b00 0000000000000206
ffffffff80600cc0 ffff81013b896b00 00007f31838c7000 ffff8101bbec6f40
ffff8101fff32160 ffff8101bbec7178 0000000100000000 ffff8101bbec7178
Call Trace:
[<ffffffff80224f1d>] default_wake_function+0x0/0xe
[<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff80467779>] error_exit+0x0/0x51
[<ffffffff8025bae5>] find_lock_page+0x29/0x87
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
[<ffffffff8026d535>] unmap_region+0x10f/0x125
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D ffff81007e8b3cf8 0 26583 8013
ffff8101bbf9fb60 0000000000000086 ffff81010c703160 0000000000000202
ffffffff80600cc0 ffff81010c703160 00007ffac6bd2000 ffff8101bbec6420
ffff81007d93fa60 ffff8101bbec6658 0000000100000000 ffff8101bbec6658
Call Trace:
[<ffffffff80224f1d>] default_wake_function+0x0/0xe
[<ffffffff80411b14>] dev_queue_xmit+0x21a/0x232
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff80467779>] error_exit+0x0/0x51
[<ffffffff8025bae5>] find_lock_page+0x29/0x87
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
[<ffffffff8026d535>] unmap_region+0x10f/0x125
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D ffff810208bf59f8 0 26589 26567
ffff810208bf59d8 0000000000000082 ffff810163233090 ffffffff802e9b05
ffffffff80600cc0 ffffffff00058011 ffff810208bf5af8 ffff81027ecce420
ffff81001ce40000 ffff81027ecce658 0000000208bf5b58 ffff81027ecce658
Call Trace:
[<ffffffff802e9b05>] ext4_ext_find_extent+0x67/0x26f
[<ffffffff80224a97>] __wake_up+0x38/0x4f
[<ffffffff803023b0>] jbd2_log_wait_commit+0xb6/0x108
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff80300c3c>] jbd2_log_do_checkpoint+0x12d/0x3bd
[<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
[<ffffffff802a3c4d>] __find_get_block+0x14c/0x15c
[<ffffffff802a3c7b>] __getblk+0x1e/0x28d
[<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
[<ffffffff802e6fb3>] ext4fs_dirhash+0x10d/0x1d2
[<ffffffff802e04cf>] dx_probe+0xa1/0x29e
[<ffffffff802e1b3a>] ext4_find_entry+0x293/0x592
[<ffffffff80300f53>] __jbd2_log_wait_for_space+0x87/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8029535a>] d_alloc+0x1f/0x189
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802e24bd>] ext4_create+0x4c/0x105
[<ffffffff802e27c3>] ext4_lookup+0x97/0xc1
[<ffffffff8028b805>] vfs_create+0x7d/0xed
[<ffffffff8028e3ac>] do_filp_open+0x1ed/0x832
[<ffffffff8028290c>] do_sys_open+0x46/0xc3
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D ffff81007e8b3cf8 0 26595 26584
ffff8101b1307ad0 0000000000000086 ffffffff8042530e ffff81014256f5d0
ffffffff80600cc0 ffff81017fa8fa00 ffffffff80242fdc ffff8101ff6d8b20
ffff81001ce46420 ffff8101ff6d8d58 000000018024153d ffff8101ff6d8d58
Call Trace:
[<ffffffff8042530e>] nf_iterate+0x41/0x7d
[<ffffffff80242fdc>] getnstimeofday+0x38/0x92
[<ffffffff8039d0b6>] loopback_xmit+0x80/0x86
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D ffff81007e8b3cf8 0 26602 26590
ffff81025dcf1c60 0000000000000082 ffffffff802e6fb3 0000000000000020
ffffffff80600cc0 ffff81014256f090 00000000000300aa ffff81027ed942c0
ffff81007d93e420 ffff81027ed944f8 000000017ae2b000 ffff81027ed944f8
Call Trace:
[<ffffffff802e6fb3>] ext4fs_dirhash+0x10d/0x1d2
[<ffffffff802e04cf>] dx_probe+0xa1/0x29e
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802e24bd>] ext4_create+0x4c/0x105
[<ffffffff802e27c3>] ext4_lookup+0x97/0xc1
[<ffffffff8028b805>] vfs_create+0x7d/0xed
[<ffffffff8028e3ac>] do_filp_open+0x1ed/0x832
[<ffffffff8028290c>] do_sys_open+0x46/0xc3
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D 00000000ffffffff 0 26605 8013
ffff81001cdc9b60 0000000000000086 ffff8101bfc0b4d0 0000000000000216
ffffffff80600cc0 ffff8101bfc0b4d0 00007f628a734000 ffff81001ce43d30
ffff8101fff33210 ffff81001ce43f68 0000000200000000 ffff81001ce43f68
Call Trace:
[<ffffffff80224f1d>] default_wake_function+0x0/0xe
[<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff80467779>] error_exit+0x0/0x51
[<ffffffff8025bae5>] find_lock_page+0x29/0x87
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
[<ffffffff8026d535>] unmap_region+0x10f/0x125
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D ffff81007e8b1690 0 26608 8013
ffff81003eb4fd70 0000000000000082 ffff81003eb4fd38 ffff81007edfcb00
ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81001ce43210
ffff81001ce410b0 ffff81001ce43448 000000018028c917 ffff81001ce43448
Call Trace:
[<ffffffff80293dd5>] dput+0x21/0x10a
[<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
[<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff8028ac53>] lock_rename+0x35/0xc5
[<ffffffff8028d545>] do_rename+0x98/0x1b2
[<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D ffff81007e8b1690 0 26609 8013
ffff81003eb63d70 0000000000000082 ffff81003eb63d38 ffff81007edfcb00
ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81001ce442c0
ffff81017faea160 ffff81001ce444f8 000000008028c917 ffff81001ce444f8
Call Trace:
[<ffffffff80293dd5>] dput+0x21/0x10a
[<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
[<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff8028ac53>] lock_rename+0x35/0xc5
[<ffffffff8028d545>] do_rename+0x98/0x1b2
[<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D ffff81007e8b1690 0 26610 8013
ffff81003e98dd70 0000000000000082 ffff81003e98dd38 ffff81007edfcb00
ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81001ce42c80
ffff81017fae8590 ffff81001ce42eb8 000000028028c917 ffff81001ce42eb8
Call Trace:
[<ffffffff80293dd5>] dput+0x21/0x10a
[<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
[<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff8028ac53>] lock_rename+0x35/0xc5
[<ffffffff8028d545>] do_rename+0x98/0x1b2
[<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
[<ffffffff8027ce42>] get_partial_node+0x15/0x78
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D 00000000ffffffff 0 26611 8013
ffff81001ced3b60 0000000000000082 ffff81002d7fa0d0 ffff81002d7fa0d0
ffffffff80600cc0 00000000486202ae 0000000210e7c865 ffff81001ce41640
ffffffff8054d350 ffff81001ce41878 0000000000000003 ffff81001ce41878
Call Trace:
[<ffffffff80411b14>] dev_queue_xmit+0x21a/0x232
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
[<ffffffff8028d0ba>] path_walk+0xaf/0xb9
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff80282afc>] __dentry_open+0x141/0x226
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D 00000000ffffffff 0 26613 8013
ffff8101e1321b30 0000000000000082 0000000000000296 00000000662df001
ffffffff80600cc0 ffff8101d40c9e90 ffffffff80242fdc ffff8101ff6dc850
ffff8101fff33210 ffff8101ff6dca88 000000028024153d ffff8101ff6dca88
Call Trace:
[<ffffffff80242fdc>] getnstimeofday+0x38/0x92
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffffa0070d34>] :nf_conntrack:nf_ct_deliver_cached_events+0x4d/0x7d
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff80269428>] do_wp_page+0x45b/0x4ba
[<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
[<ffffffff8021ed0c>] do_page_fault+0x435/0x825
[<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff80467779>] error_exit+0x0/0x51

vsftpd D ffff81007e8b3cf8 0 26614 26606
ffff8100588b1c90 0000000000000086 ffffffff802411a8 000000000000148d
ffffffff80600cc0 0000000000000286 00000000ffffffff ffff81001ce40000
ffff8101ff6dd900 ffff81001ce40238 00000002ffffffff ffff81001ce40238
Call Trace:
[<ffffffff802411a8>] hrtimer_start+0x100/0x122
[<ffffffff8022a745>] hrtick_set+0xa1/0x10a
[<ffffffff80466148>] thread_return+0x63/0xbb
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80296e64>] inode_setattr+0x103/0x10a
[<ffffffff802dc9b0>] ext4_setattr+0x1eb/0x244
[<ffffffff80296fbb>] notify_change+0x150/0x26d
[<ffffffff80283568>] do_truncate+0x5e/0x79
[<ffffffff80256c42>] audit_syscall_entry+0x11f/0x152
[<ffffffff80283664>] sys_ftruncate+0xe1/0xfd
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

bzip2 D 00000000ffffffff 0 26616 26615
ffff8102043d9ad0 0000000000000082 ffffffff80278233 ffff810256ebc660
ffffffff80600cc0 00007f5eba717000 0000000000000002 ffff81027ed91640
ffff8101fff342c0 ffff81027ed91878 000000038026a789 ffff81027ed91878
Call Trace:
[<ffffffff80278233>] alloc_page_vma+0x154/0x221
[<ffffffff802a3c7b>] __getblk+0x1e/0x28d
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff80260c63>] rmqueue_bulk+0x42/0x8a
[<ffffffff80266f91>] zone_statistics+0x3c/0x90
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80268f85>] __do_fault+0x321/0x369
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff8022a745>] hrtick_set+0xa1/0x10a
[<ffffffff80466148>] thread_return+0x63/0xbb
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp D ffff81007e8b3cf8 0 26621 8037
ffff8101b1103b60 0000000000000082 ffff81013b896b00 0000000000000206
ffffffff80600cc0 ffff81013b896b00 00007f4f1e28e000 ffff8101ff6df4d0
ffff8101bbec6f40 ffff8101ff6df708 0000000100000000 ffff8101ff6df708
Call Trace:
[<ffffffff80224f1d>] default_wake_function+0x0/0xe
[<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff80467779>] error_exit+0x0/0x51
[<ffffffff8025bae5>] find_lock_page+0x29/0x87
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
[<ffffffff8026d535>] unmap_region+0x10f/0x125
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D ffff81007e8b3cf8 0 26625 26617
ffff810039cf9b70 0000000000000082 ffff810270718a80 ffff81022646e2a0
ffffffff80600cc0 ffff810039cf9d44 0000000000000000 ffff81001ce469b0
ffff81001ce41640 ffff81001ce46be8 00000000802a3c7b ffff81001ce46be8
Call Trace:
[<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff80294b15>] __d_lookup+0xbd/0x107
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802e2b08>] ext4_rename+0x4d/0x64a
[<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
[<ffffffff8028bc5c>] vfs_rename+0x1fd/0x376
[<ffffffff8028d5fe>] do_rename+0x151/0x1b2
[<ffffffff802614c3>] __alloc_pages_internal+0xd2/0x40c
[<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
[<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D 00000000ffffffff 0 26632 26626
ffff8101dddc3c60 0000000000000082 ffffffff802e6fb3 0000000000000020
ffffffff80600cc0 ffff8101d40c8610 0000000000031014 ffff8101ff6dd900
ffff8101fff33210 ffff8101ff6ddb38 000000021d4e6000 ffff8101ff6ddb38
Call Trace:
[<ffffffff802e6fb3>] ext4fs_dirhash+0x10d/0x1d2
[<ffffffff802e04cf>] dx_probe+0xa1/0x29e
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802e24bd>] ext4_create+0x4c/0x105
[<ffffffff802e27c3>] ext4_lookup+0x97/0xc1
[<ffffffff8028b805>] vfs_create+0x7d/0xed
[<ffffffff8028e3ac>] do_filp_open+0x1ed/0x832
[<ffffffff8028290c>] do_sys_open+0x46/0xc3
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd D 00000000ffffffff 0 26635 26633
ffff8100066c9dd0 0000000000000086 ffffffff804082ac ffff8100066c9d34
ffffffff80600cc0 0000000e9aa3566f ffff81007edfcbc8 ffff81007d93e420
ffff8101fff32160 ffff81007d93e658 0000000100000000 ffff81007d93e658
Call Trace:
[<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
[<ffffffff8028d0ba>] path_walk+0xaf/0xb9
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff8028e2cb>] do_filp_open+0x10c/0x832
[<ffffffff80282837>] get_unused_fd_flags+0x7f/0x10e
[<ffffffff8028290c>] do_sys_open+0x46/0xc3
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

bzip2 D 00000000ffffffff 0 26638 26637
ffff8101427cdad0 0000000000000082 ffffffff80278233 ffff8101f6fa70c0
ffffffff80600cc0 00007f0c65f4e000 0000000000000002 ffff81017faec850
ffffffff8054d350 ffff81017faeca88 000000008026a789 ffff81017faeca88
Call Trace:
[<ffffffff80278233>] alloc_page_vma+0x154/0x221
[<ffffffff802a3c7b>] __getblk+0x1e/0x28d
[<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
[<ffffffff80466561>] mutex_lock+0xa/0xb
[<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
[<ffffffff802feac7>] start_this_handle+0x2eb/0x370
[<ffffffff80260c63>] rmqueue_bulk+0x42/0x8a
[<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
[<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
[<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
[<ffffffff80295a1f>] file_update_time+0xaf/0xf4
[<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
[<ffffffff80268f85>] __do_fault+0x321/0x369
[<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
[<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
[<ffffffff80283eb3>] do_sync_write+0xce/0x113
[<ffffffff8021ed48>] do_page_fault+0x471/0x825
[<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
[<ffffffff8022a72f>] hrtick_set+0x8b/0x10a
[<ffffffff80466148>] thread_return+0x63/0xbb
[<ffffffff802846d8>] vfs_write+0xad/0x136
[<ffffffff8028481d>] sys_write+0x45/0x6e
[<ffffffff8020b2d2>] tracesys+0xd5/0xda

I used ext4-patch-queue-b5db22ef52ed53d8e3fa978a5a29e1609c9333aa.tar.gz and
a clean linux-2.6.26-rc6 with no other patches.

Holger

2008-06-26 00:46:02

[permalink] [raw]

Subject: Re: Performance of ext4

On Wed, 2008-06-25 at 09:09 +0000, Holger Kiehl wrote:
> On Tue, 24 Jun 2008, Mingming wrote:
>
> >
> > On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
> >> Yes, with this patch applied on top of latest patch queue I no longer
> >> get truncated files, after running a short test. Tomorrow I will do some
> >> more thorough testing and use the patch you have send to me in a separate
> >> mail. The above patch did not apply but it was easy to apply by hand.
> >
> >
> > Thanks for quick response and test. I have updated the patch queue with
> > above patch merged. Please let me know if you still see apply issue and
> > file size update issue with current patch queue.
> >
> Thanks, it applies without any problems. However I still hit an oops. What
> I find strange is that I got the oops just as the benchmark is done and
> all process where shutting down. The same behaviour I reported here:
> http://www.ussg.iu.edu/hypermail/linux/kernel/0806.2/2113.html
> Only this time I got just one oops. This is on x86_64 system (4 Opteron CPU's
> and SW Raid 1+0). I have not seen this on my home system x86 (1 Dual Core
> and HW Raid). Anyway, here the dmesg output:
>
> kjournald2 starting. Commit interval 15 seconds
> EXT4 FS on md7, internal journal
> EXT4-fs: mounted filesystem with ordered data mode.
> EXT4-fs: file extents enabled
> EXT4-fs: mballoc enabled
> JBD: barrier-based sync failed on md7 - disabling barriers
> ------------[ cut here ]------------
> kernel BUG at fs/ext4/inode.c:1667!

Did not get a chance to look more closely today, but it's point to this
code in ext4_da_writepage()

page_bufs = page_buffers(page);

and appearently it's BUG_ON at
BUG_ON(!PagePrivate(page)); in page_buffers().

Mingming

> invalid opcode: 0000 [1] SMP
> CPU 3
> Modules linked in: w83627hf lm85 hwmon_vid bonding nf_conntrack_ftp ipt_REJECT xt_tcpudp nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables binfmt_misc sg floppy k8temp ohci_hcd i2c_amd756 i2c_core button usbcore
> Pid: 2661, comm: kjournald2 Not tainted 2.6.26-rc6 #1
> RIP: 0010:[<ffffffff802dd314>] [<ffffffff802dd314>] ext4_da_writepage+0x30/0xf4
> RSP: 0018:ffff81027d737c10 EFLAGS: 00010246
> RAX: 010000000001002d RBX: ffffe200017e9218 RCX: 00000000ffffffe8
> RDX: 0000000000004278 RSI: ffff81027d737e00 RDI: ffffe200017e9218
> RBP: ffff81021dad66e8 R08: ffff81027ff05440 R09: ffff81027d737b50
> R10: ffff81020fe54480 R11: ffff81020fe54400 R12: ffff81027d737e00
> R13: 0000000000000001 R14: 0000000000000000 R15: ffff8101ff0b8120
> FS: 00007f53ed10b6f0(0000) GS:ffff81027ff13000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 000000000081a000 CR3: 0000000166f03000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process kjournald2 (pid: 2661, threadinfo ffff81027d736000, task ffff81027ed93210)
> Stack: ffff81021dad67f8 ffff81027d737e00 0000000000000000 ffffffff80261816
> ffffe200017e9218 ffffffff80261f8c ffff81021dad67f8 ffffffff8026180c
> ffff81021dad67f8 0000000280224f0c 0000000000000004 000000007d737fd8
> Call Trace:
> [<ffffffff80261816>] ? __writepage+0xa/0x27
> [<ffffffff80261f8c>] ? write_cache_pages+0x174/0x2be
> [<ffffffff8026180c>] ? __writepage+0x0/0x27
> [<ffffffff802ff1ed>] ? jbd2_journal_commit_transaction+0x343/0xe6a
> [<ffffffff802411a8>] ? hrtimer_start+0x100/0x122
> [<ffffffff802357ba>] ? try_to_del_timer_sync+0x52/0x5b
> [<ffffffff80302972>] ? kjournald2+0xdf/0x202
> [<ffffffff8023e74a>] ? autoremove_wake_function+0x0/0x2e
> [<ffffffff80302893>] ? kjournald2+0x0/0x202
> [<ffffffff8023e43b>] ? kthread+0x47/0x73
> [<ffffffff8020bf78>] ? child_rip+0xa/0x12
> [<ffffffff8023e3f4>] ? kthread+0x0/0x73
> [<ffffffff8020bf6e>] ? child_rip+0x0/0x12
>
>
> Code: 55 53 48 8b 47 18 48 89 fb 48 8b 28 65 48 8b 04 25 00 00 00 00 48 83 b8 d0 04 00 00 00 75 5f 48 8b 07 48 8b 55 68 f6 c4 08 75 04 <0f> 0b eb fe 48 89 d0 81 e2 ff 0f 00 00 48 8b 77 10 48 c1 f8 0c
> RIP [<ffffffff802dd314>] ext4_da_writepage+0x30/0xf4
> RSP <ffff81027d737c10>
> ---[ end trace efaf3891d582feb1 ]---
> SysRq : Show Blocked State
> task PC stack pid father
> pdflush D 00000000ffffffff 0 242 2
> ffff81007f319bb8 0000000000000046 ffffffff8031682e 000000000000000e
> ffffffff80600cc0 000000000000000e ffff81007f319bf0 ffff81007fbbbd30
> ffff8101fff33210 ffff81007fbbbf68 00000002802aa2a6 ffff81007fbbbf68
> Call Trace:
> [<ffffffff8031682e>] submit_bio+0xc4/0xcb
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff802fde7f>] jbd2_journal_stop+0x191/0x19d
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dd7e8>] ext4_da_writepages+0x75/0x173
> [<ffffffff80262112>] do_writepages+0x20/0x2d
> [<ffffffff8029f6da>] __writeback_single_inode+0x16f/0x35e
> [<ffffffff8029fcf7>] sync_sb_inodes+0x283/0x3b8
> [<ffffffff802a0080>] writeback_inodes+0x89/0xde
> [<ffffffff80262252>] wb_kupdate+0x9f/0x116
> [<ffffffff80262ca0>] pdflush+0x122/0x1c9
> [<ffffffff802621b3>] wb_kupdate+0x0/0x116
> [<ffffffff80262b7e>] pdflush+0x0/0x1c9
> [<ffffffff8023e43b>] kthread+0x47/0x73
> [<ffffffff8022ae6a>] schedule_tail+0x27/0x5c
> [<ffffffff8020bf78>] child_rip+0xa/0x12
> [<ffffffff8023e3f4>] kthread+0x0/0x73
> [<ffffffff8020bf6e>] child_rip+0x0/0x12
>
> init_afd D 00000000ffffffff 0 7974 1
> ffff8101050d3b30 0000000000000082 ffff810055f08230 0000000000080768
> ffffffff80600cc0 ffff8101ff8509c0 ffffffff802a38d5 ffff8101050aac80
> ffff8101fff342c0 ffff8101050aaeb8 000000038023e66b ffff8101050aaeb8
> Call Trace:
> [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
> [<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff80224a97>] __wake_up+0x38/0x4f
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
> [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
> [<ffffffff8022a72f>] hrtick_set+0x8b/0x10a
> [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
> [<ffffffff80242fdc>] getnstimeofday+0x38/0x92
> [<ffffffff80240fcb>] lock_hrtimer_base+0x1b/0x3c
> [<ffffffff80241089>] hrtimer_try_to_cancel+0x61/0x6a
> [<ffffffff8024109e>] hrtimer_cancel+0xc/0x16
> [<ffffffff8046687e>] do_nanosleep+0x6c/0xa6
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff80467779>] error_exit+0x0/0x51
>
> system_log D 00000000ffffffff 0 7975 7974
> ffff81002d533ad0 0000000000000086 ffffffff8028fce8 ffff81007ed55e90
> ffffffff80600cc0 0000000000000100 ffff81027d7cae40 ffff81007ed55e90
> ffff8101fff342c0 ffff81007ed560c8 000000032d533f50 ffff81007ed560c8
> Call Trace:
> [<ffffffff8028fce8>] poll_freewait+0x36/0x87
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
> [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff80295a76>] touch_atime+0x12/0xfa
> [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8021ed48>] do_page_fault+0x471/0x825
> [<ffffffff8026f499>] mmap_region+0x41a/0x505
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> receive_log D 00000000ffffffff 0 7977 7974
> ffff81002d4ffad0 0000000000000082 ffffffff8028fce8 ffff81007ed55900
> ffffffff80600cc0 0000000000000100 ffff81007e86ed80 ffff81007ed55900
> ffffffff8054d350 ffff81007ed55b38 000000002d4fff50 ffff81007ed55b38
> Call Trace:
> [<ffffffff8028fce8>] poll_freewait+0x36/0x87
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff802ec6a2>] __ext4_journal_dirty_metadata+0x1e/0x46
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff8028ad96>] do_lookup+0x63/0x1a1
> [<ffffffff80293dd5>] dput+0x21/0x10a
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> archive_watch D 00000000ffffffff 0 7980 7974
> ffff81007ecabce0 0000000000000082 ffffffff802de7cd ffff810000000001
> ffffffff80600cc0 ffff810000000001 ffffffff8027ce42 ffff81007ed537a0
> ffff8101fff33210 ffff81007ed539d8 00000002802d9b7a ffff81007ed539d8
> Call Trace:
> [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
> [<ffffffff8027ce42>] get_partial_node+0x15/0x78
> [<ffffffff80294b15>] __d_lookup+0xbd/0x107
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8028c917>] __link_path_walk+0x6e0/0xdd4
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802e3350>] ext4_unlink+0x3f/0x1fe
> [<ffffffff8028b364>] may_delete+0x5e/0x135
> [<ffffffff8028b9af>] vfs_unlink+0x5e/0xa3
> [<ffffffff8028d985>] do_unlinkat+0xc1/0x142
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff8020b26e>] tracesys+0x71/0xda
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> amg D ffff81007e8b3cf8 0 7986 7974
> ffff81002d50bb60 0000000000000086 0000000000000000 0000000000000100
> ffffffff80600cc0 0000000000000000 0000000000000000 ffff81007ed50000
> ffff8101050acde0 ffff81007ed50238 0000000100000000 ffff81007ed50238
> Call Trace:
> [<ffffffff80224f1d>] default_wake_function+0x0/0xe
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
> [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
> [<ffffffff8028d0ba>] path_walk+0xaf/0xb9
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> dir_check D ffff81007e8b1690 0 7988 7986
> ffff81002d625d70 0000000000000082 ffff81002d625d38 ffff81007edfcb00
> ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81007ed542c0
> ffff81007ed574d0 ffff81007ed544f8 000000028028c917 ffff81007ed544f8
> Call Trace:
> [<ffffffff80293dd5>] dput+0x21/0x10a
> [<ffffffff8028cee0>] __link_path_walk+0xca9/0xdd4
> [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff8028ac53>] lock_rename+0x35/0xc5
> [<ffffffff8028d545>] do_rename+0x98/0x1b2
> [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> afd_stat D 00000000ffffffff 0 7989 7974
> ffff81002d461b30 0000000000000082 ffff81002d461da8 ffff81002d461da8
> ffffffff80600cc0 ffff81002d461da8 ffffffff802a38d5 ffff81007ed57a60
> ffffffff8054d350 ffff81007ed57c98 000000008023e66b ffff81007ed57c98
> Call Trace:
> [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
> [<ffffffff80290dfd>] __pollwait+0x0/0xe3
> [<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
> [<ffffffff80242fdc>] getnstimeofday+0x38/0x92
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff80262dda>] pdflush_operation+0x93/0x9d
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
> [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
> [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
> [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
> [<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
> [<ffffffff802411a8>] hrtimer_start+0x100/0x122
> [<ffffffff802258bb>] hrtick_start_fair+0x139/0x18a
> [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff80467779>] error_exit+0x0/0x51
>
> init_afd D ffff81007e8b3cf8 0 7997 1
> ffff81027ee41b30 0000000000000086 ffff81027ee41dd0 ffff81027ee41da8
> ffffffff80600cc0 ffff81027ee41db8 0000000000000010 ffff81027ecca160
> ffff8101bbec1640 ffff81027ecca398 0000000000000000 ffff81027ecca398
> Call Trace:
> [<ffffffff80290dfd>] __pollwait+0x0/0xe3
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
> [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
> [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
> [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
> [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff80467779>] error_exit+0x0/0x51
>
> transfer_log D 00000000ffffffff 0 8001 7997
> ffff81002d681ad0 0000000000000082 ffffffff8028fce8 ffff81007fb8f4d0
> ffffffff80600cc0 0000000000000100 ffff8101b1362540 ffff81007fb8f4d0
> ffff8101fff33210 ffff81007fb8f708 000000022d681f50 ffff81007fb8f708
> Call Trace:
> [<ffffffff8028fce8>] poll_freewait+0x36/0x87
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff80295a76>] touch_atime+0x12/0xfa
> [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8021ed48>] do_page_fault+0x471/0x825
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> archive_watch D 00000000ffffffff 0 8003 7997
> ffff81002d49dce0 0000000000000086 ffffffff802de7cd ffff810000000001
> ffffffff80600cc0 ffff810000000001 ffffffff8027ce42 ffff81007fb8ef40
> ffff8101fff32160 ffff81007fb8f178 00000001802d9b7a ffff81007fb8f178
> Call Trace:
> [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
> [<ffffffff8027ce42>] get_partial_node+0x15/0x78
> [<ffffffff80294b15>] __d_lookup+0xbd/0x107
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8028c917>] __link_path_walk+0x6e0/0xdd4
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802e3350>] ext4_unlink+0x3f/0x1fe
> [<ffffffff8028b364>] may_delete+0x5e/0x135
> [<ffffffff8028b9af>] vfs_unlink+0x5e/0xa3
> [<ffffffff8028d985>] do_unlinkat+0xc1/0x142
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff8020b26e>] tracesys+0x71/0xda
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> input_log D 00000000ffffffff 0 8004 7997
> ffff81002d419ad0 0000000000000086 ffffffff8028fce8 ffff81007fb8de90
> ffffffff80600cc0 0000000000000100 ffff81027f76aa80 ffff81007fb8de90
> ffff8101fff32160 ffff81007fb8e0c8 000000012d419f50 ffff81007fb8e0c8
> Call Trace:
> [<ffffffff8028fce8>] poll_freewait+0x36/0x87
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
> [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff80295a76>] touch_atime+0x12/0xfa
> [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> output_log D ffff81007e8b3cf8 0 8005 7997
> ffff81002d4fdad0 0000000000000086 ffffffff8028fce8 ffff81007fbbc2c0
> ffffffff80600cc0 0000000000000100 ffff81017ff649c0 ffff81007fbbc2c0
> ffff81001ce46420 ffff81007fbbc4f8 000000012d4fdf50 ffff81007fbbc4f8
> Call Trace:
> [<ffffffff8028fce8>] poll_freewait+0x36/0x87
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
> [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff80295a76>] touch_atime+0x12/0xfa
> [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> production_lo D ffff81007e8b3cf8 0 8007 7997
> ffff81002d615ad0 0000000000000082 ffffffff8028fce8 ffff81007d939bd0
> ffffffff80600cc0 0000000000000100 ffff81027f76a3c0 ffff81007d939bd0
> ffff81007ed50590 ffff81007d939e08 000000012d615f50 ffff81007d939e08
> Call Trace:
> [<ffffffff8028fce8>] poll_freewait+0x36/0x87
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
> [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff80295a76>] touch_atime+0x12/0xfa
> [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8026f499>] mmap_region+0x41a/0x505
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> distribution_ D 00000000ffffffff 0 8008 7997
> ffff81002d53fad0 0000000000000086 ffffffff8028fce8 ffff81007d93ac80
> ffffffff80600cc0 0000000000000100 ffff81007f300600 ffff81007d93ac80
> ffff8101fff33210 ffff81007d93aeb8 000000022d53ff50 ffff81007d93aeb8
> Call Trace:
> [<ffffffff8028fce8>] poll_freewait+0x36/0x87
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff80295a76>] touch_atime+0x12/0xfa
> [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> dir_check D ffff81007e8b1690 0 8011 8009
> ffff8101e1359d70 0000000000000082 ffff8101e1359d38 ffff81007edfcb00
> ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff8101bbec10b0
> ffff81007fb8de90 ffff8101bbec12e8 000000018028c917 ffff8101bbec12e8
> Call Trace:
> [<ffffffff80293dd5>] dput+0x21/0x10a
> [<ffffffff8028cee0>] __link_path_walk+0xca9/0xdd4
> [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff8028ac53>] lock_rename+0x35/0xc5
> [<ffffffff8028d545>] do_rename+0x98/0x1b2
> [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> afd_stat D ffff81007e8b3cf8 0 8012 7997
> ffff81002d69bb30 0000000000000082 ffff81002d69bda8 ffff81002d69bda8
> ffffffff80600cc0 ffff81002d69bda8 ffffffff802a38d5 ffff81007d93c2c0
> ffff81027eccd900 ffff81007d93c4f8 000000008023e66b ffff81007d93c4f8
> Call Trace:
> [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
> [<ffffffff80290dfd>] __pollwait+0x0/0xe3
> [<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff80224a97>] __wake_up+0x38/0x4f
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
> [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
> [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
> [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
> [<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
> [<ffffffff802411a8>] hrtimer_start+0x100/0x122
> [<ffffffff802258bb>] hrtick_start_fair+0x139/0x18a
> [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff80467779>] error_exit+0x0/0x51
>
> init_afd D 00000000ffffffff 0 8020 1
> ffff8101bbdd5b30 0000000000000086 ffff8101bbdd5dd0 ffff8101bbdd5da8
> ffffffff80600cc0 ffff8101bbdd5db8 0000000000000010 ffff8101bbec1640
> ffffffff8054d350 ffff8101bbec1878 0000000000000000 ffff8101bbec1878
> Call Trace:
> [<ffffffff80290dfd>] __pollwait+0x0/0xe3
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
> [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
> [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
> [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
> [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff80467779>] error_exit+0x0/0x51
>
> receive_log D 00000000ffffffff 0 8023 8020
> ffff81027ee1bad0 0000000000000086 ffffffff8028fce8 ffff81027eccc2c0
> ffffffff80600cc0 0000000000000100 ffff81007f300000 ffff81027eccc2c0
> ffff8101fff33210 ffff81027eccc4f8 000000027ee1bf50 ffff81027eccc4f8
> Call Trace:
> [<ffffffff8028fce8>] poll_freewait+0x36/0x87
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff802ec6a2>] __ext4_journal_dirty_metadata+0x1e/0x46
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff8028ad96>] do_lookup+0x63/0x1a1
> [<ffffffff80295a76>] touch_atime+0x12/0xfa
> [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> transfer_log D 00000000ffffffff 0 8024 8020
> ffff81027f799ad0 0000000000000086 ffffffff8028fce8 ffff81027eccc850
> ffffffff80600cc0 0000000000000100 ffff81027d7443c0 ffff81027eccc850
> ffffffff8054d350 ffff81027eccca88 000000007f799f50 ffff81027eccca88
> Call Trace:
> [<ffffffff8028fce8>] poll_freewait+0x36/0x87
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
> [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff80295a76>] touch_atime+0x12/0xfa
> [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8021ed48>] do_page_fault+0x471/0x825
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> archive_watch D 00000000ffffffff 0 8026 8020
> ffff81027f7ddce0 0000000000000082 ffffffff802de7cd ffff810200000001
> ffffffff80600cc0 ffff810000000001 ffffffff8027ce42 ffff81027ecc90b0
> ffff8101fff33210 ffff81027ecc92e8 00000002802d9b7a ffff81027ecc92e8
> Call Trace:
> [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
> [<ffffffff8027ce42>] get_partial_node+0x15/0x78
> [<ffffffff80294b15>] __d_lookup+0xbd/0x107
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8028c917>] __link_path_walk+0x6e0/0xdd4
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802e3350>] ext4_unlink+0x3f/0x1fe
> [<ffffffff8028b364>] may_delete+0x5e/0x135
> [<ffffffff8028b9af>] vfs_unlink+0x5e/0xa3
> [<ffffffff8028d985>] do_unlinkat+0xc1/0x142
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff8020b26e>] tracesys+0x71/0xda
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> input_log D ffff81007e8b3cf8 0 8027 8020
> ffff81027eed1ad0 0000000000000082 ffffffff8028fce8 ffff81027ecca6f0
> ffffffff80600cc0 0000000000000100 ffff81027d744e40 ffff81027ecca6f0
> ffff81007fb8de90 ffff81027ecca928 000000017eed1f50 ffff81027ecca928
> Call Trace:
> [<ffffffff8028fce8>] poll_freewait+0x36/0x87
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff80295a76>] touch_atime+0x12/0xfa
> [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> output_log D ffff81007e8b3cf8 0 8028 8020
> ffff81027d64dad0 0000000000000086 ffffffff8028fce8 ffff81027eccac80
> ffffffff80600cc0 0000000000000100 ffff81010508c840 ffff81027eccac80
> ffff81007ed574d0 ffff81027eccaeb8 000000027d64df50 ffff81027eccaeb8
> Call Trace:
> [<ffffffff8028fce8>] poll_freewait+0x36/0x87
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff80295a76>] touch_atime+0x12/0xfa
> [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> distribution_ D 00000000ffffffff 0 8031 8020
> ffff81027ee1dad0 0000000000000086 ffffffff8028fce8 ffff81027ecc8590
> ffffffff80600cc0 0000000000000100 ffff810105128a80 ffff81027ecc8590
> ffff8101fff33210 ffff81027ecc87c8 000000027ee1df50 ffff81027ecc87c8
> Call Trace:
> [<ffffffff8028fce8>] poll_freewait+0x36/0x87
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
> [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff80295a76>] touch_atime+0x12/0xfa
> [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> dir_check D 00000000ffffffff 0 8034 8032
> ffff81027ec01d70 0000000000000082 ffff81027ec01d38 ffff81007edfcb00
> ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81027eccb210
> ffff8101fff33210 ffff81027eccb448 000000028028c917 ffff81027eccb448
> Call Trace:
> [<ffffffff80293dd5>] dput+0x21/0x10a
> [<ffffffff8028cee0>] __link_path_walk+0xca9/0xdd4
> [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff8028ac53>] lock_rename+0x35/0xc5
> [<ffffffff8028d545>] do_rename+0x98/0x1b2
> [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> afd_stat D ffff81007e8b3cf8 0 8035 8020
> ffff81027edebb30 0000000000000082 ffff81027edebda8 ffff81027edebda8
> ffffffff80600cc0 ffff81027edebda8 ffffffff802a38d5 ffff81027eccd900
> ffff81007ed57a60 ffff81027eccdb38 000000008023e66b ffff81027eccdb38
> Call Trace:
> [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
> [<ffffffff80290dfd>] __pollwait+0x0/0xe3
> [<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff80224a97>] __wake_up+0x38/0x4f
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
> [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
> [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
> [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
> [<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
> [<ffffffff802411a8>] hrtimer_start+0x100/0x122
> [<ffffffff802258bb>] hrtick_start_fair+0x139/0x18a
> [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff80467779>] error_exit+0x0/0x51
>
> fd D ffff81007e8b3cf8 0 8037 8020
> ffff81027ededb30 0000000000000086 ffff81027ededdd0 ffff81027ededda8
> ffffffff80600cc0 ffff81027ededdb8 00000000000007e0 ffff81027eccde90
> ffff81027ed937a0 ffff81027ecce0c8 0000000200000000 ffff81027ecce0c8
> Call Trace:
> [<ffffffff80290dfd>] __pollwait+0x0/0xe3
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
> [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
> [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
> [<ffffffff8022f67f>] release_task+0x2cb/0x2dd
> [<ffffffff80230175>] do_wait+0xae4/0xb88
> [<ffffffff80286e19>] vfs_stat_fd+0x1b/0x4a
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff80467779>] error_exit+0x0/0x51
>
> afd D 00000000ffffffff 0 25970 2891
> ffff81017fae7ad0 0000000000000086 ffffffff8028fce8 0000000000000246
> ffffffff80600cc0 0000000000000100 ffff81020fe5dc00 ffff8101050ab210
> ffff8101fff342c0 ffff8101050ab448 000000037fae7f50 ffff8101050ab448
> Call Trace:
> [<ffffffff8028fce8>] poll_freewait+0x36/0x87
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff80268f85>] __do_fault+0x321/0x369
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8021ed48>] do_page_fault+0x471/0x825
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff80237d9a>] group_send_sig_info+0x62/0x6f
> [<ffffffff80237dd8>] kill_pid_info+0x31/0x3b
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> sf_ftp D ffff81007e8b3cf8 0 26537 8037
> ffff8101ff7b1b70 0000000000000086 ffffffff802411a8 000000000000148d
> ffffffff80600cc0 0000000000000286 00000000ffffffff ffff8101bbec26f0
> ffff81027eccb210 ffff8101bbec2928 00000002ffffffff ffff8101bbec2928
> Call Trace:
> [<ffffffff802411a8>] hrtimer_start+0x100/0x122
> [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
> [<ffffffff80466148>] thread_return+0x63/0xbb
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802e2b08>] ext4_rename+0x4d/0x64a
> [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
> [<ffffffff8028bc5c>] vfs_rename+0x1fd/0x376
> [<ffffffff8028d5fe>] do_rename+0x151/0x1b2
> [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> sf_ftp D ffff81007e8b1690 0 26543 8037
> ffff8101ddec3d70 0000000000000086 ffff8101ddec3d38 ffff81007edfcb00
> ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff8101bbec3210
> ffff81007ed542c0 ffff8101bbec3448 000000028028c917 ffff8101bbec3448
> Call Trace:
> [<ffffffff80293dd5>] dput+0x21/0x10a
> [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
> [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff8028ac53>] lock_rename+0x35/0xc5
> [<ffffffff8028d545>] do_rename+0x98/0x1b2
> [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> sf_ftp D ffff81007e8b3cf8 0 26559 8037
> ffff8101b11ddb60 0000000000000086 ffff8101bfc0b4d0 0000000000000202
> ffffffff80600cc0 ffff8101bfc0b4d0 00007f1ea612f000 ffff8101bbec74d0
> ffff81007ed574d0 ffff8101bbec7708 0000000200000000 ffff8101bbec7708
> Call Trace:
> [<ffffffff80224f1d>] default_wake_function+0x0/0xe
> [<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff80467779>] error_exit+0x0/0x51
> [<ffffffff8025bae5>] find_lock_page+0x29/0x87
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8021ed48>] do_page_fault+0x471/0x825
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
> [<ffffffff8026d535>] unmap_region+0x10f/0x125
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> sf_ftp D 00000000ffffffff 0 26569 8013
> ffff8101f40c1b60 0000000000000082 ffff81013b896b00 0000000000000206
> ffffffff80600cc0 ffff81013b896b00 00007f31838c7000 ffff8101bbec6f40
> ffff8101fff32160 ffff8101bbec7178 0000000100000000 ffff8101bbec7178
> Call Trace:
> [<ffffffff80224f1d>] default_wake_function+0x0/0xe
> [<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff80467779>] error_exit+0x0/0x51
> [<ffffffff8025bae5>] find_lock_page+0x29/0x87
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8021ed48>] do_page_fault+0x471/0x825
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
> [<ffffffff8026d535>] unmap_region+0x10f/0x125
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> sf_ftp D ffff81007e8b3cf8 0 26583 8013
> ffff8101bbf9fb60 0000000000000086 ffff81010c703160 0000000000000202
> ffffffff80600cc0 ffff81010c703160 00007ffac6bd2000 ffff8101bbec6420
> ffff81007d93fa60 ffff8101bbec6658 0000000100000000 ffff8101bbec6658
> Call Trace:
> [<ffffffff80224f1d>] default_wake_function+0x0/0xe
> [<ffffffff80411b14>] dev_queue_xmit+0x21a/0x232
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff80467779>] error_exit+0x0/0x51
> [<ffffffff8025bae5>] find_lock_page+0x29/0x87
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8021ed48>] do_page_fault+0x471/0x825
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
> [<ffffffff8026d535>] unmap_region+0x10f/0x125
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> vsftpd D ffff810208bf59f8 0 26589 26567
> ffff810208bf59d8 0000000000000082 ffff810163233090 ffffffff802e9b05
> ffffffff80600cc0 ffffffff00058011 ffff810208bf5af8 ffff81027ecce420
> ffff81001ce40000 ffff81027ecce658 0000000208bf5b58 ffff81027ecce658
> Call Trace:
> [<ffffffff802e9b05>] ext4_ext_find_extent+0x67/0x26f
> [<ffffffff80224a97>] __wake_up+0x38/0x4f
> [<ffffffff803023b0>] jbd2_log_wait_commit+0xb6/0x108
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff80300c3c>] jbd2_log_do_checkpoint+0x12d/0x3bd
> [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
> [<ffffffff802a3c4d>] __find_get_block+0x14c/0x15c
> [<ffffffff802a3c7b>] __getblk+0x1e/0x28d
> [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
> [<ffffffff802e6fb3>] ext4fs_dirhash+0x10d/0x1d2
> [<ffffffff802e04cf>] dx_probe+0xa1/0x29e
> [<ffffffff802e1b3a>] ext4_find_entry+0x293/0x592
> [<ffffffff80300f53>] __jbd2_log_wait_for_space+0x87/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8029535a>] d_alloc+0x1f/0x189
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802e24bd>] ext4_create+0x4c/0x105
> [<ffffffff802e27c3>] ext4_lookup+0x97/0xc1
> [<ffffffff8028b805>] vfs_create+0x7d/0xed
> [<ffffffff8028e3ac>] do_filp_open+0x1ed/0x832
> [<ffffffff8028290c>] do_sys_open+0x46/0xc3
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> vsftpd D ffff81007e8b3cf8 0 26595 26584
> ffff8101b1307ad0 0000000000000086 ffffffff8042530e ffff81014256f5d0
> ffffffff80600cc0 ffff81017fa8fa00 ffffffff80242fdc ffff8101ff6d8b20
> ffff81001ce46420 ffff8101ff6d8d58 000000018024153d ffff8101ff6d8d58
> Call Trace:
> [<ffffffff8042530e>] nf_iterate+0x41/0x7d
> [<ffffffff80242fdc>] getnstimeofday+0x38/0x92
> [<ffffffff8039d0b6>] loopback_xmit+0x80/0x86
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> vsftpd D ffff81007e8b3cf8 0 26602 26590
> ffff81025dcf1c60 0000000000000082 ffffffff802e6fb3 0000000000000020
> ffffffff80600cc0 ffff81014256f090 00000000000300aa ffff81027ed942c0
> ffff81007d93e420 ffff81027ed944f8 000000017ae2b000 ffff81027ed944f8
> Call Trace:
> [<ffffffff802e6fb3>] ext4fs_dirhash+0x10d/0x1d2
> [<ffffffff802e04cf>] dx_probe+0xa1/0x29e
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802e24bd>] ext4_create+0x4c/0x105
> [<ffffffff802e27c3>] ext4_lookup+0x97/0xc1
> [<ffffffff8028b805>] vfs_create+0x7d/0xed
> [<ffffffff8028e3ac>] do_filp_open+0x1ed/0x832
> [<ffffffff8028290c>] do_sys_open+0x46/0xc3
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> sf_ftp D 00000000ffffffff 0 26605 8013
> ffff81001cdc9b60 0000000000000086 ffff8101bfc0b4d0 0000000000000216
> ffffffff80600cc0 ffff8101bfc0b4d0 00007f628a734000 ffff81001ce43d30
> ffff8101fff33210 ffff81001ce43f68 0000000200000000 ffff81001ce43f68
> Call Trace:
> [<ffffffff80224f1d>] default_wake_function+0x0/0xe
> [<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff80467779>] error_exit+0x0/0x51
> [<ffffffff8025bae5>] find_lock_page+0x29/0x87
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8021ed48>] do_page_fault+0x471/0x825
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
> [<ffffffff8026d535>] unmap_region+0x10f/0x125
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> sf_ftp D ffff81007e8b1690 0 26608 8013
> ffff81003eb4fd70 0000000000000082 ffff81003eb4fd38 ffff81007edfcb00
> ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81001ce43210
> ffff81001ce410b0 ffff81001ce43448 000000018028c917 ffff81001ce43448
> Call Trace:
> [<ffffffff80293dd5>] dput+0x21/0x10a
> [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
> [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff8028ac53>] lock_rename+0x35/0xc5
> [<ffffffff8028d545>] do_rename+0x98/0x1b2
> [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> sf_ftp D ffff81007e8b1690 0 26609 8013
> ffff81003eb63d70 0000000000000082 ffff81003eb63d38 ffff81007edfcb00
> ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81001ce442c0
> ffff81017faea160 ffff81001ce444f8 000000008028c917 ffff81001ce444f8
> Call Trace:
> [<ffffffff80293dd5>] dput+0x21/0x10a
> [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
> [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff8028ac53>] lock_rename+0x35/0xc5
> [<ffffffff8028d545>] do_rename+0x98/0x1b2
> [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> sf_ftp D ffff81007e8b1690 0 26610 8013
> ffff81003e98dd70 0000000000000082 ffff81003e98dd38 ffff81007edfcb00
> ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81001ce42c80
> ffff81017fae8590 ffff81001ce42eb8 000000028028c917 ffff81001ce42eb8
> Call Trace:
> [<ffffffff80293dd5>] dput+0x21/0x10a
> [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
> [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff8028ac53>] lock_rename+0x35/0xc5
> [<ffffffff8028d545>] do_rename+0x98/0x1b2
> [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
> [<ffffffff8027ce42>] get_partial_node+0x15/0x78
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> sf_ftp D 00000000ffffffff 0 26611 8013
> ffff81001ced3b60 0000000000000082 ffff81002d7fa0d0 ffff81002d7fa0d0
> ffffffff80600cc0 00000000486202ae 0000000210e7c865 ffff81001ce41640
> ffffffff8054d350 ffff81001ce41878 0000000000000003 ffff81001ce41878
> Call Trace:
> [<ffffffff80411b14>] dev_queue_xmit+0x21a/0x232
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
> [<ffffffff8028d0ba>] path_walk+0xaf/0xb9
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff80282afc>] __dentry_open+0x141/0x226
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> sf_ftp D 00000000ffffffff 0 26613 8013
> ffff8101e1321b30 0000000000000082 0000000000000296 00000000662df001
> ffffffff80600cc0 ffff8101d40c9e90 ffffffff80242fdc ffff8101ff6dc850
> ffff8101fff33210 ffff8101ff6dca88 000000028024153d ffff8101ff6dca88
> Call Trace:
> [<ffffffff80242fdc>] getnstimeofday+0x38/0x92
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffffa0070d34>] :nf_conntrack:nf_ct_deliver_cached_events+0x4d/0x7d
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
> [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
> [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
> [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff80467779>] error_exit+0x0/0x51
>
> vsftpd D ffff81007e8b3cf8 0 26614 26606
> ffff8100588b1c90 0000000000000086 ffffffff802411a8 000000000000148d
> ffffffff80600cc0 0000000000000286 00000000ffffffff ffff81001ce40000
> ffff8101ff6dd900 ffff81001ce40238 00000002ffffffff ffff81001ce40238
> Call Trace:
> [<ffffffff802411a8>] hrtimer_start+0x100/0x122
> [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
> [<ffffffff80466148>] thread_return+0x63/0xbb
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80296e64>] inode_setattr+0x103/0x10a
> [<ffffffff802dc9b0>] ext4_setattr+0x1eb/0x244
> [<ffffffff80296fbb>] notify_change+0x150/0x26d
> [<ffffffff80283568>] do_truncate+0x5e/0x79
> [<ffffffff80256c42>] audit_syscall_entry+0x11f/0x152
> [<ffffffff80283664>] sys_ftruncate+0xe1/0xfd
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> bzip2 D 00000000ffffffff 0 26616 26615
> ffff8102043d9ad0 0000000000000082 ffffffff80278233 ffff810256ebc660
> ffffffff80600cc0 00007f5eba717000 0000000000000002 ffff81027ed91640
> ffff8101fff342c0 ffff81027ed91878 000000038026a789 ffff81027ed91878
> Call Trace:
> [<ffffffff80278233>] alloc_page_vma+0x154/0x221
> [<ffffffff802a3c7b>] __getblk+0x1e/0x28d
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff80260c63>] rmqueue_bulk+0x42/0x8a
> [<ffffffff80266f91>] zone_statistics+0x3c/0x90
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff80268f85>] __do_fault+0x321/0x369
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8021ed48>] do_page_fault+0x471/0x825
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
> [<ffffffff80466148>] thread_return+0x63/0xbb
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> sf_ftp D ffff81007e8b3cf8 0 26621 8037
> ffff8101b1103b60 0000000000000082 ffff81013b896b00 0000000000000206
> ffffffff80600cc0 ffff81013b896b00 00007f4f1e28e000 ffff8101ff6df4d0
> ffff8101bbec6f40 ffff8101ff6df708 0000000100000000 ffff8101ff6df708
> Call Trace:
> [<ffffffff80224f1d>] default_wake_function+0x0/0xe
> [<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff80467779>] error_exit+0x0/0x51
> [<ffffffff8025bae5>] find_lock_page+0x29/0x87
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8021ed48>] do_page_fault+0x471/0x825
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
> [<ffffffff8026d535>] unmap_region+0x10f/0x125
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> vsftpd D ffff81007e8b3cf8 0 26625 26617
> ffff810039cf9b70 0000000000000082 ffff810270718a80 ffff81022646e2a0
> ffffffff80600cc0 ffff810039cf9d44 0000000000000000 ffff81001ce469b0
> ffff81001ce41640 ffff81001ce46be8 00000000802a3c7b ffff81001ce46be8
> Call Trace:
> [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff80294b15>] __d_lookup+0xbd/0x107
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802e2b08>] ext4_rename+0x4d/0x64a
> [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
> [<ffffffff8028bc5c>] vfs_rename+0x1fd/0x376
> [<ffffffff8028d5fe>] do_rename+0x151/0x1b2
> [<ffffffff802614c3>] __alloc_pages_internal+0xd2/0x40c
> [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
> [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> vsftpd D 00000000ffffffff 0 26632 26626
> ffff8101dddc3c60 0000000000000082 ffffffff802e6fb3 0000000000000020
> ffffffff80600cc0 ffff8101d40c8610 0000000000031014 ffff8101ff6dd900
> ffff8101fff33210 ffff8101ff6ddb38 000000021d4e6000 ffff8101ff6ddb38
> Call Trace:
> [<ffffffff802e6fb3>] ext4fs_dirhash+0x10d/0x1d2
> [<ffffffff802e04cf>] dx_probe+0xa1/0x29e
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802e24bd>] ext4_create+0x4c/0x105
> [<ffffffff802e27c3>] ext4_lookup+0x97/0xc1
> [<ffffffff8028b805>] vfs_create+0x7d/0xed
> [<ffffffff8028e3ac>] do_filp_open+0x1ed/0x832
> [<ffffffff8028290c>] do_sys_open+0x46/0xc3
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> vsftpd D 00000000ffffffff 0 26635 26633
> ffff8100066c9dd0 0000000000000086 ffffffff804082ac ffff8100066c9d34
> ffffffff80600cc0 0000000e9aa3566f ffff81007edfcbc8 ffff81007d93e420
> ffff8101fff32160 ffff81007d93e658 0000000100000000 ffff81007d93e658
> Call Trace:
> [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
> [<ffffffff8028d0ba>] path_walk+0xaf/0xb9
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff8028e2cb>] do_filp_open+0x10c/0x832
> [<ffffffff80282837>] get_unused_fd_flags+0x7f/0x10e
> [<ffffffff8028290c>] do_sys_open+0x46/0xc3
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> bzip2 D 00000000ffffffff 0 26638 26637
> ffff8101427cdad0 0000000000000082 ffffffff80278233 ffff8101f6fa70c0
> ffffffff80600cc0 00007f0c65f4e000 0000000000000002 ffff81017faec850
> ffffffff8054d350 ffff81017faeca88 000000008026a789 ffff81017faeca88
> Call Trace:
> [<ffffffff80278233>] alloc_page_vma+0x154/0x221
> [<ffffffff802a3c7b>] __getblk+0x1e/0x28d
> [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
> [<ffffffff80466561>] mutex_lock+0xa/0xb
> [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
> [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
> [<ffffffff80260c63>] rmqueue_bulk+0x42/0x8a
> [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
> [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
> [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
> [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
> [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
> [<ffffffff80268f85>] __do_fault+0x321/0x369
> [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
> [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
> [<ffffffff80283eb3>] do_sync_write+0xce/0x113
> [<ffffffff8021ed48>] do_page_fault+0x471/0x825
> [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
> [<ffffffff8022a72f>] hrtick_set+0x8b/0x10a
> [<ffffffff80466148>] thread_return+0x63/0xbb
> [<ffffffff802846d8>] vfs_write+0xad/0x136
> [<ffffffff8028481d>] sys_write+0x45/0x6e
> [<ffffffff8020b2d2>] tracesys+0xd5/0xda
>
> I used ext4-patch-queue-b5db22ef52ed53d8e3fa978a5a29e1609c9333aa.tar.gz and
> a clean linux-2.6.26-rc6 with no other patches.
>
> Holger
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2008-06-27 09:23:39

[permalink] [raw]

Subject: Re: Performance of ext4

On Wed, Jun 25, 2008 at 05:46:39PM -0700, Mingming wrote:
>
> On Wed, 2008-06-25 at 09:09 +0000, Holger Kiehl wrote:
> > On Tue, 24 Jun 2008, Mingming wrote:
> >
> > >
> > > On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
> > >> Yes, with this patch applied on top of latest patch queue I no longer
> > >> get truncated files, after running a short test. Tomorrow I will do some
> > >> more thorough testing and use the patch you have send to me in a separate
> > >> mail. The above patch did not apply but it was easy to apply by hand.
> > >
> > >
> > > Thanks for quick response and test. I have updated the patch queue with
> > > above patch merged. Please let me know if you still see apply issue and
> > > file size update issue with current patch queue.
> > >
> > Thanks, it applies without any problems. However I still hit an oops. What
> > I find strange is that I got the oops just as the benchmark is done and
> > all process where shutting down. The same behaviour I reported here:
> > http://www.ussg.iu.edu/hypermail/linux/kernel/0806.2/2113.html
> > Only this time I got just one oops. This is on x86_64 system (4 Opteron CPU's
> > and SW Raid 1+0). I have not seen this on my home system x86 (1 Dual Core
> > and HW Raid). Anyway, here the dmesg output:
> >
> > kjournald2 starting. Commit interval 15 seconds
> > EXT4 FS on md7, internal journal
> > EXT4-fs: mounted filesystem with ordered data mode.
> > EXT4-fs: file extents enabled
> > EXT4-fs: mballoc enabled
> > JBD: barrier-based sync failed on md7 - disabling barriers
> > ------------[ cut here ]------------
> > kernel BUG at fs/ext4/inode.c:1667!
>
> Did not get a chance to look more closely today, but it's point to this
> code in ext4_da_writepage()
>
> page_bufs = page_buffers(page);
>
> and appearently it's BUG_ON at
> BUG_ON(!PagePrivate(page)); in page_buffers().
>
>

Ok so we are doing the journal_commit and meanwhile shrink_page_list
dropped the buffer. I guess what is happening is

journal_submit_inode_data_buffers
generic_writepages
write_cache_pages
pagevec_lookup_tag(..PAGECACHE_TAG_DIRTY,..)
foreach(page)
shrink_page_list
lock_page
ext4_releasepage
try_to_free_buffers
drop_buffers
cancel_dirty_page
unlock_page
lock_page()
BUG_ON(!PagePrivate(page));

How about the below ? I am not sure journalled mode writepage would need a
simillar change. We don't need to redirty the page because
that would mean the next journal_commit will hit the BUG_ON.
If we need to do a BUG_ON

diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index fd67b34..94e5ab0 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1662,8 +1662,17 @@ static int ext4_da_writepage(struct page *page,
* writing the page in case we would have to do so.
* We reach here also via journal_submit_inode_data_buffers
*/
+ if (!page_has_buffers(page)) {
+ /*
+ * This can happen when we are called via
+ * journal_submit_inode_data_buffers and
+ * shrink_page_list parallely did drop_buffers
+ * after write_cache_pages did a pagevec_lookup_tag
+ */
+ unlock_page(page);
+ return 0;
+ }
size = i_size_read(inode);

2008-06-27 09:50:43

[permalink] [raw]

Subject: Re: Performance of ext4

On Fri, Jun 27, 2008 at 02:44:59PM +0530, Aneesh Kumar K.V wrote:
> On Wed, Jun 25, 2008 at 05:46:39PM -0700, Mingming wrote:
> >
> > On Wed, 2008-06-25 at 09:09 +0000, Holger Kiehl wrote:
> > > On Tue, 24 Jun 2008, Mingming wrote:
> > >
> > > >
> > > > On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
> > > >> Yes, with this patch applied on top of latest patch queue I no longer
> > > >> get truncated files, after running a short test. Tomorrow I will do some
> > > >> more thorough testing and use the patch you have send to me in a separate
> > > >> mail. The above patch did not apply but it was easy to apply by hand.
> > > >
> > > >
> > > > Thanks for quick response and test. I have updated the patch queue with
> > > > above patch merged. Please let me know if you still see apply issue and
> > > > file size update issue with current patch queue.
> > > >
> > > Thanks, it applies without any problems. However I still hit an oops. What
> > > I find strange is that I got the oops just as the benchmark is done and
> > > all process where shutting down. The same behaviour I reported here:
> > > http://www.ussg.iu.edu/hypermail/linux/kernel/0806.2/2113.html
> > > Only this time I got just one oops. This is on x86_64 system (4 Opteron CPU's
> > > and SW Raid 1+0). I have not seen this on my home system x86 (1 Dual Core
> > > and HW Raid). Anyway, here the dmesg output:
> > >
> > > kjournald2 starting. Commit interval 15 seconds
> > > EXT4 FS on md7, internal journal
> > > EXT4-fs: mounted filesystem with ordered data mode.
> > > EXT4-fs: file extents enabled
> > > EXT4-fs: mballoc enabled
> > > JBD: barrier-based sync failed on md7 - disabling barriers
> > > ------------[ cut here ]------------
> > > kernel BUG at fs/ext4/inode.c:1667!
> >
> > Did not get a chance to look more closely today, but it's point to this
> > code in ext4_da_writepage()
> >
> > page_bufs = page_buffers(page);
> >
> > and appearently it's BUG_ON at
> > BUG_ON(!PagePrivate(page)); in page_buffers().
> >
> >
>
> Ok so we are doing the journal_commit and meanwhile shrink_page_list
> dropped the buffer. I guess what is happening is
>
>
> journal_submit_inode_data_buffers
> generic_writepages
> write_cache_pages
> pagevec_lookup_tag(..PAGECACHE_TAG_DIRTY,..)
> foreach(page)
> shrink_page_list
> lock_page
> ext4_releasepage
> try_to_free_buffers
> drop_buffers
> cancel_dirty_page
> unlock_page
> lock_page()
> BUG_ON(!PagePrivate(page));
>
>
> How about the below ?

or update write_cache_pages not to call writepage if the page is not
dirty ?

diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index ded57d5..0a13702 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -929,6 +929,11 @@ int write_cache_pages(struct address_space *mapping,
continue;
}

+ if (!PageDirty(page)) {
+ unlock_page(page);
+ continue;
+ }
+
ret = (*writepage)(page, wbc, data);

if (unlikely(ret == AOP_WRITEPAGE_ACTIVATE)) {

2008-06-27 10:00:26

by Jan Kara

[permalink] [raw]

Subject: Re: Performance of ext4

On Fri 27-06-08 15:19:13, Aneesh Kumar K.V wrote:
> On Fri, Jun 27, 2008 at 02:44:59PM +0530, Aneesh Kumar K.V wrote:
> > On Wed, Jun 25, 2008 at 05:46:39PM -0700, Mingming wrote:
> > >
> > > On Wed, 2008-06-25 at 09:09 +0000, Holger Kiehl wrote:
> > > > On Tue, 24 Jun 2008, Mingming wrote:
> > > >
> > > > >
> > > > > On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
> > > > >> Yes, with this patch applied on top of latest patch queue I no longer
> > > > >> get truncated files, after running a short test. Tomorrow I will do some
> > > > >> more thorough testing and use the patch you have send to me in a separate
> > > > >> mail. The above patch did not apply but it was easy to apply by hand.
> > > > >
> > > > >
> > > > > Thanks for quick response and test. I have updated the patch queue with
> > > > > above patch merged. Please let me know if you still see apply issue and
> > > > > file size update issue with current patch queue.
> > > > >
> > > > Thanks, it applies without any problems. However I still hit an oops. What
> > > > I find strange is that I got the oops just as the benchmark is done and
> > > > all process where shutting down. The same behaviour I reported here:
> > > > http://www.ussg.iu.edu/hypermail/linux/kernel/0806.2/2113.html
> > > > Only this time I got just one oops. This is on x86_64 system (4 Opteron CPU's
> > > > and SW Raid 1+0). I have not seen this on my home system x86 (1 Dual Core
> > > > and HW Raid). Anyway, here the dmesg output:
> > > >
> > > > kjournald2 starting. Commit interval 15 seconds
> > > > EXT4 FS on md7, internal journal
> > > > EXT4-fs: mounted filesystem with ordered data mode.
> > > > EXT4-fs: file extents enabled
> > > > EXT4-fs: mballoc enabled
> > > > JBD: barrier-based sync failed on md7 - disabling barriers
> > > > ------------[ cut here ]------------
> > > > kernel BUG at fs/ext4/inode.c:1667!
> > >
> > > Did not get a chance to look more closely today, but it's point to this
> > > code in ext4_da_writepage()
> > >
> > > page_bufs = page_buffers(page);
> > >
> > > and appearently it's BUG_ON at
> > > BUG_ON(!PagePrivate(page)); in page_buffers().
> > >
> > >
> >
> > Ok so we are doing the journal_commit and meanwhile shrink_page_list
> > dropped the buffer. I guess what is happening is
> >
> >
> > journal_submit_inode_data_buffers
> > generic_writepages
> > write_cache_pages
> > pagevec_lookup_tag(..PAGECACHE_TAG_DIRTY,..)
> > foreach(page)
> > shrink_page_list
> > lock_page
> > ext4_releasepage
> > try_to_free_buffers
> > drop_buffers
> > cancel_dirty_page
> > unlock_page
> > lock_page()
> > BUG_ON(!PagePrivate(page));
> >
> >
> > How about the below ?
>
> or update write_cache_pages not to call writepage if the page is not
> dirty ?
But that is already happening :) Look a few lines above your patch into
clear_page_dirty_for_io()...

Honza

> diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> index ded57d5..0a13702 100644
> --- a/mm/page-writeback.c
> +++ b/mm/page-writeback.c
> @@ -929,6 +929,11 @@ int write_cache_pages(struct address_space *mapping,
> continue;
> }
>
> + if (!PageDirty(page)) {
> + unlock_page(page);
> + continue;
> + }
> +
> ret = (*writepage)(page, wbc, data);
>
> if (unlikely(ret == AOP_WRITEPAGE_ACTIVATE)) {
--
Jan Kara <[email protected]>
SUSE Labs, CR

2008-06-27 17:36:04

[permalink] [raw]

Subject: Re: Performance of ext4

On Fri, Jun 27, 2008 at 12:00:24PM +0200, Jan Kara wrote:
> On Fri 27-06-08 15:19:13, Aneesh Kumar K.V wrote:
> > On Fri, Jun 27, 2008 at 02:44:59PM +0530, Aneesh Kumar K.V wrote:
> > > On Wed, Jun 25, 2008 at 05:46:39PM -0700, Mingming wrote:
> > > >
> > > > On Wed, 2008-06-25 at 09:09 +0000, Holger Kiehl wrote:
> > > > > On Tue, 24 Jun 2008, Mingming wrote:
> > > > >
> > > > > >
> > > > > > On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
> > > > > >> Yes, with this patch applied on top of latest patch queue I no longer
> > > > > >> get truncated files, after running a short test. Tomorrow I will do some
> > > > > >> more thorough testing and use the patch you have send to me in a separate
> > > > > >> mail. The above patch did not apply but it was easy to apply by hand.
> > > > > >
> > > > > >
> > > > > > Thanks for quick response and test. I have updated the patch queue with
> > > > > > above patch merged. Please let me know if you still see apply issue and
> > > > > > file size update issue with current patch queue.
> > > > > >
> > > > > Thanks, it applies without any problems. However I still hit an oops. What
> > > > > I find strange is that I got the oops just as the benchmark is done and
> > > > > all process where shutting down. The same behaviour I reported here:
> > > > > http://www.ussg.iu.edu/hypermail/linux/kernel/0806.2/2113.html
> > > > > Only this time I got just one oops. This is on x86_64 system (4 Opteron CPU's
> > > > > and SW Raid 1+0). I have not seen this on my home system x86 (1 Dual Core
> > > > > and HW Raid). Anyway, here the dmesg output:
> > > > >
> > > > > kjournald2 starting. Commit interval 15 seconds
> > > > > EXT4 FS on md7, internal journal
> > > > > EXT4-fs: mounted filesystem with ordered data mode.
> > > > > EXT4-fs: file extents enabled
> > > > > EXT4-fs: mballoc enabled
> > > > > JBD: barrier-based sync failed on md7 - disabling barriers
> > > > > ------------[ cut here ]------------
> > > > > kernel BUG at fs/ext4/inode.c:1667!
> > > >
> > > > Did not get a chance to look more closely today, but it's point to this
> > > > code in ext4_da_writepage()
> > > >
> > > > page_bufs = page_buffers(page);
> > > >
> > > > and appearently it's BUG_ON at
> > > > BUG_ON(!PagePrivate(page)); in page_buffers().
> > > >
> > > >
> > >

How about this ?

commit 174d555d8effb480a23d5dea8db698d1bc2cfa7d
Author: Aneesh Kumar K.V <[email protected]>
Date: Fri Jun 27 23:04:28 2008 +0530

ext4: call ext4_page_mkwrite even for MappedToDisk pages

We can have pages that are fully mapped to disk. The
mappedtodisk flag is used to indicate that every
buffer_head in the page have a mapping block allocated
on disk. But that doesn't gurantee that we have initialized
those buffer_head and added it to page via page->private.
This causes writepage to go BUGON when it find a page
that have NULL page->private.

The fix is to make sure we initialize the buffer_head and add
it to page when we are going to write to the page. This can
be done via ext4_page_mkwrite

Signed-off-by: Aneesh Kumar K.V <[email protected]>

diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 10f1d5d..11ebe88 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -3978,8 +3978,6 @@ int ext4_page_mkwrite(struct vm_area_struct *vma, struct page *page)
goto out_unlock;
}
ret = 0;
- if (PageMappedToDisk(page))
- goto out_unlock;

if (page->index == size >> PAGE_CACHE_SHIFT)
len = size & ~PAGE_CACHE_MASK;

2008-07-07 13:13:49

[permalink] [raw]

Subject: Re: Performance of ext4

On Thu, 19 Jun 2008, Theodore Tso wrote:

> On Wed, Jun 18, 2008 at 05:58:00AM +0000, Holger Kiehl wrote:
>> For afdbench: 5336.41 files per second 15.63 MiB/s
>>
>> So it seems that for afdbench the ext4-patch-queue is a slowdown.
>
> Can you remind me where afdbench can be downloaded? And if I remember
> correctly, it creates and deletes large numbers of small files,
> correct?
>
> It would be interesting to see which new feature introduced by the
> ext4 patch queue --- probably dellayed allocation or mballoc --- is
> responsible for the slowdown. One or the other (or both) can be
> disabled by mounting the filesystem (using a kernel with the ext4
> patch queue) with the mount options -O nomballoc or -O nodelalloc.
>
> If it turns out that nomballoc restores the speed for afdbench, for
> example, then it will tell us where we need to look more closely.
> Ideally we would not want to have one mount option needed to optimize
> filesystem operations for large amoutns of modifications to small
> files, and another mode of operation when mostly writing to large
> files. So if you could do a round of tests using the ext4 patch queue
> kernel, with -O nomballoc and -O nodelalloc (and if both seem to
> improve things, try "-O nomballoc,nodelalloc" and see if you get back
> to the pre-ext4 patch queue speed), it would be very much appreciated.
>
Here the results:
+---------+------------+
| afdbench| bonnie++ |
+---------+--------+---+
|file rate| block w|%CP|
-------------------------------------+---------+--------+---+
ext3 | 5536.79 | 212350 | 92|
ext4-patch-queue | 5054.86 | 244384 | 50|
ext4-patch-queue-nodelalloc | 4943.78 | 225819 | 92|
ext4-patch-queue-nomballoc | 3123.49 | 244535 | 52|
ext4-patch-queue-nomballoc-nodelalloc| 4931.09 | 231332 | 91|
-------------------------------------+---------+--------+---+

Test where done with 2.6.26-rc8 and
ext4-patch-queue-52c8a02a8a7b7e5915b9301e9c171b4faf22b928.tar.gz,
e2fsprogs is from git (27th April 2008). ext4 filesystem was created
with 'mke2fs -m 0 -t ext4dev /dev/md7' and ext3 'mke2fs -m 0 -j /dev/md7'.
Common mount options are: noatime,nodiratime,commit=15

Looking at the afdbench results I also notice that when I just take
the FTP results the results look as follows:

ext3 : 3465.50
ext4-patch-queue : 2785.58
ext4-patch-queue-nodelalloc : 2677.39
ext4-patch-queue-nomballoc : 219.12
ext4-patch-queue-nomballoc-nodelalloc: 2566.24

Now one can see the drop with ext4-patch-queue much clearer. The testing
of afdbench is done in two parts, one where we just link lots of small
files locally and the same test is then repeated using a network protocol
in this case FTP. So the difference is that for the filesystem lots
of new files get created. Further testing showed that when I increase
the number FTP process performance decreases in all cases but much more
for ext4-patch-queue (nearly 50% drop against ext3) as the following results
show:

ext3 : 2352.89
ext4-patch-queue : 1226.55
ext4-patch-queue-nodelalloc : 1340.80
ext4-patch-queue-nomballoc-nodelalloc: 1181.12

I did not do the ext4-patch-queue-nomballoc test since there is obviously
something wrong here when you look at the numbers above (219.12 fps).
During that test I notice that when you try to open an existing file
with vi it can take several minutes before it opens this file. The strange
thing is that vi was not in D-state but it could not be killed, even root
could not kill it with -9.

There is also some corruption in filesystem during the test with
ext4-patch-queue and ext4-patch-queue-nomballoc. It happens when after
the test I umount the test filesystem and then mount it again the
following message appears:

root@athena:~# umount /home
root@athena:~# mount /home
mount: wrong fs type, bad option, bad superblock on /dev/md7,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

EXT4-fs: ext4_check_descriptors: Inode bitmap for group 256 not in group (block 117835012)!<3>EXT4-fs: group descriptors corrupted!

Using fsck this problem could be corrected. Now that one does not think I
did those test on a corrupted file system. The filesystem was newly created
for each of the above five test runs.

Regards,
Holger

2008-07-10 08:11:23

[permalink] [raw]

Subject: Re: Performance of ext4

On Mon, 7 Jul 2008, Holger Kiehl wrote:

> On Thu, 19 Jun 2008, Theodore Tso wrote:
>
>> On Wed, Jun 18, 2008 at 05:58:00AM +0000, Holger Kiehl wrote:
>>> For afdbench: 5336.41 files per second 15.63 MiB/s
>>>
>>> So it seems that for afdbench the ext4-patch-queue is a slowdown.
>>
>> Can you remind me where afdbench can be downloaded? And if I remember
>> correctly, it creates and deletes large numbers of small files,
>> correct?
>>
>> It would be interesting to see which new feature introduced by the
>> ext4 patch queue --- probably dellayed allocation or mballoc --- is
>> responsible for the slowdown. One or the other (or both) can be
>> disabled by mounting the filesystem (using a kernel with the ext4
>> patch queue) with the mount options -O nomballoc or -O nodelalloc.
>>
>> If it turns out that nomballoc restores the speed for afdbench, for
>> example, then it will tell us where we need to look more closely.
>> Ideally we would not want to have one mount option needed to optimize
>> filesystem operations for large amoutns of modifications to small
>> files, and another mode of operation when mostly writing to large
>> files. So if you could do a round of tests using the ext4 patch queue
>> kernel, with -O nomballoc and -O nodelalloc (and if both seem to
>> improve things, try "-O nomballoc,nodelalloc" and see if you get back
>> to the pre-ext4 patch queue speed), it would be very much appreciated.
>>
> Here the results:
> +---------+------------+
> | afdbench| bonnie++ |
> +---------+--------+---+
> |file rate| block w|%CP|
> -------------------------------------+---------+--------+---+
> ext3 | 5536.79 | 212350 | 92|
> ext4-patch-queue | 5054.86 | 244384 | 50|
> ext4-patch-queue-nodelalloc | 4943.78 | 225819 | 92|
> ext4-patch-queue-nomballoc | 3123.49 | 244535 | 52|
> ext4-patch-queue-nomballoc-nodelalloc| 4931.09 | 231332 | 91|
> -------------------------------------+---------+--------+---+
>
> Test where done with 2.6.26-rc8 and
> ext4-patch-queue-52c8a02a8a7b7e5915b9301e9c171b4faf22b928.tar.gz,
> e2fsprogs is from git (27th April 2008). ext4 filesystem was created
> with 'mke2fs -m 0 -t ext4dev /dev/md7' and ext3 'mke2fs -m 0 -j /dev/md7'.
> Common mount options are: noatime,nodiratime,commit=15
>
> Looking at the afdbench results I also notice that when I just take
> the FTP results the results look as follows:
>
> ext3 : 3465.50
> ext4-patch-queue : 2785.58
> ext4-patch-queue-nodelalloc : 2677.39
> ext4-patch-queue-nomballoc : 219.12
> ext4-patch-queue-nomballoc-nodelalloc: 2566.24
>
> Now one can see the drop with ext4-patch-queue much clearer. The testing
> of afdbench is done in two parts, one where we just link lots of small
> files locally and the same test is then repeated using a network protocol
> in this case FTP. So the difference is that for the filesystem lots
> of new files get created. Further testing showed that when I increase
> the number FTP process performance decreases in all cases but much more
> for ext4-patch-queue (nearly 50% drop against ext3) as the following results
> show:
>
> ext3 : 2352.89
> ext4-patch-queue : 1226.55
> ext4-patch-queue-nodelalloc : 1340.80
> ext4-patch-queue-nomballoc-nodelalloc: 1181.12
>
> I did not do the ext4-patch-queue-nomballoc test since there is obviously
> something wrong here when you look at the numbers above (219.12 fps).
> During that test I notice that when you try to open an existing file
> with vi it can take several minutes before it opens this file. The strange
> thing is that vi was not in D-state but it could not be killed, even root
> could not kill it with -9.
>
> There is also some corruption in filesystem during the test with
> ext4-patch-queue and ext4-patch-queue-nomballoc. It happens when after
> the test I umount the test filesystem and then mount it again the
> following message appears:
>
> root@athena:~# umount /home
> root@athena:~# mount /home
> mount: wrong fs type, bad option, bad superblock on /dev/md7,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> EXT4-fs: ext4_check_descriptors: Inode bitmap for group 256 not in group
> (block 117835012)!<3>EXT4-fs: group descriptors corrupted!
>
> Using fsck this problem could be corrected. Now that one does not think I
> did those test on a corrupted file system. The filesystem was newly created
> for each of the above five test runs.
>
Any ideas what I can do to help find why performance under load is nearly
halved and the group descriptor corruption?

I did try newer patch queue (ext4-patch-queue-a5d48915447f44c3af6ce8e1c91d45b452977fcf)
from today, but I immediatly hit an oops as soon as I untar a file, see below.

Thanks,
Holger

kjournald2 starting. Commit interval 15 seconds
EXT4 FS on md7, internal journal
EXT4-fs: mounted filesystem with ordered data mode.
EXT4-fs: delayed allocation enabled
EXT4-fs: file extents enabled
EXT4-fs: mballoc enabled
------------[ cut here ]------------
kernel BUG at fs/ext4/extents.c:1817!
invalid opcode: 0000 [1] SMP
CPU 0
Modules linked in: w83627hf lm85 hwmon_vid bonding nf_conntrack_ftp ipt_REJECT xt_tcpudp nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables binfmt_misc floppy i2c_amd756 i2c_core k8temp ohci_hcd sg button usbcore
Pid: 2757, comm: tar Not tainted 2.6.26-rc9 #1
RIP: 0010:[<ffffffff802e2722>] [<ffffffff802e2722>] ext4_ext_get_blocks+0x9eb/0xde1
RSP: 0018:ffff81007a0f99b8 EFLAGS: 00010246
RAX: 0000000000000001 RBX: ffff81002cfd69c0 RCX: ffff81002cfd69a8
RDX: ffff81007f48c6fc RSI: 00000000ffffffff RDI: ffff81002cfd69c0
RBP: ffff81007a0f9b88 R08: ffff81007f48c6fc R09: 0000000000000000
R10: 000000000000a855 R11: 0000000000000000 R12: ffff81007f48c7b0
R13: 0000000000000001 R14: ffff81007f48c7b0 R15: 0000000000000001
FS: 00007f66afd3b780(0000) GS:ffffffff80570000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000000081d000 CR3: 00000001e9e86000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process tar (pid: 2757, threadinfo ffff81007a0f8000, task ffff81007d9110e0)
Stack: ffff81007d36c300 000000007f46e030 ffff81007a0f9b88 0000000000000001
000000012c815bc0 ffff81007f46e030 ffff81002cfd69c0 000000007d36c300
ffff81007a0f9bb8 ffff81007f48c6f0 000000007a0f9bc8 ffff81007f46e030
Call Trace:
[<ffffffff802d2aaa>] ? ext4_mark_inode_dirty+0x134/0x147
[<ffffffff80223c42>] ? __wake_up+0x38/0x4f
[<ffffffff802d4e0b>] ? ext4_get_blocks_wrap+0x70/0x165
[<ffffffff8031af55>] ? __up_read+0x13/0x8a
[<ffffffff802d5280>] ? ext4_getblk+0x62/0x170
[<ffffffff802d7801>] ? add_dirent_to_buf+0xcb/0x2ec
[<ffffffff802d539b>] ? ext4_bread+0xd/0x5f
[<ffffffff802d7206>] ? ext4_append+0x3a/0x88
[<ffffffff802d8042>] ? ext4_add_entry+0x620/0x87f
[<ffffffff802d12ce>] ? ext4_new_inode+0xc4e/0xc78
[<ffffffff802f58f3>] ? start_this_handle+0x2c7/0x370
[<ffffffff802d8916>] ? ext4_add_nondir+0x18/0x4e
[<ffffffff802d8ff8>] ? ext4_create+0xc2/0x105
[<ffffffff802d9288>] ? ext4_lookup+0x97/0xc1
[<ffffffff802823d6>] ? vfs_create+0x75/0xba
[<ffffffff80284e5d>] ? do_filp_open+0x1e4/0x7f6
[<ffffffff80279e7e>] ? sys_chown+0x5c/0x6b
[<ffffffff80279684>] ? do_sys_open+0x46/0xca
[<ffffffff8020b16b>] ? system_call_after_swapgs+0x7b/0x80

Code: 39 44 24 24 72 2f 66 81 fa 00 80 0f b7 c2 76 05 2d 00 80 00 00 48 8b 7c 24 30 01 f0 89 44 24 24 e8 71 d3 ff ff 3b 44 24 24 75 04 <0f> 0b eb fe 2b 44 24 24 eb 11 0f 0b eb fe c7 44 24 24 00 00 00
RIP [<ffffffff802e2722>] ext4_ext_get_blocks+0x9eb/0xde1
RSP <ffff81007a0f99b8>
---[ end trace e595ecd19e9f2f92 ]---

2008-07-10 09:25:07

[permalink] [raw]

Subject: Re: Performance of ext4

On Thu, Jul 10, 2008 at 08:11:15AM +0000, Holger Kiehl wrote:
> On Mon, 7 Jul 2008, Holger Kiehl wrote:
>
>
> I did try newer patch queue (ext4-patch-queue-a5d48915447f44c3af6ce8e1c91d45b452977fcf)
> from today, but I immediatly hit an oops as soon as I untar a file, see below.
>

I am able to reproduce this. Revert the patch Fix-EXT_MAX_BLOCK.patch.

-aneesh

2008-07-10 09:27:12

[permalink] [raw]

Subject: Revert Fix-EXT_MAX_BLOCK.patch

Sending again with different subject.

On Thu, Jul 10, 2008 at 02:54:25PM +0530, Aneesh Kumar K.V wrote:
> On Thu, Jul 10, 2008 at 08:11:15AM +0000, Holger Kiehl wrote:
> > On Mon, 7 Jul 2008, Holger Kiehl wrote:
> >
> >
> > I did try newer patch queue (ext4-patch-queue-a5d48915447f44c3af6ce8e1c91d45b452977fcf)
> > from today, but I immediatly hit an oops as soon as I untar a file, see below.
> >
>
> I am able to reproduce this. Revert the patch Fix-EXT_MAX_BLOCK.patch.
>
> -aneesh

2008-07-10 12:23:29

[permalink] [raw]

Subject: Re: Revert Fix-EXT_MAX_BLOCK.patch

Aneesh, for the record, how were you able to reproduce the problem?

Do you have Holger's afdbench setup, or did you use some other workload?

Thanks, regards,

- Ted

On Thu, Jul 10, 2008 at 02:56:35PM +0530, Aneesh Kumar K.V wrote:
> Sending again with different subject.
>
> On Thu, Jul 10, 2008 at 02:54:25PM +0530, Aneesh Kumar K.V wrote:
> > On Thu, Jul 10, 2008 at 08:11:15AM +0000, Holger Kiehl wrote:
> > > On Mon, 7 Jul 2008, Holger Kiehl wrote:
> > >
> > >
> > > I did try newer patch queue (ext4-patch-queue-a5d48915447f44c3af6ce8e1c91d45b452977fcf)
> > > from today, but I immediatly hit an oops as soon as I untar a file, see below.
> > >
> >
> > I am able to reproduce this. Revert the patch Fix-EXT_MAX_BLOCK.patch.
> >
> > -aneesh
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2008-07-10 12:24:50

[permalink] [raw]

Subject: Re: Revert Fix-EXT_MAX_BLOCK.patch

Ive commented out Fix-EXT_MAX_BLOCK.patch from the series file until
we know more.

- Ted

On Thu, Jul 10, 2008 at 02:56:35PM +0530, Aneesh Kumar K.V wrote:
> Sending again with different subject.
>
> On Thu, Jul 10, 2008 at 02:54:25PM +0530, Aneesh Kumar K.V wrote:
> > On Thu, Jul 10, 2008 at 08:11:15AM +0000, Holger Kiehl wrote:
> > > On Mon, 7 Jul 2008, Holger Kiehl wrote:
> > >
> > >
> > > I did try newer patch queue (ext4-patch-queue-a5d48915447f44c3af6ce8e1c91d45b452977fcf)
> > > from today, but I immediatly hit an oops as soon as I untar a file, see below.
> > >
> >
> > I am able to reproduce this. Revert the patch Fix-EXT_MAX_BLOCK.patch.
> >
> > -aneesh
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2008-07-10 12:38:15

[permalink] [raw]

Subject: Re: Revert Fix-EXT_MAX_BLOCK.patch

On Thu, Jul 10, 2008 at 08:22:42AM -0400, Theodore Tso wrote:
> Aneesh, for the record, how were you able to reproduce the problem?
>
> Do you have Holger's afdbench setup, or did you use some other workload?
>

cp -ax /usr .

that will cause the BUG

-aneesh

2008-07-10 13:02:26

[permalink] [raw]

Subject: Re: Revert Fix-EXT_MAX_BLOCK.patch

On Thu, Jul 10, 2008 at 06:08:03PM +0530, Aneesh Kumar K.V wrote:
> On Thu, Jul 10, 2008 at 08:22:42AM -0400, Theodore Tso wrote:
> > Aneesh, for the record, how were you able to reproduce the problem?
> >
> > Do you have Holger's afdbench setup, or did you use some other workload?
> >
>
> cp -ax /usr .
>
> that will cause the BUG

Hmm, I wasn't able to trigger it so easily, at least, while trying to
replicate Gary Hawco's, bug. (Which also had Fix-EXT_MAX_BLOCK
applied.) I wonder if he wasn't able to trigger it while doing the
partial rollback, and so ext4-fix-mb_find_next_bit-return.patch was
unfairly blamed for the BUG.

In any case, both patches have been pulled out of the patch series ,
and I'll release a new patchset until we can figure out what happened.

(Gary, this is why its important to get the kernel BUG/oops message;
it helps us correlate your bug reports with others.)

- Ted

2008-07-11 09:57:51

[permalink] [raw]

Subject: Re: Revert Fix-EXT_MAX_BLOCK.patch

On Thu, 10 Jul 2008, Aneesh Kumar K.V wrote:

> Sending again with different subject.
>
> On Thu, Jul 10, 2008 at 02:54:25PM +0530, Aneesh Kumar K.V wrote:
>> On Thu, Jul 10, 2008 at 08:11:15AM +0000, Holger Kiehl wrote:
>>> On Mon, 7 Jul 2008, Holger Kiehl wrote:
>>>
>>>
>>> I did try newer patch queue (ext4-patch-queue-a5d48915447f44c3af6ce8e1c91d45b452977fcf)
>>> from today, but I immediatly hit an oops as soon as I untar a file, see below.
>>>
>>
>> I am able to reproduce this. Revert the patch Fix-EXT_MAX_BLOCK.patch.
>>
Thanks. Reverting that patch also fixed it for me. I was able to do my
test however performance is down another 10% (compared to
ext4-patch-queue-52c8a02a8a7b7e5915b9301e9c171b4faf22b928). ext4 is getting
slower and slower :(

Also the group descriptors still get corrupted.

Holger

PS: http://repo.or.cz/w/ext4-patch-queue.git is empty, is that correct?

2008-07-11 12:43:49

[permalink] [raw]

Subject: Re: Revert Fix-EXT_MAX_BLOCK.patch

On Fri, Jul 11, 2008 at 09:57:47AM +0000, Holger Kiehl wrote:
> Thanks. Reverting that patch also fixed it for me. I was able to do my
> test however performance is down another 10% (compared to
> ext4-patch-queue-52c8a02a8a7b7e5915b9301e9c171b4faf22b928). ext4 is getting
> slower and slower :(

How reproducible are your results? That is, if you run the benchmarks
multiple times, how much variance is there between different runs?

If you are willing, this would be helpful. In your ext4 patch
repository, try out commit 179a876b. (You can do this via
"git checkout -b rc9-rebase 179a876b"; after doing the test you can
switch the working directory of the ext4 patch queue back to the master
branch via "git checkout master".) This commit is pretty much
identical to your previous 52c8a02a test, modulo rebasing to -rc9.

If you see the same results, you could try going to the next patch,
via "git checkout -b i-blocks-stat ef019f0a" which also has the fix so
that stat returns a valid i_blocks field for files that have been
freshly written when delayed allocation is enabled. Both of these
revisions rae before the patches that were causing corrupion were
added to the patch queue, so it should be fine.

The funny thing is looking at the various recent patches, I don't see
how they could be affecting performance of your patches so
significantly. I gather afdbench is very metadata intensive, with
lots of small files, but even so, none of these patches should make
that kind of difference. So that's why I'm wondering how much
variance there is between runs of afdbench, and whether that might be
a possible explanation.

> Also the group descriptors still get corrupted.

Hmm, can you send me the output of dumpe2fs before and after the
benchmark run which corrupts the group descriptors? And can you send
me the output of "e2fsck -fy /dev/XXXXX >& /tmp/log", so I can see
what got corrupted?

I also note that you are using a fairly old e2fsprogs from April 27th.
You might want to try going to the just-released e2fsprogs 1.41.0
released yesterday, as that has some flex_bg layout changes that might
help out performance for afdbench. Also note that with both the April
27th and the latest e2fsprogs 1.41.0 release, there is a mke2fs.conf
file in misc/mke2fs.conf that should be installed in /etc/mke2fs.conf
for best results.

> PS: http://repo.or.cz/w/ext4-patch-queue.git is empty, is that correct?

Nope, we're working on that. Things seem to have gotten corrupted on
repo.or.cz, as you may have seen on another e-mail thread. I have
established an git repository here:

git://git.kernel.org/pub/scm/fs/ext2/ext4-patch-queue.git

As an interim replacement.

- Ted

2008-07-11 14:57:54

[permalink] [raw]

Subject: Re: Revert Fix-EXT_MAX_BLOCK.patch

On Fri, 11 Jul 2008, Theodore Tso wrote:

> On Fri, Jul 11, 2008 at 09:57:47AM +0000, Holger Kiehl wrote:
>> Thanks. Reverting that patch also fixed it for me. I was able to do my
>> test however performance is down another 10% (compared to
>> ext4-patch-queue-52c8a02a8a7b7e5915b9301e9c171b4faf22b928). ext4 is getting
>> slower and slower :(
>
> How reproducible are your results? That is, if you run the benchmarks
> multiple times, how much variance is there between different runs?
>
I always run the benchmark 3 times and then take the average. Additionally
there are two main types of test one where files are moved locally (FILE)
and another where files are send via FTP. So its 6 runs with the following
results:

FTP FILE
2548.81 6569.68
2613.05 6480.86
2599.09 6573.62

I then took all six runs added them and diveded by six giving 4564.19 fps.
Those results where achived with a5d48915 and e2fsprogs-1.41-WIP-0707.tar.bz2.

The same was done with 52c8a02a only here I used the April 27th e2fsprogs.
There I got 5054.86 fps.

Each run takes 30 minutes and approx. 10 minutes to delete the test files
from a previous run and setup the new test files, that is approx. 4 hours
for all 6 runs. I then also always do a 2 hour test run with a lot more
files and process sending files, one for FTP and one for FILE. But I
did not mention those numbers because I always did it once. But here too
one could see the numbers going down by approx. 10%.

> If you are willing, this would be helpful. In your ext4 patch
> repository, try out commit 179a876b. (You can do this via
> "git checkout -b rc9-rebase 179a876b"; after doing the test you can
> switch the working directory of the ext4 patch queue back to the master
> branch via "git checkout master".) This commit is pretty much
> identical to your previous 52c8a02a test, modulo rebasing to -rc9.
>
That is why I did another test run with ext3 which I did not mention, sorry.
Here the results:

ext3 ext4-patch-queue
52c8a02a 5536.79 5054.86
a5d48915 5587.78 4564.19

So the result of ext3 are the same while ext4-patch-queue dropped the
nearly 10%.

> If you see the same results, you could try going to the next patch,
> via "git checkout -b i-blocks-stat ef019f0a" which also has the fix so
> that stat returns a valid i_blocks field for files that have been
> freshly written when delayed allocation is enabled. Both of these
> revisions rae before the patches that were causing corrupion were
> added to the patch queue, so it should be fine.
>
> The funny thing is looking at the various recent patches, I don't see
> how they could be affecting performance of your patches so
> significantly. I gather afdbench is very metadata intensive, with
> lots of small files, but even so, none of these patches should make
> that kind of difference. So that's why I'm wondering how much
> variance there is between runs of afdbench, and whether that might be
> a possible explanation.
>
>> Also the group descriptors still get corrupted.
>
> Hmm, can you send me the output of dumpe2fs before and after the
> benchmark run which corrupts the group descriptors? And can you send
> me the output of "e2fsck -fy /dev/XXXXX >& /tmp/log", so I can see
> what got corrupted?
>
> I also note that you are using a fairly old e2fsprogs from April 27th.
> You might want to try going to the just-released e2fsprogs 1.41.0
> released yesterday, as that has some flex_bg layout changes that might
> help out performance for afdbench.
>
Where those changes already in e2fsprogs-1.41-WIP-0707.tar.bz2 release?

> Also note that with both the April
> 27th and the latest e2fsprogs 1.41.0 release, there is a mke2fs.conf
> file in misc/mke2fs.conf that should be installed in /etc/mke2fs.conf
> for best results.
>
Since I made my own src rpm, I did use the misc/mke2fs.conf in both cases.
Just checked this.

Holger

2008-07-14 19:55:13

[permalink] [raw]

Subject: Re: Revert Fix-EXT_MAX_BLOCK.patch

Attachments:

dumpe2fs.after4.bz2 (49.70 kB)
e2fsck.log4.bz2 (959.00 B)
Download all attachments

2008-07-14 20:28:34

[permalink] [raw]

Subject: Re: Revert Fix-EXT_MAX_BLOCK.patch

On Mon, Jul 14, 2008 at 07:55:10PM +0000, Holger Kiehl wrote:
> You are right. I did compare the .config of both and noticed that
> CONFIG_FAIR_GROUP_SCHED was set in the rc9 test but not in rc8 test.
> Doing the test without CONFIG_FAIR_GROUP_SCHED gave me back 6%.
> Sorry.

You may have told us already, but can you tell me the full
configuration of your benchmark machine? (i.e., how many CPU's, how
much memory, etc.) Also, what are the current mount options you are
currently using? And have you redone the ext3 benchmark number with
barriers enabled? Or was that the original number done with default
mount options that leave barriers disabled?

Thanks!!

- Ted

2008-07-15 06:43:25