Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S966012AbcCPJoO (ORCPT ); Wed, 16 Mar 2016 05:44:14 -0400 Received: from mail-oi0-f50.google.com ([209.85.218.50]:34163 "EHLO mail-oi0-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753680AbcCPJoI (ORCPT ); Wed, 16 Mar 2016 05:44:08 -0400 MIME-Version: 1.0 X-Originating-IP: [217.173.44.24] In-Reply-To: <56E7C009.3070008@gmail.com> References: <56E7C009.3070008@gmail.com> Date: Wed, 16 Mar 2016 10:44:06 +0100 Message-ID: Subject: Re: [fuse-devel] Horrible mmap write performance (kernel writeback issue?) From: Miklos Szeredi To: Jakob Unterwurzacher Cc: fuse-devel , Linux-Fsdevel , Kernel Mailing List Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4839 Lines: 123 On Tue, Mar 15, 2016 at 8:55 AM, Jakob Unterwurzacher wrote: > Just for anybody finding this thread: This still happens in v4.4, it > just took longer to trigger. > > I have posted more details to linux-kernel (copy-pasted below), > http://thread.gmane.org/gmane.linux.kernel/2132944 Okay, so you can reproduce this relatively quickly. Can you try "git bisect" to find exactly which commit is responsible? Thanks, Miklos > > -------- copy of the email to linux-kernel ------------------- > > 2016-01-22 21:10:59 > > I have noticed an annoying regression that was introduced in 4.2 and is > still there in 4.4. mmap writes to FUSE filesystems are throttled down > to basically zero. > > Reproducer: https://github.com/rfjakob/mmapwrite , testing against encfs: > > $ mmapwrite /tmp/encfs-mnt/foo > 1 .................................................. 107.01 MB/s > 2 .................................................. 101.98 MB/s > [...] > 68 .................................................. 106.79 MB/s > 69 .................................................. 105.09 MB/s > 70 .................................................. 2.02 MB/s > 71 .................................................. 1.77 MB/s > 72 .................................................. 0.42 MB/s > 73 .................................... (hangs) > > I have tested kernels from 4.0 and this seems to have been introduced in > 4.2: > > 4.0 ....... 140MB/s permanent > 4.1 ....... 140MB/s permanent > 4.2 ....... 100MB/s at the start, sudden slowdown to 1MB/s after ~5GB > 4.3 ....... 100MB/s at the start, sudden slowdown to 1MB/s after ~1.5GB > 4.4-rc4 ... 100MB/s at the start, slowly ramps down, 0.3MB/s after ~2GB > 4.4 ....... 100MB/s at the start, sudden slowdown after ~3GB > > Is there a way to disable the throttling? Or at least exempt FUSE until > there is a proper fix? > > Thanks, > Jakob > > > > > On 17.12.2015 00:26, Jakob Unterwurzacher wrote: >> This seems to be fixed in v4.4-rc5-18-gedb42dc. mmap writes now proceed >> at solid 100MB/s with full CPU saturation. >> >> Thanks, >> Jakob >> >> On Mon, Dec 14, 2015 at 9:06 AM, Jakob Unterwurzacher >> > wrote: >> >> I am the developer of https://github.com/rfjakob/gocryptfs (an >> encrypted overlay filesystem like EncFS) >> and have ported xfstests over for regression testing >> ( https://github.com/rfjakob/fuse-xfstests ). >> >> xfstests generic/074 is how I noticed that mmap write performance >> plummeted when Fedora upgraded my kernel to 4.2.5. >> It used to complete in 10 minutes and now it will probably take days. >> I am on kernel 4.4-rc1 now and still seeing the same issue. >> >> It looks like the kernel at some point the kernel writeback >> mechanism gets stuck (or throttles?) >> >> Testing against encfs: >> >> ./mmapwrite /tmp/e2/foo # >> https://github.com/rfjakob/mmapwrite . >> .................................................................................................... >> 98.91 MB/s >> .................................................................................................... >> 93.74 MB/s >> .................................................................................................... >> 103.89 MB/s >> .................................................................................................... >> 100.20 MB/s >> .................................................................................................... >> 104.03 MB/s >> .................................................................................................... >> 98.06 MB/s >> .................................................................................................... >> 10.17 MB/s >> .................................................................................................... >> 9.50 MB/s >> .................................. (hangs) >> >> At this point no write requests are submitted to encfs and the >> mmapwrite process is >> stuck in the kernel in balance_dirty_pages.isra.22. >> >> Bisecting this will be a pain, I would appreciate any suggestions. >> >> Note that this seems to affect every FUSE filesystem, also ntfs-3g. >> >> Best regards, >> Jakob >> >> > > > ------------------------------------------------------------------------------ > Transform Data into Opportunity. > Accelerate data analysis in your applications with > Intel Data Analytics Acceleration Library. > Click to learn more. > http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 > -- > fuse-devel mailing list > To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel