Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2993698AbbEOXiX (ORCPT ); Fri, 15 May 2015 19:38:23 -0400 Received: from ipmail06.adl6.internode.on.net ([150.101.137.145]:35136 "EHLO ipmail06.adl6.internode.on.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2992566AbbEOXiN (ORCPT ); Fri, 15 May 2015 19:38:13 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BiBwD1glZVPPDOLHlcgxCBMoZMrBcBAQEBBplLAgIBAQKBLE0BAQEBAQEHAQEBAUE/hCIBAQEDATocFg0FCwgDGAklDwUlAwcaExuICQfXPgEBAQcCAR8YhX6FJIUFB4QtBZ1TgSiGYYsfg1iBBIEFIQMcgWQsMYJGAQEB Date: Sat, 16 May 2015 09:38:08 +1000 From: Dave Chinner To: Linus Torvalds Cc: Al Viro , Linux Kernel Mailing List , linux-fsdevel , Christoph Hellwig , Neil Brown Subject: Re: [RFC][PATCHSET v3] non-recursive pathname resolution & RCU symlinks Message-ID: <20150515233808.GH4316@dastard> References: <20150505052205.GS889@ZenIV.linux.org.uk> <20150511180650.GA4147@ZenIV.linux.org.uk> <20150513222533.GA24192@ZenIV.linux.org.uk> <20150514033040.GF7232@ZenIV.linux.org.uk> <20150514112304.GT15721@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 7044 Lines: 159 On Thu, May 14, 2015 at 08:51:12AM -0700, Linus Torvalds wrote: > On Thu, May 14, 2015 at 4:23 AM, Dave Chinner wrote: > > > > IIRC, ext4 readdir is not slow because of the use of the buffer > > cache, it's slow because of the way it hashes dirents across blocks > > on disk. i.e. it has locality issues, not a caching problem. > > No, you're just worrying about IO. Natural for a filesystem guy, but a > lot of loads cache really well, and IO isn't an issue. Yes, there's a > bad cold-cache case, but that's not when you get inode semaphore > contention. Right, because it's cold cache performance that everyone complains about. e.g. Workloads like gluster, ceph, fileservers, openstack (e.g. swift) etc are all mostly cold cache directory workloads with *extremely high* concurrency. Nobody is complaining about cached readdir performance - concurrency in cold cache directory operations is what everyone has been asking me for. In case you missed it, recently the Ceph developers have been talking about storing file handles in a userspace database and then using open_by_handle_at() so they can avoid the pain of cold cache directory lookup overhead (see the O_NOMTIME thread). We have a serious cold cache lookup problem on directories when people are looking to bypass the directory structure entirely.... [snip a bunch of rhetoric lacking in technical merit] > End result: readdir() wastes a *lot* of time on stupid stuff (just > that physical block number lookup is generally more expensive than > readdir itself should be), and it does so with excessive locking, > serializing everything. The most overhead in readdir is calling filldir over and over again for every dirent to copy it into the user buffer. The overhead is not from looking up the buffer in the cache. So, I just created close to a million dirents in a directory, and ran the xfs_io readdir command on it (look, a readdir performance measurement tool!). I used a ram disk to take IO out of the picture for the first read, the system has E5-4620 0 @ 2.20GHz CPUs, and I dropped caches to ensure that there was no cached metadata: $ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches" $ sudo xfs_io -c readdir /mnt/scratch read 29545648 bytes from offset 0 28 MiB, 923327 ops, 0.0000 sec (111.011 MiB/sec and 3637694.9201 ops/sec) $ sudo xfs_io -c readdir /mnt/scratch read 29545648 bytes from offset 0 28 MiB, 923327 ops, 0.0000 sec (189.864 MiB/sec and 6221628.5056 ops/sec) $ sudo xfs_io -c readdir /mnt/scratch read 29545648 bytes from offset 0 28 MiB, 923327 ops, 0.0000 sec (190.156 MiB/sec and 6231201.6629 ops/sec) $ Reading, decoding and copying dirents at 190MB/s? That's roughly 6 million dirents/second being pulled from cache, and it's doing roughly 4 million/second cold cache. That's not slow at all. What *noticable* performance gains are there to be had here for the average user? Anything that takes less than a second or two to complete is not going to be noticable to a user, and most people don't have 8-10 million inodes in a directory.... So, what did the profile look like? 10.07% [kernel] [k] __xfs_dir3_data_check 9.92% [kernel] [k] copy_user_generic_string 7.44% [kernel] [k] xfs_dir_ino_validate 6.83% [kernel] [k] filldir 5.43% [kernel] [k] xfs_dir2_leaf_getdents 4.56% [kernel] [k] kallsyms_expand_symbol.constprop.1 4.38% [kernel] [k] _raw_spin_unlock_irqrestore 4.26% [kernel] [k] _raw_spin_unlock_irq 4.02% [kernel] [k] __memcpy 3.02% [kernel] [k] format_decode 2.36% [kernel] [k] xfs_dir2_data_entsize 2.28% [kernel] [k] vsnprintf 1.99% [kernel] [k] __do_softirq 1.93% [kernel] [k] xfs_dir2_data_get_ftype 1.88% [kernel] [k] number.isra.14 1.84% [kernel] [k] _xfs_buf_find 1.82% [kernel] [k] ___might_sleep 1.61% [kernel] [k] strnlen 1.49% [kernel] [k] queue_work_on 1.48% [kernel] [k] string.isra.4 1.21% [kernel] [k] __might_sleep Oh, I'm running CONFIG_XFS_DEBUG=y, so internal runtime consistency checks consume most of the CPU (__xfs_dir3_data_check, xfs_dir_ino_validate). IOWs, real world readdir performance will be much, much faster than I've demonstrated. Other than that, the most CPU is spent on copying dirents into the user buffer (copy_user_generic_string), passing dirents to the user buffer (filldir) and extracting dirents from the on-disk buffer (xfs_dir2_leaf_getdents). The we have lock contention, ramdisk IO (memcpy), some vsnprintf stuff (includes format_decode, probably debug code) and some more dirent information extraction functions. it's not until we get to _xfs_buf_find() do we see a buffer cache lookup function, and that's actually comsuming less CPU than the __might_sleep/____might_sleep() debug annotations. That puts it in persepective just how little overhead readdir buffer caching actually has compared to everything else. IOWs, these numbers indicate that readdir caching overhead has no real impact on the performance of hot cache readdir operations. So, back to the question I asked that you didn't answer: exactly what are you proposing to cache in the VFS readdir cache? Without knowing that, I can't make any sane comment on about technical merit of your proposal.... > Both readdir() and path component lookup are technically read > operations, so why the hell do we use a mutex, rather than just > get a read-write lock for reading? Yeah, it's that (d) above. I > might trust xfs and ext4 to get their internal exclusions for > allocations etc right when called concurrently for the same > directory. But the others? They just use a write lock for everything and *nothing changes* - this is a simple problem to solve. The argument "filesystem developers are stupid" is not a compelling argument against changing locking. You're just being insulting, even though you probably don't realise it. [snip more rhetoric about the page cache being the only solution] > Basically, in computer science, pretty much all performance work > is about caching. And readdir is the one area where the VFS layer > doesn't do well, falls on its face and punts back to the > filesystem. Caching is used to hide the problems of the lower layers. If the lower layers don't have a problem, then another layer of caching is not necessary. Linus, what you haven't put together is a clear statement of the problem another layer of readdir caching is going to solve. What workload is having problems? Where are the profiles demonstrating that readdir caching is the issue, or the solution to the issue you are seeing? We know about plenty of workloads where directory access concurrency is a real problem, but I'm not seeing the problem you are trying to address... Cheers, Dave. -- Dave Chinner david@fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/