Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755598AbcDNOVI (ORCPT ); Thu, 14 Apr 2016 10:21:08 -0400 Received: from mx2.suse.de ([195.135.220.15]:39732 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753049AbcDNOVD (ORCPT ); Thu, 14 Apr 2016 10:21:03 -0400 Date: Thu, 14 Apr 2016 16:20:59 +0200 From: Jan Kara To: Waiman Long Cc: Alexander Viro , Jan Kara , Jeff Layton , "J. Bruce Fields" , Tejun Heo , Christoph Lameter , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Ingo Molnar , Peter Zijlstra , Andi Kleen , Dave Chinner , Boqun Feng , Scott J Norton , Douglas Hatch Subject: Re: [PATCH v7 4/4] vfs: Use per-cpu list for superblock's inode list Message-ID: <20160414142059.GF22053@quack2.suse.cz> References: <1460501686-37096-1-git-send-email-Waiman.Long@hpe.com> <1460501686-37096-5-git-send-email-Waiman.Long@hpe.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1460501686-37096-5-git-send-email-Waiman.Long@hpe.com> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1798 Lines: 48 On Tue 12-04-16 18:54:46, Waiman Long wrote: > When many threads are trying to add or delete inode to or from > a superblock's s_inodes list, spinlock contention on the list can > become a performance bottleneck. > > This patch changes the s_inodes field to become a per-cpu list with > per-cpu spinlocks. As a result, the following superblock inode list > (sb->s_inodes) iteration functions in vfs are also being modified: > > 1. iterate_bdevs() > 2. drop_pagecache_sb() > 3. wait_sb_inodes() > 4. evict_inodes() > 5. invalidate_inodes() > 6. fsnotify_unmount_inodes() > 7. add_dquot_ref() > 8. remove_dquot_ref() > > With an exit microbenchmark that creates a large number of threads, > attachs many inodes to them and then exits. The runtimes of that > microbenchmark with 1000 threads before and after the patch on a > 4-socket Intel E7-4820 v3 system (40 cores, 80 threads) were as > follows: > > Kernel Elapsed Time System Time > ------ ------------ ----------- > Vanilla 4.5-rc4 65.29s 82m14s > Patched 4.5-rc4 22.81s 23m03s > > Before the patch, spinlock contention at the inode_sb_list_add() > function at the startup phase and the inode_sb_list_del() function at > the exit phase were about 79% and 93% of total CPU time respectively > (as measured by perf). After the patch, the percpu_list_add() > function consumed only about 0.04% of CPU time at startup phase. The > percpu_list_del() function consumed about 0.4% of CPU time at exit > phase. There were still some spinlock contention, but they happened > elsewhere. > > Signed-off-by: Waiman Long The patch looks good to me. You can add: Reviewed-by: Jan Kara Honza -- Jan Kara SUSE Labs, CR