Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753743AbZJHFpU (ORCPT ); Thu, 8 Oct 2009 01:45:20 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1753279AbZJHFpT (ORCPT ); Thu, 8 Oct 2009 01:45:19 -0400 Received: from mga03.intel.com ([143.182.124.21]:5945 "EHLO mga03.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752940AbZJHFpQ (ORCPT ); Thu, 8 Oct 2009 01:45:16 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.44,523,1249282800"; d="scan'208";a="196405360" Date: Thu, 8 Oct 2009 13:44:21 +0800 From: Wu Fengguang To: Peter Staubach Cc: Andrew Morton , Theodore Tso , Christoph Hellwig , Dave Chinner , Chris Mason , Peter Zijlstra , "Li, Shaohua" , Myklebust Trond , "jens.axboe@oracle.com" , Jan Kara , Nick Piggin , "linux-fsdevel@vger.kernel.org" , LKML Subject: Re: [PATCH 00/45] some writeback experiments Message-ID: <20091008054421.GA20128@localhost> References: <20091007073818.318088777@intel.com> <4ACC9BE2.5070409@redhat.com> <20091007151822.GA9574@localhost> <20091008053335.GA19458@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20091008053335.GA19458@localhost> User-Agent: Mutt/1.5.18 (2008-05-17) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 11303 Lines: 202 On Thu, Oct 08, 2009 at 01:33:35PM +0800, Wu Fengguang wrote: > On Wed, Oct 07, 2009 at 11:18:22PM +0800, Wu Fengguang wrote: > > On Wed, Oct 07, 2009 at 09:47:14PM +0800, Peter Staubach wrote: > > > > > > > # vmmon -d 1 nr_writeback nr_dirty nr_unstable # (per 1-second samples) > > > > nr_writeback nr_dirty nr_unstable > > > > 11227 41463 38044 > > > > 11227 41463 38044 > > > > 11227 41463 38044 > > > > 11227 41463 38044 > > > > I guess in the above 4 seconds, either client or (more likely) server > > is blocked. A blocked server cannot send ACKs to knock down both > > Yeah the server side is blocked. The nfsd are mostly blocked in > generic_file_aio_write(), in particular, the i_mutex lock! I'm copying > one or two big files over NFS, so the i_mutex lock is heavily contented. > > I'm using the default wsize=4096 for NFS-root.. Just switched to 512k wsize, and things improved: in most time the 8 nfsd are not all blocked. However, the bumpiness still remains: nr_writeback nr_dirty nr_unstable 11105 58080 15042 11105 58080 15042 11233 54583 18626 11101 51964 22036 11105 51978 22065 11233 52362 22577 10985 58538 13500 11233 53748 19721 11047 51999 21778 11105 50262 23572 11105 50262 20441 10985 52772 20721 10977 52109 21516 11105 48296 26629 11105 48296 26629 10985 52191 21042 11166 51456 22296 10980 50681 24466 11233 45352 30488 11233 45352 30488 11105 45475 30616 11131 45313 20355 11233 51126 22637 11233 51126 22637 wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs 329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 1 0.1 S< svc_recv nfsd 4691 4691 TS - -5 24 0 0.1 S< svc_recv nfsd 4692 4692 TS - -5 24 0 0.1 R< ? nfsd 4693 4693 TS - -5 24 0 0.1 S< svc_recv nfsd 4694 4694 TS - -5 24 0 0.1 S< svc_recv nfsd 4695 4695 TS - -5 24 0 0.1 S< svc_recv nfsd 4696 4696 TS - -5 24 0 0.1 S< svc_recv nfsd 4697 4697 TS - -5 24 0 0.1 R< ? nfsd wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs 329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4691 4691 TS - -5 24 0 0.1 S< svc_recv nfsd 4692 4692 TS - -5 24 1 0.1 D< log_wait_commit nfsd 4693 4693 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4694 4694 TS - -5 24 0 0.1 S< svc_recv nfsd 4695 4695 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4696 4696 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4697 4697 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs 329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 1 0.1 S< svc_recv nfsd 4691 4691 TS - -5 24 0 0.1 S< svc_recv nfsd 4692 4692 TS - -5 24 1 0.1 R< ? nfsd 4693 4693 TS - -5 24 1 0.1 R< ? nfsd 4694 4694 TS - -5 24 1 0.1 R< ? nfsd 4695 4695 TS - -5 24 1 0.1 S< svc_recv nfsd 4696 4696 TS - -5 24 0 0.1 S< svc_recv nfsd 4697 4697 TS - -5 24 1 0.1 S< svc_recv nfsd wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs 329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd 4691 4691 TS - -5 24 0 0.1 S< svc_recv nfsd 4692 4692 TS - -5 24 1 0.1 D< nfsd_sync nfsd 4693 4693 TS - -5 24 1 0.1 D< sync_buffer nfsd 4694 4694 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd 4695 4695 TS - -5 24 1 0.1 S< svc_recv nfsd 4696 4696 TS - -5 24 0 0.1 S< svc_recv nfsd 4697 4697 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd Thanks, Fengguang > wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs > 329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod > 4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd > 4691 4691 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd > 4692 4692 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd > 4693 4693 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd > 4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd > 4695 4695 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd > 4696 4696 TS - -5 24 1 0.0 D< log_wait_commit nfsd > 4697 4697 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd > wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs > 329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod > 4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd > 4691 4691 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd > 4692 4692 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd > 4693 4693 TS - -5 24 0 0.0 D< sync_buffer nfsd > 4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd > 4695 4695 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd > 4696 4696 TS - -5 24 1 0.0 D< generic_file_aio_write nfsd > 4697 4697 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd > > wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs > 329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod > 4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd > 4691 4691 TS - -5 24 0 0.1 D< get_request_wait nfsd > 4692 4692 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd > 4693 4693 TS - -5 24 0 0.1 S< svc_recv nfsd > 4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd > 4695 4695 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd > 4696 4696 TS - -5 24 0 0.1 S< svc_recv nfsd > 4697 4697 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd > > wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs > 329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod > 4690 4690 TS - -5 24 1 0.1 D< get_write_access nfsd > 4691 4691 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd > 4692 4692 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd > 4693 4693 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd > 4694 4694 TS - -5 24 1 0.1 D< get_write_access nfsd > 4695 4695 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd > 4696 4696 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd > 4697 4697 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd > > Thanks, > Fengguang > > > nr_writeback/nr_unstable. And the stuck nr_writeback will freeze > > nr_dirty as well, because the dirtying process is throttled until > > it receives enough "PG_writeback cleared" event, however the bdi-flush > > thread is also blocked when trying to clear more PG_writeback, because > > the client side nr_writeback limit has been reached. In summary, > > > > server blocked => nr_writeback stuck => nr_writeback limit reached > > => bdi-flush blocked => no end_page_writeback() => dirtier blocked > > => nr_dirty stuck > > > > Thanks, > > Fengguang > > > > > > 11045 53987 6490 > > > > 11033 53120 8145 > > > > 11195 52143 10886 > > > > 11211 52144 10913 > > > > 11211 52144 10913 > > > > 11211 52144 10913 > > > > > > > > btrfs seems to maintain a private pool of writeback pages, which can go out of > > > > control: > > > > > > > > nr_writeback nr_dirty > > > > 261075 132 > > > > 252891 195 > > > > 244795 187 > > > > 236851 187 > > > > 228830 187 > > > > 221040 218 > > > > 212674 237 > > > > 204981 237 > > > > > > > > XFS has very interesting "bumpy writeback" behavior: it tends to wait > > > > collect enough pages and then write the whole world. > > > > > > > > nr_writeback nr_dirty > > > > 80781 0 > > > > 37117 37703 > > > > 37117 43933 > > > > 81044 6 > > > > 81050 0 > > > > 43943 10199 > > > > 43930 36355 > > > > 43930 36355 > > > > 80293 0 > > > > 80285 0 > > > > 80285 0 > > > > > > > > Thanks, > > > > Fengguang > > > > > > > > -- > > > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > > > the body of a message to majordomo@vger.kernel.org > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/