Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Sat, 16 Nov 2019 14:31:06 +0800
From:   Ming Lei <ming.lei@redhat.com>
To:     Dave Chinner <david@fromorbit.com>
Cc:     linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org,
        linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org,
        Jeff Moyer <jmoyer@redhat.com>,
        Dave Chinner <dchinner@redhat.com>,
        Eric Sandeen <sandeen@redhat.com>,
        Christoph Hellwig <hch@lst.de>, Jens Axboe <axboe@kernel.dk>,
        Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Tejun Heo <tj@kernel.org>
Subject: Re: single aio thread is migrated crazily by scheduler
Message-ID: <20191116063106.GA18194@ming.t460p>
References: <20191114113153.GB4213@ming.t460p>
 <20191114235415.GL4614@dread.disaster.area>
 <20191115010824.GC4847@ming.t460p>
 <20191115045634.GN4614@dread.disaster.area>
 <20191115070843.GA24246@ming.t460p>
 <20191115234005.GO4614@dread.disaster.area>
MIME-Version: 1.0
In-Reply-To: <20191115234005.GO4614@dread.disaster.area>
User-Agent: Mutt/1.12.1 (2019-06-15)
Content-Type: text/plain; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Sat, Nov 16, 2019 at 10:40:05AM +1100, Dave Chinner wrote:
> On Fri, Nov 15, 2019 at 03:08:43PM +0800, Ming Lei wrote:
> > On Fri, Nov 15, 2019 at 03:56:34PM +1100, Dave Chinner wrote:
> > > On Fri, Nov 15, 2019 at 09:08:24AM +0800, Ming Lei wrote:
> > I can reproduce the issue with 4k block size on another RH system, and
> > the login info of that system has been shared to you in RH BZ.
> >=20
> > 1)
> > sysctl kernel.sched_min_granularity_ns=3D10000000
> > sysctl kernel.sched_wakeup_granularity_ns=3D15000000
>=20
> So, these settings definitely influence behaviour.
>=20
> If these are set to kernel defaults (4ms and 3ms each):
>=20
> sysctl kernel.sched_min_granularity_ns=3D4000000
> sysctl kernel.sched_wakeup_granularity_ns=3D3000000
>=20
> The migration problem largely goes away - the fio task migration
> event count goes from ~2,000 a run down to 200/run.
>=20
> That indicates that the migration trigger is likely load/timing
> based. The analysis below is based on the 10/15ms numbers above,
> because it makes it so much easier to reproduce.

On another machine, './xfs_complete 512' may be migrated 11~12K/sec,
which don't need to change the above two kernel sched defaults, however
the fio io thread only takes 40% CPU.=20

'./xfs_complete 4k' on this machine, the fio IO CPU utilization is >=3D 98%=
.


Thanks,
Ming