Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Wed, 4 Dec 2019 21:50:14 +0800
From:   Ming Lei <ming.lei@redhat.com>
To:     Vincent Guittot <vincent.guittot@linaro.org>
Cc:     Phil Auld <pauld@redhat.com>, Dave Chinner <david@fromorbit.com>,
        Hillf Danton <hdanton@sina.com>,
        linux-block <linux-block@vger.kernel.org>,
        linux-fs <linux-fsdevel@vger.kernel.org>,
        linux-xfs <linux-xfs@vger.kernel.org>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        Christoph Hellwig <hch@lst.de>, Jens Axboe <axboe@kernel.dk>,
        Peter Zijlstra <peterz@infradead.org>,
        Rong Chen <rong.a.chen@intel.com>, Tejun Heo <tj@kernel.org>
Subject: Re: single aio thread is migrated crazily by scheduler
Message-ID: <20191204135014.GA21449@ming.t460p>
References: <20191115010824.GC4847@ming.t460p>
 <20191115045634.GN4614@dread.disaster.area>
 <20191115070843.GA24246@ming.t460p>
 <20191128094003.752-1-hdanton@sina.com>
 <CAKfTPtA23ErKGCEJVmg6vk-QoufkiUM3NbXd31mZmKnuwbTkFw@mail.gmail.com>
 <20191202024625.GD24512@ming.t460p>
 <20191202040256.GE2695@dread.disaster.area>
 <CAKfTPtD8Q97qJ_+hdCXQRt=gy7k96XrhnFmGYP1G88YSFW0vNA@mail.gmail.com>
 <20191202212210.GA32767@lorien.usersys.redhat.com>
 <CAKfTPtC7uycC3b2ngOFUqOh9-Fcz7h-151aaYJbLJFXrNq-gkw@mail.gmail.com>
MIME-Version: 1.0
In-Reply-To: <CAKfTPtC7uycC3b2ngOFUqOh9-Fcz7h-151aaYJbLJFXrNq-gkw@mail.gmail.com>
User-Agent: Mutt/1.12.1 (2019-06-15)
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Dec 03, 2019 at 10:45:38AM +0100, Vincent Guittot wrote:
> On Mon, 2 Dec 2019 at 22:22, Phil Auld <pauld@redhat.com> wrote:
> >
> > Hi Vincent,
> >
> > On Mon, Dec 02, 2019 at 02:45:42PM +0100 Vincent Guittot wrote:
> > > On Mon, 2 Dec 2019 at 05:02, Dave Chinner <david@fromorbit.com> wrote=
:
> >
> > ...
> >
> > > > So, we can fiddle with workqueues, but it doesn't address the
> > > > underlying issue that the scheduler appears to be migrating
> > > > non-bound tasks off a busy CPU too easily....
> > >
> > > The root cause of the problem is that the sched_wakeup_granularity_ns
> > > is in the same range or higher than load balance period. As Peter
> > > explained, This make the kworker waiting for the CPU for several load
> > > period and a transient unbalanced state becomes a stable one that the
> > > scheduler to fix. With default value, the scheduler doesn't try to
> > > migrate any task.
> >
> > There are actually two issues here.   With the high wakeup granularity
> > we get the user task actively migrated. This causes the significant
> > performance hit Ming was showing. With the fast wakeup_granularity
> > (or smaller IOs - 512 instead of 4k) we get, instead, the user task
> > migrated at wakeup to a new CPU for every IO completion.
>=20
> Ok, I haven't noticed that this one was a problem too. Do we have perf
> regression ?

Follows the test result on one server(Dell, R630: Haswell-E):

kernel.sched_wakeup_granularity_ns =3D 4000000
kernel.sched_min_granularity_ns =3D 3000000

---------------------------------------
test              =09=09        | IOPS
---------------------------------------
./xfs_complete 512      =09    | 7.8K=20
---------------------------------------
taskset -c 8 ./xfs_complete 512 | 9.8K=20
---------------------------------------

Thanks,
Ming