Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: single aio thread is migrated crazily by scheduler
To:     Boaz Harrosh <boaz@plexistor.com>, Phil Auld <pauld@redhat.com>,
        Ming Lei <ming.lei@redhat.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Dave Chinner <david@fromorbit.com>,
        linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org,
        linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org,
        Jeff Moyer <jmoyer@redhat.com>,
        Dave Chinner <dchinner@redhat.com>,
        Eric Sandeen <sandeen@redhat.com>,
        Christoph Hellwig <hch@lst.de>, Ingo Molnar <mingo@redhat.com>,
        Tejun Heo <tj@kernel.org>,
        Vincent Guittot <vincent.guittot@linaro.org>
References: <20191114235415.GL4614@dread.disaster.area>
 <20191115010824.GC4847@ming.t460p>
 <20191115045634.GN4614@dread.disaster.area>
 <20191115070843.GA24246@ming.t460p>
 <20191115234005.GO4614@dread.disaster.area>
 <20191118092121.GV4131@hirez.programming.kicks-ass.net>
 <20191118204054.GV4614@dread.disaster.area>
 <20191120191636.GI4097@hirez.programming.kicks-ass.net>
 <20191120220313.GC18056@pauld.bos.csb> <20191121041218.GK24548@ming.t460p>
 <20191121141207.GA18443@pauld.bos.csb>
 <93de0f75-3664-c71e-9947-5b37ae935ddc@plexistor.com>
From:   Jens Axboe <axboe@kernel.dk>
Message-ID: <8c02ab43-3880-68db-f293-9958510fb29e@kernel.dk>
Date:   Thu, 21 Nov 2019 09:19:29 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.9.0
MIME-Version: 1.0
In-Reply-To: <93de0f75-3664-c71e-9947-5b37ae935ddc@plexistor.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 11/21/19 8:02 AM, Boaz Harrosh wrote:
> On 21/11/2019 16:12, Phil Auld wrote:
> <>
>>
>> The scheduler doesn't know if the queued_work submitter is going to go to sleep.
>> That's why I was singling out AIO. My understanding of it is that you submit the IO
>> and then keep going. So in that case it might be better to pick a node-local nearby
>> cpu instead. But this is a user of work queue issue not a scheduler issue.
>>
> 
> We have a very similar long standing problem in our system (zufs), that we had to do
> hacks to fix.
> 
> We have seen these CPU bouncing exacly as above in fio and more
> benchmarks, Our final analysis was:
>
> One thread is in wait_event() if the wake_up() is on the same CPU as
> the waiter, on some systems usually real HW and not VMs, would bounce
> to a different CPU.  Now our system has an array of worker-threads
> bound to each CPU. an incoming thread chooses a corresponding cpu
> worker-thread, let it run, waiting for a reply, then when the
> worker-thread is done it will do a wake_up(). Usually its fine and the
> wait_event() stays on the same CPU. But on some systems it will wakeup
> in a different CPU.
> 
> Now this is a great pity because in our case and the work_queue case
> and high % of places the thread calling wake_up() will then
> immediately go to sleep on something.  (Work done lets wait for new
> work)
> 
> I wish there was a flag to wake_up() or to the event object that says
> to relinquish the remaning of the time-slice to the waiter on same
> CPU, since I will be soon sleeping.

Isn't that basically what wake_up_sync() is?

-- 
Jens Axboe