Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Fri, 23 Mar 2018 13:21:31 -0400
From:   joe.korty@concurrent-rt.com
To:     Julia Cartwright <julia@ni.com>
Cc:     <bigeasy@linutronix.de>, <mingo@redhat.com>, <tglx@linutronix.de>,
        <rostedt@goodmis.org>, <linux-rt-users@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH RT] Defer migrate_enable migration while task state !=
 TASK_RUNNING
Message-ID: <20180323172131.GA2670@zipoli.concurrent-rt.com>
Reply-To: "Joe Korty" <joe.korty@concurrent-rt.com>
References: <20180323150959.GA16131@zipoli.concurrent-rt.com>
 <20180323165921.GG10942@jcartwri.amer.corp.natinst.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20180323165921.GG10942@jcartwri.amer.corp.natinst.com>
User-Agent: Mutt/1.8.3 (2017-05-23)
Received-SPF: None (protection.outlook.com: concurrent-rt.com does not
 designate permitted sender hosts)
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Mar 2018 17:21:35.0992 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: d10d0742-4dab-452f-d62f-08d590e287b7
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 38747689-e6b0-4933-86c0-1116ee3ef93e
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB2570
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Julia,
Thanks for the quick response!

On Fri, Mar 23, 2018 at 11:59:21AM -0500, Julia Cartwright wrote:
> Hey Joe-
> 
> Thanks for the writeup.
> 
> On Fri, Mar 23, 2018 at 11:09:59AM -0400, joe.korty@concurrent-rt.com wrote:
> > I see the below kernel splat in 4.9-rt when I run a test program that
> > continually changes the affinity of some set of running pids:
> > 
> >    do not call blocking ops when !TASK_RUNNING; state=2 set at ...
> >       ...
> >       stop_one_cpu+0x60/0x80
> >       migrate_enable+0x21f/0x3e0
> >       rt_spin_unlock+0x2f/0x40
> >       prepare_to_wait+0x5c/0x80
> >       ...
> 
> This is clearly a problem.
> 
> > The reason is that spin_unlock, write_unlock, and read_unlock call
> > migrate_enable, and since 4.4-rt, migrate_enable will sleep if it discovers
> > that a migration is in order.  But sleeping in the unlock services is not
> > expected by most kernel developers,
> 
> I don't buy this, see below:
> 
> > and where that counts most is in code sequences like the following:
> >
> >   set_current_state(TASK_UNINTERRUPIBLE);
> >   spin_unlock(&s);
> >   schedule();
> 
> The analog in mainline is CONFIG_PREEMPT and the implicit
> preempt_enable() in spin_unlock().  In this configuration, a kernel
> developer should _absolutely_ expect their task to be suspended (and
> potentially migrated), _regardless of the task state_ if there is a
> preemption event on the CPU on which this task is executing.
> 
> Similarly, on RT, there is nothing _conceptually_ wrong on RT with
> migrating on migrate_enable(), regardless of task state, if there is a
> pending migration event.

My understanding is, in standard Linux and in rt, setting
task state to anything other than TASK_RUNNING in of itself
blocks preemption.  A preemption is not really needed here
as it is expected that there is a schedule() written in that
will shortly be executed.  And if a 'involuntary schedule'
(ie, preemption) were allowed to occur between the task
state set and the schedule(), that would change the task
state back to TASK_RUNNING, which would cause the schedule
to NOP.  Thus we risk not having paused long enough here
for the condition we were waiting for to become true.

> 
> It's clear, however, that the mechanism used here is broken ...
> 
>    Julia

Thanks,
Joe