Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Sat, 17 Sep 2022 21:55:45 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Peter Zijlstra <peterz@infradead.org>
CC:     Vincent Guittot <vincent.guittot@linaro.org>,
        Tim Chen <tim.c.chen@intel.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        Juri Lelli <juri.lelli@redhat.com>,
        Rik van Riel <riel@surriel.com>,
        Aaron Lu <aaron.lu@intel.com>,
        Abel Wu <wuyun.abel@bytedance.com>,
        K Prateek Nayak <kprateek.nayak@amd.com>,
        Yicong Yang <yangyicong@hisilicon.com>,
        "Gautham R . Shenoy" <gautham.shenoy@amd.com>,
        Ingo Molnar <mingo@redhat.com>,
        "Dietmar Eggemann" <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH] sched/fair: Choose the CPU where short task is
 running during wake up
Message-ID: <YyXR4YgHeTR4Negc@chenyu5-mobl1>
References: <20220915165407.1776363-1-yu.c.chen@intel.com>
 <YyRhvEGS4ivPGf+z@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <YyRhvEGS4ivPGf+z@hirez.programming.kicks-ass.net>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?UhgTQLgV+nGpah7UIDNojqB4OSg/Zx6HLleOBnpQMJX1UF/btQ7bRG45Howc?=
 =?us-ascii?Q?1ov0bncnb90exLn0U06C/tycPNN7YoV4W0yFXC9D9c6wGLUm4eYrJF/xjxpo?=
 =?us-ascii?Q?m+BDGoMEniK8y2XnfmZa+aRoaRMd9LHGGzskSnm9WvOrIzx/HLmegrRXd+Zx?=
 =?us-ascii?Q?HR0HrBlMjQwKUcyxOspKb/+kRsT5wgovDUFiajrzrZQOToI1T+eFNMrR3kLH?=
 =?us-ascii?Q?Yt8kAGZTA6DWxnfUemWclrhngwrqO6OEaFk3DuWrHw7TGRyYqLcHk17GkfNb?=
 =?us-ascii?Q?DDI76ipFJHhGiEk7fqX3aLBdjVZlSxMzDdo/8eEmog0GIjEIbAZ0NOCO/hsh?=
 =?us-ascii?Q?kE+GGVVh0HC4+H4/6v9M3PvCs0DyD+6k9J6ISpAV4ECd5iiyb9Ie1fk0qocT?=
 =?us-ascii?Q?Rvbr1mTBUhDvH0aYtrTRmjGPR76fpxE8/dKgZP9JjIUoiWaGzGc7DNMA+6CJ?=
 =?us-ascii?Q?N3cZDhVK+56JS1tl8yKN0+i5AbyECFCOhjJpKFV5wu1NUj5iGm3/+BEniXEn?=
 =?us-ascii?Q?MMs70H7b4IoLA2vMVPD7qMSCdn3TlaUxiy5SjgmXK+lZ3KONNZR9TMkFfKiO?=
 =?us-ascii?Q?R8ek30Fc577HjqkqOEbqJN/AdAXfb9Rm1TULh2OVXKnn5lsQdUbgSwsXFInm?=
 =?us-ascii?Q?3VDgdShSQ0Bt1HXBeJY6cpSSYdDgoP5+4Vxuk5nQAzIouTnOhRj8i/zCDmSR?=
 =?us-ascii?Q?wVbzowM6oMdaP53Agnrf7094xP7PFj18XTQcXizUVAiLCUcXZs8Q7hvFLn5B?=
 =?us-ascii?Q?HPskUndlcAK55ZMrj0qbJe1tEtvOPnGhP8ROUXYTxnm6V/w+EzSmBVaH3Ohf?=
 =?us-ascii?Q?BDNQUg2PCWSoUCRvGP/IyIcpDhIRiBvH4apFtn85DyRHBbq7LZybuaVRL+L4?=
 =?us-ascii?Q?njBb6H7J3BA2ql1i31/UsxyB256YeVltRlks8WdY+FZNeYkGl6MUWmFewyc0?=
 =?us-ascii?Q?aW/2hlFYrqCK5DXZk08ZuFZLVArUMxunYt5bRMBueK7pbnSIX/auPyHpjOhb?=
 =?us-ascii?Q?9n1jV2UgHbFQO8CZnfH4CtnGXydz3+tHtkqsGj1+0SGsm6rMTf43syrnFVYz?=
 =?us-ascii?Q?jYUzUC5CQ99C2hdkT4F//yU4K2YsF96YOD454JCDdQ7XMcVByJ8FtsoSf7GF?=
 =?us-ascii?Q?WeTsvsE9sn7jauFFzfrxi4xZCrOTLtJapQNqHZctOUa6qnmUvSca6lNo9HH6?=
 =?us-ascii?Q?+lQ6CecCrti8xQmHiodejSFu1HrvEcnF4BRNDr/FYn6EzDj9MjXxUwkY2Jmw?=
 =?us-ascii?Q?wZHDmgLBljxfzAfLnndRCPJdPx18OQe2wOO47RCzy4SX6xK/+WH8cRRw6vlh?=
 =?us-ascii?Q?SUN445CgnXZckm5IW2o/wR6wO12nusdOikGI+pYIAkwINSEVIOIv8JenXo/A?=
 =?us-ascii?Q?k7JZlCsmwwEP5A2CscZgIOQkedmvzmUQUWfO3RbmDN7gkWaJSa96q2Z2L20Y?=
 =?us-ascii?Q?cP5g57Nzyfu2wA8iW+22akOZJMRvy3QJCSevQQoXNXZyov8eCv1SND5ZrUgw?=
 =?us-ascii?Q?kYz7YQivCpSepuKEVw6tlbiRAQnuUsKi9JrlE9HgjYQWjjSJmjn0a6XlSgBN?=
 =?us-ascii?Q?yd6mB29eq00ezfXzd7TTqzcu75sO4xhHLdH8VrDb?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d28e78b2-4f9e-4f25-1883-08da98b45d72
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Sep 2022 13:56:05.7683
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LHsHgrU2ZPM4ExTeY3wjtQtnhMAYkah+wDybxQ5FJSW7nHFJF7feyaIZXUjy0o2BiDx1Tzny4LXJmxrVV/7+Og==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR11MB6028
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,
        SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2022-09-16 at 13:45:00 +0200, Peter Zijlstra wrote:
> On Fri, Sep 16, 2022 at 12:54:07AM +0800, Chen Yu wrote:
> > And the rq lock bottleneck is composed of two paths(perf profile):
> > 
> > (path1):
> > raw_spin_rq_lock_nested.constprop.0;
> > try_to_wake_up;
> > default_wake_function;
> > autoremove_wake_function;
> > __wake_up_common;
> > __wake_up_common_lock;
> > __wake_up_sync_key;
> > pipe_write;
> > new_sync_write;
> > vfs_write;
> > ksys_write;
> > __x64_sys_write;
> > do_syscall_64;
> > entry_SYSCALL_64_after_hwframe;write
> 
> Can you please addr2line -i the raw_spin_rq_lock callsite so we know which is
> the one causing grief?
> 
> Specifically; I'm worried about PSI, psi_ttwu_dequeue() can cause ttwu()
> to take _2_ rq->lock, which absolutely blows for this case.
Above perf profile result was captured with 'psi=0' appended in the boot
commandline, and with NO_TTWU_QUEUE on 6.0-rc4. To narrow down we disabled
psi the first time we saw a rq lock contention. But even with psi=0 we still
observe the rq lock contention.

To confirm this, the 'perf report -F+period,srcline' was used to leverage
addr2line to parse the line. However it seems that with DWARF v4 enabled
in the kernel, the rq lock issue could not be reproduced. So I hacked the
code to make ttwu_queue() non-static, and perf profile shows that it grabs
the rq lock:

raw_spin_rq_lock_nested.constprop.0;
ttwu_queue;    <----------
try_to_wake_up;
default_wake_function;
autoremove_wake_function;
__wake_up_common;
__wake_up_common_lock;
__wake_up_sync_key;
pipe_write;
vfs_write;
ksys_write;
__x64_sys_write;
do_syscall_64;
entry_SYSCALL_64_after_hwframe;
write

Then if TTWU_QUEUE is enabled, the rq lock contention issue could
not be reproduced, but long idle duration was still observed due to
sched_ttwu_pending(as descibed in the commit log).


thanks,
Chenyu