Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6;
Message-ID: <5fe8b838-ffe6-8f5d-e23d-61c58bc7f6b3@amd.com>
Date:   Thu, 14 Dec 2023 10:45:18 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.13.1
Subject: Re: [PATCH v6 00/20] Proxy Execution: A generalized form of Priority
 Inheritance v6
Content-Language: en-US
To:     John Stultz <jstultz@google.com>
Cc:     LKML <linux-kernel@vger.kernel.org>,
        Joel Fernandes <joelaf@google.com>,
        Qais Yousef <qyousef@google.com>,
        Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Valentin Schneider <vschneid@redhat.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>,
        Zimuzo Ezeozue <zezeozue@google.com>,
        Youssef Esmat <youssefesmat@google.com>,
        Mel Gorman <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Will Deacon <will@kernel.org>,
        Waiman Long <longman@redhat.com>,
        Boqun Feng <boqun.feng@gmail.com>,
        "Paul E . McKenney" <paulmck@kernel.org>, kernel-team@android.com,
        Metin Kaya <Metin.Kaya@arm.com>
References: <20231106193524.866104-1-jstultz@google.com>
 <c6787831-f659-12cb-4954-fd13a05ed590@amd.com>
 <CANDhNCqcviNOZTi2G9R21K-p32uM0U_LLiiEBjveJ4gikk8wrQ@mail.gmail.com>
From:   K Prateek Nayak <kprateek.nayak@amd.com>
In-Reply-To: <CANDhNCqcviNOZTi2G9R21K-p32uM0U_LLiiEBjveJ4gikk8wrQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YkJkWkVHNEFpTGNuQWhYeVk3NEtzSG91dlZ2eWlKaWNsWmQyTWtlMzAwZFly?=
 =?utf-8?B?cTM0V1NBMHcrZHVPd0EwU2JmWFZwUDh3ZEZWTU5KelNXdkhoenJoTVdFNjN4?=
 =?utf-8?B?YXB1aE1ua3Q4NUpBdzYzWUNCYWdQeHpkUW1TdWpIUXBXK2NYbnVjZW5EaXpU?=
 =?utf-8?B?bUxNc0ZLaWFWZHpscVg3NVJyT3lxTVF3VUlxWnMxL2JPbHVjdDg4c1A1RHhh?=
 =?utf-8?B?czVCT2Y4dDlZQWtlMGZwWkV5SFZDOGxGVUNDaHhFMTBoTW16dHBBUS9DeVNJ?=
 =?utf-8?B?cmFtMGRCcEF2TzBUNkdGcXJPZUtqZ0E1VjdIWCs4UnArRXBGVEVqZnpzSHVt?=
 =?utf-8?B?allNQ2VkVUF4ZmxtZWcyRStUL0V0MUc1Q0hGT09oMGE3VkV5ODB2eHBQSHk2?=
 =?utf-8?B?SGxENGdaekl2SUprblJkd0xjQmdrbTNzZThoU3BsM0lFNXhSVHpocU4yY0Zk?=
 =?utf-8?B?ZUQvVkV2U252Y3RVK1ZsckJGODJtdjNPZ2lPRjZEQWU4TjBEdG54S3pZTFha?=
 =?utf-8?B?WHFMREpjNkV6TWxKTHpBdVljRXgxdlk3cG1mZXU0Tlc3dm5qRVc2Sng4WmFV?=
 =?utf-8?B?WVFHUTdJbmhYZk9EUXh5VjZTU2dzY1Zxd0tsb3pTczB5VTVmM3d6QWJqSko3?=
 =?utf-8?B?emtPTlJFYm9aaGx2N2lPRko4c2pETThXMFVXUDdIK0R6eitrQU9CNVhKY1M5?=
 =?utf-8?B?R3NLQjYySjJpdi84SHMrZ0tNUTdkVW16M2YzRHM1ck5BR292MFpNYjNYTzd0?=
 =?utf-8?B?dThnNjEyNEtrNUlWRWhUVXUxVzI3VTdrV3FhYlAvU3hmamNEN01hVEdRbjJ4?=
 =?utf-8?B?Y0Q1OTNiRUJEOW5veGl1WTlMWFJUMUZVeTg3MWdzTDFReTFZL3lQUjhFL1c5?=
 =?utf-8?B?ejRiZEw3dDc1Q0E3eE83TUZMa2UrWkxDZGp5eDY5MzEwTFBQSlp1OHA3TDZT?=
 =?utf-8?B?QTlKL29GYzZRSjBuRXIyejZ4SWh2cDZ3ckYxYkFZOGZVendrdWtZSnB3MTZk?=
 =?utf-8?B?SE9RR0Q0dWd2ZTBVRytYYTNlMWRZbE9ETlNXVkhBd2VGNG9XRThWN2h3RkZQ?=
 =?utf-8?B?cDBUenZrN1dxMitNK3dwbDh2bGZWYlFKWEJ2KzR3ckF3Q1p0bHRlZmE2L3Vn?=
 =?utf-8?B?YWxGVWYzUGU3YkJxd0JyTXNxbkY0VVFxYXJ1bUR5aHJuSG9ZOXQxQllHSVF1?=
 =?utf-8?B?dmhKaUQ1OGFZTW9GZ2NjSUtlTjJZVS9qbFFPaWs4VldqN1Aya0IzYnU2eTU3?=
 =?utf-8?B?dTNpV0MwbGhFMkRUc2pFV2NYQUd3UUpoNkxCOG12SEs2RmVWa3VmWGxOcCtj?=
 =?utf-8?B?SlhhTy8zbUQxZHhWQldnWnprZlg0eDZ0bGd1OTR0UGFrclIyN3FWS2x3WWZx?=
 =?utf-8?B?Y0NybUxvZU96WTNadTlBZjg1SmhUdko5cCtoZ2JlYjk4RjFIQ0RjajZManJs?=
 =?utf-8?B?TnRVQUhrVDd4TXdzcllib0hMejJBMFAxUGRrTGFIczFWVlliVE9WbEdMdHY3?=
 =?utf-8?B?YVVHck15VWxyVzZiQlJ3dUlRam16Y0ltWGJLM2JacFlRWm9GNEI3cFVLZ08v?=
 =?utf-8?B?TWQxRmlrTTZ0NE9LMklKSDlhWTMyU3IxNCt2K2oyN0haN3gzWTBIVloxcXVT?=
 =?utf-8?B?TUdJVlZTeU5NSHhqcFZ4NUhjWHliRlFrcGM2QVEzTHNseGwzSWc5RTFWZG5O?=
 =?utf-8?B?d1BreFNOaHZ3aHFHQlVuZ004V0FpUmJPYzdITjBEeVovT3dvUTA1VlVhT3N3?=
 =?utf-8?B?TmtEcy9aNHIrQVEwVk02VEZ4a3JabGptaEI5S3B3bmFmSEVaVTNoUXZGSjlM?=
 =?utf-8?B?d29qb3BCUEFjQ2Y3ak0wMGd2TDh4d3had3ByRjV0dktLVjRNRkhVVnFPeitU?=
 =?utf-8?B?SFJUVEpCRWV0TmgvdEdIdGVoNWNhTjFkUFAyTEpIRmt6L3pmSzdmVWxONTRM?=
 =?utf-8?B?UnBDbkRKZ3NtazdGdzlKZ3llU2hkV1FrS01XZHR5eFBxNkZUdThLeGJWZ1lt?=
 =?utf-8?B?QWJFNXBOWG5ZU1ROc25VMExPdnlhZUxFQ1pSMU9hT3VHU1lrOVdnay9UbExN?=
 =?utf-8?B?QWRnT1Q3NStQanJkMVdSZHIvNE5aTmhkL1FzeG5oNHltTFdOOUdraW1peHBE?=
 =?utf-8?Q?FIsTefkFUC9KQqjVT/zXg8M/q?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1cec9207-5ab9-4865-d273-08dbfc63b16a
X-MS-Exchange-CrossTenant-AuthSource: MW2PR12MB2379.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Dec 2023 05:15:32.5741
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: RE/QSzpARTjj17P8P7leNJPwPIqcxBgllkS9k1zDUCZD5fY1nUCqIS31AFdj+pjv37hyLrswEGFSs5UtKfjkBA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4969
X-Spam-Status: No, score=-1.6 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 13 Dec 2023 21:15:47 -0800 (PST)

Hello John,

Thank you for taking a look at the report.

On 12/14/2023 12:41 AM, John Stultz wrote:
> On Tue, Dec 12, 2023 at 10:37 PM K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>>
>> Hello John,
>>
>> I may have some data that might help you debug a potential performance
>> issue mentioned below.
> 
> Hey Prateek,
>   Thank you so much for taking the time to try this out and providing
> such helpful analysis!
> More below.
> 
>> On 11/7/2023 1:04 AM, John Stultz wrote:
>>> [..snip..]
>>>
>>> Performance:
>>> —----------
>>> This patch series switches mutexes to use handoff mode rather
>>> than optimistic spinning. This is a potential concern where locks
>>> are under high contention. However, earlier performance analysis
>>> (on both x86 and mobile devices) did not see major regressions.
>>> That said, Chenyu did report a regression[3], which I’ll need to
>>> look further into.
>>
>> I too see this as the most notable regression. Some of the other
>> benchmarks I've tested (schbench, tbench, netperf, ycsb-mongodb,
>> DeathStarBench) show little to no difference when running with Proxy
> 
> This is great to hear! Thank you for providing this input!
> 
>> Execution, however sched-messaging sees a 10x blowup in the runtime.
>> (taskset -c 0-7,128-125 perf bench sched messaging -p -t -l 100000 -g 1)
> 
> Oof. I appreciate you sharing this!
> 
>> While investigating, I drew up the runqueue length when running
>> sched-messaging pinned to 1CCX (CPUs 0-7,128-125 on my 3rd Generation
>> EPYC system) using the following bpftrace script that dumps it csv
>> format:
> 
> Just so I'm following you properly on the processor you're using, cpus
> 0-7 and 128-125 are in the same CCX?
> (I thought there were only 8 cores per CCX?)

Sorry about that! It should be 0-7,128-135 (16 threads of 8 cores in the
same CCX) The pinning was added so that I could only observe a subset of
the total CPUs since analyzing the behavior of 40 tasks on 256 CPUs was
much harder than analyzing it on 16 CPUs :)

> 
>> rqlen.bt
>> ---
> <snip>
>> --
>>
>> I've attached the csv for tip (rqlen50-tip-pinned.csv) and proxy
>> execution (rqlen50-pe-pinned.csv) below.
>>
>> The trend I see with hackbench is that the chain migration leads
>> to a single runqueue being completely overloaded, followed by some
>> amount of the idling on the entire CCX and a similar chain appearing
>> on a different CPU. The trace for tip show a lot more CPUs being
>> utilized.
>>
>> Mathieu has been looking at hackbench and the effect of task migration
>> on the runtime and it appears that lowering the migrations improves
>> the hackbench performance significantly [1][2][3]
>>
>> [1] https://lore.kernel.org/lkml/20230905072141.GA253439@ziqianlu-dell/
>> [2] https://lore.kernel.org/lkml/20230905171105.1005672-1-mathieu.desnoyers@efficios.com/
>> [3] https://lore.kernel.org/lkml/20231019160523.1582101-1-mathieu.desnoyers@efficios.com/
>>
>> Since migration itself is not cheap, I believe the chain migration at
>> the current scale hampers the performance since sched-messaging
>> emulates a worst-case scenario for proxy-execution.
> 
> Hrm.
> 
>> I'll update the thread once I have more information. I'll continue
>> testing and take a closer look at the implementation.
>>
>>> I also briefly re-tested with this v5 series
>>> and saw some average latencies grow vs v4, suggesting the changes
>>> to return-migration (and extra locking) have some impact. With v6
>>> the extra overhead is reduced but still not as nice as v4. I’ll
>>> be digging more there, but my priority is still stability over
>>> speed at this point (it’s easier to validate correctness of
>>> optimizations if the baseline isn’t crashing).
>>>
>>>
>>> If folks find it easier to test/tinker with, this patch series
>>> can also be found here:
>>>   https://github.com/johnstultz-work/linux-dev/commits/proxy-exec-v6-6.6
>>>   https://github.com/johnstultz-work/linux-dev.git proxy-exec-v6-6.6
>>
>> P.S. I was using the above tree.
> 
> Ok, I've been working on getting v7 ready which includes two main things:
> 1) I've reworked the return migration back into the ttwu path to avoid
> the lock juggling
> 2) Working to properly conditionalize and re-add Connor's
> chain-migration feature (which when a migration happens pulls the full
> blocked_donor list with it)
> 
> So I'll try to reproduce your results and see if these help any with
> this particular case, and then I'll start to look closer at what can
> be done.
> 
> Again, thanks so much, I've got so much gratitude for your testing and
> analysis here. I really appreciate your feedback!
> Do let me know if you find anything further!

Sure thing! I'll keep you updated of any finding. Thank you for digging
further into this issue.

> 
> thanks
> -john

--
Thanks and Regards,
Prateek