Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <b9f72ee9-328c-9de3-05e7-ea7d1339984b@oracle.com>
Date:   Mon, 10 Apr 2023 16:51:38 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.9.1
Subject: Re: [RFC PATCH v4] sched: Fix performance regression introduced by
 mm_cid
Content-Language: en-US
To:     Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        Peter Zijlstra <peterz@infradead.org>
Cc:     linux-kernel@vger.kernel.org, Aaron Lu <aaron.lu@intel.com>,
        Olivier Dion <odion@efficios.com>
References: <20230410150150.2179062-1-mathieu.desnoyers@efficios.com>
From:   michael.christie@oracle.com
In-Reply-To: <20230410150150.2179062-1-mathieu.desnoyers@efficios.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YmRxUHBpbXVMZW55cExZUUVKOHowYUVnTjczelJqQ2tUUzVsVWhHVk5icnMw?=
 =?utf-8?B?b1RyY3dTcjdXYTVFbksyT1VuUS9jTFF1V2lpNS9rR1dkSWZCSWRmSHJJbGhG?=
 =?utf-8?B?K1l2Zk9lcFdibURodGZUdkpyN2xkbUtIZVUwUHpHalA4ZHcwWWZKSDlKRzQ3?=
 =?utf-8?B?MnAyb1owRmdWRFJrcEZPYjhEdjR3V3NlWFBMbmRkWVBaM0RSYy93T1o2TU5I?=
 =?utf-8?B?YzBDMHNvMFFuUHBTUG92MUJ5TGorY0Y2U0g1RUZpYUhkSm80VmpxbmZHQlhT?=
 =?utf-8?B?YTlGNVRtZTl5Yi8yNlJWVFkxcGp3d3hwdU5WUkpBQ1d3elFDMExtVGFrYWVj?=
 =?utf-8?B?cVh3bUhtSk1NWXplbW5TTXp2clowV245TGU0WHV1RzRsYUNidFJBR0pFcVNh?=
 =?utf-8?B?b2taVzVtTDQreU9lWU9uTCs5VmNaV2ROMFBhZ09mclhTSHlmOEJXVElQVzdM?=
 =?utf-8?B?aG9ZTXVZNks2YXVxTmpPVndNYTZvdVA1SHd0L1o2blBsb3dOOStoUnU1czVy?=
 =?utf-8?B?Y1pyWE1zWjdjV002RUQzL3kxVnNURUd6czdZUGEveXBmY2ZqUjREMktyZnpi?=
 =?utf-8?B?cXZPTWRXZS9zNElTZkp6ZUNUQ1k2N0dMYjB3WVQwZTRzUHlENlE3a3NveWlJ?=
 =?utf-8?B?bFNQcndrOXluWFFTU2UzVnFlRFRZQi8yQ2J4dnp2OXBaSzYwR0FwUStPdDJF?=
 =?utf-8?B?blFNUVVGSjBsL25XNUUyMkdYRHo3TXJ4WU1lSDZqUlhCY0pGc1NHSEE2VXZH?=
 =?utf-8?B?eUpoa3pNQkNjRXZKeDNPd0tQUXNZelpXTHc1ak5TQUtaTzFuNUNBRUNnaUo3?=
 =?utf-8?B?ZEphMXlVVS83TXYwMUVpdXJxOG1jYnZoSjRyNERDWi9YcCt4TUs4ZlBJZEtX?=
 =?utf-8?B?Wmh3cmVDd3dGMDB0UEhCQ3VUSGQzZE1YNkZ3YzVrTS92a3VhMWZSbWM2QVMv?=
 =?utf-8?B?R0RQRU1JK3N0SWVKVVNydlBXbVA5VkdXYTRTSVMyUGl0ZXlNMkljYkw3bHNP?=
 =?utf-8?B?bnRzaDdMa0d1UG5oSFFRL0xPUjhZTlRPck1lTlhyREJ3QnFGbGU2b09QTUlD?=
 =?utf-8?B?VG9DeS9vS3pESEVFdmtpZFNSOGtYanhoNGorblZWUHFrbXY2enJ3ZWxnMWpj?=
 =?utf-8?B?UjZFV1Jidmcwc2Nld3ZveWhvM1JITUdxcGhOa3ljbjhJdmdXSHNma1NvaS8x?=
 =?utf-8?B?TEZPaHVkcHpjNHA4M3RCVXJOYTdFbjl3NEp6N2VYbHlvYzBBdkdHbTM3aFYz?=
 =?utf-8?B?cUFFSmpnUmpYdEExMFFOYko0L054RDZkZkgxdG92cHo4NlA0UWZ6aGlDMDFu?=
 =?utf-8?B?MjNwbGhBQjgyaWVEUnBlVlFnWFBKcmo4c0tGUXFvQmUxeld6WTFiTjJtd3Zm?=
 =?utf-8?B?UG1nRFhBM3Rxd1FSNHVVazdTeXh1VFYzTGV0U3dxSnV4RjBSTTB2d2M4d2lN?=
 =?utf-8?B?cEFDM1JwTHZXRDd6aHhqRlV1NmY3QUhmb1ZyMzlGMDNGYzY1REU3RTQrNk5q?=
 =?utf-8?B?ckd1WFd5cXlidVhRTFk0WE1yMnRKSERiVnU2ZjE0YVlrV2V3bnNtdUZncFFi?=
 =?utf-8?B?OVpNZFp3SlpyVzZDekZ0VTBBYTJDcVAyc1ZwVUZUMmdXZVFXWVZ6UnE1azYx?=
 =?utf-8?B?Znl2YjY2MUVveEYxMnJrbWhyTEx0N3JTQTBOdUlLVTFnZ05iTDVMRDNRWklu?=
 =?utf-8?B?RWxVOFh6T1gzanFKZmRCd1dzcGQycTVRNWx2VDhRSVBINFBPTEFMOG10OURk?=
 =?utf-8?B?RFgvMnNsSitxZk05YzhnRXhQbHczT1V0ZXh0cnJrdGV1b0xvbCtIM2txbHdV?=
 =?utf-8?B?TndtMGNtSW1TMEltUWpiR1Y1WW1odjBFcEtWak9aaW1MOUlDK3ZhTVVIMUJ6?=
 =?utf-8?B?Y2k0eVJkZDdXSXhqamdneFNZOUFHTHVQYk5OSGpraElxMFpyeEFRclUwOGUw?=
 =?utf-8?B?SU9nMTVpTjVNaXNBdU9LV2JzYTZWby9MTk5HK2dBME02a0RJWUhXNjh1T0ZC?=
 =?utf-8?B?OFhaK1pZbXJjZFA3cWdMME5hUUxYTmU1Tm5KdVd5blUzZjhTNjhLTktNTVNz?=
 =?utf-8?B?UHh6SThNRTMxLy9TWWVYbzJteDl4bFZBWHZTaWZXS2tiVFBzWSsyeEgvOWZG?=
 =?utf-8?B?V1NZM3BZZ05kMFNVdGZwdGwyQWhkY3JZcW53Smc2R3YrcFFPL0VSdGFyeG1u?=
 =?utf-8?B?NUE9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: as75urqHHebg2L4hZuwQA5jqeC0JyYsel7z/AUa3NS//+ZjeeVjAp0ZUSy06WDR0OLzKH4fVIKqKUlZWbR+sQY7ySkGLaC4anUbPLUS3gy6ZcL9/PbSHTpQx1wa7ihY8QUe9kPTAnynYw5602AElO6b6y6bikJZ+xZMEJUhYQldqj3l9GGBpaOyqX/7d21WZNPF7DClXQJW4dNAgYJLeKuYz7KxMSptD+jVceptjTDPSQvJcgsta6zdZzEzrHI+sBVQIgqI8d3pZ26Py5YE0d7oS0YBgLtd+ml4vrDmrH//MIqPrHJHu5+1Pz8LIeHlMD7xz9c6+WXLE4GOg6en+sinY0Is7zwB6/mC9Uw1SHveEJTYIBb75/HGMfOYrLZhMEDVgOoeecJIMbq8C1lW2pgAQaQOjdtgj9xT2IZkgt/S1JhL8Ez6PILIOfD4pmBPOzHFPK6pXKTIVdp+FI3GGZgyTLeikavOzlX6b/k7CYqVZLFhsHrBKRaj8vQmEBpAVc/NfmoIBskcDTlpHmbcePEQS+3aZin2lCRBZoshWPaJ78qStvu+us3dB2Vzr39iAo2z2jHCqWUmYjM6o5KHRH/2sBN4CZDaLAUBEwQgmcSjtvI1dG5xZaj1D+RZRFIsLyTaUoSJlbHYxar6nQeazpgs98/u/lL8too2V+/xNPb4DCWrC+GSRvFKJpf5v7nUzYfTYTyyFEignma/Tp2nl3bujQjtDkUvt28Z84wSZbfXvikR4mUVfypLTH78qrrfymINwWp6HntjEnzJMQEdlJgF/S8kBVrmSEDOqFcmKs7zMwQwwzGlLZhW+24UcJqM0rF+A5kd0RJTm1pMxllHRjO9m2BAGT7cSgWLnxUQbMGc=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a84bb386-dfc5-48c8-2744-08db3a0dc3fb
X-MS-Exchange-CrossTenant-AuthSource: DM5PR10MB1466.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Apr 2023 21:51:40.2049
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /A3F4PpbJqjVnrNfw8idZIDeEw4PkQyLyVOB4m+7y6E7HNlh42J/AO5YEdCOwy6poslN0QGo/w0J2qIeGVWl6NRP/KpXamxi+iIeLoGaFv0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB5593
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22
 definitions=2023-04-10_16,2023-04-06_03,2023-02-09_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0
 mlxlogscore=999 spamscore=0 suspectscore=0 bulkscore=0 adultscore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2303200000 definitions=main-2304100190
X-Proofpoint-GUID: De4bU0xC4ugJcztNyh2XDxNmWLF4s2aO
X-Proofpoint-ORIG-GUID: De4bU0xC4ugJcztNyh2XDxNmWLF4s2aO
X-Spam-Status: No, score=-4.1 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
        DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 4/10/23 10:01 AM, Mathieu Desnoyers wrote:
> Introduce per-mm/cpu current concurrency id (mm_cid) to fix a PostgreSQL
> sysbench regression reported by Aaron Lu.
> 
> Keep track of the currently allocated mm_cid for each mm/cpu rather than
> freeing them immediately on context switch. This eliminates most atomic
> operations when context switching back and forth between threads
> belonging to different memory spaces in multi-threaded scenarios (many
> processes, each with many threads). The per-mm/per-cpu mm_cid values are
> serialized by their respective runqueue locks.
> 
> Thread migration is handled by introducing invocation to
> sched_mm_cid_migrate_from() in set_task_cpu() and to
> sched_mm_cid_migrate_to() (with destination runqueue lock held) in
> activate_task() for migrating tasks. set_task_cpu() is invoked with and
> without source rq lock held: the wakeup path does not hold the source rq
> lock.
> 
> sched_mm_cid_migrate_from() clears the mm_cid from the task's mm per-cpu
> index corresponding to the source runqueue if it matches the last mm_cid
> observed by the migrated task. This last mm_cid value is returned as a
> hint to conditionally clear the mm's per-cpu mm_cid on the destination
> cpu.
> 
> Then, in sched_mm_cid_migrate_to(), if the last mm_cid is smaller than
> the mm's destination cpu current mm_cid, clear the mm's destination cpu
> current mm_cid. If the migrated task's mm is in use on the destination
> cpu, the reclaim of the mm_cid will be done lazily on the next
> destination cpu context switch, else it is performed immediately.
> 
> The source cpu's mm_cid is _not_ simply moved to the destination cpu on
> migration, because passing ownership of the mm_cid value to the
> destination cpu while an actively running tasks also has its own
> mm_cid value (in case of lazy reclaim on next context switch) would
> over-allocate mm_cid values beyond the number of possible cpus.
> 
> Because we want to ensure the mm_cid converges towards the smaller
> values as migrations happen, the prior optimization that was done when
> context switching between threads belonging to the same mm is removed,
> because it could delay the lazy release of the destination runqueue
> mm_cid after it has been replaced by a migration. Removing this prior
> optimization is not an issue performance-wise because the introduced
> per-mm/per-cpu mm_cid tracking also covers this more specific case.
> 
> This patch is based on v6.3-rc6 with this patch applied:
> 
> ("mm: Fix memory leak on mm_init error handling")
> 
> https://lore.kernel.org/lkml/20230330133822.66271-1-mathieu.desnoyers@efficios.com/
> 
> Fixes: af7f588d8f73 ("sched: Introduce per-memory-map concurrency ID")
> Link: https://lore.kernel.org/lkml/20230327080502.GA570847@ziqianlu-desk2/
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Cc: Aaron Lu <aaron.lu@intel.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Olivier Dion <odion@efficios.com>
> Cc: michael.christie@oracle.com

Hey thanks for fixing this.

When testing linux-next with vhost devices, without this patch IOPs get stuck at around
1.3 million IOPs total when using 8 or more devices (you get a worker thread per device)
per VM. With this patch applied IOPs scale again, and we get up to 2.4M iops when using
up to 16 devices per VM.

Tested-by: Mike Christie <michael.christie@oracle.com>