Received-SPF: pass (google.com: domain of linux-kernel+bounces-134622-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <98b3144d-4519-4378-9121-c1538b4c5b68@intel.com>
Date: Sun, 7 Apr 2024 12:13:09 -0700
User-Agent: Mozilla Thunderbird
From: Reinette Chatre <reinette.chatre@intel.com>
Subject: Re: [PATCH v1 0/6] x86/resctrl: Avoid searching tasklist during
 mongrp_reparent
To: Peter Newman <peternewman@google.com>
CC: Fenghua Yu <fenghua.yu@intel.com>, James Morse <james.morse@arm.com>,
	Stephane Eranian <eranian@google.com>, Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>, Dave Hansen
	<dave.hansen@linux.intel.com>, <x86@kernel.org>, "H. Peter Anvin"
	<hpa@zytor.com>, Peter Zijlstra <peterz@infradead.org>, Juri Lelli
	<juri.lelli@redhat.com>, Vincent Guittot <vincent.guittot@linaro.org>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>, Steven Rostedt
	<rostedt@goodmis.org>, Ben Segall <bsegall@google.com>, Mel Gorman
	<mgorman@suse.de>, Daniel Bristot de Oliveira <bristot@redhat.com>, "Valentin
 Schneider" <vschneid@redhat.com>, Uros Bizjak <ubizjak@gmail.com>, "Mike
 Rapoport" <rppt@kernel.org>, "Kirill A. Shutemov"
	<kirill.shutemov@linux.intel.com>, Rick Edgecombe
	<rick.p.edgecombe@intel.com>, Xin Li <xin3.li@intel.com>, Babu Moger
	<babu.moger@amd.com>, Shaopeng Tan <tan.shaopeng@fujitsu.com>, "Maciej
 Wieczor-Retman" <maciej.wieczor-retman@intel.com>, Jens Axboe
	<axboe@kernel.dk>, Christian Brauner <brauner@kernel.org>, Oleg Nesterov
	<oleg@redhat.com>, Andrew Morton <akpm@linux-foundation.org>, Tycho Andersen
	<tandersen@netflix.com>, Nicholas Piggin <npiggin@gmail.com>, Beau Belgrave
	<beaub@linux.microsoft.com>, "Matthew Wilcox (Oracle)" <willy@infradead.org>,
	<linux-kernel@vger.kernel.org>
References: <20240325172707.73966-1-peternewman@google.com>
 <4681c9c5-3364-4628-a581-0366d0024211@intel.com>
 <CALPaoCh0SbG1+VbbgcxjubE7Cc2Pb6QqhG3NH6X=WwsNfqNjtA@mail.gmail.com>
Content-Language: en-US
In-Reply-To: <CALPaoCh0SbG1+VbbgcxjubE7Cc2Pb6QqhG3NH6X=WwsNfqNjtA@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UXRNVU1ES1lpYUgwYW1DTnVmMU8vdGRBYUhCRnBMRW1PbW1mNzFwTVNSRmRF?=
 =?utf-8?B?K2MvTTZURkJJMGp0Vm5mc1ZQZjhSSzF1QkVpTDhUMG1CVGxBdGQzZFhRK0Z0?=
 =?utf-8?B?Uk1tVkFqZzF6ditNRStPMjhISDhvZUtOTzdaV1lOcnp1SnRlb2NPSStYaktW?=
 =?utf-8?B?OHFvSnlsZHVQdjRtOFErWG5pa1RPeExOZTZQRDByRUFpUDBCcGFkblc0Y2ZQ?=
 =?utf-8?B?c0dhL1JGUUhqZDR4dVBOenhMaXI1S2dHSzFJNlBzUm52L2lEeWpsREN0TVRL?=
 =?utf-8?B?UTVVZmNDYjdYY0labDFJVFd1YXlRUGZldlhkYTZ0M3hJZmpWZEU4ek5WQW9a?=
 =?utf-8?B?dTdxS2h2eEpjbFh0MGJQa2RkSmpNbFBEeEE2Q1UzWW4yZzVMN3ZJUXI3YVR6?=
 =?utf-8?B?UG1uWmdHeWVIK2tpV213cG1Gbkt6amN5TDFRbGFTVkhua0lQZHVOa01TN3Fi?=
 =?utf-8?B?eVAyNXNWT1p4RU9HeDNkTDNBOVpyTnlibFNheFJqTkpUUm94aHpCbVBwMlJI?=
 =?utf-8?B?cHVlYmdvMkZ1M0pXQUl4b1VTTXovSWx5Tk9tVFQ2SGRkajBQcFU5U2ZLY1NU?=
 =?utf-8?B?OHdKbnNEamNKNnBpOUw5RE9sc2FHemtCMmQ0YWxoUWw3ODE0ZXhJbDZKaUQ4?=
 =?utf-8?B?VHZDbVYydkZSVkMvVUROM2FBbTgxV3RFL3hYcFpTM1h1dGN2U3cvWWxuSnZu?=
 =?utf-8?B?Q3E1NVI0WU4zRzFZNEFTc1grSVdnWnVsU3FlNnN1Zmo0YU9vQ0drN0RKSjlJ?=
 =?utf-8?B?QlNwNWx2SzY3a2tkK0Z5dWRKTWpMT0JCMEtiMGNjNkZ6dGZxZndNd0lyMUwv?=
 =?utf-8?B?WlRvc1lkTGxhMzFNSzhVOGl5b2NGSWVCUUdiKzdYMDN3VUxBRkFPeUIxUWM2?=
 =?utf-8?B?N05QYm1pWjlMd2Q4b2phSjArUWRSSFFOMkU4a2pqeWlRbS9nS0pvWS9Hbnh5?=
 =?utf-8?B?MUQrOWNrK0t4dmJxN05HNjlWOU9oNlQxSDJxTjBDdGxDRW5RUTAyU2xSNDg1?=
 =?utf-8?B?Rk1pNGZBMFF5dE5tbGpWNmQvYXJBcGFKaEFrQU1UbXVFMVlRdy9kcFllNWJL?=
 =?utf-8?B?MWJZblN5UnVvckQ0akdJbUk4cGk4ZFIrOFhnUGlxL2dRU25jek9ZdXF2c2Fn?=
 =?utf-8?B?ZUlLVGk5YTM4R1duQlJHNVplZHBJVWZjZUt0MkM0UnJ2WHpMT04xWUEya2Rl?=
 =?utf-8?B?SjkreHVGTmhQNEpmUnA0QXhUOFlWWVRMWEdmSExwTlRNWFUrd21rUU9hMXlq?=
 =?utf-8?B?cVUxMjZ6bGdOZ1A3bGp3RnhieWZ1dVdZRFlSSlVYL21zL1JKdU1xdGdJRm5t?=
 =?utf-8?B?UThuQU1RcDY5dHRFTVpYNEpIS2F5MlIzZ2dTL05Od1F4ZTlvek9iNysxWmRh?=
 =?utf-8?B?cVFIdGZnNU9wU2p3dWtNelZOK2tBMGZKcUJZVGpDSFhUTlhXbkE3L2xJUGRP?=
 =?utf-8?B?Ry9lYWlWdjJmUStTZE1MRzh4L0c2R1RhNmhzNDJkYWw3VnRkRlY4Z2FRbW0r?=
 =?utf-8?B?N3dvcGxKWXNZdHc1R0FJS2s5WFN4OFRJWHRqU2cvN29CSStyVUhkZll1NTN2?=
 =?utf-8?B?OFp5ekZ6bzVvTDZWTG9RWnIzM2lyMjJpUUhwS1BVRjIyMEthQXV3aEV3K093?=
 =?utf-8?B?MXNOUCtMNS83bDZPeUtIOThLZk5idCt1UlZ6SWUvbDlzamNTd2lLa2M5N1h4?=
 =?utf-8?B?SDd0OEZoK0haZ2pRNDFOMXZheXJBcEhDbWYwNElpU2dPYlBTOVJxTHdDcXFJ?=
 =?utf-8?B?WXovdlZJRThzVmFkck1ROUJmYUpTMGJPNDVBL2N1Q3B0TXFnNFI4T3RVeXlT?=
 =?utf-8?B?Z1FRUmUyNlQzdWN1RGsxOXRhQ3dsU2U0U1NVSzZ0elVhY0ZKRFRwY0JTcFJE?=
 =?utf-8?B?bytIWGEwS1h5ZUZVMVBma3hKY0QrMlpkNUdJaEs1Y2J2bDE4UWVkUmdLZDNO?=
 =?utf-8?B?eXZ6NUUwT0xqTkpFdDg5MjBCaWVicTQ0em1McGxqeWx6TEpJUFhUMElIWFFC?=
 =?utf-8?B?Mi9LRVY4cnFlWlh5S2xpSWZqY1QyQkhNa0UxaEtMd29YUUtkUzJEL2tlRUM3?=
 =?utf-8?B?cVlRTmRGRTB0K1c4dmlHVFFjNHlaUDJteXBtb0dZMEtPdUVZTHlmRFl4ZUpy?=
 =?utf-8?B?OXRJNjB4dnBYRUkzVnh3eHRNeUFkQXhuWVp4S1RIT3dZRmZHT1UxUzVaNkFo?=
 =?utf-8?B?QlE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: bd4eb9ea-d57e-43f7-9d1c-08dc5736c4df
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Apr 2024 19:13:12.4279
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 6zPbtB46t4yTDqsC/fMG1vy+ob6uQ8d9H/NBpWLuEMdySyTPvmmoLubP7oNP8rESnDi4XprOvG1jKvJ17pCqOawAiS5nL7Cy37oQc5z58e8=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB7569
X-OriginatorOrg: intel.com

Hi Peter,

On 4/5/2024 2:30 PM, Peter Newman wrote:
> Hi Reinette,
> 
> On Thu, Apr 4, 2024 at 4:09 PM Reinette Chatre
> <reinette.chatre@intel.com> wrote:
>> On 3/25/2024 10:27 AM, Peter Newman wrote:
>>> I've been working with users of the recently-added mongroup rename
>>> operation[1] who have observed the impact of back-to-back operations on
>>> latency-sensitive, thread pool-based services. Because changing a
>>> resctrl group's CLOSID (or RMID) requires all task_structs in the system
>>> to be inspected with the tasklist_lock read-locked, a series of move
>>> operations can block out thread creation for long periods of time, as
>>> creating threads needs to write-lock the tasklist_lock.
>>
>> Could you please give some insight into the delays experienced? "long
>> periods of time" mean different things to different people and this
>> series seems to get more ominous as is progresses with the cover letter
>> starting with "long periods of time" and by the time the final patch
>> appears it has become "disastrous".
> 
> There was an incident where 99.999p tail latencies of a service
> increased from 100 milliseconds to over 2 seconds when the container
> manager switched from our legacy downstream CLOSID-reuse technique[1]
> to mongrp_rename().
> 
> A more focused study benchmarked creating 128 threads with
> pthread_create() on a production host found that moving mongroups
> unrelated to any of the benchmark threads increased the completion cpu
> time from 30ms to 183ms. Profiling the contention on the tasklist_lock
> showed that the average contention time on the tasklist_lock was about
> 70ms when mongroup move operations were taking place.
> 
> It's difficult for me to access real production workloads, but I
> estimated a crude figure by measuring the task time of "wc -l
> /sys/fs/resctrl" with perf stat on a relatively idle Intel(R) Xeon(R)
> Platinum 8273CL CPU @ 2.20GHz. As I increased the thread count, it
> converged to a line where every additional 1000 threads added about 1
> millisecond.

Thank you very much for capturing this. Could you please include this in
next posting? This data motivates this work significantly more than
terms that are not measurable.

> Incorporating kernfs_rename() into the solution for changing a group's
> class of service also contributes a lot of overhead (about 90% of a
> mongroup rename seems to be spent here), but the global impact is far
> less than that of the tasklist_lock contention.

Is the kernfs_rename() overhead in an acceptable range?

>>> context switch. Updating a group's ID then only requires the current
>>> task to be switched back in on all CPUs. On server hosts with very large
>>> thread counts, this is much less disruptive than making thread creation
>>> globally unavailable. However, this is less desirable on CPU-isolated,
>>> realtime workloads, so I am interested in suggestions on how to reach a
>>> compromise for the two use cases.
>>
>> As I understand this only impacts moving a monitor group? To me this sounds
>> like a user space triggered event associated with a particular use case that
>> may not be relevant to the workloads that you refer to. I think this could be
>> something that can be documented for users with this type of workloads.
>> (please see patch #6)
> 
> All of the existing rmdir cases seem to have the same problem, but
> they must not be used frequently enough for any concerns to be raised.
> 
> It seems that it's routine for the workload of hosts to be increased
> until memory bandwidth saturation, so applying and unapplying
> allocation restrictions happens rather frequently.

Thank you. 

Reinette