Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5;
Message-ID: <9a66b7f8-097b-45af-97b2-4c403c295301@intel.com>
Date:   Thu, 14 Dec 2023 10:53:24 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v7 21/24] x86/resctrl: Allow overflow/limbo handlers to be
 scheduled on any-but cpu
Content-Language: en-US
To:     James Morse <james.morse@arm.com>, <x86@kernel.org>,
        <linux-kernel@vger.kernel.org>
CC:     Fenghua Yu <fenghua.yu@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        H Peter Anvin <hpa@zytor.com>,
        Babu Moger <Babu.Moger@amd.com>,
        <shameerali.kolothum.thodi@huawei.com>,
        D Scott Phillips OS <scott@os.amperecomputing.com>,
        <carl@os.amperecomputing.com>, <lcherian@marvell.com>,
        <bobo.shaobowang@huawei.com>, <tan.shaopeng@fujitsu.com>,
        <baolin.wang@linux.alibaba.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Xin Hao <xhao@linux.alibaba.com>, <peternewman@google.com>,
        <dfustini@baylibre.com>, <amitsinght@marvell.com>
References: <20231025180345.28061-1-james.morse@arm.com>
 <20231025180345.28061-22-james.morse@arm.com>
 <9084cb79-22bd-4cb3-b48d-f0d8d71aa47c@intel.com>
 <b077b38f-b42c-f679-1e08-70b55d116e17@arm.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <b077b38f-b42c-f679-1e08-70b55d116e17@arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dS91QUgxYXFmUEN2em44TFI1blJNVGk5OUMxcHM4WmZVamczUVdQWHF4eVpZ?=
 =?utf-8?B?RG1DTEdrZm8zSEMvVTZvbGFYNFk5OGdCRnNLcnZ5cU1iTjJrNjVac1JwSmZ1?=
 =?utf-8?B?SXF1NU1nNVJVM1lMQUNZbjk4WnBEVzAxRm9rZWdlN3hhRkpoN2c0RWRYREp6?=
 =?utf-8?B?R1FrakVYM2NmbSsxa0UzVUp5WTQyejJoYXpXTU1SUFVOWlNOTHVYK3E1Qnky?=
 =?utf-8?B?K0dHN0dhQmxqWXJYYnRIL0NMUDJHTEZZdTdnN1ltVGs5NmRSeC8xWmFiS2Ja?=
 =?utf-8?B?clpqNjhQaUtMNTB5MS9EODFnUDBVRGRTNzVlNHlKN0tCT29Ra2RJbkUvbXhW?=
 =?utf-8?B?cHY1TkJVUFNDcEo2eWIvN3JyUTVDOVAyR0ptamZQbDF6aktxbTdJS1Y5MC9L?=
 =?utf-8?B?VUVxRkd3alRybGtIMmw1MnQvcGhvcGtPOTl0OTZsM0hOMWgrWERHTXVFS2Qx?=
 =?utf-8?B?NW50bkRhdmxQOVFhTVNCaFFiNzZra29RN0t0ZWt3TXdNZ1NZSmNWZWRZaEJM?=
 =?utf-8?B?bkhMNTN5MjhwVDl2OEVvNldoYlRSZWJBU2hDQXVmakgzR21QZ292OHQ3S2hC?=
 =?utf-8?B?cDlYbzBHZUlBanhtRXR3MWRDRG1BZldTS002Yko3V1RJeVdxbzZJaCtmZ3FG?=
 =?utf-8?B?M0JWakpvaG1IaWJ5bG9GdFFUeW1Db1ZkenNnZ0JpaXZ4bXBhN0tObE15QjNR?=
 =?utf-8?B?WWxKL1NTOUtET2tWdUVlaVNBTVJ2SFo3a1Zka1U2M21OdVVKcWdRTWlKU1RY?=
 =?utf-8?B?N0xzdnloRTlEVkwwVS94TXJUZVdJMmN1Wmxxd3RIRjA0L0ZmM0xRRTdRRU94?=
 =?utf-8?B?Tzh0SFM2RW0vd2luS21hS01zTjE0dm9POU9YcU1BYnJQamRrT1VESGJNcjNR?=
 =?utf-8?B?Sis4Q25nSWlRUEZyM0phekdaaitVVlJQR0VPbE11OVpnYzVhZmZvUE5BOHZz?=
 =?utf-8?B?bHFvMGxPZVpIOXRUd0tRK215T244amZJbHZadk5ZK0hUVHZjN3dOVjBtNkh0?=
 =?utf-8?B?UTIwK0xMQnFWYXJSOHFOazBVYUhVU3FYOFNETjd6OWt3cWtQc3p6NnRhSlJk?=
 =?utf-8?B?UVp2V3hrQUd1ZUJzK0x0RmlMWjFFTzgrOEFXTU94NGNIaWNHSHdtMWxPNFBL?=
 =?utf-8?B?YTdFZFl4R3l5ZXV4dEpYMm9jemVId0o4VXI5QVJvZE8xSnh4ZmdqS3FuZzFI?=
 =?utf-8?B?V3NaTE56L2laVmRZblpJUmp4NXRlRUNESkUrK0FzakkvbXc4bXJ5ekJlRGlJ?=
 =?utf-8?B?TThkT0FpNmZzK0U1Q3diTXBSVGcvWUt2QzBtK0dHS1JxUFIzd3I2a2ovTnVm?=
 =?utf-8?B?WnoyNjF5WHRsM2RaVnRoS0xOL1BOWFNBL1dGb2hmOGZGcDFQVkNKSUJpaVgy?=
 =?utf-8?B?WERtVzlVUm9SRy93TUtpblIxWXlvb0ZKSW1YVkdkaXA4QzdkNTUyMzZHVk5O?=
 =?utf-8?B?cGNTSmE3YUdRTGNic2Z3eStVRmM3WitoWnJrR3RDeldBbE5MYWVNNmVYQjZu?=
 =?utf-8?B?Z0Q2dkFCZzJoQ3JBYkVtbDBMeE83WDBpVTVFc1NJaDdxU081WVVOeTlReVJm?=
 =?utf-8?B?V3lDaFJVVUlnSGhURkJjZThXVWFDWDRDS29kNGVocURCOFJFNWcxMFk3Kzdj?=
 =?utf-8?B?UzZjak1HQy9lc1JTblVGdFY0SmlQcElQSi9CYi9neWRqcVF1bURuVGI0TDdJ?=
 =?utf-8?B?Z294QTFMUC9RemhtMnpIUnpPYTVZRkVqcXA2VHREbHR6bVIzUGpJRDA5VTVV?=
 =?utf-8?B?T3RzWVJ4bkN3aGJqUVpUSlhnVkZUckM5VmF5bUUvdEhiNTlpZEZEc0UyelJS?=
 =?utf-8?B?QWNOZXcrVG1jRkdUZEtWVGNWZGcyeDh5bGUrbjBLS0Z2THRTT2FkNjN3SGYr?=
 =?utf-8?B?ZkVSZEVrejByWW16NzZIOEtDaDh3QnlrWnZhRjh1MnBabm5yU0tPUzNXOTFr?=
 =?utf-8?B?aDZ3WDFINnM5bi9KUTJZWW4wd0ZhdWQ2NmNYRWpJc3hoc1E5NUk5YnBSa1FR?=
 =?utf-8?B?UDFiSldNK0ZJRVR4ZWZOYXBIZFQ5aWZtaGpDcWJjcnNWUnNrY050RSs3Rmc3?=
 =?utf-8?B?WE1laHZuZFhLSGN0anlHc1owdjVBZFZISmdCbThrelBSRjltWFpnOUxZczNU?=
 =?utf-8?B?aGxTYmllOE82R1B5czEyREMwSzVkVHJjTW9xMkIwOHdVeG1uZVdVK1RYWG1H?=
 =?utf-8?B?WUE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 88447f1b-abcd-49f8-9ed4-08dbfcd5f3e9
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Dec 2023 18:53:25.4709
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: a0QSVHmmubGx9XB/A55JHfOaI4WCElovub4BrsQE8cayAz0DMM7LuNDmwRL+v8+pOf5v12O52/zeCt8cSNWAwinXjWDdJGCafIFAv0rehsg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB6242
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Thu, 14 Dec 2023 10:53:47 -0800 (PST)

Hi James,

On 12/14/2023 3:38 AM, James Morse wrote:
> On 09/11/2023 17:48, Reinette Chatre wrote:
>> On 10/25/2023 11:03 AM, James Morse wrote:

>>> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
>>> index c4c1e1909058..f5fff2f0d866 100644
>>> --- a/arch/x86/kernel/cpu/resctrl/internal.h
>>> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
>>> @@ -61,19 +61,36 @@
>>>   * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that
>>>   *			        aren't marked nohz_full
>>>   * @mask:	The mask to pick a CPU from.
>>> + * @exclude_cpu:The CPU to avoid picking.
>>>   *
>>> - * Returns a CPU in @mask. If there are housekeeping CPUs that don't use
>>> - * nohz_full, these are preferred.
>>> + * Returns a CPU from @mask, but not @exclude_cpu. If there are housekeeping
>>> + * CPUs that don't use nohz_full, these are preferred. Pass
>>> + * RESCTRL_PICK_ANY_CPU to avoid excluding any CPUs.
>>> + *
>>> + * When a CPU is excluded, returns >= nr_cpu_ids if no CPUs are available.
>>>   */
>>> -static inline unsigned int cpumask_any_housekeeping(const struct cpumask *mask)
>>> +static inline unsigned int
>>> +cpumask_any_housekeeping(const struct cpumask *mask, int exclude_cpu)
>>>  {
>>>  	unsigned int cpu, hk_cpu;
>>>  
>>> -	cpu = cpumask_any(mask);
>>> -	if (!tick_nohz_full_cpu(cpu))
>>> +	if (exclude_cpu == RESCTRL_PICK_ANY_CPU)
>>> +		cpu = cpumask_any(mask);
>>> +	else
>>> +		cpu = cpumask_any_but(mask, exclude_cpu);
>>> +
>>> +	if (!IS_ENABLED(CONFIG_NO_HZ_FULL))
>>> +		return cpu;
>>
>> It is not clear to me how cpumask_any_but() failure is handled.
>>
>> cpumask_any_but() returns ">= nr_cpu_ids if no cpus set" ...
> 
> It wasn't a satisfiable request, there are no CPUs for this domain other than the one that
> was excluded. cpumask_any_but() also describes its errors as "returns >= nr_cpu_ids if no
> CPUs are available".
> 
> The places this can happen in resctrl are:
> cqm_setup_limbo_handler(), where it causes the schedule_delayed_work_on() call to be skipped.
> mbm_setup_overflow_handler(), which does similar.
> 
> These two cases are triggered from resctrl_offline_cpu() when the last CPU in a domain is
> going offline, and the domain is about to be free()d. This is how the limbo/overflow
> 'threads' stop.

Right ... yet this is a generic function, if there are any requirements on when/how it should
be called then it needs to be specified in the function comments. I do not expect this to
be the case for this function.

>>> +
>>> +	/* If the CPU picked isn't marked nohz_full, we're done */
>>
>> Please don't impersonate code.
>>
>>> +	if (cpu <= nr_cpu_ids && !tick_nohz_full_cpu(cpu))
>>>  		return cpu;
>>
>> Is this intended to be "cpu < nr_cpu_ids"?
> 
> Yes, fixed - thanks!
> 
> 
>> But that would have
>> code continue ... so maybe it needs explicit error check of
>> cpumask_any_but() failure with an earlier exit?
> 
> I'm not sure what the problem you refer to here is.
> If 'cpu' is valid, and not marked nohz_full, nothing more needs doing.
> If 'cpu' is invalid or a CPU marked nohz_full, then a second attempt is made to find a
> housekeeping CPU into 'hk_cpu'. If the second attempt is valid, it's used in preference.

Considering that the second attempt can only be on the same or smaller set of CPUs,
how could the second attempt ever succeed if the first attempt failed? I do not see
why it is worth continuing.

> An error is returned if the request couldn't be satisfied, i.e. an empty mask was passed,
> or the only CPU set in the mask was excluded.
> There is a second attempt in this case for a housekeeping CPU - but that will fail too.
> As above, this only happens when CPUs are going offline, and this error is handled by the
> caller.

Reinette