Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <5e6a2e0a-6f31-c9b0-5eec-346fd072d286@intel.com>
Date:   Fri, 31 Mar 2023 16:25:40 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.9.1
Subject: Re: [PATCH v3 09/19] x86/resctrl: Queue mon_event_read() instead of
 sending an IPI
Content-Language: en-US
To:     James Morse <james.morse@arm.com>, <x86@kernel.org>,
        <linux-kernel@vger.kernel.org>
CC:     Fenghua Yu <fenghua.yu@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        H Peter Anvin <hpa@zytor.com>,
        Babu Moger <Babu.Moger@amd.com>,
        <shameerali.kolothum.thodi@huawei.com>,
        D Scott Phillips OS <scott@os.amperecomputing.com>,
        <carl@os.amperecomputing.com>, <lcherian@marvell.com>,
        <bobo.shaobowang@huawei.com>, <tan.shaopeng@fujitsu.com>,
        <xingxin.hx@openanolis.org>, <baolin.wang@linux.alibaba.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Xin Hao <xhao@linux.alibaba.com>, <peternewman@google.com>
References: <20230320172620.18254-1-james.morse@arm.com>
 <20230320172620.18254-10-james.morse@arm.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20230320172620.18254-10-james.morse@arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cG1TREc4MWZFU1FRbHpZQW9tdlVSSEhZRDNxblFtakNBMGloVlkra1VKMlhD?=
 =?utf-8?B?SStjWUNjeXNLQVhtdjJPb3ZMNkphMng1ck9WQlhpZjV0eDc3RzVQMFlDQ0dt?=
 =?utf-8?B?TUtQMEk3TFpnN3N6OEhkVWdXRVNjV25MUGYyenZHL2d5OW1EUGZwem96S2U5?=
 =?utf-8?B?cng0Q3U1SmN5azBHRVRGbnpNQXZydjBIdHd2S1k3U3lOKzRQU2hjY1dsNkMy?=
 =?utf-8?B?bUgxeGRPTjl3K2xVazNBSFQyd0lXL2FJYlRUdHppdnk5V2FLM3hKOXgwQzBR?=
 =?utf-8?B?VXVzU2dXZkJmQUNSaFVGeFBBZWxYeHFQSGVaYlNVaGJaN1lESWdJWnBwQnlI?=
 =?utf-8?B?VVUxdTl3YkY0LzgwZXY1ZElDR3ZmR0JkcWlnK2NxNkxmWTJrQi9GakRBZ2R4?=
 =?utf-8?B?MVRaYlFoQ0hVQ0tMRlhVZEpnVEoyZWQvZkVJSjdsR210d2RkMERqT0w4YjQw?=
 =?utf-8?B?TkFlY2JBOFljOWEzSkJ3VnVsWmdIak1WMjEydEJTY0NPMlhGdGZlSW1kRUxr?=
 =?utf-8?B?bnJoaTROdzRBSmRHeWx2RnJjcnlZK0Z0N2ppSDdMRnoxWnJZUDVtWnZVbXJV?=
 =?utf-8?B?SFdtL2kxb1BJd1VvYitYTExYOVMybHR1WjhpdldNamdoK0liaEx4U21pb295?=
 =?utf-8?B?RXpMclRIRGlQQkw3UFBiY291ZDB2T3lpWVpQUmsxNmo4RXViRk40Rmt4U0tK?=
 =?utf-8?B?VGttZzdlSE9hRGZvd0dpT1R3TE5WNTNnWFhQYjhrdENhaWM5TThCdDBtWGlW?=
 =?utf-8?B?Q2taRndzNFJBN1dwLzRXVFEwVHgzejJZY1VJSUZnYkt6NS9LTVlock1ncGdI?=
 =?utf-8?B?YnVNQXZUMHppS3pXclBKNWdpcnFRdC9BVWV4MHFGOE9FczNGS05EZDZEMHhB?=
 =?utf-8?B?cnE3QVhwaXVGSkoxem8rQlNlWHUwZDQ1VTdPYjN4MzQrLytzT3JDYnRyQ2dh?=
 =?utf-8?B?c1g4bEFjRVlZQko5NGVxZE1QTTJhaWhRb2lXd240MzJHbzVyUzhrZk5PUS8v?=
 =?utf-8?B?azhiUkd3UVBQMWpqYlB1cXNFSmp6WTVvN09ObE43czB4b29jdVJuUlEvb2FR?=
 =?utf-8?B?WVUreHZBZW5DZjllUWZqcDczQi9zMzluYk9wODhsODJZZVN4Y3B1eExvWHd5?=
 =?utf-8?B?Qm84ejJOa0M0Uy95cmxDSHhFVG03TElVUkU1LzJOcGEzR0t2TVNPalcybHRB?=
 =?utf-8?B?NmI4L1BOUGdFejcvbWJ1YU1QcXBqcXZPeURRNFd1cWJPVHB1MGo4VkNCR1Bo?=
 =?utf-8?B?dW9zTHdsb09HcWpsTloyaGFVZUxEcE9JOVAvT0VPL242U3pIeitVZE9JQ0NV?=
 =?utf-8?B?UHBuR29qakNnTDIrQWQ0OHZ1ZVpTa3V6Y3JzTE9NQnNwaDFja245RHNHV3N2?=
 =?utf-8?B?NU5ZMTdFZXFBVnlnUkdzNUdDQzRoa0tRN2xhemRvODNjNmhwdGtpYzZHVGQw?=
 =?utf-8?B?T1RobWtKTTYvc1Jxc1hhdStFRmxhSXpSSnpnWGYwNE1sZjg4MzhYODEwYTdh?=
 =?utf-8?B?N0haUThlb21SWGVuWjMzOGhhYkVvbFZPU05Ic3ErR1l0eklvMGgzWEZsUS9U?=
 =?utf-8?B?Tk4yNjUxM0NJZHdSbUhXZExoWk1VYlY1b291SVNjNjRGcm5vdHpmNVJzVTZR?=
 =?utf-8?B?SThBR3NsSm5zSVM4WjFHUzlOakhCTDNhQVNrZktUc1krS25jK1I1dmFETGtS?=
 =?utf-8?B?cGdHS1BUWXpVYjZOVXFXZUNCSTc2ZnpDNXNCNlhxMnFld2o5RzlOWkh0bTNM?=
 =?utf-8?B?T2xmV1JYenNhVy9Hd2JCUUx5N2pVQlY2QVoydUVMSjI4ZzhOQ09VTmdZUFFl?=
 =?utf-8?B?VEdlQlNFeFA3M25FbUlrKzVsUVZhaWpjZW0yekpTMlhCWjlkZ1RJYTAyYmtx?=
 =?utf-8?B?ZFVIMkdRaU1STldQN3diUmFkVmNFTmRPZmJxYVhRWjdJRDNJYUFXZWtwYVIw?=
 =?utf-8?B?cjVsRkpMLytJNnZETWVYeGR5ZEVZL2ozcCtqVUtuSXQyNHIyRm1sV3Bwemky?=
 =?utf-8?B?UWgybjNQS2krUWd1My9qTnI1UmhpQjg4VVloLzUxVE0vZVBMSkZVcE5hMW9i?=
 =?utf-8?B?Ui9LSlJobUcxTVFLSmhqa1Fnck1FQ2ZZbk1uaTQzYndpcmRqZDNCNGV5Y3JI?=
 =?utf-8?B?Z0dWKzZmYk1zRkpvMW13aGZ5eVRqL081SEtQemtzMkJidzcvRUp2Rno5U1Jq?=
 =?utf-8?B?UlE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: dc59abf0-3911-4b66-0ce2-08db323f3f56
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Mar 2023 23:25:43.2335
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: nHyGqhFLm+5FqOZEI+A0ouclBK5tOGbN9ObjAxtnGl0R5klNDiNzF0X0iU8vJF4KUxEyouWK1w2bIwJP4GfEIBkgXksptzkAgOynhOJ7VJg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6426
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi James,

On 3/20/2023 10:26 AM, James Morse wrote:
> x86 is blessed with an abundance of monitors, one per RMID, that can be
> read from any CPU in the domain. MPAMs monitors reside in the MMIO MSC,
> the number implemented is up to the manufacturer. This means when there are
> fewer monitors than needed, they need to be allocated and freed.
> 
> Worse, the domain may be broken up into slices, and the MMIO accesses
> for each slice may need performing from different CPUs.
> 
> These two details mean MPAMs monitor code needs to be able to sleep, and
> IPI another CPU in the domain to read from a resource that has been sliced.
> 
> mon_event_read() already invokes mon_event_count() via IPI, which means
> this isn't possible. On systems using nohz-full, some CPUs need to be
> interrupted to run kernel work as they otherwise stay in user-space
> running realtime workloads. Interrupting these CPUs should be avoided,
> and scheduling work on them may never complete.
> 
> Change mon_event_read() to pick a housekeeping CPU, (one that is not using
> nohz_full) and schedule mon_event_count() and wait. If all the CPUs
> in a domain are using nohz-full, then an IPI is used as the fallback.

It is not clear to me where in this solution an IPI is used as fallback ...
(see below) 

> +	int cpu;
> +
> +	/* When picking a CPU from cpu_mask, ensure it can't race with cpuhp */
> +	lockdep_assert_held(&rdtgroup_mutex);
> +
>  	/*
> -	 * setup the parameters to send to the IPI to read the data.
> +	 * setup the parameters to pass to mon_event_count() to read the data.
>  	 */
>  	rr->rgrp = rdtgrp;
>  	rr->evtid = evtid;
> @@ -537,7 +543,16 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
>  	rr->val = 0;
>  	rr->first = first;
>  
> -	smp_call_function_any(&d->cpu_mask, mon_event_count, rr, 1);
> +	cpu = get_cpu();
> +	if (cpumask_test_cpu(cpu, &d->cpu_mask)) {
> +		mon_event_count(rr);
> +		put_cpu();
> +	} else {
> +		put_cpu();
> +
> +		cpu = cpumask_any_housekeeping(&d->cpu_mask);
> +		smp_call_on_cpu(cpu, mon_event_count, rr, false);
> +	}
>  }
>  

... from what I can tell there is no IPI fallback here. As per previous
patch I understand cpumask_any_housekeeping() could still return
a nohz_full CPU and calling smp_call_on_cpu() on it would not send
an IPI but instead queue the work to it. What did I miss?

Reinette