Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <c9bf21a0-3dab-5605-688d-d9de634e88c4@intel.com>
Date:   Fri, 16 Dec 2022 16:59:29 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.6.0
Subject: Re: [PATCH v2 1/2] x86/resctrl: Fix event counts regression in reused
 RMIDs
Content-Language: en-US
To:     Peter Newman <peternewman@google.com>, <fenghua.yu@intel.com>
CC:     <Babu.Moger@amd.com>, <bp@alien8.de>,
        <dave.hansen@linux.intel.com>, <eranian@google.com>,
        <hpa@zytor.com>, <james.morse@arm.com>,
        <linux-kernel@vger.kernel.org>, <mingo@redhat.com>,
        <quic_jiles@quicinc.com>, <tan.shaopeng@fujitsu.com>,
        <tglx@linutronix.de>, <x86@kernel.org>
References: <20221214160856.2164207-1-peternewman@google.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20221214160856.2164207-1-peternewman@google.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MXZuM1IvdmpLblRVdnlrOEYrV1BvVXllS2F3VWo2WXBmMjkzdWVnMkYvUFVR?=
 =?utf-8?B?bVRQWnRERlY0ZEtXNHFPc0w2M0YvN2ZoQmhsMlhOblFrZURLMGZOdmxFejFy?=
 =?utf-8?B?b2NuZkVhQUQweHc3OHNFL3ZGWTg2ZzdIMWJGck8yY2xaRXZzR0c3TVRYMjc3?=
 =?utf-8?B?YUNZaWQrY1V1b3VNZnNiRHNzMW5FMTlxSmVBdHhPYUQvVUcvM3lYZDRVeUtj?=
 =?utf-8?B?Y2t5K1ZSSVU4YzdIWGhtN25VMzNEMitvb0VVM3ZKMmJ0eCtBcTQvWTU0TjVC?=
 =?utf-8?B?TWRjNDdEQjBGTFE0V3MvcGxqYnNJTVlDb0dtSmlUUEIyWUF2NEUwV2JWL01S?=
 =?utf-8?B?RGd4NDhMMnpkTk1kdWgvRWp5dkdHNVZsdmpDQnZGT0JaY0M1RGY4ek1jV1BH?=
 =?utf-8?B?TWkxZFo5TXNtbU13VFo1b29VUGROUy85WUREODlCSTQ3SkI2RDdlVUZhZHha?=
 =?utf-8?B?L1l2bzZDL0R0NFZtTFAwV0dkWXBud090UnFiSk9tTHp5elFKSDRnTHRqcUJ0?=
 =?utf-8?B?bHJtSjMzbEdrM2svakxqdnZGUWt3c0xNSzVsUmhvakJKNm14WGlkeWVoWlpL?=
 =?utf-8?B?NjZGM2NmUHczY21kN1QyV0ZpTE1KaURyaldCOEN6RzJPc1o2bGV0NU8wVU40?=
 =?utf-8?B?bHRWYnN4V1ZJcXdDYzBpcXJMTDduUmo5T0tIZ1FGYUNYby82MmQ4THNHcUZI?=
 =?utf-8?B?Ukphb0U3RmdreFZPajRJWVA4WjBYSTRRNlhRbVBiSlRlS2ZTcUpINW85Vks2?=
 =?utf-8?B?eU9pbThMSmplZEhuNmgyVzJoT3J1N3B2NUM5VEUwUVdZMGlCMjBYdloyV1JN?=
 =?utf-8?B?NkVMOGdMcnNsWEl4YnRTQ2ZJakt6RUZmZFZFY2RlcUNJNFlEUkFTNXVuMkRD?=
 =?utf-8?B?ZXZBaldQaEh2ZUpqTzB5cWN6YTJDR3liS09FUDhNUWdUWkxmWmRqZmZ3Y25w?=
 =?utf-8?B?Z3Z6dU56Z29Cd0wzZ1F0MFZCdkxlRjAyY3ZmTlVoR0xqMGlQUC9RTzdwQTNv?=
 =?utf-8?B?MStuZ201cG9qcVlEOCtBeStRM0lUZmJoMXAvNnpiMm9RU1RQNFpidnQ0Tk5T?=
 =?utf-8?B?V3Y1SDFRd21QdktOZGZGVFMvRU5WRmh1UGdJTkE2STFpSzVKOVpnRStqMDdz?=
 =?utf-8?B?OFJscm9OTHNvY2ZCaTJRWWxHU29GUlZhQ0h4R2FBMEVnVUtGVjVtbG9pbG1r?=
 =?utf-8?B?VUJBTlA3MERmZHlWVjlaWFVQemN5bElWV2JHQnpxVmlDaDJTZzNzSkc2SnRm?=
 =?utf-8?B?ZytROTU2WExCUjhwZlcwTDc5VE5TOExNT3VROEFnWFNyUlIxSTc4a1QyMEZ0?=
 =?utf-8?B?anFwdzF3MlFxcVczeTZPcVNxRTBCS1NnNjFQWko3OSt3Um5ENkFiTzhvcnVJ?=
 =?utf-8?B?Y2FYTS81Y2Y5U1hDc0hnaWxpelJHRlhqWWdYRko3Qll4OU9rWUFCTHRvUjBo?=
 =?utf-8?B?MmR0YUN6U3lWM3dxS29OcXNiYy9WQ2ZkeXJOeEtBOGdxdC9rQmVmOU9VNTI4?=
 =?utf-8?B?U2N6NlRRODVqL202S2U3cnVKTDl3c1IycHMyK1hNbmZGb1R2WXIvKzU1TTVX?=
 =?utf-8?B?NnlHckt0MVFza1lxWGxwckhiaUdnQTQrUzRQTmlUTE9VZjJEOFRoYkhReGdk?=
 =?utf-8?B?eUZEZmtVby9MOHpVK3ZocWZ3TmdsMi9ROENlZzhhKzNHSEpFd3MvbXdka2xX?=
 =?utf-8?B?QXFTWlNpM3JZMjJiV29ucnVHWVQ3RXNoQ1FSd09PK2ZnbHdwZ3I4VHhIakp2?=
 =?utf-8?B?Q3FHZlM4bHlLNEVXVVpNQVBBZ0x6czcxY1RDdjJnZjR2aVZuMEFwS0dqTU5I?=
 =?utf-8?B?d3FXMndmNDM5Y3Y5UzFNaWNuZ3pWaXRMVFc5eW5xRUcxamphTklHMEZRbzJO?=
 =?utf-8?B?ZmhYOS9TUWhKZEl5Szh5YzFVaXlvSVlySHZQMnAvTTIrNW9NRk9WV2hwR3VR?=
 =?utf-8?B?YnQvTHZ6TEFkWHNvbUhrUWRBdGo0NGZEK1UwYjFmUmg5WWhySkJtUzVnbGFm?=
 =?utf-8?B?dHJ3TE9YREZmZi9xazB0UHRrN3RVSlFLdEU2NllKS3Vta3puWWlzUkJhR2RS?=
 =?utf-8?B?S1RrREc4S01nSjhGMFZaWFdlK29ablIrUHkwbEV1SWMvZHpTRVBtcWZ6SjZN?=
 =?utf-8?B?aU1qN2w1M1BxMENBMFdBSEZVZTFyeS9JOEQ0aGxMUjNqZVBmaERObXBwUk1V?=
 =?utf-8?B?UEE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ca709f80-5fc8-4707-542d-08dadfc9f526
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Dec 2022 00:59:32.3665
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ezAS97acman0cSKLsAMb1x+QUNZc6RShPmOw0KMRyVyQmeAyDd/XULWcICuiGsZ1eJ7PzjP3gaum72dq16K5FMPxc07jmMQlJLrk0ESLflU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR11MB7071
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Peter,

On 12/14/2022 8:08 AM, Peter Newman wrote:
> When creating a new monitoring group, the RMID allocated for it may have
> been used by a group which was previously removed. In this case, the
> hardware counters will have non-zero values which should be deducted
> from what is reported in the new group's counts.
> 
> resctrl_arch_reset_rmid() initializes the prev_msr value for counters to
> 0, causing the initial count to be charged to the new group. Resurrect
> __rmid_read() and use it to initialize prev_msr correctly.
> 
> Unlike before, __rmid_read() checks for error bits in the MSR read so
> that callers don't need to.
> 
> Fixes: 1d81d15db39c ("x86/resctrl: Move mbm_overflow_count() into resctrl_arch_rmid_read()")
> Signed-off-by: Peter Newman <peternewman@google.com>

This does look like a candidate for stable?

> ---

It is helpful to have a summary here of what changed since previous version.

>  arch/x86/kernel/cpu/resctrl/monitor.c | 49 ++++++++++++++++++---------
>  1 file changed, 33 insertions(+), 16 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
> index efe0c30d3a12..77538abeb72a 100644
> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
> @@ -146,6 +146,30 @@ static inline struct rmid_entry *__rmid_entry(u32 rmid)
>  	return entry;
>  }
>  
> +static int __rmid_read(u32 rmid, enum resctrl_event_id eventid, u64 *val)
> +{
> +	u64 msr_val;
> +
> +	/*
> +	 * As per the SDM, when IA32_QM_EVTSEL.EvtID (bits 7:0) is configured
> +	 * with a valid event code for supported resource type and the bits
> +	 * IA32_QM_EVTSEL.RMID (bits 41:32) are configured with valid RMID,
> +	 * IA32_QM_CTR.data (bits 61:0) reports the monitored data.
> +	 * IA32_QM_CTR.Error (bit 63) and IA32_QM_CTR.Unavailable (bit 62)
> +	 * are error bits.
> +	 */
> +	wrmsr(MSR_IA32_QM_EVTSEL, eventid, rmid);
> +	rdmsrl(MSR_IA32_QM_CTR, msr_val);
> +
> +	if (msr_val & RMID_VAL_ERROR)
> +		return -EIO;
> +	if (msr_val & RMID_VAL_UNAVAIL)
> +		return -EINVAL;
> +
> +	*val = msr_val;
> +	return 0;
> +}
> +
>  static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_domain *hw_dom,
>  						 u32 rmid,
>  						 enum resctrl_event_id eventid)
> @@ -172,8 +196,12 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_domain *d,
>  	struct arch_mbm_state *am;
>  
>  	am = get_arch_mbm_state(hw_dom, rmid, eventid);
> -	if (am)
> +	if (am) {
>  		memset(am, 0, sizeof(*am));
> +
> +		/* Record any initial, non-zero count value. */
> +		__rmid_read(rmid, eventid, &am->prev_msr);
> +	}
>  }
>  
>  static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width)
> @@ -191,25 +219,14 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain *d,
>  	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
>  	struct arch_mbm_state *am;
>  	u64 msr_val, chunks;
> +	int ret;
>  
>  	if (!cpumask_test_cpu(smp_processor_id(), &d->cpu_mask))
>  		return -EINVAL;
>  
> -	/*
> -	 * As per the SDM, when IA32_QM_EVTSEL.EvtID (bits 7:0) is configured
> -	 * with a valid event code for supported resource type and the bits
> -	 * IA32_QM_EVTSEL.RMID (bits 41:32) are configured with valid RMID,
> -	 * IA32_QM_CTR.data (bits 61:0) reports the monitored data.
> -	 * IA32_QM_CTR.Error (bit 63) and IA32_QM_CTR.Unavailable (bit 62)
> -	 * are error bits.
> -	 */
> -	wrmsr(MSR_IA32_QM_EVTSEL, eventid, rmid);
> -	rdmsrl(MSR_IA32_QM_CTR, msr_val);
> -
> -	if (msr_val & RMID_VAL_ERROR)
> -		return -EIO;
> -	if (msr_val & RMID_VAL_UNAVAIL)
> -		return -EINVAL;
> +	ret = __rmid_read(rmid, eventid, &msr_val);
> +	if (ret)
> +		return ret;
>  
>  	am = get_arch_mbm_state(hw_dom, rmid, eventid);
>  	if (am) {
> 
> base-commit: 830b3c68c1fb1e9176028d02ef86f3cf76aa2476


Thank you very much for catching and fixing this.

Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>

Reinette