Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <853b9cfd-504d-0dcb-2a2c-8b8df75a9b60@intel.com>
Date:   Wed, 16 Mar 2022 14:52:04 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.7.0
Subject: Re: [PATCH v3 15/21] x86/resctrl: Abstract __rmid_read()
Content-Language: en-US
To:     James Morse <james.morse@arm.com>, <x86@kernel.org>,
        <linux-kernel@vger.kernel.org>
CC:     Fenghua Yu <fenghua.yu@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        H Peter Anvin <hpa@zytor.com>,
        Babu Moger <Babu.Moger@amd.com>,
        <shameerali.kolothum.thodi@huawei.com>,
        Jamie Iles <jamie@nuviainc.com>,
        "D Scott Phillips OS" <scott@os.amperecomputing.com>,
        <lcherian@marvell.com>, <bobo.shaobowang@huawei.com>,
        <tan.shaopeng@fujitsu.com>
References: <20220217182110.7176-1-james.morse@arm.com>
 <20220217182110.7176-16-james.morse@arm.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20220217182110.7176-16-james.morse@arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Y3pQN1dBTHhCdEQxT3BrYlc0cGM5YTI3Y0RSTDZvVVgydEpraUV1ejNXK2FS?=
 =?utf-8?B?a2ZrdHFMblR5OFdSTktNSHo1N1N3WXFSSENaWnliOS9SMnEyNVNMSEJUZnl6?=
 =?utf-8?B?YWRqemFtUTlEc0JwS01ZaEJXanIxYk4vZzlmT1dxaWsyWGtKaDZ6OHJyV2RX?=
 =?utf-8?B?VTQ2U0xyOUFhWHpQSHJVbUlYWDBXWVhJODhqZlArdktsSXprSHBPSUYyN3VW?=
 =?utf-8?B?VDFqQUs2eHBGK3UxL3BmV0txZ29xMVFqMFNQUWVGVUZCdE9FdnNZWk00V2FM?=
 =?utf-8?B?S0ErWXgyaGpzWXZkMGZiZ09tYzJoYVFtWVN3eVJjRjIxbUxmMFJsQ3o1YVBv?=
 =?utf-8?B?d0JibjZRZVhBY0lMNHRHcFh4dE12OFZYdHoxM09Qb0RROTZFUGd4enMyQUpX?=
 =?utf-8?B?THIycStmM3U5L21CZ0J3VmErdGduS0FsUFd3d0JkcTR0MDNUVkdnZXZKTWpy?=
 =?utf-8?B?NlBRWmk4RVd2RlB5VUhIRlpyMzdwWm90WnB5dlVXdWZ2UzcwUTUyQVV4K3BE?=
 =?utf-8?B?MlQxR2lCeHU0N21XUmRMc1Yvd3VTVHFnMkdFV0NadGNrcTM3dVdyZzFYbzhL?=
 =?utf-8?B?c25VRC9hTms0Z0lEVVJVYzh6K3U3REFPSDhyRFpWT3BNcHMvN3hLZWpSREp5?=
 =?utf-8?B?VW85TjUvWFlsUDNjaVd6U3FLUUR1SjdwQk8wQjZqVEZ3R0FrOVZqU3dEU3FK?=
 =?utf-8?B?Sk5GcFA5YWRxRUxuNEhxM3VVcjBIYzFGS1dkSjQzdThySGVVSmwwbHFnRS9S?=
 =?utf-8?B?bUliTkFTdjh4c2ZmTXlMUGhxRTg1NCtWZEpPNHBHQk5XVUZ5NDFCS1hmR2pT?=
 =?utf-8?B?Y09RNFdzbU1JVVk0NC85ZDYrekozV2VQSUhLUm4zODdFbVM2WERtREh6bDhQ?=
 =?utf-8?B?dTZXSGFITUpaVkIzOXM4S2YvR3VkUXU0YURxT1dMZ3JYdUsrWnpTM2xYc3ZT?=
 =?utf-8?B?c1lSdlRUaTJhcU1CM2FrcEluUmJSWThKWjQxS3Fad3lPOTM4a2s3VG05bEts?=
 =?utf-8?B?cFRmUE5mU0x6WjAwMTJiaW1XWURQUGs2THpVSGd3WmxTTUJVeUcxSVJjSEdh?=
 =?utf-8?B?YVZZTzNCc0RyYjN6VWZJaWZld2o1YWhHYS9tREp6NEhzTTZSVlVEa0FYUGt1?=
 =?utf-8?B?MWp6d1l1NTl4d3h1c2dnUlBkazhjQ3VPUDMwS1NPT3FORmlSVjdLVTRJSmlx?=
 =?utf-8?B?M0I4TThEdEgvdGMyOTI3STd4QnhPejRsV1JBc2JkNGtyN1ZjT2I4RkFTaW90?=
 =?utf-8?B?Rkl6clE1L3JFRGp5OWVEVXVUdHByMXltOUdQWmptUnFjZTVSQ0dXdm5Rdm5l?=
 =?utf-8?B?WkwyOHI4azRBSklRZTE3R2o1MWp4ejF3RnB1Tk9RSnk1RTVkaDZjWDlnZXV0?=
 =?utf-8?B?cHoyVEl6bnVGZjlueDBEQXZEUytkbDgySW9yb0U4bmpKRTdvNktEQWI3SkVU?=
 =?utf-8?B?N1dPRkZYL3VtYmxDWXN2UGVSWWdlUGlKVmdIdGF5ZW1tTzNHdzNIamVnMXM2?=
 =?utf-8?B?MkIwNmtXOGZISk5FZ3U5amNMSGZwUC9oN3VpSDZFN2RwaFg2M0pzREs5U2E3?=
 =?utf-8?B?T2hmemNNOTIrczRQckEybncrTWFMQ1JnZHkxT2VSdjFtelAyTkp5QWFVeHI1?=
 =?utf-8?B?SytTUnlaT2x6SFFyWS9aSTZrS3EvMi9oYUs3WldjWWFIYlJSNlZSNVVtNXJx?=
 =?utf-8?B?UFJuZmdEUDQzdmIrdm0zZFBiQ2RYakppOXlISzhLTUpOOGVpS3lCS0RiRmJy?=
 =?utf-8?B?WndUeE9ZZXRKQkhGcTl4b3dQTjB1M21mcTNrTHM3dld0QS9UQkJxdkVIWGlq?=
 =?utf-8?B?SUYzODdrellZN0EyVHJ1MEliVmk1U0p4d3NPSzdtZXQxbDVtMWV4SW0rRVBx?=
 =?utf-8?B?MUg4dkEyd2ZEUEhSdlUxRjRlZWNKemtyUGY0UzBScHRhbU5ndXBpN3NtRk1o?=
 =?utf-8?B?MURKREMyWXc0dzk4eTBlWnpRWmthdWtkd09yS3NZOThDcUJqd2trdHlmVEJm?=
 =?utf-8?B?NEdzNGh5RDhnPT0=?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 6aabff12-c64e-4f1d-d6d7-08da079737be
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Mar 2022 21:52:08.5712
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: rAI87cgtGrgQXYUeD4CLBqogNe8fbKcYFbD0KqaspjSf+KvHlFjfRZ6HNK+0/THRBn5+FxR83BHu9icpKxoQ5ulXAP5t42xDlWlt68xwtaE=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR11MB3038
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-3.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi James,

On 2/17/2022 10:21 AM, James Morse wrote:
> __rmid_read() selects the specified eventid and returns the counter
> value from the MSR. The error handling is architecture specific, and
> handled by the callers, rdtgroup_mondata_show() and __mon_event_count().
> 
> Error handling should be handled by architecture specific code, as
> a different architecture may have different requirements. MPAM's
> counters can report that they are 'not ready', requiring a second
> read after a short delay. This should be hidden from resctrl.
> 
> Make __rmid_read() the architecture specific function for reading
> a counter. Rename it resctrl_arch_rmid_read() and move the error
> handling into it.
> 
> Signed-off-by: James Morse <james.morse@arm.com>

...

> @@ -167,9 +167,9 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_domain *d,
>  		memset(am, 0, sizeof(*am));
>  }
>  
> -static u64 __rmid_read(u32 rmid, enum resctrl_event_id eventid)
> +int resctrl_arch_rmid_read(u32 rmid, enum resctrl_event_id eventid, u64 *val)
>  {
> -	u64 val;
> +	u64 msr_val;
>  
>  	/*
>  	 * As per the SDM, when IA32_QM_EVTSEL.EvtID (bits 7:0) is configured
> @@ -180,14 +180,24 @@ static u64 __rmid_read(u32 rmid, enum resctrl_event_id eventid)
>  	 * are error bits.
>  	 */
>  	wrmsr(MSR_IA32_QM_EVTSEL, eventid, rmid);
> -	rdmsrl(MSR_IA32_QM_CTR, val);
> +	rdmsrl(MSR_IA32_QM_CTR, msr_val);
>  
> -	return val;
> +	if (msr_val & RMID_VAL_ERROR)
> +		return -EIO;
> +	if (msr_val & RMID_VAL_UNAVAIL)
> +		return -EINVAL;
> +
> +	*val = msr_val;
> +
> +	return 0;
>  }

From above we see that resctrl_arch_rmid_read() returns an int that could be
-EIO or -EINVAL ...

...

> @@ -319,15 +331,15 @@ static u64 __mon_event_count(u32 rmid, struct rmid_read *rr)
>  {
>  	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(rr->r);
>  	struct mbm_state *m;
> -	u64 chunks, tval;
> +	u64 chunks, tval = 0;
>  
>  	if (rr->first)
>  		resctrl_arch_reset_rmid(rr->r, rr->d, rmid, rr->evtid);
>  
> -	tval = __rmid_read(rmid, rr->evtid);
> -	if (tval & (RMID_VAL_ERROR | RMID_VAL_UNAVAIL)) {
> -		return tval;
> -	}
> +	rr->err = resctrl_arch_rmid_read(rmid, rr->evtid, &tval);
> +	if (rr->err)
> +		return rr->err;
> +

Setting rr->err, an int, to the return of resctrl_arch_rmid_read() is ok and
can handle the negative error codes, but returning it here means that
__mon_event_count()'s return type should be changed,
it is currently u64.

>  	switch (rr->evtid) {
>  	case QOS_L3_OCCUP_EVENT_ID:
>  		rr->val += tval;
> @@ -343,7 +355,7 @@ static u64 __mon_event_count(u32 rmid, struct rmid_read *rr)
>  		 * Code would never reach here because an invalid
>  		 * event id would fail the __rmid_read.
>  		 */
> -		return RMID_VAL_ERROR;
> +		return -EINVAL;

Same here ... __mon_event_count() is u64 still ...

>  	}
>  
>  	if (rr->first) {
> @@ -419,9 +431,14 @@ void mon_event_count(void *info)
>  		}
>  	}
>  

Also take care here ... ret_val in mon_event_count() is still u64 while
__mon_event_count() attempts to return negative errors.

> -	/* Report error if none of rmid_reads are successful */
> -	if (ret_val)
> -		rr->val = ret_val;
> +	/*
> +	 * __mon_event_count() calls for newly created monitor groups may
> +	 * report -EINVAL/Unavailable if the monitor hasn't seen any traffic.
> +	 * If the first call for the control group succeed, discard any error
> +	 * set by reads of monitor groups.
> +	 */

Additionally, if the first call fails, but a following read of monitor group
succeeds then the first call's error is discarded.

How about if the last sentence is replaced with:
"Discard error if any of the monitor event reads succeeded."

> +	if (ret_val == 0)
> +		rr->err = 0;
>  }
>  
>  /*
> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
> index 0b48239f5920..70112dbfa128 100644
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -219,6 +219,7 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_domain *d,
>  			    u32 closid, enum resctrl_conf_type type);
>  int resctrl_online_domain(struct rdt_resource *r, struct rdt_domain *d);
>  void resctrl_offline_domain(struct rdt_resource *r, struct rdt_domain *d);
> +int resctrl_arch_rmid_read(u32 rmid, enum resctrl_event_id eventid, u64 *res);
>  
>  /**
>   * resctrl_arch_reset_rmid() - Reset any private state associated with rmid


Reinette