Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <32cfa982-037c-6edf-702a-c59d9c5d2f57@intel.com>
Date:   Fri, 16 Sep 2022 09:17:41 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.2.2
Subject: Re: [PATCH v4 11/13] x86/resctrl: Add sysfs interface to write the
 event configuration
Content-Language: en-US
To:     Babu Moger <babu.moger@amd.com>, <corbet@lwn.net>,
        <tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>
CC:     <fenghua.yu@intel.com>, <dave.hansen@linux.intel.com>,
        <x86@kernel.org>, <hpa@zytor.com>, <paulmck@kernel.org>,
        <akpm@linux-foundation.org>, <quic_neeraju@quicinc.com>,
        <rdunlap@infradead.org>, <damien.lemoal@opensource.wdc.com>,
        <songmuchun@bytedance.com>, <peterz@infradead.org>,
        <jpoimboe@kernel.org>, <pbonzini@redhat.com>,
        <chang.seok.bae@intel.com>, <pawan.kumar.gupta@linux.intel.com>,
        <jmattson@google.com>, <daniel.sneddon@linux.intel.com>,
        <sandipan.das@amd.com>, <tony.luck@intel.com>,
        <james.morse@arm.com>, <linux-doc@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <bagasdotme@gmail.com>,
        <eranian@google.com>
References: <166257348081.1043018.11227924488792315932.stgit@bmoger-ubuntu>
 <166257367603.1043018.15774268739007323339.stgit@bmoger-ubuntu>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <166257367603.1043018.15774268739007323339.stgit@bmoger-ubuntu>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QzExTHY3Q00ra1FjSXhoYXlmNDhkWXIyZVI5bThhREdCVjhob0ZacUl4NzNt?=
 =?utf-8?B?Q2hUQlR4UkxjS3F1STljcHNzRUwrMXM2dlNHMlpSUkxiQXo1eVBiQmdOTjd1?=
 =?utf-8?B?VlpXWTBFZjRMSUwxSVQ5ZFpJMjVhZk5BbUF0clpUSDhyZkxBSFhkSUNOSkZZ?=
 =?utf-8?B?VlcrNnpsb0dNb0Mva1VJWEZqOTVoUUVTUXpHK2Y1eFM2OHJlWWlrdnpjcE8v?=
 =?utf-8?B?Q2k1YzRaUUt4ZzljTjVHOVVoaUdkMzFnV1F1Z09QTklwYzhOREF3Wk5iRGtr?=
 =?utf-8?B?SjdIT0x6VHlvM1J3UHhMK25TNGRva2Q2VWhCZHcybzdZYVdka3FnRHNrRG1U?=
 =?utf-8?B?TlR6ZFR1bERQVUx3M1VGNEtsM1cxSTA0MDNHVDdHOTRNMDdKa3JsQVpGYi9D?=
 =?utf-8?B?NVBkc2lqTWgrVm8yekdTbFNYZTZjbWpWdGtsY0NtLzQzclhtb2d3OTBQRUY1?=
 =?utf-8?B?eDNyZG13TnFkZnJiYnY3cTB2UUhRV0d2aGhtYlhZWWhadlVmZWk3SURNbjF2?=
 =?utf-8?B?NnFCN0p1dlhza3h3b2NlUUVWSXo5UE9jZXZBT05hbnd3Zkloejh6d1hwYk5u?=
 =?utf-8?B?TTkzMVpQc0lCVGE3Tksxc3lDTXZaNTFBSDdtSVRSSUdNK2xnYURVSHdRb01L?=
 =?utf-8?B?L1hKcVF3UDR2bWsvL0w0ZVd5ay9kaGd1N3VyYjRBandtTStFVnhvSVExdFRM?=
 =?utf-8?B?WHZZK3JxdEdPWGFTUDdibldVWWEzSi81MXRMZmVMSjNrMnRaaU9rRXlqa1d4?=
 =?utf-8?B?L0luSVRmOUNyRnNsbEx2aVJId2dFbWoxcmJEY1ZJelVTZGtuZy85VnFuMUF1?=
 =?utf-8?B?YytpdGRHR0RhK1ZzSFcwejhZeis2WmdUdm8wRERkOVhLS0FRU05JTkRwUjNl?=
 =?utf-8?B?eWhTdzlUU2FDeXpVdFRmeGowNStYOFNHL3F5WTYwbE5qUExsWEd0LzVJWDNz?=
 =?utf-8?B?dHBhd3JoSGNQK015U2RRTWhCUCs4RUdCb0l4OWNoL0l6aEF0WDZuM2xid3RI?=
 =?utf-8?B?SVNPbWRDdHNzT2taelB6ajc5cXFrYUFFb09GanA4cnR0dUNHbzZqdHlBMStN?=
 =?utf-8?B?bGxLN1R1S1BZSGFmUzM2VFVZSmN2YTBzNS9SdzE5eERZOHlSZVlaOXNPMmNi?=
 =?utf-8?B?TXdLMDNOSHA2SFJiQUlRR2pMSElSTG5BemxQeU5MaG5rK3dBbmJOd3R4MTA2?=
 =?utf-8?B?UDF3UVRZL0M1d1JPSElRa3hyUXdhaW15UHpySXFucVltQ2x4dU12bnE1cTJ1?=
 =?utf-8?B?NWdiVURQNHJnS1Q4YXFQNWFiMVBUMlZoR1p0c1RXSWNVUXB0dStrQ0VlUTFa?=
 =?utf-8?B?MVZIb0xVUGVqNU44eC9UNjJ4MUd0NU83QTdWcFd6YTVPaEpIUjVvMTlRQUpl?=
 =?utf-8?B?ZXprQ2hXb2tadVc5WitiV1FyTEIyNlpaY2FOb3V5UnRuSnBlVTROZTgxVncw?=
 =?utf-8?B?T1loUEZzUUg2WWFVY0xaQ2tnYjBtMEt6TVQ2QWo2aXl3d1NjdU5kVWlxWXdw?=
 =?utf-8?B?QlNJS3creVByOEN3UTdLRVB0eGp1U05GNDV0eEZGZHg0WEVvL1dtMVRxaVVl?=
 =?utf-8?B?VEJJbVNxVlB5ZFFQNEtJNDg5YW1vbnFQa0NxaFVFWmJDOFBsanFweGdLTm1n?=
 =?utf-8?B?dTFOZW16YWZITWVESEJMdVdvcWtYL1NORjBMWGZTNjNXbnVDRHBXZmFUTWV6?=
 =?utf-8?B?SGRXMVgvNml1VlRiRXVzQjB1Nkl0dmR2eXRSNjBKSHFJYnZMRHlGT2VBNW02?=
 =?utf-8?B?ZU4yRTF5NHYrNlR0ZGJGYWhOc0ZqYW9qVW9PUEJQeENIMUhlbnppTEJydGUr?=
 =?utf-8?B?MWZFUHk5ZG0yalUrSlVnVk5ITkFMcDZqK25nVjl1QVFWT1E4REZLbk8ySWpr?=
 =?utf-8?B?SkRxREVTWkU2V1J3c0dhSlhScDNocW9RYzVjRWVSMjhNc3pZbXprQU1hWVVn?=
 =?utf-8?B?eFFJR3krRUt4Tm93d0hTWVRZNlVqdlhiVWtXTE1IYmp2em9seGlXdG5nemZH?=
 =?utf-8?B?TUczeEZhZCszOElaZHlJNzVvUFBBSHUwL01ldUMvbzBwcFRFK0lUY1FLM2Zk?=
 =?utf-8?B?MTlXcEFVQ05SUk9Jc2d0d0dtRnRpTUJBVlA2ZCtVMXBHalRGdFZna3hBM0xZ?=
 =?utf-8?B?V2RCcFdqa2phVk5kK0NYZ3hFNlRlQWRaSWZMeHVhcHhtZ05SYmJzTWtCaWE1?=
 =?utf-8?B?WlE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ac19df6d-b09e-48da-2522-08da97fefd0f
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Sep 2022 16:17:45.2586
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pBLkhEDEvowdk7dPWt2voD2mPaSg5zPDqwLBHuRIxcxNVx7DIcna7rrtNoyRvuWSGjbH8f7QCkX2Kc2Ejw5+mIxaXG+EgFPKZDLUXkV5Ebc=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB6325
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-6.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,
        SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Babu,

On 9/7/2022 11:01 AM, Babu Moger wrote:

...

> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index 6f067c1ac7c1..59b484eb1267 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> @@ -330,9 +330,121 @@ int rdtgroup_mondata_config_show(struct seq_file *m, void *arg)
>  	return ret;
>  }
>  
> +/*
> + * This is called via IPI to read the CQM/MBM counters
> + * in a domain.

copy&paste from previous patch?

> + */
> +void mon_event_config_write(void *info)
> +{
> +	struct mon_config_info *mon_info = info;
> +	u32 msr_index;
> +
> +	switch (mon_info->evtid) {
> +	case QOS_L3_MBM_TOTAL_EVENT_ID:
> +		msr_index = 0;
> +		break;
> +	case QOS_L3_MBM_LOCAL_EVENT_ID:
> +		msr_index = 1;
> +		break;
> +	default:
> +		/* Not expected to come here */
> +		return;
> +	}
> +
> +	wrmsr(MSR_IA32_EVT_CFG_BASE + msr_index, mon_info->mon_config, 0);
> +}
> +
> +ssize_t  rdtgroup_mondata_config_write(struct kernfs_open_file *of,
> +				       char *buf, size_t nbytes, loff_t off)
> +{
> +	struct mon_config_info mon_info;
> +	struct rdt_hw_resource *hw_res;
> +	struct rdtgroup *rdtgrp;
> +	struct rdt_resource *r;
> +	unsigned int mon_config;
> +	cpumask_var_t cpu_mask;
> +	union mon_data_bits md;
> +	struct rdt_domain *d;
> +	u32 resid, domid;
> +	int ret = 0, cpu;
> +
> +	ret = kstrtouint(buf, 0, &mon_config);
> +	if (ret)
> +		return ret;
> +
> +	rdt_last_cmd_clear();
> +
> +	/* mon_config cannot be more than the supported set of events */
> +	if (mon_config > MAX_EVT_CONFIG_BITS) {
> +		rdt_last_cmd_puts("Invalid event configuration\n");
> +		return -EINVAL;
> +	}
> +
> +	cpus_read_lock();
> +	rdtgrp = rdtgroup_kn_lock_live(of->kn);
> +	if (!rdtgrp) {
> +		return -ENOENT;
> +		goto e_unlock;
> +	}
> +
> +	if (!zalloc_cpumask_var(&cpu_mask, GFP_KERNEL)) {
> +		ret = -ENOMEM;
> +		goto e_unlock;
> +	}
> +
> +	md.priv = of->kn->priv;
> +	resid = md.u.rid;
> +	domid = md.u.domid;
> +
> +	hw_res = &rdt_resources_all[resid];
> +	r = &hw_res->r_resctrl;
> +	d = rdt_find_domain(r, domid, NULL);
> +	if (IS_ERR_OR_NULL(d)) {
> +		ret = -ENOENT;
> +		goto e_cpumask;
> +	}
> +
> +	/*
> +	 * Read the current config value first. If both are same
> +	 * then we dont need to write again
> +	 */
> +	mon_info.evtid = md.u.evtid;
> +	mondata_config_read(d, &mon_info);
> +	if (mon_info.mon_config == mon_config)
> +		goto e_cpumask;
> +
> +	mon_info.mon_config = mon_config;
> +
> +	/* Pick all the CPUs in the domain instance */
> +	for_each_cpu(cpu, &d->cpu_mask)
> +		cpumask_set_cpu(cpu, cpu_mask);
> +
> +	/* Update MSR_IA32_EVT_CFG_BASE MSR on all the CPUs in cpu_mask */
> +	on_each_cpu_mask(cpu_mask, mon_event_config_write, &mon_info, 1);

If this is required then could you please add a comment why every CPU in
the domain needs to be updated? Until now configuration changes
only needed to be made on one CPU per domain. 
Even in the previous patch when the user reads the current configuration
value it is only done on one CPU in the domain ... to me that implies
that the scope is per domain and only one CPU in the domain needs to be
changed.

> +
> +	/*
> +	 * When an Event Configuration is changed, the bandwidth counters
> +	 * for all RMIDs and Events will be cleared, and the U-bit for every
> +	 * RMID will be set on the next read to any BwEvent for every RMID.
> +	 * Clear the mbm_local and mbm_total counts for all the RMIDs.
> +	 */

This is a snippet that was copied from the hardware spec and since it is
inserted into the kernel driver the context makes it hard to understand.
Could it be translated into what it means in this context?
Perhaps something like:

	/*
	 * When an Event Configuration is changed, the bandwidth counters
	 * for all RMIDs and Events will be cleared by the hardware. The
         * hardware also sets MSR_IA32_QM_CTR.Unavailable (bit 62) for every
	 * RMID on the next read to any event for every RMID. Subsequent
	 * reads will have MSR_IA32_QM_CTR.Unavailable (bit 62) cleared
	 * while it is tracked by the hardware.
	 * Clear the mbm_local and mbm_total counts for all the RMIDs.
	 */

Please fixup where I got it wrong.

> +	memset(d->mbm_local, 0, sizeof(struct mbm_state) * r->num_rmid);
> +	memset(d->mbm_total, 0, sizeof(struct mbm_state) * r->num_rmid);
> +

Reinette