Received-SPF: pass (google.com: domain of linux-kernel+bounces-181553-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <95b0ee31-8746-474f-a15b-91c893fdce28@amd.com>
Date: Thu, 16 May 2024 15:03:33 -0500
User-Agent: Mozilla Thunderbird
Reply-To: babu.moger@amd.com
Subject: Re: [RFC PATCH v3 07/17] x86/resctrl: Add support to enable/disable
 ABMC feature
To: Peter Newman <peternewman@google.com>
Cc: corbet@lwn.net, fenghua.yu@intel.com, reinette.chatre@intel.com,
 tglx@linutronix.de, mingo@redhat.com, bp@alien8.de,
 dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com,
 paulmck@kernel.org, rdunlap@infradead.org, tj@kernel.org,
 peterz@infradead.org, yanjiewtw@gmail.com, kim.phillips@amd.com,
 lukas.bulwahn@gmail.com, seanjc@google.com, jmattson@google.com,
 leitao@debian.org, jpoimboe@kernel.org, rick.p.edgecombe@intel.com,
 kirill.shutemov@linux.intel.com, jithu.joseph@intel.com,
 kai.huang@intel.com, kan.liang@linux.intel.com,
 daniel.sneddon@linux.intel.com, pbonzini@redhat.com, sandipan.das@amd.com,
 ilpo.jarvinen@linux.intel.com, maciej.wieczor-retman@intel.com,
 linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, eranian@google.com,
 james.morse@arm.com
References: <cover.1711674410.git.babu.moger@amd.com>
 <0db75c94886da62b8da498ef159d8fe27b0b3811.1711674410.git.babu.moger@amd.com>
 <CALPaoCgRtYLnzpkL6uVjb+LvBbxD8ANRuzACP1UQ=rkzk8TCqg@mail.gmail.com>
Content-Language: en-US
From: "Moger, Babu" <babu.moger@amd.com>
In-Reply-To: <CALPaoCgRtYLnzpkL6uVjb+LvBbxD8ANRuzACP1UQ=rkzk8TCqg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?SjdYYnVjSlNkcXVwU3FPaWQ0SnpucW9jd241cE9yTkQ4d3czL2dsTy95blhn?=
 =?utf-8?B?Y3MxQ3prN3NuNjFMTU9BZDltU29WWk1wUW9iWU40ZlNYQktqTHpFcGtyRkpn?=
 =?utf-8?B?QTgwUU9QdUhKWkp4a2IvMFJlS1ZQdDlWalJYSkJoYnM5ZHVxa1BEN0N0UExm?=
 =?utf-8?B?VUE2bUYyOThGSXNTb0FFenpES2lzVG9zN3N2aXE4SGljSjdUbkFIeURyT1Bx?=
 =?utf-8?B?UmRXRDM3dnhub3ZvK3VjU1N2RTRnbXMweVdLYXVHQzFOeDFJNk5aRGg1d09q?=
 =?utf-8?B?aTYzbE5yTGNkZGRyWkVUdUsybXl4OWhYa2hDOS9ya08yTW9TOGJ6dzVRdUVI?=
 =?utf-8?B?NXFiaVdKclFrZWJwc3BocEpwZHVuaThBbWRIN29SSTJXNEhtQ2Z4Tmo2RnU4?=
 =?utf-8?B?Kzk0ZUhHeS9JeWpFeTZBY3h1TVRVYy9BM091OTFXVElOMXp1TFAvV3lJTmlm?=
 =?utf-8?B?V1V5NnU3ZnVITC9FaXBYbkhsaWU5MmFIZ2piV1Rva2MwOTdpakJDQjZ0d1Nv?=
 =?utf-8?B?VEUrYlpCd2NJUUwzMlRVenFGeUFQY0FuT3RXWVUzVkQ1Znk4eUhlbVZpeXZN?=
 =?utf-8?B?K0ZOWmlXeHgzdTFHWGhON3MwNXMzMU1mUDNkL2ZVTUhVSmVGczRJSnI3ZThy?=
 =?utf-8?B?SHRIVkFnQ1FyRWlGem0vcXVMbmJpdFg3T2htUWR4OFFOZWR6R2ZiOVV4eDdT?=
 =?utf-8?B?UmZRNXJsb2MxNkQ0clh0M0RBb01veDBXR2NaWnJkZjk3b1B6Q2RnUHlRSllh?=
 =?utf-8?B?cEJNMU02bVZxRk4zb29tU2RDRVczQlU4UmR6V2twVlltZnJWWVVMT3VZK3RC?=
 =?utf-8?B?SVlyc3RPUkZrcnlkRmN4ajlQMmtlMjYydGVXUUFVU1dORnFSaTd5QlFWNE9a?=
 =?utf-8?B?d2J5alFVZ0FwYlRYVmZ5RlFQZkR3S0JvT1VPN3lnZ1JFcVI5MkMwNHFacHM0?=
 =?utf-8?B?NFpyWk5ESEdoa3U0K1dlTytwdkVEWC92cUROMG92NlVHa2hVZ1RmWWZQQmZu?=
 =?utf-8?B?ZkxIQktnR25QZ0ZOeWxSWmkyTkQyMHFZSU55cms1aXZaL3grSzZkSWZTNEw0?=
 =?utf-8?B?MnBiS0pVeVRMckgwa3B1b29TdldtYUw0VjJ3Z0JMbU1KTnNtL2R1Rm1yU3Aw?=
 =?utf-8?B?UHc0VnJRSkpoeVRtUDFWUDJKMm8zTWNPNUxkYmxleHZTWmJHM1k3ak5wYmFR?=
 =?utf-8?B?eTliVU1jZFdkdVJFaUV6QXVWS2VSOEQ3bUhaR0dlK3lhR2lhZ3c1R0tLZGow?=
 =?utf-8?B?RUlKUjlFVmZpQUpjMTdyVjJ3ZU5pajFzOGViMEI0ZkFGZXdhVW9EWkovaTZ2?=
 =?utf-8?B?UElqUDJ4RkRGNTk1YWtwbmJlWi84dThCWm55Y2JMU3pOa3RTamZvL0pIdE9S?=
 =?utf-8?B?dmN6KzNzQk5GSnFtNWtQSFVYWkVQRk91K2FyeTRNQ215c1lmc3Bud1MreVBL?=
 =?utf-8?B?bmVoS1VCYVYyNWNSVDNZejVJVkV4emV4VjM2aFRSQjBXZmtwRjNneEc5WDV4?=
 =?utf-8?B?QksySjhxOVpNNmdFMDRhWktqQ2JvUjdDVzRKUHdqVWM5dnZzR0xITExZQk9R?=
 =?utf-8?B?d3VUMENrSGNnSmJGSjRFR3MzeXl3REF0Y3VFRzNzQUgvYWFXUTRUMnJZeS9V?=
 =?utf-8?B?SEdwWktFWUZxRlZJcVBVTjZoT3AxcFN3ZnVCSzNsczJnV3JkWi9rTS83bkFy?=
 =?utf-8?B?ZDBKaFdVV2laU29DMjBveFFmWUZ6bWVYbnVzV3hhTGc4U0kxN1lnMkJjdFpY?=
 =?utf-8?B?dUFOa1ZyZEltSk5PQkZ1eTRYNEg3RFYzamNvSG1WMjI2dFEvNG52MzB4aDRz?=
 =?utf-8?B?Vy9GdWVLM2tPRThZaWN4aUxKMDJoUGpVOUd3S1pYTlZXL0Q5OWJlWjBqdy9a?=
 =?utf-8?B?Tmp1OXNENWE5ME1TUUl0N1BXQ29Zb0tmZTdCUUNFL0V5T2ZQVXgybkRsNjc4?=
 =?utf-8?B?WkRQbW1VOU1XekhtREVtdE10M0dDTDFSOUpXdlYyYkZSSSs1VXFOek1ENFdq?=
 =?utf-8?B?eGR6N2FEd09TV3NsZEFLV2RoK1laZXk5OFJ5TS84NmNGR0ZqUUhtY01OWEMv?=
 =?utf-8?B?ZUx2dkJFaU1icnRKRkpOMFZVV01BakJYMWJlTGxaRThKeHFSWElyZnVSUGhi?=
 =?utf-8?Q?/Qfo=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 73ae8c33-3713-433e-9806-08dc75e345da
X-MS-Exchange-CrossTenant-AuthSource: MW3PR12MB4553.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2024 20:03:37.1014
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: M9C8DliQ57p4ppWmyQICUVQSMufKAB5gbv3RH5Hwi7Lf2NO6z9QlTXqc5zBs84JS
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8339

Hi Peter,

While working on v4, found few things. Just wanted you to know (mostly FYI.).

On 4/3/24 19:30, Peter Newman wrote:
> Hi Babu,
> 
> On Thu, Mar 28, 2024 at 6:07 PM Babu Moger <babu.moger@amd.com> wrote:
>>  struct rdt_fs_context {
>>         struct kernfs_fs_context        kfc;
>>         bool                            enable_cdpl2;
>> @@ -433,6 +436,7 @@ struct rdt_parse_data {
>>   * @mbm_cfg_mask:      Bandwidth sources that can be tracked when Bandwidth
>>   *                     Monitoring Event Configuration (BMEC) is supported.
>>   * @cdp_enabled:       CDP state of this resource
>> + * @abmc_enabled:      ABMC feature is enabled
>>   *
>>   * Members of this structure are either private to the architecture
>>   * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g.
>> @@ -448,6 +452,7 @@ struct rdt_hw_resource {
>>         unsigned int            mbm_width;
>>         unsigned int            mbm_cfg_mask;
>>         bool                    cdp_enabled;
>> +       bool                    abmc_enabled;
>>  };
>>
>>  static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r)
>> @@ -491,6 +496,13 @@ static inline bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l)
>>
>>  int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable);
>>
>> +static inline bool resctrl_arch_get_abmc_enabled(enum resctrl_res_level l)
>> +{
>> +       return rdt_resources_all[l].abmc_enabled;
>> +}
> 
> This inline definition will not work in either this file or
> fs/resctrl/internal.h, following James's change[1] moving the code.
> 
> resctrl_arch-definitions are either declared in linux/resctrl.h or
> defined inline in a file like asm/resctrl.h.

Yes, These definitions need to moved to asm/resctrl.h.
Moving that will involve moving few data structure as well.
It is better it is done during fs and arch restructure.
> 
> 
>> +
>> +int resctrl_arch_set_abmc_enabled(enum resctrl_res_level l, bool enable);
>> +
>>  /*
>>   * To return the common struct rdt_resource, which is contained in struct
>>   * rdt_hw_resource, walk the resctrl member of struct rdt_hw_resource.
>> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> index 05f551bc316e..f49073c86884 100644
>> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> @@ -850,9 +850,15 @@ static int rdtgroup_mbm_assign_show(struct kernfs_open_file *of,
>>                                     struct seq_file *s, void *v)
>>  {
>>         struct rdt_resource *r = of->kn->parent->priv;
>> +       struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
>>
>> -       if (r->mbm_assign_capable)
>> +       if (r->mbm_assign_capable && hw_res->abmc_enabled) {
>> +               seq_puts(s, "[abmc]\n");
>> +               seq_puts(s, "legacy_mbm\n");
>> +       } else if (r->mbm_assign_capable) {
>>                 seq_puts(s, "abmc\n");
>> +               seq_puts(s, "[legacy_mbm]\n");
>> +       }
> 
> This looks like it would move to fs/resctrl/rdtgroup.c where it's not
> possible to dereference an rdt_hw_resource struct.

There are two rdtgroup.c files.
0 arch/x86/kernel/cpu/resctrl/rdtgroup.c
1 fs/resctrl/rdtgroup.c

I think this should move to arch rdtgroup.c. It is better it is done
during fs and arch restructure.

> 
> It might be helpful to try building your changes on top of James's
> change[1] to get an idea of how this would fit in post-refactoring.
> I'll stop pointing out inconsistencies with his portability scheme
> now.
> 
>>
>>         return 0;
>>  }
>> @@ -2433,6 +2439,74 @@ int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable)
>>         return 0;
>>  }
>>
>> +static void resctrl_abmc_msrwrite(void *arg)
>> +{
>> +       bool *enable = arg;
>> +       u64 msrval;
>> +
>> +       rdmsrl(MSR_IA32_L3_QOS_EXT_CFG, msrval);
>> +
>> +       if (*enable)
>> +               msrval |= ABMC_ENABLE;
>> +       else
>> +               msrval &= ~ABMC_ENABLE;
>> +
>> +       wrmsrl(MSR_IA32_L3_QOS_EXT_CFG, msrval);
>> +}
>> +
>> +static int resctrl_abmc_setup(enum resctrl_res_level l, bool enable)
>> +{
>> +       struct rdt_resource *r = &rdt_resources_all[l].r_resctrl;
>> +       struct rdt_domain *d;
>> +
>> +       /* Update QOS_CFG MSR on all the CPUs in cpu_mask */
>> +       list_for_each_entry(d, &r->domains, list) {
>> +               on_each_cpu_mask(&d->cpu_mask, resctrl_abmc_msrwrite, &enable, 1);
>> +               resctrl_arch_reset_rmid_all(r, d);
>> +       }
>> +
>> +       return 0;
>> +}
>> +
>> +static int resctrl_abmc_enable(enum resctrl_res_level l)
>> +{
>> +       struct rdt_hw_resource *hw_res = &rdt_resources_all[l];
>> +       int ret = 0;
>> +
>> +       if (!hw_res->abmc_enabled) {
>> +               ret = resctrl_abmc_setup(l, true);
>> +               if (!ret)
>> +                       hw_res->abmc_enabled = true;
> 
> Presumably this would be called holding the rdtgroup_mutex? Perhaps a
> lockdep assertion somewhere would be appropriate?

Yes. I have taken care of this.

> 
> Thanks!
> -Peter
> 
> [1] https://lore.kernel.org/lkml/20240321165106.31602-32-james.morse@arm.com/
> 

-- 
Thanks
Babu Moger