Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH v2 17/23] x86/resctrl: Abstract __rmid_read()
To:     Babu Moger <babu.moger@amd.com>, James Morse <james.morse@arm.com>,
        <x86@kernel.org>, <linux-kernel@vger.kernel.org>
CC:     Fenghua Yu <fenghua.yu@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        H Peter Anvin <hpa@zytor.com>,
        <shameerali.kolothum.thodi@huawei.com>,
        Jamie Iles <jamie@nuviainc.com>,
        D Scott Phillips OS <scott@os.amperecomputing.com>,
        <lcherian@marvell.com>, <bobo.shaobowang@huawei.com>,
        <tan.shaopeng@fujitsu.com>
References: <20211001160302.31189-1-james.morse@arm.com>
 <20211001160302.31189-18-james.morse@arm.com>
 <887f8946-6d2b-27bf-a49b-f83af05cbc68@amd.com>
 <dfe2f33c-6103-9699-42f9-73983fa62057@intel.com>
 <81826a4f-c3de-787f-8059-4808815b4800@amd.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
Message-ID: <71efc804-4ca0-f31b-4485-005b3ad6e4d4@intel.com>
Date:   Wed, 20 Oct 2021 13:28:50 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Firefox/78.0 Thunderbird/78.14.0
In-Reply-To: <81826a4f-c3de-787f-8059-4808815b4800@amd.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QXYwSi9kaTJJelMwc2Q3K3o0ZUdFYjg3bWRBZjlpNWNUd0R1UDNFbm9xcloy?=
 =?utf-8?B?V2NSNlgzRUptU2ZmYzhnclF6dDNUWHNjeVlmTHVZSEF4YWVSdlcxKzdLcmtG?=
 =?utf-8?B?Nm52QjBSd2psS2xVSHVEUWRTZVhSTGQrTUQ0dEpGSEovT2NsVm8vMlZKMTJi?=
 =?utf-8?B?eWREVytHZVBTMEFSZUVtUDlkOU9YQjkwWDM4amVESGZlNTRyQlh2dklUOWtr?=
 =?utf-8?B?SUpaMU5FRXBud0t1bUU1dzFMU1N5elV3VFVESGxmWGlLeHdLMzI4NjcrUUh6?=
 =?utf-8?B?c3hNbTZCSm90a1JoR2tPUVUwZFJiTng4cFhkbGJaUkF3S2YrYVpWUkswVVFn?=
 =?utf-8?B?VDZIWG5ORmxwS2tZV2VQbTdELzZWOXVoVmEydHpZNTZXOWhuQU1lOFhHYlYy?=
 =?utf-8?B?Wkw1eUQ2anJjRGhZdWZwUk1UUGdsSnNCWWtxVnZFd000anQ3dzJmZWl2U2ly?=
 =?utf-8?B?NHZMaHF0UHdsWnZ5YkFGdDc4S0VmcjE1SDBLM0tnWFpDdjVDY2kyQ2Y2QTJw?=
 =?utf-8?B?QlBNSTg0TWE4amU5TndBZjdyQWh3YXR4bCtLMmFmbFdObWlHRVNTM3JnNnJt?=
 =?utf-8?B?S3QxS2dFeUo2QVBpRWZ3ZldkUGJEZ3lhYzFISk5oUnJuWVZpbi8yNlphOHlu?=
 =?utf-8?B?aHJ2dVRyTXYrUEhKN29LbXU5bXFQK0ZwOXNKeUVIcTZ6SHZuR3Z3cTF5eTFL?=
 =?utf-8?B?VHNCSjU0bHRKcnN6UmZoZURjVWNkdGJ2TmVWWkJ6OVFVSFdHWkJCVnVaT0Uz?=
 =?utf-8?B?QmlOZEJkUmxGZElSTGxRYVoxVG9wNmhNQmliSGlKcXdZZ3h6Ri9NVHp3Q2ph?=
 =?utf-8?B?ZDdlYW9ob29SSkdZTDJWRVI3d2ZVMm0vNzJBRDRmaG1hTHFVSmNCdHBUSzJQ?=
 =?utf-8?B?Yy8xS0JJdnUxSnN5aStwSDdYdjVXN3hYOWFPZUxBTDFjN0Rva1MvTVlmV3Bs?=
 =?utf-8?B?MGVhOUNTUll1NU9HNC9QR21UQ1BPdWZ3cDVDQnFxTU12ZzdyQ0FvVllUaXFM?=
 =?utf-8?B?eG16S0I0VUIwVVY4RHpBRGp0YmVTdENsU0NML0V3MjJRdDRUYjN0Q1pOSlpS?=
 =?utf-8?B?aGtqcmtpS3RiZmhjVldMV09VRUpGVk5LdnNlNDNVY1d0c1QxVkVIU3B5dSto?=
 =?utf-8?B?R0NDa2dsNFRMZi9VOUprckFDQTZzZExxOExqWWVUSXpmMDVZM1h0eXMxWjJP?=
 =?utf-8?B?bGR6U2MxVzRVRUZOSTkwNDFrdjdaMnl4NVBpQ254aGNiN0ttRkppOUE4MXIr?=
 =?utf-8?B?elA4QlNsMGtsMEUyQjB4QlJ1SXZsMGlEanZ3bHltekM5R1pMb0JhQjdJY3VS?=
 =?utf-8?B?MkhnWlVia2Y1N2x6Vm9KSFJjdUxNVGpvODZTSUNZS2FBdnZ5MXo1REhMSmoz?=
 =?utf-8?B?cC92eDY4czZLSUsyb2hRY2xBcnNENjdRQTV5Sjl5d3RxTGR5Uk1SaSs4dUNu?=
 =?utf-8?B?dlNkK3ZSWnZEdXJ2TWU4MlFMSnEyOHFrV05PL1VSZnVxR0Yza1o1bWhsVGZq?=
 =?utf-8?B?TC9yZjIwRGExMGdtSDdmOXNWZUM4NVlzMmxpTzhNcEVoQ2xNbTYzVGhIaXM1?=
 =?utf-8?B?NTdDZ2pYQWErcG5WUkhYMUtwS01vZVJ5Sy9JSXhrcXdkdzZVUnF4RFRzQmwv?=
 =?utf-8?B?ZDdoRUNoZE1KSzd0KzBTN3hsb2ltSi9lOHBxa1drSldtNFVJc2tUTWxNTTFT?=
 =?utf-8?B?NStGTUcrYTBVTjYwVWdPb1Y2azlIV3lIRTJMUDU5aTZpL3hHeTV2M3ptbFhJ?=
 =?utf-8?B?VDljUWtjTmtQWkpLMWdQWVR6TVRsTFd0TUJqYlRndk5xOVFyR2lQSFJZNHhF?=
 =?utf-8?B?d0kxQzhWUHFydE5qNDlYc1U0Q2ppV0dXSjRUU1NZN2kwTFZjVjNQQ05ETEVW?=
 =?utf-8?B?cVRSMXJ3cncrTTNuYklQWjZLUytwYjRaL3h0ZjQzU2JzWnVoQUZGci9GaGN4?=
 =?utf-8?Q?35HO5yQwFP8sLpbknKy+zg40rKnbgl+N?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e37c0224-32d8-4ec0-96ea-08d994083cab
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Oct 2021 20:28:54.9816
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: reinette.chatre@intel.com
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB1300
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Babu,

On 10/20/2021 12:22 PM, Babu Moger wrote:
> On 10/20/21 1:15 PM, Reinette Chatre wrote:
>> On 10/19/2021 4:20 PM, Babu Moger wrote:
>>> On 10/1/21 11:02 AM, James Morse wrote:
>>>> __rmid_read() selects the specified eventid and returns the counter
>>>> value from the msr. The error handling is architecture specific, and
>>>> handled by the callers, rdtgroup_mondata_show() and __mon_event_count().
>>>>
>>>> Error handling should be handled by architecture specific code, as
>>>> a different architecture may have different requirements. MPAM's
>>>> counters can report that they are 'not ready', requiring a second
>>>> read after a short delay. This should be hidden from resctrl.
>>>>
>>>> Make __rmid_read() the architecture specific function for reading
>>>> a counter. Rename it resctrl_arch_rmid_read() and move the error
>>>> handling into it.
>>>>
>>>> Signed-off-by: James Morse <james.morse@arm.com>
>>>> ---
>>>> Changes since v1:
>>>>    * Return EINVAL from the impossible case in __mon_event_count() instead
>>>>      of an x86 hardware specific value.
>>>> ---
>>>>    arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  4 +--
>>>>    arch/x86/kernel/cpu/resctrl/internal.h    |  2 +-
>>>>    arch/x86/kernel/cpu/resctrl/monitor.c     | 42 +++++++++++++++--------
>>>>    include/linux/resctrl.h                   |  1 +
>>>>    4 files changed, 31 insertions(+), 18 deletions(-)
>>>>
>>>> diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>>>> b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>>>> index 25baacd331e0..c8ca7184c6d9 100644
>>>> --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>>>> +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>>>> @@ -579,9 +579,9 @@ int rdtgroup_mondata_show(struct seq_file *m, void
>>>> *arg)
>>>>          mon_event_read(&rr, r, d, rdtgrp, evtid, false);
>>>>    -    if (rr.val & RMID_VAL_ERROR)
>>>> +    if (rr.err == -EIO)
>>>>            seq_puts(m, "Error\n");
>>>> -    else if (rr.val & RMID_VAL_UNAVAIL)
>>>> +    else if (rr.err == -EINVAL)
>>>>            seq_puts(m, "Unavailable\n");
>>>>        else
>>>>            seq_printf(m, "%llu\n", rr.val * hw_res->mon_scale);
>>>
>>> This patch breaks the earlier fix
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Fcommit%2F%3Fh%3Dv5.15-rc6%26id%3D064855a69003c24bd6b473b367d364e418c57625&amp;data=04%7C01%7Cbabu.moger%40amd.com%7C85219a5827114935cdaa08d993f59fa0%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637703505420472920%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=yP8awDgGGZ%2BWj5ZItdTNJItTVuK828yGnibwq%2BrVaf0%3D&amp;reserved=0
>>>
>>>
>>> When the user reads the events on the default monitoring group with
>>> multiple subgroups, the events on all subgroups are consolidated
>>> together. In case if the last rmid read was resulted in error then whole
>>> group will be reported as error. The err field needs to be cleared.
>>>
>>> Please add this patch to clear the error.
>>>
>>> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c
>>> b/arch/x86/kernel/cpu/resctrl/monitor.c
>>> index 14bc843043da..0e4addf237ec 100644
>>> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
>>> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
>>> @@ -444,6 +444,8 @@ void mon_event_count(void *info)
>>>           /* Report error if none of rmid_reads are successful */
>>>           if (ret_val)
>>>                   rr->val = ret_val;
>>> +       else
>>> +               rr->err  = 0;
>>>    }
>>>
>>>    /*
>>>
>>
>> Good catch, thank you.
>>
>> Even so, I do not think mon_event_count()'s usage of __mon_event_count()
>> was taken into account by this patch and needs a bigger rework than the
>> above fixup. For example, if I understand correctly ret_val is the error
>> and rr->val no longer expected to contain the error after this patch. So
>> keeping that assignment to rr->val is not correct.
> 
> Yes. You are right. rr->val is not expected to contain the error.
> Hopefully, this should help.
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c
> b/arch/x86/kernel/cpu/resctrl/monitor.c
> index 14bc843043da..105d972cc511 100644
> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
> @@ -441,9 +441,9 @@ void mon_event_count(void *info)
>                  }
>          }
> 
> -       /* Report error if none of rmid_reads are successful */
> -       if (ret_val)
> -               rr->val = ret_val;
> +       /* Clear the error if at least one of the rmid reads succeed */
> +       if (ret_val == 0)
> +               rr->err = 0;
>   }
> 
>   /*
> 

Yes, this looks good. If the first __mon_event_count() succeeds but a 
following one fails then the data still needs to be reported so the 
error code needs to be fixed up afterwards and cannot be done inside 
__mon_event_count(). Thank you very much.

Reinette