Message-ID: <7c9a8ba2-f671-fb31-8146-58e972f1cd5a@intel.com>
Date:   Fri, 10 Mar 2023 12:00:34 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.7.2
Subject: Re: [PATCH v2 08/18] x86/resctrl: Queue mon_event_read() instead of
 sending an IPI
Content-Language: en-US
To:     James Morse <james.morse@arm.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "x86@kernel.org" <x86@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
CC:     Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        H Peter Anvin <hpa@zytor.com>,
        Babu Moger <Babu.Moger@amd.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        D Scott Phillips OS <scott@os.amperecomputing.com>,
        "carl@os.amperecomputing.com" <carl@os.amperecomputing.com>,
        "lcherian@marvell.com" <lcherian@marvell.com>,
        "bobo.shaobowang@huawei.com" <bobo.shaobowang@huawei.com>,
        "tan.shaopeng@fujitsu.com" <tan.shaopeng@fujitsu.com>,
        "xingxin.hx@openanolis.org" <xingxin.hx@openanolis.org>,
        "baolin.wang@linux.alibaba.com" <baolin.wang@linux.alibaba.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Xin Hao <xhao@linux.alibaba.com>,
        "peternewman@google.com" <peternewman@google.com>
References: <20230113175459.14825-1-james.morse@arm.com>
 <20230113175459.14825-9-james.morse@arm.com>
 <IA1PR11MB6097ED9919B408116EFE161E9BC69@IA1PR11MB6097.namprd11.prod.outlook.com>
 <ab8c7a00-8c7e-97ab-25db-92053c7b45cc@arm.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <ab8c7a00-8c7e-97ab-25db-92053c7b45cc@arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SUtyYW1pT2g0dkx4V05vKzYreWg4eU8ybHJkY1JHa055TVhHci84UWNSbEpU?=
 =?utf-8?B?OVJsTGxtVThXdGU4Q0VBTzBVa0ZFVkZRVDN3OVR6YVZBNENlSGdsMTFqaENS?=
 =?utf-8?B?eStFTFhZaERCOHVHRHRQbGd4TUhQaUFDZGsxZ1dLYXh4ZWRXUDRBVGpwVElH?=
 =?utf-8?B?QmJCL2U3N2Zza0ZtY1d2Y1ZST1dCd3NxUEE5Sjl4R2NidmhrRlVXczV3eDlv?=
 =?utf-8?B?WDBWRk1SajU0eXJnUHRxRk1xMXdtalNkNEs5VHcxYXlCajdESEQzQnRMdWFm?=
 =?utf-8?B?Kyt3OTBkS3QwaHdhc2lFVjBDRUVKRS9rM3JLanZwTG5QVDRDRWNjdVBJbXMy?=
 =?utf-8?B?enkrSkR3SDZRMWxjcUVmTlg4eXltZGpIOEdkb0RHanJwZjVlSmVDL01MSGJq?=
 =?utf-8?B?VUxXMmY2clJPMXZ2QVlvNUFBMWp3NVdDR3BQU1NmMERYY1c5N09sZjE2Y050?=
 =?utf-8?B?UVNXYlpDTHBWYVhwSnNoaUc1eWNSTkVhZlZIaEFVaElEUWNjbk5nMG5ld1Fq?=
 =?utf-8?B?YUF1bWUreFovZ1Z5aWhEem9xYXl1N253ODdac3Q2enpwREhibDZXaXY3ZXJQ?=
 =?utf-8?B?WGdzcllaVm04dFRZLzVjZS9OMzlUdVZjbUxEVFd3aGNoU0pnb0xyWkIzUnor?=
 =?utf-8?B?L0FrNnU4V2VMbkR0RjNzbTlaMmUvSWJlenV4TTY3Uzg0WEE1aTlUczV1OW5H?=
 =?utf-8?B?dkxCWU9McjRGRDZMMWZ6b3dtMk9vVnAvSTg2dldSRDEvcG5oZjBDbUE4UUFi?=
 =?utf-8?B?bktzMlF4cVVHVGt0QzlVbERCU1IvMGo1bVY4QVdhY01KUUpIQkhlckxGVDRs?=
 =?utf-8?B?NVVCWmlFY1A1a3FXcEVCcVNZRjVjNDMyWngvT2Z0VjVwR2ViMUVESHA3NU9J?=
 =?utf-8?B?eEN3akVTM3VHUEhteDZzTkRRcnRvWWpTcldYNGU5ckZpeTN6Z3RGclk1V05P?=
 =?utf-8?B?RmdyM1VuazdUajNuQm1rcXFuRWM3V1ljSWJFbDR1NEM1WTBDRnluMWloN0ZD?=
 =?utf-8?B?d0t0TDdiTHRJMTFYUDRma25aaWFYMTZubzl1NHN2Q0llS0hmWndYMUFEN1Y5?=
 =?utf-8?B?aWN5S0UzYXljbkU0SmpQRVU2UEcvcDYwNEk0eUlXUlAwR2x6RkVqWk5DMmhH?=
 =?utf-8?B?aVJFVUFZMytOWG1aTE9EV1J5bTkyeWRNSzJlYnRpckRTdW9VaUxNZndEVDdi?=
 =?utf-8?B?WHdZM3BIaVhkazl5MTZDaXkyLzNKdjJSR3o1cWxzMTRhWDN2Rk0zQ3hlTWdZ?=
 =?utf-8?B?TEY4cHBHYlcyTkVRa1VDR3pQY3J6dGNJNWoxa3lrVUVnK2UyYWlPMWUxcGx0?=
 =?utf-8?B?VkFsMG1HbTVSUVgrM3RKUVpzM2JxZFp6YXl2Nm9pMSs2SUlrMDRqTUYwWGpH?=
 =?utf-8?B?b1RFcmhxN2F6OTF1UTdpeFRuamF6NVkwZGNEdERVR01HemorUjE2amQxWTZE?=
 =?utf-8?B?SHAwUS9JQUVXOTB4RXB3b0puZnZLNkxUZUhNbUt2TlhRTlR5QUZQNElGMVVY?=
 =?utf-8?B?M1doMU5KTVZramZ4V1Z4YnhLRnRIelZTS2luQmQrRXp6dHVDVEZhdzYyYzk4?=
 =?utf-8?B?N3p6M0QzMzRLcWxHK25weGV2R2lRS2tSVTNQNjVWVnZMTjNyR2MwNndIbHZm?=
 =?utf-8?B?NlBBQkxzRmU1bSt6ZDliZjhiRlg3VW8yYU5Ob1JIeVp1ZEVxK0FoUE5BREFG?=
 =?utf-8?B?WFQvRjd6U2pNYk9mVlVtRkZKakY4Z1dmSS9wNUU4dnpCTjRqbnNPd05EOEJE?=
 =?utf-8?B?S0IwZ093OVkxSzgvNjd2RzA5ZFBzRUd6TDR6R3pKQXFTRFJaS2JQaS8zVHlv?=
 =?utf-8?B?N0VyYlZ2djBlenpnTDJzWmlrOHNkSlFnZzg0bjlMTWxlbUlQNGFYY29wNVNz?=
 =?utf-8?B?SEVjMmQrSnZ6R1lqUEF4ZmVMWTJPMHN3OUtHSGgydE5hRmc2dDBXL0VYNnNO?=
 =?utf-8?B?U1VjTzBudWdtUFV2N3R0eGVsSUx6dTNSYkZ1Z0wzTFRibTJHQjRDVjAyRVY0?=
 =?utf-8?B?WnRTU2QwNHlQTytianlTQzhnM015b2kvOHBMSEFiRE5GWTRZUVZQbWd6ZzZr?=
 =?utf-8?B?dGh1WTVPVE5waXlIYTh0dHNWR2p5cWgxZ0hVaG8vWStBYnl2OXVVdTJJTUds?=
 =?utf-8?B?cVA3RlVQZEYrUEJIZGNuOFNDUUFBeU9OaEZZbUJ6eElMN3BUQVM0TS9BYUI1?=
 =?utf-8?B?NGc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 6837faad-8957-4af2-a718-08db21a21ddf
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Mar 2023 20:00:37.5015
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mEqu4FUoVUIEWtopcnwvPW6b0h/lYBLOeWbVyHYfIx+z1jxXJtRrE1+K6/gB24PPbxRtYjZOIDPPaEl0Yb4LCXC+W8xYhS90PY22twmmBNY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6353
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi James and Fenghua,

On 3/6/2023 3:32 AM, James Morse wrote:
> On 17/01/2023 18:29, Yu, Fenghua wrote:
>>> x86 is blessed with an abundance of monitors, one per RMID, that can be read
>>> from any CPU in the domain. MPAMs monitors reside in the MMIO MSC, the
>>> number implemented is up to the manufacturer. This means when there are
>>> fewer monitors than needed, they need to be allocated and freed.
>>>
>>> Worse, the domain may be broken up into slices, and the MMIO accesses for
>>> each slice may need performing from different CPUs.
>>>
>>> These two details mean MPAMs monitor code needs to be able to sleep, and IPI
>>> another CPU in the domain to read from a resource that has been sliced.
>>>
>>> mon_event_read() already invokes mon_event_count() via IPI, which means this
>>> isn't possible.
>>>
>>> Change mon_event_read() to schedule mon_event_count() on a remote CPU
>>> and wait, instead of sending an IPI. This function is only used in response to a
>>> user-space filesystem request (not the timing sensitive overflow code).
> 
>> But mkdir mon group needs mon_event_count() to reset RMID state.
>> If mon_event_count() is called much later, the RMID state may be used
>> before it's reset. E.g. prev_msr might be non-0 value. That will cause
>> overflow code failure.
>>
>> Seems this may happen on both x86 and arm64. At least need to make sure
>> RMID state reset happens before it's used.
> 
> On the architecture side, there is a patch from Peter that records the MSR value on the
> architecture side when an RMID is reset/re-allocated. 2a81160d29d6 ("x86/resctrl: Fix
> event counts regression in reused RMIDs")
> 
> For the filesystem, the 'first' value is passed through and handled by the CPU that reads
> the MSR. I don't see what problem any extra delay due to scheduling would cause.
> 

Both the monitor directory creation and the overflow handler rely on the
rdtgroup mutex so I do not see how an overflow code failure could sneak in here.

Reinette