Message-ID: <16e7fd97-49e4-0180-86bc-9dfd5bf90940@intel.com>
Date:   Fri, 10 Mar 2023 16:22:16 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.7.2
Subject: Re: [PATCH v2 18/18] x86/resctrl: Separate arch and fs resctrl locks
Content-Language: en-US
To:     James Morse <james.morse@arm.com>, <x86@kernel.org>,
        <linux-kernel@vger.kernel.org>
CC:     Fenghua Yu <fenghua.yu@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        H Peter Anvin <hpa@zytor.com>,
        Babu Moger <Babu.Moger@amd.com>,
        <shameerali.kolothum.thodi@huawei.com>,
        D Scott Phillips OS <scott@os.amperecomputing.com>,
        <carl@os.amperecomputing.com>, <lcherian@marvell.com>,
        <bobo.shaobowang@huawei.com>, <tan.shaopeng@fujitsu.com>,
        <xingxin.hx@openanolis.org>, <baolin.wang@linux.alibaba.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Xin Hao <xhao@linux.alibaba.com>, <peternewman@google.com>
References: <20230113175459.14825-1-james.morse@arm.com>
 <20230113175459.14825-19-james.morse@arm.com>
 <d7d43c53-a482-100f-8a25-4dae6500d184@intel.com>
 <cc4eed18-1966-ebcf-8ae1-81b7e0833299@arm.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <cc4eed18-1966-ebcf-8ae1-81b7e0833299@arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?K0JUNmoySWphMTljYlh2bnFWZlZhNSt4RWRLZXNLUW4xT2VYMFRnZXJGSXVk?=
 =?utf-8?B?Yk9kdjR3SjJITHhxSkhWU2VmaWdublRCcnd0Y1gxT090cW1JMHJSdDY5Rm5Y?=
 =?utf-8?B?YWVJdFFWTUxESnZvQzBwVFBBejAvLzZiNVdkK2M2L2pPSFVDbEJPcE91V0Qz?=
 =?utf-8?B?TVJPYzcxdzF3a1pNU1dJT3pueUgzS1ZSYm04cUZkbzM4MDJkbjZWdllQU3k5?=
 =?utf-8?B?bHFXZnZPZlVZMkJZbEdmVGtaM1ROMlRYdjNRUmN2NXFydHcwL2xheTBPZStV?=
 =?utf-8?B?T29aRmczSjBTWnRqL0Q1ZnB6Tjg0VHNja2NnVjh0SndTcXVGc2ppSllhQVRs?=
 =?utf-8?B?M3dRRHFyL0l4bU1WVTJJcFlqZkxpa2gwMVFjelc0eEdOeis2eW5OTk04elBs?=
 =?utf-8?B?M0MvOGpJQjZ6UzE4Uno4VGluOHR5VFBsc05ROGJGUFlYVGZUZTFLZEZzaUpo?=
 =?utf-8?B?WFhacHRzbzlXb1gvc3lpWlJGcDgxLzNpdnJsYTl1cDhINTNaSG12VG5hMnlU?=
 =?utf-8?B?dGNTVldwVS9RM24zYi82QXZweitBV0lkU29ya2djeThYaC9WRDNuZTJmSUpY?=
 =?utf-8?B?RFlkSjlyZEY2Z3NXMWo3YklaQXBmUTNKamVtNi81MXQ2M2h6MTVzYm1Ydkhi?=
 =?utf-8?B?T0RWSXpWYUlidHVGNVJVNGlPc3JWWndld0hrcWJWeGJvV002d0J3aTc3VFQ3?=
 =?utf-8?B?TDExZ1VONm1DdTAyd3k0UHFYSWRJVG5vRCtjZzF0V2ovMThNODdxRzNLM2x0?=
 =?utf-8?B?dlNKOFZaRXlWTHJYYVZ5YlNwc2F0a3NUdTUxQmdVWUFDTXBaNUhiemZTZTFq?=
 =?utf-8?B?THJTeUhDT1M4RC9rYzIxcUxEY0gvQ21OcUpORzYySXgvcjkrQit1SWV1VGha?=
 =?utf-8?B?RGpRY1NLcXY3Q2k2dnNNdW9Za0dkcUYrb3RIUU9NdEVqNUQ4Q0lTU29pUFNr?=
 =?utf-8?B?UGN6eUNLL243WmFnQnNVVUxkdzFQSUNhQVBwdU1PU3R4Y0U1cENia1V0d3BY?=
 =?utf-8?B?VmhCV1MzaHJBZWJGLzcybFZGeUFabDFGU05jakdET25icDZEMWNpcXA5b1F2?=
 =?utf-8?B?bm9KWklQSDlMaWhSWXI3SVpKUUI5YzZpMHQ1cjM1YjhFLys0UzJLV2w2QVRi?=
 =?utf-8?B?b1ozQ1pIQlAvUFpBTE1sdGR0Yk1haXZoRHFVbzY1Qk1kNHpOeXhSRGZKZG9L?=
 =?utf-8?B?ajdVcTZQcW9XUUZTNTdSaDN2Q0lZeGJoNGR1ZTAyT0FNNEs2Kzg1cUIrN1pw?=
 =?utf-8?B?VEcxc3hQVE45MlRmREppR2FBTlFuWnZGMDFrQytwUmpnTU04QnhPMGZjbFBQ?=
 =?utf-8?B?UUtLZHFRUTBtbnVsMFpVZ2dacW9aRFdGOGN0Q09mRkJUTmpoblBqZHFQSGdO?=
 =?utf-8?B?S1B6Z2gzYXg3YzJpdEM5VEk0UDlwOWFvd2RDOWpXZjZ1cTRnbS82RytQSkpq?=
 =?utf-8?B?ZG5Oc3FLSWRXOTJNRjhzRGMxS1VZdzBBbkNUc2lYT1JXOXU4OC9KN0dUaXZC?=
 =?utf-8?B?VFlXYkRNSVRWaHpkTWtBUGh1Y3NWYk1rNWtFenNKZExyYWt2RGpLaUZuUURZ?=
 =?utf-8?B?emNOWHVoL2VLN2VieHBMSlE0S0I0bTFuT2UvQ2pyK1VxWEZreHRWTmtjMkhZ?=
 =?utf-8?B?WlV0K25uYkFTaVhqeDhNUGF3NEtmY0t3R01xMUhLTFRWNDI5M2t6UlJQcVI4?=
 =?utf-8?B?d2ljOVBkWE1RY2ZQZHVIRWNUOFZTdFhSbEwrMEQwUm1pY2ZkWkxTMG5hK0tH?=
 =?utf-8?B?Uk5LYi8rN1VWVXhtYndPOFRlRHlYektVTXlHa1EwenF1N1FuQ05MU3pxcldz?=
 =?utf-8?B?SFQ5bGpBWmNqbFlEMnBlZy96dVRJam9FQ2tqNE5jQit0SWRDNVRkNzJLYzE2?=
 =?utf-8?B?YUpwTHdyV0tVV2ZvUHdHRSsybEZURzY5d3A2Y1ltaU9LeHZRa1o4b3FHeEc0?=
 =?utf-8?B?c2xRSy9ZbVJyZnhCNy9pR0pXUVduQ2s5MGRtZGV2WlQ5V1hpQytjc3VQQnhv?=
 =?utf-8?B?S0tOZy9KWlhYbXhUdkMreEw0RUlKWU01d1Ezb1JJeG93TVFUSFBGT1R0ZndD?=
 =?utf-8?B?Slg2eDhmdWI3TUxudG5NcnM3ZXUxTjNKeTVTTUNmSlpqTUlMRWE5aUhtK21M?=
 =?utf-8?B?bWhYVUs1V2o2THdaUHBLdFBXQ0lwZzYzTnZxVEc0MzZLR3JkeWwyU0U4a203?=
 =?utf-8?B?SkE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 8022a86e-c6a9-4e27-79d6-08db21c6ad45
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Mar 2023 00:22:20.0903
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: SHdbfv2ksoeY5gzxMP7C7j2R3y8NgXcZydruMOHtGUabSnV05Co2EPCqGAqfNboFfIfAq7ydg4gKTCCiqxbk90hgOoYqGRu5KnKuWlJrUQU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB4530
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi James,

On 3/6/2023 3:34 AM, James Morse wrote:
> Hi Reinette,
> 
> On 02/02/2023 23:50, Reinette Chatre wrote:
>> On 1/13/2023 9:54 AM, James Morse wrote:
>>> resctrl has one mutex that is taken by the architecture specific code,
>>> and the filesystem parts. The two interact via cpuhp, where the
>>> architecture code updates the domain list. Filesystem handlers that
>>> walk the domains list should not run concurrently with the cpuhp
>>> callback modifying the list.
>>>
>>> Exposing a lock from the filesystem code means the interface is not
>>> cleanly defined, and creates the possibility of cross-architecture
>>> lock ordering headaches. The interaction only exists so that certain
>>> filesystem paths are serialised against cpu hotplug. The cpu hotplug
>>> code already has a mechanism to do this using cpus_read_lock().
>>>
>>> MPAM's monitors have an overflow interrupt, so it needs to be possible
>>> to walk the domains list in irq context. RCU is ideal for this,
>>> but some paths need to be able to sleep to allocate memory.
>>>
>>> Because resctrl_{on,off}line_cpu() take the rdtgroup_mutex as part
>>> of a cpuhp callback, cpus_read_lock() must always be taken first.
>>> rdtgroup_schemata_write() already does this.
>>>
>>> All but one of the filesystem code's domain list walkers are
>>> currently protected by the rdtgroup_mutex taken in
>>> rdtgroup_kn_lock_live(). The exception is rdt_bit_usage_show()
>>> which takes the lock directly.
>>
>> The new BMEC code also. You can find it on tip's x86/cache branch,
>> see mbm_total_bytes_config_write() and mbm_local_bytes_config_write().
>>
>>>
>>> Make the domain list protected by RCU. An architecture-specific
>>> lock prevents concurrent writers. rdt_bit_usage_show() can
>>> walk the domain list under rcu_read_lock().
>>> The other filesystem list walkers need to be able to sleep.
>>> Add cpus_read_lock() to rdtgroup_kn_lock_live() so that the
>>> cpuhp callbacks can't be invoked when file system operations are
>>> occurring.
>>>
>>> Add lockdep_assert_cpus_held() in the cases where the
>>> rdtgroup_kn_lock_live() call isn't obvious.
>>>
>>> Resctrl's domain online/offline calls now need to take the
>>> rdtgroup_mutex themselves.
> 
> 
>>> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
>>> index 7896fcf11df6..dc1ba580c4db 100644
>>> --- a/arch/x86/kernel/cpu/resctrl/core.c
>>> +++ b/arch/x86/kernel/cpu/resctrl/core.c
>>> @@ -25,8 +25,14 @@
>>>  #include <asm/resctrl.h>
>>>  #include "internal.h"
>>>  
>>> -/* Mutex to protect rdtgroup access. */
>>> -DEFINE_MUTEX(rdtgroup_mutex);
>>> +/*
>>> + * rdt_domain structures are kfree()d when their last cpu goes offline,
>>> + * and allocated when the first cpu in a new domain comes online.
>>> + * The rdt_resource's domain list is updated when this happens. The domain
>>> + * list is protected by RCU, but callers can also take the cpus_read_lock()
>>> + * to prevent modification if they need to sleep. All writers take this mutex:
>>
>> Using "callers can" is not specific (compare to "callers should"). Please provide
>> clear guidance on how the locks should be used. Reader may wonder "why take cpus_read_lock()
>> to prevent modification, why not just take the mutex to prevent modification?"
> 
> 'if they need to sleep' is the answer to this. I think a certain amount of background
> knowledge can be assumed. My aim here wasn't to write an essay, but indicate not all
> readers do the same thing. This is already the case in resctrl, and the MPAM pmu stuff
> makes that worse.
> 
> Is this more robust:
> | * rdt_domain structures are kfree()d when their last cpu goes offline,
> | * and allocated when the first cpu in a new domain comes online.
> | * The rdt_resource's domain list is updated when this happens. Readers of
> | * the domain list must either take cpus_read_lock(), or rely on an RCU
> | * read-side critical section, to avoid observing concurrent modification.
> | * For information about RCU, see Docuemtation/RCU/rcu.rst.
> | * All writers take this mutex:
> 
> ?

Yes, I do think this is more robust. Since you do mention, "'if they need to sleep'
is the answer to this", how about "... must take cpus_read_lock() if they need to
sleep, or otherwise rely on an RCU read-side critical section, ..."? I do not
think it is necessary to provide a link to the documentation. If you do prefer
to keep it, please note the typo.

Also, please cpu -> CPU. 

>>> @@ -569,30 +579,27 @@ static void clear_closid_rmid(int cpu)
>>>  static int resctrl_arch_online_cpu(unsigned int cpu)
>>>  {
>>>  	struct rdt_resource *r;
>>> -	int err;
>>>  
>>> -	mutex_lock(&rdtgroup_mutex);
>>> +	mutex_lock(&domain_list_lock);
>>>  	for_each_capable_rdt_resource(r)
>>>  		domain_add_cpu(cpu, r);
>>>  	clear_closid_rmid(cpu);
>>> +	mutex_unlock(&domain_list_lock);
> 
>> Why is clear_closid_rmid(cpu) protected by mutex?
> 
> It doesn't need to be, its just an artefact of changing the lock, then moving the
> filesystem calls out. (its doesn't need to be protected by rdtgroup_mutex today).
> 
> If you don't think its churn, I'll move it to make it clearer.
> 

I do not see a problem with keeping the lock/unlock as before but
if you do find that you can make the locking clearer then
please do.

Reinette