Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <3b020456-6312-89c7-1391-06241fa0f3de@intel.com>
Date:   Tue, 13 Dec 2022 10:33:55 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.5.1
Subject: Re: [PATCH v4 1/2] x86/resctrl: Update task closid/rmid with
 task_call_func()
Content-Language: en-US
To:     Peter Newman <peternewman@google.com>
CC:     <bp@alien8.de>, <derkling@google.com>, <eranian@google.com>,
        <fenghua.yu@intel.com>, <hpa@zytor.com>, <james.morse@arm.com>,
        <jannh@google.com>, <kpsingh@google.com>,
        <linux-kernel@vger.kernel.org>, <mingo@redhat.com>,
        <tglx@linutronix.de>, <x86@kernel.org>
References: <cdcfcd64-c76f-0d2d-6653-0229c956f2bc@intel.com>
 <20221212173638.1858573-1-peternewman@google.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20221212173638.1858573-1-peternewman@google.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bFd5MXpVVmVCbEhwaUVlMnI5STVOa21RSmk4QlZNZXZ6MzNNSS81TGh0QklG?=
 =?utf-8?B?eHY1R2RBK0VNVnZUZW1xQWJQWmFPaER4bk82cnR0a2p1ajdLZnl1QlJJWEdS?=
 =?utf-8?B?SEdqbmVWMk80cEU0TktuYkZWeXc2cWxNUlVNcVRhSlBEbnFQYjRNdzcyd0tS?=
 =?utf-8?B?MWlYRy9GL21UbXA1M0FwOEtRMzd0VTVRL0hPbU9pOU54OThiWHg3c0xHVUFu?=
 =?utf-8?B?TlJSNVZBajlCSWYrSzVYeU4rajBHQUMwZW9GWkJPdDNYYmpmQlVPOThOTWhT?=
 =?utf-8?B?Z2FET2VhRGh0eEg2TFQrVk83UzBMQWlpSmgzTjdXcUVVUTZkQ0ZCOHQvWEhH?=
 =?utf-8?B?eU1TcVNFeURDUkduL25KTDJqekM0dmljNHRGZ2hUdHpkR3NEd2pwREIvY0lD?=
 =?utf-8?B?STVjM0JPWVhuMWMreGtHZlg3bU1iREhyTXAvRSsvcS8wc0Q4QXNPQlZBd3RF?=
 =?utf-8?B?YkxnalR6bUprbjNvL2w4aVdxbWZVWkV3bk9IalZ6clMrOCs1NFMzL3NRVHhL?=
 =?utf-8?B?UEFuSEJhWmx5WHVSYmt3T1VXWFNibjFLY2FmZDdBbE1iUE5aQ2tRL0x1Y1VL?=
 =?utf-8?B?d090cXR6Qjk2Y0twT2VCSFo2V0VreGtzZmdZUzVPMEtJSU9yZHNwbkVwMHFp?=
 =?utf-8?B?bmhvaTRVV1VjZG1pNlJqL0FyUkQ1Ymo3a3F5THZVdjRuc2RQd1dleVUzcVdS?=
 =?utf-8?B?VVVnQ29LODc2U3NKUklKVGpPaGpoMmxvR2dBdUc1cGwyOXZBdlVXamtJMzVN?=
 =?utf-8?B?UnpxNk5qTGZoTlo5L1NLZ0M4QTFGTzVwR3U3bERzTS9PalU3eHgyajgrK1FI?=
 =?utf-8?B?a09LOW4yZTZFemJVS2lsdkRvVDFMSE5uWU55RzF4VUIwYi9GTk84MW4rYnNV?=
 =?utf-8?B?TlJiTVZsSTVZRVVxbk1yblhQc044OTB6clR2c2FrZ1llU1l3UkhrVXdLcDky?=
 =?utf-8?B?Mm5aeDJaNmM5Y0VTUW1KV21GM0NCanZiS1d5TUtGWlFaYVdnanFLQnM0K3VJ?=
 =?utf-8?B?ZTJFTWgyaFZLTk9mSVFmRjZ2ZmxEZ0Y3aXo3K2NRWDhDajVqbUN3c21pSlFv?=
 =?utf-8?B?VGU2RWhBRmZXYW13VFhOZWtMR3FpRFpJdFdzOFRNM3BXOGZsYWVTWG1QZzhv?=
 =?utf-8?B?S3JYS05qajlhdWJkU2JuYTlQblA3VnFRWVBNM1FMZ2ZXNGFhdU5vUDR1cVRn?=
 =?utf-8?B?SU5GUitrMUtzZXBtWUEzTUQxQ09aempoZGp5bTVjRk5ybGtvcnN4d2wvZ3p5?=
 =?utf-8?B?ODZzMGg5Wkg5OGl1U1czWFBVSmdUZEpWVUJmdU42cEYwc0grN05RQjE5M3Nm?=
 =?utf-8?B?R2wzUlpMZlBKc2JlOTI5aFVwdWswODdQOXhoQ3lqajRtK0I4SjZrYTRlck4r?=
 =?utf-8?B?clNpa2ZxWk40VHdVZGpGOWsxYTZEOWVaYkZBZkFnMnpwcjhFU3Y2QWpTQ3Fu?=
 =?utf-8?B?b1ZQNmdaVHphdDEvY21hMkxuc3dGN2d4cSt5aTdIM05CRit2UUJ5TnlNQnhq?=
 =?utf-8?B?aXJUWC9FTDZkUXdzcWMzRVRSSDFja0Uyc1RBaFZBMlFjZm43UW9uTmcrdUZO?=
 =?utf-8?B?bU8zMmNyOHJ3bnZWQis0cXJMdlNpejhzbkt4VTM2cGlacUJVM2Fmb0NVRmxo?=
 =?utf-8?B?S0xSZjJsNWM4VlNZZkxwbllEc0ZJelVuRHFXU0NMazRCVnBKRmFtZHNMWWQz?=
 =?utf-8?B?ZHF3REU3VzFvTlBiNkdLYURTbkdaRXdpSzVKRkZTQnJ1eCtKMkdnZXBUTFp2?=
 =?utf-8?B?RnpnSzQrYlFBUS9YL1Y2VGd4ZEMzeXVMWVF6VjVudmhSRmMxS05wdGV0S2ow?=
 =?utf-8?B?b1hmZEgwTWFsUUFVdWtjL1FoVzYxNG9zdXJXNU91Z3dHNGh6clBRZ1lVbHlr?=
 =?utf-8?B?c0hZc3lTMHowV0lDUndLSTUyazhQaTNWcWFpMGJsalJOdTJ0Nm9DVlB1VHNn?=
 =?utf-8?B?WUNnRWlMRkR2a0RoMThWbUV3NG96TkhIbHhLMHI2MCszRzVFWUIyNVovWXVO?=
 =?utf-8?B?K0FueVlQVGJVZndVU0hQejRCQjFEbGdodXBDK0ZtMXdNZU1zT25jdExDalhO?=
 =?utf-8?B?aHN2bjNweUVOcVh0a05ET21qMkhtUVErVkUraWtJTFM2L25tK243aC9kcldX?=
 =?utf-8?B?bkpDejFGNzljMFpxaExMTVFQY0xxMTEzVzFoSEZOcitRSDJlaVdLZkdVa3Rj?=
 =?utf-8?B?YVE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e13d2853-fb0a-4a92-2507-08dadd389986
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Dec 2022 18:33:59.2146
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 13g3ksWvx4bfbtgb4hMNR8ECBOvWOKsZkeqi2rOn+CFG9qzanznk5dkTbjpv47Q8UGmW3o8Vl8gO2y9QrOpnbhRAQt0W13KLjPNxAik5n+0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR11MB7346
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Peter,

On 12/12/2022 9:36 AM, Peter Newman wrote:
> On Sat, Dec 10, 2022 at 12:54 AM Reinette Chatre <reinette.chatre@intel.com> wrote:
>> On 12/8/2022 2:30 PM, Peter Newman wrote:
>>> Based on this, I'll just sketch out the first scenario below and drop
>>> (2) from the changelog. This also implies that the group update cases
>>
>> ok, thank you for doing that analysis.
>>
>>> can use a single smp_mb() to provide all the necessary ordering, because
>>> there's a full barrier on context switch for it to pair with, so I don't
>>> need to broadcast IPI anymore.  I don't know whether task_call_func() is
>>
>> This is not clear to me because rdt_move_group_tasks() seems to have the
>> same code as shown below as vulnerable to re-ordering. Only difference
>> is that it uses the "//false" checks to set a bit in the cpumask for a
>> later IPI instead of an immediate IPI.
> 
> An smp_mb() between writing the new task_struct::{closid,rmid} and
> calling task_curr() would prevent the reordering I described, but I
> was worried about the cost of executing a full barrier for every
> matching task.

So for moving a single task the solution may just be to change
the current barrier() to smp_mb()?

> 
> I tried something like this:
> 
> for_each_process_thread(p, t) {
> 	if (!from || is_closid_match(t, from) ||
> 	    is_rmid_match(t, from)) {
> 		WRITE_ONCE(t->closid, to->closid);
> 		WRITE_ONCE(t->rmid, to->mon.rmid);
> 		/* group moves are serialized by rdt */
> 		t->resctrl_dirty = true;
> 	}
> }
> if (IS_ENABLED(CONFIG_SMP) && mask) {
> 	/* Order t->{closid,rmid} stores before loads in task_curr() */
> 	smp_mb();
> 	for_each_process_thread(p, t) {
> 		if (t->resctrl_dirty) {
> 			if (task_curr(t))
> 				cpumask_set_cpu(task_cpu(t), mask);
> 			t->resctrl_dirty = false;
> 		}
> 	}
> }
> 

struct task_struct would not welcome a new member dedicated to resctrl's
rare use for convenience. Another option may be to use a flag within
the variables themselves but that seems like significant complication
(flag need to be dealt with during scheduling) for which the benefit
is not clear to me. I would prefer that we start with the simplest 
solution first (I see that as IPI to all CPUs). The changelog needs clear
description of the problem needing to be solved and the solution chosen, noting
the tradeoffs with other possible solutions. You can submit that, as an RFC
if the "IPI to all CPUs" remains a concern, after which we can bring that
submission to the attention of the experts who would have needed information then
to point us in the right direction.

Reinette