Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <b931062a-1a70-2331-03cc-6bfa69a51bee@intel.com>
Date:   Wed, 19 Oct 2022 16:54:30 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.3.3
Subject: Re: [RFD] resctrl: reassigning a running container's CTRL_MON group
Content-Language: en-US
To:     Peter Newman <peternewman@google.com>
CC:     Tony Luck <tony.luck@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "Eranian, Stephane" <eranian@google.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        "James Morse" <james.morse@arm.com>,
        Babu Moger <Babu.Moger@amd.com>,
        "Gaurang Upasani" <gupasani@google.com>
References: <CALPaoCj-zav4x6H3ffXo_O+RFan8Qb-uLy-DdtkaQTfuxY4a0w@mail.gmail.com>
 <b2e020b1-f6b2-e236-a042-4eb2fd27d8b0@intel.com>
 <IA1PR11MB6097236CFF891041DBA42ECB9B5F9@IA1PR11MB6097.namprd11.prod.outlook.com>
 <Y0BhzKkksSjSeE3W@agluck-desk3.sc.intel.com>
 <81a7b4f6-fbb5-380e-532d-f2c1fc49b515@intel.com>
 <CALPaoCjdeRjyX5L6BBX688ZM21eMwetuL9QLF1+GEDUskGcU2w@mail.gmail.com>
 <7b09fb62-e61a-65b9-a71e-ab725f527ded@intel.com>
 <CALPaoCg4zrODVoXF2y2b+LRYq_+jnV8yv5qB+T_3Z264cV82GQ@mail.gmail.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <CALPaoCg4zrODVoXF2y2b+LRYq_+jnV8yv5qB+T_3Z264cV82GQ@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RFAzSlgxZHRUNTF4MUpmeEptR1Rlc0NFczRXMnRnTVZrem12Q0wxeEN0TmRP?=
 =?utf-8?B?WmhqR0t3WDc3SmNIMEVZYkF4bXlBTHpOejBuMHBSVFpGMWFUSUJNUHNabnlW?=
 =?utf-8?B?REpWSXJEbm95OU1IQWJiYnN3ek1NOEQ2NzliSWVDZmtrTWZ2YUd5S253ZDhh?=
 =?utf-8?B?NjR2YmZCOVd5SDVsZWR2aXJIcTZDSG1ZTE1CTE82c3l5SUxzb1paZk1MclV1?=
 =?utf-8?B?TDJOS1ZKS212R0tyWnlBbG9ZRmtabmlCOUNRTHNQWEZJRk91eG53eGpGZDc0?=
 =?utf-8?B?bjZTNEd6dE9iUW1qaWxMMk9tL3JiMUZ1SDdtTTZMYjMxQUxQaGQ0UlFpYjN1?=
 =?utf-8?B?UVd3NzJEaC9mZTFuR1lHN2o2M0tRZjhiOXFXczlGQWpXVmpPYmVmRXl5Rkgr?=
 =?utf-8?B?bUtpVWRYQzU2Y2NPR1dGRnpmSU5yYjBmTlUvakZmaVIycHJnVE9GMWdQaWpm?=
 =?utf-8?B?R2RiSmxHd1hPcUw5emtZNEs5b0svNXcrd2VuWVN3MmpRejBQTVhzWnBYaTFj?=
 =?utf-8?B?QVJUSU9SYzlsSmV0MDBwSmJWZkJQaXg5TThlMEw1RFE5UG9NYUxyQ01lOHpF?=
 =?utf-8?B?Vk50L0FPSEgxdVdSNGxsLzh4U1pNN2hhMEZMYTY5N1RpMVJwa1BjWnRvazQx?=
 =?utf-8?B?VmgxbEdXblZ6T1YzR1B3NkovRmc0QmNudjVGeFA0dERsZkdXSHJTSXVYbHVD?=
 =?utf-8?B?YlY3S01NNzNEZHlYd0V1cXZEQXNNd2k4WUNTS1c4Zis4WEhlVGg5VlZUczN2?=
 =?utf-8?B?MVBWT1h6VWUzenRFcm1lQWk4VGlMMm1ZWnNDZm0rWDJBaWFRamloUkVlalpZ?=
 =?utf-8?B?R05TbjJSS0hJbUltdEtsbjQ1cnh4dFJ5QWFqRTY4NnZCSTdtcVhzSitidHgv?=
 =?utf-8?B?VFhORWl5eTJvaUlPelkydkFzUEZRNEl5SzFVVmc0WjZyVzk4R0NTdkVkNkVt?=
 =?utf-8?B?Z0xYT0tsRTBmMzBVdDNycWJUci8wdXBySDZja0RtRjhHcVp3cnE5VWFzS1pO?=
 =?utf-8?B?aXV0cG5WMmFJZDA1Ynp2ZGJiK1Qvd2VMaytFWTdrU2pYd1B6UlBkRXZZUzJQ?=
 =?utf-8?B?ekhGaGFKUmRscE5Kb0daQTJkc3hTNldkeEE0bGpPOXgwU1NQWnljRVp5WkVN?=
 =?utf-8?B?ajBRc3AxeENCbysxOTArK2pqRnNFeUd2aTZ3Y0dUL0xaSlU1NmhGRTRkN2Rw?=
 =?utf-8?B?YVd2dmZaMGF4Mkt6RFdQMGVHYmJETG9MMTVVdHJXTGV1N3Z3RllvTkVZbm4w?=
 =?utf-8?B?aXVGYXJ5d1I4cy9MS0FGNDZsZWUxYVJrYWVrTUhEamdrK25RQldxZlZqanFy?=
 =?utf-8?B?VTNkYnVFQUluV0ZRWnhqREp1QnVQNG5md2hNazVWdUIzZGNrK054eHZuUW02?=
 =?utf-8?B?ZnBhc0R4bWdDSHdKdDdmTFZ5RktRcVZYenArQlYzdEJIZlVneEcxcVhpZEgr?=
 =?utf-8?B?MmV1LytYcU1QTldYWlkwUXpGSVdrNkZjcnE1VkV3Q1JqT3lteU1Na2FDVmM4?=
 =?utf-8?B?RFJ0ZDRRcDFzYk52RnBPNDQxdWRqdDdIL2F5UDZyQkoxbElVUm5nWDRXVkdu?=
 =?utf-8?B?S3hDQjFyNmZVeUhJRkRCYVJYVUZwOUtCNDJqbGpTNFJZOHBOcmExWFRXKytp?=
 =?utf-8?B?bjAyT09Dc1lSSXhxaFlCRjE5RVAvcXp3V0ZHakpmRkhHU3lMSGs0VGxpVWRz?=
 =?utf-8?B?ZW9PRWIxTGpudjZxVnVzRzEvNVNVNElIQkE3Q01Gd1RJd2ZjWjVJRHBzempQ?=
 =?utf-8?B?NDBjNmE0VnJ2OHZ2VWFTd1FaeERaUDNkbEhwMi9icEFMUkVtWmJrQldmd2ow?=
 =?utf-8?B?elRnb3h0c2NzZzR0L2h4OFVlZnVWb25NOFNjNXcvZGFxdFdNMkEwajdnbmFV?=
 =?utf-8?B?enR6ajlLajdkc2I0QytRS1A5K253dkh6MDdkUTZBZStLc3VzK3kzS2k0N3VC?=
 =?utf-8?B?dWt6ajBOZkM3N0daeXBaa1JoTnU4TkgwUmhVa2tUZEt6UmRFR3BNaHVST0xP?=
 =?utf-8?B?d3NSSFVuMkhjVWJkUFhZenRXSFo2Vnh0VG9GbHpMSFhCY0ZKeE5jdURqb3Vj?=
 =?utf-8?B?MzA4WlNKaUZrTkU3Q0czR2pXVWxnMUFsMEV6b1NnaXlROW5nbGNsd1ViajhH?=
 =?utf-8?B?TkwzQmE3dVRBM3pJZVlEbjFVR29NdU0rT2dXWmJ6VFZ4ZWwrTG9MTk80STVU?=
 =?utf-8?B?T0E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0462b296-4500-424f-503c-08dab22d44e3
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2022 23:54:32.7397
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: vlyQeUTP40ERy6e4ytIzsvKrsqLnlZ4GPcXtV9gUnMBTY0WU58E8gbQdNO4ZhWSintQ6RRTJotA4AfE+5/e4lQbxxGvT4RfvyFPQ+XF35Ag=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB5341
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Peter,

On 10/19/2022 2:08 AM, Peter Newman wrote:
> Hi Reinette,
> 
> On Wed, Oct 12, 2022 at 7:23 PM Reinette Chatre
> <reinette.chatre@intel.com> wrote:
>> What if resctrl adds support to rdtgroup_kf_syscall_ops for
>> the .rename callback?
>>
>> It seems like doing so could enable users to do something like:
>> mv /sys/fs/resctrl/groupA/mon_groups/containerA /sys/fs/resctrl/groupB/mon_groups/
>>
>> Such a user request would trigger the "containerA" monitor group
>> to be moved to another control group. All tasks within it could be moved to
>> the new control group (their CLOSIDs are changed) while their RMIDs
>> remain intact.
> 
> I think this will be the best approach for us, since we need separate
> counters for every job. Unless you were planning to implement this very
> soon, I will prototype it for the container manager team to try out and
> submit patches for review if it works for them.

I do not have plans for work in this area.

It is still not clear to me how palatable this will be on Arm systems. 
This solution also involves changing the CLOSID/PARTID like your original
proposal and James highlighted that it would "mess up the bandwidth counters"
because of the way PARTID.PMG is used for monitoring. Perhaps even a new
PMG would need to be assigned during such a monitor group move. One requirement
for this RFD was to keep usage counts intact and from what I understand
this will not be possible on Arm systems.  There could be software mechanisms
to help reduce the noise during the transition. For example, some new limbo
mechanism that avoids re-assigning the old PARTID.PMG, while perhaps still
using the old PARTID.PMG to read usage counts for a while? Or would the
guidance just be that the counters will have some noise after the move?

Reinette