Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <81a7b4f6-fbb5-380e-532d-f2c1fc49b515@intel.com>
Date:   Mon, 10 Oct 2022 16:35:36 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.3.1
Subject: Re: [RFD] resctrl: reassigning a running container's CTRL_MON group
Content-Language: en-US
To:     Tony Luck <tony.luck@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>
CC:     Peter Newman <peternewman@google.com>,
        "Eranian, Stephane" <eranian@google.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        "James Morse" <james.morse@arm.com>,
        Babu Moger <Babu.Moger@amd.com>
References: <CALPaoCj-zav4x6H3ffXo_O+RFan8Qb-uLy-DdtkaQTfuxY4a0w@mail.gmail.com>
 <b2e020b1-f6b2-e236-a042-4eb2fd27d8b0@intel.com>
 <IA1PR11MB6097236CFF891041DBA42ECB9B5F9@IA1PR11MB6097.namprd11.prod.outlook.com>
 <Y0BhzKkksSjSeE3W@agluck-desk3.sc.intel.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <Y0BhzKkksSjSeE3W@agluck-desk3.sc.intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?LzVrR25DbzVxaXZrT0dMYWFReTNSd3JQaGZCN09BaTZWRnY5U0dSNmh6R2tR?=
 =?utf-8?B?VS9RSEJiU3FBS1JacU5NUEhkZFd4dGNVQmFIckVFRVNIRlc0VEZMaHRPQ3pa?=
 =?utf-8?B?WmNlYVBpa1VnVWNCbmxQeWJ4Y3gzYWRFSGovQmpUUTdzOUk3UHNQK0hjeW5U?=
 =?utf-8?B?YWlPVkFEMytRaFFhWk5UM2w5U0NEK2NmT1ovbTNFendIbkFQay84cGhEbmpT?=
 =?utf-8?B?M0NreFZMYkUyZjRXZGpGZ2JYdnNXWDNsWEdmcmEycXVSZ2F4S0FHZElWQkxL?=
 =?utf-8?B?bS9TUkVLSm1jS0o2Qm5nNzhPVW91citlbGJYT2lRYXd6ZENJZVl1R0pMMFRj?=
 =?utf-8?B?aU02RURoMGthNVdpOUdCem01VXdYNU94ckxOTkNwbDh3S2VZMmRIMzQ1MWlN?=
 =?utf-8?B?Q3diR0hHU1JUVE02eHBqVlhZTzFMK2p4ZDNKb1U0MG41bTRVaUlNWS9TN2xP?=
 =?utf-8?B?UXVXbVVtakttTUhnSWhpU3dMME1LMElNQkk0aVNtSEI1RUVROHdudnJHTDBC?=
 =?utf-8?B?eXl0ZGpxSnhEWHNjL01qN0RpSGhkZ0JLeS9pTFd3Sk13d1RWZFlQRGhvTlAx?=
 =?utf-8?B?ODBJdVJ6ZGEwN0xTZEFsSFBuTTIxU0RTcHRhcU1tRFk0VDgwZlhXRDc5MTBn?=
 =?utf-8?B?WDgwLzJsNThEaTM1c2xZc2p6b1htWkZsZnF6UFZ3WVlFZy9ybXJ3bGltN0Yy?=
 =?utf-8?B?QlJQb3gzTGJQaWtHUElvTHJ2dHZsOWo5UWlnb0Uzd2xrTXNSd20xMnZaWnZP?=
 =?utf-8?B?VzlJdEw4TXhROWxFUDVIUEQxekQ3blBFR2ppUm1wdE93YW54WU1jL24zUTJ1?=
 =?utf-8?B?OHJGcm5reC93VHFPMURTYXZNVkNSeTZLTE5MdFBDK1RrTkkvOHJhem1GUnRh?=
 =?utf-8?B?OTdkSWpJOXlGaHFRdlhYd2FRSk04MnpLYkJqS0VmRDNvNnFHb3lEZzB4Yk1I?=
 =?utf-8?B?SHcybS9HUXpIelFEcVE0aklMbTJrbndOVW9ua21XeG9IeWNVdjNrUmZkUVA1?=
 =?utf-8?B?UUdxWnlrSnRuOHh1eXZCQ3NRbWxDRnR2T2JOeDYzRm1sL1BHc0Y1bkhuUElm?=
 =?utf-8?B?NXJkS0phOGtTaGFoeGtJTzl0Q01CSFpyT2IwdGNNM1JRNzBQZFVSZ20xNEtq?=
 =?utf-8?B?RmRjMjZzeW1hMzBtbEVNUFJJMStxbHFib04reUtRUUZVajdqNTRic1BDVmFF?=
 =?utf-8?B?Vm8rWnRVdTJocmdnU3F1WnRJK25COEY2YVM0RGNoOGpnYm5WU3grQjlxNVpz?=
 =?utf-8?B?aTBFc1pMUzR2dXR4ZGoxNzAzWkJRRml0OFFOZFRkeStxSEFWVkwzcjZCSzRN?=
 =?utf-8?B?MzhDNlB1YXVDdmtyQ2ROcWZxOTdXQWJoZUhLV0g2QWxOa21UbnlpcDBUZEVv?=
 =?utf-8?B?aXNSUWZMdTRwMHhiN3k3RkhRVEl3dWdCTjR5anV3cCt3a2FFdXFPV3hXMEpq?=
 =?utf-8?B?YlBrUXVMbThNekJ0NVZIN1BkbytMRUQvazBMTDF5eWQ2U1JTc1BuazFicndG?=
 =?utf-8?B?c0dKWEtYS0t4WkZkU1pzR1dkMVoySkZ3ZFNQb3MvN1lNT09RaHJsUjZ0Ykxj?=
 =?utf-8?B?RjIydHNsZTNxblFDRlFmMmo1L3pYaXowdEZObzJsWjFibER3S1pFeHlqaGFU?=
 =?utf-8?B?bjRYUXBLWlFjRGhZOS9FcDJQL1ZlUWVtUWwzb3Myejh6aHQvVWRBZ3RFblBa?=
 =?utf-8?B?d0FlelQzU2FXNFdvUW9ZVHdMZ2cvb2huVWFvRlRZZUVXaTJFSHlaVkEwSDIz?=
 =?utf-8?B?ZDg0d0ozYjVKMjkyeXE5amlWSkVHUkJEcXlSSEIwZ1dRQWo3cWw1TGExeU1L?=
 =?utf-8?B?M2RRaU5UdWtzdEZzQW1nZWQvVUQrQi9vb0tYbFl6NVhmSUlQeEllVmZXTEcy?=
 =?utf-8?B?eGliRGZxdkFLM0Z6Y3N1RHU0NzRoeEw4WUhxOTEzUVUrY1hpRVJEZmtuTVJD?=
 =?utf-8?B?Q1cybmlQNHI1M0Z6ZW9idTV5L3pIZ2F6SGYrQWxDTnFvZW5QK0NEQkRkaGx6?=
 =?utf-8?B?WWhkTVgwUy9RblRSODh1RERxSVBQWjlRVUd6a3ZhYnNYRzVtY3VoUGxlNzEw?=
 =?utf-8?B?NFJtcXhHYkZ1M09HSDkrOWU5WGVRdDl6Tm50L1VwQ0JKaXRZMkppNTZoaDJN?=
 =?utf-8?B?UmozT0dxLzl5YktaYlFXRUdBUzVjN3hwUFNkVGpWUWVrSVlwMmhqR0w2aGFV?=
 =?utf-8?B?aFE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 6451372f-cc28-4eb7-1221-08daab182385
X-MS-Exchange-CrossTenant-AuthSource: DM5PR11MB1867.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Oct 2022 23:35:39.1865
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 9vQ77RfRLkYjLvGqvJgRQ+cpZqHz+BAHFUc69J3/YINEiJLt60EXBEhmqeMc8LD6t3doth2SkdhDUPazdeJ+5kbfEuQ8UWSsut8GWpfu2Jk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR11MB6126
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-6.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 10/7/2022 10:28 AM, Tony Luck wrote:
> On Fri, Oct 07, 2022 at 08:44:53AM -0700, Yu, Fenghua wrote:
>> Hi, Peter,
>>
>>> On 10/7/2022 3:39 AM, Peter Newman wrote:
> 
>>>> The CLOSID management rules would roughly be:
>>>>
>>>>  1. If an update would cause a CTRL_MON group's config to match that of
>>>>     an existing group, the CTRL_MON group's CLOSID should change to that
>>>>     of the existing group, where the definition of "match" is: all
>>>>     control values match in all domains for all resources, as well as
>>>>     the cpu masks matching.
> 
> So the micro steps are:
> 
> # mkdir newgroup
> 	# New groups are created with maximum resources. So this might
> 	# match the root/default group (if the root schemata had not
> 	# been edited) ... so you could re-use CLOSID=0 for this, or
> 	# perhaps allocate a new CLOSID
> # edit newgroup/schemata
> 	# if this update makes this schemata match some other group,
> 	# then update the CLOSID for this group to be same as the other
> 	# group.
>>>>
>>>>  2. If an update to a CTRL_MON group sharing a CLOSID with another group
>>>>     causes that group to no longer match any others, a new CLOSID must
>>>>     be allocated.
> 	# So you have reference counts for CLOSIDs for how many groups
> 	# share it. In above example the change to the schemata and
> 	# alloction of a new CLOSID would decrement the reference count
> 	# and free the old CLOSID if it goes to zero
>>>>
>>>>  3. An update to a CTRL_MON group using a non-shared CLOSID which
>>>>     continues to not match any others follows the current resctrl
>>>>     behavior.
> 	# An update to a CTRL_MON group that has a CLOSID reference
> 	# count > 1 would try to allocate a new CLOSID if the new
> 	# schemata doesn't match any other group. If all CLOSIDs are
> 	# already in use, the write(2) to the schemata file must fail
> 	# ... maybe -ENOSPC is the right error code?
> 
> Note that if the root/default CTRL_MON had been editted you might not be
> able to create a new group (even though you intend to make to match some
> existing group and share a CLOSID). Perhaps we could change existing
> semantics so that new groups copy the root group schemata instead of
> being maximally permissibe with all resources?
>>>>
>>>> Before I prepare any patches for review, I'm interested in any
>>>> comments or suggestions on the use case and solution.
>>>>
>>>> Are there simpler strategies for reassigning a running container's
>>>> tasks to a different CTRL_MON group that we should be considering first?
> 
> Do tasks in a container share a "process group"? If they do, then a
> simpler option would be some syntax to assign a group to a resctrl group
> (perhaps as a negative task-id? or with a "G" prefix??).
> 
> Or is there some other simple way to enumerate all the tasks in a
> container with some syntax that is convenient for both the user and the
> kernel? If there is, then add code to allow something like:
> 	# echo C{containername} > tasks
> and have the resctrl code move all tasks en masse.
> 
> Yet another option would be syntax to apply the move recursively to all
> descendents of the given task id.
> 
> 	# echo R{process-id} > tasks
> 
> I don't know how complex it would for the kernel to implement this. Or
> whether it would meet Google's needs.
> 

How about moving monitor groups from one control group to another?

Based on the initial description I got the impression that there is 
already a monitor group for every container. (Please correct me if I am
wrong). If this is the case then it may be possible to create an interface
that could move an entire monitor group to another control group. This would
keep the benefit of usage counts remaining intact, tasks get a new closid, but
keep their rmid. There would be no need for the user to specify process-ids.

Reinette