Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <7b09fb62-e61a-65b9-a71e-ab725f527ded@intel.com>
Date:   Wed, 12 Oct 2022 10:23:16 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.3.2
Subject: Re: [RFD] resctrl: reassigning a running container's CTRL_MON group
Content-Language: en-US
To:     Peter Newman <peternewman@google.com>
CC:     Tony Luck <tony.luck@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "Eranian, Stephane" <eranian@google.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        "James Morse" <james.morse@arm.com>,
        Babu Moger <Babu.Moger@amd.com>,
        Gaurang Upasani <gupasani@google.com>
References: <CALPaoCj-zav4x6H3ffXo_O+RFan8Qb-uLy-DdtkaQTfuxY4a0w@mail.gmail.com>
 <b2e020b1-f6b2-e236-a042-4eb2fd27d8b0@intel.com>
 <IA1PR11MB6097236CFF891041DBA42ECB9B5F9@IA1PR11MB6097.namprd11.prod.outlook.com>
 <Y0BhzKkksSjSeE3W@agluck-desk3.sc.intel.com>
 <81a7b4f6-fbb5-380e-532d-f2c1fc49b515@intel.com>
 <CALPaoCjdeRjyX5L6BBX688ZM21eMwetuL9QLF1+GEDUskGcU2w@mail.gmail.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <CALPaoCjdeRjyX5L6BBX688ZM21eMwetuL9QLF1+GEDUskGcU2w@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UytWMUJXaWdyRWVtMkJ2eTJVdXVOemwzb2xueGYwSVdJRnJMd2dlS2s5TWhP?=
 =?utf-8?B?T1hSeDBpM09TcjhxVHRYSG1sam1Dc2Q1WEtPRlVTSU1LcEdGRDZmRUJKYVd2?=
 =?utf-8?B?WUdTVDIvZ2F2QXF6cTFKSTdhK0hSQUFCSW1JM2xieG1OdmdUMlhiYStsRjBq?=
 =?utf-8?B?RU5TVjlFNnFGdWRKMForVXJJWHZDak5IZE5heG83N1ZVQXBlNXNHS3M2ejZa?=
 =?utf-8?B?dE94bWQ0MGd0bHRMRENJYVRZT2Y5Q1B0T2l3WUFQVlhqcVpCZktMQ2FySWxq?=
 =?utf-8?B?RXFuWXErQUVuNUtQWERKc2JEblBFMW41R1B6NGE1NUtRQmI2c0plZ2J4SjJh?=
 =?utf-8?B?Z2ZPa3owWnFkaTdzUTR4SGVLYS9qUkhqS2FaRDdyMGJVd0tmK0tZZUhFMnA5?=
 =?utf-8?B?V3VPV1ZFeW9XMUpYbXVubUlrWVhmT1A0Qmo1NkhlNk55bndFV2s5K01sZ2Q0?=
 =?utf-8?B?L0hGZ1RObER3OXl2U2dEdkduVHRWSisrU1VOaDFyNkEybHBDbUhWNUFFbmV2?=
 =?utf-8?B?cmlrV3BpWldFdlNrZElxRTBsZFNzUWtQblVpKzFCZnZjbHpoZjRtVXVxdUM3?=
 =?utf-8?B?dlR2VTV2L1dXTTUrNXBmQnZmNEtZeEdrUmtJajMrdXFLWXBHVWIyMkY4bHpa?=
 =?utf-8?B?OFJSbWZMMDdhekI2a1V6T001RVkraXB3QWMzYVMxUjBrSGFuWkRhSTBab1BJ?=
 =?utf-8?B?eXNhMjE0bjRMSDNWT01WeWdCMENYcTAvMG94SEk3WHdvWHl6cmtuVERacW1Y?=
 =?utf-8?B?SU9pcmIvVjhvV1FpbllQdWRwVHN5U3FXYzRpcEZHUUxjem1iMWZzQXBhUUVv?=
 =?utf-8?B?bzJsa0hvc0h1MTBZNEFkc3pYbmlGTzU3ZFc5akhhc0NJVlZUazNScDhNWTYr?=
 =?utf-8?B?eWdqak9OdjhjWWFxYmw4Z1phYnc2U3JLVGxVdGJSL2IwUlo1VWliQklDOGZY?=
 =?utf-8?B?ZTJJUllTMnV0TkN3bWxmQS9HM0s5NXBoM0RTNkFpcTduOExDbnh1K2VzTU14?=
 =?utf-8?B?aGphTWFQbzVNb2FiUjBjNTNnUXB1WHpueDhVRW9qL2poS3orTkh2WFUybnRW?=
 =?utf-8?B?cE9KblZvWlJkVDZzZzNveHRET2Yxc21GbE54eTBtR0JkcGlOOWpTN2hiVWxF?=
 =?utf-8?B?UTlyd0tXeGMvMXA3SitQNjZVUm9TNis4dVpZcEh4bFg1V3NTWERoWGFwa2hl?=
 =?utf-8?B?UjZCUnlWSnVGbEppSUZyQVRhMURzMzVZWXdLbFZXaGl3R2pHcHEzSEVISC96?=
 =?utf-8?B?bnpLclpvcDcwc2NSZG4vdEp0eWZhUEZ6ejZxY1JNdkl3bVZoQ0RSWlJpSmdx?=
 =?utf-8?B?ZVdKbHY4bzV5T2lEdTQ1cEp1S2VHV1lOK1d6bUtiSTVjSDdPOXFMTnpYUHFs?=
 =?utf-8?B?WFQzK3k0QkxNa3paRE1CYk52ZkpMK2h6bWhIOXF0UzBROUJOMVNaL2FEUHli?=
 =?utf-8?B?dXRXK0JKOWFTdjk0RmdtditEOVZXSk1qK1IvTy9pY3FkS0J0TXN3elV1dWIv?=
 =?utf-8?B?QURHcDBkOFZSNm5zOGhiQkMwT1prQzZlcFNZUHU1ZTh6V2svdWhrajZTRE42?=
 =?utf-8?B?Vk9aQytUbEtPT0pIeWJXZUcwVkMyYkJON0NmZ1VobWJWcVJRRTFsT0FIS0t6?=
 =?utf-8?B?ODBnOFdZNnd2NExkRjVSK3hpZDNjdEtwWkIrcHFyMG16RzlpVlhPWTNPUjRG?=
 =?utf-8?B?UkVNUDRjUzNKRHBzVWNpVlhTZS9XNzhjYkVPZE9GWXFCR0xwWHNaWTk5Nzc3?=
 =?utf-8?B?YXJtTitmN0czWlhuZzlYYzZNaFlQclJnQ0FZREttMHE4M0ROUE1vRGRqS2RJ?=
 =?utf-8?B?bWlvdy94bU5yQm1xaDJsNFFNd1JiL0cyU0s1eXlFc2ppeFRPN0RvakpNUUJv?=
 =?utf-8?B?NkkrdWU3dWRKSEVHWVo0a3U1WWIyUkVoczZ2RXZyV3FnWnFWcDQwaThzM3Vq?=
 =?utf-8?B?MlI3bnJPN0twV0Y2eVMxS1hlMlNZMXZBTVJJRGdkNWFoQnpncEZRZFdvLzFH?=
 =?utf-8?B?aFgxMVFBV2VSNytmWTExOFpHdTI4VTBrQ0UzM2J4REhvVXNzZUE4MVNXZ09m?=
 =?utf-8?B?QjIzS2VJcVBId25SMWFYc0orbXhwandaZ2JZWlU5ams0WTRRV1JpWG9QUEU1?=
 =?utf-8?B?WmYybWVxTkJkN3JqKytSa1BRUnhQTmU3bXVmVEdXYmZtdi9hSkJvSXdvbjlG?=
 =?utf-8?B?NlE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 7421e545-1807-478f-8f4b-08daac7674dc
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Oct 2022 17:23:19.4795
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JpoU2AOYVZWb6EK+gkQUmwinoRwH8xMteFAwhJ4VYIIag0+Ao+Ucc1q9vpvmAssfZ8FaebtQKc7oI7GsF/dC15oECCv83ki3wh7DfjnJo08=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR11MB7058
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.9 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Peter,

On 10/12/2022 4:21 AM, Peter Newman wrote:
> [Adding Gaurang to CC]
> 
> On Tue, Oct 11, 2022 at 1:35 AM Reinette Chatre
> <reinette.chatre@intel.com> wrote:
>>
>> On 10/7/2022 10:28 AM, Tony Luck wrote:
>>> I don't know how complex it would for the kernel to implement this. Or
>>> whether it would meet Google's needs.
>>>
>>
>> How about moving monitor groups from one control group to another?
>>
>> Based on the initial description I got the impression that there is
>> already a monitor group for every container. (Please correct me if I am
>> wrong). If this is the case then it may be possible to create an interface
>> that could move an entire monitor group to another control group. This would
>> keep the benefit of usage counts remaining intact, tasks get a new closid, but
>> keep their rmid. There would be no need for the user to specify process-ids.
> 
> Yes, Stephane also pointed out the importance of maintaining RMID assignments
> as well and I don't believe I put enough emphasis on it during my
> original email.
> 
> We need to maintain accurate memory bandwidth usage counts on all
> containers, so it's important to be able to maintain an RMID assignment
> and its event counts across a CoS downgrade. The solutions Tony
> suggested do solve the races in moving the tasks, but the container
> would need to temporarily join the default MON group in the new CTRL_MON
> group before it can be moved to its replacement MON group.
> 
> Being able to re-parent a MON group would allow us to change the CLOSID
> independently of the RMID in a container and would address the issue.

What if resctrl adds support to rdtgroup_kf_syscall_ops for
the .rename callback?

It seems like doing so could enable users to do something like:
mv /sys/fs/resctrl/groupA/mon_groups/containerA /sys/fs/resctrl/groupB/mon_groups/

Such a user request would trigger the "containerA" monitor group
to be moved to another control group. All tasks within it could be moved to
the new control group (their CLOSIDs are changed) while their RMIDs
remain intact.

I just read James's response and I do not know how this could be made to
work with the Arm monitoring when it arrives. Potentially there
could be an architecture specific "move monitor group" call.

> The only other point I can think of to differentiate it from the
> automatic CLOSID management solution is whether the 1:1 CTRL_MON:CLOSID
> approach will become too limiting going forward. For example, if there
> are configurations where one resource has far fewer CLOSIDs than others
> and we want to start assigning CLOSIDs on-demand, per-resource to avoid
> wasting other resources' available CLOSID spaces. If we can foresee this
> becoming a concern, then automatic CLOSID management would be
> inevitable.

I think Fenghua answered this well.

Reinette