Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <da89ea90-8453-f848-38d1-a14195faa95e@intel.com>
Date:   Thu, 20 Oct 2022 12:08:43 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.3.3
Subject: Re: [RFD] resctrl: reassigning a running container's CTRL_MON group
Content-Language: en-US
To:     Peter Newman <peternewman@google.com>
CC:     Tony Luck <tony.luck@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "Eranian, Stephane" <eranian@google.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        "James Morse" <james.morse@arm.com>,
        Babu Moger <Babu.Moger@amd.com>,
        Gaurang Upasani <gupasani@google.com>
References: <CALPaoCj-zav4x6H3ffXo_O+RFan8Qb-uLy-DdtkaQTfuxY4a0w@mail.gmail.com>
 <b2e020b1-f6b2-e236-a042-4eb2fd27d8b0@intel.com>
 <IA1PR11MB6097236CFF891041DBA42ECB9B5F9@IA1PR11MB6097.namprd11.prod.outlook.com>
 <Y0BhzKkksSjSeE3W@agluck-desk3.sc.intel.com>
 <81a7b4f6-fbb5-380e-532d-f2c1fc49b515@intel.com>
 <CALPaoCjdeRjyX5L6BBX688ZM21eMwetuL9QLF1+GEDUskGcU2w@mail.gmail.com>
 <7b09fb62-e61a-65b9-a71e-ab725f527ded@intel.com>
 <CALPaoCg4zrODVoXF2y2b+LRYq_+jnV8yv5qB+T_3Z264cV82GQ@mail.gmail.com>
 <b931062a-1a70-2331-03cc-6bfa69a51bee@intel.com>
 <CALPaoCj8ps2YRH5T1eCRY-8MLwGi25cfbd0KNtjYXTq5baAXOQ@mail.gmail.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <CALPaoCj8ps2YRH5T1eCRY-8MLwGi25cfbd0KNtjYXTq5baAXOQ@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QTJlUGJaQ0NNNUZPdlRmS1BITVMzbXp3aDVZM0FuOTBQYXJYaGZFQVQ5VUJp?=
 =?utf-8?B?MjB0M25EUTFWNm04QUdicGRDeHVMOXVOY1lEZHVLR2xwa1cvUzFPU0dvM0dN?=
 =?utf-8?B?dGJJY2RsZ0E5U2xBYmpBbDl1UnFsa0d0Q0VteGt3ZU9STVEwbjd3cjd4LzZ6?=
 =?utf-8?B?WFVWbHBwMmp2NU5WNlNaRnhSdXFLMDg4bmd6dXEwV051aW1RZDJIUnJSRVdw?=
 =?utf-8?B?SEhXeW1oV1VYeGszY2hkOEF6Zk5ZcnUrdW05ekFHTXA1bThaRjNkWkQycGli?=
 =?utf-8?B?SFRka29GKytNNGJiSk9VR3hCNS8raFNsS3BzcGVkQXk0RWU2OERJUWs0bVp4?=
 =?utf-8?B?dExlbkRaL3RTS2wvK1pYeWdqUUJLdGNoTnlOZlZRRGFReHhENURFM2IyUDBI?=
 =?utf-8?B?eGFFZ08rOEtzT3pHZ1hCdDR5V0hGT05UUUhzS3NlcGpSTWRZejgwSVVhVEJh?=
 =?utf-8?B?bEt4a0djckdBYk8rWitZTVErL2tDd016SGtEYW5MQ2VUajkvZ2JjMHdqaEox?=
 =?utf-8?B?SlBweThza1ZwMzc5aTRLSVVteHJDOE1NMXlqVG9leFNuYytFV0FkbUpCSlJE?=
 =?utf-8?B?S0pSOEtUbFhLWTIrREJ1VVVaNjQ4TWhhMWx4NVRPSFpJcXVESGRZSGFjNlVS?=
 =?utf-8?B?alQrTmpkZXRUMUVMaU1nazNPa0w1aXdaNStYeGFrQUF4NTh3WWFGcHhPbE5I?=
 =?utf-8?B?TE5ib0ZvazJYRHdSa21Ucm90aGdnT3hWUkV6SFAvNkhWTTJVUXdXbEMyakEv?=
 =?utf-8?B?dzJHQlBrMThWWm1GK3FDbjBNb01MaTRIZUEvR0tyWkQzQlNhVWJkQ2dLbUxD?=
 =?utf-8?B?SmhSOVhxYmNzVk1CaDRKRVVPSjUyQ2ZtWmlwSVFJVnB2S1h1Snk3YUxNTG9B?=
 =?utf-8?B?ME5ob2NBekdFYjZzdDRQamxPRHlTNnlWYzN1eDNUME90bnpqR0txZnp0N1NP?=
 =?utf-8?B?ZWYwZWU5TzlQakF2anRZSUZpWHArZjBHOFZoOUZMV0lIS0IySy95OEM0d1cr?=
 =?utf-8?B?c2dPOW80SU1jS3pUdTNXUFB3Y0tlVnIzVHNlcDNjUE5nbFZxcEM5Sm9aV3Zj?=
 =?utf-8?B?WjB5SnFoK2VPOVFvQ1NaNjNseXhlQmFLMENLM2FGZDVsc0pDdTEyempaaUZU?=
 =?utf-8?B?c0YweVJXZkpmTGRMUE9BS1pKNG1VaFNISnhVQW1TdFJ2MWIyWFZxUHBCQ2Ri?=
 =?utf-8?B?NUoxQ3NVb01rTXFmaFJoVU1Qa3kwRFpwUFZjT0Ryc3JLaTRVcDR1dERLMjVM?=
 =?utf-8?B?UzV5dEJUQUlzTUpwV3B2eU5wbEV0ZkY3QUcrUnZhekJwRHY0VUVvQjd4RTNw?=
 =?utf-8?B?eDZrb0hxKzVWWkViOTh2YUw5NldYYU1XQ0dlZG5SREROZDdVNzFPUFpIcWVw?=
 =?utf-8?B?ODVUd1RXdXFTSllMa29DQkdGR2dUWXltaFhHRWp4Z21nL2c2bDhvcllPV3pN?=
 =?utf-8?B?K29uek5hTGlXWVorRGxKcnZHQk5QazJ1WGE3eERDajVsNUttblY3Q2ZsTWtq?=
 =?utf-8?B?dE00UWlMcHZqZElBdHNPeVpwU0RpbGp4OW93VnBPZnVOMFFQREl5Ym5HUzlV?=
 =?utf-8?B?SEw0bjgwT3Y2MnRYb0x2dStwZkd4WkplbUdlcWprTUxNS1A5R2xIZi92d0Rv?=
 =?utf-8?B?QzdKbzhDc1BTZkJZMjBuU1U0aVRHQ21VNi83U2ZiYlN5L1YvdllpVFFWcWNK?=
 =?utf-8?B?ZFB1WEY0SWZvM2paTDBTTXVsYk9MaEtGS1FFelVDNFROUE5TKzJTdTZ3eXVJ?=
 =?utf-8?B?WDl5RFB2SmgxT1FtdW9nQTN5M09hdng2THFLWDVOTkFIWkxPUXFZbzFlcU5P?=
 =?utf-8?B?QmIwNHNOZ3FuS0F3UlR4RjZUTElja3RaczdnRjJvcFlTMnNmL2k0WWZldnZY?=
 =?utf-8?B?U3hHWTFLeDR4dDlEbnJoeDczakhnVFA0K0NhQXEyNW5ZeDZLaHBMYnB5NzVU?=
 =?utf-8?B?VDZSZXFvdkhyY3p0Ti9HK1FHMzVzTEJNQUpoQmZjSWU5c1dBNXZudnpwUTN4?=
 =?utf-8?B?NktLM3NNblR6QnBhKyt2aFpiMHNMcC81SXZMSVVnSFp3M2R0akw2UDNYeGsx?=
 =?utf-8?B?Q29Jbm1Cb2ZmU0x1WXd3bFdLY04zTnVpZFg3bVpaZGYyM2lOT014Z0t0SGYw?=
 =?utf-8?B?dTBPNXhOTkRQNzBNWjc4WTdtYkhYcFY3b0Y0S1dMZzlvV3NCZUk1ZXlPaGJa?=
 =?utf-8?B?RFE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 60512188-80ba-4065-a7ee-08dab2ce82f7
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Oct 2022 19:08:45.8496
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Wr9quJWU7B/89F9qqbOr7cWEisIKAAWbXqNGhLBJ5V/gvPkj/zBS56IkSS+4brvStZgeMD/HdTTgJZ/TgugZwSr4LvFJmZd0a4ZSZ1i5sDc=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB6075
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_NONE,URIBL_BLOCKED autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 10/20/2022 1:48 AM, Peter Newman wrote:
> On Thu, Oct 20, 2022 at 1:54 AM Reinette Chatre
> <reinette.chatre@intel.com> wrote:
>> It is still not clear to me how palatable this will be on Arm systems.
>> This solution also involves changing the CLOSID/PARTID like your original
>> proposal and James highlighted that it would "mess up the bandwidth counters"
>> because of the way PARTID.PMG is used for monitoring. Perhaps even a new
>> PMG would need to be assigned during such a monitor group move. One requirement
>> for this RFD was to keep usage counts intact and from what I understand
>> this will not be possible on Arm systems.  There could be software mechanisms
>> to help reduce the noise during the transition. For example, some new limbo
>> mechanism that avoids re-assigning the old PARTID.PMG, while perhaps still
>> using the old PARTID.PMG to read usage counts for a while? Or would the
>> guidance just be that the counters will have some noise after the move?
> 
> I'm going to have to follow up on the details of this in James's thread.
> It sounded like we probably won't be able to create enough mon_groups
> under a single control group for the rename feature to even be useful.
> Rather, we expect the PARTID counts to be so much larger than the PMG
> counts that creating more mon_groups to reduce the number of control
> groups wouldn't make sense.
> 
> At least in our use case, we're literally creating "classes of service"
> to prioritize memory traffic, so we want a small number of control
> groups to represent the small number of priority levels, but enough
> RMIDs to count every job's traffic independently. For MPAM to support
> this MBM/MBA use case in exactly this fashion, we'd have to develop the
> monitors-not-matching-on-PARTID use case better in the MPAM
> architecture. But before putting much effort into that, I'd want to know
> if there's any payoff beyond being able to use resctrl the same way on
> both implementations.

If the expectation is that PARTID counts are very high then how about
a solution where multiple PARTIDs are associated with the same CTRL_MON group?
A CTRL_MON group presents a resource allocation to user space, CLOSIDs/PARTIDs
are not exposed. So using multiple PARTIDs for a resource group (all with the
same allocation) seems conceptually ok to me. (Please note, I did not do an
audit to see if there are any hidden assumption or look into lifting required
to support his.)

So, if a user moves a MON group to a new CTRL_MON group, if there are no
PARTID.PMG available in the destination CTRL_MON group to support the move
then one of the free PARTID can be used, automatically assigned with the
allocation of the destination CTRL_MON, and a new monitor group created using
the new PMG range brought with the new PARTID.

There may also be a way to guide resctrl to do something like this (use
available PARTID) when a user creates a new MON group. This may be a way
to address the earlier concern of how applications can decide to create
lots of MON groups vs CTRL_MON groups.

Reinette