Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <b2e020b1-f6b2-e236-a042-4eb2fd27d8b0@intel.com>
Date:   Fri, 7 Oct 2022 08:36:37 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.3.1
Subject: Re: [RFD] resctrl: reassigning a running container's CTRL_MON group
Content-Language: en-US
To:     Peter Newman <peternewman@google.com>,
        Fenghua Yu <fenghua.yu@intel.com>
CC:     Stephane Eranian <eranian@google.com>,
        <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        James Morse <james.morse@arm.com>,
        Babu Moger <Babu.Moger@amd.com>,
        "Luck, Tony" <tony.luck@intel.com>
References: <CALPaoCj-zav4x6H3ffXo_O+RFan8Qb-uLy-DdtkaQTfuxY4a0w@mail.gmail.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <CALPaoCj-zav4x6H3ffXo_O+RFan8Qb-uLy-DdtkaQTfuxY4a0w@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?T2o3QlQ3VU1lM1krc0FZb3hGbzFYdTlBM05wK1ZvUWJXSGUxNVk3ejRsV0tN?=
 =?utf-8?B?WFMzdWpkZ1BQWXNiT2pQK0dRRXR5aldVYW5rVjNzUjhYV2RyY212WG05UTgr?=
 =?utf-8?B?bm5VQUNUbklFUElpU3JIRHNnSkFIeXk3dDE3U3Q2b3RWQmFkL01ENUpiTnRG?=
 =?utf-8?B?U3c3emR3Ky85ZkdJcTQrZFpORkE0YWNtb2R2a0o2QUpsZWh0T1UwVmVvRlMw?=
 =?utf-8?B?VG11Mys2eWdzM0gwM0ZEVTR0Z2Vkb1FGMmlqRmRCY0JMSkYwejQwWDNUbHhV?=
 =?utf-8?B?SGJxNEZnaXdMTlZoUVhTZVcwRVc4ZFN0MnNIVmdIcVdzTU9PYStva3htKzlt?=
 =?utf-8?B?RTdqcC9UUi9NZ0hxRWJuMFoxUElSZ3lyWVVtRmZ4cG1YbTJyMFFUa0lnMWlM?=
 =?utf-8?B?Tm1aNjlibzNsck05SjlWalRGc2RrTFVMVSt5MHFLUDU5anF1amx4OFQ4eFZo?=
 =?utf-8?B?ODRYK2QxaGFVWkU1TWIvYlhua0lTS21aWFpQZHAzSWptZHdpaERaU0hlZnVi?=
 =?utf-8?B?dXlMK3RhdkpnUFpvN01PaVA5YWlMSVJZRk0xSlB6VWxnWmpGKzFHTmJrZWZw?=
 =?utf-8?B?M2JrclpVR0ozb2Z1VkMrWkZXUlB0aTZCejVRNjFmWjBpeTM0RitaRzJoaENB?=
 =?utf-8?B?ZlJsbS9KN3daV1ZxQWI0SjdXQi9pNXU5OE13T2xsdnBXdGlFRWsyalVTbkxo?=
 =?utf-8?B?Y1BQZEJYOTdqeHpZOUs3bEFtMTRYbU40R1hXYjhNN1RZMk9kbGtwdDA0c3l5?=
 =?utf-8?B?Wk5Fbmc5aE9YcFVVMEVBRFpXYmhhajV4TlVDeG03UU5tSXhzOVNOQUlyUTZq?=
 =?utf-8?B?dnk2a1dEN2RKSytaejgvYjdZSEpmZWxhK25CbGo3LzY4aExKeHQ5WE1vWHQw?=
 =?utf-8?B?b0hvWUtCTzNwSXd2SFNLY0JURVNtTnNyR2hmaGRBdmRQL3ZRaTZhZUdlYzdN?=
 =?utf-8?B?Q1U4QWl1V1JOSjlWNVpJaFQrQ3o3dTBOOXEyUzJsWDJ0STZnOXIzRjM2NzdY?=
 =?utf-8?B?b3I1SVd2L3BTdGxiK2x6dzlNL3dtdzM0YStXTThVK0xPS1FWdEk1SGtVMnhh?=
 =?utf-8?B?UWpuTmd3Ryt4djhRUnZVeTk1WUF5dGE2SFIvNXhhbHhIQlQ5UnZEcUV6ZGk2?=
 =?utf-8?B?TEpsMGVkY1pCSW5zL2UvS2RueWR0aTdva3NBT2k0OUt3eDJXNWJzbFJkODlZ?=
 =?utf-8?B?MXA0bHhRYStSdFU4ZSt1blE4OEdpQzYxTCtZVFgvcWQyZXdPSkNqZ2VvZmdX?=
 =?utf-8?B?cmtLSVQwSmk5ekg4eURpN3hTZjRRN3doSW4zeWMwdXlpOW9oQlcvWTJIVlBw?=
 =?utf-8?B?bVh3MC90OHJIcGhmVllybUhkVDNxMDJvNEtFdTdNREt4NUgxSjk3VTRRNCtX?=
 =?utf-8?B?UjQ3MGhUNE01UFhibE1COUVIa2NvNkJtaFBnZ3h5QzhUU3JPTTZyb3daYitX?=
 =?utf-8?B?dFE2Y2pnVDhndG5rL1JFcHBIM3pEVGg3VU5KQWRoVGhmQjBzUitKL0MzVHlW?=
 =?utf-8?B?RENKRGt2bWd4UThjOHI4TGhKZ1NNeTFHMnRsa3p0K21SK1Z1TFU4YnVyYnc5?=
 =?utf-8?B?UGJxNU8xVmptajlxdnVabGlERHczYWIrbElmV0hheVNlc05nMUJST2hQMFNQ?=
 =?utf-8?B?dkRlOEpiTDRCTnA4ajJpWUJ5eHdVMFpoWGZoRTlwbXZhUW40RDR4THlxM0Ju?=
 =?utf-8?B?TGh2N3lEUGQwVnlPRjYvY05CZ3E0QkJscnBjdzVZT1RwbWczRGxPWElxSFFI?=
 =?utf-8?B?L1RObGtWcjZvaks2ays5RkxDWkFqNEFzNlJXY0R2ZS9xUG5XTE5QUUN2cUNu?=
 =?utf-8?B?MnZLc3BCYTZiNkV5VWE5SzY1Z1N1NVJhUldoUWQ2T21ncDBnV21hSTgxZU83?=
 =?utf-8?B?aXlUT2pPaU1UUVhRa3F3T2pycHFBa1R5NTVsZkFuek9UdXlvZWVKUzNUU2VM?=
 =?utf-8?B?VGVqL2FyTEZ0T3hOT1U3WU1lWk42TlJqOU5UbGtwUjVUZ0cyQmE4TTFrUTFj?=
 =?utf-8?B?V3BlQTc5ajREcWdHeXJaVG9qNHhhK1pPeWJxT1ErUmZpdkliOFRKVUZGL1ZB?=
 =?utf-8?B?NldNMUdiYWtYbEE0Qmh1bFZSdjJ6T0JhVlp6S3lGRDdvMU1KUXlKLzRNQnJn?=
 =?utf-8?B?ck15alppdnc3VjUzendia1VLUzdwWXRLcVU4WVlwOUFoYjdyRzB3dk5Wai9i?=
 =?utf-8?B?Snc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 7b45de0f-c6c5-4912-0384-08daa879ba4a
X-MS-Exchange-CrossTenant-AuthSource: DM5PR11MB1867.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2022 15:36:39.8299
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FDNsgBjoUnHRqU6CgRKAj47UFOjQU0Poo4YoR3E0Qte3KNXGqEIB2wC9n1iSHmVYkutSH3YNO4fAGzv7E59YGQZqxWV0JlBMezPcH9UEO6w=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR11MB5907
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

+Tony

On 10/7/2022 3:39 AM, Peter Newman wrote:
> Hi Reinette, Fenghua,
> 
> I'd like to talk about the tasks file interface in CTRL_MON and MON
> groups.
> 
> For some background, we are using the memory-bandwidth monitoring and
> allocation features of resctrl to maintain QoS on external memory
> bandwidth for latency-sensitive containers to help enable batch
> containers to use up leftover CPU/memory resources on a machine.  We
> also monitor the external memory bandwidth usage of all hosted
> containers to identify ones which are misusing their latency-sensitive
> CoS assignment and downgrade them to the batch CoS.
> 
> The trouble is, container manager developers working with the tasks
> interface have complained that it's not usable for them because it takes
> many (or an unbounded number of) passes to move all tasks from a
> container over, as the list is always changing.
> 
> Our solution for them is to remove the need for moving tasks between
> CTRL_MON groups. Because we are mainly using MB throttling to implement
> QoS, we only need two classes of service. Therefore we've modified
> resctrl to reuse existing CLOSIDs for CTRL_MON groups with identical
> configurations, allowing us to create a CTRL_MON group for every
> container. Instead of moving the tasks over, we only need to update
> their CTRL_MON group's schemata. Another benefit for us is that we do
> not need to also move all of the tasks over to a new monitoring group in
> the batch CTRL_MON group, and the usage counts remain intact.
> 
> The CLOSID management rules would roughly be:
> 
>  1. If an update would cause a CTRL_MON group's config to match that of
>     an existing group, the CTRL_MON group's CLOSID should change to that
>     of the existing group, where the definition of "match" is: all
>     control values match in all domains for all resources, as well as
>     the cpu masks matching.
> 
>  2. If an update to a CTRL_MON group sharing a CLOSID with another group
>     causes that group to no longer match any others, a new CLOSID must
>     be allocated.
> 
>  3. An update to a CTRL_MON group using a non-shared CLOSID which
>     continues to not match any others follows the current resctrl
>     behavior.
> 
> Before I prepare any patches for review, I'm interested in any comments
> or suggestions on the use case and solution.
> 
> Are there simpler strategies for reassigning a running container's tasks
> to a different CTRL_MON group that we should be considering first?
> 
> Any concerns about the CLOSID-reusing behavior? The hope is existing
> users who aren't creating identically-configured CTRL_MON groups would
> be minimally impacted. Would it help if the proposed behavior were
> opt-in at mount-time?
> 
> Thanks!
> -Peter