Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <09029c7a-489a-7054-1ab5-01fa879fb42f@intel.com>
Date:   Fri, 21 Oct 2022 13:09:25 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.3.3
Subject: Re: [RFD] resctrl: reassigning a running container's CTRL_MON group
Content-Language: en-US
To:     James Morse <james.morse@arm.com>,
        Peter Newman <peternewman@google.com>
CC:     Tony Luck <tony.luck@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "Eranian, Stephane" <eranian@google.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        "Babu Moger" <Babu.Moger@amd.com>,
        Gaurang Upasani <gupasani@google.com>
References: <CALPaoCj-zav4x6H3ffXo_O+RFan8Qb-uLy-DdtkaQTfuxY4a0w@mail.gmail.com>
 <b2e020b1-f6b2-e236-a042-4eb2fd27d8b0@intel.com>
 <IA1PR11MB6097236CFF891041DBA42ECB9B5F9@IA1PR11MB6097.namprd11.prod.outlook.com>
 <Y0BhzKkksSjSeE3W@agluck-desk3.sc.intel.com>
 <81a7b4f6-fbb5-380e-532d-f2c1fc49b515@intel.com>
 <CALPaoCjdeRjyX5L6BBX688ZM21eMwetuL9QLF1+GEDUskGcU2w@mail.gmail.com>
 <76bb4dc9-ab7c-4cb6-d1bf-26436c88c6e2@arm.com>
 <CALPaoCiKUQC+LxDwKQ0gE5AQniJi_nbzrXi_HA9ZBRtiXdw_dg@mail.gmail.com>
 <835d769b-3662-7be5-dcdd-804cb1f3999a@arm.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <835d769b-3662-7be5-dcdd-804cb1f3999a@arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TFEvaHpDSVJlQThobW9OQUpobnVrT3FadjF5UmNoY2g5cGk5SGtrSzBUeVFo?=
 =?utf-8?B?eG9tS1VPbXJEa0tzWHh3VDIwNmxSSGI3bGJ4UjVPRUtXMXNXbVZ5bHRzdHNI?=
 =?utf-8?B?SzMxSHZwbC9Zc3g3c1p2RFdGRm8zWHg3UXNPZy9PZjdtN3V4bUZEOENUdE1l?=
 =?utf-8?B?TmRiU2tNeEt6TTVMVjJHNzBYbGNja3NNbjk3aHJpbm1QbEJPMzhQcTlzc21n?=
 =?utf-8?B?UHhMRTR3b0pwTDBjOFIzeGttTXo2VlNOclNjalVEY2NVUmRJd2xITkhpa0RU?=
 =?utf-8?B?WGcxYUNMUktQd2hxcmxlVkdQL0hpLzhRb0FzbFhPRmJLZzI4dG1ZTGd1b0wr?=
 =?utf-8?B?WXh1Ly9GdlBJVGZIc3ZEMHVZK0VJMFZjbks0S0o1VGp4dG93R1NLOWRVYVJF?=
 =?utf-8?B?RkZKb3VwenF1RFVvUEtPRWpxMFRCblFnYnd4dGI0cXVhR2NIUkIwQTBDdGJN?=
 =?utf-8?B?U3R2SXlOTVY5OGE3N1VGUG0vRE82YWczQW5QMHVHdVAwMlh3ajR4bW15dndB?=
 =?utf-8?B?SVBqdWRJUFZ4US9idzUxaWZHWHkzWTYrZVlIUEJXdzBBSi9iTDQ1UHdSMndR?=
 =?utf-8?B?eVI0R1NrL3VpS0cvRXRETXFwTk9FUXk4UUkxRnFxVWNqNlpRNFBaazFTRXda?=
 =?utf-8?B?dUsxTmNtY2hFQU5LN2E4NTg2UXpuUUtUazRoQUg5dVNIUWFVajRNTnhjbWw2?=
 =?utf-8?B?djlOMUxVV2lvTkJZUS8vNThHZEpsYjE3NnJMV1phL3JBUmI5WTFMQ0VmYnFi?=
 =?utf-8?B?NDI5MG9sTEJCd3V6QVFURWNobVJkVjVjNmx5Q09ON2U4Q2x3UTJUTGlEdThy?=
 =?utf-8?B?cFRTRnZKUnhJL2Vwa3lZcVBuak9MckRZOEgyOXMzcU5rOUhkK05Pbko5NnA0?=
 =?utf-8?B?bGdkUjVNZGhsRFpXZk96aHp5NWd2WlltTDVTMmZLeDBZcGk2N3N0OExsMC9k?=
 =?utf-8?B?L2RXNUlZdkFpTmZZM2UxZTdEdXM4OGpzZzFleFFZeHlPQ0VES3MwUWZpVmVJ?=
 =?utf-8?B?ZlNObkR0UGMzdGN3d2wvMTRUV1FCQlN5b0M4S0VFL1hLLytaaXJ5WXg3TGRV?=
 =?utf-8?B?d0RZZ0pjcGk1eS9qM2I4Rzd4d0FrRjlNRVIxejVwcnRJYWpKYUxFZmppVVE1?=
 =?utf-8?B?Yys3UE9rS3dUamlPTVZzTkIxSnRWRlc2MDcvWVFkc2VEOGh2OEVmanlvYlVu?=
 =?utf-8?B?OVF6TEVwWFhyV3RZUFZIaC9CbEZZU1ZsUzFvZ01vMUpkYUdzTWhoek1vVDhq?=
 =?utf-8?B?RHZ2T2VJYkNLMWYvK0NueElMWTcrNTVGengvYlNpcFJqM2cyYmt2TklCaFo3?=
 =?utf-8?B?MUxvTk0rcnp3citZTHlSdFBEWU5CWmcrcDRDTDQ4TEl5ZUM4YWJRLzU0cmZL?=
 =?utf-8?B?S2hpZk1pcjUwcFRhS1dJY3NDdndaYVJ2aFlEYTl5VHQwZGFlYkg3b09UQ1Vm?=
 =?utf-8?B?SmtQTy9HSU1CZmI4UGRYT240K1RxZTlXcGQvYjBUQTdYbjIxQjIzT3BpNVQ1?=
 =?utf-8?B?NE9ueEEzUlNBcmF0akxESHZuNjJEeWdldlpxeGJPR3VRYkxZZjhSaE5CTGlB?=
 =?utf-8?B?UUdNQ2hRclpyZlFKaFBGbGd6R0gxdEMwR1MvRmszRDhkS3BGUTRhUWpKODl3?=
 =?utf-8?B?R1Z4bFdCTHZPb2kzWEFJcVoyQWdrcnJkVjBXSFB5N0dRcUZxUXVycHdqZVNQ?=
 =?utf-8?B?VmxKdk5jcGxPS0pKTFNwakJsSVpGQ3dmcXN0MmFCYkg4emMzNzJjenByVit0?=
 =?utf-8?B?bG9qRi9UemMzd0J4c3UrNDdjZnZnZUhlQTMybFJjcDBMMnZIb2RHTjE0Mi9h?=
 =?utf-8?B?ZFYvTzJOWkEzbDFoK1g1STNoTGVJQlBCUnExL2lqMVprZC9RV2NNTUtPWHg1?=
 =?utf-8?B?Wm5BdWNONkNxQmJ1SjYwU01WMEYzTW1OS25DY01ZamJ6TmZuY1hDaXAvQ2xY?=
 =?utf-8?B?QnpQVk8vZXVNWGV1SGhEY3ZqQzNPejA1bDZmTkNTaW9VNnM0MzFPUWdCNUl2?=
 =?utf-8?B?Snp0NmtCM0FHT3JsQjQ2cXhGYUZ6YjA2djRVbWlRMWM3dUpPMnM0N1IwT3Za?=
 =?utf-8?B?MnVkczhvUHBXNFN4b2ZSZUR0cjdXSWVxOGR2UE9oOXBicUFJNCtMbS8vUlZZ?=
 =?utf-8?B?K1NrYjBGaDF5cXdvSk9GUjd4alhiSHJrSTZYT2xQUmRvOE5JRWVBYjQ4TGxo?=
 =?utf-8?B?a1E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 11549ea6-1358-43b8-3712-08dab3a0289d
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Oct 2022 20:09:28.6221
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: r9A/XvL5ryAVGOVxwBu57s31zcOBb2wf5dWMm5iR+VXE/nZcmRjHoKcZl4ZhajDy1YHZ/QwXUTrD09isPIA8CeWIpaOy7RLMSBx0vNV+KKE=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR11MB5461
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi James,

On 10/19/2022 6:57 AM, James Morse wrote:
> Hi Peter,
> 
> On 17/10/2022 11:15, Peter Newman wrote:
>> On Wed, Oct 12, 2022 at 6:55 PM James Morse <james.morse@arm.com> wrote:
>>> You originally asked:
>>> | Any concerns about the CLOSID-reusing behavior?
>>>
>>> I don't think this will work well with MPAM ... I expect it will mess up the bandwidth
>>> counters.
>>>
>>> MPAM's equivalent to RMID is PMG. While on x86 CLOSID and RMID are independent numbers,
>>> this isn't true for PARTID (MPAM's version of CLOSID) and PMG. The PMG bits effectively
>>> extended the PARTID with bits that aren't used to look up the configuration.
>>>
>>> x86's monitors match only on RMID, and there are 'enough' RMID... MPAMs monitors are more
>>> complicated. I've seen details of a system that only has 1 bit of PMG space.
>>>
>>> While MPAM's bandwidth monitors can match just the PMG, there aren't expected to be enough
>>> unique PMG for every control/monitor group to have a unique value. Instead, MPAM's
>>> monitors are expected to be used with both the PARTID and PMG.
>>>
>>> ('bandwidth monitors' is relevant here, MPAM's 'cache storage utilisation' monitors can't
>>> match on just PMG at all - they have to be told the PARTID too)
>>>
>>>
>>> If you're re-using CLOSID like this, I think you'll end up with noisy measurements on MPAM
>>> systems as the caches hold PARTID/PMG values from before the re-use pattern changed, and
>>> the monitors have to match on both.
> 
>> Yes, that sounds like it would be an issue.
>>
>> Following your refactoring changes, hopefully the MPAM driver could
>> offer alternative methods for managing PARTIDs and PMGs depending on the
>> available hardware resources.
> 
> Mmmm, I don't think anything other than one-partid per control group and one-pmg per
> monitor group makes much sense.
> 
> 
>> If there are a lot more PARTIDs than PMGs, then it would fit well with a
>> user who never creates child MON groups. In case the number of MON
>> groups gets ahead of the number of CTRL_MON groups and you've run out of
>> PMGs, perhaps you would just try to allocate another PARTID and program
>> the same partitioning configuration before giving up.
> 
> User-space can choose to do this.
> If the kernel tries to be clever and do this behind user-space's back, it needs to
> allocate two monitors for this secretly-two-control-groups, and always sum the counters
> before reporting them to user-space.

If I understand this scenario correctly, the kernel is already doing this.
As implemented in mon_event_count() the monitor data of a CTRL_MON group is
the sum of the parent CTRL_MON group and all its child MON groups.

> If monitors are a contended resource, then you may be unable to monitor the
> secretly-two-control-groups group once the kernel has done this.

I am not viewing this as "secretly-two-control-groups" - there would still be
only one parent CTRL_MON group that dictates all the allocations. MON groups already
have a CLOSID (PARTID) property but at this time it is always identical to the parent
CTRL_MON group. The difference introduced is that some of the child MON groups
may have a different CLOSID (PARTID) from the parent.

> 
> I don't think the kernel should try to be too clever here.
> 

That is a fair concern but it may be worth exploring as it seems to address
a few ABI concerns and user space seems to be eyeing using a future "num_closid"
info as a check of "RDT/PQoS" vs "MPAM".

Reinette