Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CALPaoCj-zav4x6H3ffXo_O+RFan8Qb-uLy-DdtkaQTfuxY4a0w@mail.gmail.com>
 <b2e020b1-f6b2-e236-a042-4eb2fd27d8b0@intel.com> <IA1PR11MB6097236CFF891041DBA42ECB9B5F9@IA1PR11MB6097.namprd11.prod.outlook.com>
 <Y0BhzKkksSjSeE3W@agluck-desk3.sc.intel.com> <81a7b4f6-fbb5-380e-532d-f2c1fc49b515@intel.com>
 <CALPaoCjdeRjyX5L6BBX688ZM21eMwetuL9QLF1+GEDUskGcU2w@mail.gmail.com>
 <76bb4dc9-ab7c-4cb6-d1bf-26436c88c6e2@arm.com> <CALPaoCiKUQC+LxDwKQ0gE5AQniJi_nbzrXi_HA9ZBRtiXdw_dg@mail.gmail.com>
 <835d769b-3662-7be5-dcdd-804cb1f3999a@arm.com> <09029c7a-489a-7054-1ab5-01fa879fb42f@intel.com>
 <f80299a4-7eaf-46a0-89e6-b9f5385f183c@arm.com> <c227a0df-7ac8-91f3-cada-0ca5ec047579@intel.com>
In-Reply-To: <c227a0df-7ac8-91f3-cada-0ca5ec047579@intel.com>
From:   Peter Newman <peternewman@google.com>
Date:   Wed, 9 Nov 2022 10:50:38 +0100
Message-ID: <CALPaoCidd+WwGTyE3D74LhoL13ce+EvdTmOnyPrQN62j+zZ1fg@mail.gmail.com>
Subject: Re: [RFD] resctrl: reassigning a running container's CTRL_MON group
To:     Reinette Chatre <reinette.chatre@intel.com>
Cc:     James Morse <james.morse@arm.com>, Tony Luck <tony.luck@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "Eranian, Stephane" <eranian@google.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Babu Moger <Babu.Moger@amd.com>,
        Gaurang Upasani <gupasani@google.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

Hi Reinette,

On Tue, Nov 8, 2022 at 10:28 PM Reinette Chatre
<reinette.chatre@intel.com> wrote:
> On 11/3/2022 10:06 AM, James Morse wrote:
> > That is true. MPAM has an additional headache here as it needs to allocate a monitor in
> > order to read the counters. If there are enough monitors for each CLOSID*RMID to have one,
> > then MPAM can export the counter files in the same way RDT does.
> >
> > While there are systems that have enough monitors, I don't think this is going to be the
> > norm. To allow systems that don't have a surfeit of monitors to use the counters, I plan
> > to export the values from resctrl_arch_rmid_read() via perf. (but only for bandwidth counters)
>
> This sounds related to the way monitoring was done in earlier kernels. This was
> long before I become involved with this work. Unfortunately I am not familiar with
> all the history involved that ended in it being removed from the kernel. Looks like
> this was around v4.6, here is a sample commit that may help point to what was done:

Sort of related, this is a problem we have to work around on AMD
implementations that I will be sharing a patch for soon.

Note the second paragraph at the top of page 13:

https://developer.amd.com/wp-content/resources/56375_1.00.pdf

AMD QoS often provides less counters than RMIDs, but the architecture
promises there will be at least as many counters in a QoS domain as
CPUs. Using this we can permanently pin RMIDs to CPUs and read the
counters on every task switch to implement MBM RMIDs in software.

This has the caveats that evictions while one task is running could have
resulted from a previous task on the current CPU, but will be counted
against the new task's software-RMID, and that CMT doesn't work.

I will propose making this available as a mount option for cloud container
use cases which need to monitor a large number of tasks on B/W counter-poor
systems, and of course don't need CMT.

> [...]
>
> > I think the solution to all this is:
> >  * Add rename support to move a monitor group between two control groups.
> >  ** On x86, this is guaranteed to preserve the RMID, so the destination counter continues
> > unaffected.
> >  ** On arm64, the PARTID is also relevant to the monitors, so the old counters will
> > continue to count.
>
> This looks like the solution to me also.
>
> The details of the arm64 support is not clear to me though. The destination
> group may not have enough PMG to host the new group so failures need to be
> handled. As you mention also, the old counters will continue to count.
> I assume that you mean the hardware will still have a record of the occupancy
> and that needs some time to dissipate? I assume this would fall under the
> limbo handling so in some scenarios (for example the just moved monitor
> group used the last PMG) it may take some time for the source control
> group to allow a new monitor group? The new counters will also not
> reflect the task's history.
>
> Moving an arm64  monitor group may thus have a few surprises for user
> space while sounding complex to support. Would adding all this additional
> support be worth it if the guidance to user space is to instead create many
> control groups in such a control-group-rich environment?
>
> > Whether this old counters keep counting needs exposing to user-space so that it is aware.
>
> Could you please elaborate? Do old counters not always keep counting?

Based on this, is it even worth it to allocate PMGs given that the
systems James has seen so far only have a single PMG bit? All this will
get us is the ability to create a single child mon_group in each control
group. This seems too limiting for the feature to be useful.

-Peter