Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CALPaoCj-zav4x6H3ffXo_O+RFan8Qb-uLy-DdtkaQTfuxY4a0w@mail.gmail.com>
 <MW3PR12MB45531D396A2010666F3E238F955F9@MW3PR12MB4553.namprd12.prod.outlook.com>
In-Reply-To: <MW3PR12MB45531D396A2010666F3E238F955F9@MW3PR12MB4553.namprd12.prod.outlook.com>
From:   Stephane Eranian <eranian@google.com>
Date:   Tue, 11 Oct 2022 08:00:20 -0700
Message-ID: <CABPqkBS_=V4D-3wRvucEBZqEsT6VppYutg8ExQoLpKjbzKvZoA@mail.gmail.com>
Subject: Re: [RFD] resctrl: reassigning a running container's CTRL_MON group
To:     "Moger, Babu" <Babu.Moger@amd.com>
Cc:     Peter Newman <peternewman@google.com>,
        Reinette Chatre <reinette.chatre@intel.com>,
        Fenghua Yu <fenghua.yu@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        James Morse <james.morse@arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Fri, Oct 7, 2022 at 10:57 AM Moger, Babu <Babu.Moger@amd.com> wrote:
>
> [AMD Official Use Only - General]
>
> Hi Peter,
>
> > -----Original Message-----
> > From: Peter Newman <peternewman@google.com>
> > Sent: Friday, October 7, 2022 5:40 AM
> > To: Reinette Chatre <reinette.chatre@intel.com>; Fenghua Yu
> > <fenghua.yu@intel.com>
> > Cc: Stephane Eranian <eranian@google.com>; linux-kernel@vger.kernel.org=
;
> > Thomas Gleixner <tglx@linutronix.de>; James Morse
> > <james.morse@arm.com>; Moger, Babu <Babu.Moger@amd.com>
> > Subject: [RFD] resctrl: reassigning a running container's CTRL_MON grou=
p
> >
> > Hi Reinette, Fenghua,
> >
> > I'd like to talk about the tasks file interface in CTRL_MON and MON gro=
ups.
> >
> > For some background, we are using the memory-bandwidth monitoring and
> > allocation features of resctrl to maintain QoS on external memory bandw=
idth
> > for latency-sensitive containers to help enable batch containers to use=
 up
> > leftover CPU/memory resources on a machine.  We also monitor the extern=
al
> > memory bandwidth usage of all hosted containers to identify ones which =
are
> > misusing their latency-sensitive CoS assignment and downgrade them to t=
he
> > batch CoS.
> >
> > The trouble is, container manager developers working with the tasks int=
erface
> > have complained that it's not usable for them because it takes many (or=
 an
> > unbounded number of) passes to move all tasks from a container over, as=
 the
> > list is always changing.
> >
> > Our solution for them is to remove the need for moving tasks between
> > CTRL_MON groups. Because we are mainly using MB throttling to implement
> > QoS, we only need two classes of service. Therefore we've modified resc=
trl to
> > reuse existing CLOSIDs for CTRL_MON groups with identical configuration=
s,
> > allowing us to create a CTRL_MON group for every container. Instead of
> > moving the tasks over, we only need to update their CTRL_MON group's
> > schemata. Another benefit for us is that we do not need to also move al=
l of the
> > tasks over to a new monitoring group in the batch CTRL_MON group, and t=
he
> > usage counts remain intact.
> >
> > The CLOSID management rules would roughly be:
> >
> >  1. If an update would cause a CTRL_MON group's config to match that of
> >     an existing group, the CTRL_MON group's CLOSID should change to tha=
t
> >     of the existing group, where the definition of "match" is: all
> >     control values match in all domains for all resources, as well as
> >     the cpu masks matching.
> >
> >  2. If an update to a CTRL_MON group sharing a CLOSID with another grou=
p
> >     causes that group to no longer match any others, a new CLOSID must
> >     be allocated.
> >
> >  3. An update to a CTRL_MON group using a non-shared CLOSID which
> >     continues to not match any others follows the current resctrl
> >     behavior.
> >
> > Before I prepare any patches for review, I'm interested in any comments=
 or
> > suggestions on the use case and solution.
> >
> > Are there simpler strategies for reassigning a running container's task=
s to a
> > different CTRL_MON group that we should be considering first?
> >
> > Any concerns about the CLOSID-reusing behavior? The hope is existing us=
ers
> > who aren't creating identically-configured CTRL_MON groups would be
> > minimally impacted. Would it help if the proposed behavior were opt-in =
at
> > mount-time?
>
> I am still trying to understand. I would think when creating a new group,=
 the new CLOS id will be used. Basically, it will remain as default behavio=
r.  You probably don=E2=80=99t need to create two identical group during th=
e group creation.
> New behavior of changing the CLOS id will happen when the "match check" i=
s triggered. How is the match check is triggered? Is it triggered by user?

The matching check is triggered when the schemata is changed for any
resctrl group.

> Thanks
> Babu
>
>