Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1;
MIME-Version: 1.0
References: <20231201005720.235639-1-babu.moger@amd.com>
In-Reply-To: <20231201005720.235639-1-babu.moger@amd.com>
From:   Peter Newman <peternewman@google.com>
Date:   Mon, 4 Dec 2023 16:13:50 -0800
Message-ID: <CALPaoCgZ=eXJj4xVsy9x8a39cpPv+3LDF0KS0RKuTknn5k=9VA@mail.gmail.com>
Subject: Re: [PATCH 00/15] x86/resctrl : Support AMD QoS RMID Pinning feature
To:     James Morse <james.morse@arm.com>
Cc:     corbet@lwn.net, fenghua.yu@intel.com, reinette.chatre@intel.com,
        tglx@linutronix.de, mingo@redhat.com, bp@alien8.de,
        dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com,
        paulmck@kernel.org, rdunlap@infradead.org, tj@kernel.org,
        peterz@infradead.org, seanjc@google.com, kim.phillips@amd.com,
        jmattson@google.com, ilpo.jarvinen@linux.intel.com,
        jithu.joseph@intel.com, kan.liang@linux.intel.com, nikunj@amd.com,
        daniel.sneddon@linux.intel.com, pbonzini@redhat.com,
        rick.p.edgecombe@intel.com, rppt@kernel.org,
        maciej.wieczor-retman@intel.com, linux-doc@vger.kernel.org,
        linux-kernel@vger.kernel.org, eranian@google.com, dhagiani@amd.com,
        Babu Moger <babu.moger@amd.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

[+James]

Hi James,

On Thu, Nov 30, 2023 at 4:57=E2=80=AFPM Babu Moger <babu.moger@amd.com> wro=
te:
>
> These series adds the support for AMD QoS RMID Pinning feature. It is als=
o
> called ABMC (Assignable Bandwidth Monitoring Counters) feature.
>
> The feature details are available in APM listed below [1].
> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
> Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth
> Monitoring (ABMC). The documentation is available at
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=3D206537
>
> The patches are based on top of commit
> 346887b65d89ae987698bc1efd8e5536bd180b3f (tip/master)
>
> # Introduction
>
> AMD hardware can support 256 or more RMIDs. However, bandwidth monitoring
> feature only guarantees that RMIDs currently assigned to a processor will
> be tracked by hardware. The counters of any other RMIDs which are no
> longer being tracked will be reset to zero. The MBM event counters return
> "Unavailable" for the RMIDs that are not active.
>
> Users can create 256 or more monitor groups. But there can be only limite=
d
> number of groups that can be give guaranteed monitoring numbers. With eve=
r
> changing system configuration, there is no way to definitely know which o=
f
> these groups will be active for certain point of time. Users do not have
> the option to monitor a group or set of groups for certain period of time
> without worrying about RMID being reset in between.
>
> The ABMC feature provides an option to pin (or assign) the RMID to the
> hardware counter and monitor the bandwidth for a longer duration. The
> pinned RMID will be active until the user unpins (or unassigns) it.  Ther=
e
> is no need to worry about counters being reset during this period.
> Additionally, the user can specify a bitmask identifying the specific
> bandwidth types from the given source to track with the counter.
>
> # Linux Implementation
>
> Hardware provides total of 32 counters available for assignment.
> Each Linux resctrl group can be assigned a maximum of 2 counters. One for
> mbm_total_bytes and one for mbm_local_bytes. Users also have the option t=
o
> assign only one counter to the group. If the system runs out of assignabl=
e
> counters, the kernel will display the error when the user attempts a new
> counter assignment. Users need to unassign already used counters for new
> assignments.
>
> # Examples
>
> a. Check if ABMC support is available
>         #mount -t resctrl resctrl /sys/fs/resctrl/
>         #cat /sys/fs/resctrl/info/L3_MON/mon_features
>         llc_occupancy
>         mbm_total_bytes
>         mbm_total_bytes_config
>         mbm_local_bytes
>         mbm_local_bytes_config
>         abmc_capable =E2=86=90  Linux kernel detected ABMC feature.
>
> b. Mount with ABMC support
>         #umount /sys/fs/resctrl/
>         #mount  -o abmc -t resctrl resctrl /sys/fs/resctrl/
>
> c. Read the monitor states. There will be new file "monitor_state"
>    for each monitor group when ABMC feature is enabled. By default,
>    both total and local MBM events are in "unassign" state.
>
>         #cat /sys/fs/resctrl/monitor_state
>         total=3Dunassign;local=3Dunassign
>
> d. Read the event mbm_total_bytes and mbm_local_bytes. Note that MBA
>    events are not available until the user assigns the events explicitly.
>    Users need to assign the counters to monitor the events in this mode.
>
>         #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
>         Unavailable
>
>         #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes
>         Unavailable
>
> e. Assign a h/w counter to the total event and read the monitor_state.
>
>         #echo total=3Dassign > /sys/fs/resctrl/monitor_state
>         #cat /sys/fs/resctrl/monitor_state
>         total=3Dassign;local=3Dunassign
>
> f. Now that the total event is assigned. Read the total event.
>
>         #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
>         6136000
>
> g. Assign a h/w counter to both total and local events and read the monit=
or_state.
>
>         #echo "total=3Dassign;local=3Dassign" > /sys/fs/resctrl/monitor_s=
tate
>         #cat /sys/fs/resctrl/monitor_state
>         total=3Dassign;local=3Dassign
>
> h. Now that both total and local events are  assigned, read the events.
>
>         #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
>         6136000
>         #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes
>         58694

We had briefly discussed this topic of explicit counter assignment in
resctrl earlier this year[1], but you didn't want it to be unique to
MPAM.

Now that a similar capability exists on AMD and an interface is being
proposed, we can talk about this in the context of MPAM again.

With some generalization and refinements, I expect this proposal could
be applied to assigning a limited number of MBWU monitors to
monitoring groups.

Also, I had proposed in another thread[2] applying such an interface
to previous AMD hardware where the monitor assignments cannot be
directly controlled to avoid or reduce the overhead in my soft RMID
proposal.

Thanks!
-Peter

[1] https://lore.kernel.org/all/f8a25b5f-4a7d-0891-1152-33f349059b5d@arm.co=
m/
[2] https://lore.kernel.org/all/CALPaoCjg-W3w8OKLHP_g6Evoo03fbgaOQZrGTLX6vd=
SLp70=3DSA@mail.gmail.com/