Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230525180209.19497-1-james.morse@arm.com> <20230525180209.19497-13-james.morse@arm.com>
 <CALPaoCjLjngabG32m4X8sSwK-bbZ28oL6BVKSn_dFnDwPzkDkg@mail.gmail.com> <7ff7fcf9-725b-fb3d-43f3-b80a1df3a001@arm.com>
In-Reply-To: <7ff7fcf9-725b-fb3d-43f3-b80a1df3a001@arm.com>
From:   Peter Newman <peternewman@google.com>
Date:   Wed, 7 Jun 2023 14:51:40 +0200
Message-ID: <CALPaoChW=3T2EmrcW+eLEnDUY00rsRSQarTN3c0hSDX-FDRqvw@mail.gmail.com>
Subject: Re: [PATCH v4 12/24] x86/resctrl: Make resctrl_arch_rmid_read() retry
 when it is interrupted
To:     James Morse <james.morse@arm.com>
Cc:     x86@kernel.org, linux-kernel@vger.kernel.org,
        Fenghua Yu <fenghua.yu@intel.com>,
        Reinette Chatre <reinette.chatre@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        H Peter Anvin <hpa@zytor.com>,
        Babu Moger <Babu.Moger@amd.com>,
        shameerali.kolothum.thodi@huawei.com,
        D Scott Phillips OS <scott@os.amperecomputing.com>,
        carl@os.amperecomputing.com, lcherian@marvell.com,
        bobo.shaobowang@huawei.com, tan.shaopeng@fujitsu.com,
        xingxin.hx@openanolis.org, baolin.wang@linux.alibaba.com,
        Jamie Iles <quic_jiles@quicinc.com>,
        Xin Hao <xhao@linux.alibaba.com>, dfustini@baylibre.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Hi James,

On Tue, Jun 6, 2023 at 7:03=E2=80=AFPM James Morse <james.morse@arm.com> wr=
ote:
> On 06/06/2023 09:49, Peter Newman wrote:
> > It looks like if __rmid_read() is interrupted by an occupancy counter
> > read between writing QM_EVTSEL and reading QM_CTR, it will not perform
> > any update to am->prev_msr, and the interrupted read will return the
> > same counter value as in the interrupting read.
>
> Yup, that's a problem. I was only looking at the mbm state in memory, not=
 the CPU register.
> I think the fix is to read back QM_EVTSEL after reading QM_CTR. I'll do t=
his in
> __rmid_read() to avoid returning -EINTR. It creates two retry loops which=
 is annoying, but
> making the window larger means you're more likely to see false positives.
>
> ----------------------------%<----------------------------
> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/=
resctrl
> /monitor.c
> index e24390d2e661..aeba035bb680 100644
> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
> @@ -101,6 +101,7 @@ static inline u64 get_corrected_mbm_count(u32 rmid, u=
nsigned
>  long val)
>
>  static int __rmid_read(u32 rmid, enum resctrl_event_id eventid, u64 *val=
)
>  {
> +       u32 _rmid, _eventid;
>         u64 msr_val;
>
>         /*
> @@ -110,9 +111,15 @@ static int __rmid_read(u32 rmid, enum resctrl_event_=
id eventid, u64 *val)
>          * IA32_QM_CTR.data (bits 61:0) reports the monitored data.
>          * IA32_QM_CTR.Error (bit 63) and IA32_QM_CTR.Unavailable (bit 62=
)
>          * are error bits.
> +        * QM_EVTSEL is re-read to detect if this function was interrupte=
d by
> +        * another call, meaning the QM_CTR value may belong to a differe=
nt
> +        * event.
>          */
> -       wrmsr(MSR_IA32_QM_EVTSEL, eventid, rmid);
> -       rdmsrl(MSR_IA32_QM_CTR, msr_val);
> +       do {
> +               wrmsr(MSR_IA32_QM_EVTSEL, eventid, rmid);
> +               rdmsrl(MSR_IA32_QM_CTR, msr_val);
> +               rdmsr(MSR_IA32_QM_EVTSEL, _eventid, _rmid);
> +       } while (eventid !=3D _eventid || rmid !=3D _rmid);
>
>         if (msr_val & RMID_VAL_ERROR)
>                 return -EIO;

I happen to be tracking the cost of resctrl_arch_rmid_read() calls, so
I measured the impact of your fix on my AMD EPYC 7B12:

with both this and the soft RMID series[1] applied:

Base switch         7955     0.23%
Hard RMID Switch    8476     6.80%
Soft RMID Switch    10173   28.17%
CLOSID+Soft RMID    10740   35.32%

then adding EVTSEL read-back patch:

Base switch         7985
Hard RMID Switch    8540     6.96%
Soft RMID Switch    11015   37.95%
CLOSID+Soft RMID    11590   45.16%

The Soft RMID switches contain two __rmid_read() calls, so this
implies each QM_EVTSEL read-back is around 420 cycles on this AMD
implementation.

Even if you don't agree with my plan to add resctrl_arch_rmid_read()
calls to context switches, there should be cheaper ways to handle
this.

-Peter

[1] https://lore.kernel.org/lkml/20230421141723.2405942-4-peternewman@googl=
e.com/