Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <CAM_iQpXvuYKn94WjJ9nSjzhk8DzYAvDmdgxsi6cc9CdBfkdTnw@mail.gmail.com>
 <20180223121456.GZ25201@hirez.programming.kicks-ass.net> <20180226203937.GA21543@tassilo.jf.intel.com>
 <CAKA=qzYOU-VtEC5p6djRNmVS0xGe=jpTd3ZgUr++1G3Jj1=PTg@mail.gmail.com>
 <alpine.DEB.2.21.1908121933310.7324@nanos.tec.linutronix.de>
 <CAKA=qzY=F-wj8YXhb-B7RahNceeab0rSA=06qBc8+7V-SyY-+Q@mail.gmail.com>
 <alpine.DEB.2.21.1908122133310.7324@nanos.tec.linutronix.de>
 <CAKA=qzYTax6ECQBChUrNWMRp5iY9F2SezMY2Ma_zmWxiDgjOSA@mail.gmail.com> <CAKA=qzY_2S11MTMEjLtQHJLgHV_nY8893EhBm4-gBmS+duYBDA@mail.gmail.com>
In-Reply-To: <CAKA=qzY_2S11MTMEjLtQHJLgHV_nY8893EhBm4-gBmS+duYBDA@mail.gmail.com>
From:   Josh Hunt <joshhunt00@gmail.com>
Date:   Mon, 19 Aug 2019 16:16:34 -0700
Message-ID: <CAKA=qzann=Ex3owK7mJezq3JfkD1HxPiVAqQSSA=u7tPYVVXZA@mail.gmail.com>
Subject: Re: Long standing kernel warning: perfevents: irq loop stuck!
To:     Thomas Gleixner <tglx@linutronix.de>
Cc:     Andi Kleen <ak@linux.intel.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Cong Wang <xiyou.wangcong@gmail.com>,
        "Liang, Kan" <kan.liang@intel.com>, jolsa@redhat.com,
        bigeasy@linutronix.de, "H. Peter Anvin" <hpa@zytor.com>,
        Ingo Molnar <mingo@redhat.com>, x86 <x86@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, Aug 19, 2019 at 2:17 PM Josh Hunt <joshhunt00@gmail.com> wrote:
>
> On Mon, Aug 12, 2019 at 12:42 PM Josh Hunt <joshhunt00@gmail.com> wrote:
> >
> > On Mon, Aug 12, 2019 at 12:34 PM Thomas Gleixner <tglx@linutronix.de> wrote:
> > >
> > > On Mon, 12 Aug 2019, Josh Hunt wrote:
> > > > On Mon, Aug 12, 2019 at 10:55 AM Thomas Gleixner <tglx@linutronix.de> wrote:
> > > > >
> > > > > On Mon, 12 Aug 2019, Josh Hunt wrote:
> > > > > > Was there any progress made on debugging this issue? We are still
> > > > > > seeing it on 4.19.44:
> > > > >
> > > > > I haven't seen anyone looking at this.
> > > > >
> > > > > Can you please try the patch Ingo posted:
> > > > >
> > > > >   https://lore.kernel.org/lkml/20150501070226.GB18957@gmail.com/
> > > > >
> > > > > and if it fixes the issue decrease the value from 128 to the point where it
> > > > > comes back, i.e. 128 -> 64 -> 32 ...
> > > > >
> > > > > Thanks,
> > > > >
> > > > >         tglx
> > > >
> > > > I just checked the machines where this problem occurs and they're both
> > > > Nehalem boxes. I think Ingo's patch would only help Haswell machines.
> > > > Please let me know if I misread the patch or if what I'm seeing is a
> > > > different issue than the one Cong originally reported.
> > >
> > > Find the NHM hack below.
> > >
> > > Thanks,
> > >
> > >         tglx
> > >
> > > 8<----------------
> > >
> > > diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
> > > index 648260b5f367..93c1a4f0e73e 100644
> > > --- a/arch/x86/events/intel/core.c
> > > +++ b/arch/x86/events/intel/core.c
> > > @@ -3572,6 +3572,11 @@ static u64 bdw_limit_period(struct perf_event *event, u64 left)
> > >         return left;
> > >  }
> > >
> > > +static u64 nhm_limit_period(struct perf_event *event, u64 left)
> > > +{
> > > +       return max(left, 128ULL);
> > > +}
> > > +
> > >  PMU_FORMAT_ATTR(event, "config:0-7"    );
> > >  PMU_FORMAT_ATTR(umask, "config:8-15"   );
> > >  PMU_FORMAT_ATTR(edge,  "config:18"     );
> > > @@ -4606,6 +4611,7 @@ __init int intel_pmu_init(void)
> > >                 x86_pmu.pebs_constraints = intel_nehalem_pebs_event_constraints;
> > >                 x86_pmu.enable_all = intel_pmu_nhm_enable_all;
> > >                 x86_pmu.extra_regs = intel_nehalem_extra_regs;
> > > +               x86_pmu.limit_period = nhm_limit_period;
> > >
> > >                 mem_attr = nhm_mem_events_attrs;
> > >
> > Thanks Thomas. Will try this and let you know.
> >
> > --
> > Josh
>
> Thomas
>
> I found on my setup that setting the value to 32 was the lowest value
> I could use to keep the problem from happening. Let me know if you
> want me to send a patch with the updated value, etc.
>
> I saw in the original thread from Ingo and Vince that this was seen on
> Haswell, but I checked our Haswell boxes and so far we have not
> reproduced the problem there.
>
> --
> Josh

I went ahead and sent this patch with the value set to 32:
https://lore.kernel.org/lkml/1566256411-18820-1-git-send-email-johunt@akamai.com/T/#u

I wasn't sure how/who to give credit to for the change, so please
resubmit if what I did is incorrect or if you wanted to debug further.
If you decide to resubmit the patch please add my tested-by and
Bhupesh's reported-by. I'm able to reproduce the problem within about
2 hours if there's anything else you wanted to look into before going
with this approach.

Thanks!
-- 
Josh