Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752551AbbHRKKe (ORCPT ); Tue, 18 Aug 2015 06:10:34 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35433 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752444AbbHRKKb (ORCPT ); Tue, 18 Aug 2015 06:10:31 -0400 Date: Tue, 18 Aug 2015 12:10:25 +0200 From: Jiri Olsa To: Andy Lutomirski Cc: Borislav Petkov , "linux-kernel@vger.kernel.org" , X86 ML , Peter Zijlstra , Ingo Molnar , Robert Richter , "H. Peter Anvin" , Thomas Gleixner , Arnaldo Carvalho de Melo , Namhyung Kim , Jan Stancek , Suravee Suthikulpanit , Sherry Hurwitz Subject: Re: [BUG/RFC] perf test fails on AMD CPUs Message-ID: <20150818101025.GA15340@krava.brq.redhat.com> References: <20150816222956.GA14290@krava.brq.redhat.com> <20150817043603.GB9387@nazgul.tnic> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 5416 Lines: 136 On Mon, Aug 17, 2015 at 09:06:59AM -0700, Andy Lutomirski wrote: > On Sun, Aug 16, 2015 at 9:36 PM, Borislav Petkov wrote: > > On Mon, Aug 17, 2015 at 12:29:56AM +0200, Jiri Olsa wrote: > >> hi, > >> 'perf test 18' is failing on systems with AMD processor. > > > > Hmm, still using that b0rked test box? :-) > > > > Also, which kernel? > > > > There have been substantial changes to the entry code recently. Although > > I don't see anything being done differently on AMD there except > > X86_BUG_SYSRET_SS_ATTRS but that should be unrelated. > > > >> The only reason I could find is that AMD does not set 'resume flag' > >> in RFLAGS register the way the Intel CPU does. > >> > >> (simplified) test scenario: > >> > >> - create breakpoint (on test_function) perf event with SIGIO signal > >> to be delivered any time the breakpoint is hit > >> - run test_function > >> > >> > >> expected course of actions is: > >> 1) CPU hits 'test_function' > >> 2) DB exception is triggered, with RFLAGS.RF=0 > >> 3) DB exception handler sets regs->RFLAGS.RF=1 and perf handler > >> triggers irq_work pending work > >> 4) DB exception executes iretd > >> 5) irq_work interrupt is triggered, with RFLAGS.RF=1 > >> 6) irq_work interrupt calls kill_fasync with SIGIO signal > >> 7) irq_work interrupt on return to userspace calls prepare_exit_to_usermode > >> which actually delivers the SIGIO signal > >> 8) sigreturn syscall prepare registers to return to the > >> instruction from step 1) and sets RFLAGS.RF to the its original > >> value from step 5) (RFLAGS.RF=1) > >> 9) CPU hits 'test_function' and DB exception is NOT triggered > >> due to RFLAGS.RF=1 > >> > >> this is how I see it works on Intel > >> > >> But AMD gives me RFLAGS.RF=0 on step 5, which makes the step 9 to > >> trigger the DB exception once again and makes the test fail. > > > > Adding Andy, he might have an idea. Leaving in the rest for reference. > > Gee thanks :-p > > Jiri, did you instrument the code and observe do_IRQ sees RF clear in > its pt_regs? Also, it might be worth checking that regs->ip in the > irq_work matches regs->ip. yep, thats what I saw.. once irq_work interrupt was triggered the regs->ip was same as for the previous debug exception but the RFLAGS.RF was 0 > > It's *possible* that I messed up and broke RF restore with > opportunistic sysret, but the code looks correct: > > testq $(X86_EFLAGS_RF|X86_EFLAGS_TF), %r11 > jnz opportunistic_sysret_failed AFAICS the problematic paths did not hit syscalls buuuuuut anyway, it looks like latest AMD firmware issue: [root@amd-pike-07 ~]# cat /sys/devices/system/cpu/cpu0/microcode/version 0x6000822 [root@amd-pike-07 perf]# ./perf test 18 18: Test breakpoint overflow signal handler : Ok [root@amd-pike-07 perf]# cat /sys/devices/system/cpu/cpu0/microcode/version 0x6000832 [root@amd-pike-07 perf]# ./perf test 18 18: Test breakpoint overflow signal handler : FAILED! [root@amd-pike-07 ~]# cat /proc/cpuinfo processor : 7 vendor_id : AuthenticAMD cpu family : 21 model : 2 model name : AMD Opteron(tm) Processor 3380 stepping : 0 microcode : 0x6000832 SNIP > >> AMD description of RF flag (SDM 3.1.6): > >> ======================================= > >> Resume Flag (RF) Bit. Bit 16. The RF bit allows an instruction to be restarted following an > >> instruction breakpoint resulting in a debug exception (#DB). This bit prevents multiple debug > >> exceptions from occurring on the same instruction. > >> The processor clears the RF bit after every instruction is successfully executed, except when the > >> instruction is: > >> • > >> • > >> An IRET that sets the RF bit. > >> JMP, CALL, or INTn through a task gate. > >> In both of the above cases, RF is not cleared to 0 until the next instruction successfully executes. > >> When an exception occurs (or when a string instruction is interrupted), the processor normally sets > >> RF=1 in the RFLAGS image saved on the interrupt stack. However, when a #DB exception occurs as a > >> result of an instruction breakpoint, the processor clears the RF bit to 0 in the interrupt-stack RFLAGS > >> image. > > That's a little weird, I think. Shouldn't RF be zero on #DB due to a > *watchpoint* so that a watchpoint followed immediately by a breakpoint > works? the AMD description looked to be more vague (compared to Intels) > > >> • For other cases, the value pushed for RF is the value that was in EFLAG.RF at the time the event handler was > >> called. This includes: > >> — Debug exceptions generated in response to instruction breakpoints > >> — Hardware-generated interrupts arriving between instructions (including those arriving after the last > >> iteration of a repeated string instruction) > > This appears to be why it works on Intel. Does AMD not do that? We > could probably work around this in software (by not using irq work for > this), but yuck. yep, but hopefuly it's the issue microcode ;-) Cc-ing guys from linux-firmware git Sherry, Suravee, any idea? thanks, jirka -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/