Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757521AbYHTOzz (ORCPT ); Wed, 20 Aug 2008 10:55:55 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752762AbYHTOzr (ORCPT ); Wed, 20 Aug 2008 10:55:47 -0400 Received: from py-out-1112.google.com ([64.233.166.183]:15264 "EHLO py-out-1112.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752402AbYHTOzq (ORCPT ); Wed, 20 Aug 2008 10:55:46 -0400 DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=wAileS8ABHfENi4sN3mP7HiqftE+jh2GSrWsHjazNRmA0Uz6iiVpE7mC/xqrh/BofP Xo8O0RI7bI+hDQVKZ5w0dr7IxztlybyNxQsP+dX8jwm5DnxW4L0MA0nnM6I2L1JEYeEM bvEXLJbrb5VdOrCoOJxHf5N/GVk9y3z87xxbw= Message-ID: <9e4733910808200755y1128ae56p6a1235684bfbb3ec@mail.gmail.com> Date: Wed, 20 Aug 2008 10:55:45 -0400 From: "Jon Smirl" To: "Eran Liberty" Subject: Re: ftrace introduces instability into kernel 2.6.27(-rc2,-rc3) Cc: "Steven Rostedt" , "Mathieu Desnoyers" , linux-kernel@vger.kernel.org, linuxppc-dev@ozlabs.org, "Steven Rostedt" , "Alan Modra" , "Scott Wood" , "Paul E. McKenney" In-Reply-To: <48AC23F4.80900@extricom.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <48591941.4070408@extricom.com> <1219114600.8062.15.camel@pasglop> <1219119431.8062.35.camel@pasglop> <1219216705.21386.46.camel@pasglop> <48AC1DD8.9080702@extricom.com> <48AC23F4.80900@extricom.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4185 Lines: 144 On 8/20/08, Eran Liberty wrote: > Steven Rostedt wrote: > > > On Wed, 20 Aug 2008, Eran Liberty wrote: > > > > > > > > > Steven Rostedt wrote: > > > > > > > > > > On Wed, 20 Aug 2008, Steven Rostedt wrote: > > > > > > > > > > > > > > > > > On Wed, 20 Aug 2008, Benjamin Herrenschmidt wrote: > > > > > > > > > > > > > > > > > > > > > Found the problem (or at least -a- problem), it's a gcc bug. > > > > > > > > > > > > Well, first I must say the code generated by -pg is just plain > > > > > > horrible :-) > > > > > > > > > > > > Appart from that, look at the exit of, for example, __d_lookup, as > > > > > > generated by gcc when ftrace is enabled: > > > > > > > > > > > > c00c0498: 38 60 00 00 li r3,0 > > > > > > c00c049c: 81 61 00 00 lwz r11,0(r1) > > > > > > c00c04a0: 80 0b 00 04 lwz r0,4(r11) > > > > > > c00c04a4: 7d 61 5b 78 mr r1,r11 > > > > > > c00c04a8: bb 0b ff e0 lmw r24,-32(r11) > > > > > > c00c04ac: 7c 08 03 a6 mtlr r0 > > > > > > c00c04b0: 4e 80 00 20 blr > > > > > > > > > > > > As you can see, it restores r1 -before- it pops r24..r31 off > > > > > > the stack ! I let you imagine what happens if an interrupt happens > > > > > > just in between those two instructions (mr and lmw). We don't do > > > > > > redzones on our ABI, so basically, the registers end up corrupted > > > > > > by the interrupt. > > > > > > > > > > > > > > > > > Ouch! You've disassembled this without -pg too, and it does not > have this > > > > > bug? What version of gcc do you have? > > > > > > > > > > > > > > > > > > > I have: > > > > gcc (Debian 4.3.1-2) 4.3.1 > > > > > > > > c00c64c8: 81 61 00 00 lwz r11,0(r1) > > > > c00c64cc: 7f 83 e3 78 mr r3,r28 > > > > c00c64d0: 80 0b 00 04 lwz r0,4(r11) > > > > c00c64d4: ba eb ff dc lmw r23,-36(r11) > > > > c00c64d8: 7d 61 5b 78 mr r1,r11 > > > > c00c64dc: 7c 08 03 a6 mtlr r0 > > > > c00c64e0: 4e 80 00 20 blr > > > > > > > > > > > > My version looks fine. I'm thinking that this is a separate issue > than what > > > > Eran is seeing. > > > > > > > > Eran, can you do an "objdump -dr vmlinux" and search for __d_lookup, > and > > > > print out the end of the function dump. > > > > > > > > Thanks, > > > > > > > > -- Steve > > > > > > > > > > > > > > > > > > > > > > > powerpc-linux-gnu-objdump -dr --start-address=0xc00bb584 vmlinux | head > -n 100 > > > > > > vmlinux: file format elf32-powerpc > > > > > > Disassembly of section .text: > > > > > > c00bb584 <__d_lookup>: > > > > > > > > > > [...] > > > > > > > > > c00bb670: 41 9e 00 50 beq- cr7,c00bb6c0 <__d_lookup+0x13c> > > > c00bb674: 83 de 00 00 lwz r30,0(r30) > > > c00bb678: 2f 9e 00 00 cmpwi cr7,r30,0 > > > c00bb67c: 40 9e ff 98 bne+ cr7,c00bb614 <__d_lookup+0x90> > > > c00bb680: 38 60 00 00 li r3,0 > > > c00bb684: 81 61 00 00 lwz r11,0(r1) > > > c00bb688: 80 0b 00 04 lwz r0,4(r11) > > > c00bb68c: 7d 61 5b 78 mr r1,r11 > > > > > > > > > > [ BUG HERE IF INTERRUPT HAPPENS ] > > > > > > > > > c00bb690: bb 0b ff e0 lmw r24,-32(r11) > > > c00bb694: 7c 08 03 a6 mtlr r0 > > > c00bb698: 4e 80 00 20 blr > > > > > > > > > > Yep, you have the same bug in your compiler. > > > > -- Steve > > > > > Hmm... so whats now? > > Is there a way to prove this scenario is indeed the one that caused the > opps? Manually edit the broken binary to change the order of the restore and see if the problem disappears. That will keep everything else constant. > > -- Liberty > > _______________________________________________ > Linuxppc-dev mailing list > Linuxppc-dev@ozlabs.org > https://ozlabs.org/mailman/listinfo/linuxppc-dev > -- Jon Smirl jonsmirl@gmail.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/