2008-10-30 21:45:29

by Luis R. Rodriguez

[permalink] [raw]
Subject: [MEMO] ftrace suckage++

As johannes pointed out to me the culprit to the e1000e corruption was
determined to be ftrace related [1]. Well it seems I hit an oops with
ftrace using iwlagn after a simple load of the module using
wireless-testing. I'm not sure if john's tree yet has ftrace fixes.
Both Fedora 10 and Ubuntu 8.10 had dynamic ftrace enabled so users on
these first kernels and using compat-wireless may run into this so
just putting this out there. I'll can check for this flag in
compat-wireless for now and bitch/exit if its enabled unless someone
has a better idea.

[1] http://lwn.net/Articles/303390/

Oops, using wireless-testing and iwlagn with these options enabled:

CONFIG_HAVE_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_FTRACE=y
CONFIG_DYNAMIC_FTRACE=y


[ 294.209022] BUG: soft lockup - CPU#1 stuck for 61s! [insmod:9196]
[ 294.209022] Modules linked in: iwlcore mac80211 cfg80211 af_packet
binfmt_misc rfcomm l2cap uinput ipv6 acpi_cpufreq cpufreq_userspace
cpufreq_conservative cpufreq_ondemand cpufreq_powersave cpufreq_stats
freq_table pci_slot sbs sbshc container iptable_filter ip_tables
x_tables
sbp2 snd_hda_intel joydev snd_pcm_oss arc4 snd_mixer_oss ecb snd_pcm
snd_seq_dummy snd_seq_oss pcmcia snd_seq_midi sdhci_pci sdhci psmouse
serio_raw yenta_socket rsrc_nonstatic pcmcia_core ricoh_mmc mmc_core
snd_rawmidi iTCO_wdt iTCO_vendor_support btusb bluetooth ac battery
video
output snd_seq_midi_event snd_seq snd_timer snd_seq_device shpchp
pci_hotplug snd soundcore wmi snd_page_alloc intel_agp agpgart button
thinkpad_acpi rfkill hwmon led_class evdev nvram ext3 jbd mbcache
sr_mod cdrom sg ata_piix sd_mod crc_t10dif pata_acpi ohci1394 ieee1394
ata_generic ahci libata scsi_mod ehci_hcd uhci_hcd e1000e usbcore
thermal processor fan fuse [last unloaded: cfg80211]
[ 294.209022] irq event stamp: 208522
[ 294.209022] hardirqs last enabled at (208521): [<c02674ac>]
trace_hardirqs_on_thunk+0xc/0x10
[ 294.209022] hardirqs last disabled at (208522): [<c02674bc>]
trace_hardirqs_off_thunk+0xc/0x10
[ 294.209022] softirqs last enabled at (208520): [<c013b39b>]
__do_softirq+0x13b/0x190
[ 294.209022] softirqs last disabled at (208511): [<c013b47d>]
do_softirq+0x8d/0xa0
[ 294.209022]
[ 294.209022] Pid: 9196, comm: insmod Not tainted (2.6.28-rc2-wl #9) 6460DWU
[ 294.209022] EIP: 0060:[<c018c96d>] EFLAGS: 00000282 CPU: 1
[ 294.209022] EIP is at ftrace_record_ip+0x9d/0x1d0
[ 294.209022] EAX: c209a068 EBX: f80d9c36 ECX: 03380000 EDX: c2714738
[ 294.209022] ESI: c054cda0 EDI: c092d4f8 EBP: f2311e0c ESP: f2311dec
[ 294.209022] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[ 294.209022] CR0: 8005003b CR2: b7c4e007 CR3: 32222000 CR4: 00000690
[ 294.209022] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 294.209022] DR6: ffff0ff0 DR7: 00000400
[ 294.209022] Call Trace:
[ 294.209022] [<c018cdff>] ftrace_convert_nops+0x2f/0xd0
[ 294.209022] [<c018ceb6>] ftrace_init_module+0x16/0x20
[ 294.209022] [<c01671fc>] load_module+0x15dc/0x1c10
[ 294.209022] [<c019608c>] ? generic_file_aio_read+0x24c/0x6b0
[ 294.209022] [<c015cecb>] ? trace_hardirqs_on+0xb/0x10
[ 294.209022] [<c0394da4>] ? mutex_lock_interruptible_nested+0x274/0x340
[ 294.209022] [<c0167883>] sys_init_module+0x53/0x1b0
[ 294.209022] [<c015ce65>] ? trace_hardirqs_on_caller+0x115/0x170
[ 294.209022] [<c02674ac>] ? trace_hardirqs_on_thunk+0xc/0x10
[ 294.209022] [<c010407b>] sysenter_do_call+0x12/0x3f

Luis


2008-10-30 23:03:13

by Tim Gardner

[permalink] [raw]
Subject: Re: [MEMO] ftrace suckage++

Luis R. Rodriguez wrote:
> As johannes pointed out to me the culprit to the e1000e corruption was
> determined to be ftrace related [1]. Well it seems I hit an oops with
> ftrace using iwlagn after a simple load of the module using
> wireless-testing. I'm not sure if john's tree yet has ftrace fixes.
> Both Fedora 10 and Ubuntu 8.10 had dynamic ftrace enabled so users on
> these first kernels and using compat-wireless may run into this so
> just putting this out there. I'll can check for this flag in
> compat-wireless for now and bitch/exit if its enabled unless someone
> has a better idea.
>
> [1] http://lwn.net/Articles/303390/
>

I disabled it in the 2.6.27 Intrepid Ubuntu kernel prior to release.

rtg
--
Tim Gardner [email protected]

2008-10-31 02:05:39

by Luis R. Rodriguez

[permalink] [raw]
Subject: Re: [MEMO] ftrace suckage++

On Thu, Oct 30, 2008 at 06:54:00PM -0700, Steven Rostedt wrote:
>
> On Thu, 30 Oct 2008, Luis R. Rodriguez wrote:
>
> > As johannes pointed out to me the culprit to the e1000e corruption was
> > determined to be ftrace related [1]. Well it seems I hit an oops with
> > ftrace using iwlagn after a simple load of the module using
> > wireless-testing. I'm not sure if john's tree yet has ftrace fixes.
> > Both Fedora 10 and Ubuntu 8.10 had dynamic ftrace enabled so users on
> > these first kernels and using compat-wireless may run into this so
> > just putting this out there. I'll can check for this flag in
> > compat-wireless for now and bitch/exit if its enabled unless someone
> > has a better idea.
> >
> > [1] http://lwn.net/Articles/303390/
> >
> > Oops, using wireless-testing and iwlagn with these options enabled:
> >
> > CONFIG_HAVE_FTRACE=y
> > CONFIG_HAVE_DYNAMIC_FTRACE=y
> > CONFIG_FTRACE=y
> > CONFIG_DYNAMIC_FTRACE=y
> >
>
> Yes, ftrace in 2.6.27 is known to be broken. That's why 2.6.27.1 labeled
> it as such. What is currently in Linus's 2.6.28 git tree is a completely
> rewritten version of the dynamic ftracer, which addresses all of the
> issues that caused breakage in the 2.6.27 release.
>
> Do not use the 2.6.27 dynamic ftracer, it is known bad. 2.6.28 is fine. If
> you find issues with what is currently in 2.6.28, please let me know.
> There is currently some, "make allyesconfig" breakage that ftrace caused
> on archs other than x86,ppc and sparc, but patches to fix them are
> in queue to Linus.

OK then in that case if *and only if* on 27 and if dynamic tracer is enabled
I'll set compat stuff to simply bail out at compile time.

Thanks,

Luis

2008-10-31 01:55:56

by Steven Rostedt

[permalink] [raw]
Subject: Re: [MEMO] ftrace suckage++


On Thu, 30 Oct 2008, Luis R. Rodriguez wrote:

> As johannes pointed out to me the culprit to the e1000e corruption was
> determined to be ftrace related [1]. Well it seems I hit an oops with
> ftrace using iwlagn after a simple load of the module using
> wireless-testing. I'm not sure if john's tree yet has ftrace fixes.
> Both Fedora 10 and Ubuntu 8.10 had dynamic ftrace enabled so users on
> these first kernels and using compat-wireless may run into this so
> just putting this out there. I'll can check for this flag in
> compat-wireless for now and bitch/exit if its enabled unless someone
> has a better idea.
>
> [1] http://lwn.net/Articles/303390/
>
> Oops, using wireless-testing and iwlagn with these options enabled:
>
> CONFIG_HAVE_FTRACE=y
> CONFIG_HAVE_DYNAMIC_FTRACE=y
> CONFIG_FTRACE=y
> CONFIG_DYNAMIC_FTRACE=y
>

Yes, ftrace in 2.6.27 is known to be broken. That's why 2.6.27.1 labeled
it as such. What is currently in Linus's 2.6.28 git tree is a completely
rewritten version of the dynamic ftracer, which addresses all of the
issues that caused breakage in the 2.6.27 release.

Do not use the 2.6.27 dynamic ftracer, it is known bad. 2.6.28 is fine. If
you find issues with what is currently in 2.6.28, please let me know.
There is currently some, "make allyesconfig" breakage that ftrace caused
on archs other than x86,ppc and sparc, but patches to fix them are
in queue to Linus.

-- Steve