Hi,
I've been running 2.6.21.3-rt9 and a couple of -rc predecessors for
a few weeks and this minor problem has been consistent on all versions
I've run. On this AMD64 machine I still have 2.6.17-rt8 and 2.6.18-rt7
and neither of those versions cause the problem. I am currently
building the matching vanilla kernel as well as the 2.6.22-rc3 kernel
to see if it's in those versions.
I see today that the same problem has been reported by another
AMD64 user on a distribution mail list so I think the problem isn't
specific to my hardware anymore. I had previously thought so and
therefore didn't report this right away thinking I'd do more debug.
http://search.gmane.org/?query=IRQ+23&group=gmane.linux.distributions.64studio
Since my IRQ23 is shared on my machine by Ethernet and USB I have
tried detaching all network cables and turning off all USB devices but
the problem persists.
I'll report back what happens with the vanilla-kernels. If a binary
regression search is required of everything between 2.6.18 and 2.6.21
to look for where the problem crept in that will take some time so any
guidance on where to start would be helpful.
If more info is desired please don't hesitate to ask.
Cheers,
Mark
mark@lightning ~ $ uname -a
Linux lightning 2.6.21.3-rt9 #1 PREEMPT RT Sat Jun 9 09:49:32 PDT 2007
x86_64 AMD Athlon(tm) 64 Processor 3000+ AuthenticAMD GNU/Linux
mark@lightning ~ $
mark@lightning ~ $ top
top - 10:02:17 up 3 min, 2 users, load average: 0.00, 0.00, 0.00
Tasks: 88 total, 2 running, 86 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.3%us, 0.0%sy, 0.0%ni, 95.7%id, 0.0%wa, 3.0%hi, 0.0%si, 0.0%st
Mem: 1026612k total, 190316k used, 836296k free, 8124k buffers
Swap: 2008084k total, 0k used, 2008084k free, 74616k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
947 root -76 -5 0 0 0 S 2.7 0.0 0:05.58 IRQ-23
9533 root 15 0 324m 13m 7420 S 1.3 1.4 0:03.24 X
1 root 15 0 2668 588 500 S 0.0 0.1 0:00.27 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.00 posix_cpu_timer
3 root -51 0 0 0 0 S 0.0 0.0 0:00.00 softirq-high/0
mark@lightning ~ $ cat /proc/interrupts
CPU0
0: 212 IO-APIC-edge timer
1: 296 IO-APIC-edge i8042
6: 3 IO-APIC-edge floppy
7: 0 IO-APIC-edge lpptest
8: 0 IO-APIC-edge rtc
9: 0 IO-APIC-fasteoi acpi
12: 8690 IO-APIC-edge i8042
14: 62 IO-APIC-edge ide0
16: 0 IO-APIC-fasteoi hdsp
18: 14496 IO-APIC-fasteoi ohci1394, radeon@pci:0000:01:00.0
20: 2 IO-APIC-fasteoi ehci_hcd:usb1
21: 7724 IO-APIC-fasteoi libata
22: 0 IO-APIC-fasteoi libata, NVidia CK804
23: 1739277 IO-APIC-fasteoi ohci_hcd:usb2, eth0
NMI: 0
LOC: 59688
ERR: 0
mark@lightning ~ $
mark@lightning ~ $ uname -a
Linux lightning 2.6.18-rt7 #4 PREEMPT Sun Apr 8 17:32:31 PDT 2007
x86_64 AMD Athlon(tm) 64 Processor 3000+ AuthenticAMD GNU/Linux
mark@lightning ~ $
mark@lightning ~ $ top
top - 10:09:51 up 1 min, 2 users, load average: 0.00, 0.00, 0.00
Tasks: 83 total, 1 running, 82 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.7%us, 0.3%sy, 0.0%ni, 95.4%id, 2.6%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1026840k total, 185688k used, 841152k free, 7904k buffers
Swap: 2008084k total, 0k used, 2008084k free, 74172k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9024 root 15 0 324m 13m 7072 S 1.7 1.3 0:01.00 X
7 root -2 0 0 0 0 S 0.3 0.0 0:01.02 softirq-block/0
9912 mark 15 0 10600 1304 960 R 0.3 0.1 0:00.01 top
1 root 15 0 2676 588 500 S 0.0 0.1 0:00.31 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.00 posix_cpu_timer
3 root -2 0 0 0 0 S 0.0 0.0 0:00.00 softirq-high/0
lightning ~ # lspci
00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3)
00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2)
00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3)
00:04.0 Multimedia audio controller: nVidia Corporation CK804 AC'97
Audio Controller (rev a2)
00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2)
00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:08.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev a2)
00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev a3)
00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0c.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0d.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Miscellaneous Control
01:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B60
[Radeon X300 (PCIE)]
01:00.1 Display controller: ATI Technologies Inc RV370 [Radeon X300SE]
05:06.0 Multimedia audio controller: Xilinx Corporation RME Hammerfall
DSP (rev 68)
05:08.0 FireWire (IEEE 1394): Texas Instruments TSB82AA2 IEEE-1394b
Link Layer Controller (rev 01)
lightning ~ #
lightning ~ # lsmod
Module Size Used by
it87 22032 0
hwmon_vid 2880 1 it87
i2c_isa 6144 1 it87
eeprom 7248 0
snd_seq_midi 7296 0
snd_pcm_oss 40160 0
snd_mixer_oss 15168 1 snd_pcm_oss
snd_seq_dummy 3460 0
snd_seq_oss 29696 0
snd_seq_midi_event 7040 2 snd_seq_midi,snd_seq_oss
snd_seq 48000 6
snd_seq_midi,snd_seq_dummy,snd_seq_oss,snd_seq_midi_event
realtime 9416 0
sbp2 22084 0
radeon 115488 2
drm 84392 3 radeon
snd_hdsp 47812 0
snd_rawmidi 21216 2 snd_seq_midi,snd_hdsp
snd_seq_device 7252 5
snd_seq_midi,snd_seq_dummy,snd_seq_oss,snd_seq,snd_rawmidi
snd_hwdep 8456 1 snd_hdsp
ohci1394 31368 0
ieee1394 91024 2 sbp2,ohci1394
snd_intel8x0 32296 1
snd_ac97_codec 107288 1 snd_intel8x0
ac97_bus 3712 1 snd_ac97_codec
snd_pcm 74440 4 snd_pcm_oss,snd_hdsp,snd_intel8x0,snd_ac97_codec
snd_timer 20424 2 snd_seq,snd_pcm
snd 52264 14
snd_pcm_oss,snd_mixer_oss,snd_seq_oss,snd_seq,snd_hdsp,snd_rawmidi,snd_seq_device,snd_hwdep,snd_intel8x0,snd_ac97_codec,snd_pcm,snd_timer
snd_page_alloc 8208 3 snd_hdsp,snd_intel8x0,snd_pcm
k8temp 5888 0
i2c_nforce2 6016 0
i2c_core 21056 4 it87,i2c_isa,eeprom,i2c_nforce2
lightning ~ #
* Mark Knecht <[email protected]> wrote:
> 23: 1739277 IO-APIC-fasteoi ohci_hcd:usb2, eth0
> 00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev a3)
ok, that's forcedeth. Does that behavior go away if you revert the
change below, via patch -p0 -R ?
but in general, -rt will track IRQ cpu usage accurately, while mainline
does not measure it at all when in idle. So a frequent IRQ on mainline
might be using up similar amounts of of CPU time as well, but it's not
reported.
Ingo
---
drivers/net/forcedeth.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
Index: linux-rt.q/drivers/net/forcedeth.c
===================================================================
--- linux-rt.q.orig/drivers/net/forcedeth.c
+++ linux-rt.q/drivers/net/forcedeth.c
@@ -800,7 +800,7 @@ struct fe_priv {
* Maximum number of loops until we assume that a bit in the irq mask
* is stuck. Overridable with module param.
*/
-static int max_interrupt_work = 5;
+static int max_interrupt_work = 50;
/*
* Optimization can be either throuput mode or cpu mode
@@ -812,7 +812,7 @@ enum {
NV_OPTIMIZATION_MODE_THROUGHPUT,
NV_OPTIMIZATION_MODE_CPU
};
-static int optimization_mode = NV_OPTIMIZATION_MODE_THROUGHPUT;
+static int optimization_mode = NV_OPTIMIZATION_MODE_CPU;
/*
* Poll interval for timer irq
@@ -3108,9 +3108,17 @@ static int nv_napi_poll(struct net_devic
int retcode;
if (np->desc_ver == DESC_VER_1 || np->desc_ver == DESC_VER_2) {
+ spin_lock_irqsave(&np->lock, flags);
+ nv_tx_done(dev);
+ spin_unlock_irqrestore(&np->lock, flags);
+
pkts = nv_rx_process(dev, limit);
retcode = nv_alloc_rx(dev);
} else {
+ spin_lock_irqsave(&np->lock, flags);
+ nv_tx_done_optimized(dev, np->tx_ring_size);
+ spin_unlock_irqrestore(&np->lock, flags);
+
pkts = nv_rx_process_optimized(dev, limit);
retcode = nv_alloc_rx_optimized(dev);
}
On 6/10/07, Ingo Molnar <[email protected]> wrote:
>
> * Mark Knecht <[email protected]> wrote:
>
> > 23: 1739277 IO-APIC-fasteoi ohci_hcd:usb2, eth0
>
> > 00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev a3)
>
> ok, that's forcedeth. Does that behavior go away if you revert the
> change below, via patch -p0 -R ?
>
> but in general, -rt will track IRQ cpu usage accurately, while mainline
> does not measure it at all when in idle. So a frequent IRQ on mainline
> might be using up similar amounts of of CPU time as well, but it's not
> reported.
>
> Ingo
>
Ingo,
GMail is being kranky. Can you please send the patch as a zipped attachment?
Thanks,
Mark
* Mark Knecht <[email protected]> wrote:
> GMail is being kranky. Can you please send the patch as a zipped
> attachment?
you can pick it up from:
http://people.redhat.com/mingo/realtime-preempt/testing/forcedeth-rt-tweak.patch
Ingo
On 6/10/07, Ingo Molnar <[email protected]> wrote:
>
> * Mark Knecht <[email protected]> wrote:
>
> > GMail is being kranky. Can you please send the patch as a zipped
> > attachment?
>
> you can pick it up from:
>
> http://people.redhat.com/mingo/realtime-preempt/testing/forcedeth-rt-tweak.patch
>
> Ingo
>
Hi Ingo,
Yep, that solved it on my machine. I did have to patch using 'patch
-p1 -R' I think that's because the paths in your patch file are maybe
based on your work setup? I don't know. I'm amazed most of the time
that I can even do this stuff but with your help it seems to work out.
Thanks.
Anyway, top no longer reports the 2.7% CPU usage from IRQ23. I do
see a little bit of CPU consumption from X and firefox but that seems
pretty reasonable.
If there is more you'd like me to look at or do on my end please let me know.
Cheers,
Mark
* Mark Knecht <[email protected]> wrote:
> Hi Ingo,
> Yep, that solved it on my machine. I did have to patch using 'patch
> -p1 -R' I think that's because the paths in your patch file are maybe
> based on your work setup? I don't know. I'm amazed most of the time
> that I can even do this stuff but with your help it seems to work out.
> Thanks.
>
> Anyway, top no longer reports the 2.7% CPU usage from IRQ23. I do
> see a little bit of CPU consumption from X and firefox but that seems
> pretty reasonable.
ok, thanks - i took that change out from -rt.
Ingo
On 6/11/07, Ingo Molnar <[email protected]> wrote:
>
> * Mark Knecht <[email protected]> wrote:
>
> > Hi Ingo,
> > Yep, that solved it on my machine. I did have to patch using 'patch
> > -p1 -R' I think that's because the paths in your patch file are maybe
> > based on your work setup? I don't know. I'm amazed most of the time
> > that I can even do this stuff but with your help it seems to work out.
> > Thanks.
> >
> > Anyway, top no longer reports the 2.7% CPU usage from IRQ23. I do
> > see a little bit of CPU consumption from X and firefox but that seems
> > pretty reasonable.
>
> ok, thanks - i took that change out from -rt.
>
> Ingo
>
Thanks Ingo. Have a great week.
- Mark