Hi!
Right after I compiled my first 2.6.31 for my server I got a crash, machine
stops at user level, network, keyboard, ... still work, seems to me that
only disk access crashes. I know 2.6.31.1 is out but the changelog didn't
seem to show anything related to this, I'll try to test it anyway just to be
sure.
It took me a while to get the crash to happen again and get a photo of the
screen which then I passed through gocr and then I corrected (sorry if there
are any typos left). It seems that the bug tends to happen on weekends or
so, as I had the machine working throughout the week without any problem and
then it crashed again on the next weekend, but I haven't identified any
special disk related jobs, other than the typical weekly jobs that the
distro (Debian) runs, which didn't seem relevant to me.
Machine is a Pentium III and had previously been running 2.6.30.5 without
any problem at all.
I don't know what else to add, so I leave here the trace I got, please don't
hesitate to contact me if you need any other info.
------------[ cut here ]------------
kernel BUG at drivers/ide/ide-disk.c:187!
invalid opcode; 0000 [#1]
last sysfs file: /sys/devices/pci0000:00/0000:00:0c.0/i2c-adapter/i2c-0/name
Modules linked in: gl518sm fue smbfs zd1201 tuner tea5767 tda8290 tuner_xc2028
snd_sbawe snd_opl3_lib snd_hwdep xc5000 tda9887 snd_sb16_dsp snd_sb_common tuner
_simple snd_mpu401_uart tuner_types mt20xx tea5761 snd_rawmidi snd_seq_device tv
audio snd_pcm snd_timer snd snd_page_alloc tda7432 msp3400 bttv ir_common i2c_al
go_bit v4l2_common uhci_hcd ehci_hcd ohci_hcd videodev v4l1_compat videobuf_dma_
sg videobuf_core btcx_risc tveeprom i2c_viapro i2c_core dl2k usbcore parport_pc
via_agp parport agpgart
Pid: 77, comm: kblockd/0 Not tainted (2.6.31 #1)
EIP: 0060:[<c0205bca>] EFLAGS: 00010206 CPU: 0
EIP is at ide_do_rw_disk+0x26/0x266
EAX: ef8d5e00 EBX: 000c2d3f ECX: 000c2d3f EDX: c7911834
ESI: ef9e3c00 EDI: 00000088 EBP: c7911834 ESP: ef90beb8
DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process kblockd/0 (pid: 77, ti=ef90a000 task=ef8feec0 task.ti=ef90a000)
Stack:
c019df97 c7911834 c791187c c01a5a42 ef8d5200 ef8d9f04 c7911834 ef8d9f04
<0> efb090e8 ef8d5200 c01a5b09 00000000 c01ab97d c01fea56 4088c800 00000000
<0> ef9e3c00 ef90bf78 c7911834 c01ff098 000004e2 ef90bf13 c02b2334 ef9e3c00
Call Trace:
[<c019df97>] ? elv_rb_del+0x20/0x2e
[<c01a5a42>] ? cfq_remove_request+0xbf/0x162
[<c01a5b09>] ? cfq_dispatch_insert+0x24/0x32
[<c01ab97d>] ? __const_udelay+0x15/0x16
[<c01fea56>] ? __ide_wait_stat+0x81/0xb0
[<c01ff098>] ? ide_wait_stat+0x3f/0x6f
[<c02055e9>] ? ide_gd_do_request+0x7/0x9
[<c01fe7f2>] ? do_ide_request+0x316/0x488
[<c010f865>] ? dequeue_task+x90/0x9e
[<c029d777>] ? schedule+0x2ad/0x2d9
[<c019f63a>] ? __blk_run_queue+0x39/0x60
[<c0la4f97>] ? cfq_kick_queue+0x0/0xb
[<c01a4fa0>] ? cfq_kick_queue+0x9/0xb
[<c011dd82>] ? worker_thread+0xae/0x11c
[<c0120354>] ? autoremove_wake_function+0x0/0x2d
[<c011dcd4>] ? worker_thread+0x0/0x11c
[<c0120084>] ? kthread+0x6b/0x70
[<c0120019>] ? kthread+0x0/0x70
[<c0102d43>] ? kernel_thread_helper+0x7/0x10
Code: 00 c3 31 c0 c3 55 57 56 89 c6 53 89 cb 83 ec 58 f6 46 2a 02 89 54 24 04 8b
40 20 74 04 0f 0b eb fe 8b 54 24 04 83 7a 28 01 74 04 <0f> 0b eb fe 8b 48 6c 85
c9 74 08 8b 54 24 04 89 f0 ff d1 8b 44
EIP: [<c0205bca>] ide_do_rw_disk+0x26/0x266 SS:ESP 0068:ef90beb8
---[ end trace 76b3e81fa9e97b6f ]---
Regards...
--
Manty/BestiaTester -> http://manty.net
(cc linux-ide)
On Wed, 30 Sep 2009 13:05:29 +0200 Santiago Garcia Mantinan <[email protected]> wrote:
> Hi!
>
> Right after I compiled my first 2.6.31 for my server I got a crash, machine
> stops at user level, network, keyboard, ... still work, seems to me that
> only disk access crashes. I know 2.6.31.1 is out but the changelog didn't
> seem to show anything related to this, I'll try to test it anyway just to be
> sure.
>
> It took me a while to get the crash to happen again and get a photo of the
> screen which then I passed through gocr and then I corrected (sorry if there
> are any typos left). It seems that the bug tends to happen on weekends or
> so, as I had the machine working throughout the week without any problem and
> then it crashed again on the next weekend, but I haven't identified any
> special disk related jobs, other than the typical weekly jobs that the
> distro (Debian) runs, which didn't seem relevant to me.
>
> Machine is a Pentium III and had previously been running 2.6.30.5 without
> any problem at all.
>
> I don't know what else to add, so I leave here the trace I got, please don't
> hesitate to contact me if you need any other info.
>
> ------------[ cut here ]------------
> kernel BUG at drivers/ide/ide-disk.c:187!
> invalid opcode; 0000 [#1]
> last sysfs file: /sys/devices/pci0000:00/0000:00:0c.0/i2c-adapter/i2c-0/name
> Modules linked in: gl518sm fue smbfs zd1201 tuner tea5767 tda8290 tuner_xc2028
> snd_sbawe snd_opl3_lib snd_hwdep xc5000 tda9887 snd_sb16_dsp snd_sb_common tuner
> _simple snd_mpu401_uart tuner_types mt20xx tea5761 snd_rawmidi snd_seq_device tv
> audio snd_pcm snd_timer snd snd_page_alloc tda7432 msp3400 bttv ir_common i2c_al
> go_bit v4l2_common uhci_hcd ehci_hcd ohci_hcd videodev v4l1_compat videobuf_dma_
> sg videobuf_core btcx_risc tveeprom i2c_viapro i2c_core dl2k usbcore parport_pc
> via_agp parport agpgart
>
> Pid: 77, comm: kblockd/0 Not tainted (2.6.31 #1)
> EIP: 0060:[<c0205bca>] EFLAGS: 00010206 CPU: 0
> EIP is at ide_do_rw_disk+0x26/0x266
> EAX: ef8d5e00 EBX: 000c2d3f ECX: 000c2d3f EDX: c7911834
> ESI: ef9e3c00 EDI: 00000088 EBP: c7911834 ESP: ef90beb8
> DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
> Process kblockd/0 (pid: 77, ti=ef90a000 task=ef8feec0 task.ti=ef90a000)
> Stack:
> c019df97 c7911834 c791187c c01a5a42 ef8d5200 ef8d9f04 c7911834 ef8d9f04
> <0> efb090e8 ef8d5200 c01a5b09 00000000 c01ab97d c01fea56 4088c800 00000000
> <0> ef9e3c00 ef90bf78 c7911834 c01ff098 000004e2 ef90bf13 c02b2334 ef9e3c00
> Call Trace:
> [<c019df97>] ? elv_rb_del+0x20/0x2e
> [<c01a5a42>] ? cfq_remove_request+0xbf/0x162
> [<c01a5b09>] ? cfq_dispatch_insert+0x24/0x32
> [<c01ab97d>] ? __const_udelay+0x15/0x16
> [<c01fea56>] ? __ide_wait_stat+0x81/0xb0
> [<c01ff098>] ? ide_wait_stat+0x3f/0x6f
> [<c02055e9>] ? ide_gd_do_request+0x7/0x9
> [<c01fe7f2>] ? do_ide_request+0x316/0x488
> [<c010f865>] ? dequeue_task+x90/0x9e
> [<c029d777>] ? schedule+0x2ad/0x2d9
> [<c019f63a>] ? __blk_run_queue+0x39/0x60
> [<c0la4f97>] ? cfq_kick_queue+0x0/0xb
> [<c01a4fa0>] ? cfq_kick_queue+0x9/0xb
> [<c011dd82>] ? worker_thread+0xae/0x11c
> [<c0120354>] ? autoremove_wake_function+0x0/0x2d
> [<c011dcd4>] ? worker_thread+0x0/0x11c
> [<c0120084>] ? kthread+0x6b/0x70
> [<c0120019>] ? kthread+0x0/0x70
> [<c0102d43>] ? kernel_thread_helper+0x7/0x10
> Code: 00 c3 31 c0 c3 55 57 56 89 c6 53 89 cb 83 ec 58 f6 46 2a 02 89 54 24 04 8b
> 40 20 74 04 0f 0b eb fe 8b 54 24 04 83 7a 28 01 74 04 <0f> 0b eb fe 8b 48 6c 85
> c9 74 08 8b 54 24 04 89 f0 ff d1 8b 44
> EIP: [<c0205bca>] ide_do_rw_disk+0x26/0x266 SS:ESP 0068:ef90beb8
> ---[ end trace 76b3e81fa9e97b6f ]---
Hi Manty,
Andrew Morton wrote:
> On Wed, 30 Sep 2009 13:05:29 +0200 Santiago Garcia Mantinan wrote:
>> kernel BUG at drivers/ide/ide-disk.c:187!
Looks like this is a deliberate test for unknown requests that was added in
2.6.31 with the following commit:
commit 2c7eaa43c3bb7b3b9fe2051d17f308c1f0728c78
Author: Bartlomiej Zolnierkiewicz <[email protected]>
Date: Mon Jun 15 22:16:10 2009 +0200
ide: BUG() on unknown requests
Unsupported requests should be never handed down to device drivers
and the best thing we can do upon discovering such request inside
driver's ->do_request method is to just BUG().
In previous kernels the code would not dump, but still fail and print the
request flags, identified by "ide_do_rw_disk - bad command", to the kernel
log.
Manty: Have you ever seen such messages with previous kernels?
Question for IDE maintainers: should maybe the old printing of request info
be reinstated, or can the request flags also be obtained from the BUG
info?
Cheers,
FJP
From: Frans Pop <[email protected]>
Date: Thu, 1 Oct 2009 10:26:14 +0200
> Question for IDE maintainers: should maybe the old printing of request info
> be reinstated, or can the request flags also be obtained from the BUG
> info?
Using a BUG for this doesn't make it any easier to track down the
problem. WARN_ON_ONCE() or similar is much more appropriate here.
BUG() is for situations where the system's state is completely
irrecoverably corrupted, and we cannot continue, and that is not the
case here at all.
On Thursday 01 October 2009 10:30:30 David Miller wrote:
> From: Frans Pop <[email protected]>
> Date: Thu, 1 Oct 2009 10:26:14 +0200
>
> > Question for IDE maintainers: should maybe the old printing of request info
> > be reinstated, or can the request flags also be obtained from the BUG
> > info?
>
> Using a BUG for this doesn't make it any easier to track down the
> problem. WARN_ON_ONCE() or similar is much more appropriate here.
>
> BUG() is for situations where the system's state is completely
> irrecoverably corrupted, and we cannot continue, and that is not the
> case here at all.
The problem is that you simply cannot know what is the system state here.
Thus when the unknown block layer request is encountered the best thing
you can do is to BUG early instead of allowing the situation when some
requests are silently dropped and possibly causing the data corruption.
> In previous kernels the code would not dump, but still fail and print the
> request flags, identified by "ide_do_rw_disk - bad command", to the kernel
> log.
> Manty: Have you ever seen such messages with previous kernels?
Not at all.
Regards...
--
Manty/BestiaTester -> http://manty.net
From: Bartlomiej Zolnierkiewicz <[email protected]>
Date: Thu, 1 Oct 2009 11:25:40 +0200
> The problem is that you simply cannot know what is the system state here.
>
> Thus when the unknown block layer request is encountered the best thing
> you can do is to BUG early instead of allowing the situation when some
> requests are silently dropped and possibly causing the data corruption.
Yes, but if you BUG() in this kind of location, the chance of getting
the debugging information from the user can be close to zero. We were
very lucky this time :-)
If we're tossing a request, signal an error to the submitter.
I hear we have infrastructure for that :-)
On Thursday 01 October 2009 18:40:34 David Miller wrote:
> From: Bartlomiej Zolnierkiewicz <[email protected]>
> Date: Thu, 1 Oct 2009 11:25:40 +0200
>
> > The problem is that you simply cannot know what is the system state here.
> >
> > Thus when the unknown block layer request is encountered the best thing
> > you can do is to BUG early instead of allowing the situation when some
> > requests are silently dropped and possibly causing the data corruption.
>
> Yes, but if you BUG() in this kind of location, the chance of getting
> the debugging information from the user can be close to zero. We were
> very lucky this time :-)
Do you mean that there is higher chance of user noticing some WARN_ON
warning somewhere in the log than OOPS? I don't quite believe it..
> If we're tossing a request, signal an error to the submitter.
>
> I hear we have infrastructure for that :-)
It has its own problems (see blk_execute_rq() overriding error values
for one of many examples)..
Anyway this is completely besides the point here (however please don't
let it stop you from fixing the aforementioned infrastructure) until
the whole issue gets debugged properly and I'd be quite happy to do it
under normal circumstances but since:
- you are always jumping the gun with your strong opinions before people
even had chance to debug the issue properly and find real root causes
(vide infamous sparc cmd64x problems, which were caused by the real
bugfixes and were completely fixed within 48h from the initial report)
- for the last three months you haven't debugged/fixed a single IDE issue
and you keep dodging every single bug-report
- I have neither time nor interest for this kind of silly corporate-style
games (I had some really good laugh few times though :)
- I really do not have to work on IDE (never had, it was always a hobby)
- I'm not the maintainer any longer :)
I simply do not even want to be cc:ed on IDE problems unless it is a paid
job or mail comes from the person who in the past proved the ability to
work well with others.
Thank you for understanding.
From: Bartlomiej Zolnierkiewicz <[email protected]>
Date: Thu, 1 Oct 2009 20:21:24 +0200
> - for the last three months you haven't debugged/fixed a single IDE issue
> and you keep dodging every single bug-report
I mostly do the same with networking, and I'm the maintainer there
too. :-)
And regardless of my workload, all reasonable patches and bug fixes
submitted for IDE flowed freely during this time, or were given
a review NACK.
I depend upon subsystem contributors to fix the bugs, that's how this
works. I merely mediate, review patches, apply patches, and chip in
with some work of my own from time to time as the situation and my
workload dictate.
Nobody is forcing you to fix bugs or work on anything, you can freely
ignore every IDE related email. Create a filter for it if you like.
:-)
From: Santiago Garcia Mantinan <[email protected]>
Date: Wed, 30 Sep 2009 13:05:29 +0200
> [<c010f865>] ? dequeue_task+x90/0x9e
> [<c029d777>] ? schedule+0x2ad/0x2d9
> [<c019f63a>] ? __blk_run_queue+0x39/0x60
> [<c0la4f97>] ? cfq_kick_queue+0x0/0xb
> [<c01a4fa0>] ? cfq_kick_queue+0x9/0xb
> [<c011dd82>] ? worker_thread+0xae/0x11c
So it does look like a normal block I/O request to the disk
going through the CFQ scheduler.
But ->cmd_type of the request is corrupted, but we have no
idea in what way.
Well, we know it's not a special request, because one layer
up the IDE I/O layer driver does special processing for
blk_special_request() by calling ide_special_rq().
I suspect the request structure has been freed already and
we're referencing free'd memory.
Please add this test patch and let us know what messages
you end up with in the logs. It won't BUG() any more,
so you have to watch for the messages.
Thanks!
-DaveM (the IDE bug dodger)
diff --git a/drivers/ide/ide-disk.c b/drivers/ide/ide-disk.c
index 7f87801..54b9dbc 100644
--- a/drivers/ide/ide-disk.c
+++ b/drivers/ide/ide-disk.c
@@ -184,7 +184,11 @@ static ide_startstop_t ide_do_rw_disk(ide_drive_t *drive, struct request *rq,
ide_hwif_t *hwif = drive->hwif;
BUG_ON(drive->dev_flags & IDE_DFLAG_BLOCKED);
- BUG_ON(!blk_fs_request(rq));
+ if (!blk_fs_request(rq)) {
+ pr_alert("IDE: Non-FS req in ide_do_rw_disk(), cmd_type %d\n",
+ rq->cmd_type);
+ ide_kill_rq(drive, rq);
+ }
ledtrig_ide_activity();
On Thursday 01 October 2009 20:34:18 David Miller wrote:
> From: Bartlomiej Zolnierkiewicz <[email protected]>
> Date: Thu, 1 Oct 2009 20:21:24 +0200
>
> > - for the last three months you haven't debugged/fixed a single IDE issue
> > and you keep dodging every single bug-report
>
> I mostly do the same with networking, and I'm the maintainer there
> too. :-)
>
> And regardless of my workload, all reasonable patches and bug fixes
> submitted for IDE flowed freely during this time, or were given
> a review NACK.
>
> I depend upon subsystem contributors to fix the bugs, that's how this
> works. I merely mediate, review patches, apply patches, and chip in
> with some work of my own from time to time as the situation and my
> workload dictate.
IIRC you were not very content with exactly this style of my management
of IDE some years ago.. However I'm really glad that it works for you
so well in the networking, and now in IDE too! :)
From: David Miller <[email protected]>
Date: Thu, 01 Oct 2009 11:47:55 -0700 (PDT)
> Please add this test patch and let us know what messages
> you end up with in the logs. It won't BUG() any more,
> so you have to watch for the messages.
Sorry, there's a small bug in the test patch, please use this one
instead.
diff --git a/drivers/ide/ide-disk.c b/drivers/ide/ide-disk.c
index 7f87801..6e0dfa9 100644
--- a/drivers/ide/ide-disk.c
+++ b/drivers/ide/ide-disk.c
@@ -184,7 +184,12 @@ static ide_startstop_t ide_do_rw_disk(ide_drive_t *drive, struct request *rq,
ide_hwif_t *hwif = drive->hwif;
BUG_ON(drive->dev_flags & IDE_DFLAG_BLOCKED);
- BUG_ON(!blk_fs_request(rq));
+ if (!blk_fs_request(rq)) {
+ pr_alert("IDE: Non-FS req in ide_do_rw_disk(), cmd_type %d\n",
+ rq->cmd_type);
+ ide_kill_rq(drive, rq);
+ return ide_stopped;
+ }
ledtrig_ide_activity();
> Sorry, there's a small bug in the test patch, please use this one
> instead.
Applied it to 2.6.31.1, I'll wait and see what the weekend tells us.
BTW, I forgot to include info on the controller, disks, ... on the report,
so here they go just in case they make any difference.
Uniform Multi-Platform E-IDE driver
via82cxxx 0000:00:07.1: VIA vt82c596b (rev 23) IDE UDMA66
via82cxxx 0000:00:07.1: IDE controller (0x1106:0x0571 rev 0x10)
via82cxxx 0000:00:07.1: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0xe800-0xe807
ide1: BM-DMA at 0xe808-0xe80f
Probing IDE interface ide0...
hda: HL-DT-STDVD-RAM GH22NP20, ATAPI CD/DVD-ROM drive
hdb: ST3160021A, ATA DISK drive
hda: host max PIO5 wanted PIO255(auto-tune) selected PIO4
hda: UDMA/66 mode selected
hdb: host max PIO5 wanted PIO255(auto-tune) selected PIO4
hdb: UDMA/66 mode selected
Probing IDE interface ide1...
hdc: ST3160021A, ATA DISK drive
hdc: host max PIO5 wanted PIO255(auto-tune) selected PIO4
hdc: UDMA/66 mode selected
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
ide-gd driver 1.18
hdb: max request size: 512KiB
hdb: Host Protected Area detected.
current capacity is 312579695 sectors (160040 MB)
native capacity is 312581808 sectors (160041 MB)
hdb: 312579695 sectors (160040 MB) w/2048KiB Cache, CHS=19457/255/63
hdb: cache flushes supported
hdb: hdb1 hdb2 hdb3
hdc: max request size: 512KiB
hdc: Host Protected Area detected.
current capacity is 312579695 sectors (160040 MB)
native capacity is 312581808 sectors (160041 MB)
hdc: 312579695 sectors (160040 MB) w/2048KiB Cache, CHS=19457/255/63
hdc: cache flushes supported
hdc: hdc1 hdc2 hdc3
ide-cd driver 5.00
ide-cd: hda: ATAPI 48X DVD-ROM DVD-R/RAM CD-R/RW drive, 2048kB Cache
Uniform CD-ROM driver Revision: 3.20
Regards...
--
Manty/BestiaTester -> http://manty.net
> Applied it to 2.6.31.1, I'll wait and see what the weekend tells us.
up 2 days, 8:27 and still nothing, maybe I shouldn't have applied it to .1
as some of the fixes there could be masking the problem and I should be
testing it on 2.6.31 :-?
What do you think about, that, should I compile a plain 2.6.31? or should I
give 2.6.31.1 some more time?
On the other hand when I tried to compile a kernel for a non IDE machine out
of the same 2.6.31.1 patched sources I found this error:
ERROR: "ide_kill_rq" [drivers/ide/ide-gd_mod.ko] undefined!
make[2]: *** [__modpost] Erro 1
make[1]: *** [modules] Erro 2
make[1]: Sa?ndo do directorio `/usr/src/linux-2.6.31'
After reverting the patch it compiled ok, of course, I was wondering if this
only affects the modules build and for the test we are having I have IDE in
the kernel, so no problem with that, or if that can affect the IDE kernel as
well and it may break because of the patch or something.
Regards...
--
Manty/BestiaTester -> http://manty.net
From: Santiago Garcia Mantinan <[email protected]>
Date: Mon, 5 Oct 2009 00:37:05 +0200
>> Applied it to 2.6.31.1, I'll wait and see what the weekend tells us.
>
> up 2 days, 8:27 and still nothing, maybe I shouldn't have applied it to .1
> as some of the fixes there could be masking the problem and I should be
> testing it on 2.6.31 :-?
If 2.6.31.1 fixes your bug, that would be good news :-)