Using rootfstype=ext4 with today's linux-2.6.git causes a panic on my
two amd64 machines (haven't tested it on any others). The steps to
reproduce go something like:
- add rootfstype=ext4 kernel param
- boot
- do some I/O
My root fs is actually ext3 (if that's relevant). Booting without
rootfstype=ext4 seems stable. Let me know if you need more info or patch
testing. This trace was obtained over netconsole:
[ 281.714039] ------------[ cut here ]------------
[ 281.714207] kernel BUG at mm/slub.c:2802!
[ 281.714366] invalid opcode: 0000 [#1] PREEMPT SMP
[ 281.714709] last sysfs file: /sys/class/net/lo/operstate
[ 281.714865] CPU 0
[ 281.715017] Modules linked in:
[ 281.715017] Pid: 0, comm: swapper Not tainted 2.6.30-rc2-diogenes7 #1 System Product Name
[ 281.715017] RIP: 0010:[<ffffffff802b3d4e>] [<ffffffff802b3d4e>] kfree+0x12e/0x140
[ 281.715017] RSP: 0018:ffff88000100ddf8 EFLAGS: 00010246
[ 281.715017] RAX: 0100000000080000 RBX: ffff88003e828908 RCX: 0000000000000000
[ 281.715017] RDX: ffffe20000000000 RSI: ffffe20000dac8c0 RDI: ffff88003e828908
[ 281.715017] RBP: ffffffff8041f885 R08: 0000000002202000 R09: 0000000000000001
[ 281.715017] R10: 0000000000000016 R11: ffffffff80224b60 R12: ffff88003e828908
[ 281.715017] R13: 00000000000000e7 R14: 0000000000000000 R15: ffff88003e828908
[ 281.715017] FS: 00007ffe72c136f0(0000) GS:ffff88000100a000(0000) knlGS:0000000000000000
[ 281.715017] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[ 281.715017] CR2: 00007ffe725c48f0 CR3: 000000003f2ac000 CR4: 00000000000006e0
[ 281.715017] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 281.715017] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 281.715017] Process swapper (pid: 0, threadinfo ffffffff806e2000, task ffffffff80688360)
[ 281.715017] Stack:
[ 281.715017] ffff88003e828908 ffff88003fb8a800 0000000000000000 ffffffff8041f885
[ 281.715017] 0000000000000050 ffff88003fb8a800 ffff88003e804d88 0000000000000000
[ 281.715017] 0000000000000050 ffff88003fb8a800 ffff88003e804d88 ffffffff80423801
[ 281.715017] Call Trace:
[ 281.715017] <IRQ> <0> [<ffffffff8041f885>] ? ide_complete_cmd+0x75/0x100
[ 281.715017] [<ffffffff80423801>] ? ide_finish_cmd+0x51/0xa0
[ 281.715017] [<ffffffff80424101>] ? task_no_data_intr+0xf1/0x170
[ 281.715017] [<ffffffff80424010>] ? task_no_data_intr+0x0/0x170
[ 281.715017] [<ffffffff8041f4ec>] ? ide_intr+0x1ec/0x250
[ 281.715017] [<ffffffff8026c3dd>] ? handle_IRQ_event+0xad/0x220
[ 281.715017] [<ffffffff8026e081>] ? handle_edge_irq+0xc1/0x160
[ 281.715017] [<ffffffff8020e167>] ? handle_irq+0x17/0x20
[ 281.715017] [<ffffffff8020d8d5>] ? do_IRQ+0x65/0xf0
[ 281.715017] [<ffffffff8020bd53>] ? ret_from_intr+0x0/0xa
[ 281.715017] <EOI> <0> [<ffffffff8021321b>] ? default_idle+0x9b/0x150
[ 281.715017] [<ffffffff802590f7>] ? notifier_call_chain+0x37/0x70
[ 281.715017] [<ffffffff8020a64a>] ? cpu_idle+0x5a/0xc0
[ 281.715017] [<ffffffff806ebc55>] ? start_kernel+0x342/0x408
[ 281.715017] [<ffffffff806eb378>] ? x86_64_start_kernel+0xe1/0xf2
[ 281.715017] Code: 14 49 8b 00 49 89 04 d4 4d 89 20 eb cc e8 3b 1f 2c 00 e9 57 ff ff ff 66 a9 00 c0 66 90 74 48 89 f7 fd ff <0f> 0b eb fe 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 81 ef
[ 281.715017] RIP [<ffffffff802b3d4e>] kfree+0x12e/0x140
[ 281.715017] RSP <ffff88000100ddf8>
[ 281.715017] ---[ end trace 0e7df544696dff6e ]---
[ 281.715017] Kernel panic - not syncing: Fatal exception in interrupt
[ 281.715017] Pid: 0, comm: swapper Tainted: G D 2.6.30-rc2-diogenes7 #1
[ 281.715017] Call Trace:
[ 281.715017] <IRQ> [<ffffffff80574c4a>] ? panic+0x95/0x152
[ 281.715017] [<ffffffff8020e371>] ? show_registers+0x91/0x2f0
[ 281.715017] [<ffffffff802596f9>] ? __atomic_notifier_call_chain+0x19/0x50
[ 281.715017] [<ffffffff80258f16>] ? up+0x16/0x50
[ 281.715017] [<ffffffff8023e195>] ? release_console_sem+0x1a5/0x1f0
[ 281.715017] [<ffffffff8020f77d>] ? oops_end+0x8d/0xa0
[ 281.715017] [<ffffffff8020d1c4>] ? do_invalid_op+0x84/0xa0
[ 281.715017] [<ffffffff802b3d4e>] ? kfree+0x12e/0x140
[ 281.715017] [<ffffffff80231350>] ? activate_task+0x40/0x70
[ 281.715017] [<ffffffff80239506>] ? try_to_wake_up+0x116/0x250
[ 281.715017] [<ffffffff8041f885>] ? ide_complete_cmd+0x75/0x100
[ 281.715017] [<ffffffff8020c145>] ? invalid_op+0x15/0x20
[ 281.715017] [<ffffffff8041f885>] ? ide_complete_cmd+0x75/0x100
[ 281.715017] [<ffffffff80224b60>] ? native_apic_mem_write+0x0/0x10
[ 281.715017] [<ffffffff802b3d4e>] ? kfree+0x12e/0x140
[ 281.715017] [<ffffffff802b3c9f>] ? kfree+0x7f/0x140
[ 281.715017] [<ffffffff8041f885>] ? ide_complete_cmd+0x75/0x100
[ 281.715017] [<ffffffff80423801>] ? ide_finish_cmd+0x51/0xa0
[ 281.715017] [<ffffffff80424101>] ? task_no_data_intr+0xf1/0x170
[ 281.715017] [<ffffffff80424010>] ? task_no_data_intr+0x0/0x170
[ 281.715017] [<ffffffff8041f4ec>] ? ide_intr+0x1ec/0x250
[ 281.715017] [<ffffffff8026c3dd>] ? handle_IRQ_event+0xad/0x220
[ 281.715017] [<ffffffff8026e081>] ? handle_edge_irq+0xc1/0x160
[ 281.715017] [<ffffffff8020e167>] ? handle_irq+0x17/0x20
[ 281.715017] [<ffffffff8020d8d5>] ? do_IRQ+0x65/0xf0
[ 281.715017] [<ffffffff8020bd53>] ? ret_from_intr+0x0/0xa
[ 281.715017] <EOI> [<ffffffff8021321b>] ? default_idle+0x9b/0x150
[ 281.715017] [<ffffffff802590f7>] ? notifier_call_chain+0x37/0x70
[ 281.715017] [<ffffffff8020a64a>] ? cpu_idle+0x5a/0xc0
[ 281.715017] [<ffffffff806ebc55>] ? start_kernel+0x342/0x408
[ 281.715017] [<ffffffff806eb378>] ? x86_64_start_kernel+0xe1/0xf2
--
Andrew Price
On Wed, Apr 15, 2009 at 09:59:26PM +0100, Andrew Price wrote:
> Using rootfstype=ext4 with today's linux-2.6.git causes a panic on my
> two amd64 machines (haven't tested it on any others).
Hmm, git commit id, please? The stack traces are in the IDE interrupt
handler, so it seems surprising that ext4 would trigger it but ext3
would not. Have you tried ext4 on any earlier kernel?
The main difference I can think of is that ext4 enables barriers by
default; maybe that's the case of the IDE breakage? Can you try
booting with the boot command option "rootfsflags=barrier=0" as well
as "rootfstype=ext4", and see if that helps?
If so, it's a bug in the IDE code in that it's not handling barriers
correctly.
- Ted
On Thu, Apr 16, 2009 at 12:19:45AM -0400, Theodore Tso wrote:
> On Wed, Apr 15, 2009 at 09:59:26PM +0100, Andrew Price wrote:
> > Using rootfstype=ext4 with today's linux-2.6.git causes a panic on my
> > two amd64 machines (haven't tested it on any others).
>
> Hmm, git commit id, please?
commit 3ee8da87ba6151ec91b2b8bbd27633bb248ea0d5
Merge: a2c252e 9dd175f
Author: Linus Torvalds <[email protected]>
Date: Wed Apr 15 09:11:11 2009 -0700
> The stack traces are in the IDE interrupt
> handler, so it seems surprising that ext4 would trigger it but ext3
> would not. Have you tried ext4 on any earlier kernel?
It happened with linux-2.6.git kernels earlier in the week when I
started trying rootfstype=ext4 but I haven't tried properly bisecting it
yet.
> The main difference I can think of is that ext4 enables barriers by
> default; maybe that's the case of the IDE breakage? Can you try
> booting with the boot command option "rootfsflags=barrier=0" as well
> as "rootfstype=ext4", and see if that helps?
I added rootflags=barrier=0 ...
(aside: the ext4 docs say the param is "barriers" with an s; it isn't).
... and it doesn't panic.
> If so, it's a bug in the IDE code in that it's not handling barriers
> correctly.
Bingo.
--
Andrew Price
On Thu, Apr 16, 2009 at 11:47:58AM +0100, Andrew Price wrote:
> On Thu, Apr 16, 2009 at 12:19:45AM -0400, Theodore Tso wrote:
> > The stack traces are in the IDE interrupt
> > handler, so it seems surprising that ext4 would trigger it but ext3
> > would not. Have you tried ext4 on any earlier kernel?
>
> It happened with linux-2.6.git kernels earlier in the week when I
> started trying rootfstype=ext4 but I haven't tried properly bisecting it
> yet.
>
> > The main difference I can think of is that ext4 enables barriers by
> > default; maybe that's the case of the IDE breakage? Can you try
> > booting with the boot command option "rootfsflags=barrier=0" as well
> > as "rootfstype=ext4", and see if that helps?
>
> I added rootflags=barrier=0 ...
>
> ... and it doesn't panic.
>
> > If so, it's a bug in the IDE code in that it's not handling barriers
> > correctly.
>
> Bingo.
OK, can you confirm that you tried using ext4 with either 2.6.29 or
2.6.28, and it was working previously? That would make it a
regression which Rafael would track.
Also, I'd suggest sending Bartolmiej details about what IDE controller
you have (the dmesg during a boot under ext3 or ext4 w/ barrier=0
would also be useful), so he can track it down. It might be specific
to a particular IDE controller/device, and not be a generalized IDE
problem, after all.
Anyway, I'll hand this bug off to Raefael's and/or Bartlomiej's very
capable hands, since I suspect you could also reproduce this using
rootfstype=ext3 rootfsflags=barrier=1, and that this is purely an IDE
driver problem.
Regards,
- Ted
> (aside: the ext4 docs say the param is "barriers" with an s; it isn't).
P.S. Where in the documentation file do we have the mount option as
"barriers=0". I went looking for it, and I couldn't find it. What I
see is:
barrier=<0|1(*)> This enables/disables the use of write barriers in
the jbd code. barrier=0 disables, barrier=1 enables.
This also requires an IO stack which can support
barriers, and if jbd gets an error on a barrier
write, it will disable again with a warning.
Write barriers enforce proper on-disk ordering
of journal commits, making volatile disk write caches
safe to use, at some performance penalty. If
your disks are battery-backed in one way or another,
disabling barriers may safely improve performance.
On Thursday 16 April 2009 16:53:57 Theodore Tso wrote:
> On Thu, Apr 16, 2009 at 11:47:58AM +0100, Andrew Price wrote:
> > On Thu, Apr 16, 2009 at 12:19:45AM -0400, Theodore Tso wrote:
> > > The stack traces are in the IDE interrupt
> > > handler, so it seems surprising that ext4 would trigger it but ext3
> > > would not. Have you tried ext4 on any earlier kernel?
> >
> > It happened with linux-2.6.git kernels earlier in the week when I
> > started trying rootfstype=ext4 but I haven't tried properly bisecting it
> > yet.
> >
> > > The main difference I can think of is that ext4 enables barriers by
> > > default; maybe that's the case of the IDE breakage? Can you try
> > > booting with the boot command option "rootfsflags=barrier=0" as well
> > > as "rootfstype=ext4", and see if that helps?
> >
> > I added rootflags=barrier=0 ...
> >
> > ... and it doesn't panic.
> >
> > > If so, it's a bug in the IDE code in that it's not handling barriers
> > > correctly.
> >
> > Bingo.
Freeing non-slab objects is bad.
Andrew, does this patch help?
---
drivers/ide/ide-io.c | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
Index: b/drivers/ide/ide-io.c
===================================================================
--- a/drivers/ide/ide-io.c
+++ b/drivers/ide/ide-io.c
@@ -102,11 +102,14 @@ void ide_complete_cmd(ide_drive_t *drive
drive->dev_flags |= IDE_DFLAG_PARKED;
}
- if (rq && rq->cmd_type == REQ_TYPE_ATA_TASKFILE)
- memcpy(rq->special, cmd, sizeof(*cmd));
+ if (rq && rq->cmd_type == REQ_TYPE_ATA_TASKFILE) {
+ struct ide_cmd *orig_cmd = rq->special;
- if (cmd->tf_flags & IDE_TFLAG_DYN)
- kfree(cmd);
+ if (cmd->tf_flags & IDE_TFLAG_DYN)
+ kfree(orig_cmd);
+ else
+ memcpy(orig_cmd, cmd, sizeof(*cmd));
+ }
}
/* obsolete, blk_rq_bytes() should be used instead */
On Thu, Apr 16, 2009 at 10:53:57AM -0400, Theodore Tso wrote:
> OK, can you confirm that you tried using ext4 with either 2.6.29 or
> 2.6.28, and it was working previously? That would make it a
> regression which Rafael would track.
I've tested this with a local GFS2 mount (which uses barriers too) and
the same bug occurs. Apologies for making it seem like an ext4 bug. I'm
currently waiting for a kernel to build with Bartlomiej's patch so we'll
see how that goes.
> Also, I'd suggest sending Bartolmiej details about what IDE controller
> you have (the dmesg during a boot under ext3 or ext4 w/ barrier=0
> would also be useful), so he can track it down. It might be specific
> to a particular IDE controller/device, and not be a generalized IDE
> problem, after all.
00:0f.1 IDE interface [0101]: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE [1106:0571] (rev 06) (prog-if 8a [Master SecP PriP])
Subsystem: ASUSTeK Computer Inc. A7V600/K8V-X/A8V Deluxe motherboard [1043:80ed]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 32
Interrupt: pin A routed to IRQ 20
Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable) [size=1]
Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable) [size=1]
Region 4: I/O ports at fc00 [size=16]
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Kernel driver in use: VIA_IDE
> Anyway, I'll hand this bug off to Raefael's and/or Bartlomiej's very
> capable hands, since I suspect you could also reproduce this using
> rootfstype=ext3 rootfsflags=barrier=1, and that this is purely an IDE
> driver problem.
Thanks Ted.
> P.S. Where in the documentation file do we have the mount option as
> "barriers=0". I went looking for it, and I couldn't find it. What I
> see is:
> <snip>
Further up in that file in the last paragraph of the 'Quick usage
instructions' it says:
[...]
'-o barriers=[0|1]' mount option for both ext3 and ext4 filesystems
[...]
(I searched the file for barrier and it was the first chunk of text
which matched.)
--
Andrew Price
On Thu, Apr 16, 2009 at 05:38:36PM +0100, Andrew Price wrote:
> On Thu, Apr 16, 2009 at 10:53:57AM -0400, Theodore Tso wrote:
> > OK, can you confirm that you tried using ext4 with either 2.6.29 or
> > 2.6.28, and it was working previously? That would make it a
> > regression which Rafael would track.
>
> I've tested this with a local GFS2 mount (which uses barriers too) and
> the same bug occurs. Apologies for making it seem like an ext4 bug. I'm
> currently waiting for a kernel to build with Bartlomiej's patch so we'll
> see how that goes.
No, no, that's fine. That sort of thing we get all the time; it's
part of trying to find the root cause of the bug. What I was more
interested in was trying to figure out when (at which kernel version)
the bug had introduced; if it's a recent regression then Rafael would
be interested in tracking it as such.
> Further up in that file in the last paragraph of the 'Quick usage
> instructions' it says:
>
> [...]
> '-o barriers=[0|1]' mount option for both ext3 and ext4 filesystems
> [...]
>
OK, got it. Thanks. I'll queue up a patch to fix that up.
- Ted
On Thu, Apr 16, 2009 at 06:02:21PM +0200, Bartlomiej Zolnierkiewicz wrote:
> Freeing non-slab objects is bad.
>
> Andrew, does this patch help?
Yes, that seems to fix it. I tested it with gfs2 and ext4 the usual way
and I couldn't reproduce the panic.
Thanks Bartlomiej.
> ---
> drivers/ide/ide-io.c | 11 +++++++----
> 1 file changed, 7 insertions(+), 4 deletions(-)
>
> Index: b/drivers/ide/ide-io.c
> ===================================================================
> --- a/drivers/ide/ide-io.c
> +++ b/drivers/ide/ide-io.c
> @@ -102,11 +102,14 @@ void ide_complete_cmd(ide_drive_t *drive
> drive->dev_flags |= IDE_DFLAG_PARKED;
> }
>
> - if (rq && rq->cmd_type == REQ_TYPE_ATA_TASKFILE)
> - memcpy(rq->special, cmd, sizeof(*cmd));
> + if (rq && rq->cmd_type == REQ_TYPE_ATA_TASKFILE) {
> + struct ide_cmd *orig_cmd = rq->special;
>
> - if (cmd->tf_flags & IDE_TFLAG_DYN)
> - kfree(cmd);
> + if (cmd->tf_flags & IDE_TFLAG_DYN)
> + kfree(orig_cmd);
> + else
> + memcpy(orig_cmd, cmd, sizeof(*cmd));
> + }
> }
>
> /* obsolete, blk_rq_bytes() should be used instead */
>
--
Andrew Price
On Thursday 16 April 2009 19:05:56 Andrew Price wrote:
> On Thu, Apr 16, 2009 at 06:02:21PM +0200, Bartlomiej Zolnierkiewicz wrote:
> > Freeing non-slab objects is bad.
> >
> > Andrew, does this patch help?
>
> Yes, that seems to fix it. I tested it with gfs2 and ext4 the usual way
> and I couldn't reproduce the panic.
>
> Thanks Bartlomiej.
Great, I'll queue the following patch for the next round of IDE fixes.
From: Bartlomiej Zolnierkiewicz <[email protected]>
Subject: [PATCH] ide: fix barriers support
Freeing non-slab objects is bad and results in an oops. Fix it.
Reported-and-tested-by: Andrew Price <[email protected]>
Cc: Theodore Tso <[email protected]>
Cc: "Rafael J. Wysocki" <[email protected]>
Signed-off-by: Bartlomiej Zolnierkiewicz <[email protected]>
---
drivers/ide/ide-io.c | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
Index: b/drivers/ide/ide-io.c
===================================================================
--- a/drivers/ide/ide-io.c
+++ b/drivers/ide/ide-io.c
@@ -102,11 +102,14 @@ void ide_complete_cmd(ide_drive_t *drive
drive->dev_flags |= IDE_DFLAG_PARKED;
}
- if (rq && rq->cmd_type == REQ_TYPE_ATA_TASKFILE)
- memcpy(rq->special, cmd, sizeof(*cmd));
+ if (rq && rq->cmd_type == REQ_TYPE_ATA_TASKFILE) {
+ struct ide_cmd *orig_cmd = rq->special;
- if (cmd->tf_flags & IDE_TFLAG_DYN)
- kfree(cmd);
+ if (cmd->tf_flags & IDE_TFLAG_DYN)
+ kfree(orig_cmd);
+ else
+ memcpy(orig_cmd, cmd, sizeof(*cmd));
+ }
}
/* obsolete, blk_rq_bytes() should be used instead */