Is a fix for this going to make 2.6.31?
To replicate
cat /proc/kcore >/dev/null
See also
http://bugzilla.kernel.org/show_bug.cgi?id=13850
To get
BUG: unable to handle kernel paging request at eda08000
IP: [<c01a0e98>] read_kcore+0x27f/0x341
*pdpt = 0000000071809027
Oops: 0000 [#1] SMP
last sysfs file: /sys/class/net/lo/operstate
Pid: 2361, comm: cat Not tainted (2.6.31.git-x1 #1)
EIP: 0061:[<c01a0e98>] EFLAGS: 00010286 CPU: 1
EIP is at read_kcore+0x27f/0x341
EAX: c04404fc EBX: 00001000 ECX: 00000400 EDX: eda09000
ESI: eda08000 EDI: ec129000 EBP: ec60bf48 ESP: ec60bf10
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
Process cat (pid: 2361, ti=ec60a000 task=ecf800b0 task.ti=ec60a000)
Stack:
00001000 08051000 00000000 00001000 00000003 eda08000 ec129000 c04404fc
<0> eda09000 00000000 00001000 ecd39380 fffffffb c01a0c19 ec60bf6c c019a907
<0> ec60bf98 00001000 08051000 eccbab00 eccbab00 c019a8af 00001000 ec60bf8c
Call Trace:
[<c01a0c19>] ? read_kcore+0x0/0x341
[<c019a907>] ? proc_reg_read+0x58/0x6c
[<c019a8af>] ? proc_reg_read+0x0/0x6c
[<c016905e>] ? vfs_read+0x87/0x110
[<c0169180>] ? sys_read+0x3b/0x60
[<c0106a4d>] ? syscall_call+0x7/0xb
Code: d3 29 f3 89 f8 29 f0 39 d7 0f 46 d8 8d 0c 33 89 4d e8 29 5d ec 8b 45 e4 f6 40 0c 01 75 18 89 d9 c1 e9 02 89 f7 2b 7d dc 03 7d e0 <f3> a5 89 d9 83 e1 03 74 02 f3 a4 8b 55 e4 8b 12 89 55 e4 83 7d
EIP: [<c01a0e98>] read_kcore+0x27f/0x341 SS:ESP 0069:ec60bf10
CR2: 00000000eda08000
---[ end trace 4387f828fd1590eb ]---
I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT)
under Xen as a domU. The bugzilla report states it happens on non xen
machines also.
I know reading /proc/kcore isn't such a good idea, but badly written
backup scripts are triggering this on our customer's servers :-(
--
Nick Craig-Wood <[email protected]> -- http://www.craig-wood.com/nick
On Thu, Sep 3, 2009 at 2:34 PM, Nick Craig-Wood<[email protected]> wrote:
> Is a fix for this going to make 2.6.31?
>
> To replicate
>
> ?cat /proc/kcore >/dev/null
>
> See also
>
> ?http://bugzilla.kernel.org/show_bug.cgi?id=13850
>
> To get
>
> BUG: unable to handle kernel paging request at eda08000
> IP: [<c01a0e98>] read_kcore+0x27f/0x341
> *pdpt = 0000000071809027
> Oops: 0000 [#1] SMP
> last sysfs file: /sys/class/net/lo/operstate
>
> Pid: 2361, comm: cat Not tainted (2.6.31.git-x1 #1)
> EIP: 0061:[<c01a0e98>] EFLAGS: 00010286 CPU: 1
> EIP is at read_kcore+0x27f/0x341
> EAX: c04404fc EBX: 00001000 ECX: 00000400 EDX: eda09000
> ESI: eda08000 EDI: ec129000 EBP: ec60bf48 ESP: ec60bf10
> ?DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
> Process cat (pid: 2361, ti=ec60a000 task=ecf800b0 task.ti=ec60a000)
> Stack:
> ?00001000 08051000 00000000 00001000 00000003 eda08000 ec129000 c04404fc
> <0> eda09000 00000000 00001000 ecd39380 fffffffb c01a0c19 ec60bf6c c019a907
> <0> ec60bf98 00001000 08051000 eccbab00 eccbab00 c019a8af 00001000 ec60bf8c
> Call Trace:
> ?[<c01a0c19>] ? read_kcore+0x0/0x341
> ?[<c019a907>] ? proc_reg_read+0x58/0x6c
> ?[<c019a8af>] ? proc_reg_read+0x0/0x6c
> ?[<c016905e>] ? vfs_read+0x87/0x110
> ?[<c0169180>] ? sys_read+0x3b/0x60
> ?[<c0106a4d>] ? syscall_call+0x7/0xb
> Code: d3 29 f3 89 f8 29 f0 39 d7 0f 46 d8 8d 0c 33 89 4d e8 29 5d ec 8b 45 e4 f6 40 0c 01 75 18 89 d9 c1 e9 02 89 f7 2b 7d dc 03 7d e0 <f3> a5 89 d9 83 e1 03 74 02 f3 a4 8b 55 e4 8b 12 89 55 e4 83 7d
> EIP: [<c01a0e98>] read_kcore+0x27f/0x341 SS:ESP 0069:ec60bf10
> CR2: 00000000eda08000
> ---[ end trace 4387f828fd1590eb ]---
>
> I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT)
> under Xen as a domU. ?The bugzilla report states it happens on non xen
> machines also.
>
> I know reading /proc/kcore isn't such a good idea, but badly written
> backup scripts are triggering this on our customer's servers :-(
AFAICT the bug was fixed but I can't seem to find the patches in
Linus' git either. Lets CC Andrew and Hiroyuki-san.
Pekka
On Thu, 3 Sep 2009 15:14:55 +0300
Pekka Enberg <[email protected]> wrote:
> On Thu, Sep 3, 2009 at 2:34 PM, Nick Craig-Wood<[email protected]> wrote:
> > Is a fix for this going to make 2.6.31?
> >
> > To replicate
> >
> > cat /proc/kcore >/dev/null
> >
> > See also
> >
> > http://bugzilla.kernel.org/show_bug.cgi?id=13850
> >
> > To get
> >
> > BUG: unable to handle kernel paging request at eda08000
> > IP: [<c01a0e98>] read_kcore+0x27f/0x341
> > *pdpt = 0000000071809027
> > Oops: 0000 [#1] SMP
> > last sysfs file: /sys/class/net/lo/operstate
> >
> > Pid: 2361, comm: cat Not tainted (2.6.31.git-x1 #1)
> > EIP: 0061:[<c01a0e98>] EFLAGS: 00010286 CPU: 1
> > EIP is at read_kcore+0x27f/0x341
> > EAX: c04404fc EBX: 00001000 ECX: 00000400 EDX: eda09000
> > ESI: eda08000 EDI: ec129000 EBP: ec60bf48 ESP: ec60bf10
> > DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
> > Process cat (pid: 2361, ti=ec60a000 task=ecf800b0 task.ti=ec60a000)
> > Stack:
> > 00001000 08051000 00000000 00001000 00000003 eda08000 ec129000 c04404fc
> > <0> eda09000 00000000 00001000 ecd39380 fffffffb c01a0c19 ec60bf6c c019a907
> > <0> ec60bf98 00001000 08051000 eccbab00 eccbab00 c019a8af 00001000 ec60bf8c
> > Call Trace:
> > [<c01a0c19>] ? read_kcore+0x0/0x341
> > [<c019a907>] ? proc_reg_read+0x58/0x6c
> > [<c019a8af>] ? proc_reg_read+0x0/0x6c
> > [<c016905e>] ? vfs_read+0x87/0x110
> > [<c0169180>] ? sys_read+0x3b/0x60
> > [<c0106a4d>] ? syscall_call+0x7/0xb
> > Code: d3 29 f3 89 f8 29 f0 39 d7 0f 46 d8 8d 0c 33 89 4d e8 29 5d ec 8b 45 e4 f6 40 0c 01 75 18 89 d9 c1 e9 02 89 f7 2b 7d dc 03 7d e0 <f3> a5 89 d9 83 e1 03 74 02 f3 a4 8b 55 e4 8b 12 89 55 e4 83 7d
> > EIP: [<c01a0e98>] read_kcore+0x27f/0x341 SS:ESP 0069:ec60bf10
> > CR2: 00000000eda08000
> > ---[ end trace 4387f828fd1590eb ]---
> >
> > I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT)
> > under Xen as a domU. The bugzilla report states it happens on non xen
> > machines also.
> >
> > I know reading /proc/kcore isn't such a good idea, but badly written
> > backup scripts are triggering this on our customer's servers :-(
>
> AFAICT the bug was fixed but I can't seem to find the patches in
> Linus' git either. Lets CC Andrew and Hiroyuki-san.
>
Ah, it's now tested under mmotm. please wait.
Thanks,
-Kame
On Fri, Sep 04, 2009 at 09:06:42AM +0900, KAMEZAWA Hiroyuki wrote:
> On Thu, 3 Sep 2009 15:14:55 +0300
> Pekka Enberg <[email protected]> wrote:
> > On Thu, Sep 3, 2009 at 2:34 PM, Nick Craig-Wood<[email protected]> wrote:
> > > Is a fix for this going to make 2.6.31?
> > >
> > > To replicate
> > >
> > > ?cat /proc/kcore >/dev/null
> > >
> > > See also
> > >
> > > ?http://bugzilla.kernel.org/show_bug.cgi?id=13850
[snip]
> > > I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT)
> > > under Xen as a domU. ?The bugzilla report states it happens on non xen
> > > machines also.
> > >
> > > I know reading /proc/kcore isn't such a good idea, but badly written
> > > backup scripts are triggering this on our customer's servers :-(
> >
> > AFAICT the bug was fixed but I can't seem to find the patches in
> > Linus' git either. Lets CC Andrew and Hiroyuki-san.
>
> Ah, it's now tested under mmotm. please wait.
I tried mmotm but I couldn't get it to boot under Xen :-(
If you send me a patch against latest git I'm willing to test it (I
tried to extract the relevant patch from mmotm but failed dismally)
Thanks
Nick
--
Nick Craig-Wood <[email protected]> -- http://www.craig-wood.com/nick
On Fri, 4 Sep 2009 10:00:39 +0100
Nick Craig-Wood <[email protected]> wrote:
> On Fri, Sep 04, 2009 at 09:06:42AM +0900, KAMEZAWA Hiroyuki wrote:
> > On Thu, 3 Sep 2009 15:14:55 +0300
> > Pekka Enberg <[email protected]> wrote:
> > > On Thu, Sep 3, 2009 at 2:34 PM, Nick Craig-Wood<[email protected]> wrote:
> > > > Is a fix for this going to make 2.6.31?
> > > >
> > > > To replicate
> > > >
> > > > cat /proc/kcore >/dev/null
> > > >
> > > > See also
> > > >
> > > > http://bugzilla.kernel.org/show_bug.cgi?id=13850
> [snip]
> > > > I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT)
> > > > under Xen as a domU. The bugzilla report states it happens on non xen
> > > > machines also.
> > > >
> > > > I know reading /proc/kcore isn't such a good idea, but badly written
> > > > backup scripts are triggering this on our customer's servers :-(
> > >
> > > AFAICT the bug was fixed but I can't seem to find the patches in
> > > Linus' git either. Lets CC Andrew and Hiroyuki-san.
> >
> > Ah, it's now tested under mmotm. please wait.
>
> I tried mmotm but I couldn't get it to boot under Xen :-(
>
> If you send me a patch against latest git I'm willing to test it (I
> tried to extract the relevant patch from mmotm but failed dismally)
>
> Thanks
>
If you already downloaded mmotm, use these patches.
vmalloc-unmap-vmalloc-area-after-hiding-it.patch
kcore-fix-vread-vwrite-to-be-aware-of-holes.patch
kcore-fix-vread-vwrite-to-be-aware-of-holes-update.patch
kcore-proc-kcore-should-use-vread.patch
All I tested was x86-32/x86-64. then more tests are welcomed.
-Kame
> Nick
> --
> Nick Craig-Wood <[email protected]> -- http://www.craig-wood.com/nick
>
On Fri, Sep 04, 2009 at 06:16:45PM +0900, KAMEZAWA Hiroyuki wrote:
> On Fri, 4 Sep 2009 10:00:39 +0100
> Nick Craig-Wood <[email protected]> wrote:
>
> > On Fri, Sep 04, 2009 at 09:06:42AM +0900, KAMEZAWA Hiroyuki wrote:
> > > On Thu, 3 Sep 2009 15:14:55 +0300
> > > Pekka Enberg <[email protected]> wrote:
> > > > On Thu, Sep 3, 2009 at 2:34 PM, Nick Craig-Wood<[email protected]> wrote:
> > > > > Is a fix for this going to make 2.6.31?
> > > > >
> > > > > To replicate
> > > > >
> > > > > ?cat /proc/kcore >/dev/null
> > > > >
> > > > > See also
> > > > >
> > > > > ?http://bugzilla.kernel.org/show_bug.cgi?id=13850
> > [snip]
> > > > > I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT)
> > > > > under Xen as a domU. ?The bugzilla report states it happens on non xen
> > > > > machines also.
> > > > >
> > > > > I know reading /proc/kcore isn't such a good idea, but badly written
> > > > > backup scripts are triggering this on our customer's servers :-(
> > > >
> > > > AFAICT the bug was fixed but I can't seem to find the patches in
> > > > Linus' git either. Lets CC Andrew and Hiroyuki-san.
> > >
> > > Ah, it's now tested under mmotm. please wait.
> >
> > I tried mmotm but I couldn't get it to boot under Xen :-(
> >
> > If you send me a patch against latest git I'm willing to test it (I
> > tried to extract the relevant patch from mmotm but failed dismally)
>
> If you already downloaded mmotm, use these patches.
>
> vmalloc-unmap-vmalloc-area-after-hiding-it.patch
> kcore-fix-vread-vwrite-to-be-aware-of-holes.patch
> kcore-fix-vread-vwrite-to-be-aware-of-holes-update.patch
> kcore-proc-kcore-should-use-vread.patch
>
> All I tested was x86-32/x86-64. then more tests are welcomed.
I can confirm that these patches fix the problem on x86-32 running
under Xen DomU - thank you very much.
--
Nick Craig-Wood <[email protected]> -- http://www.craig-wood.com/nick
On Fri, 4 Sep 2009 18:16:45 +0900 KAMEZAWA Hiroyuki <[email protected]> wrote:
> On Fri, 4 Sep 2009 10:00:39 +0100
> Nick Craig-Wood <[email protected]> wrote:
>
> > On Fri, Sep 04, 2009 at 09:06:42AM +0900, KAMEZAWA Hiroyuki wrote:
> > > On Thu, 3 Sep 2009 15:14:55 +0300
> > > Pekka Enberg <[email protected]> wrote:
> > > > On Thu, Sep 3, 2009 at 2:34 PM, Nick Craig-Wood<[email protected]> wrote:
> > > > > Is a fix for this going to make 2.6.31?
> > > > >
> > > > > To replicate
> > > > >
> > > > > __cat /proc/kcore >/dev/null
> > > > >
> > > > > See also
> > > > >
> > > > > __http://bugzilla.kernel.org/show_bug.cgi?id=13850
> > [snip]
> > > > > I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT)
> > > > > under Xen as a domU. __The bugzilla report states it happens on non xen
> > > > > machines also.
Is this a regression? I've lost track..
> > > > > I know reading /proc/kcore isn't such a good idea, but badly written
> > > > > backup scripts are triggering this on our customer's servers :-(
> > > >
> > > > AFAICT the bug was fixed but I can't seem to find the patches in
> > > > Linus' git either. Lets CC Andrew and Hiroyuki-san.
> > >
> > > Ah, it's now tested under mmotm. please wait.
> >
> > I tried mmotm but I couldn't get it to boot under Xen :-(
> >
> > If you send me a patch against latest git I'm willing to test it (I
> > tried to extract the relevant patch from mmotm but failed dismally)
> >
> > Thanks
> >
>
> If you already downloaded mmotm, use these patches.
>
> vmalloc-unmap-vmalloc-area-after-hiding-it.patch
> kcore-fix-vread-vwrite-to-be-aware-of-holes.patch
> kcore-fix-vread-vwrite-to-be-aware-of-holes-update.patch
> kcore-proc-kcore-should-use-vread.patch
>
> All I tested was x86-32/x86-64. then more tests are welcomed.
That's a lot of stuff for 2.6.31. Is there some simple quickfix we can do?
On Fri, 4 Sep 2009 10:31:56 -0700
Andrew Morton <[email protected]> wrote:
> On Fri, 4 Sep 2009 18:16:45 +0900 KAMEZAWA Hiroyuki <[email protected]> wrote:
>
> > On Fri, 4 Sep 2009 10:00:39 +0100
> > Nick Craig-Wood <[email protected]> wrote:
> >
> > > On Fri, Sep 04, 2009 at 09:06:42AM +0900, KAMEZAWA Hiroyuki wrote:
> > > > On Thu, 3 Sep 2009 15:14:55 +0300
> > > > Pekka Enberg <[email protected]> wrote:
> > > > > On Thu, Sep 3, 2009 at 2:34 PM, Nick Craig-Wood<[email protected]> wrote:
> > > > > > Is a fix for this going to make 2.6.31?
> > > > > >
> > > > > > To replicate
> > > > > >
> > > > > > __cat /proc/kcore >/dev/null
> > > > > >
> > > > > > See also
> > > > > >
> > > > > > __http://bugzilla.kernel.org/show_bug.cgi?id=13850
> > > [snip]
> > > > > > I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT)
> > > > > > under Xen as a domU. __The bugzilla report states it happens on non xen
> > > > > > machines also.
>
> Is this a regression? I've lost track..
>
regression. (this comes from new per-cpu area implemantation, which uses
vmalloc area with memory holes.)
> > > > > > I know reading /proc/kcore isn't such a good idea, but badly written
> > > > > > backup scripts are triggering this on our customer's servers :-(
> > > > >
> > > > > AFAICT the bug was fixed but I can't seem to find the patches in
> > > > > Linus' git either. Lets CC Andrew and Hiroyuki-san.
> > > >
> > > > Ah, it's now tested under mmotm. please wait.
> > >
> > > I tried mmotm but I couldn't get it to boot under Xen :-(
> > >
> > > If you send me a patch against latest git I'm willing to test it (I
> > > tried to extract the relevant patch from mmotm but failed dismally)
> > >
> > > Thanks
> > >
> >
> > If you already downloaded mmotm, use these patches.
> >
> > vmalloc-unmap-vmalloc-area-after-hiding-it.patch
> > kcore-fix-vread-vwrite-to-be-aware-of-holes.patch
> > kcore-fix-vread-vwrite-to-be-aware-of-holes-update.patch
> > kcore-proc-kcore-should-use-vread.patch
> >
> > All I tested was x86-32/x86-64. then more tests are welcomed.
>
> That's a lot of stuff for 2.6.31. Is there some simple quickfix we can do?
>
I don't like this patch. But here, my 1st version which wasn't sent out.
-Kame
==
/proc/kcore may access in memory holes in vmalloc area. skip it
by using copy_from_user().
Signed-off-by: KAMEZAWA Hiroyuki <[email protected]>
---
fs/proc/kcore.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
Index: linux-2.6.31-rc8/fs/proc/kcore.c
===================================================================
--- linux-2.6.31-rc8.orig/fs/proc/kcore.c
+++ linux-2.6.31-rc8/fs/proc/kcore.c
@@ -361,7 +361,9 @@ read_kcore(struct file *file, char __use
/* don't dump ioremap'd stuff! (TA) */
if (m->flags & VM_IOREMAP)
continue;
- memcpy(elf_buf + (vmstart - start),
+ /* we may access memory holes */
+ __copy_from_user_inatomic(
+ elf_buf + (vmstart - start),
(char *)vmstart, vmsize);
}
read_unlock(&vmlist_lock);
On Mon, 7 Sep 2009 09:38:56 +0900
KAMEZAWA Hiroyuki <[email protected]> wrote:
> > That's a lot of stuff for 2.6.31. Is there some simple quickfix we can do?
> >
> I don't like this patch. But here, my 1st version which wasn't sent out.
> -Kame
Fixed warnings. (I confirmed this patch works well on x86-64.)
==
To access vmalloc area which may have memory holes, copy_from_user is
useful. By this,
# cat /proc/kcore > /dev/null
will not panic.
Signed-off-by: KAMEZAWA Hiroyuki <[email protected]>
---
fs/proc/kcore.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
Index: linux-2.6.31-rc8/fs/proc/kcore.c
===================================================================
--- linux-2.6.31-rc8.orig/fs/proc/kcore.c
+++ linux-2.6.31-rc8/fs/proc/kcore.c
@@ -361,7 +361,13 @@ read_kcore(struct file *file, char __use
/* don't dump ioremap'd stuff! (TA) */
if (m->flags & VM_IOREMAP)
continue;
- memcpy(elf_buf + (vmstart - start),
+ /*
+ * we may access memory holes, then use
+ * ex_table. checking return value just for
+ * avoid warnings.
+ */
+ vmsize = __copy_from_user_inatomic(
+ elf_buf + (vmstart - start),
(char *)vmstart, vmsize);
}
read_unlock(&vmlist_lock);
On Mon, Sep 07, 2009 at 03:14:22PM +0900, KAMEZAWA Hiroyuki wrote:
> On Mon, 7 Sep 2009 09:38:56 +0900
> KAMEZAWA Hiroyuki <[email protected]> wrote:
> > > That's a lot of stuff for 2.6.31. Is there some simple quickfix we can do?
> > >
> > I don't like this patch. But here, my 1st version which wasn't sent out.
> > -Kame
>
> Fixed warnings. (I confirmed this patch works well on x86-64.)
> ==
> To access vmalloc area which may have memory holes, copy_from_user is
> useful. By this,
> # cat /proc/kcore > /dev/null
> will not panic.
>
> Signed-off-by: KAMEZAWA Hiroyuki <[email protected]>
> ---
> fs/proc/kcore.c | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> Index: linux-2.6.31-rc8/fs/proc/kcore.c
> ===================================================================
> --- linux-2.6.31-rc8.orig/fs/proc/kcore.c
> +++ linux-2.6.31-rc8/fs/proc/kcore.c
> @@ -361,7 +361,13 @@ read_kcore(struct file *file, char __use
> /* don't dump ioremap'd stuff! (TA) */
> if (m->flags & VM_IOREMAP)
> continue;
> - memcpy(elf_buf + (vmstart - start),
> + /*
> + * we may access memory holes, then use
> + * ex_table. checking return value just for
> + * avoid warnings.
> + */
> + vmsize = __copy_from_user_inatomic(
> + elf_buf + (vmstart - start),
> (char *)vmstart, vmsize);
> }
> read_unlock(&vmlist_lock);
I gave this patch a thrashing and it works fine for me too (Xen domU x86)
It would be great if we could have this quick fix for 2.6.31.
Perhaps it should also go out to 2.6.30-stable also?
I can confirm that this patch applies to 2.6.30.5 and fixes the problem there too.
--
Nick Craig-Wood <[email protected]> -- http://www.craig-wood.com/nick