Hi Linus,
This is the SLUB fix for m68k, which also applies to stable.
The following changes since commit 8e10cd74342c7f5ce259cceca36f6eba084f5d58:
Linus Torvalds (1):
Linux 2.6.39-rc5
are available in the git repository at:
master.kernel.org:/pub/scm/linux/kernel/git/geert/linux-m68k.git for-2.6.39
Michael Schmitz (1):
m68k/mm: Set all online nodes in N_NORMAL_MEMORY
arch/m68k/mm/motorola.c | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
Thanks for pulling!
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
On Thu, 28 Apr 2011, Geert Uytterhoeven wrote:
> Hi Linus,
>
> This is the SLUB fix for m68k, which also applies to stable.
>
> The following changes since commit 8e10cd74342c7f5ce259cceca36f6eba084f5d58:
> Linus Torvalds (1):
> Linux 2.6.39-rc5
>
> are available in the git repository at:
>
> master.kernel.org:/pub/scm/linux/kernel/git/geert/linux-m68k.git for-2.6.39
>
> Michael Schmitz (1):
> m68k/mm: Set all online nodes in N_NORMAL_MEMORY
>
> arch/m68k/mm/motorola.c | 2 ++
> 1 files changed, 2 insertions(+), 0 deletions(-)
>
Thanks for pushing this, Geert.
Since 4a5fa3590f09 ([PARISC] slub: fix panic with DISCONTIGMEM) from
2.6.39-rc4, you can't actually select slub on m68k without CONFIG_ADVANCED
and CONFIG_SINGLE_MEMORY_CHUNK because it otherwises defaults to
discontigmem.
James tested hppa64 with my N_NORMAL_MEMORY fix and found that it turned
an SMP box into UP. If you've tested slub on m68k without regressions,
then perhaps you'd like to add a "|| M68K" to CONFIG_SLUB?
On Thu, 2011-04-28 at 14:25 -0700, David Rientjes wrote:
> On Thu, 28 Apr 2011, Geert Uytterhoeven wrote:
>
> > Hi Linus,
> >
> > This is the SLUB fix for m68k, which also applies to stable.
> >
> > The following changes since commit 8e10cd74342c7f5ce259cceca36f6eba084f5d58:
> > Linus Torvalds (1):
> > Linux 2.6.39-rc5
> >
> > are available in the git repository at:
> >
> > master.kernel.org:/pub/scm/linux/kernel/git/geert/linux-m68k.git for-2.6.39
> >
> > Michael Schmitz (1):
> > m68k/mm: Set all online nodes in N_NORMAL_MEMORY
> >
> > arch/m68k/mm/motorola.c | 2 ++
> > 1 files changed, 2 insertions(+), 0 deletions(-)
> >
>
> Thanks for pushing this, Geert.
>
> Since 4a5fa3590f09 ([PARISC] slub: fix panic with DISCONTIGMEM) from
> 2.6.39-rc4, you can't actually select slub on m68k without CONFIG_ADVANCED
> and CONFIG_SINGLE_MEMORY_CHUNK because it otherwises defaults to
> discontigmem.
>
> James tested hppa64 with my N_NORMAL_MEMORY fix and found that it turned
> an SMP box into UP. If you've tested slub on m68k without regressions,
> then perhaps you'd like to add a "|| M68K" to CONFIG_SLUB?
To be honest, I really don't see that fixing it. As soon as you
allocate memory beyond range zero, you move onto a non-zero node as far
as slub is concerned, and that will oops.
I think what the N_NORMAL_MEMORY patch did is just make it take a whiile
before you start allocating from that range. Try executing a memory
balloon on the platform; that was how we first demonstrated the problem
on parisc.
James
On Thu, 28 Apr 2011, James Bottomley wrote:
> > Since 4a5fa3590f09 ([PARISC] slub: fix panic with DISCONTIGMEM) from
> > 2.6.39-rc4, you can't actually select slub on m68k without CONFIG_ADVANCED
> > and CONFIG_SINGLE_MEMORY_CHUNK because it otherwises defaults to
> > discontigmem.
> >
> > James tested hppa64 with my N_NORMAL_MEMORY fix and found that it turned
> > an SMP box into UP. If you've tested slub on m68k without regressions,
> > then perhaps you'd like to add a "|| M68K" to CONFIG_SLUB?
>
> To be honest, I really don't see that fixing it. As soon as you
> allocate memory beyond range zero, you move onto a non-zero node as far
> as slub is concerned, and that will oops.
>
Possible nodes are represented in slub with N_NORMAL_MEMORY, so the
kmem_cache_node structures are allocated and initialized based on this
nodemask. As long as the memory ranges map to nodes set in the nodemask,
this should be fine.
> I think what the N_NORMAL_MEMORY patch did is just make it take a whiile
> before you start allocating from that range. Try executing a memory
> balloon on the platform; that was how we first demonstrated the problem
> on parisc.
>
With parisc, you encountered an oops in add_partial() because the
kmem_cache_node structure for the memory range returned by page_to_nid()
was not allocated. init_kmem_cache_nodes() takes care of this for all
memory ranges set in N_NORMAL_MEMORY.
Adding Christoph and Pekka to the cc if there is additional concerns about
slub on this architecture.
On Thu, Apr 28, 2011 at 23:41, David Rientjes <[email protected]> wrote:
> On Thu, 28 Apr 2011, James Bottomley wrote:
>
>> > Since 4a5fa3590f09 ([PARISC] slub: fix panic with DISCONTIGMEM) from
>> > 2.6.39-rc4, you can't actually select slub on m68k without CONFIG_ADVANCED
>> > and CONFIG_SINGLE_MEMORY_CHUNK because it otherwises defaults to
>> > discontigmem.
>> >
>> > James tested hppa64 with my N_NORMAL_MEMORY fix and found that it turned
>> > an SMP box into UP. If you've tested slub on m68k without regressions,
>> > then perhaps you'd like to add a "|| M68K" to CONFIG_SLUB?
>>
>> To be honest, I really don't see that fixing it. As soon as you
>> allocate memory beyond range zero, you move onto a non-zero node as far
>> as slub is concerned, and that will oops.
>>
>
> Possible nodes are represented in slub with N_NORMAL_MEMORY, so the
> kmem_cache_node structures are allocated and initialized based on this
> nodemask. As long as the memory ranges map to nodes set in the nodemask,
> this should be fine.
>
>> I think what the N_NORMAL_MEMORY patch did is just make it take a whiile
>> before you start allocating from that range. Try executing a memory
>> balloon on the platform; that was how we first demonstrated the problem
>> on parisc.
>>
>
> With parisc, you encountered an oops in add_partial() because the
> kmem_cache_node structure for the memory range returned by page_to_nid()
> was not allocated. init_kmem_cache_nodes() takes care of this for all
> memory ranges set in N_NORMAL_MEMORY.
>
> Adding Christoph and Pekka to the cc if there is additional concerns about
> slub on this architecture.
My ARAnyM instance has
System Memory: 276480K
14 MB at 0x00000000 (ST-RAM)
256 MB at 0x01000000 (alternate RAM)
and 137800KIB of swap, and survived the following program just fine:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(int argc, char *argv[])
{
size_t size = 1048576;
size_t total = 0;
void *p;
while (size) {
p = malloc(size);
if (!p) {
printf("Failed to allocate %zu bytes\n", size);
size /= 2;
}
memset(p, 0xaa, size);
total += size;
printf("Using %zu / 0x%zx bytes of memory\n", total, total);
}
printf("Finished!\n");
return 0;
}
i.e. the OOM-killer just killed the program after it consumed all
available virtual
memory:
Out of memory: Kill process 1727 (malloctest) score 854 or sacrifice child
Killed process 1727 (malloctest) total-vm:361160kB, anon-rss:224164kB,
file-rss:0kB
malloctest: page allocation failure. order:0, mode:0x84d0
So SLUB really seems to work now.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Geert Uytterhoeven wrote:
> On Thu, Apr 28, 2011 at 23:41, David Rientjes <[email protected]> wrote:
>
>> On Thu, 28 Apr 2011, James Bottomley wrote:
>>
>>
>>
>>> I think what the N_NORMAL_MEMORY patch did is just make it take a whiile
>>> before you start allocating from that range. Try executing a memory
>>> balloon on the platform; that was how we first demonstrated the problem
>>> on parisc.
>>>
>>>
>> With parisc, you encountered an oops in add_partial() because the
>> kmem_cache_node structure for the memory range returned by page_to_nid()
>> was not allocated. init_kmem_cache_nodes() takes care of this for all
>> memory ranges set in N_NORMAL_MEMORY.
>>
>> Adding Christoph and Pekka to the cc if there is additional concerns about
>> slub on this architecture.
>>
>
> My ARAnyM instance has
>
> System Memory: 276480K
> 14 MB at 0x00000000 (ST-RAM)
> 256 MB at 0x01000000 (alternate RAM)
>
> and 137800KIB of swap, and survived the following program just fine:
>
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
>
> int main(int argc, char *argv[])
> {
> size_t size = 1048576;
> size_t total = 0;
> void *p;
>
> while (size) {
> p = malloc(size);
> if (!p) {
> printf("Failed to allocate %zu bytes\n", size);
> size /= 2;
> }
> memset(p, 0xaa, size);
> total += size;
> printf("Using %zu / 0x%zx bytes of memory\n", total, total);
> }
>
> printf("Finished!\n");
> return 0;
> }
>
> i.e. the OOM-killer just killed the program after it consumed all
> available virtual
> memory:
>
> Out of memory: Kill process 1727 (malloctest) score 854 or sacrifice child
> Killed process 1727 (malloctest) total-vm:361160kB, anon-rss:224164kB,
> file-rss:0kB
> malloctest: page allocation failure. order:0, mode:0x84d0
>
> So SLUB really seems to work now.
>
Forgot to mention what I did for tests, on all kernels that I could
actually boot: I ran slabinfo -l and slabinfo -T (saved the output in
case anyone wants to analyze that), any kernel that survived this was
considered good in the original bisect. The current fix was also tested
on the actual hardware.
There were quite a few kernels that initially booted but died on the
first slabinfo invocation. They invariably died at the e2fsck stage when
rebooted after that.
Using your test, ARAnyM, 14MB/128MB RAM and 134MB swap:
Out of memory: Kill process 1376 (malloctest) score 944 or sacrifice child
Killed process 1376 (malloctest) total-vm:272756kB, anon-rss:135232kB,
file-rss:0kB
Falcon CT60, 14MB/512MB no swap:
Out of memory: Kill process 8644 (malloctest) score 967 or sacrifice child
Killed process 8644 (malloctest) total-vm:512244kB, anon-rss:510088kB,
file-rss:284kB
HTH,
Michael
On Fri, 29 Apr 2011, Geert Uytterhoeven wrote:
> My ARAnyM instance has
>
> System Memory: 276480K
> 14 MB at 0x00000000 (ST-RAM)
> 256 MB at 0x01000000 (alternate RAM)
>
> and 137800KIB of swap, and survived the following program just fine:
>
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
>
> int main(int argc, char *argv[])
> {
> size_t size = 1048576;
> size_t total = 0;
> void *p;
>
> while (size) {
> p = malloc(size);
> if (!p) {
> printf("Failed to allocate %zu bytes\n", size);
> size /= 2;
> }
> memset(p, 0xaa, size);
> total += size;
> printf("Using %zu / 0x%zx bytes of memory\n", total, total);
> }
>
> printf("Finished!\n");
> return 0;
> }
>
> i.e. the OOM-killer just killed the program after it consumed all
> available virtual
> memory:
>
> Out of memory: Kill process 1727 (malloctest) score 854 or sacrifice child
> Killed process 1727 (malloctest) total-vm:361160kB, anon-rss:224164kB,
> file-rss:0kB
> malloctest: page allocation failure. order:0, mode:0x84d0
>
> So SLUB really seems to work now.
>
So we're in the unfortunate position where slub works fine for some
architectures with DISCONTIGMEM and not with others. It seems like the
problems originating on James' hppa aren't related to slab allocation at
all, though, so I'm wondering if we should rethink disallowing SLUB as it
sits in Linus' tree right now for everything that uses DISCONTINGMEM
without NUMA and not force them to enable CONFIG_BROKEN?
Perhaps change the kconfig entry to only block slub for parisc instead?
On Tue, 3 May 2011, David Rientjes wrote:
> So we're in the unfortunate position where slub works fine for some
> architectures with DISCONTIGMEM and not with others. It seems like the
> problems originating on James' hppa aren't related to slab allocation at
> all, though, so I'm wondering if we should rethink disallowing SLUB as it
> sits in Linus' tree right now for everything that uses DISCONTINGMEM
> without NUMA and not force them to enable CONFIG_BROKEN?
>
> Perhaps change the kconfig entry to only block slub for parisc instead?
As i have explained multiple times before: This is a generic issue with a
kernel configuration that has DISCONTIGMEM on and NUMA configured off.
Core code in various subsystems makes various assumptions in the !NUMA
case. F.e. page_to_nid(page) == 0. Slub is one of them.
DISCONTIGMEM works fine on !NUMA if it just has a single node which is 0.
But in James' hppa we have multiple nodes and thus a fundamental problem
with node 1 existing in a non NUMA environment. We then have a strange
mixture of NUMA nodes existing in DISCONTIGMEM code and the core code
assuming there are none. This can lead to numerous weird problems.
IMHO A config broken for DISCONTIG and !NUMA for arches that can actually
use multiple DISCONTIG nodes would be the proper thing.
On Thu, 2011-04-28 at 14:41 -0700, David Rientjes wrote:
> On Thu, 28 Apr 2011, James Bottomley wrote:
>
> > > Since 4a5fa3590f09 ([PARISC] slub: fix panic with DISCONTIGMEM) from
> > > 2.6.39-rc4, you can't actually select slub on m68k without CONFIG_ADVANCED
> > > and CONFIG_SINGLE_MEMORY_CHUNK because it otherwises defaults to
> > > discontigmem.
> > >
> > > James tested hppa64 with my N_NORMAL_MEMORY fix and found that it turned
> > > an SMP box into UP. If you've tested slub on m68k without regressions,
> > > then perhaps you'd like to add a "|| M68K" to CONFIG_SLUB?
> >
> > To be honest, I really don't see that fixing it. As soon as you
> > allocate memory beyond range zero, you move onto a non-zero node as far
> > as slub is concerned, and that will oops.
> >
>
> Possible nodes are represented in slub with N_NORMAL_MEMORY, so the
> kmem_cache_node structures are allocated and initialized based on this
> nodemask. As long as the memory ranges map to nodes set in the nodemask,
> this should be fine.
>
> > I think what the N_NORMAL_MEMORY patch did is just make it take a whiile
> > before you start allocating from that range. Try executing a memory
> > balloon on the platform; that was how we first demonstrated the problem
> > on parisc.
> >
>
> With parisc, you encountered an oops in add_partial() because the
> kmem_cache_node structure for the memory range returned by page_to_nid()
> was not allocated. init_kmem_cache_nodes() takes care of this for all
> memory ranges set in N_NORMAL_MEMORY.
Yes, but I also encountered it after I applied you patch, which is why I
still pushed the Kconfig patch. It's possible, since there were a huge
number of patches flying around that the kernel base was contaminated,
so I'll strip down to just linus HEAD + parisc coherence patches,
reverting the Kconfig one and try again.
James
On Wed, 4 May 2011, Christoph Lameter wrote:
> As i have explained multiple times before: This is a generic issue with a
> kernel configuration that has DISCONTIGMEM on and NUMA configured off.
> Core code in various subsystems makes various assumptions in the !NUMA
> case. F.e. page_to_nid(page) == 0. Slub is one of them.
>
Agreed, but James is still reporting that there is a slub oops with at
least parisc even with the N_NORMAL_MEMORY fix applied. He's currently
retrying on a clean kernel with the latest git (and presumably
CONFIG_BROKEN to enable CONFIG_SLUB). If he reports that slub is no
longer an issue with such a configuration, then I agree that we can remove
the Kconfig change for slub so it no longer forces CONFIG_NUMA for
CONFIG_DISCONTIGMEM (or CONFIG_BROKEN).
If that's the case, and we'll have to wait for James' report back first,
then we can probably make adjustments in the per-arch Kconfigs that enable
discontigmem without NUMA and then make it a strict requirement (or
CONFIG_BROKEN) until the additional problems are solved in other places in
the kernel.
On Wed, 4 May 2011, James Bottomley wrote:
> Yes, but I also encountered it after I applied you patch, which is why I
> still pushed the Kconfig patch. It's possible, since there were a huge
> number of patches flying around that the kernel base was contaminated,
> so I'll strip down to just linus HEAD + parisc coherence patches,
> reverting the Kconfig one and try again.
>
Great, and if that works out successfully this time around I think we'll
either need to fix each individual arch Kconfig that we know doesn't work
well (at least parisc because of the scheduling issue) so that it at least
enables CONFIG_NUMA implicitly for discontigmem unless CONFIG_BROKEN is
set.
The ideal solution is probably to rely on CONFIG_NEED_MULTIPLE_NODES
rather than CONFIG_NUMA, which is why it was introduced in the first place
since it was duplicating data structures for both NUMA and discontigmem.
That's apparently broken somewhere in the kernel that turned your SMP box
into an UP.
On Wed, 2011-05-04 at 12:07 -0700, David Rientjes wrote:
> On Wed, 4 May 2011, James Bottomley wrote:
>
> > Yes, but I also encountered it after I applied you patch, which is why I
> > still pushed the Kconfig patch. It's possible, since there were a huge
> > number of patches flying around that the kernel base was contaminated,
> > so I'll strip down to just linus HEAD + parisc coherence patches,
> > reverting the Kconfig one and try again.
> >
>
> Great, and if that works out successfully this time around I think we'll
> either need to fix each individual arch Kconfig that we know doesn't work
> well (at least parisc because of the scheduling issue) so that it at least
> enables CONFIG_NUMA implicitly for discontigmem unless CONFIG_BROKEN is
> set.
OK, I confirm that the N_NORMAL_MEMORY patch on its own fixes slub for
us. We can revert the mark slub BROKEN in DISCONTIGMEM && !NUMA patch.
> The ideal solution is probably to rely on CONFIG_NEED_MULTIPLE_NODES
> rather than CONFIG_NUMA, which is why it was introduced in the first place
> since it was duplicating data structures for both NUMA and discontigmem.
> That's apparently broken somewhere in the kernel that turned your SMP box
> into an UP.
Sure ... either that or accelerate a conversion to something like
SPARSEMEM.
James
On Mon, 9 May 2011, James Bottomley wrote:
> > Great, and if that works out successfully this time around I think we'll
> > either need to fix each individual arch Kconfig that we know doesn't work
> > well (at least parisc because of the scheduling issue) so that it at least
> > enables CONFIG_NUMA implicitly for discontigmem unless CONFIG_BROKEN is
> > set.
>
> OK, I confirm that the N_NORMAL_MEMORY patch on its own fixes slub for
> us. We can revert the mark slub BROKEN in DISCONTIGMEM && !NUMA patch.
>
Ok, so we need to revert 4a5fa3590f09 ([PARISC] slub: fix panic with
DISCONTIGMEM) that did not allow CONFIG_SLUB to be set for architectures
that use DISCONTIGMEM without NUMA support unless they have CONFIG_BROKEN
set from Linus' tree _and_ from the stable trees.
slub: Revert "[PARISC] slub: fix panic with DISCONTIGMEM"
4a5fa3590f09 ([PARISC] slub: fix panic with DISCONTIGMEM) did not allow
SLUB to be used on architectures that use DISCONTIGMEM without compiling
NUMA support without CONFIG_BROKEN also set.
The slub panic that it was intended to prevent is addressed by
d9b41e0b54fd ([PARISC] set memory ranges in N_NORMAL_MEMORY when onlined)
on parisc so there is no further slub issues with such a configuration.
This reverts the former commit so that SLUB may now be used on such
architectures since there haven't been any reports of additional errors.
Cc: James Bottomley <[email protected]>
Signed-off-by: David Rientjes <[email protected]>
---
init/Kconfig | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)
diff --git a/init/Kconfig b/init/Kconfig
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1226,7 +1226,6 @@ config SLAB
per cpu and per node queues.
config SLUB
- depends on BROKEN || NUMA || !DISCONTIGMEM
bool "SLUB (Unqueued Allocator)"
help
SLUB is a slab allocator that minimizes cache line usage
On Wed, May 11, 2011 at 02:08, David Rientjes <[email protected]> wrote:
> On Mon, 9 May 2011, James Bottomley wrote:
>> > Great, and if that works out successfully this time around I think we'll
>> > either need to fix each individual arch Kconfig that we know doesn't work
>> > well (at least parisc because of the scheduling issue) so that it at least
>> > enables CONFIG_NUMA implicitly for discontigmem unless CONFIG_BROKEN is
>> > set.
>>
>> OK, I confirm that the N_NORMAL_MEMORY patch on its own fixes slub for
>> us. We can revert the mark slub BROKEN in DISCONTIGMEM && !NUMA patch.
>>
>
> Ok, so we need to revert 4a5fa3590f09 ([PARISC] slub: fix panic with
> DISCONTIGMEM) that did not allow CONFIG_SLUB to be set for architectures
> that use DISCONTIGMEM without NUMA support unless they have CONFIG_BROKEN
> set from Linus' tree _and_ from the stable trees.
>
>
>
> slub: Revert "[PARISC] slub: fix panic with DISCONTIGMEM"
>
> 4a5fa3590f09 ([PARISC] slub: fix panic with DISCONTIGMEM) did not allow
> SLUB to be used on architectures that use DISCONTIGMEM without compiling
> NUMA support without CONFIG_BROKEN also set.
>
> The slub panic that it was intended to prevent is addressed by
> d9b41e0b54fd ([PARISC] set memory ranges in N_NORMAL_MEMORY when onlined)
> on parisc so there is no further slub issues with such a configuration.
Please also refer to the N_NORMAL_MEMORY fixes for the other arches,
esp. for stable.
> This reverts the former commit so that SLUB may now be used on such
> architectures since there haven't been any reports of additional errors.
>
> Cc: James Bottomley <[email protected]>
> Signed-off-by: David Rientjes <[email protected]>
Acked-by: Geert Uytterhoeven <[email protected]>
Cc: [email protected]
> ---
> init/Kconfig | 1 -
> 1 files changed, 0 insertions(+), 1 deletions(-)
>
> diff --git a/init/Kconfig b/init/Kconfig
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1226,7 +1226,6 @@ config SLAB
> per cpu and per node queues.
>
> config SLUB
> - depends on BROKEN || NUMA || !DISCONTIGMEM
> bool "SLUB (Unqueued Allocator)"
> help
> SLUB is a slab allocator that minimizes cache line usage
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds