Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933409Ab0GAVmj (ORCPT ); Thu, 1 Jul 2010 17:42:39 -0400 Received: from kroah.org ([198.145.64.141]:32906 "EHLO coco.kroah.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1759126Ab0GAVO0 (ORCPT ); Thu, 1 Jul 2010 17:14:26 -0400 X-Mailbox-Line: From gregkh@clark.site Thu Jul 1 10:42:56 2010 Message-Id: <20100701174256.555019192@clark.site> User-Agent: quilt/0.48-10.1 Date: Thu, 01 Jul 2010 10:43:30 -0700 From: Greg KH To: linux-kernel@vger.kernel.org, stable@kernel.org Cc: stable-review@kernel.org, torvalds@linux-foundation.org, akpm@linux-foundation.org, alan@lxorguk.ukuu.org.uk, Yanmin Zhang , Christoph Lameter , Alexander Duyck , Pekka Enberg Subject: [120/200] slub: move kmem_cache_node into its own cacheline In-Reply-To: <20100701175201.GA2149@kroah.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3525 Lines: 118 2.6.34-stable review patch. If anyone has any objections, please let me know. ------------------ From: Alexander Duyck commit 73367bd8eef4f4eb311005886aaa916013073265 upstream. This patch is meant to improve the performance of SLUB by moving the local kmem_cache_node lock into it's own cacheline separate from kmem_cache. This is accomplished by simply removing the local_node when NUMA is enabled. On my system with 2 nodes I saw around a 5% performance increase w/ hackbench times dropping from 6.2 seconds to 5.9 seconds on average. I suspect the performance gain would increase as the number of nodes increases, but I do not have the data to currently back that up. Bugzilla-Reference: http://bugzilla.kernel.org/show_bug.cgi?id=15713 Reported-by: Alex Shi Tested-by: Alex Shi Acked-by: Yanmin Zhang Acked-by: Christoph Lameter Signed-off-by: Alexander Duyck Signed-off-by: Pekka Enberg Signed-off-by: Greg Kroah-Hartman --- include/linux/slub_def.h | 9 +++------ mm/slub.c | 33 +++++++++++---------------------- 2 files changed, 14 insertions(+), 28 deletions(-) --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -75,12 +75,6 @@ struct kmem_cache { int offset; /* Free pointer offset. */ struct kmem_cache_order_objects oo; - /* - * Avoid an extra cache line for UP, SMP and for the node local to - * struct kmem_cache. - */ - struct kmem_cache_node local_node; - /* Allocation and freeing of slabs */ struct kmem_cache_order_objects max; struct kmem_cache_order_objects min; @@ -102,6 +96,9 @@ struct kmem_cache { */ int remote_node_defrag_ratio; struct kmem_cache_node *node[MAX_NUMNODES]; +#else + /* Avoid an extra cache line for UP */ + struct kmem_cache_node local_node; #endif }; --- a/mm/slub.c +++ b/mm/slub.c @@ -2141,7 +2141,7 @@ static void free_kmem_cache_nodes(struct for_each_node_state(node, N_NORMAL_MEMORY) { struct kmem_cache_node *n = s->node[node]; - if (n && n != &s->local_node) + if (n) kmem_cache_free(kmalloc_caches, n); s->node[node] = NULL; } @@ -2150,33 +2150,22 @@ static void free_kmem_cache_nodes(struct static int init_kmem_cache_nodes(struct kmem_cache *s, gfp_t gfpflags) { int node; - int local_node; - - if (slab_state >= UP && (s < kmalloc_caches || - s >= kmalloc_caches + KMALLOC_CACHES)) - local_node = page_to_nid(virt_to_page(s)); - else - local_node = 0; for_each_node_state(node, N_NORMAL_MEMORY) { struct kmem_cache_node *n; - if (local_node == node) - n = &s->local_node; - else { - if (slab_state == DOWN) { - early_kmem_cache_node_alloc(gfpflags, node); - continue; - } - n = kmem_cache_alloc_node(kmalloc_caches, - gfpflags, node); - - if (!n) { - free_kmem_cache_nodes(s); - return 0; - } + if (slab_state == DOWN) { + early_kmem_cache_node_alloc(gfpflags, node); + continue; + } + n = kmem_cache_alloc_node(kmalloc_caches, + gfpflags, node); + if (!n) { + free_kmem_cache_nodes(s); + return 0; } + s->node[node] = n; init_kmem_cache_node(n, s); } -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/