The following patch changes the dequeueing to select a huge page near
the node executing instead of always beginning to check for free
nodes from node 0. This will result in a placement of the huge pages near
the executing processor improving performance.
The existing implementation can place the huge pages far away from
the executing processor causing significant degradation of performance.
The search starting from zero also means that the lower zones quickly
run out of memory. Selecting a huge page near the process distributed the
huge pages better.
Signed-off-by: Christoph Lameter <[email protected]>
Index: linux-2.6.14-mm1/mm/hugetlb.c
===================================================================
--- linux-2.6.14-mm1.orig/mm/hugetlb.c 2005-11-09 10:47:37.000000000 -0800
+++ linux-2.6.14-mm1/mm/hugetlb.c 2005-11-10 15:02:05.000000000 -0800
@@ -36,14 +36,16 @@ static struct page *dequeue_huge_page(vo
{
int nid = numa_node_id();
struct page *page = NULL;
+ struct zonelist *zonelist = NODE_DATA(nid)->node_zonelists;
+ struct zone **z;
- if (list_empty(&hugepage_freelists[nid])) {
- for (nid = 0; nid < MAX_NUMNODES; ++nid)
- if (!list_empty(&hugepage_freelists[nid]))
- break;
+ for (z = zonelist->zones; *z; z++) {
+ nid = (*z)->zone_pgdat->node_id;
+ if (!list_empty(&hugepage_freelists[nid]))
+ break;
}
- if (nid >= 0 && nid < MAX_NUMNODES &&
- !list_empty(&hugepage_freelists[nid])) {
+
+ if (z) {
page = list_entry(hugepage_freelists[nid].next,
struct page, lru);
list_del(&page->lru);
Christoph Lameter wrote on Thursday, November 10, 2005 3:27 PM
> The following patch changes the dequeueing to select a huge page near
> the node executing instead of always beginning to check for free
> nodes from node 0. This will result in a placement of the huge pages near
> the executing processor improving performance.
>
> The existing implementation can place the huge pages far away from
> the executing processor causing significant degradation of performance.
> The search starting from zero also means that the lower zones quickly
> run out of memory. Selecting a huge page near the process distributed the
> huge pages better.
Looks great!
- Ken
On Thu, 10 Nov 2005, Chen, Kenneth W wrote:
> Looks great!
Well in that case, we may do even more:
Make huge pages obey cpusets.
Signed-off-by: Christoph Lameter <[email protected]>
Index: linux-2.6.14-mm1/mm/hugetlb.c
===================================================================
--- linux-2.6.14-mm1.orig/mm/hugetlb.c 2005-11-10 15:02:05.000000000 -0800
+++ linux-2.6.14-mm1/mm/hugetlb.c 2005-11-10 16:29:16.000000000 -0800
@@ -11,6 +11,7 @@
#include <linux/highmem.h>
#include <linux/nodemask.h>
#include <linux/pagemap.h>
+#include <linux/cpuset.h>
#include <asm/page.h>
#include <asm/pgtable.h>
@@ -41,7 +42,8 @@ static struct page *dequeue_huge_page(vo
for (z = zonelist->zones; *z; z++) {
nid = (*z)->zone_pgdat->node_id;
- if (!list_empty(&hugepage_freelists[nid]))
+ if (cpuset_zone_allowed(*z, GFP_HIGHUSER) &&
+ !list_empty(&hugepage_freelists[nid]))
break;
}
On Thu, Nov 10, 2005 at 03:27:12PM -0800, Christoph Lameter wrote:
> The following patch changes the dequeueing to select a huge page near
> the node executing instead of always beginning to check for free
> nodes from node 0. This will result in a placement of the huge pages near
> the executing processor improving performance.
> The existing implementation can place the huge pages far away from
> the executing processor causing significant degradation of performance.
> The search starting from zero also means that the lower zones quickly
> run out of memory. Selecting a huge page near the process distributed the
> huge pages better.
> Signed-off-by: Christoph Lameter <[email protected]>
Long intended to have been corrected. Thanks.
Acked-by: William Irwin <[email protected]>
-- wli
On Thu, Nov 10, 2005 at 04:44:40PM -0800, Christoph Lameter wrote:
> Well in that case, we may do even more:
> Make huge pages obey cpusets.
> Signed-off-by: Christoph Lameter <[email protected]>
Simple enough.
Acked-by: William Irwin <[email protected]>
-- wli
On Thu, 2005-11-10 at 15:27 -0800, Christoph Lameter wrote:
> The following patch changes the dequeueing to select a huge page near
> the node executing instead of always beginning to check for free
> nodes from node 0. This will result in a placement of the huge pages near
> the executing processor improving performance.
>
> The existing implementation can place the huge pages far away from
> the executing processor causing significant degradation of performance.
> The search starting from zero also means that the lower zones quickly
> run out of memory. Selecting a huge page near the process distributed the
> huge pages better.
>
> Signed-off-by: Christoph Lameter <[email protected]>
I'll add my voice to the chorus of aye's.
--
Adam Litke - (agl at us.ibm.com)
IBM Linux Technology Center
On Fri, 11 Nov 2005, Adam Litke wrote:
> On Thu, 2005-11-10 at 15:27 -0800, Christoph Lameter wrote:
> > The following patch changes the dequeueing to select a huge page near
> > the node executing instead of always beginning to check for free
> > nodes from node 0. This will result in a placement of the huge pages near
> > the executing processor improving performance.
> >
> > The existing implementation can place the huge pages far away from
> > the executing processor causing significant degradation of performance.
> > The search starting from zero also means that the lower zones quickly
> > run out of memory. Selecting a huge page near the process distributed the
> > huge pages better.
> >
> > Signed-off-by: Christoph Lameter <[email protected]>
>
> I'll add my voice to the chorus of aye's.
There is a slight problem with the patch. We need to check *z instead of
z. Here is a fixed patch. Thanks to Paul T. Darga to point that out.
Index: linux-2.6.14-mm1/mm/hugetlb.c
===================================================================
--- linux-2.6.14-mm1.orig/mm/hugetlb.c 2005-11-09 10:47:37.000000000 -0800
+++ linux-2.6.14-mm1/mm/hugetlb.c 2005-11-11 09:31:02.000000000 -0800
@@ -36,14 +36,16 @@ static struct page *dequeue_huge_page(vo
{
int nid = numa_node_id();
struct page *page = NULL;
+ struct zonelist *zonelist = NODE_DATA(nid)->node_zonelists;
+ struct zone **z;
- if (list_empty(&hugepage_freelists[nid])) {
- for (nid = 0; nid < MAX_NUMNODES; ++nid)
- if (!list_empty(&hugepage_freelists[nid]))
- break;
+ for (z = zonelist->zones; *z; z++) {
+ nid = (*z)->zone_pgdat->node_id;
+ if (!list_empty(&hugepage_freelists[nid]))
+ break;
}
- if (nid >= 0 && nid < MAX_NUMNODES &&
- !list_empty(&hugepage_freelists[nid])) {
+
+ if (*z) {
page = list_entry(hugepage_freelists[nid].next,
struct page, lru);
list_del(&page->lru);