2006-08-25 12:49:36

by Ulrich Windl

[permalink] [raw]
Subject: FYI: 2.6.16-smp: DMA memory inbalance for NUMA?

Hi,

my apologies if this is an issue already solved:
I have no idea what these messages exactly say, but for reasons of symmetry I
think there's something wrong:
For a Sun Fire X4100 with two Dual_Core Operon processors, the kernel (SLES10
kernel (x86_64, 2.6.16.21-0.15-smp)) says during boot:

<6>SRAT: PXM 0 -> APIC 0 -> Node 0
<6>SRAT: PXM 0 -> APIC 1 -> Node 0
<6>SRAT: PXM 1 -> APIC 2 -> Node 1
<6>SRAT: PXM 1 -> APIC 3 -> Node 1
<6>SRAT: Node 0 PXM 0 100000-f4000000
<6>SRAT: Node 1 PXM 1 20c000000-40c000000
<6>SRAT: Node 0 PXM 0 100000-20c000000
<6>SRAT: Node 0 PXM 0 0-20c000000

[[ Note: "Node 0" is mentioned three times, but "Node 1" is only mentioned once.
If this is intended to be some address assignments, they look quite odd to me.
Does the following really mean that only one node can do DMA (the other has zero
DMA pages)? ]]

<6>ACPI: SLIT table looks invalid. Not used.
<7>NUMA: Using 26 for the hash shift.
<6>Bootmem setup node 0 0000000000000000-000000020c000000
<6>Bootmem setup node 1 000000020c000000-000000040c000000
<7>On node 0 totalpages: 2066745
<7> DMA zone: 2993 pages, LIFO batch:0
<7> DMA32 zone: 981032 pages, LIFO batch:31
<7> Normal zone: 1082720 pages, LIFO batch:31
<7> HighMem zone: 0 pages, LIFO batch:0
<7>On node 1 totalpages: 2068480
<7> DMA zone: 0 pages, LIFO batch:0
<7> DMA32 zone: 0 pages, LIFO batch:0
<7> Normal zone: 2068480 pages, LIFO batch:31
<7> HighMem zone: 0 pages, LIFO batch:0

As usual, please CC: replies to my address as I'm not subscribed here.

Regards,
Ulrich


2006-08-25 23:16:15

by Robert Hancock

[permalink] [raw]
Subject: Re: FYI: 2.6.16-smp: DMA memory inbalance for NUMA?

Ulrich Windl wrote:
> Hi,
>
> my apologies if this is an issue already solved:
> I have no idea what these messages exactly say, but for reasons of symmetry I
> think there's something wrong:
> For a Sun Fire X4100 with two Dual_Core Operon processors, the kernel (SLES10
> kernel (x86_64, 2.6.16.21-0.15-smp)) says during boot:
>
> <6>SRAT: PXM 0 -> APIC 0 -> Node 0
> <6>SRAT: PXM 0 -> APIC 1 -> Node 0
> <6>SRAT: PXM 1 -> APIC 2 -> Node 1
> <6>SRAT: PXM 1 -> APIC 3 -> Node 1
> <6>SRAT: Node 0 PXM 0 100000-f4000000
> <6>SRAT: Node 1 PXM 1 20c000000-40c000000
> <6>SRAT: Node 0 PXM 0 100000-20c000000
> <6>SRAT: Node 0 PXM 0 0-20c000000
>
> [[ Note: "Node 0" is mentioned three times, but "Node 1" is only mentioned once.
> If this is intended to be some address assignments, they look quite odd to me.
> Does the following really mean that only one node can do DMA (the other has zero
> DMA pages)? ]]

DMA is not done only to the DMA zones. The DMA zone is for devices that
can only address 24-bit addresses (ISA/LPC devices). The DMA32 zone is
for devices that can only address 4GB of memory.

I suppose you could say there is a bit of an imbalance between nodes but
hopefully these are both (especially ZONE_DMA) rarely used in any sane
setup so it wouldn't make much difference. Since these zones are based
on bus-visible physical addresses I'm not sure if this could be fixed in
any case.

>
> <6>ACPI: SLIT table looks invalid. Not used.
> <7>NUMA: Using 26 for the hash shift.
> <6>Bootmem setup node 0 0000000000000000-000000020c000000
> <6>Bootmem setup node 1 000000020c000000-000000040c000000
> <7>On node 0 totalpages: 2066745
> <7> DMA zone: 2993 pages, LIFO batch:0
> <7> DMA32 zone: 981032 pages, LIFO batch:31
> <7> Normal zone: 1082720 pages, LIFO batch:31
> <7> HighMem zone: 0 pages, LIFO batch:0
> <7>On node 1 totalpages: 2068480
> <7> DMA zone: 0 pages, LIFO batch:0
> <7> DMA32 zone: 0 pages, LIFO batch:0
> <7> Normal zone: 2068480 pages, LIFO batch:31
> <7> HighMem zone: 0 pages, LIFO batch:0

--
Robert Hancock Saskatoon, SK, Canada
To email, remove "nospam" from [email protected]
Home Page: http://www.roberthancock.com/