2005-10-03 15:05:04

by David Lang

[permalink] [raw]
Subject: Re: [PATCH 00/07][RFC] i386: NUMA emulation

On Mon, 3 Oct 2005, Martin J. Bligh wrote:

> But that's not the same at all! ;-) PAE memory is the same speed as
> the other stuff. You just have a 3rd level of pagetables for everything.
> One could (correctly) argue it made *all* memory slower, but it does so
> in a uniform fashion.

is it? I've seen during the memory self-test at boot that machines slow
down noticably as they pass the 4G mark.

David Lang

--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare


2005-10-03 15:09:14

by Martin Bligh

[permalink] [raw]
Subject: Re: [PATCH 00/07][RFC] i386: NUMA emulation



--David Lang <[email protected]> wrote (on Monday, October 03, 2005 08:03:44 -0700):

> On Mon, 3 Oct 2005, Martin J. Bligh wrote:
>
>> But that's not the same at all! ;-) PAE memory is the same speed as
>> the other stuff. You just have a 3rd level of pagetables for everything.
>> One could (correctly) argue it made *all* memory slower, but it does so
>> in a uniform fashion.
>
> is it? I've seen during the memory self-test at boot that machines slow down noticably as they pass the 4G mark.

Not noticed that, and I can't see why it should be the case in general,
though I suppose some machines might be odd. Got any numbers?

M.

2005-10-03 15:14:14

by David Lang

[permalink] [raw]
Subject: Re: [PATCH 00/07][RFC] i386: NUMA emulation

On Mon, 3 Oct 2005, Martin J. Bligh wrote:

> --David Lang <[email protected]> wrote (on Monday, October 03, 2005 08:03:44 -0700):
>
>> On Mon, 3 Oct 2005, Martin J. Bligh wrote:
>>
>>> But that's not the same at all! ;-) PAE memory is the same speed as
>>> the other stuff. You just have a 3rd level of pagetables for everything.
>>> One could (correctly) argue it made *all* memory slower, but it does so
>>> in a uniform fashion.
>>
>> is it? I've seen during the memory self-test at boot that machines slow down noticably as they pass the 4G mark.
>
> Not noticed that, and I can't see why it should be the case in general,
> though I suppose some machines might be odd. Got any numbers?

just the fact that the system boot memory test takes 3-4 times as long
with 8G or ram then with 4G of ram. I then boot a 64 bit kernel on the
system and never use PAE mode again :-)

if you can point me at a utility that will test the speed of the memory in
different chunks I'll do some testing on the Opteron systems I have
available. unfortunantly I don't have any Xeon systems to test this on.

David Lang

--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare

2005-10-03 15:25:28

by Martin Bligh

[permalink] [raw]
Subject: Re: [PATCH 00/07][RFC] i386: NUMA emulation



--David Lang <[email protected]> wrote (on Monday, October 03, 2005 08:13:09 -0700):

> On Mon, 3 Oct 2005, Martin J. Bligh wrote:
>
>> --David Lang <[email protected]> wrote (on Monday, October 03, 2005 08:03:44 -0700):
>>
>>> On Mon, 3 Oct 2005, Martin J. Bligh wrote:
>>>
>>>> But that's not the same at all! ;-) PAE memory is the same speed as
>>>> the other stuff. You just have a 3rd level of pagetables for everything.
>>>> One could (correctly) argue it made *all* memory slower, but it does so
>>>> in a uniform fashion.
>>>
>>> is it? I've seen during the memory self-test at boot that machines slow down noticably as they pass the 4G mark.
>>
>> Not noticed that, and I can't see why it should be the case in general,
>> though I suppose some machines might be odd. Got any numbers?
>
> just the fact that the system boot memory test takes 3-4 times as long with 8G or ram then with 4G of ram. I then boot a 64 bit kernel on the system and never use PAE mode again :-)
>
> if you can point me at a utility that will test the speed of the memory in different chunks I'll do some testing on the Opteron systems I have available. unfortunantly I don't have any Xeon systems to test this on.

Mmm. 64-bit uniproc systems, with > 4GB of RAM, running a 32 bit kernel
don't really strike me as a huge market segment ;-)

M.

2005-10-03 15:33:13

by David Lang

[permalink] [raw]
Subject: Re: [PATCH 00/07][RFC] i386: NUMA emulation

On Mon, 3 Oct 2005, Martin J. Bligh wrote:

> --David Lang <[email protected]> wrote (on Monday, October 03, 2005 08:13:09 -0700):
>
>> On Mon, 3 Oct 2005, Martin J. Bligh wrote:
>>
>>> --David Lang <[email protected]> wrote (on Monday, October 03, 2005 08:03:44 -0700):
>>>
>>>> On Mon, 3 Oct 2005, Martin J. Bligh wrote:
>>>>
>>>>> But that's not the same at all! ;-) PAE memory is the same speed as
>>>>> the other stuff. You just have a 3rd level of pagetables for everything.
>>>>> One could (correctly) argue it made *all* memory slower, but it does so
>>>>> in a uniform fashion.
>>>>
>>>> is it? I've seen during the memory self-test at boot that machines slow down noticably as they pass the 4G mark.
>>>
>>> Not noticed that, and I can't see why it should be the case in general,
>>> though I suppose some machines might be odd. Got any numbers?
>>
>> just the fact that the system boot memory test takes 3-4 times as long with 8G or ram then with 4G of ram. I then boot a 64 bit kernel on the system and never use PAE mode again :-)
>>
>> if you can point me at a utility that will test the speed of the memory in different chunks I'll do some testing on the Opteron systems I have available. unfortunantly I don't have any Xeon systems to test this on.
>
> Mmm. 64-bit uniproc systems, with > 4GB of RAM, running a 32 bit kernel
> don't really strike me as a huge market segment ;-)

true, but there are a lot of 32-bit uniproc systems sold by Intel that
have (or can have) more then 4G of ram. These are the machines I was
thinking of.

David Lang

--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare

2005-10-03 15:54:30

by Martin Bligh

[permalink] [raw]
Subject: Re: [PATCH 00/07][RFC] i386: NUMA emulation



--David Lang <[email protected]> wrote (on Monday, October 03, 2005 08:32:47 -0700):

> On Mon, 3 Oct 2005, Martin J. Bligh wrote:
>
>> --David Lang <[email protected]> wrote (on Monday, October 03, 2005 08:13:09 -0700):
>>
>>> On Mon, 3 Oct 2005, Martin J. Bligh wrote:
>>>
>>>> --David Lang <[email protected]> wrote (on Monday, October 03, 2005 08:03:44 -0700):
>>>>
>>>>> On Mon, 3 Oct 2005, Martin J. Bligh wrote:
>>>>>
>>>>>> But that's not the same at all! ;-) PAE memory is the same speed as
>>>>>> the other stuff. You just have a 3rd level of pagetables for everything.
>>>>>> One could (correctly) argue it made *all* memory slower, but it does so
>>>>>> in a uniform fashion.
>>>>>
>>>>> is it? I've seen during the memory self-test at boot that machines slow down noticably as they pass the 4G mark.
>>>>
>>>> Not noticed that, and I can't see why it should be the case in general,
>>>> though I suppose some machines might be odd. Got any numbers?
>>>
>>> just the fact that the system boot memory test takes 3-4 times as long with 8G or ram then with 4G of ram. I then boot a 64 bit kernel on the system and never use PAE mode again :-)
>>>
>>> if you can point me at a utility that will test the speed of the memory in different chunks I'll do some testing on the Opteron systems I have available. unfortunantly I don't have any Xeon systems to test this on.
>>
>> Mmm. 64-bit uniproc systems, with > 4GB of RAM, running a 32 bit kernel
>> don't really strike me as a huge market segment ;-)
>
> true, but there are a lot of 32-bit uniproc systems sold by Intel that have (or can have) more then 4G of ram. These are the machines I was thinking of.

Does your opteron box have more than 1 socket? that'd explain it.

Anyway, it shouldn't happen on any normal platform. Until we get
numbers that prove that it does (and understand why), I don't think
we need NUMA for PAE.

M.

2005-10-03 16:45:10

by David Lang

[permalink] [raw]
Subject: Re: [PATCH 00/07][RFC] i386: NUMA emulation

On Mon, 3 Oct 2005, Martin J. Bligh wrote:

>>>>>
>>>>> Not noticed that, and I can't see why it should be the case in general,
>>>>> though I suppose some machines might be odd. Got any numbers?
>>>>
>>>> just the fact that the system boot memory test takes 3-4 times as long with 8G or ram then with 4G of ram. I then boot a 64 bit kernel on the system and never use PAE mode again :-)
>>>>
>>>> if you can point me at a utility that will test the speed of the memory in different chunks I'll do some testing on the Opteron systems I have available. unfortunantly I don't have any Xeon systems to test this on.
>>>
>>> Mmm. 64-bit uniproc systems, with > 4GB of RAM, running a 32 bit kernel
>>> don't really strike me as a huge market segment ;-)
>>
>> true, but there are a lot of 32-bit uniproc systems sold by Intel that have (or can have) more then 4G of ram. These are the machines I was thinking of.
>
> Does your opteron box have more than 1 socket? that'd explain it.

yes, but I see the same 4G breakpoint no matter what the memory config
(including one dual proc machine with 16G, if it was a matter of hitting
memory connected to the other socket I would expect the slowdown at 8G,
not at 4G)

> Anyway, it shouldn't happen on any normal platform. Until we get
> numbers that prove that it does (and understand why), I don't think
> we need NUMA for PAE.

Ok, if nobody else is seeing any slowdown.

David Lang

--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare