Hello,
For the sake of an old prototype peripheral I'm using a non PAE 32 bit
x86 kernel and I'm having trouble accessing memory above 2 GB. The
system has 4GB installed and all is well with a PAE kernel.
I'm obviously expecting to lose some memory due to memory mapped devices
but I wasn't expecting to lose 2GB. Instead I'm suspecting a BIOS bug.
The system reports:
free -m
total used free shared buffers
cached
Mem: 2012 491 1521 0 40
277
The mtrr table looked odd so I enabled sanitisation:
[ 0.000000] original variable MTRRs
[ 0.000000] reg 0, base: 2GB, range: 2GB, type UC
[ 0.000000] reg 1, base: 0GB, range: 4GB, type WB
[ 0.000000] reg 2, base: 4GB, range: 2GB, type WB
[ 0.000000] total RAM covered: 4096M
[ 0.000000] Found optimal setting for mtrr clean up
[ 0.000000] gran_size: 64K chunk_size: 64K num_reg: 2
lose cover RAM: 0G
[ 0.000000] New variable MTRRs
[ 0.000000] reg 0, base: 0GB, range: 2GB, type WB
[ 0.000000] reg 1, base: 4GB, range: 2GB, type WB
I don't understand the gap in the new table.
The motherboard is an Asus P5K with an Intel? P35 chipset. I'm using the
ubuntu lucid kernel (2.6.32) but the problem is also present with the
ubuntu precise kernel.
I'm at the limit of my understanding, can anyone advise how to debug
further?
Thanks,
Simon
On 06/12/2013 06:54 PM, Simon Brown wrote:
> Hello,
>
> For the sake of an old prototype peripheral I'm using a non PAE 32 bit
> x86 kernel and I'm having trouble accessing memory above 2 GB. The
> system has 4GB installed and all is well with a PAE kernel.
>
I have also 3 or 4 32-bit boxes running with more than 2Gb, I will try to
give you some ideas...
> I'm obviously expecting to lose some memory due to memory mapped devices
> but I wasn't expecting to lose 2GB. Instead I'm suspecting a BIOS bug.
> The system reports:
> free -m
> total used free shared buffers
> cached
> Mem: 2012 491 1521 0 40
> 277
>
> The mtrr table looked odd so I enabled sanitisation:
> [ 0.000000] original variable MTRRs
> [ 0.000000] reg 0, base: 2GB, range: 2GB, type UC
> [ 0.000000] reg 1, base: 0GB, range: 4GB, type WB
> [ 0.000000] reg 2, base: 4GB, range: 2GB, type WB
> [ 0.000000] total RAM covered: 4096M
> [ 0.000000] Found optimal setting for mtrr clean up
> [ 0.000000] gran_size: 64K chunk_size: 64K num_reg: 2
> lose cover RAM: 0G
> [ 0.000000] New variable MTRRs
> [ 0.000000] reg 0, base: 0GB, range: 2GB, type WB
> [ 0.000000] reg 1, base: 4GB, range: 2GB, type WB
>
> I don't understand the gap in the new table.
>
MTRR is set initially like:
- 4GB at address 0x, from which the last 2GB are unusable = 2GB cacheable
- 2GB more starting at address 4GB, normally cacheable
After the cleanup, the 4-2 part is cleaned and left as just 2:
- 2GB at 0
- a hole from 2GB to 4GB
- 2GB at 4Gb
It looks like your mobo BIOS maps the second GB pair up from address 4GB,
so you can not use them without PAE. If you find an option
in your BIOS to force it to map them under 4GB, the a normal kernel
with CONFIG_HIGHMEM4G=y will see it.
Look in BIOS for something like 'endable/disable continous/discrete mtrr'
or 'harware memory hole' or something like that.
> The motherboard is an Asus P5K with an Intel® P35 chipset. I'm using the
> ubuntu lucid kernel (2.6.32) but the problem is also present with the
> ubuntu precise kernel.
> I'm at the limit of my understanding, can anyone advise how to debug
> further?
>
--
J.A. Magallon <jamagallon()ono!com> \ Winter is coming...
On 06/12/2013 12:54 PM, Simon Brown wrote:
> Hello,
>
> For the sake of an old prototype peripheral I'm using a non PAE 32 bit
> x86 kernel and I'm having trouble accessing memory above 2 GB. The
> system has 4GB installed and all is well with a PAE kernel.
>
> I'm obviously expecting to lose some memory due to memory mapped devices
> but I wasn't expecting to lose 2GB. Instead I'm suspecting a BIOS bug.
> The system reports:
> free -m
> total used free shared buffers
> cached
> Mem: 2012 491 1521 0 40
> 277
>
> The mtrr table looked odd so I enabled sanitisation:
> [ 0.000000] original variable MTRRs
> [ 0.000000] reg 0, base: 2GB, range: 2GB, type UC
> [ 0.000000] reg 1, base: 0GB, range: 4GB, type WB
> [ 0.000000] reg 2, base: 4GB, range: 2GB, type WB
> [ 0.000000] total RAM covered: 4096M
> [ 0.000000] Found optimal setting for mtrr clean up
> [ 0.000000] gran_size: 64K chunk_size: 64K num_reg: 2
> lose cover RAM: 0G
> [ 0.000000] New variable MTRRs
> [ 0.000000] reg 0, base: 0GB, range: 2GB, type WB
> [ 0.000000] reg 1, base: 4GB, range: 2GB, type WB
>
> I don't understand the gap in the new table.
Check the e820 table. Chances are the BIOS is reserving 2GB to
map various devices (especially video cards) below the 4GB limit.
--
All rights reversed
On 06/12/2013 10:10 PM, Rik van Riel wrote:
> On 06/12/2013 12:54 PM, Simon Brown wrote:
>> Hello,
>>
>> For the sake of an old prototype peripheral I'm using a non PAE 32 bit
>> x86 kernel and I'm having trouble accessing memory above 2 GB. The
>> system has 4GB installed and all is well with a PAE kernel.
>>
>> I'm obviously expecting to lose some memory due to memory mapped devices
>> but I wasn't expecting to lose 2GB. Instead I'm suspecting a BIOS bug.
>> The system reports:
>> free -m
>> total used free shared buffers
>> cached
>> Mem: 2012 491 1521 0 40
>> 277
>>
>> The mtrr table looked odd so I enabled sanitisation:
>> [ 0.000000] original variable MTRRs
>> [ 0.000000] reg 0, base: 2GB, range: 2GB, type UC
>> [ 0.000000] reg 1, base: 0GB, range: 4GB, type WB
>> [ 0.000000] reg 2, base: 4GB, range: 2GB, type WB
>> [ 0.000000] total RAM covered: 4096M
>> [ 0.000000] Found optimal setting for mtrr clean up
>> [ 0.000000] gran_size: 64K chunk_size: 64K num_reg: 2
>> lose cover RAM: 0G
>> [ 0.000000] New variable MTRRs
>> [ 0.000000] reg 0, base: 0GB, range: 2GB, type WB
>> [ 0.000000] reg 1, base: 4GB, range: 2GB, type WB
>>
>> I don't understand the gap in the new table.
>
> Check the e820 table. Chances are the BIOS is reserving 2GB to
> map various devices (especially video cards) below the 4GB limit.
>
>
Acording to manual, that mobo has an option to "Memory remap feature"
in BIOS that looks like that...
--
J.A. Magallon <jamagallon()ono!com> \ Winter is coming...
On Wednesday 12 Jun 2013 16:10:03 Rik van Riel wrote:
> On 06/12/2013 12:54 PM, Simon Brown wrote:
> > Hello,
> >
> > For the sake of an old prototype peripheral I'm using a non PAE 32 bit
> > x86 kernel and I'm having trouble accessing memory above 2 GB. The
> > system has 4GB installed and all is well with a PAE kernel.
> >
> > I'm obviously expecting to lose some memory due to memory mapped devices
> > but I wasn't expecting to lose 2GB. Instead I'm suspecting a BIOS bug.
> > The system reports:
> > free -m
> >
> > total used free shared buffers
> > cached
> >
> > Mem: 2012 491 1521 0 40
> > 277
> >
> > The mtrr table looked odd so I enabled sanitisation:
> > [ 0.000000] original variable MTRRs
> > [ 0.000000] reg 0, base: 2GB, range: 2GB, type UC
> > [ 0.000000] reg 1, base: 0GB, range: 4GB, type WB
> > [ 0.000000] reg 2, base: 4GB, range: 2GB, type WB
> > [ 0.000000] total RAM covered: 4096M
> > [ 0.000000] Found optimal setting for mtrr clean up
> > [ 0.000000] gran_size: 64K chunk_size: 64K num_reg: 2
> > lose cover RAM: 0G
> > [ 0.000000] New variable MTRRs
> > [ 0.000000] reg 0, base: 0GB, range: 2GB, type WB
> > [ 0.000000] reg 1, base: 4GB, range: 2GB, type WB
> >
> > I don't understand the gap in the new table.
>
> Check the e820 table. Chances are the BIOS is reserving 2GB to
> map various devices (especially video cards) below the 4GB limit.
The table looks like this:
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
[ 0.000000] BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
[ 0.000000] BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 000000007ff80000 (usable)
[ 0.000000] BIOS-e820: 000000007ff80000 - 000000007ff8e000 (ACPI data)
[ 0.000000] BIOS-e820: 000000007ff8e000 - 000000007ffe0000 (ACPI NVS)
[ 0.000000] BIOS-e820: 000000007ffe0000 - 0000000080000000 (reserved)
[ 0.000000] BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
[ 0.000000] BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
[ 0.000000] BIOS-e820: 0000000100000000 - 0000000180000000 (usable)
So the BIOS has reserved the entire upper half. Can I do anything about that?
Simon
On Wednesday 12 Jun 2013 23:33:26 JA Magall?n wrote:
> On 06/12/2013 10:10 PM, Rik van Riel wrote:
> > On 06/12/2013 12:54 PM, Simon Brown wrote:
> >> Hello,
> >>
> >> For the sake of an old prototype peripheral I'm using a non PAE 32 bit
> >> x86 kernel and I'm having trouble accessing memory above 2 GB. The
> >> system has 4GB installed and all is well with a PAE kernel.
> >>
> >> I'm obviously expecting to lose some memory due to memory mapped devices
> >> but I wasn't expecting to lose 2GB. Instead I'm suspecting a BIOS bug.
> >> The system reports:
> >> free -m
> >>
> >> total used free shared buffers
> >> cached
> >>
> >> Mem: 2012 491 1521 0 40
> >> 277
> >>
> >> The mtrr table looked odd so I enabled sanitisation:
> >> [ 0.000000] original variable MTRRs
> >> [ 0.000000] reg 0, base: 2GB, range: 2GB, type UC
> >> [ 0.000000] reg 1, base: 0GB, range: 4GB, type WB
> >> [ 0.000000] reg 2, base: 4GB, range: 2GB, type WB
> >> [ 0.000000] total RAM covered: 4096M
> >> [ 0.000000] Found optimal setting for mtrr clean up
> >> [ 0.000000] gran_size: 64K chunk_size: 64K num_reg: 2
> >> lose cover RAM: 0G
> >> [ 0.000000] New variable MTRRs
> >> [ 0.000000] reg 0, base: 0GB, range: 2GB, type WB
> >> [ 0.000000] reg 1, base: 4GB, range: 2GB, type WB
> >>
> >> I don't understand the gap in the new table.
> >
> > Check the e820 table. Chances are the BIOS is reserving 2GB to
> > map various devices (especially video cards) below the 4GB limit.
>
> Acording to manual, that mobo has an option to "Memory remap feature"
> in BIOS that looks like that...
I don't understand that option in the BIOS. If I disable the option the e820
table is the same as before except missing the last line and the BIOS boot
screen only reports 2GB. The choice seems to be map it high or lose it.
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
[ 0.000000] BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
[ 0.000000] BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 000000007ff80000 (usable)
[ 0.000000] BIOS-e820: 000000007ff80000 - 000000007ff8e000 (ACPI data)
[ 0.000000] BIOS-e820: 000000007ff8e000 - 000000007ffe0000 (ACPI NVS)
[ 0.000000] BIOS-e820: 000000007ffe0000 - 0000000080000000 (reserved)
[ 0.000000] BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
[ 0.000000] BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
Apologies for the slow follow up,
Simon
On 06/13/2013 06:32 PM, Simon Brown wrote:
> On Wednesday 12 Jun 2013 16:10:03 Rik van Riel wrote:
>> On 06/12/2013 12:54 PM, Simon Brown wrote:
>>> Hello,
>>>
>>> For the sake of an old prototype peripheral I'm using a non PAE 32 bit
>>> x86 kernel and I'm having trouble accessing memory above 2 GB. The
>>> system has 4GB installed and all is well with a PAE kernel.
>>>
>>> I'm obviously expecting to lose some memory due to memory mapped devices
>>> but I wasn't expecting to lose 2GB. Instead I'm suspecting a BIOS bug.
>>> The system reports:
>>> free -m
>>>
>>> total used free shared buffers
>>> cached
>>>
>>> Mem: 2012 491 1521 0 40
>>> 277
>>>
>>> The mtrr table looked odd so I enabled sanitisation:
>>> [ 0.000000] original variable MTRRs
>>> [ 0.000000] reg 0, base: 2GB, range: 2GB, type UC
>>> [ 0.000000] reg 1, base: 0GB, range: 4GB, type WB
>>> [ 0.000000] reg 2, base: 4GB, range: 2GB, type WB
>>> [ 0.000000] total RAM covered: 4096M
>>> [ 0.000000] Found optimal setting for mtrr clean up
>>> [ 0.000000] gran_size: 64K chunk_size: 64K num_reg: 2
>>> lose cover RAM: 0G
>>> [ 0.000000] New variable MTRRs
>>> [ 0.000000] reg 0, base: 0GB, range: 2GB, type WB
>>> [ 0.000000] reg 1, base: 4GB, range: 2GB, type WB
>>>
>>> I don't understand the gap in the new table.
>>
>> Check the e820 table. Chances are the BIOS is reserving 2GB to
>> map various devices (especially video cards) below the 4GB limit.
>
> The table looks like this:
> [ 0.000000] BIOS-provided physical RAM map:
> [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
> [ 0.000000] BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
> [ 0.000000] BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
> [ 0.000000] BIOS-e820: 0000000000100000 - 000000007ff80000 (usable)
> [ 0.000000] BIOS-e820: 000000007ff80000 - 000000007ff8e000 (ACPI data)
> [ 0.000000] BIOS-e820: 000000007ff8e000 - 000000007ffe0000 (ACPI NVS)
> [ 0.000000] BIOS-e820: 000000007ffe0000 - 0000000080000000 (reserved)
> [ 0.000000] BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
> [ 0.000000] BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
> [ 0.000000] BIOS-e820: 0000000100000000 - 0000000180000000 (usable)
>
> So the BIOS has reserved the entire upper half. Can I do anything about that?
Besides use a 64 bit kernel? No.
--
All rights reversed
On Thu, Jun 13, 2013 at 5:14 PM, Rik van Riel <[email protected]> wrote:
> On 06/13/2013 06:32 PM, Simon Brown wrote:
>>
>> On Wednesday 12 Jun 2013 16:10:03 Rik van Riel wrote:
>>>
>>> On 06/12/2013 12:54 PM, Simon Brown wrote:
>>>>
>>>> Hello,
>>>>
>>>> For the sake of an old prototype peripheral I'm using a non PAE 32 bit
>>>> x86 kernel and I'm having trouble accessing memory above 2 GB. The
>>>> system has 4GB installed and all is well with a PAE kernel.
>>>>
>>>> I'm obviously expecting to lose some memory due to memory mapped devices
>>>> but I wasn't expecting to lose 2GB. Instead I'm suspecting a BIOS bug.
>>>> The system reports:
>>>> free -m
>>>>
>>>> total used free shared buffers
>>>> cached
>>>>
>>>> Mem: 2012 491 1521 0 40
>>>> 277
>>>>
>>>> The mtrr table looked odd so I enabled sanitisation:
>>>> [ 0.000000] original variable MTRRs
>>>> [ 0.000000] reg 0, base: 2GB, range: 2GB, type UC
>>>> [ 0.000000] reg 1, base: 0GB, range: 4GB, type WB
>>>> [ 0.000000] reg 2, base: 4GB, range: 2GB, type WB
>>>> [ 0.000000] total RAM covered: 4096M
>>>> [ 0.000000] Found optimal setting for mtrr clean up
>>>> [ 0.000000] gran_size: 64K chunk_size: 64K num_reg: 2
>>>> lose cover RAM: 0G
>>>> [ 0.000000] New variable MTRRs
>>>> [ 0.000000] reg 0, base: 0GB, range: 2GB, type WB
>>>> [ 0.000000] reg 1, base: 4GB, range: 2GB, type WB
>>>>
>>>> I don't understand the gap in the new table.
>>>
>>>
>>> Check the e820 table. Chances are the BIOS is reserving 2GB to
>>> map various devices (especially video cards) below the 4GB limit.
>>
>>
>> The table looks like this:
>> [ 0.000000] BIOS-provided physical RAM map:
>> [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
>> [ 0.000000] BIOS-e820: 000000000009fc00 - 00000000000a0000
>> (reserved)
>> [ 0.000000] BIOS-e820: 00000000000e4000 - 0000000000100000
>> (reserved)
>> [ 0.000000] BIOS-e820: 0000000000100000 - 000000007ff80000 (usable)
>> [ 0.000000] BIOS-e820: 000000007ff80000 - 000000007ff8e000 (ACPI
>> data)
>> [ 0.000000] BIOS-e820: 000000007ff8e000 - 000000007ffe0000 (ACPI
>> NVS)
>> [ 0.000000] BIOS-e820: 000000007ffe0000 - 0000000080000000
>> (reserved)
>> [ 0.000000] BIOS-e820: 00000000fee00000 - 00000000fee01000
>> (reserved)
>> [ 0.000000] BIOS-e820: 00000000fff00000 - 0000000100000000
>> (reserved)
>> [ 0.000000] BIOS-e820: 0000000100000000 - 0000000180000000 (usable)
>>
>> So the BIOS has reserved the entire upper half. Can I do anything about
>> that?
>
>
> Besides use a 64 bit kernel? No.
>
or use 64bit kernel + kvm , qemu will have own e820 map
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000dfffdfff] usable
so it looks like 3G ram under 4G.
but you need to make sure mb support vt-d/dmar, so you could use
pci_passthrough with your old prototype peripheral in your guest 32bit
kernel
without PAE.
Yinghai