Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of
RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By
default, my system boots up with these MTRR settings:
reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
The X server and various other programs try to add a mapping for my video
card's buffer, at 0xd0000000, size=256MB, type=write-combining, and this always
fails with a type mismatch error (old type is write-back). Apparently it's
conflicting with mapping register 0. I can't just disable the existing settings
and re-add them; the system hangs soon after disabling reg01.
I guess the kernel must be getting the initial setup from the BIOS. I've hacked
around this in mtrr/generic.c by explicitly changing the MTRR state in
get_mtrr_state to split the first mapping into two; one at base 0 size 2048M
and one at base 2048M size 1024M. So now I have this, which is pretty much what
I wanted:
reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg03: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg04: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
reg05: base=0xd0000000 (3328MB), size= 256MB: write-combining, count=1
So the question is - was there an easier/correct way to do this?
It might have been nice if the MTRR ioctls allowed the register number to be
specified on the Set commands, though I'm not sure that would have helped in
this case.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
On 9/14/07, Howard Chu <[email protected]> wrote:
> Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of
> RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By
> default, my system boots up with these MTRR settings:
>
> reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
> reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
BIOS should have another setup option about MTRR (continous?)
then you could get
> reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
> reg01: base=0x100000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
===> if you have rev E...instead of rev F.
YH
--- generic.c.O 2007-08-30 23:21:01.000000000 -0700
+++ generic.c 2007-09-16 09:09:34.000000000 -0700
@@ -78,6 +78,8 @@
base, base + step - 1, mtrr_attrib_to_str(*types));
}
+static int set_mtrr_var_ranges(unsigned int index, struct mtrr_var_range *vr);
+
/* Grab all of the MTRR state for this CPU into *state */
void get_mtrr_state(void)
{
@@ -105,6 +107,21 @@
mtrr_state.def_type = (lo & 0xff);
mtrr_state.enabled = (lo & 0xc00) >> 10;
+ /* Check for initial 4GB range, it should only be 3GB */
+ if (!mtrr_state.var_ranges[0].base_hi && !(mtrr_state.var_ranges[0].base_lo & 0xffffff00) &&
+ !(mtrr_state.var_ranges[0].mask_lo & 0x80000000)) {
+ /* split initial 4GB range into 2GB and 1GB */
+ for (i = num_var_ranges-1; i>0; i--)
+ mtrr_state.var_ranges[i] = mtrr_state.var_ranges[i-1];
+ mtrr_state.var_ranges[0].mask_lo |= 0x80000000;
+ mtrr_state.var_ranges[1].base_hi = 0;
+ mtrr_state.var_ranges[1].base_lo = 0x80000000 | MTRR_TYPE_WRBACK ;
+ mtrr_state.var_ranges[1].mask_hi = mtrr_state.var_ranges[2].mask_hi;
+ mtrr_state.var_ranges[1].mask_lo = mtrr_state.var_ranges[2].mask_lo;
+ for (i = 0; i < num_var_ranges; i++)
+ set_mtrr_var_ranges(i, &mtrr_state.var_ranges[i]);
+ }
+
if (mtrr_show) {
int high_width;
On 9/16/07, Howard Chu <[email protected]> wrote:
> Yinghai Lu wrote:
> > On 9/14/07, Howard Chu <[email protected]> wrote:
> >> Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of
> >> RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By
> >> default, my system boots up with these MTRR settings:
> >>
> >> reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
> >> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> >> reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
> >> reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
if rev F before cpu aka Rev E is installed, and 8G RAM installed, it will be
> >> reg00: base=0x00000000 ( 0MB), size=8192MB: write-back, count=1
> >> reg01: base=0x100000000 (8192MB), size=1024MB: write-back, count=1
> >> reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
> >> reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
and you will have problem...
So good way is get the ram top, and hole size...and reset the whole
set ot var mtrr.
you could refer the code in LinuxBIOS about setting that...
http://www.openbios.org/viewvc/trunk/LinuxBIOSv2/src/cpu/x86/mtrr/mtrr.c?revision=2616&view=markup
esp the part about CONFIG_VAR_MTRR_HOLE...
but Andi and Eric said resetting mtrr is not good... when someone from
intel try to trim the MTRR for intel CPU.
YH
"Yinghai Lu" <[email protected]> writes:
> On 9/16/07, Howard Chu <[email protected]> wrote:
>> Yinghai Lu wrote:
>> > On 9/14/07, Howard Chu <[email protected]> wrote:
>> >> Hi, was wondering if anyone else has been tripped up by this... I've got
> 4GB of
>> >> RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By
>> >> default, my system boots up with these MTRR settings:
>> >>
>> >> reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
>> >> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
>> >> reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
>> >> reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
>
> if rev F before cpu aka Rev E is installed, and 8G RAM installed, it will be
>> >> reg00: base=0x00000000 ( 0MB), size=8192MB: write-back, count=1
>> >> reg01: base=0x100000000 (8192MB), size=1024MB: write-back, count=1
>> >> reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
>> >> reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
> and you will have problem...
>
> So good way is get the ram top, and hole size...and reset the whole
> set ot var mtrr.
>
> you could refer the code in LinuxBIOS about setting that...
> http://www.openbios.org/viewvc/trunk/LinuxBIOSv2/src/cpu/x86/mtrr/mtrr.c?revision=2616&view=markup
> esp the part about CONFIG_VAR_MTRR_HOLE...
>
> but Andi and Eric said resetting mtrr is not good... when someone from
> intel try to trim the MTRR for intel CPU.
There are a couple issues with changing the MTRR configuration.
- You may not have perfect information on the cpu, the AMD revF is a good
example.
- Code in SMM mode may actually depend on the current mtrr configuration.
- The BIOS's need to fixed to setup MTRRs properly.
So the sanest approach appears to be.
- In linux only use ram that is mapped by a write-back mtrr.
This preserves performance and is always safe.
- If you need write-combining set it up in the page tables with PAT.
There is some difficulty there but software can always do those things
safely.
Eric
Eric W. Biederman wrote:
>> but Andi and Eric said resetting mtrr is not good... when someone from
>> intel try to trim the MTRR for intel CPU.
>
> There are a couple issues with changing the MTRR configuration.
> - You may not have perfect information on the cpu, the AMD revF is a good
> example.
> - Code in SMM mode may actually depend on the current mtrr configuration.
> - The BIOS's need to fixed to setup MTRRs properly.
Well the BIOS is definitely doing it wrong here. As I mentioned before, it was
setting up
0-0x100000000 WB
0xc0000000 - 0x100000000 UC
0xc0000000 - 0xd0000000 WC
But the Intel Architecture Software Developer's Manual states that whenever
any variable MTRR range overlaps with an UC MTRR range, the range remains UC.
(Section 9.12.2.3). So in fact what I needed to set was
0-2GB WB
2-3GB WB
3-3.25GB WC
and delete the 3-4GB UC range to get the behavior that the BIOS seems to have
been intending to set up. (Relying on the default of UC for the unspecified
ranges.)
> So the sanest approach appears to be.
> - In linux only use ram that is mapped by a write-back mtrr.
> This preserves performance and is always safe.
> - If you need write-combining set it up in the page tables with PAT.
>
> There is some difficulty there but software can always do those things
> safely.
Hm. Section 9.5.1 of the doc (table 9-5) says that anything marked UC is
always UC regardless of the bits in the page table. So with the MTRR setup
that the BIOS left me with, this is still a no-go. There's no way to get the
desired effect without completely reinitializing the MTRRs.
Of course, this isn't the only problem with these Asus BIOSs...
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
Howard Chu <[email protected]> writes:
> Eric W. Biederman wrote:
>>> but Andi and Eric said resetting mtrr is not good... when someone from
>>> intel try to trim the MTRR for intel CPU.
>>
>> There are a couple issues with changing the MTRR configuration.
>> - You may not have perfect information on the cpu, the AMD revF is a good
>> example.
>> - Code in SMM mode may actually depend on the current mtrr configuration.
>> - The BIOS's need to fixed to setup MTRRs properly.
>
> Well the BIOS is definitely doing it wrong here. As I mentioned before, it was
> setting up
> 0-0x100000000 WB
> 0xc0000000 - 0x100000000 UC
> 0xc0000000 - 0xd0000000 WC
Ok. I do agree that is wrong, and really weird to.
> But the Intel Architecture Software Developer's Manual states that whenever any
> variable MTRR range overlaps with an UC MTRR range, the range remains
> UC. (Section 9.12.2.3). So in fact what I needed to set was
> 0-2GB WB
> 2-3GB WB
> 3-3.25GB WC
> and delete the 3-4GB UC range to get the behavior that the BIOS seems to have
> been intending to set up. (Relying on the default of UC for the unspecified
> ranges.)
Which is certainly a good way to go if your hole sizes allow and you
have enough MTRRs.
>> So the sanest approach appears to be.
>> - In linux only use ram that is mapped by a write-back mtrr.
>> This preserves performance and is always safe.
>> - If you need write-combining set it up in the page tables with PAT.
>>
>> There is some difficulty there but software can always do those things
>> safely.
>
> Hm. Section 9.5.1 of the doc (table 9-5) says that anything marked UC is always
> UC regardless of the bits in the page table. So with the MTRR setup that the
> BIOS left me with, this is still a no-go. There's no way to get the desired
> effect without completely reinitializing the MTRRs.
Please look a little more closely. WC is a flavor of UC. There is a
specific exception that allows page tables to promote UC to WC.
I don't seem to have the same version of this document so I can't
refer you to specific sections. But I have looked it up several
times and every time I have looked I have found the exception that
allows PAT to promote UC to WC.
> Of course, this isn't the only problem with these Asus BIOSs...
Sure.
My point was that when we try to solve the general problem rather then
a specific case there are some very practical limits on what can be
done.
Now it should be mentioned that you can go in with /proc/mtrr and fix
things manually on a specific machine.
Eric
On Friday, September 14, 2007 9:33 am Howard Chu wrote:
> So the question is - was there an easier/correct way to do this?
>
> It might have been nice if the MTRR ioctls allowed the register
> number to be specified on the Set commands, though I'm not sure that
> would have helped in this case.
To do this in a nicer way (and be less vulnerable to similar BIOS
funkiness) the kernel really needs full PAT support. That should allow
WC over WB and WC over UC mappings to occur, at least if I'm
remembering the docs right...
Jesse
Jesse Barnes <[email protected]> writes:
>
> To do this in a nicer way (and be less vulnerable to similar BIOS
> funkiness) the kernel really needs full PAT support. That should allow
> WC over WB and WC over UC mappings to occur, at least if I'm
> remembering the docs right...
PAT only really helps for device driver performance optimizations.
But if the basic WB MTRRs are wrong PAT cannot really salvage it.
-Andi
Andi Kleen wrote:
> Jesse Barnes <[email protected]> writes:
>
>> To do this in a nicer way (and be less vulnerable to similar BIOS
>> funkiness) the kernel really needs full PAT support. That should allow
>> WC over WB and WC over UC mappings to occur, at least if I'm
>> remembering the docs right...
>>
>
> PAT only really helps for device driver performance optimizations.
> But if the basic WB MTRRs are wrong PAT cannot really salvage it.
>
Is there any reason not to set the MTRRs to define the entire memory as
write back, and use PAT exclusively for setting cacheability?
On my home machine for instance, the BIOS uses all 8 MTRRs leaving none
for X. I hacked it by merging a couple of MTRRs but this isn't a
generic solution.
--
error compiling committee.c: too many arguments to function
> Is there any reason not to set the MTRRs to define the entire memory as
> write back, and use PAT exclusively for setting cacheability?
That risks breaking SMM or BIOS code.
-Andi
Andi Kleen wrote:
>> Is there any reason not to set the MTRRs to define the entire memory as
>> write back, and use PAT exclusively for setting cacheability?
>>
>
> That risks breaking SMM or BIOS code.
>
>
Yes. Too bad.
--
error compiling committee.c: too many arguments to function
On Wednesday, September 19, 2007 11:50:23 pm Andi Kleen wrote:
> Jesse Barnes <[email protected]> writes:
> > To do this in a nicer way (and be less vulnerable to similar BIOS
> > funkiness) the kernel really needs full PAT support. That should allow
> > WC over WB and WC over UC mappings to occur, at least if I'm
> > remembering the docs right...
>
> PAT only really helps for device driver performance optimizations.
> But if the basic WB MTRRs are wrong PAT cannot really salvage it.
Look at the original bug report. There's a bug WB region so any subsequent WC
mappings via the MTRRs won't work, so you need PAT to do WC (unless you want
to rewrite all your MTRR settings as Howard has to do now).
Seems like more and more machines are being setup this way, and until we have
PAT users will have to jump through hoops to get good performance.
Jesse
On Thursday, September 20, 2007, Jesse Barnes wrote:
> On Wednesday, September 19, 2007 11:50:23 pm Andi Kleen wrote:
> > Jesse Barnes <[email protected]> writes:
> > > To do this in a nicer way (and be less vulnerable to similar BIOS
> > > funkiness) the kernel really needs full PAT support. That should
> > > allow WC over WB and WC over UC mappings to occur, at least if
> > > I'm remembering the docs right...
> >
> > PAT only really helps for device driver performance optimizations.
> > But if the basic WB MTRRs are wrong PAT cannot really salvage it.
>
> Look at the original bug report. There's a bug WB region so any
^^^
err big
On Fri, Sep 14, 2007 at 09:33:30AM -0700, Howard Chu wrote:
> Hi, was wondering if anyone else has been tripped up by this... I've got
> 4GB of
> RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By
> default, my system boots up with these MTRR settings:
>
> reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
> reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
>
> The X server and various other programs try to add a mapping for my video
> card's buffer, at 0xd0000000, size=256MB, type=write-combining, and this
> always
> fails with a type mismatch error (old type is write-back). Apparently it's
> conflicting with mapping register 0. I can't just disable the existing
> settings
> and re-add them; the system hangs soon after disabling reg01.
>
> I guess the kernel must be getting the initial setup from the BIOS. I've
> hacked
> around this in mtrr/generic.c by explicitly changing the MTRR state in
> get_mtrr_state to split the first mapping into two; one at base 0 size
> 2048M
> and one at base 2048M size 1024M. So now I have this, which is pretty much
> what
> I wanted:
>
> reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
> reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
> reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> reg03: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
> reg04: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
> reg05: base=0xd0000000 (3328MB), size= 256MB: write-combining, count=1
BTW, having overlapping WC, UC regions make the end result UC. So in this
case, you may not be getting the desired performance.
>
> So the question is - was there an easier/correct way to do this?
>
> It might have been nice if the MTRR ioctls allowed the register number to
> be
> specified on the Set commands, though I'm not sure that would have helped
> in
> this case.
Siddha, Suresh B wrote:
> On Fri, Sep 14, 2007 at 09:33:30AM -0700, Howard Chu wrote:
>> So now I have this, which is pretty much
>> what
>> I wanted:
>>
>> reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
>> reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
>> reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
>> reg03: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
>> reg04: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
>> reg05: base=0xd0000000 (3328MB), size= 256MB: write-combining, count=1
>
> BTW, having overlapping WC, UC regions make the end result UC. So in this
> case, you may not be getting the desired performance.
Thanks, I noticed that later. I simply deleted the UC mapping since it was no
longer needed.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/