On Mon, 29 Sep 2008 18:54:12 -0700, Yinghai Lu <[email protected]> wrote:
> one have gran < 1M
>
> reg00: base=0xd8000000 (3456MB), size= 128MB: uncachable, count=1
> reg01: base=0xe0000000 (3584MB), size= 512MB: uncachable, count=1
> reg02: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
> reg03: base=0x100000000 (4096MB), size= 512MB: write-back, count=1
> reg04: base=0x120000000 (4608MB), size= 128MB: write-back, count=1
> reg05: base=0xd7f80000 (3455MB), size= 512KB: uncachable, count=1
>
> will get
>
> Found optimal setting for mtrr clean up
> gran_size: 512K chunk_size: 2M num_reg: 7 lose RAM: 0G
> range0: 0000000000000000 - 00000000d8000000
> Setting variable MTRR 0, base: 0GB, range: 2GB, type WB
> Setting variable MTRR 1, base: 2GB, range: 1GB, type WB
> Setting variable MTRR 2, base: 3GB, range: 256MB, type WB
> Setting variable MTRR 3, base: 3328MB, range: 128MB, type WB
> hole: 00000000d7f00000 - 00000000d7f80000
> Setting variable MTRR 4, base: 3455MB, range: 512KB, type UC
> rangeX: 0000000100000000 - 0000000128000000
> Setting variable MTRR 5, base: 4GB, range: 512MB, type WB
> Setting variable MTRR 6, base: 4608MB, range: 128MB, type WB
>
> so start from 64k instead of 1M
>
>
Also, on a 64 bit box with 4Gb, it gives this:
cicely:~# cat /proc/mtrr
reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0x140000000 (5120MB), size= 512MB: write-back, count=1
reg03: base=0x160000000 (5632MB), size= 256MB: write-back, count=1
reg04: base=0x80000000 (2048MB), size=2048MB: uncachable, count=1
Patch below cleans up formatting, with space for big bases and sizes (64 Gb).
diff -p -up arch/x86/kernel/cpu/mtrr/if.c.orig arch/x86/kernel/cpu/mtrr/if.c
--- arch/x86/kernel/cpu/mtrr/if.c.orig 2008-10-03 00:16:37.000000000 +0200
+++ arch/x86/kernel/cpu/mtrr/if.c 2008-10-03 00:22:34.000000000 +0200
@@ -16,7 +16,7 @@
static const char *const mtrr_strings[MTRR_NUM_TYPES] =
{
"uncachable", /* 0 */
- "write-combining", /* 1 */
+ "write-combine", /* 1 */
"?", /* 2 */
"?", /* 3 */
"write-through", /* 4 */
@@ -405,9 +405,9 @@ static int mtrr_seq_show(struct seq_file
}
/* RED-PEN: base can be > 32bit */
len += seq_printf(seq,
- "reg%02i: base=0x%05lx000 (%4luMB), size=%4lu%cB: %s, count=%d\n",
+ "reg%02i: base=0x%06lx000 (%5luMB), size=%5lu%cB, count=%d: %s\n",
i, base, base >> (20 - PAGE_SHIFT), size, factor,
- mtrr_attrib_to_str(type), mtrr_usage_table[i]);
+ mtrr_usage_table[i], mtrr_attrib_to_str(type));
}
}
return 0;
--
J.A. Magallon <jamagallon()ono!com> \ Software is like sex:
\ It's better when it's free
Mandriva Linux release 2009.0 (Cooker) for i586
Linux 2.6.25-jam18 (gcc 4.3.1 20080626 (GCC) #1 SMP
On Thu, Oct 2, 2008 at 3:32 PM, J.A. Magall?n <[email protected]> wrote:
> On Mon, 29 Sep 2008 18:54:12 -0700, Yinghai Lu <[email protected]> wrote:
>
> Also, on a 64 bit box with 4Gb, it gives this:
>
> cicely:~# cat /proc/mtrr
> reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> reg02: base=0x140000000 (5120MB), size= 512MB: write-back, count=1
> reg03: base=0x160000000 (5632MB), size= 256MB: write-back, count=1
> reg04: base=0x80000000 (2048MB), size=2048MB: uncachable, count=1
>
boundary handling may have problem...
should have
> reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> reg02: base=0x140000000 (5120MB), size= 512MB: write-back, count=1
> reg03: base=0x160000000 (5632MB), size= 256MB: write-back, count=1
can you post /proc/mtrr with disable_mtrr_cleanup?
YH
On Thu, Oct 2, 2008 at 3:39 PM, Yinghai Lu <[email protected]> wrote:
> On Thu, Oct 2, 2008 at 3:32 PM, J.A. Magall?n <[email protected]> wrote:
>> On Mon, 29 Sep 2008 18:54:12 -0700, Yinghai Lu <[email protected]> wrote:
>>
>> Also, on a 64 bit box with 4Gb, it gives this:
>>
>> cicely:~# cat /proc/mtrr
>> reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
>> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
>> reg02: base=0x140000000 (5120MB), size= 512MB: write-back, count=1
>> reg03: base=0x160000000 (5632MB), size= 256MB: write-back, count=1
>> reg04: base=0x80000000 (2048MB), size=2048MB: uncachable, count=1
>>
> boundary handling may have problem...
can you try
diff --git a/arch/x86/kernel/cpu/mtrr/main.c b/arch/x86/kernel/cpu/mtrr/main.c
index ef64128..70beb13 100644
--- a/arch/x86/kernel/cpu/mtrr/main.c
+++ b/arch/x86/kernel/cpu/mtrr/main.c
@@ -1044,7 +1044,7 @@ second_try:
hole_sizek = range0_sizek - state->range_sizek - second_sizek;
/* hole size should be less than half of range0 size */
- if (hole_sizek > (range0_sizek >> 1) &&
+ if (hole_sizek >= (range0_sizek >> 1) &&
range0_sizek >= chunk_sizek) {
range0_sizek -= chunk_sizek;
second_sizek = 0;
YH
On Thu, 2 Oct 2008 15:39:34 -0700, "Yinghai Lu" <[email protected]> wrote:
> On Thu, Oct 2, 2008 at 3:32 PM, J.A. Magallón <[email protected]> wrote:
> > On Mon, 29 Sep 2008 18:54:12 -0700, Yinghai Lu <[email protected]> wrote:
> >
> > Also, on a 64 bit box with 4Gb, it gives this:
> >
> > cicely:~# cat /proc/mtrr
> > reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
> > reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> > reg02: base=0x140000000 (5120MB), size= 512MB: write-back, count=1
> > reg03: base=0x160000000 (5632MB), size= 256MB: write-back, count=1
> > reg04: base=0x80000000 (2048MB), size=2048MB: uncachable, count=1
> >
> boundary handling may have problem...
>
> should have
> > reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
should not this ^^^^^ be 4096MB ??
> > reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> > reg02: base=0x140000000 (5120MB), size= 512MB: write-back, count=1
> > reg03: base=0x160000000 (5632MB), size= 256MB: write-back, count=1
>
> can you post /proc/mtrr with disable_mtrr_cleanup?
>
Oops, sorry, this is without cleanup. This is a distro kernel and is built
but not enable by deafult. As it is rc7, I will use 'enble_mtrr_cleanup' ;):
cicely:~# cat /proc/mtrr
reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0x140000000 (5120MB), size= 512MB: write-back, count=1
reg03: base=0x160000000 (5632MB), size= 256MB: write-back, count=1
I have lost 2Gb ?
cicely:~# free
total used free shared buffers cached
Mem: 3755568 182348 3573220 0 14024 72716
-/+ buffers/cache: 95608 3659960
I can't easily try your patch, this is a distro kernel.
I will get the src.rpm...
Ahhhhh....
This is a dual opteron board. dmidecode says:
Handle 0x0026, DMI type 16, 15 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: Single-bit ECC
Maximum Capacity: 8 GB
Error Information Handle: Not Provided
Number Of Devices: 8
So it maps one Opteron memory in first 4Gb and the other on the second 4Gb.
So I should have 2Gb@0 and 2Gb@4Gb.
What I don't know is why the bios eats up 256Mb.
--
J.A. Magallon <jamagallon()ono!com> \ Software is like sex:
\ It's better when it's free
Mandriva Linux release 2009.0 (Cooker) for i586
Linux 2.6.25-jam18 (gcc 4.3.1 20080626 (GCC) #1 SMP
On Thu, Oct 2, 2008 at 4:20 PM, J.A. Magall?n <[email protected]> wrote:
> On Thu, 2 Oct 2008 15:39:34 -0700, "Yinghai Lu" <[email protected]> wrote:
>
> Oops, sorry, this is without cleanup. This is a distro kernel and is built
> but not enable by deafult. As it is rc7, I will use 'enble_mtrr_cleanup' ;):
>
> cicely:~# cat /proc/mtrr
> reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> reg02: base=0x140000000 (5120MB), size= 512MB: write-back, count=1
> reg03: base=0x160000000 (5632MB), size= 256MB: write-back, count=1
>
> I have lost 2Gb ?
that is memhole
>
> cicely:~# free
> total used free shared buffers cached
> Mem: 3755568 182348 3573220 0 14024 72716
> -/+ buffers/cache: 95608 3659960
>
> I can't easily try your patch, this is a distro kernel.
> I will get the src.rpm...
just pull tip/master, and use your config from /boot/config....
>
> Ahhhhh....
>
> This is a dual opteron board. dmidecode says:
>
> Handle 0x0026, DMI type 16, 15 bytes
> Physical Memory Array
> Location: System Board Or Motherboard
> Use: System Memory
> Error Correction Type: Single-bit ECC
> Maximum Capacity: 8 GB
> Error Information Handle: Not Provided
> Number Of Devices: 8
>
> So it maps one Opteron memory in first 4Gb and the other on the second 4Gb.
> So I should have 2Gb@0 and 2Gb@4Gb.
> What I don't know is why the bios eats up 256Mb.
check if you enable memhole remapping in BIOS.
YH
* J.A. Magall?n <[email protected]> wrote:
> Also, on a 64 bit box with 4Gb, it gives this:
>
> cicely:~# cat /proc/mtrr
> reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> reg02: base=0x140000000 (5120MB), size= 512MB: write-back, count=1
> reg03: base=0x160000000 (5632MB), size= 256MB: write-back, count=1
> reg04: base=0x80000000 (2048MB), size=2048MB: uncachable, count=1
>
> Patch below cleans up formatting, with space for big bases and sizes (64 Gb).
applied to tip/x86/mtrr, thanks!
> +++ arch/x86/kernel/cpu/mtrr/if.c 2008-10-03 00:22:34.000000000 +0200
> @@ -16,7 +16,7 @@
> static const char *const mtrr_strings[MTRR_NUM_TYPES] =
> {
> "uncachable", /* 0 */
> - "write-combining", /* 1 */
> + "write-combine", /* 1 */
hm, maybe this bit could confuse versions of Xorg that modifies
/proc/mtrr - i.e. this could be part of the ABI towards user-space.
We'll see.
Ingo
* Yinghai Lu <[email protected]> wrote:
> > boundary handling may have problem...
>
> can you try
>
> diff --git a/arch/x86/kernel/cpu/mtrr/main.c b/arch/x86/kernel/cpu/mtrr/main.c
> index ef64128..70beb13 100644
> --- a/arch/x86/kernel/cpu/mtrr/main.c
> +++ b/arch/x86/kernel/cpu/mtrr/main.c
> @@ -1044,7 +1044,7 @@ second_try:
> hole_sizek = range0_sizek - state->range_sizek - second_sizek;
>
> /* hole size should be less than half of range0 size */
> - if (hole_sizek > (range0_sizek >> 1) &&
> + if (hole_sizek >= (range0_sizek >> 1) &&
> range0_sizek >= chunk_sizek) {
applied to tip/x86/mtrr, thanks Yinghai!
Ingo
Ingo Molnar wrote:
> * J.A. Magallón <[email protected]> wrote:
>
>> Also, on a 64 bit box with 4Gb, it gives this:
>>
>> cicely:~# cat /proc/mtrr
>> reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
>> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
>> reg02: base=0x140000000 (5120MB), size= 512MB: write-back, count=1
>> reg03: base=0x160000000 (5632MB), size= 256MB: write-back, count=1
>> reg04: base=0x80000000 (2048MB), size=2048MB: uncachable, count=1
>>
>> Patch below cleans up formatting, with space for big bases and sizes (64 Gb).
>
> applied to tip/x86/mtrr, thanks!
>
>> +++ arch/x86/kernel/cpu/mtrr/if.c 2008-10-03 00:22:34.000000000 +0200
>> @@ -16,7 +16,7 @@
>> static const char *const mtrr_strings[MTRR_NUM_TYPES] =
>> {
>> "uncachable", /* 0 */
>> - "write-combining", /* 1 */
>> + "write-combine", /* 1 */
>
> hm, maybe this bit could confuse versions of Xorg that modifies
> /proc/mtrr - i.e. this could be part of the ABI towards user-space.
> We'll see.
>
This *IS* part of the ABI toward userspace, although I think Xorg uses
the ioctl() interface.
-hpa
Ingo Molnar wrote:
>
>> +++ arch/x86/kernel/cpu/mtrr/if.c 2008-10-03 00:22:34.000000000 +0200
>> @@ -16,7 +16,7 @@
>> static const char *const mtrr_strings[MTRR_NUM_TYPES] =
>> {
>> "uncachable", /* 0 */
>> - "write-combining", /* 1 */
>> + "write-combine", /* 1 */
>
> hm, maybe this bit could confuse versions of Xorg that modifies
> /proc/mtrr - i.e. this could be part of the ABI towards user-space.
> We'll see.
>
By the way, this is just plain wrong; the cachability name is "Write
Combining (WC)" not "write-combine".
-hpa
On Fri, 3 Oct 2008 09:37:27 +0200, Ingo Molnar <[email protected]> wrote:
>
> * J.A. Magallón <[email protected]> wrote:
>
> > Also, on a 64 bit box with 4Gb, it gives this:
> >
> > cicely:~# cat /proc/mtrr
> > reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
> > reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> > reg02: base=0x140000000 (5120MB), size= 512MB: write-back, count=1
> > reg03: base=0x160000000 (5632MB), size= 256MB: write-back, count=1
> > reg04: base=0x80000000 (2048MB), size=2048MB: uncachable, count=1
> >
> > Patch below cleans up formatting, with space for big bases and sizes (64 Gb).
>
> applied to tip/x86/mtrr, thanks!
>
> > +++ arch/x86/kernel/cpu/mtrr/if.c 2008-10-03 00:22:34.000000000 +0200
> > @@ -16,7 +16,7 @@
> > static const char *const mtrr_strings[MTRR_NUM_TYPES] =
> > {
> > "uncachable", /* 0 */
> > - "write-combining", /* 1 */
> > + "write-combine", /* 1 */
>
> hm, maybe this bit could confuse versions of Xorg that modifies
> /proc/mtrr - i.e. this could be part of the ABI towards user-space.
> We'll see.
>
> Ingo
Oops, it's not only used for messages, also for writes in /proc !!!
Forget this part.
--
J.A. Magallon <jamagallon()ono!com> \ Software is like sex:
\ It's better when it's free
Mandriva Linux release 2009.0 (Cooker) for i586
Linux 2.6.25-jam18 (gcc 4.3.1 20080626 (GCC) #1 SMP
* H. Peter Anvin <[email protected]> wrote:
> Ingo Molnar wrote:
>> * J.A. Magall?n <[email protected]> wrote:
>>
>>> Also, on a 64 bit box with 4Gb, it gives this:
>>>
>>> cicely:~# cat /proc/mtrr
>>> reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
>>> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
>>> reg02: base=0x140000000 (5120MB), size= 512MB: write-back, count=1
>>> reg03: base=0x160000000 (5632MB), size= 256MB: write-back, count=1
>>> reg04: base=0x80000000 (2048MB), size=2048MB: uncachable, count=1
>>>
>>> Patch below cleans up formatting, with space for big bases and sizes (64 Gb).
>>
>> applied to tip/x86/mtrr, thanks!
>>
>>> +++ arch/x86/kernel/cpu/mtrr/if.c 2008-10-03 00:22:34.000000000 +0200
>>> @@ -16,7 +16,7 @@
>>> static const char *const mtrr_strings[MTRR_NUM_TYPES] =
>>> {
>>> "uncachable", /* 0 */
>>> - "write-combining", /* 1 */
>>> + "write-combine", /* 1 */
>>
>> hm, maybe this bit could confuse versions of Xorg that modifies
>> /proc/mtrr - i.e. this could be part of the ABI towards user-space.
>> We'll see.
>>
>
> This *IS* part of the ABI toward userspace, although I think Xorg uses
> the ioctl() interface.
yeah, but aspects of the ABI that applications do not rely on can
generally be changed. OTOH, i've undone this aspect of the patch -
/proc/mtrr is a legacy interface and changes to it are unnecessary. I
kept the boot printout cleanups.
Ingo
* H. Peter Anvin <[email protected]> wrote:
> Ingo Molnar wrote:
>>
>>> +++ arch/x86/kernel/cpu/mtrr/if.c 2008-10-03 00:22:34.000000000 +0200
>>> @@ -16,7 +16,7 @@
>>> static const char *const mtrr_strings[MTRR_NUM_TYPES] =
>>> {
>>> "uncachable", /* 0 */
>>> - "write-combining", /* 1 */
>>> + "write-combine", /* 1 */
>>
>> hm, maybe this bit could confuse versions of Xorg that modifies
>> /proc/mtrr - i.e. this could be part of the ABI towards user-space.
>> We'll see.
>>
>
> By the way, this is just plain wrong; the cachability name is "Write
> Combining (WC)" not "write-combine".
and "uncachable" should be "uncacheable" i guess.
anyway, agreed, lets not touch this.
Ingo
Ingo Molnar wrote:
>>>
>>>> +++ arch/x86/kernel/cpu/mtrr/if.c 2008-10-03 00:22:34.000000000 +0200
>>>> @@ -16,7 +16,7 @@
>>>> static const char *const mtrr_strings[MTRR_NUM_TYPES] =
>>>> {
>>>> "uncachable", /* 0 */
>>>> - "write-combining", /* 1 */
>>>> + "write-combine", /* 1 */
>>> hm, maybe this bit could confuse versions of Xorg that modifies
>>> /proc/mtrr - i.e. this could be part of the ABI towards user-space.
>>> We'll see.
>>>
>> This *IS* part of the ABI toward userspace, although I think Xorg uses
>> the ioctl() interface.
>
> yeah, but aspects of the ABI that applications do not rely on can
> generally be changed. OTOH, i've undone this aspect of the patch -
> /proc/mtrr is a legacy interface and changes to it are unnecessary. I
> kept the boot printout cleanups.
>
Great. It's worth noting that these strings aren't just used for
readout, but also for setting.
-hpa
| From: H. Peter Anvin <[email protected]>
| Ingo Molnar wrote:
| > * J.A. Magallón <[email protected]> wrote:
| > > +++ arch/x86/kernel/cpu/mtrr/if.c 2008-10-03 00:22:34.000000000 +0200
| > > @@ -16,7 +16,7 @@
| > > static const char *const mtrr_strings[MTRR_NUM_TYPES] =
| > > {
| > > "uncachable", /* 0 */
| > > - "write-combining", /* 1 */
| > > + "write-combine", /* 1 */
| >
| > hm, maybe this bit could confuse versions of Xorg that modifies /proc/mtrr -
| > i.e. this could be part of the ABI towards user-space. We'll see.
| >
|
| This *IS* part of the ABI toward userspace, although I think Xorg uses the
| ioctl() interface.
My mtrr-uncover program uses this interface.
One reason is that the ioctl interface is broken. See
http://lkml.org/lkml/2008/8/5/62
I too understand that Xorg drivers uses the ioctl interface. Since
they only use it on video device buffers, and those buffers always
seem to start and end below 4GiB, the ioctl bugs probably don't affect
Xorg.
The ioctl bugs do mean that Xorg drivers cannot even see the MTRR
nesting problems let alone address them.
The ib_path driver apparently uses MTRRs in a way analogous to Xorg
drivers. There is a script that is part of Fedora that uses the
/proc/mtrr interface to attempt to fix nested MTRRs:
http://cvs.fedora.redhat.com/viewvc/devel/rdma/rdma-fixup-mtrr.awk
There may well be other users of this API. Or is it an ABI. I'm not sure.
It would be useful to know what code exists that uses the
MTRR-changing interfaces. I only know about the ones I've mentioned
above. Does anyone know of others?