2011-02-03 19:43:50

by Brian D. McGrew

[permalink] [raw]
Subject: Memset very slow

Hello All!

We're using Dell T5500 workstations and a kernel-2.6.18.194 in CentOS
5.5_x86-64. When we get these Dells, we have to flash the BIOS back to rev
A02 or else everything is so slow its unusable. Now, the "new" T5500's are
here and we don't have the option to flash the BIOS back to A02 since it's a
faster processor and memory and a new motherboard.

We do memset in lots of places in our code, and in a couple places, the
memset is as large as 3GB. Using the old machines with old BIOS, we were
fine but with a newer (current) BIOS on the new machines or the old
machines, memset is so slow it renders the system unusable. But also after
using memset and that taking too long, everything else is slow, especially
system() calls.

On a properly configured "old" machine with the "old" BIOS, I can memset
3GBx2 in about .45 seconds and system calls return in under .25 seconds.
However, sing the new BIOS on either machine, that same memset takes about
20 seconds and system calls 6 to 7 seconds.

I grabbed the bleeding edge 2.6.37 from Elrepo and everything works fine,
new or old boxes and/or BIOS.

So, my question is, can someone explain to me what is going on here? I am
not convinced this is a kernel problem just because a new kernel fixed it;
but, if there were changes related to this in the kernel, then what is the
oldest version I should be looking at to fix my issues?

Thanks!

-brian


2011-02-03 21:29:28

by Xose Vazquez Perez

[permalink] [raw]
Subject: Re: Memset very slow

Brian D. McGrew wrote:

> So, my question is, can someone explain to me what is going on here? I am
> not convinced this is a kernel problem just because a new kernel fixed it;
> but, if there were changes related to this in the kernel, then what is the
> oldest version I should be looking at to fix my issues?

changelog of BIOS:

--cut--
A07
1. Enhanced security device support

A06
1. Updated Intel Xeon?Processor 5600/3600 Series microcode to rev 10.
2. Updated Intel Xeon Processor 5500/5600 Platform Reference Code to revision P2.91.
3. Enhanced TPM remotely provision.
4. Enhance VT-d.

A05
1. Updated to the latest Intel (R) Xeon (R) Processor 5600 Series microcode

A04
1. Supported E5620, E5630, E5640, X5650, X5660, X5667, X5670, X5677, X5680, W3680 CPUs.
2. Updated Intel Xeon?500/5600 Platform Reference Code to revision P2.7
3. Removed S1 support.
4. Removed "Optional HDD fan" support.
5. Reported riser's DIMMs information(asset tag, serail number) in SMBIOS.
6. Used the same fan setting before and after S3.
7. Enhanced the compatibility with certain PCI-Express Gen1 cards.
8. Enhanced NUMA under RHEL5.3.
9. Removed HDD Acoustic support.

A03
1. Added error detection for bad monitors or cables when entering setup menu.
2. Enhanced PCIe bar allocation.
3. Enhanced algorithm for Graphics card with multiple OPROMs.
4. Updated BIOS fan descriptions.
5. Fixed BIOS cannot boot from SATA CD/DVD when setting USB controller to "No Boot" in BIOS setup menu.

A02
1. Implemented fix for intermittant boot issue with 6.4 GT/s CPUs.
2. Added feature to display a message if the DIMM configuration is not optimal.
3. Updated Intel(R) Memory Reference Code.
4. Added microcode update revision 11 for Nehalem D0-step.
5. Fixed boot issue with RAID and ATI FireMV 2450.
6. Added TCM support.
7. Added updated IO programming.
8. Fixed possible hang condition when VT-d is enabled.
9. Moved "Optional HDD Fan" to the "Post Behavior" section in Setup.
10. Updated SMBIOS tables.
11. Added support for Windows 7.
12. Updated fan settings.
13. Corrected memory channel information in Setup.
14. Improved the allocation of system resources.
--end--

What latest bios is sane ? Run the test again in the
OLD machine with A03, A04, and so on...

Also check numactl.

--
?All? muevan feroz guerra, ciegos reyes por un palmo m?s de tierra;
que yo aqu? tengo por m?o cuanto abarca el mar brav?o, a quien nadie
impuso leyes. Y no hay playa, sea cualquiera, ni bandera de esplendor,
que no sienta mi derecho y d? pecho a mi valor.?

2011-02-03 21:29:33

by Alan

[permalink] [raw]
Subject: Re: Memset very slow

> So, my question is, can someone explain to me what is going on here? I am
> not convinced this is a kernel problem just because a new kernel fixed it;
> but, if there were changes related to this in the kernel, then what is the
> oldest version I should be looking at to fix my issues?

The newer kernel does several things that may be relevant - the obvious
one is that it rewrites and corrects MTRR data if it doesn't like what
the BIOS handed out. That would fit with dramatic slowdowns in memory
performance being fixed.

2011-02-03 21:58:17

by Brian D. McGrew

[permalink] [raw]
Subject: Re: Memset very slow

On 2/3/11 1:31 PM, "Alan Cox" <[email protected]> wrote:

>> So, my question is, can someone explain to me what is going on here? I am
>> not convinced this is a kernel problem just because a new kernel fixed it;
>> but, if there were changes related to this in the kernel, then what is the
>> oldest version I should be looking at to fix my issues?
>
> The newer kernel does several things that may be relevant - the obvious
> one is that it rewrites and corrects MTRR data if it doesn't like what
> the BIOS handed out. That would fit with dramatic slowdowns in memory
> performance being fixed.

The Dell precsion-linux mailing list almost immediately pointed me here:
https://bugzilla.redhat.com/show_bug.cgi?id=581933

Which seems like it is my problem and the fix seems to work for me.

-brian