Forgot to attach the config, here it is now
tia,
marek
>>> Willy Tarreau <[email protected]> 01/04/05 5:05 PM >>>
>> Oh, while I'm at it, are you using hyperthreading, and if so, could
you
>> disable it ? I have seen many cases where it degrades performances
>> significantly (eg: highly loaded user space network applications).
>Willy
Indeed. AIX (sorry) 5.3 on POWER5 explicitly disables SMT (IBM
hyperthreading) if the load doesn't warrant it.
(Now how about that for Linux?) :)
Nik
On Tue, Jan 04, 2005 at 08:56:36PM +0100, Marek Habersack wrote:
> Hello,
>
> We have several machines with similar configurations
>
> 0000:00:00.0 Host bridge: Intel Corp. 82875P Memory Controller Hub
(rev 02)
> 0000:00:01.0 PCI bridge: Intel Corp. 82875P Processor to AGP
Controller (rev 02)
> 0000:00:1e.0 PCI bridge: Intel Corp. 82801 PCI Bridge (rev c2)
> 0000:00:1f.0 ISA bridge: Intel Corp. 82801EB/ER (ICH5/ICH5R) LPC
Bridge (rev 02)
> 0000:00:1f.2 IDE interface: Intel Corp. 82801EB (ICH5) Serial ATA 150
Storage Controller (rev 02)
> 0000:00:1f.3 SMBus: Intel Corp. 82801EB/ER (ICH5/ICH5R) SMBus
Controller (rev 02)
> 0000:02:09.0 VGA compatible controller: ATI Technologies Inc Rage XL
(rev 27)
> 0000:02:0a.0 Ethernet controller: Intel Corp. 82541EI Gigabit Ethernet
Controller (Copper)
> 0000:02:0b.0 Ethernet controller: Intel Corp. 82541EI Gigabit Ethernet
Controller (Copper)
>
> and
>
> 0000:00:00.0 Host bridge: Intel Corp. 82845G/GL[Brookdale-G]/GE/PE
DRAM Controller/Host-Hub Interface (rev 03)
> 0000:00:02.0 VGA compatible controller: Intel Corp.
82845G/GL[Brookdale-G]/GE Chipset Integrated Graphics Device (rev 03)
> 0000:00:1e.0 PCI bridge: Intel Corp. 82801 PCI Bridge (rev 82)
> 0000:00:1f.0 ISA bridge: Intel Corp. 82801DB/DBL (ICH4/ICH4-L) LPC
Bridge (rev 02)
> 0000:00:1f.1 IDE interface: Intel Corp. 82801DB/DBL (ICH4/ICH4-L)
UltraATA-100 IDE Controller (rev 02)
> 0000:00:1f.3 SMBus: Intel Corp. 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M)
SMBus Controller (rev 02)
> 0000:01:05.0 Ethernet controller: Intel Corp. 82540EM Gigabit Ethernet
Controller (rev 02)
> 0000:01:06.0 Ethernet controller: Intel Corp. 82540EM Gigabit Ethernet
Controller (rev 02)
>
> equipped with 2.6Ghz P4 CPUs, 1Gb of ram, 2-4gb of swap, the kernel
config
> is attached. The machines have normal load averages hovering not
higher than
> 7.0, depending on the time of the day etc. Two of the machines run
2.4.25,
> one 2.4.27 and they work fine. When booted with 2.4.28, though
(compiled
> with Debian's gcc 2.3.5, with p3 or p4 CPU selected in the config),
the load
> is climbing very fast and hovers around a value 3-4 times higher than
with
> the older kernels. Booted back in the old kernel, the load comes to
its
> usual level. The logs suggest nothing, no errors, nothing unusual is
> happening.
>
> Has anyone had similar problems with 2.4.28 in an environment
resembling the
> above? Could it be a problem with highmem i/o?
>
> tia,
>
> marek
**********************************************************
Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues.
On Tue, 4 Jan 2005, Nicholas Berry wrote:
> >>> Willy Tarreau <[email protected]> 01/04/05 5:05 PM >>>
> >> Oh, while I'm at it, are you using hyperthreading, and if so, could
> you
> >> disable it ? I have seen many cases where it degrades performances
> >> significantly (eg: highly loaded user space network applications).
>
> >Willy
>
> Indeed. AIX (sorry) 5.3 on POWER5 explicitly disables SMT (IBM
> hyperthreading) if the load doesn't warrant it.
>
Heh, yeah, that's pretty funky. I was initially pretty baffled first time
topas on a AIX 5.3 box showed me I had 6 CPU's but quitting it and
starting it 2sec later showed me 8 CPU's (box has 4 physical CPUs). That
send me hunting through docs. On the surface it seems like a nice feature,
but if it makes any real difference or not I've had difficulty in
determining.
--
Jesper Juhl
On Maw, 2005-01-04 at 22:41, Nicholas Berry wrote:
> Indeed. AIX (sorry) 5.3 on POWER5 explicitly disables SMT (IBM
> hyperthreading) if the load doesn't warrant it.
>
> (Now how about that for Linux?) :)
It would be very nice to do but AFAIK no current processor with
hypedthreading lets you do dynamic disabling. We do try and land tasks
on the real processors before other SMT threads and to leave the other
threads idle. I'm not sure we could do much more unless flipping the
cache control bits on packages when idle is a win (which I doubt)
Nicholas Berry wrote:
>>>>Willy Tarreau <[email protected]> 01/04/05 5:05 PM >>>
>>>
>>>Oh, while I'm at it, are you using hyperthreading, and if so, could
>
> you
>
>>>disable it ? I have seen many cases where it degrades performances
>>>significantly (eg: highly loaded user space network applications).
>
>
> Indeed. AIX (sorry) 5.3 on POWER5 explicitly disables SMT (IBM
> hyperthreading) if the load doesn't warrant it.
>
> (Now how about that for Linux?) :)
Didn't he say that it degrades performance under load? You're asking to
disable it under low load. The 2.6 scheduler is already supposed to try
and move tasks to full cores if they're empty instead of smt siblings.
Con
> Indeed. AIX (sorry) 5.3 on POWER5 explicitly disables SMT (IBM
> hyperthreading) if the load doesn't warrant it.
>
> (Now how about that for Linux?) :)
Its all there in ppc64 2.6 mainline:
for i in `seq 0 127`
do
echo XXX > /sys/devices/system/cpu/cpu$i/smt_snooze_delay
done
Will enable dynamic ST-SMT switching (XXX is the number of microseconds
we wait in the idle loop before sleeping the thread). Since the SMT
scheduler in 2.6 fills up primary threads first, we will keep SMT
disabled until we have enough work to do. This is probably what you are
referring to here.
To switch SMT off permanently on the fly:
for i in `seq 0 2 127`
do
echo 0 > /sys/devices_system/cpu/cpu$i/online
done
(ie hotplug cpu disable every second thread) and to switch SMT on again:
for i in `seq 0 2 127`
do
echo 1 > /sys/devices_system/cpu/cpu$i/online
done
Switching off SMT using either mode will give you a gain in performance
on some things, particularly single threaded stuff. The POWER5 chip
actually reconfigures itself on the fly and reallocates all the
resources to the one thread. eg on a single CPU, two thread box:
SMT enabled:
# ./lat_syscall -N 1
Simple syscall: 0.3360 microseconds
now disable SMT:
# echo 0 > /sys/devices/system/cpu/cpu1/online
# ./lat_syscall -N 1 null
Simple syscall: 0.2970 microseconds
Anton
> On Maw, 2005-01-04 at 22:41, Nicholas Berry wrote:
> >/ Indeed. AIX (sorry) 5.3 on POWER5 explicitly disables SMT (IBM/
>>/ hyperthreading) if the load doesn't warrant it./
>>/ /
> >/ (Now how about that for Linux?) :)/
> It would be very nice to do but AFAIK no current processor with
> hypedthreading lets you do dynamic disabling.
Maybe I am victim of marketing (or poor memory) but wasn't it so that
x86 instruction HLT was possible to use for single logical processor?
Indrek
> Maybe I am victim of marketing (or poor memory) but wasn't it so that
> x86 instruction HLT was possible to use for single logical processor?
You can easily park one of the threads ("rep nop" is used as a hint to
give the CPU back to the other thread so
1: hlt
rep nop
jr 1b
ought to do it, but in HT mode other stuff like the cache behaviour is
often changed