IBM has a brand new xSeries PC server called the x360.
It has 1-4 Xeon MP's, DDR SDRAM, PCI-X backplane, IBM Summit chipset.
Full specs are here:
http://www.pc.ibm.com/us/eserver/xseries/x360.html
Has anyone tried running Linux on one of these ?
I have tried a few versions and found:
2.2.18 fails in boot
2.2.20 fails in boot
2.2.21rc2 fails in boot
2.4.9 works fine!
I know the hardware is in good shape because a 2.4.9 kernel works
fine on this machine. I have scoured the IBM site, linux-kernel,
and Google for clues, but to no avail.
The boot sequence failure under the 2.2.x versions that I tried is
always the same, it fails to recognize the IDE and SCSI devices. From
the messages, the system appears to be deaf to interrupts and so it
cannot recognize its devices. Notable messages from the boot sequence
that support this idea are:
hda: IRQ probe failed (0)
hda: lost interrupt
floppy0: no floppy controllers found
keyboard: Timeout - AT keyboard not present?
scsi : aborting command due to timeout : pid 0, scsi0, channel 0, id 0,
lun 0 Test Unit Ready 00 00 00 00 00
Below is the full boot sequence output as best I could reconstruct it.
It is a bit fuzzy because the x360 lacks a serial port (maybe it has
one accessible through the USB, but I don't have a cable). I pulled
this output together through their ASM remote console redirect feature,
which is less than convenient, leaving me with an error-probe cut and
paste exercise.
Unfortunately I have this machine for a limited eval and must return
it on Monday. I do hope to get it back ASAP to continue debugging,
but that is at IBM's discretion.
Any suggestions welcome, I will post more as I learn it.
-=-=-=-=-=-=-
LILO boot:
Loading linux-2220...............
Uncompressing Linux... Ok, booting the kernel.
per-CPU timeslice cutoff: 0.00 usecs.
CPU0: Intel Intel(R) Xeon(TM) CPU 1.60GHz stepping 01
calibrating APIC timer ...
..... CPU clock speed is 1600.2578 MHz.
..... system bus clock speed is 100.0160 MHz.
Booting processor 4 eip 2000
Calibrating delay loop... 3198.15 BogoMIPS
Intel machine check reporting enabled on CPU#4.
8K L1 data cache
1024K L3 cache
12K L1 instruction cache
8192K L1 instruction cache
CPU: L1 I Cache: 8204K L1 D Cache: 8K
CPU: L3 Cache: 1024K
CPU: Intel(R) Xeon(TM) CPU 1.60GHz
OK.
CPU4: Intel Intel(R) Xeon(TM) CPU 1.60GHz stepping 01
Total of 2 processors activated (6389.76 BogoMIPS).
enabling symmetric IO mode... ...done.
ENABLING IO-APIC IRQs
init IO_APIC IRQs
IO-APIC (apicid-pin) 14-0, 14-3, 14-5, 14-11, 14-13, 14-15,
14-18WARNING: ASSIGN_IRQ_VECTOR wrapped back to 52
Linux version 2.2.20 ([email protected]) (gcc version
egcs-2.91.66 1
9990314/Linux (egcs-1.1.2 release)) #1 SMP Fri Mar 15 14:12:38 EST 2002
BIOS-provided physical RAM map:
BIOS-e820: 0009d000 @ 00000000 (usable)
BIOS-e820: 7efea400 @ 00100000 (usable)
Warning only 960MB will be used.
Intel MultiProcessor Specification v1.4
Virtual Wire compatibility mode.
OEM ID: IBM ENSW Product ID: NF 6000R SMP APIC at: 0xFEE00000
Processor #0 Unknown CPU [15:1] APIC version 20
Processor #4 Unknown CPU [15:1] APIC version 20
I/O APIC #14 Version 17 at 0xFEC00000.
Processors: 2
WARNING: MP table in the EBDA can be UNSAFE, contact
[email protected] if you experience SMP problems!
mapped APIC to ffffe000 (fee00000)
mapped IOAPIC to ffffd000 (fec00000)
Detected 1600371 kHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop...
PCI: Probing PCI hardware
PCI: Discovered primary peer bus 01
PCI->APIC IRQ transform: (B0,I2,P0) -> 17
PCI->APIC IRQ transform: (B0,I3,P0) -> 42
PCI->APIC IRQ transform: (B0,I4,P0) -> 41
PCI->APIC IRQ transform: (B0,I15,P0) -> 16
PCI->APIC IRQ transform: (B1,I1,P0) -> 43
PCI->APIC IRQ transform: (B1,I2,P0) -> 24
PCI: BIOS reporting unknown device 0a:00
Linux NET4.0 for Linux 2.2
Based upon Swansea University Computer Society NET3.039
NET4: Unix domain sockets 1.0 for Linux NET4.0.
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
TCP: Hash tables configured (ehash 524288 bhash 65536)
Linux IP multicast router 0.06 plus PIM-SM
Initializing RT netlink socket
Starting kswapd v 1.5
Detected PS/2 Mouse Port.
Serial driver version 4.27 with no serial options enabled
ttyS00 at 0x03f8 (irq = 4) is a 16550A
pty: 256 Unix98 ptys configured
PCI_IDE: unknown IDE controller on PCI bus 00 device 79, VID=1166,
DID=0211
PCI_IDE: not 100% native mode: will probe irqs later
keyboard: Timeout - AT keyboard not present?
ide0: BM-DMA at 0x0700-0x0707, BIOS settings: hda:DMA, hdb:DMA
ide1: BM-DMA at 0x0708-0x070f, BIOS settings: hdc:pio, hdd:pio
hda: IRQ probe failed (0)
hda: IRQ probe failed (0)
hda: LG CD-ROM CRN-8245B, ATAPI CDROM drive
hda: IRQ probe failed (0)
hdb: IRQ probe failed (0)
hdb: IRQ probe failed (0)
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hda: lost interrupt
hda: lost interrupt
hda: ATAPI 24X CD-ROM drive, 128kB Cache
Uniform CD-ROM driver Revision: 3.11
Floppy drive(s): fd0 is 1.44M
floppy0: no floppy controllers found
sim710: No NCR53C710 adapter found.
(scsi0) <Adaptec AIC-7892 Ultra 160/m SCSI host adapter> found at PCI
0/4/0
(scsi0) Wide Channel, SCSI ID=7, 32/255 SCBs
(scsi0) Downloading sequencer code... 396 instructions downloaded
NCR53c406a: no available ports found
sym53c416.c: Version 1.0.0
DC390: 0 adapters found
scsi0 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.33/3.2.4
<Adaptec AIC-7892 Ultra 160/m SCSI host adapter>
scsi : 1 host.
scsi : aborting command due to timeout : pid 0, scsi0, channel 0, id 0,
lun 0 Test Unit Ready 00 00 00 00 00
scsi0 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.33/3.2.4
<Adaptec AIC-7892 Ultra 160/m SCSI host adapter>
scsi : 1 host.
scsi : aborting command due to timeout : pid 0, scsi0, channel 0, id 0,
lun 0 Test Unit Ready 00 00 00 00 00
On Fri, Mar 15, 2002 at 06:09:50PM -0500, Gordon J Lee wrote:
> IBM has a brand new xSeries PC server called the x360.
> It has 1-4 Xeon MP's, DDR SDRAM, PCI-X backplane, IBM Summit chipset.
> Full specs are here:
>
> http://www.pc.ibm.com/us/eserver/xseries/x360.html
Eeek, these machines are now in the wild? Didn't realize this :)
> Has anyone tried running Linux on one of these ?
Yes.
> I have tried a few versions and found:
>
> 2.2.18 fails in boot
> 2.2.20 fails in boot
> 2.2.21rc2 fails in boot
Ouch :(
> 2.4.9 works fine!
Glad to see this.
> I know the hardware is in good shape because a 2.4.9 kernel works
> fine on this machine. I have scoured the IBM site, linux-kernel,
> and Google for clues, but to no avail.
>
> The boot sequence failure under the 2.2.x versions that I tried is
> always the same, it fails to recognize the IDE and SCSI devices. From
> the messages, the system appears to be deaf to interrupts and so it
> cannot recognize its devices. Notable messages from the boot sequence
> that support this idea are:
I don't know if anyone ever tried a 2.2.x kernel on these boxes :)
Is there a reason you _really_ need a 2.2.x kernel for this machine?
You also might try a UP 2.2.x kernel on this box to see if the problem
is in the parsing of the APIC tables (as I think it is.)
thanks,
greg k-h
On Fri, Mar 15, 2002 at 06:09:50PM -0500, Gordon J Lee wrote:
> 2.4.9 works fine!
Forgot to mention, how many processors does this kernel show you having?
I think you need to run the latest 2.4.19-ac kernel to get the second
processors to show up properly.
thanks,
greg k-h
On Friday 15 March 2002 03:34 pm, Greg KH wrote:
> On Fri, Mar 15, 2002 at 06:09:50PM -0500, Gordon J Lee wrote:
> > IBM has a brand new xSeries PC server called the x360.
> > It has 1-4 Xeon MP's, DDR SDRAM, PCI-X backplane, IBM Summit chipset.
> > Full specs are here:
> >
> > http://www.pc.ibm.com/us/eserver/xseries/x360.html
>
> Eeek, these machines are now in the wild? Didn't realize this :)
They've officially been released for a while now. (December? January?
Something like that.)
> > Has anyone tried running Linux on one of these ?
>
> Yes.
>
> > I have tried a few versions and found:
> >
> > 2.2.18 fails in boot
> > 2.2.20 fails in boot
> > 2.2.21rc2 fails in boot
>
> Ouch :(
Not surprising. x360s have XAPICs in them and need reasonably up-to-date
APIC handling code. Try 2.4.14 or higher. That also includes Intel's
"hyperthreading" code, should you want to try that. (Well, minus my
forthcoming patch to that. It's in Alan's 2.4 tree.)
Recent 2.4.x also properly detects and uses APICs without going through a PIC
phase. This is important as many x360 IRQ sources do not run through a PIC,
and will fail any attempt to use PIC mode interrupts.
Also, I've got some patches that avoid a problem on SMP XAPIC boxes: all the
external interrupts hit on CPU 0.
I use a simple static round robin routine to bind the irqs roughly evenly
across all CPUs. Ingo has a different patch that rotates the irqs randomly
to idle CPUs on every received interrupt. This spreads the load much more
evenly, but has a bit more overhead and limits cache warmth for the handler
code/data. If you're going to be beating on the x360 much, you'll probably
want to install one patch or the other.
> > 2.4.9 works fine!
>
> Glad to see this.
>
> > I know the hardware is in good shape because a 2.4.9 kernel works
> > fine on this machine. I have scoured the IBM site, linux-kernel,
> > and Google for clues, but to no avail.
> >
> > The boot sequence failure under the 2.2.x versions that I tried is
> > always the same, it fails to recognize the IDE and SCSI devices. From
> > the messages, the system appears to be deaf to interrupts and so it
> > cannot recognize its devices. Notable messages from the boot sequence
> > that support this idea are:
See above; 2.2.x is not expected to work.
> I don't know if anyone ever tried a 2.2.x kernel on these boxes :)
> Is there a reason you _really_ need a 2.2.x kernel for this machine?
>
> You also might try a UP 2.2.x kernel on this box to see if the problem
> is in the parsing of the APIC tables (as I think it is.)
>
> thanks,
>
> greg k-h
> -
--
James Cleverdon, IBM xSeries Platform (NUMA), Beaverton
[email protected]
Thanks for the rapid replies,
> Eeek, these machines are now in the wild? Didn't realize this :)
Yes. They are still ramping up production, and evals are scarce.
I am pretty excited about it, because on paper, even without
the hyperthreading, they should run pretty fast for I/O intensive
workloads. My current eval project is to get some empirical
performance numbers on a particular application.
> I don't know if anyone ever tried a 2.2.x kernel on these boxes :)
I'm first! Lucky me! :-)
> Is there a reason you _really_ need a 2.2.x kernel for this machine?
Longterm no, shortterm yes,
We have some modifications to the 2.2.x kernel/drivers that would cost
us some time to migrate to 2.4.x. We expect to do this, but not within
the short eval period during which I have the box. My immediate goal
is to get it running enough to take performance measurements so we
can clearly quantify the cost/benefit of migrating to this box.
> You also might try a UP 2.2.x kernel on this box to see if the problem
> is in the parsing of the APIC tables (as I think it is.)
As a matter of fact, we did try a UP 2.2.x kernel, and it worked. But then
we only have one CPU, and where is the fun in that ? :-)
So I suppose this gives further support to the mishandled APIC table
theory.
I am interested and motivated to understand the details of APIC's further.
If I were to attempt to patch up a 2.2.x kernel to workaround this problem,
what documentation should I have on hand ? I have an Intel SMP 1.4
doc, although I haven't studied it in detail yet. Is this sufficient or
are
there other Must Have documents that I will need ?
- GL
> > 2.4.9 works fine!
>
> Forgot to mention, how many processors does this kernel show you having?
It has two physical packages, and shows two processors. See below.
> I think you need to run the latest 2.4.19-ac kernel to get the second
> processors to show up properly.
I suppose that you mean the second logical processors, right ?
It appears to show only one logical processor for each physical package.
Is there anything else you would like to see while I have an xterm open
on this ?
Straight from the horses mouth, an x360 running 2.4.9:
[root@x360-gw /root]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 1
model name : Intel(R) Xeon(TM) CPU 1.60GHz
stepping : 1
cpu MHz : 1600.337
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss tm
bogomips : 3191.60
processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 1
model name : Intel(R) Xeon(TM) CPU 1.60GHz
stepping : 1
cpu MHz : 1600.337
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss tm
bogomips : 3198.15
On Fri, Mar 15, 2002 at 09:46:02PM -0500, Gordon J Lee wrote:
> > > 2.4.9 works fine!
> >
> > Forgot to mention, how many processors does this kernel show you having?
>
> It has two physical packages, and shows two processors. See below.
Ah, can you try the latest 2.4.19-ac tree and make sure that the rest of
your processors (the "evil" twins) show up?
thanks,
greg k-h
On Fri, Mar 15, 2002 at 08:50:35PM -0500, Gordon J Lee wrote:
>
> > Eeek, these machines are now in the wild? Didn't realize this :)
>
> Yes. They are still ramping up production, and evals are scarce.
> I am pretty excited about it, because on paper, even without
> the hyperthreading, they should run pretty fast for I/O intensive
> workloads. My current eval project is to get some empirical
> performance numbers on a particular application.
Great, it will be nice to see some real world use of these machines to
see how well the HyperThreading works out.
> As a matter of fact, we did try a UP 2.2.x kernel, and it worked. But then
> we only have one CPU, and where is the fun in that ? :-)
> So I suppose this gives further support to the mishandled APIC table
> theory.
Yes it does, thanks for testing this.
> I am interested and motivated to understand the details of APIC's further.
> If I were to attempt to patch up a 2.2.x kernel to workaround this problem,
>
> what documentation should I have on hand ? I have an Intel SMP 1.4
> doc, although I haven't studied it in detail yet. Is this sufficient or
> are there other Must Have documents that I will need ?
James would be the best person for this, as he got this machine up and
working on Linux properly.
thanks,
greg k-h
> > > > 2.4.9 works fine!
> > >
> > > Forgot to mention, how many processors does this kernel show you having?
> >
> > It has two physical packages, and shows two processors. See below.
>
> Ah, can you try the latest 2.4.19-ac tree and make sure that the rest of
> your processors (the "evil" twins) show up?
Yes, they show up. I tried 'cat /proc/cpuinfo' on the following:
2.4.18 shows two processors
2.4.19-pre3 shows two processors
2.4.19-pre3-ac1 shows four processors
Here is the output from 2.4.19-pre3-ac1:
[root@x360 /root]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 1
model name : Intel(R) Xeon(TM) CPU 1.60GHz
stepping : 1
cpu MHz : 1600.400
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips : 3191.60
processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 1
model name : Intel(R) Xeon(TM) CPU 1.60GHz
stepping : 1
cpu MHz : 1600.400
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips : 3198.15
processor : 2
vendor_id : GenuineIntel
cpu family : 15
model : 1
model name : Intel(R) Xeon(TM) CPU 1.60GHz
stepping : 1
cpu MHz : 1600.400
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips : 3198.15
processor : 3
vendor_id : GenuineIntel
cpu family : 15
model : 1
model name : Intel(R) Xeon(TM) CPU 1.60GHz
stepping : 1
cpu MHz : 1600.400
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips : 3198.15
On Sat, Mar 16, 2002 at 12:27:48PM -0500, Gordon J Lee wrote:
> > > > > 2.4.9 works fine!
> > > >
> > > > Forgot to mention, how many processors does this kernel show you having?
> > >
> > > It has two physical packages, and shows two processors. See below.
> >
> > Ah, can you try the latest 2.4.19-ac tree and make sure that the rest of
> > your processors (the "evil" twins) show up?
>
> Yes, they show up. I tried 'cat /proc/cpuinfo' on the following:
>
> 2.4.18 shows two processors
> 2.4.19-pre3 shows two processors
> 2.4.19-pre3-ac1 shows four processors
Great, thanks for testing. I'd recommend using this hardware :)
greg k-h
> > 2.4.18 shows two processors
> > 2.4.19-pre3 shows two processors
> > 2.4.19-pre3-ac1 shows four processors
>
> Great, thanks for testing. I'd recommend using this hardware :)
>From your earlier post, I presume that the bug here was simply a presentation
layer bug in /proc/cpuinfo, and that kernel versions previous to 2.4.19-pre3-ac1
can actually use all of the logical processors. Is this correct ?
If so, at which 2.4.x kernel did support for hyperthreading show up?
- GL
On Sat, Mar 16, 2002 at 12:53:23PM -0500, Gordon J Lee wrote:
> > > 2.4.18 shows two processors
> > > 2.4.19-pre3 shows two processors
> > > 2.4.19-pre3-ac1 shows four processors
> >
> > Great, thanks for testing. I'd recommend using this hardware :)
Sorry, I meant to say, "I'd recommend using this kernel version".
> From your earlier post, I presume that the bug here was simply a presentation
> layer bug in /proc/cpuinfo, and that kernel versions previous to 2.4.19-pre3-ac1
> can actually use all of the logical processors. Is this correct ?
No, the other processors are not recognized by Linux at all. You need
that kernel version to properly use all of the logical processors.
> If so, at which 2.4.x kernel did support for hyperthreading show up?
In one of the 2.4.19-ac kernels from what I remember, sorry I don't know
the exact version.
Hope this helps,
greg k-h
On Sat, Mar 16, 2002 at 11:40:25AM -0800, Greg KH wrote:
> > If so, at which 2.4.x kernel did support for hyperthreading show up?
> In one of the 2.4.19-ac kernels from what I remember, sorry I don't know
> the exact version.
Interesting changelog entries..
2.4.14: hyperthreaded P4's
2.4.17:- Pentium IV Hyperthreading support (Alan Cox)
2.4.18ac:o Hyperthreading awareness for MTRR driver
Shame that .14 and .17 aren't more descriptive. I'm guessing
that they provided different bits. Not sure from memory what order
things happened though.
Maybe .14 was "Boot, and don't do anything silly" patches, whilst
.17 was the actual "take advantage of this feature" patch.
*shrug*
--
| Dave Jones. http://www.codemonkey.org.uk
| SuSE Labs
> >From your earlier post, I presume that the bug here was simply a presentation
> layer bug in /proc/cpuinfo, and that kernel versions previous to 2.4.19-pre3-ac1
> can actually use all of the logical processors. Is this correct ?
Not exactly no.
> If so, at which 2.4.x kernel did support for hyperthreading show up?
2.4.19pre, and you want -ac patches for autodetect right now - that should
all be in the main tree for 2.4.19 proper
On Friday 15 March 2002 09:58 pm, Greg KH wrote:
> On Fri, Mar 15, 2002 at 08:50:35PM -0500, Gordon J Lee wrote:
> > > Eeek, these machines are now in the wild? Didn't realize this :)
> >
> > Yes. They are still ramping up production, and evals are scarce.
> > I am pretty excited about it, because on paper, even without
> > the hyperthreading, they should run pretty fast for I/O intensive
> > workloads. My current eval project is to get some empirical
> > performance numbers on a particular application.
>
> Great, it will be nice to see some real world use of these machines to
> see how well the HyperThreading works out.
>
> > As a matter of fact, we did try a UP 2.2.x kernel, and it worked. But
> > then we only have one CPU, and where is the fun in that ? :-)
> > So I suppose this gives further support to the mishandled APIC table
> > theory.
>
> Yes it does, thanks for testing this.
>
> > I am interested and motivated to understand the details of APIC's
> > further. If I were to attempt to patch up a 2.2.x kernel to workaround
> > this problem,
> >
> > what documentation should I have on hand ? I have an Intel SMP 1.4
> > doc, although I haven't studied it in detail yet. Is this sufficient or
> > are there other Must Have documents that I will need ?
>
> James would be the best person for this, as he got this machine up and
> working on Linux properly.
The x360 doesnt require James' current patches (that arent in the mainstream
quite yet), but what it DOES require that you wont have in the 2.2 stream
is ACPI parsing.
The Linus kernel has support to boot fully HT on these boxes as of 2.4.17.
There are a number of changes related to the processor in it. In order to
detect HT you will need the acpitable code. In order to have the machine
function well beyond that you will want changes that Ingo put into the
scheduler :).
Personally I cant tell how much work you will have porting drivers to 2.4
but it might be easier than getting all the support you need back into
the 2.2 kernel, and even if you do backport it I dont know that you'd
get it in officially :)
Cheers,
--
Jack F. Vogel
IBM Linux Solutions
[email protected] (work)
[email protected] (home)