by Nakajima, Jun

[permalink] [raw]

Subject: RE: [PATCH][2.5][RFC] Using xAPIC apic address space on !Summit

Do you have performance data when you program TPR? Today, in 2.5, for
example, we are distributing interrupts among CPUs with Ingo's patch in
software. Doing that in software is fine because it's relatively simple, but
the chipset has more information on the on-going interrupt activities. So
using TPR might work better in general.

One issue with lowest priority delivery mode (as well as Ingo's) is that it
could have negative impacts on L2 caches (actually we saw such instances).
So we also have a patch to balance interrupts load (using statistic data)
without moving them around frequently (mostly static) and I think we can
combine that patch and lowest priority delivery mode.

Jun

> -----Original Message-----
> From: James Cleverdon [mailto:[email protected]]
> Sent: Thursday, December 12, 2002 7:32 PM
> To: Zwane Mwaikambo; Nakajima, Jun
> Cc: Martin Bligh; John Stultz; Linux Kernel
> Subject: Re: [PATCH][2.5][RFC] Using xAPIC apic address space on !Summit
>
> On Thursday 12 December 2002 07:26 pm, Zwane Mwaikambo wrote:
> > On Thu, 12 Dec 2002, Nakajima, Jun wrote:
> > > BTW, we are working on a xAPIC patch that supports more than 8 CPUs in
> a
> > > generic fashion (don't use hardcode OEM checking). We already tested
> it
> > > on two OEM systems with 16 CPUs.
> > > - It uses clustered mode. We don't want to use physical mode because
> it
> > > does not support lowest priority delivery mode.
> >
> > Wouldn't that only be for all including self? Or is the documentation
> > incorrect?
> >
> > Thanks,
> > Zwane
>
> I'm not sure I understand your question. Lowest Priority delivery mode
> only
> works with logical interrupts. (I've tried it with physical intrs. It
> fails
> miserably.) The "all including self" and "all excluding self" destination
> shorthands don't do lowest priority arbitration. They always deliver the
> interrupt to the CPUs mentioned in the shortand.
>
> Lowest priority delivery mode isn't _too_ useful in Linux yet. It would
> be
> nice to preferentially target idle CPUs with interrupts in real time.
> That
> means changing each CPU's Task Priority Register (TPR) to represent how
> busy
> it is. I've got some patches to do that, but haven't posted them as
> anything
> more than a RFC.
>
> --
> James Cleverdon
> IBM xSeries Linux Solutions
> {jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com

2002-12-17 02:22:39

by Nakajima, Jun

[permalink] [raw]

Subject: RE: [PATCH][2.5][RFC] Using xAPIC apic address space on !Summit

> I've never worked on a system that tries to use MSIs. Do they use lowest
> priority delivery?

They should provide lowest priority delivery as well as direct.

Jun

> -----Original Message-----
> From: James Cleverdon [mailto:[email protected]]
> Sent: Friday, December 13, 2002 11:40 AM
> To: Nakajima, Jun; Zwane Mwaikambo
> Cc: Martin Bligh; John Stultz; Linux Kernel
> Subject: Re: [PATCH][2.5][RFC] Using xAPIC apic address space on !Summit
>
> On Friday 13 December 2002 07:43 am, Nakajima, Jun wrote:
> > I/O APIC? I'm talking about xAPIC, i.e. local APIC which is part of the
> > chip. Also note that I/O subsystem can generate interrupts (e.g. MSI,
> > Message Signaled Interrupt), bypassing I/O APICs.
> >
> > Jun
>
> Part of the "x" in xAPIC is that it can communicate over the classic 2
> data /
> 1 clock serial bus or send interrupts over the system bus. If operating
> in
> serial mode, the old priority arbitration protocol works just fine. It's
> parallel mode where we have problems. All the I/O interrupts tend to hit
> CPU
> 0, thanks to the XTPR HW's tie-breaker logic.
>
> Intel has added a new register to I/O APICs that tells whether they're
> operating in serial or parallel mode. It may be useful to read that bit
> before activating some of the code in my patches.
>
> I've never worked on a system that tries to use MSIs. Do they use lowest
> priority delivery?
>
> (FYI, Summit boxes are hardwired into parallel mode. There's no provision
> for
> a serial APIC bus connecting the PCI expansion box to the CPUs.)
>
> > > -----Original Message-----
> > > From: James Cleverdon [mailto:[email protected]]
> > > Sent: Thursday, December 12, 2002 7:22 PM
> > > To: Nakajima, Jun; Zwane Mwaikambo
> > > Cc: Martin Bligh; John Stultz; Linux Kernel
> > > Subject: Re: [PATCH][2.5][RFC] Using xAPIC apic address space on
> !Summit
> > >
> > > On Thursday 12 December 2002 07:05 pm, Nakajima, Jun wrote:
> > > > BTW, we are working on a xAPIC patch that supports more than 8 CPUs
> in
> > > > a generic fashion (don't use hardcode OEM checking). We already
> tested
> > > > it
> > >
> > > on
> > >
> > > > two OEM systems with 16 CPUs.
> > > > - It uses clustered mode. We don't want to use physical mode because
> it
> > > > does not support lowest priority delivery mode.
> > > > - We also check the version of the local APIC if it's xAPIC or not.
> > > > It's possible that some other x86 architecture (other than P4P) uses
> > > > xAPIC.
> > > >
> > > > Stay tuned.
> > > >
> > > > Jun
> > >
> > > See my 2.5 summit patch for one that uses logical mode and lowest
> > > priority delivery. I haven't submitted that for 2.4 because the
> physical
> > > patch has the most run time on it, both in Alan's tree and SuSE 8.0+.
> > >
> > > The hardcoded OEM checking was necessary because the Summit I/O APICs
> > > were still using the older version number. (The HW folks claim that
> > > Intel didn't
> > > specify the new number soon enough for them.)
> > >
> > > I'm hesitant to key xAPIC vs. flat off the local APIC version number
> > > because
> > > it's possible to build a flat system out of P4 CPUs. I/O APIC version
> > > numbers combined with the parallel vs. serial bit would be safer
> (except
> > > for
> > > the Summit problem above). I've also tried checking all the CPU's
> > > physical
> > > APIC IDs to see if they use multiple APIC clusters.
> > >
> > > > > -----Original Message-----
> > > > > From: James Cleverdon [mailto:[email protected]]
> > > > > Sent: Thursday, December 12, 2002 6:42 PM
> > > > > To: Zwane Mwaikambo
> > > > > Cc: Martin Bligh; John Stultz; Linux Kernel
> > > > > Subject: Re: [PATCH][2.5][RFC] Using xAPIC apic address space on
> > >
> > > !Summit
> > >
> > > > > On Thursday 12 December 2002 06:21 pm, Zwane Mwaikambo wrote:
> > > > > > On Thu, 12 Dec 2002, James Cleverdon wrote:
> > > > > > > On Thursday 12 December 2002 05:44 pm, Zwane Mwaikambo wrote:
> > > > > > > > Hi,
> > > > > > > > I've got an 32x SMP system which has an xAPIC but
> utilises
> > > > >
> > > > > flat
> > > > >
> > > > > > > > addressing. This patch is to rename what was formerly
> > > > > > > > x86_summit
> > >
> > > to
> > >
> > > > > > > > x86_xapic (just to avoid confusion) and then select mask
> > >
> > > depending
> > >
> > > > > on
> > > > >
> > > > > > > > that.
> > > > > > > >
> > > > > > > > Untested/uncompiled patch
> > > > > > >
> > > > > > > Hi Zwane,
> > > > > > >
> > > > > > > How can you have a 32-way SMP system with flat addressing?
> There
> > >
> > > are
> > >
> > > > > > > only 8 bits in the destination address field. Even if you
> work
> > > > > > > around that by assigning a set of CPUs to each dest addr bit,
> > >
> > > there
> > >
> > > > > > > can only
> > > > >
> > > > > be
> > > > >
> > > > > > > 15 physical APIC IDs in flat mode. To get to 32 you must
> switch
> > >
> > > into
> > >
> > > > > > > clustered mode.
> > > > > > >
> > > > > > > Please tell me more. I'm intrigued how this can be done.
> > > > > >
> > > > > > Hi James,
> > > > > > with the xAPIC we can use the 8bit address space everywhere
> >
> > in
> >
> > > > > > physical destination mode. For example the ICR now has an 8bit
> > > > > > space for destination.
> > > > > >
> > > > > > "Specifies the target processor or processors. This field is
> only
> > >
> > > used
> > >
> > > > > > when the destination shorthand field is set to 00B. If the
> > >
> > > destination
> > >
> > > > > > mode is set to physical, then bits 56 through 59 contain the
> APIC
> > > > > > ID
> > >
> > > of
> > >
> > > > > > the target processor for Pentium and P6 family processors and
> bits
> > >
> > > 56
> > >
> > > > > > through 63 contain the APIC ID of the target processor the for
> > >
> > > Pentium
> > >
> > > > > > 4 and Intel Xeon processors. If the destination mode is set to
> > >
> > > logical,
> > >
> > > > > the
> > > > >
> > > > > > interpretation of the 8-bit destination field depends on the
> > >
> > > settings
> > >
> > > > > > of the DFR and LDR registers of the local APICs in all the
> > >
> > > processors
> > >
> > > > > > in
> > > > >
> > > > > the
> > > > >
> > > > > > system (see Section 8.6.2., Determining IPI Destination )."
> > > > > > - System Developer's Manual vol3 p291
> > > > > >
> > > > > > Regards,
> > > > > > Zwane
> > > > >
> > > > > Sure you can physically address them, if you assign IDs using
> Intel's
> > > > > official
> > > > > xAPIC numbering scheme (which must be clustered for more than 7
> > > > > CPUs). But,
> > > > > you still don't have enough destination address bits to go around.
> > > > > In flat
> > > > > mode, the kernel assumes you have one bit per CPU and phys IDs
> will
> > > > > be
> > >
> > > <
> > >
> > > > > 0xF.
> > > > >
> > > > > Bill tells me that you may be doing this for an emulator. Why not
> > > > > emulate clusered APIC mode, like the real hardware uses?
> > > > >
> > > > > I know the name x86_summit doesn't really fit. The summit patch
> > >
> > > should
> > >
> > > > > work
> > > > > for any xAPIC box that uses the system bus for interrupt delivery
> and
> > >
> > > has
> > >
> > > > > multiple APIC clusters. Is that what you're working towards?
> > > > >
> > > > > --
> > > > > James Cleverdon
> > > > > IBM xSeries Linux Solutions
> > > > > {jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
> > >
> > > --
> > > James Cleverdon
> > > IBM xSeries Linux Solutions
> > > {jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
>
> --
> James Cleverdon
> IBM xSeries Linux Solutions
> {jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com