* Michael S. Tsirkin <[email protected]> wrote:
> I'm looking at reducing the interrupt overhead for virtualized guests:
> some workloads spend a large part of their time processing interrupts.
> This patchset supplies infrastructure to reduce the IRQ ack overhead on
> x86: the idea is to add an eoi_write callback that we can then optimize
> without touching other apic functionality.
>
> The main user will be kvm: on kvm, an EOI write from the guest causes an
> expensive exit to host; we can avoid this using shared memory as the
> last patch in the series demonstrates.
>
> But I also wrote a micro-optimized version for the regular x2apic: this
> shaves off a branch and about 9 instructions from EOI when x2apic is
> used, and a comment in ack_APIC_irq implies that someone counted
> instructions there, at some point.
>
> Also included in the patchset are a couple of trivial macro fixes.
>
> The patches work fine on my boxes and I did look at the
> objdump output to verify that the generated code
> for the micro-optimization patch looks right
> and actually is shorter.
>
> Some benchmark results below (not sure what kind of
> testing is the most appropriate) show a tiny
> but measureable improvement. The tests were run on
> an AMD box with 24 cpus.
>
> - A clean kernel build after reboot shows
> a tiny but measureable improvement in system time
> which means lower CPU overhead (though not measureable
> in total time - that is dominated by user time and fluctuates
> too much):
>
> linux# reboot -f
> ...
> linux# make clean
> linux# time make -j 64 LOCALVERSION= 2>&1 > /dev/null
>
> Before:
>
> real 2m52.244s
> user 35m53.833s
> sys 6m7.194s
>
> After:
>
> real 2m52.827s
> user 35m48.916s
> sys 6m2.305s
>
> - perf micro-benchmarks seem to consistently show
> a tiny improvement in total time as well but it's below
> the confidence level of 3 std deviations:
>
> # ./tools/perf/perf stat --sync --repeat 100 --null perf bench sched messaging
> ...
> 0.414666797 seconds time elapsed ( +- 1.29% )
>
> Performance counter stats for 'perf bench sched messaging' (100 runs):
>
> 0.395370891 seconds time elapsed
> ( +- 1.04% )
>
>
> # ./tools/perf/perf stat --sync --repeat 100 --null perf bench sched pipe -l 10000
> 0.307019664 seconds time elapsed
> ( +- 0.10% )
>
> 0.304738024 seconds time elapsed
> ( +- 0.08% )
>
> The patches are against 3.4-rc3 - let me know if
> I need to rebase.
>
> I think patches 1-2 are definitely a good idea,
> and patches 3-4 might be a good idea.
> Please review, and consider patches 1-4 for linux 3.5.
>
> Thanks,
> MST
>
> Michael S. Tsirkin (5):
> apic: fix typo EIO_ACK -> EOI_ACK and document
> apic: use symbolic APIC_EOI_ACK
> x86: add apic->eoi_write callback
> x86: eoi micro-optimization
> kvm_para: guest side for eoi avoidance
>
> arch/x86/include/asm/apic.h | 22 ++++++++++++--
> arch/x86/include/asm/apicdef.h | 2 +-
> arch/x86/include/asm/bitops.h | 6 ++-
> arch/x86/include/asm/kvm_para.h | 2 +
> arch/x86/kernel/apic/apic_flat_64.c | 2 +
> arch/x86/kernel/apic/apic_noop.c | 1 +
> arch/x86/kernel/apic/apic_numachip.c | 1 +
> arch/x86/kernel/apic/bigsmp_32.c | 1 +
> arch/x86/kernel/apic/es7000_32.c | 2 +
> arch/x86/kernel/apic/numaq_32.c | 1 +
> arch/x86/kernel/apic/probe_32.c | 1 +
> arch/x86/kernel/apic/summit_32.c | 1 +
> arch/x86/kernel/apic/x2apic_cluster.c | 1 +
> arch/x86/kernel/apic/x2apic_phys.c | 1 +
> arch/x86/kernel/apic/x2apic_uv_x.c | 1 +
> arch/x86/kernel/kvm.c | 51 ++++++++++++++++++++++++++++++--
> arch/x86/platform/visws/visws_quirks.c | 2 +-
> 17 files changed, 88 insertions(+), 10 deletions(-)
No objections from the x86 side.
In terms of advantages, could you please create perf stat runs
that counts the number of MMIOs or so? That should show a pretty
obvious improvement - and that is enough as proof, no need to
try to reproduce the performance win in a noisy benchmark.
Thanks,
Ingo
On Mon, May 07, 2012 at 12:35:12PM +0200, Ingo Molnar wrote:
> > Michael S. Tsirkin (5):
> > apic: fix typo EIO_ACK -> EOI_ACK and document
> > apic: use symbolic APIC_EOI_ACK
> > x86: add apic->eoi_write callback
> > x86: eoi micro-optimization
> > kvm_para: guest side for eoi avoidance
> >
> > arch/x86/include/asm/apic.h | 22 ++++++++++++--
> > arch/x86/include/asm/apicdef.h | 2 +-
> > arch/x86/include/asm/bitops.h | 6 ++-
> > arch/x86/include/asm/kvm_para.h | 2 +
> > arch/x86/kernel/apic/apic_flat_64.c | 2 +
> > arch/x86/kernel/apic/apic_noop.c | 1 +
> > arch/x86/kernel/apic/apic_numachip.c | 1 +
> > arch/x86/kernel/apic/bigsmp_32.c | 1 +
> > arch/x86/kernel/apic/es7000_32.c | 2 +
> > arch/x86/kernel/apic/numaq_32.c | 1 +
> > arch/x86/kernel/apic/probe_32.c | 1 +
> > arch/x86/kernel/apic/summit_32.c | 1 +
> > arch/x86/kernel/apic/x2apic_cluster.c | 1 +
> > arch/x86/kernel/apic/x2apic_phys.c | 1 +
> > arch/x86/kernel/apic/x2apic_uv_x.c | 1 +
> > arch/x86/kernel/kvm.c | 51 ++++++++++++++++++++++++++++++--
> > arch/x86/platform/visws/visws_quirks.c | 2 +-
> > 17 files changed, 88 insertions(+), 10 deletions(-)
>
> No objections from the x86 side.
Is kvm.git a good tree to merge this through?
> In terms of advantages, could you please create perf stat runs
> that counts the number of MMIOs or so? That should show a pretty
> obvious improvement - and that is enough as proof, no need to
> try to reproduce the performance win in a noisy benchmark.
You mean with kvm PV, right? On real hardware the micro-optimization
removes branches and maybe cache-misses but I don't see why would it
reduce MMIOs.
--
MST
* Michael S. Tsirkin <[email protected]> wrote:
> On Mon, May 07, 2012 at 12:35:12PM +0200, Ingo Molnar wrote:
> > > Michael S. Tsirkin (5):
> > > apic: fix typo EIO_ACK -> EOI_ACK and document
> > > apic: use symbolic APIC_EOI_ACK
> > > x86: add apic->eoi_write callback
> > > x86: eoi micro-optimization
> > > kvm_para: guest side for eoi avoidance
> > >
> > > arch/x86/include/asm/apic.h | 22 ++++++++++++--
> > > arch/x86/include/asm/apicdef.h | 2 +-
> > > arch/x86/include/asm/bitops.h | 6 ++-
> > > arch/x86/include/asm/kvm_para.h | 2 +
> > > arch/x86/kernel/apic/apic_flat_64.c | 2 +
> > > arch/x86/kernel/apic/apic_noop.c | 1 +
> > > arch/x86/kernel/apic/apic_numachip.c | 1 +
> > > arch/x86/kernel/apic/bigsmp_32.c | 1 +
> > > arch/x86/kernel/apic/es7000_32.c | 2 +
> > > arch/x86/kernel/apic/numaq_32.c | 1 +
> > > arch/x86/kernel/apic/probe_32.c | 1 +
> > > arch/x86/kernel/apic/summit_32.c | 1 +
> > > arch/x86/kernel/apic/x2apic_cluster.c | 1 +
> > > arch/x86/kernel/apic/x2apic_phys.c | 1 +
> > > arch/x86/kernel/apic/x2apic_uv_x.c | 1 +
> > > arch/x86/kernel/kvm.c | 51 ++++++++++++++++++++++++++++++--
> > > arch/x86/platform/visws/visws_quirks.c | 2 +-
> > > 17 files changed, 88 insertions(+), 10 deletions(-)
> >
> > No objections from the x86 side.
>
> Is kvm.git a good tree to merge this through?
Fine to me, but I haven't checked how widely it conflicts with
existing bits: by the looks of it most of the linecount is on
the core x86 side, while the kvm change is well concentrated.
> > In terms of advantages, could you please create perf stat
> > runs that counts the number of MMIOs or so? That should show
> > a pretty obvious improvement - and that is enough as proof,
> > no need to try to reproduce the performance win in a noisy
> > benchmark.
>
> You mean with kvm PV, right? On real hardware the
> micro-optimization removes branches and maybe cache-misses but
> I don't see why would it reduce MMIOs.
Yeah, on KVM. On real hw I doubt it's measurable.
Thanks,
Ingo
On 05/07/2012 02:40 PM, Ingo Molnar wrote:
> > >
> > > No objections from the x86 side.
> >
> > Is kvm.git a good tree to merge this through?
>
> Fine to me, but I haven't checked how widely it conflicts with
> existing bits: by the looks of it most of the linecount is on
> the core x86 side, while the kvm change is well concentrated.
I don't see a problem with merging though tip.git - we're close to the
next merge window, and the guest side rarely causes conflicts. But
please don't apply the last patch yet, I want to review it more closely
(esp. with the host side).
--
error compiling committee.c: too many arguments to function
* Avi Kivity <[email protected]> wrote:
> On 05/07/2012 02:40 PM, Ingo Molnar wrote:
> > > >
> > > > No objections from the x86 side.
> > >
> > > Is kvm.git a good tree to merge this through?
> >
> > Fine to me, but I haven't checked how widely it conflicts with
> > existing bits: by the looks of it most of the linecount is on
> > the core x86 side, while the kvm change is well concentrated.
>
> I don't see a problem with merging though tip.git - we're close to the
> next merge window, and the guest side rarely causes conflicts. But
> please don't apply the last patch yet, I want to review it more closely
> (esp. with the host side).
That last patch was marked "don't apply yet", so I definitely
planned on another iteration that incorporates all the feedback
that has been given.
Thanks,
Ingo