Hi,
In MSI-HOWTO, it's said:
"Using MSI enables the device functions to support two or more vectors, which can be configured to target different CPUs to increase scalability."
So how can I set up MSI-X vectors to target different CPUs? I want to allocate the same number of MSI-X vectors as CPUs, and equally distribute them to every CPU.
Is it automatically done by Linux when I call pci_enable_msix()? If yes, how? If not, what should I do? My guess is to set the affinity of the interrupts manually. Am I right?
Please CC'ed me ([email protected]) answers/comments in response to this posting.
Thanks,
Ying
[email protected] wrote:
> Hi,
>
> In MSI-HOWTO, it's said:
>
> "Using MSI enables the device functions to support two or more vectors, which can be configured to target different CPUs to increase scalability."
>
> So how can I set up MSI-X vectors to target different CPUs? I want to allocate the same number of MSI-X vectors as CPUs, and equally distribute them to every CPU.
>
> Is it automatically done by Linux when I call pci_enable_msix()? If yes, how? If not, what should I do? My guess is to set the affinity of the interrupts manually. Am I right?
>
> Please CC'ed me ([email protected]) answers/comments in response to this posting.
>
> Thanks,
> Ying
If the device actually supports multiple vectors (not all do), I think
they should show up as separate interrupts in /proc/interrupts and you
can either set the affinity manually, or maybe irqbalance is smart
enough for this.
Careful, though, as in some cases this may reduce performance due to
causing more cache line bouncing between CPUs.
Thanks, Robert. My device does support multiple vectors.
When looking into functions called by pci_enable_msix(), I found msi_compose_msg() in arch/i386/kernel/io_apic.c. It tries to get destination CPU (TARGET_CPUS) and set this information to msg->address_lo. My question is about TARGET_CPUS. Under the asm-i386/mach-default, it is the cpu_online_map. Under asm-i386/mach-bigsmp, it is the cpumask_of_cpu(cpu), where cpu is a single one. I would guess if a single CPU is set as destination, only that CPU will be interrupted. But what will happen when the cpu_online_map is set as destination? Any CPU can be interrupted then? Or depending on affinity of the corresponding irq?
Please
CC'ed
me
([email protected])
answers/comments
in
response
to
this
posting.
Thanks,
Ying
----- Original Message ----
From: Robert Hancock <[email protected]>
To: [email protected]
Cc: [email protected]
Sent: Thursday, February 21, 2008 7:59:14 PM
Subject: Re: Configure MSI-X vectors to target different CPUs
[email protected]
wrote:
>
Hi,
>
>
In
MSI-HOWTO,
it's
said:
>
>
"Using
MSI
enables
the
device
functions
to
support
two
or
more
vectors,
which
can
be
configured
to
target
different
CPUs
to
increase
scalability."
>
>
So
how
can
I
set
up
MSI-X
vectors
to
target
different
CPUs?
I
want
to
allocate
the
same
number
of
MSI-X
vectors
as
CPUs,
and
equally
distribute
them
to
every
CPU.
>
>
Is
it
automatically
done
by
Linux
when
I
call
pci_enable_msix()?
If
yes,
how?
If
not,
what
should
I
do?
My
guess
is
to
set
the
affinity
of
the
interrupts
manually.
Am
I
right?
>
>
Please
CC'ed
me
([email protected])
answers/comments
in
response
to
this
posting.
>
>
Thanks,
>
Ying
If
the
device
actually
supports
multiple
vectors
(not
all
do),
I
think
they
should
show
up
as
separate
interrupts
in
/proc/interrupts
and
you
can
either
set
the
affinity
manually,
or
maybe
irqbalance
is
smart
enough
for
this.
Careful,
though,
as
in
some
cases
this
may
reduce
performance
due
to
causing
more
cache
line
bouncing
between
CPUs.