LinuxLists.cc - >10% performance degradation since 2.6.18

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

Matthew Wilcox <[email protected]> writes:
>
> ======oprofile CPU_CLK_UNHALTED for top 30 functions
> Cycles% 2.6.18-92.el5-op Cycles% 2.6.30
> 70.1409 <database> 67.0207 <database>
> 1.3556 mpt_interrupt 1.7029 mpt_interrupt

It's strange that mpt_interrupt is that more costly in 2.6.30
than in 2.6.18. I diffed 2.6.30's drivers/message/fusion/mptbase.c
to a rhel 5.3s and they seem to be about the same.

So why does it cost 0.5% more in 2.6.30?

[adding MPT maintainers]

> 1.1622 __blockdev_direct_IO 1.1443 kmem_cache_alloc

It would be interesting to find out why kmem_cache_alloc
is that more expensive. Either it is called more or the allocator
is slower. Any chance of a callgraph profile run so we
can see the callers?

(you would need to rebuild the kernel with frame pointers)

An interesting would be to simply count them on both kernels, e.g. with a
simple systemtap script (see script at end of mail)

> 0.8246 kmem_cache_free 0.8801 kmem_cache_free
> 0.7108 schedule 0.7774 __blockdev_direct_IO
> 0.6733 scsi_request_fn 0.7031 scsi_request_fn
> 0.6114 kmem_cache_alloc 0.5317 __schedule
> 0.4207 follow_hugetlb_page 0.3922 task_rq_lock
> 0.4062 list_del 0.3629 sd_prep_fn
> 0.3400 __switch_to 0.3504 list_del

A callgraph for the list_del/list_dels would be interesting.
Presumably that's cache misses, maybe we can do something
here with some strategic prefetches or more batching.

> 0.3339 generic_make_request 0.3382 __sigsetjmp

__sigsetjmp is in glibc, isn't it?

> 0.3204 memmove 0.3270 __switch_to
> 0.3088 __sigsetjmp 0.3257 generic_make_request
> 0.2848 get_request 0.3116 kfree
> 0.2804 lock_timer_base 0.2895 memmove

memmove? Is that glibc or the kernel? (it would be good if that
was marked in the listing)

Back when I wrote the x86-64 string functions I didn't consider memmove
too performance critical. So the x86-64 kernel memmove is actually
not very optimized, especially the backwards case is very dumb
and copies byte by byte.

If it's that there might be some low hanging fruit here,
on the other hand it's not a lot of CPU time. The systemtap
script at the end measures this also.

A callgraph for it would be useful if it's kernel.

> 0.2572 __end_that_request_first 0.2402 aio_complete
> 0.2567 fget_light 0.2382 mptscsih_qcmd
> 0.2531 submit_page_section 0.2342 fget
> 0.2428 mempool_alloc 0.2277 gup_huge_pmd
> 0.2428 __generic_file_aio_read 0.2264 submit_page_section
> 0.2368 touch_atime 0.2204 touch_atime

Did you not run with noatime? If it runs with noatime and touch_atime
still takes that many cycles something must be wrong.

Ah it might be the lock in mnt_want_write(). How about the
appended dumb (untested) patch to first check for all the noatime
cases before taking the lock? Even without noatime, but standard
relatime that will hopefully eliminate the locking cost
in most cases.

> 0.2270 __aio_get_req 0.2165 __list_add
> 0.2223 mptscsih_qcmd 0.2063 scsi_dispatch_cmd
> 0.2198 init_request_from_bio 0.2040 lock_timer_base
> 0.2191 fget 0.2036 irq_entries_start
> 0.2141 device_not_available 0.2036 plist_del

That plist_del is new. calltrace?

> 0.2125 try_to_wake_up 0.2004 elv_queue_empty
> 0.2065 mptscsih_io_done 0.2004 get_user_pages_fast
> 0.2059 math_state_restore 0.1997 copy_user_generic_string
> 0.2035 __errno_location 0.1925 kref_get
> 0.2022 find_vma 0.1905 scsi_finish_command
> 0.1967 _setjmp 0.1888 aio_rw_vect_retry
> 0.1966 kref_get 0.1882 __errno_location

That's not kernel, but it's weird that __errno_location is that
costly. AFAIK it should be a simple thread local data lookup

> 0.1952 memset 0.1849 scsi_device_unbusy
> 0.1924 __list_add 0.1846 pick_next_highest_task_rt
> 0.1917 copy_user_generic 0.1826 memset_c
> 0.1907 acpi_os_read_port 0.1819 _setjmp
> 0.1842 elv_queue_empty 0.1816 ipc_lock

So it looks like ipc_lock needs so more scalability work.

> 0.1809 scsi_dispatch_cmd 0.1809 mod_timer
> 0.1808 sd_init_command 0.1800 noop_queue_empty
> 0.1789 swiotlb_unmap_sg 0.1796 scsi_softirq_done
> 0.1766 rw_verify_area 0.1757 scsi_run_queue
> -----------------------------------

---

Systemtap script to measure some interesting metrics. Run
with stap script.stp and then ctrl-c to get data out.

---

global kmem_cache_alloc
global kmem_cache_free

probe kernel.function("kmem_cache_alloc") {
kmem_cache_alloc++

}

probe kernel.function("kmem_cache_free") {
kmem_cache_free++
}

global memmove_total, memmove_backwards

probe kernel.function("memmove") {
memmove_total++
if (ulong_arg(1) < ulong_arg(2)) {
memmove_backwards++
}
}

probe end {
printf("%u kmem_cache_allocs\n", kmem_cache_alloc)
printf("%u kmem_cache_frees\n", kmem_cache_free)
printf("%u memmoves, %u backwards\n", memmove_total, memmove_backwards);
}

---

Only do mnt_want_write after all other options to avoid atime update are exhausted

mnt_want_write takes a lock and can be fairly expensive. So do it as late
as possible.

Signed-off-by: Andi Kleen <[email protected]>

---
fs/inode.c | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)

Index: linux-2.6.30-ak/fs/inode.c
===================================================================
--- linux-2.6.30-ak.orig/fs/inode.c
+++ linux-2.6.30-ak/fs/inode.c
@@ -1361,31 +1361,31 @@ void touch_atime(struct vfsmount *mnt, s
struct inode *inode = dentry->d_inode;
struct timespec now;

- if (mnt_want_write(mnt))
- return;
if (inode->i_flags & S_NOATIME)
- goto out;
+ return;
if (IS_NOATIME(inode))
- goto out;
+ return;
if ((inode->i_sb->s_flags & MS_NODIRATIME) && S_ISDIR(inode->i_mode))
- goto out;
+ return;

if (mnt->mnt_flags & MNT_NOATIME)
- goto out;
+ return;
if ((mnt->mnt_flags & MNT_NODIRATIME) && S_ISDIR(inode->i_mode))
- goto out;
+ return;

now = current_fs_time(inode->i_sb);

if (!relatime_need_update(mnt, inode, now))
- goto out;
+ return;

if (timespec_equal(&inode->i_atime, &now))
- goto out;
+ return;
+
+ if (mnt_want_write(mnt))
+ return;

inode->i_atime = now;
mark_inode_dirty_sync(inode);
-out:
mnt_drop_write(mnt);
}
EXPORT_SYMBOL(touch_atime);

--
[email protected] -- Speaking for myself only.

2009-07-03 18:54:19

by Jens Axboe

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Fri, Jul 03 2009, Andi Kleen wrote:
>
> Matthew Wilcox <[email protected]> writes:
> >
> > ======oprofile CPU_CLK_UNHALTED for top 30 functions
> > Cycles% 2.6.18-92.el5-op Cycles% 2.6.30
> > 70.1409 <database> 67.0207 <database>
> > 1.3556 mpt_interrupt 1.7029 mpt_interrupt
>
> It's strange that mpt_interrupt is that more costly in 2.6.30
> than in 2.6.18. I diffed 2.6.30's drivers/message/fusion/mptbase.c
> to a rhel 5.3s and they seem to be about the same.
>
> So why does it cost 0.5% more in 2.6.30?
>
> [adding MPT maintainers]

Look at the irqs/sec rate, it's higher by about the same percentage. So
it's likely not a more costly irq handler, it's likely just called that
much more. It could be IO pattern, causing more commands to be issued
(which leads to more interrupts, etc).

> > 1.1622 __blockdev_direct_IO 1.1443 kmem_cache_alloc
>
> It would be interesting to find out why kmem_cache_alloc
> is that more expensive. Either it is called more or the allocator
> is slower. Any chance of a callgraph profile run so we
> can see the callers?

Could be more IO as well, that hits the allocate often.

I agree with some callgraph data, that would at least eliminate the
guessing here. And some detailed IO statistics, amount of data
transferred as well as iostat info to see if the pattern is
significantly worse.

--
Jens Axboe

2009-07-03 19:13:34

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Fri, Jul 03, 2009 at 08:54:14PM +0200, Jens Axboe wrote:
> On Fri, Jul 03 2009, Andi Kleen wrote:
> >
> > Matthew Wilcox <[email protected]> writes:
> > >
> > > ======oprofile CPU_CLK_UNHALTED for top 30 functions
> > > Cycles% 2.6.18-92.el5-op Cycles% 2.6.30
> > > 70.1409 <database> 67.0207 <database>
> > > 1.3556 mpt_interrupt 1.7029 mpt_interrupt
> >
> > It's strange that mpt_interrupt is that more costly in 2.6.30
> > than in 2.6.18. I diffed 2.6.30's drivers/message/fusion/mptbase.c
> > to a rhel 5.3s and they seem to be about the same.
> >
> > So why does it cost 0.5% more in 2.6.30?
> >
> > [adding MPT maintainers]
>
> Look at the irqs/sec rate, it's higher by about the same percentage. So
> it's likely not a more costly irq handler, it's likely just called that
> much more. It could be IO pattern, causing more commands to be issued
> (which leads to more interrupts, etc).

Yes, but the irqs/sec increase doesn't appear to be due to MPT interrupts.
In the /proc/interrupt summaries, RH5 did 388666895 IOC interrupts and
2.6.30 did 378419042. As a percentage of interrupts, the IOC interrupts
were 59.4% with RH and 51.8% with 2.6.30.

This isn't quite conclusive since the collection of /proc/interrupts is
over the entire life of the system, not during the measurement period.
But I do find it persuasive.

--
Matthew Wilcox Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."

2009-07-03 19:22:41

by Jens Axboe

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Fri, Jul 03 2009, Matthew Wilcox wrote:
> On Fri, Jul 03, 2009 at 08:54:14PM +0200, Jens Axboe wrote:
> > On Fri, Jul 03 2009, Andi Kleen wrote:
> > >
> > > Matthew Wilcox <[email protected]> writes:
> > > >
> > > > ======oprofile CPU_CLK_UNHALTED for top 30 functions
> > > > Cycles% 2.6.18-92.el5-op Cycles% 2.6.30
> > > > 70.1409 <database> 67.0207 <database>
> > > > 1.3556 mpt_interrupt 1.7029 mpt_interrupt
> > >
> > > It's strange that mpt_interrupt is that more costly in 2.6.30
> > > than in 2.6.18. I diffed 2.6.30's drivers/message/fusion/mptbase.c
> > > to a rhel 5.3s and they seem to be about the same.
> > >
> > > So why does it cost 0.5% more in 2.6.30?
> > >
> > > [adding MPT maintainers]
> >
> > Look at the irqs/sec rate, it's higher by about the same percentage. So
> > it's likely not a more costly irq handler, it's likely just called that
> > much more. It could be IO pattern, causing more commands to be issued
> > (which leads to more interrupts, etc).
>
> Yes, but the irqs/sec increase doesn't appear to be due to MPT interrupts.
> In the /proc/interrupt summaries, RH5 did 388666895 IOC interrupts and
> 2.6.30 did 378419042. As a percentage of interrupts, the IOC interrupts
> were 59.4% with RH and 51.8% with 2.6.30.

OK. So where are the extra irqs from?

> This isn't quite conclusive since the collection of /proc/interrupts is
> over the entire life of the system, not during the measurement period.
> But I do find it persuasive.

Since the total is so high, it's probably good enough and sampling only
during the measurement likely wouldn't change that picture a lot.

--
Jens Axboe

2009-07-03 19:46:02

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Fri, Jul 03, 2009 at 09:22:35PM +0200, Jens Axboe wrote:
> On Fri, Jul 03 2009, Matthew Wilcox wrote:
> > Yes, but the irqs/sec increase doesn't appear to be due to MPT interrupts.
> > In the /proc/interrupt summaries, RH5 did 388666895 IOC interrupts and
> > 2.6.30 did 378419042. As a percentage of interrupts, the IOC interrupts
> > were 59.4% with RH and 51.8% with 2.6.30.
>
> OK. So where are the extra irqs from?

Let's see:

Source 2.6.18 2.6.30 Delta
qla 0.8% 0.8% 0
eth 20% 27.6% +7.6%
ioc 59.4% 51.8% -7.6%
NMI 7.6% 7.9% +0.3%
LOC 12.2% 10% -2.2%
RES - 1.8% +1.8%

I wouldn't be surprised to find out that 2.6.18 accounted rescheduling
interrupts as 'LOC'. So the difference in interrupts is all about
the ethernet card. I believe these systems have an igb card.

The big difference between 2.6.18 and 2.6.30 is that the cards now have
eight interrupts in use each, instead of one each (four for rx queues
and four for tx queues). Distressingly, these interrupts are all affine
to the same CPUs (eth1's eight interrupts are all on CPU 9 and eth0's
interrupts are all on CPU 1). That would seem to be a fruitful avenue of
investigation -- whether limiting the cards to a single RX/TX interrupt
would be advantageous, or whether spreading the eight interrupts out
over the CPUs would be advantageous.

--
Matthew Wilcox Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."

2009-07-03 19:55:05

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

> That would seem to be a fruitful avenue of
> investigation -- whether limiting the cards to a single RX/TX interrupt
> would be advantageous, or whether spreading the eight interrupts out
> over the CPUs would be advantageous.

The kernel should really do the per cpu binding of MSIs by default.

-Andi

--
[email protected] -- Speaking for myself only.

2009-07-03 20:03:35

by Arjan van de Ven

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Fri, 3 Jul 2009 21:54:58 +0200
Andi Kleen <[email protected]> wrote:

> > That would seem to be a fruitful avenue of
> > investigation -- whether limiting the cards to a single RX/TX
> > interrupt would be advantageous, or whether spreading the eight
> > interrupts out over the CPUs would be advantageous.
>
> The kernel should really do the per cpu binding of MSIs by default.

... so that you can't do power management on a per socket basis?
hardly a good idea.

just need to use a new enough irqbalance and it will spread out the
interrupts unless your load is low enough to go into low power mode.

--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2009-07-03 23:35:18

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

> ... so that you can't do power management on a per socket basis?
> hardly a good idea.

Interrupts should only happen when the socket did spent time submitting
the work for them. Typically on drivers I'm familiar with the return
MSI is tied to the CPU that did submit the work item

> just need to use a new enough irqbalance and it will spread out the
> interrupts unless your load is low enough to go into low power mode.

When that load is low then likely you won't get many interrupts
anyways.

-Andi

--
[email protected] -- Speaking for myself only.

2009-07-04 06:04:36

by Arjan van de Ven

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Sat, 4 Jul 2009 01:35:05 +0200
Andi Kleen <[email protected]> wrote:

> > ... so that you can't do power management on a per socket basis?
> > hardly a good idea.
>
> Interrupts should only happen when the socket did spent time
> submitting the work for them. Typically on drivers I'm familiar with
> the return MSI is tied to the CPU that did submit the work item

for networking, especially for incoming data such as new connections,
that isn't the case.. that's more or less randomly (well hash based)
distributed.

--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2009-07-04 08:44:38

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

> for networking, especially for incoming data such as new connections,
> that isn't the case.. that's more or less randomly (well hash based)
> distributed.

Ok. Still binding them all to a single CPU all is quite dumb. It
makes MSI-X quite useless and probably even harmful.

We don't default to socket power saving for normal scheduling either,
but only when you specify a special knob. I don't see why interrupts
should be different.

-Andi

--
[email protected] -- Speaking for myself only.

2009-07-04 09:22:34

by Jeff Garzik

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

Andi Kleen wrote:
>> for networking, especially for incoming data such as new connections,
>> that isn't the case.. that's more or less randomly (well hash based)
>> distributed.
>
> Ok. Still binding them all to a single CPU all is quite dumb. It
> makes MSI-X quite useless and probably even harmful.
>
> We don't default to socket power saving for normal scheduling either,
> but only when you specify a special knob. I don't see why interrupts
> should be different.

In the pre-MSI-X days, you'd have cachelines bouncing all over the place
if you distributed networking interrupts across CPUs, particularly given
that NAPI would run some things on a single CPU anyway.

Today, machines are faster, we have multiple interrupts per device, and
we have multiple RX/TX queues. I would be interested to see hard
numbers (as opposed to guesses) about various new ways to distributed
interrupts across CPUs.

What's the best setup for power usage?
What's the best setup for performance?
Are they the same?
Is it most optimal to have the interrupt for socket $X occur on the same
CPU as where the app is running?
If yes, how to best handle when the scheduler moves app to another CPU?
Should we reprogram the NIC hardware flow steering mechanism at that point?

Interesting questions, and I hope we'd see some hard number comparisons
before solutions start flowing into the kernel.

Jeff

2009-07-05 04:02:36

by Herbert Xu

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

Jeff Garzik <[email protected]> wrote:
>
> What's the best setup for power usage?
> What's the best setup for performance?
> Are they the same?

Yes.

> Is it most optimal to have the interrupt for socket $X occur on the same
> CPU as where the app is running?

Yes.

> If yes, how to best handle when the scheduler moves app to another CPU?
> Should we reprogram the NIC hardware flow steering mechanism at that point?

Not really. For now the best thing to do is to pin everything
down and not move at all, because we can't afford to move.

The only way for moving to work is if we had the ability to get
the sockets to follow the processes. That means, we must have
one RX queue per socket.

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

2009-07-05 11:00:37

by Daniel J Blueman

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Jul 3, 9:10 pm, Arjan van de Ven <[email protected]> wrote:
> On Fri, 3 Jul 2009 21:54:58 +0200
>
> Andi Kleen <[email protected]> wrote:
> > > That would seem to be a fruitful avenue of
> > > investigation -- whether limiting the cards to a single RX/TX
> > > interrupt would be advantageous, or whether spreading the eight
> > > interrupts out over the CPUs would be advantageous.
>
> > The kernel should really do the per cpu binding of MSIs by default.
>
> ... so that you can't do power management on a per socket basis?
> hardly a good idea.
>
> just need to use a new enough irqbalance and it will spread out the
> interrupts unless your load is low enough to go into low power mode.

I was finding newer kernels (>~2.6.24) would set the Redirection Hint
bit in the MSI address vector, allowing the processors to deliver the
interrupt to the lowest interrupt priority (eg idle, no powersave)
core (http://www.intel.com/Assets/PDF/manual/253668.pdf pp10-66) and
older irqbalance daemons would periodically naively rewrite the
bitmask of cores, delivering the interrupt to a static one.

Thus, it may be worth checking if disabling any older irqbalance
daemon gives any win.

Perhaps there is value in writing different subsets of cores to the
MSI address vector core bitmask (with the redirection hint enabled)
for different I/O queues on heavy interrupt sources? By default, it's
all cores.

Daniel
--
Daniel J Blueman

2009-07-05 13:09:34

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Sun, Jul 05, 2009 at 12:01:37PM +0800, Herbert Xu wrote:
> > If yes, how to best handle when the scheduler moves app to another CPU?
> > Should we reprogram the NIC hardware flow steering mechanism at that point?
>
> Not really. For now the best thing to do is to pin everything
> down and not move at all, because we can't afford to move.
>
> The only way for moving to work is if we had the ability to get
> the sockets to follow the processes. That means, we must have
> one RX queue per socket.

Maybe not one RX queue per socket -- sockets belonging to the same
thread could share the same RX queue. I'm fairly ignorant of the way
networking works these days; is it possible to dynamically reassign a
socket between RX queues, so we'd only need one RX queue per CPU?

It seems the 82575 device has four queues per port, and it's a dual-port
card, so that's eight queues in the system. We'd need hundreds of queues
to get one queue per client process. The 82576 has sixteen queues per
port, but that's still not enough (funnily, the driver still limits you
to four per port).

For what it's worth, I believe the current setup pins the client tasks to
a package, but they are allowed to move between cores on that package.
My information may be out of date; hopefully Doug, Chinang or Terry can
clarify how the tasks are currently bound.

I know this test still uses SCHED_RR for the client tasks (using
SCHED_OTHER results in another couple of percentage points off the
overall performance).

--
Matthew Wilcox Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."

2009-07-05 16:12:40

by Herbert Xu

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Sun, Jul 05, 2009 at 07:09:26AM -0600, Matthew Wilcox wrote:
>
> Maybe not one RX queue per socket -- sockets belonging to the same
> thread could share the same RX queue. I'm fairly ignorant of the way
> networking works these days; is it possible to dynamically reassign a
> socket between RX queues, so we'd only need one RX queue per CPU?

Not reliably. You can tweak the hash in the NIC to redirect
traffic (aka flow director) but ultimately if you've got more
sockets than queues then it's a losing game.

A better strategy for now is to pin everything down and try to
get user-space involved by using threads and distributing the
sockets based on which queue they're associated with.

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

2009-07-05 20:46:26

by Jeff Garzik

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

Herbert Xu wrote:
> Jeff Garzik <[email protected]> wrote:
>> What's the best setup for power usage?
>> What's the best setup for performance?
>> Are they the same?
>
> Yes.

Is this a blind guess, or is there real world testing across multiple
setups behind this answer?

Consider a 2-package, quad-core system with 3 userland threads actively
performing network communication, plus periodic, low levels of network
activity from OS utilities (such as nightly 'yum upgrade').

That is essentially an under-utilized 8-CPU system. For such a case, it
seems like a power win to idle or power down a few cores, or maybe even
an entire package.

Efficient power usage means scaling _down_ when activity decreases. A
blind "distribute network activity across all CPUs" policy does not
appear to be responsive to that type of situation.

>> Is it most optimal to have the interrupt for socket $X occur on the same
>> CPU as where the app is running?
>
> Yes.

Same question: blind guess, or do you have numbers?

Consider two competing CPU hogs: a kernel with tons of netfilter tables
and rules, plus an application that uses a lot of CPU.

Can you not reach a threshold where it makes more sense to split kernel
and userland work onto different CPUs?

>> If yes, how to best handle when the scheduler moves app to another CPU?
>> Should we reprogram the NIC hardware flow steering mechanism at that point?
>
> Not really. For now the best thing to do is to pin everything
> down and not move at all, because we can't afford to move.
>
> The only way for moving to work is if we had the ability to get
> the sockets to follow the processes. That means, we must have
> one RX queue per socket.

That seems to presume it is impossible to reprogram the NIC RX queue
selection rules?

If you can add a new 'flow' to a NIC hardware RX queue, surely you can
imagine a remove + add operation for a migrated 'flow'... and thus, at
least on the NIC hardware level, flows can follow processes.

Jeff

2009-07-06 01:20:44

by Herbert Xu

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Sun, Jul 05, 2009 at 04:44:41PM -0400, Jeff Garzik wrote:
>
> Efficient power usage means scaling _down_ when activity decreases. A
> blind "distribute network activity across all CPUs" policy does not
> appear to be responsive to that type of situation.

I didn't suggest distributing network activity across all CPUs.
It should be distributed across all active CPUs.

> Consider two competing CPU hogs: a kernel with tons of netfilter tables
> and rules, plus an application that uses a lot of CPU.
>
> Can you not reach a threshold where it makes more sense to split kernel
> and userland work onto different CPUs?

In that case the best split would be split the application into
different threads which can then move onto different CPUs. Doing
what you said above will probably work assuming the CPUs share
cache, otherwise it will suck.

> That seems to presume it is impossible to reprogram the NIC RX queue
> selection rules?
>
> If you can add a new 'flow' to a NIC hardware RX queue, surely you can
> imagine a remove + add operation for a migrated 'flow'... and thus, at
> least on the NIC hardware level, flows can follow processes.

Right, that would work as long as you can add a rule for each socket
you cared about. Though it's interesting to know whether the number
of rules can keep up with the number of sockets that we usually
encounter on busy servers.

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

2009-07-06 08:38:39

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Sun, Jul 05, 2009 at 07:09:26AM -0600, Matthew Wilcox wrote:
> On Sun, Jul 05, 2009 at 12:01:37PM +0800, Herbert Xu wrote:
> > > If yes, how to best handle when the scheduler moves app to another CPU?
> > > Should we reprogram the NIC hardware flow steering mechanism at that point?
> >
> > Not really. For now the best thing to do is to pin everything
> > down and not move at all, because we can't afford to move.
> >
> > The only way for moving to work is if we had the ability to get
> > the sockets to follow the processes. That means, we must have
> > one RX queue per socket.
>
> Maybe not one RX queue per socket -- sockets belonging to the same
> thread could share the same RX queue. I'm fairly ignorant of the way
> networking works these days; is it possible to dynamically reassign a
> socket between RX queues, so we'd only need one RX queue per CPU?

That is how it is supposed to work (ignoring some special setups
with QoS) in theory.

a You have per CPU RX queues (or if the NIC has less than CPUs, then on a
subset of CPUs)
b The NIC uses a hash function on the stream (= socket) to map an
incoming packet to a specific RX queue.
c The interrupt handler is supposed to be bound on a specific CPU.
d The CPU then does wakeups and the scheduler biases the process/thread
using the sockets towards the CPU that always does the wakeups.

Ideally then the process/thread doing the socket IO should be on
the receiving CPU. It doesn't always work out like this in practice,
but it should.

(c) seems to be the part that is broken right now.

-Andi

--
[email protected] -- Speaking for myself only.

2009-07-06 08:45:45

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

> That seems to presume it is impossible to reprogram the NIC RX queue
> selection rules?
>
> If you can add a new 'flow' to a NIC hardware RX queue, surely you can
> imagine a remove + add operation for a migrated 'flow'... and thus, at
> least on the NIC hardware level, flows can follow processes.

The standard on modern NIC hardware is stateless hashing: as in you
don't program in flows, but the hardware just uses a fixed hash
to map the header to a rx queue. You can't really significantly influence
the hash on a per flow basis there (in theory you could chose
specific local port numbers, but you can't do that for the remote
ports or for well known sockets)

There are a few highend NICs that optionally support arbitary flow matching,
but it typically only supports a very limited number of flows
(so you need some fallback anyways) and of course it would
be costly to reprogram the NIC on every socket state change.

-Andi
--
[email protected] -- Speaking for myself only.

2009-07-06 17:00:25

by Rick Jones

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

Herbert Xu wrote:
> Jeff Garzik <[email protected]> wrote:
>
>>What's the best setup for power usage?
>>What's the best setup for performance?
>>Are they the same?
>
>
> Yes.
>
>
>>Is it most optimal to have the interrupt for socket $X occur on the same
>>CPU as where the app is running?
>
>
> Yes.

Well... Yes, if the goal is lowest service demand/latency, but not always if
the goal is to have highest throughput. For example, basic netperf TCP_RR
between a pair of systems with NIC interrupts pinned to CPU0 for my convenience :)

Pin netperf/netserver to CPU0 as well:
sbs133b15:~ # netperf -H sbs133b16 -t TCP_RR -T 0 -c -C
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
sbs133b16.west (10.208.1.50) port 0 AF_INET : first burst 0 : cpu bind
Local /Remote
Socket Size Request Resp. Elapsed Trans. CPU CPU S.dem S.dem
Send Recv Size Size Time Rate local remote local remote
bytes bytes bytes bytes secs. per sec % S % S us/Tr us/Tr

16384 87380 1 1 10.00 16396.22 0.39 0.55 3.846 5.364
16384 87380

Now pin it to the peer thread in that same core:

sbs133b15:~ # netperf -H sbs133b16 -t TCP_RR -T 8 -c -C
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
sbs133b16.west (10.208.1.50) port 0 AF_INET : first burst 0 : cpu bind
Local /Remote
Socket Size Request Resp. Elapsed Trans. CPU CPU S.dem S.dem
Send Recv Size Size Time Rate local remote local remote
bytes bytes bytes bytes secs. per sec % S % S us/Tr us/Tr

16384 87380 1 1 10.00 14078.23 0.67 0.87 7.604 9.863
16384 87380

Now pin it to another core in that same processor:

sbs133b15:~ # netperf -H sbs133b16 -t TCP_RR -T 2 -c -C
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
sbs133b16.west (10.208.1.50) port 0 AF_INET : first burst 0 : cpu bind
Local /Remote
Socket Size Request Resp. Elapsed Trans. CPU CPU S.dem S.dem
Send Recv Size Size Time Rate local remote local remote
bytes bytes bytes bytes secs. per sec % S % S us/Tr us/Tr

16384 87380 1 1 10.00 14649.57 1.76 0.64 19.213 7.036
16384 87380

Certainly seems to support "run on the same core as interrupts." Now though lets
look at bulk throughput:

sbs133b15:~ # netperf -H sbs133b16 -T 0 -c -C
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to sbs133b16.west
(10.208.1.50) port 0 AF_INET : cpu bind
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB

87380 16384 16384 10.00 9384.11 3.39 2.19 0.474 0.306

In this case, I'm running on Nehalems (two quad-cores with threads enabled) so I
have enough "oomph" to hit link-rate on a classic throughput test so all these
next two will show is the CPU hit and some of the run to run variablity:

sbs133b15:~ # for t in 8 2; do netperf -P 0 -H sbs133b16 -T $t -c -C -B "bind to
core $t"; done
87380 16384 16384 10.00 9383.67 4.23 5.21 0.591 0.728
bind to core 8
87380 16384 16384 10.00 9383.12 3.03 5.35 0.423 0.747
bind to core 2

So apart from the thing on the top of my head what is my point? Let's look at a
less conventional but still important case - bulk small packet throughput.
First, find the limit for a single connection when bound to the interrupt core:

sbs133b15:~ # for b in 0 4 16 64 128 256; do netperf -P 0 -t TCP_RR -T 0 -H
sbs133b16 -c -C -B "$b added simultaneous trans" -- -D -b $b; done
16384 87380 1 1 10.00 16336.52 0.69 0.91 6.715 8.944 0
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 61324.84 2.23 2.27 5.825 5.910 4
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 152221.78 2.81 3.49 2.956 3.664 16
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 291247.72 4.86 5.07 2.670 2.788 64
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 292257.59 3.99 5.91 2.183 3.236 128
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 291734.00 5.55 5.32 3.043 2.920 256
added simultaneous trans
16384 87380

Now, when bound to the peer thread:
sbs133b15:~ # for b in 0 4 16 64 128 256; do netperf -P 0 -t TCP_RR -T 8 -H
sbs133b16 -c -C -B "$b added simultaneous trans" -- -D -b $b; done
16384 87380 1 1 10.00 14367.40 0.78 1.75 8.652 19.477 0
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 54820.22 2.73 4.78 7.956 13.948 4
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 159305.92 4.61 6.84 4.627 6.874 16
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 260227.55 6.26 8.36 3.851 5.140 64
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 256336.50 6.23 8.00 3.891 4.993 128
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 250543.92 6.24 6.29 3.985 4.014 256
added simultaneous trans
16384 87380

Things still don't look good for running on another CPU, but wait :) Bind to
another core in the same processor:

sbs133b15:~ # for b in 0 4 16 64 128 256; do netperf -P 0 -t TCP_RR -T 2 -H
sbs133b16 -c -C -B "$b added simultaneous trans" -- -D -b $b; done
16384 87380 1 1 10.00 14697.98 0.89 1.53 9.689 16.700 0
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 58201.08 2.11 4.21 5.804 11.585 4
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 158999.50 3.87 6.20 3.899 6.240 16
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 379243.72 6.24 9.04 2.634 3.815 64
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 384823.34 6.15 9.50 2.556 3.949 128
added simultaneous trans
16384 87380
16384 87380 1 1 10.00 375001.50 6.07 9.63 2.588 4.109 256
added simultaneous trans
16384 87380

When the CPU does not have enough "oomph" for link-rate 10G, then what we see
above with the aggregate TCP_RR holds true for a plain TCP_STREAM test as well -
getting the second core involved, while indeed increasing CPU util, also
provides the additional cycles required to get higher thoughput. So what is
optimal depends on what one wishes to optimize.

>
>>If yes, how to best handle when the scheduler moves app to another CPU?
>>Should we reprogram the NIC hardware flow steering mechanism at that point?
>
>
> Not really. For now the best thing to do is to pin everything
> down and not move at all, because we can't afford to move.
>
> The only way for moving to work is if we had the ability to get
> the sockets to follow the processes. That means, we must have
> one RX queue per socket.

Well, or assign sockets to per-core RX queues and be able to move them around.
If it weren't for all the smarts in the NICs getting in the way :), we'd
probably do the "lookup where the socket was last accessed and run there" thing
somewhere in the inbound path a la TOPS.

rick jones

>
> Cheers,

2009-07-06 17:36:25

[permalink] [raw]

Subject: RE: >10% performance degradation since 2.6.18

For OLTP workload we are not pushing much network throughput. Lower network latency is more important for OLTP performance. For the original Nehalem 2 sockets OLTP result in this mail thread, we bound the two NIC interrupts to cpu1 and cpu9 (one NIC per sockets). Database processes are divided into two groups and pinned to socket and each processe only received request from the NIC it bound to. This binding scheme gave us >1% performance boost pre-Nehalem date. We also see positive impact on this NHM system.
-Chinang

>-----Original Message-----
>From: Rick Jones [mailto:[email protected]]
>Sent: Monday, July 06, 2009 10:00 AM
>To: Herbert Xu
>Cc: Jeff Garzik; [email protected]; [email protected]; [email protected];
>[email protected]; [email protected]; Styner, Douglas W; Ma,
>Chinang; Prickett, Terry O; Wilcox, Matthew R; [email protected]; DL-
>[email protected]; [email protected]
>Subject: Re: >10% performance degradation since 2.6.18
>
>Herbert Xu wrote:
>> Jeff Garzik <[email protected]> wrote:
>>
>>>What's the best setup for power usage?
>>>What's the best setup for performance?
>>>Are they the same?
>>
>>
>> Yes.
>>
>>
>>>Is it most optimal to have the interrupt for socket $X occur on the same
>>>CPU as where the app is running?
>>
>>
>> Yes.
>
>Well... Yes, if the goal is lowest service demand/latency, but not always
>if
>the goal is to have highest throughput. For example, basic netperf TCP_RR
>between a pair of systems with NIC interrupts pinned to CPU0 for my
>convenience :)
>
>Pin netperf/netserver to CPU0 as well:
>sbs133b15:~ # netperf -H sbs133b16 -t TCP_RR -T 0 -c -C
>TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
>sbs133b16.west (10.208.1.50) port 0 AF_INET : first burst 0 : cpu bind
>Local /Remote
>Socket Size Request Resp. Elapsed Trans. CPU CPU S.dem S.dem
>Send Recv Size Size Time Rate local remote local remote
>bytes bytes bytes bytes secs. per sec % S % S us/Tr us/Tr
>
>16384 87380 1 1 10.00 16396.22 0.39 0.55 3.846 5.364
>16384 87380
>
>Now pin it to the peer thread in that same core:
>
>sbs133b15:~ # netperf -H sbs133b16 -t TCP_RR -T 8 -c -C
>TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
>sbs133b16.west (10.208.1.50) port 0 AF_INET : first burst 0 : cpu bind
>Local /Remote
>Socket Size Request Resp. Elapsed Trans. CPU CPU S.dem S.dem
>Send Recv Size Size Time Rate local remote local remote
>bytes bytes bytes bytes secs. per sec % S % S us/Tr us/Tr
>
>16384 87380 1 1 10.00 14078.23 0.67 0.87 7.604 9.863
>16384 87380
>
>Now pin it to another core in that same processor:
>
>sbs133b15:~ # netperf -H sbs133b16 -t TCP_RR -T 2 -c -C
>TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
>sbs133b16.west (10.208.1.50) port 0 AF_INET : first burst 0 : cpu bind
>Local /Remote
>Socket Size Request Resp. Elapsed Trans. CPU CPU S.dem S.dem
>Send Recv Size Size Time Rate local remote local remote
>bytes bytes bytes bytes secs. per sec % S % S us/Tr us/Tr
>
>16384 87380 1 1 10.00 14649.57 1.76 0.64 19.213 7.036
>16384 87380
>
>Certainly seems to support "run on the same core as interrupts." Now though
>lets
>look at bulk throughput:
>
>sbs133b15:~ # netperf -H sbs133b16 -T 0 -c -C
>TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to sbs133b16.west
>(10.208.1.50) port 0 AF_INET : cpu bind
>Recv Send Send Utilization Service
>Demand
>Socket Socket Message Elapsed Send Recv Send Recv
>Size Size Size Time Throughput local remote local
>remote
>bytes bytes bytes secs. 10^6bits/s % S % S us/KB
>us/KB
>
> 87380 16384 16384 10.00 9384.11 3.39 2.19 0.474
>0.306
>
>In this case, I'm running on Nehalems (two quad-cores with threads enabled)
>so I
>have enough "oomph" to hit link-rate on a classic throughput test so all
>these
>next two will show is the CPU hit and some of the run to run variablity:
>
>sbs133b15:~ # for t in 8 2; do netperf -P 0 -H sbs133b16 -T $t -c -C -B
>"bind to
>core $t"; done
> 87380 16384 16384 10.00 9383.67 4.23 5.21 0.591
>0.728
>bind to core 8
> 87380 16384 16384 10.00 9383.12 3.03 5.35 0.423
>0.747
>bind to core 2
>
>So apart from the thing on the top of my head what is my point? Let's look
>at a
>less conventional but still important case - bulk small packet throughput.
>First, find the limit for a single connection when bound to the interrupt
>core:
>
>sbs133b15:~ # for b in 0 4 16 64 128 256; do netperf -P 0 -t TCP_RR -T 0 -H
>sbs133b16 -c -C -B "$b added simultaneous trans" -- -D -b $b; done
>16384 87380 1 1 10.00 16336.52 0.69 0.91 6.715 8.944
>0
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 61324.84 2.23 2.27 5.825 5.910
>4
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 152221.78 2.81 3.49 2.956 3.664
>16
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 291247.72 4.86 5.07 2.670 2.788
>64
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 292257.59 3.99 5.91 2.183 3.236
>128
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 291734.00 5.55 5.32 3.043 2.920
>256
>added simultaneous trans
>16384 87380
>
>Now, when bound to the peer thread:
>sbs133b15:~ # for b in 0 4 16 64 128 256; do netperf -P 0 -t TCP_RR -T 8 -H
>sbs133b16 -c -C -B "$b added simultaneous trans" -- -D -b $b; done
>16384 87380 1 1 10.00 14367.40 0.78 1.75 8.652 19.477
>0
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 54820.22 2.73 4.78 7.956 13.948
>4
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 159305.92 4.61 6.84 4.627 6.874
>16
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 260227.55 6.26 8.36 3.851 5.140
>64
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 256336.50 6.23 8.00 3.891 4.993
>128
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 250543.92 6.24 6.29 3.985 4.014
>256
>added simultaneous trans
>16384 87380
>
>Things still don't look good for running on another CPU, but wait :) Bind
>to
>another core in the same processor:
>
>sbs133b15:~ # for b in 0 4 16 64 128 256; do netperf -P 0 -t TCP_RR -T 2 -H
>sbs133b16 -c -C -B "$b added simultaneous trans" -- -D -b $b; done
>16384 87380 1 1 10.00 14697.98 0.89 1.53 9.689 16.700
>0
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 58201.08 2.11 4.21 5.804 11.585
>4
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 158999.50 3.87 6.20 3.899 6.240
>16
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 379243.72 6.24 9.04 2.634 3.815
>64
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 384823.34 6.15 9.50 2.556 3.949
>128
>added simultaneous trans
>16384 87380
>16384 87380 1 1 10.00 375001.50 6.07 9.63 2.588 4.109
>256
>added simultaneous trans
>16384 87380
>
>When the CPU does not have enough "oomph" for link-rate 10G, then what we
>see
>above with the aggregate TCP_RR holds true for a plain TCP_STREAM test as
>well -
>getting the second core involved, while indeed increasing CPU util, also
>provides the additional cycles required to get higher thoughput. So what
>is
>optimal depends on what one wishes to optimize.
>
>>
>>>If yes, how to best handle when the scheduler moves app to another CPU?
>>>Should we reprogram the NIC hardware flow steering mechanism at that
>point?
>>
>>
>> Not really. For now the best thing to do is to pin everything
>> down and not move at all, because we can't afford to move.
>>
>> The only way for moving to work is if we had the ability to get
>> the sockets to follow the processes. That means, we must have
>> one RX queue per socket.
>
>Well, or assign sockets to per-core RX queues and be able to move them
>around.
>If it weren't for all the smarts in the NICs getting in the way :), we'd
>probably do the "lookup where the socket was last accessed and run there"
>thing
>somewhere in the inbound path a la TOPS.
>
>rick jones
>
>>
>> Cheers,

2009-07-06 17:42:18

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Mon, Jul 06, 2009 at 10:36:11AM -0700, Ma, Chinang wrote:
> For OLTP workload we are not pushing much network throughput. Lower network latency is more important for OLTP performance. For the original Nehalem 2 sockets OLTP result in this mail thread, we bound the two NIC interrupts to cpu1 and cpu9 (one NIC per sockets). Database processes are divided into two groups and pinned to socket and each processe only received request from the NIC it bound to. This binding scheme gave us >1% performance boost pre-Nehalem date. We also see positive impact on this NHM system.

So you've tried spreading the four RX and TX interrupts for each card
out over, say, CPUs 1, 3, 5, 7 for eth1 and then 9, 11, 13, 15 for eth0,
and it produces worse performance than having all four tied to CPUs 1
and 9? Interesting.

Can you try changing IGB_MAX_RX_QUEUES (in drivers/net/igb/igb.h, about
line 60) to 1, and seeing if performance improves that way?

--
Matthew Wilcox Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."

2009-07-06 17:57:20

[permalink] [raw]

Subject: RE: >10% performance degradation since 2.6.18

>-----Original Message-----
>From: Matthew Wilcox [mailto:[email protected]]
>Sent: Monday, July 06, 2009 10:42 AM
>To: Ma, Chinang
>Cc: Rick Jones; Herbert Xu; Jeff Garzik; [email protected];
>[email protected]; [email protected]; [email protected];
>Styner, Douglas W; Prickett, Terry O; Wilcox, Matthew R;
>[email protected]; Brandeburg, Jesse
>Subject: Re: >10% performance degradation since 2.6.18
>
>On Mon, Jul 06, 2009 at 10:36:11AM -0700, Ma, Chinang wrote:
>> For OLTP workload we are not pushing much network throughput. Lower
>network latency is more important for OLTP performance. For the original
>Nehalem 2 sockets OLTP result in this mail thread, we bound the two NIC
>interrupts to cpu1 and cpu9 (one NIC per sockets). Database processes are
>divided into two groups and pinned to socket and each processe only
>received request from the NIC it bound to. This binding scheme gave us >1%
>performance boost pre-Nehalem date. We also see positive impact on this NHM
>system.
>
>So you've tried spreading the four RX and TX interrupts for each card
>out over, say, CPUs 1, 3, 5, 7 for eth1 and then 9, 11, 13, 15 for eth0,
>and it produces worse performance than having all four tied to CPUs 1
>and 9? Interesting.

I was comparing 2 NIC on 2 sockets to 2 NIC on the same socket. I have not tried spreading out the interrupt for a NIC to cpus in the same sockets. Is there good reason for trying this?

>
>Can you try changing IGB_MAX_RX_QUEUES (in drivers/net/igb/igb.h, about
>line 60) to 1, and seeing if performance improves that way?
>

I suppose this should wait until we find out whether spread out NIC interrupt in socket helps or not.

>--
>Matthew Wilcox Intel Open Source Technology Centre
>"Bill, look, we understand that you're interested in selling us this
>operating system, but compare it to ours. We can't possibly take such
>a retrograde step."

2009-07-06 18:05:38

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Mon, Jul 06, 2009 at 10:57:09AM -0700, Ma, Chinang wrote:
> >-----Original Message-----
> >From: Matthew Wilcox [mailto:[email protected]]
> >Sent: Monday, July 06, 2009 10:42 AM
> >To: Ma, Chinang
> >Cc: Rick Jones; Herbert Xu; Jeff Garzik; [email protected];
> >[email protected]; [email protected]; [email protected];
> >Styner, Douglas W; Prickett, Terry O; Wilcox, Matthew R;
> >[email protected]; Brandeburg, Jesse
> >Subject: Re: >10% performance degradation since 2.6.18
> >
> >On Mon, Jul 06, 2009 at 10:36:11AM -0700, Ma, Chinang wrote:
> >> For OLTP workload we are not pushing much network throughput. Lower
> >network latency is more important for OLTP performance. For the original
> >Nehalem 2 sockets OLTP result in this mail thread, we bound the two NIC
> >interrupts to cpu1 and cpu9 (one NIC per sockets). Database processes are
> >divided into two groups and pinned to socket and each processe only
> >received request from the NIC it bound to. This binding scheme gave us >1%
> >performance boost pre-Nehalem date. We also see positive impact on this NHM
> >system.
> >
> >So you've tried spreading the four RX and TX interrupts for each card
> >out over, say, CPUs 1, 3, 5, 7 for eth1 and then 9, 11, 13, 15 for eth0,
> >and it produces worse performance than having all four tied to CPUs 1
> >and 9? Interesting.
>
> I was comparing 2 NIC on 2 sockets to 2 NIC on the same socket. I have not tried spreading out the interrupt for a NIC to cpus in the same sockets. Is there good reason for trying this?

If spreading the network load from one CPU to two CPUs increases the
performance, spreading the network load from two to eight might get even
better performance.

Are the clients now pinned to the CPU package on which they receive their
network interrupts?

> >Can you try changing IGB_MAX_RX_QUEUES (in drivers/net/igb/igb.h, about
> >line 60) to 1, and seeing if performance improves that way?
>
> I suppose this should wait until we find out whether spread out NIC interrupt in socket helps or not.

Yes, let's try having the driver work the way that LAD designed it
first ;-)

They may not have been optimising for the database client workload,
of course.

--
Matthew Wilcox Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."

2009-07-06 18:49:15

[permalink] [raw]

Subject: RE: >10% performance degradation since 2.6.18

>-----Original Message-----
>From: Matthew Wilcox [mailto:[email protected]]
>Sent: Monday, July 06, 2009 11:06 AM
>To: Ma, Chinang
>Cc: Rick Jones; Herbert Xu; Jeff Garzik; [email protected];
>[email protected]; [email protected]; [email protected];
>Styner, Douglas W; Prickett, Terry O; Wilcox, Matthew R;
>[email protected]; Brandeburg, Jesse
>Subject: Re: >10% performance degradation since 2.6.18
>
>On Mon, Jul 06, 2009 at 10:57:09AM -0700, Ma, Chinang wrote:
>> >-----Original Message-----
>> >From: Matthew Wilcox [mailto:[email protected]]
>> >Sent: Monday, July 06, 2009 10:42 AM
>> >To: Ma, Chinang
>> >Cc: Rick Jones; Herbert Xu; Jeff Garzik; [email protected];
>> >[email protected]; [email protected]; [email protected];
>> >Styner, Douglas W; Prickett, Terry O; Wilcox, Matthew R;
>> >[email protected]; Brandeburg, Jesse
>> >Subject: Re: >10% performance degradation since 2.6.18
>> >
>> >On Mon, Jul 06, 2009 at 10:36:11AM -0700, Ma, Chinang wrote:
>> >> For OLTP workload we are not pushing much network throughput. Lower
>> >network latency is more important for OLTP performance. For the original
>> >Nehalem 2 sockets OLTP result in this mail thread, we bound the two NIC
>> >interrupts to cpu1 and cpu9 (one NIC per sockets). Database processes
>are
>> >divided into two groups and pinned to socket and each processe only
>> >received request from the NIC it bound to. This binding scheme gave us
>>1%
>> >performance boost pre-Nehalem date. We also see positive impact on this
>NHM
>> >system.
>> >
>> >So you've tried spreading the four RX and TX interrupts for each card
>> >out over, say, CPUs 1, 3, 5, 7 for eth1 and then 9, 11, 13, 15 for eth0,
>> >and it produces worse performance than having all four tied to CPUs 1
>> >and 9? Interesting.
>>
>> I was comparing 2 NIC on 2 sockets to 2 NIC on the same socket. I have
>not tried spreading out the interrupt for a NIC to cpus in the same sockets.
>Is there good reason for trying this?
>
>If spreading the network load from one CPU to two CPUs increases the
>performance, spreading the network load from two to eight might get even
>better performance.
>

On the distributing interrupt load subject. Can we do the same thing with the IOC interrupts. We have 4 LSI 3801, The number of i/o interrupt is huge on 4 of the cpus. Is there a way to divide the IOC irq number so we can spread out the i/o interrupt to more cpu?

>Are the clients now pinned to the CPU package on which they receive their
>network interrupts?

Yes. The database processes in server are pinning to the socket on which they received network interrupts.

>
>> >Can you try changing IGB_MAX_RX_QUEUES (in drivers/net/igb/igb.h, about
>> >line 60) to 1, and seeing if performance improves that way?
>>
>> I suppose this should wait until we find out whether spread out NIC
>interrupt in socket helps or not.
>
>Yes, let's try having the driver work the way that LAD designed it
>first ;-)
>
>They may not have been optimising for the database client workload,
>of course.
>
>--
>Matthew Wilcox Intel Open Source Technology Centre
>"Bill, look, we understand that you're interested in selling us this
>operating system, but compare it to ours. We can't possibly take such
>a retrograde step."

2009-07-06 18:54:03

[permalink] [raw]

Subject: Re: >10% performance degradation since 2.6.18

On Mon, Jul 06, 2009 at 11:48:47AM -0700, Ma, Chinang wrote:
> On the distributing interrupt load subject. Can we do the same thing with the IOC interrupts. We have 4 LSI 3801, The number of i/o interrupt is huge on 4 of the cpus. Is there a way to divide the IOC irq number so we can spread out the i/o interrupt to more cpu?

I think the LSI 3801 only has one interrupt per ioc, but they could
perhaps be better spread-out. Right now, they're delivering interrupts
to CPUs 2, 3, 4 and 5. Possibly spreading them out to CPUs 2, 6, 10
and 14 would help. Or maybe it would hurt ...

(nb: the ioc interrupts are similarly tied to CPUs 2, 3, 4 and 5 with
2.6.18, so this isn't a likely cause of/solution to the regression,
it may just be a path to better numbers).

--
Matthew Wilcox Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."

2009-07-06 21:17:01

[permalink] [raw]

Subject: RE: >10% performance degradation since 2.6.18

Trying to answer Andi's questions below

>-----Original Message-----
>From: Andi Kleen [mailto:[email protected]]
>Sent: Friday, July 03, 2009 11:19 AM
>To: Matthew Wilcox
>Cc: [email protected]; Styner, Douglas W; Ma, Chinang; Prickett,
>Terry O; Wilcox, Matthew R; [email protected]; [email protected]
>Subject: Re: >10% performance degradation since 2.6.18
>
>
>Matthew Wilcox <[email protected]> writes:
>>
>> ======oprofile CPU_CLK_UNHALTED for top 30 functions
>> Cycles% 2.6.18-92.el5-op Cycles% 2.6.30
>> 70.1409 <database> 67.0207 <database>
>> 1.3556 mpt_interrupt 1.7029 mpt_interrupt
>
>It's strange that mpt_interrupt is that more costly in 2.6.30
>than in 2.6.18. I diffed 2.6.30's drivers/message/fusion/mptbase.c
>to a rhel 5.3s and they seem to be about the same.
>
>So why does it cost 0.5% more in 2.6.30?
>
>[adding MPT maintainers]
>
>> 1.1622 __blockdev_direct_IO 1.1443 kmem_cache_alloc
>
>It would be interesting to find out why kmem_cache_alloc
>is that more expensive. Either it is called more or the allocator
>is slower. Any chance of a callgraph profile run so we
>can see the callers?
>

We will try collecting the oprofile callgraph data.

>(you would need to rebuild the kernel with frame pointers)
>
>An interesting would be to simply count them on both kernels, e.g. with a
>simple systemtap script (see script at end of mail)
>
>> 0.8246 kmem_cache_free 0.8801 kmem_cache_free
>> 0.7108 schedule 0.7774 __blockdev_direct_IO
>> 0.6733 scsi_request_fn 0.7031 scsi_request_fn
>> 0.6114 kmem_cache_alloc 0.5317 __schedule
>> 0.4207 follow_hugetlb_page 0.3922 task_rq_lock
>> 0.4062 list_del 0.3629 sd_prep_fn
>> 0.3400 __switch_to 0.3504 list_del
>
>A callgraph for the list_del/list_dels would be interesting.
>Presumably that's cache misses, maybe we can do something
>here with some strategic prefetches or more batching.
>
>> 0.3339 generic_make_request 0.3382 __sigsetjmp
>
>__sigsetjmp is in glibc, isn't it?

Yes, __sigsetjmp from glibc

>
>> 0.3204 memmove 0.3270 __switch_to
>> 0.3088 __sigsetjmp 0.3257 generic_make_request
>> 0.2848 get_request 0.3116 kfree
>> 0.2804 lock_timer_base 0.2895 memmove
>
>memmove? Is that glibc or the kernel? (it would be good if that
>was marked in the listing)

Memmove is from glibc. I will filter out all user space libraries in the future report

>
>Back when I wrote the x86-64 string functions I didn't consider memmove
>too performance critical. So the x86-64 kernel memmove is actually
>not very optimized, especially the backwards case is very dumb
>and copies byte by byte.
>
>If it's that there might be some low hanging fruit here,
>on the other hand it's not a lot of CPU time. The systemtap
>script at the end measures this also.
>
>A callgraph for it would be useful if it's kernel.
>
>> 0.2572 __end_that_request_first 0.2402 aio_complete
>> 0.2567 fget_light 0.2382 mptscsih_qcmd
>> 0.2531 submit_page_section 0.2342 fget
>> 0.2428 mempool_alloc 0.2277 gup_huge_pmd
>> 0.2428 __generic_file_aio_read 0.2264 submit_page_section
>> 0.2368 touch_atime 0.2204 touch_atime
>
>Did you not run with noatime? If it runs with noatime and touch_atime
>still takes that many cycles something must be wrong.
>

Did not try noatime option yet. Will try it next time.

>Ah it might be the lock in mnt_want_write(). How about the
>appended dumb (untested) patch to first check for all the noatime
>cases before taking the lock? Even without noatime, but standard
>relatime that will hopefully eliminate the locking cost
>in most cases.
>
>> 0.2270 __aio_get_req 0.2165 __list_add
>> 0.2223 mptscsih_qcmd 0.2063 scsi_dispatch_cmd
>> 0.2198 init_request_from_bio 0.2040 lock_timer_base
>> 0.2191 fget 0.2036 irq_entries_start
>> 0.2141 device_not_available 0.2036 plist_del
>
>That plist_del is new. calltrace?

Not sure who use this. It is part of vmlinux

>
>> 0.2125 try_to_wake_up 0.2004 elv_queue_empty
>> 0.2065 mptscsih_io_done 0.2004 get_user_pages_fast
>> 0.2059 math_state_restore 0.1997 copy_user_generic_string
>> 0.2035 __errno_location 0.1925 kref_get
>> 0.2022 find_vma 0.1905 scsi_finish_command
>> 0.1967 _setjmp 0.1888 aio_rw_vect_retry
>> 0.1966 kref_get 0.1882 __errno_location
>
>That's not kernel, but it's weird that __errno_location is that
>costly. AFAIK it should be a simple thread local data lookup
>
>> 0.1952 memset 0.1849 scsi_device_unbusy
>> 0.1924 __list_add 0.1846 pick_next_highest_task_rt
>> 0.1917 copy_user_generic 0.1826 memset_c
>> 0.1907 acpi_os_read_port 0.1819 _setjmp
>> 0.1842 elv_queue_empty 0.1816 ipc_lock
>
>So it looks like ipc_lock needs so more scalability work.
>
Ipc_lock is not too high in cycles in this case. We can reduce the semaphore array size to try lower the contention.

>> 0.1809 scsi_dispatch_cmd 0.1809 mod_timer
>> 0.1808 sd_init_command 0.1800 noop_queue_empty
>> 0.1789 swiotlb_unmap_sg 0.1796 scsi_softirq_done
>> 0.1766 rw_verify_area 0.1757 scsi_run_queue
>> -----------------------------------
>
>---
>
>Systemtap script to measure some interesting metrics. Run
>with stap script.stp and then ctrl-c to get data out.
>
>---

Will try collect this data.

>
>global kmem_cache_alloc
>global kmem_cache_free
>
>probe kernel.function("kmem_cache_alloc") {
> kmem_cache_alloc++
>
>}
>
>probe kernel.function("kmem_cache_free") {
> kmem_cache_free++
>}
>
>global memmove_total, memmove_backwards
>
>probe kernel.function("memmove") {
> memmove_total++
> if (ulong_arg(1) < ulong_arg(2)) {
> memmove_backwards++
> }
>}
>
>probe end {
> printf("%u kmem_cache_allocs\n", kmem_cache_alloc)
> printf("%u kmem_cache_frees\n", kmem_cache_free)
> printf("%u memmoves, %u backwards\n", memmove_total,
>memmove_backwards);
>}
>
>
>---
>
>Only do mnt_want_write after all other options to avoid atime update are
>exhausted
>
>mnt_want_write takes a lock and can be fairly expensive. So do it as late
>as possible.
>
>Signed-off-by: Andi Kleen <[email protected]>
>
>---
> fs/inode.c | 20 ++++++++++----------
> 1 file changed, 10 insertions(+), 10 deletions(-)
>
>Index: linux-2.6.30-ak/fs/inode.c
>===================================================================
>--- linux-2.6.30-ak.orig/fs/inode.c
>+++ linux-2.6.30-ak/fs/inode.c
>@@ -1361,31 +1361,31 @@ void touch_atime(struct vfsmount *mnt, s
> struct inode *inode = dentry->d_inode;
> struct timespec now;
>
>- if (mnt_want_write(mnt))
>- return;
> if (inode->i_flags & S_NOATIME)
>- goto out;
>+ return;
> if (IS_NOATIME(inode))
>- goto out;
>+ return;
> if ((inode->i_sb->s_flags & MS_NODIRATIME) && S_ISDIR(inode->i_mode))
>- goto out;
>+ return;
>
> if (mnt->mnt_flags & MNT_NOATIME)
>- goto out;
>+ return;
> if ((mnt->mnt_flags & MNT_NODIRATIME) && S_ISDIR(inode->i_mode))
>- goto out;
>+ return;
>
> now = current_fs_time(inode->i_sb);
>
> if (!relatime_need_update(mnt, inode, now))
>- goto out;
>+ return;
>
> if (timespec_equal(&inode->i_atime, &now))
>- goto out;
>+ return;
>+
>+ if (mnt_want_write(mnt))
>+ return;
>
> inode->i_atime = now;
> mark_inode_dirty_sync(inode);
>-out:
> mnt_drop_write(mnt);
> }
> EXPORT_SYMBOL(touch_atime);
>
>
>--
>[email protected] -- Speaking for myself only.

2009-07-06 21:59:09

by Chetan.Loke

[permalink] [raw]

Subject: RE: >10% performance degradation since 2.6.18

> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of
> Daniel J Blueman
> Sent: Sunday, July 05, 2009 7:01 AM
> To: Matthew Wilcox; Andi Kleen
> Cc: Linux Kernel; Jens Axboe; Arjan van de Ven
> Subject: Re: >10% performance degradation since 2.6.18
>
> On Jul 3, 9:10 pm, Arjan van de Ven <[email protected]> wrote:
> > On Fri, 3 Jul 2009 21:54:58 +0200
> >
> > Andi Kleen <[email protected]> wrote:
> > > > That would seem to be a fruitful avenue of investigation --
> > > > whether limiting the cards to a single RX/TX interrupt would be
> > > > advantageous, or whether spreading the eight interrupts
> out over
> > > > the CPUs would be advantageous.
> >
> > > The kernel should really do the per cpu binding of MSIs
> by default.
> >
> > ... so that you can't do power management on a per socket basis?
> > hardly a good idea.
> >
> > just need to use a new enough irqbalance and it will spread out the
> > interrupts unless your load is low enough to go into low power mode.
>
> I was finding newer kernels (>~2.6.24) would set the
> Redirection Hint bit in the MSI address vector, allowing the
> processors to deliver the interrupt to the lowest interrupt
> priority (eg idle, no powersave) core
> (http://www.intel.com/Assets/PDF/manual/253668.pdf pp10-66)
> and older irqbalance daemons would periodically naively
> rewrite the bitmask of cores, delivering the interrupt to a
> static one.
>
> Thus, it may be worth checking if disabling any older
> irqbalance daemon gives any win.
>
> Perhaps there is value in writing different subsets of cores
> to the MSI address vector core bitmask (with the redirection
> hint enabled) for different I/O queues on heavy interrupt
> sources? By default, it's all cores.
>

Possible enhancement -

1) Drain the responses in the xmit_frame() path. That is, post the TX-request() and just before returning see if there are
any more responses in the RX-queue. This will minimize(only if the NIC f/w coalesces) interrupt load.
The n/w core should drain the responses rather than calling the drain-routine from the adapter's xmit_frame() handler. This way there won't be any need to
modify individual xmit_frame handlers.

PS - I'm not familiar with the networking code.

Chetan Loke-

2009-07-07 08:16:22