The regression is:
1)stoakley with 2 qual-core processors: 11%;
2)Tulsa with 4 dual-core(+hyperThread) processors:13%;
The test command is:
#sudo taskset -c 7 ./netserver
#sudo taskset -c 0 ./netperf -t TCP_RR -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -r 1,1
As a matter of fact, 2.6.23 has about 6% regression and 2.6.24-rc's
regression is between 16%~11%.
I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
but the bisected kernel wasn't stable and went crazy.
I tried both CONFIG_SLUB=y and CONFIG_SLAB=y to make sure SLUB isn't the
culprit.
The oprofile data of CONFIG_SLAB=y. Top cpu utilizations are:
1) 2.6.22
2067379 9.4888 vmlinux schedule
1873604 8.5994 vmlinux mwait_idle
1568131 7.1974 vmlinux resched_task
1066976 4.8972 vmlinux tcp_v4_rcv
986641 4.5285 vmlinux tcp_rcv_established
979518 4.4958 vmlinux find_busiest_group
767069 3.5207 vmlinux sock_def_readable
736808 3.3818 vmlinux tcp_sendmsg
595889 2.7350 vmlinux task_rq_lock
557193 2.5574 vmlinux tcp_ack
470570 2.1598 vmlinux __mod_timer
392220 1.8002 vmlinux __alloc_skb
358106 1.6436 vmlinux skb_release_data
313372 1.4383 vmlinux skb_clone
2) 2.6.24-rc7
2668426 12.4497 vmlinux vmlinux schedule
955698 4.4589 vmlinux vmlinux skb_release_data
836311 3.9018 vmlinux vmlinux tcp_v4_rcv
762398 3.5570 vmlinux vmlinux skb_release_all
728907 3.4007 vmlinux vmlinux task_rq_lock
705037 3.2894 vmlinux vmlinux __wake_up
694206 3.2388 vmlinux vmlinux __mod_timer
617616 2.8815 vmlinux vmlinux mwait_idle
It looks like tcp in 2.6.22 sends more packets, but frees far less skb than 2.6.24-rc6.
tcp_rcv_established in 2.6.22 is highlighted on cpu utilization.
-yanmin
Nobody is going to look directly into networking regressions on lkml,
please at least CC: [email protected] for networking issues.
Thank you.
On Wed, 2008-01-09 at 17:35 +0800, Zhang, Yanmin wrote:
> The regression is:
> 1)stoakley with 2 qual-core processors: 11%;
> 2)Tulsa with 4 dual-core(+hyperThread) processors:13%;
I have new update on this issue and also cc to netdev maillist.
Thank David Miller for pointing me the netdev maillist.
>
> The test command is:
> #sudo taskset -c 7 ./netserver
> #sudo taskset -c 0 ./netperf -t TCP_RR -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -r 1,1
>
> As a matter of fact, 2.6.23 has about 6% regression and 2.6.24-rc's
> regression is between 16%~11%.
>
> I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> but the bisected kernel wasn't stable and went crazy.
>
> I tried both CONFIG_SLUB=y and CONFIG_SLAB=y to make sure SLUB isn't the
> culprit.
>
> The oprofile data of CONFIG_SLAB=y. Top cpu utilizations are:
> 1) 2.6.22
> 2067379 9.4888 vmlinux schedule
> 1873604 8.5994 vmlinux mwait_idle
> 1568131 7.1974 vmlinux resched_task
> 1066976 4.8972 vmlinux tcp_v4_rcv
> 986641 4.5285 vmlinux tcp_rcv_established
> 979518 4.4958 vmlinux find_busiest_group
> 767069 3.5207 vmlinux sock_def_readable
> 736808 3.3818 vmlinux tcp_sendmsg
> 595889 2.7350 vmlinux task_rq_lock
> 557193 2.5574 vmlinux tcp_ack
> 470570 2.1598 vmlinux __mod_timer
> 392220 1.8002 vmlinux __alloc_skb
> 358106 1.6436 vmlinux skb_release_data
> 313372 1.4383 vmlinux skb_clone
>
> 2) 2.6.24-rc7
> 2668426 12.4497 vmlinux vmlinux schedule
> 955698 4.4589 vmlinux vmlinux skb_release_data
> 836311 3.9018 vmlinux vmlinux tcp_v4_rcv
> 762398 3.5570 vmlinux vmlinux skb_release_all
> 728907 3.4007 vmlinux vmlinux task_rq_lock
> 705037 3.2894 vmlinux vmlinux __wake_up
> 694206 3.2388 vmlinux vmlinux __mod_timer
> 617616 2.8815 vmlinux vmlinux mwait_idle
>
> It looks like tcp in 2.6.22 sends more packets, but frees far less skb than 2.6.24-rc6.
> tcp_rcv_established in 2.6.22 is highlighted on cpu utilization.
I instrumented kernel to capure the function call numbers.
1) 2.6.22
skb_release_data:50148649
tcp_ack: 25062858
tcp_transmit_skb:25063150
tcp_v4_rcv: 25063279
2) 2.6.24-rc6
skb_release_data:21429692
tcp_ack: 10707710
tcp_transmit_skb:10707866
tcp_v4_rcv: 10707959
The data doesn't show that 2.6.22 sends more packets while freeing far less skb than
2.6.24-rc6.
The data showed skb_release_data of kernel 2.6.22 is more than double of the one of
2.6.24-rc6. But netperf result just showed about 10% regression.
As the packet only has 1 byte, so I suspect 2.6.24-rc6 tries to merge packets after waiting for
a latency. 2.6.22 might haven't the wait latency or the latency is very small, so 2.6.22 almost
sends the packets immediately. I will check the source codes later.
-yanmin
>>The test command is:
>>#sudo taskset -c 7 ./netserver
>>#sudo taskset -c 0 ./netperf -t TCP_RR -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -r 1,1
A couple of comments/questions on the command lines:
*) netperf/netserver support CPU affinity within themselves with the
global -T option to netperf. Is the result with taskset much different?
The equivalent to the above would be to run netperf with:
./netperf -T 0,7 ...
The one possibly salient difference between the two is that when done
within netperf, the initial process creation will take place wherever
the scheduler wants it.
*) The -i option to set the confidence iteration count will silently cap
the max at 30.
happy benchmarking,
rick jones
On Fri, 2008-01-11 at 09:56 -0800, Rick Jones wrote:
> >>The test command is:
> >>#sudo taskset -c 7 ./netserver
> >>#sudo taskset -c 0 ./netperf -t TCP_RR -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -r 1,1
>
> A couple of comments/questions on the command lines:
Thanks for your kind comments.
>
> *) netperf/netserver support CPU affinity within themselves with the
> global -T option to netperf. Is the result with taskset much different?
> The equivalent to the above would be to run netperf with:
>
> ./netperf -T 0,7 ..
I checked the source codes and didn't find this option.
I use netperf V2.3 (I found the number in the makefile).
> .
>
> The one possibly salient difference between the two is that when done
> within netperf, the initial process creation will take place wherever
> the scheduler wants it.
>
> *) The -i option to set the confidence iteration count will silently cap
> the max at 30.
Indeed, you are right.
-yanmin
On Fri, 11 Jan 2008, Zhang, Yanmin wrote:
> On Wed, 2008-01-09 at 17:35 +0800, Zhang, Yanmin wrote:
> > The regression is:
> > 1)stoakley with 2 qual-core processors: 11%;
> > 2)Tulsa with 4 dual-core(+hyperThread) processors:13%;
> I have new update on this issue and also cc to netdev maillist.
> Thank David Miller for pointing me the netdev maillist.
>
> >
> > The test command is:
> > #sudo taskset -c 7 ./netserver
> > #sudo taskset -c 0 ./netperf -t TCP_RR -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -r 1,1
> >
> > As a matter of fact, 2.6.23 has about 6% regression and 2.6.24-rc's
> > regression is between 16%~11%.
> >
> > I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> > but the bisected kernel wasn't stable and went crazy.
TCP work between that is very much non-existing.
Using git-reset's to select a nearby merge point instead of default
commit where bisection lands might be help in case the bisected kernel
breaks.
Also, limiting bisection under a subsystem might reduce probability of
brokeness (might at least be able to narrow it down quite a lot), e.g.
git bisect start net/
--
i.
On Mon, 14 Jan 2008, Ilpo J?rvinen wrote:
> On Fri, 11 Jan 2008, Zhang, Yanmin wrote:
>
> > On Wed, 2008-01-09 at 17:35 +0800, Zhang, Yanmin wrote:
> > >
> > > As a matter of fact, 2.6.23 has about 6% regression and 2.6.24-rc's
> > > regression is between 16%~11%.
> > >
> > > I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> > > but the bisected kernel wasn't stable and went crazy.
>
> TCP work between that is very much non-existing.
I _really_ meant 2.6.22 - 2.6.23-rc1, not 2.6.24-rc1 in case you had a
typo there which is not that uncommon while typing kernel versions... :-)
--
i.
On Mon, 2008-01-14 at 11:21 +0200, Ilpo J?rvinen wrote:
> On Mon, 14 Jan 2008, Ilpo J?rvinen wrote:
>
> > On Fri, 11 Jan 2008, Zhang, Yanmin wrote:
> >
> > > On Wed, 2008-01-09 at 17:35 +0800, Zhang, Yanmin wrote:
> > > >
> > > > As a matter of fact, 2.6.23 has about 6% regression and 2.6.24-rc's
> > > > regression is between 16%~11%.
> > > >
> > > > I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> > > > but the bisected kernel wasn't stable and went crazy.
> >
> > TCP work between that is very much non-existing.
>
> I _really_ meant 2.6.22 - 2.6.23-rc1, not 2.6.24-rc1 in case you had a
> typo
I did bisect 2.6.22 - 2.6.23-rc1. I also tested it on the latest 2.6.24-rc.
> there which is not that uncommon while typing kernel versions... :-)
Thanks. I will retry bisect and bind the server/client to the same logical processor, where
I hope the result is stable this time when bisecting.
Manual testing showed there is still same or more regression if I bind the
processes on the same cpu.
Thanks a lot!
-yanmin
On Mon, Jan 14, 2008 at 08:44:40AM +0000, Ilpo J?rvinen wrote:
>
> > > I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> > > but the bisected kernel wasn't stable and went crazy.
>
> TCP work between that is very much non-existing.
Make sure you haven't switched between SLAB/SLUB while testing this.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
>>*) netperf/netserver support CPU affinity within themselves with the
>>global -T option to netperf. Is the result with taskset much different?
>> The equivalent to the above would be to run netperf with:
>>
>>./netperf -T 0,7 ..
>
> I checked the source codes and didn't find this option.
> I use netperf V2.3 (I found the number in the makefile).
Indeed, that version pre-dates the -T option. If you weren't already
chasing a regression I'd suggest an upgrade to 2.4.mumble. Once you are
at a point where changing another variable won't muddle things you may
want to consider upgrading.
happy benchmarking,
rick jones
On Mon, 2008-01-14 at 21:53 +1100, Herbert Xu wrote:
> On Mon, Jan 14, 2008 at 08:44:40AM +0000, Ilpo J?rvinen wrote:
> >
> > > > I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> > > > but the bisected kernel wasn't stable and went crazy.
> >
> > TCP work between that is very much non-existing.
>
> Make sure you haven't switched between SLAB/SLUB while testing this.
I can make sure. In addition, I tried both SLAB and SLUB and make sure the
regression is still there if CONFIG_SLAB=y.
Thanks,
-yanmin
On Wed, 2008-01-16 at 08:34 +0800, Zhang, Yanmin wrote:
> On Mon, 2008-01-14 at 21:53 +1100, Herbert Xu wrote:
> > On Mon, Jan 14, 2008 at 08:44:40AM +0000, Ilpo Jrvinen wrote:
> > >
> > > > > I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> > > > > but the bisected kernel wasn't stable and went crazy.
> > >
> > > TCP work between that is very much non-existing.
> >
> > Make sure you haven't switched between SLAB/SLUB while testing this.
> I can make sure. In addition, I tried both SLAB and SLUB and make sure the
> regression is still there if CONFIG_SLAB=y.
I retried bisect between 2.6.22 and 2.6.23-rc1. This time, I enabled CONFIG_SLAB=y,
and deleted the warmup procedure in the testing scripts. In addition, bind the 2
processes on the same logical processor. The regression is about 20% which is larger
than the one when binding 2 processes to different core.
The new bisect reported cfs core patch causes it. The results of every step look
stable.
dd41f596cda0d7d6e4a8b139ffdfabcefdd46528 is first bad commit
commit dd41f596cda0d7d6e4a8b139ffdfabcefdd46528
Author: Ingo Molnar <[email protected]>
Date: Mon Jul 9 18:51:59 2007 +0200
sched: cfs core code
apply the CFS core code.
this change switches over the scheduler core to CFS's modular
design and makes use of kernel/sched_fair/rt/idletask.c to implement
Linux's scheduling policies.
thanks to Andrew Morton and Thomas Gleixner for lots of detailed review
feedback and for fixlets.
Signed-off-by: Ingo Molnar <[email protected]>
Signed-off-by: Mike Galbraith <[email protected]>
Signed-off-by: Dmitry Adamushko <[email protected]>
Signed-off-by: Srivatsa Vaddagiri <[email protected]>
-yanmin