As usual:
ftp://ftp.kernel.org/pub/linux/kernel/people/davem/zerocopy-2.4.2p2-1.diff.gz
It's updated to be against the latest (2.4.2-pre2) and I've removed
the non-zerocopy related fixes from the patch (because I've sent them
under seperate cover to Linus).
Enjoy. As usual, I am very seriously interested in any bugs or
performance problems introduced by this patch. Thanks.
Later,
David S. Miller
[email protected]
"David S. Miller" wrote:
>
> As usual:
>
> ftp://ftp.kernel.org/pub/linux/kernel/people/davem/zerocopy-2.4.2p2-1.diff.gz
>
> It's updated to be against the latest (2.4.2-pre2) and I've removed
> the non-zerocopy related fixes from the patch (because I've sent them
> under seperate cover to Linus).
>
Changing the memory copy function did make some difference
in my setup. But the performance drop on send(8k) is only approx 10%,
partly because I changed the way I'm testing it - `cyclesoak' is
now penalised more heavily by cache misses, and amount of cache
missing which networking causes cyclesoak is basically the same,
whether or not the ZC patch is applied.
I tried a number of things to try to optimise this situation
on an SG-capable NIC with the ZC patch:
while (more_to_send) {
read(fd, buf, 8192);
send(sock, buf, 8192);
}
Things I tried:
- Use the csum_copy() functions
- Use copy_from_user()
- Use copy_from_user if src and dest are 8-byte aligned,
else use csum_copy.
- Force data alignment.
Explain: If an application sends a few bytes to a connection
(say, some headers) and then starts pumping bulk data down the
same connection, we end up in the situation where the source of
a copy_from_user is poorly aligned, and it *stays* that way for
the whole operation. This is because new, incoming data is always
tacked onto the end of the socket write buffer.
Copying from a poorly aligned source address takes 1.5 to 2 times
as long, depending upon the combination of source-cached and
dest-cached.
So I special-cased this in tcp_sendmsg: if we see a large write
from userspace and we're poorly aligned then just send out a single
undersized frame so we can drop back into alignment.
This didn't make a lot of difference, which perhaps indicates
that the dominating factor is misses, not alignment. If it
_is_ misses, they're probably due to aliasing - Ingo said his
toy has 2 megs of full-speed L2.
- skbuff_cache.
Explain: When we build an skbuff for ZC transmit it is always
the same size - it only holds the headers. The data is put
into the fragment buffer. So I created a slab cache for
skbuffs whose data length is <= 256 bytes, and used that.
This didn't make much difference.
send(8k), no SG 19.2%
send(8k), SG, csum_copy 20.3%
send(8k), SG, copy_from_user 20.9%
send(8k), SG, choose copy 20.6% (huh?)
send(8k), SG, page-aligned, choose copy 20.3%
send(8k), SG, page-aligned, csum_copy 20.2%
send(8k), SG, csum_copy, skbuff_cache 20.5% (huh?)
send(8k), SG, csum_copy, skbuff_cache, page-aligned 20.2%
send(8k), SG, copy_from_user, skbuff_cache, page-aligned 20.2%
That's all pretty uninteresting, except for the observation
that not using Pentium string ops on un-8byte-aligned is the
biggest win. And the two huhs, the first of which is
bizarre. I've checked that code over and over:
if (((long)_from | (long)_to) & 7)
csum_and_copy()
else
copy_from_user()
and it's slower than an unconditional csum_and_copy(). Wierd.
The profiles are more interesting:
send(8k), no SG 18.2%
=========================================
c0224734 tcp_transmit_skb 47 0.0347
c01127dc schedule 54 0.0340
c021599c ip_output 54 0.1688
c010a768 handle_IRQ_event 55 0.4583
c02041ec skb_release_data 60 0.5357
c0211068 ip_route_input 69 0.1938
c022abac tcp_v4_rcv 75 0.0470
c0215adc ip_queue_xmit 76 0.0571
c0204410 skb_clone 85 0.1986
c0219a54 tcp_sendmsg_copy 99 0.0270
c02209fc tcp_clean_rtx_queue 101 0.1153
c02042c4 __kfree_skb 113 0.3404
c024a3cc csum_partial_copy_generic 436 1.7581
c0125580 file_read_actor 548 6.5238
00000000 total 2874 0.0021
send(8k), SG, csum copy 20.3%
=========================================
c0211068 ip_route_input 47 0.1320
c011be60 del_timer 49 0.6806
c021599c ip_output 49 0.1531
c010a768 handle_IRQ_event 56 0.4667
c022abac tcp_v4_rcv 66 0.0414
c02041ec skb_release_data 69 0.6161
c0215adc ip_queue_xmit 69 0.0518
c0204410 skb_clone 70 0.1636
c02042c4 __kfree_skb 96 0.2892
c02209fc tcp_clean_rtx_queue 100 0.1142
c021b6fc tcp_sendmsg 109 0.0439
c021a8ac do_tcp_sendpages 152 0.0440
c024a3cc csum_partial_copy_generic 520 2.0968
c0125580 file_read_actor 615 7.3214
00000000 total 3222 0.0024
Note that in each and every profile which I've taken with
the ZC patch, file_read_actor() took a big hit. That's
the read from the kernel into userspace. Which confirms
that ZC is more cache-hungry.
Also, note that the sum of ZC's do_tcp_sendpages and tcp_sendmsg
is considerably higher than non-ZC's tcp_sendmsg_copy.
These profiles also show us the aggregate impact of the
ZC patch: 3222/2874 = 1.12. Twelve percent. The device
driver cost isn't shown here because it's in a module.
Here is a profile with a non-modular driver:
c020833c skb_release_data 54 0.3857
c0218d4c ip_queue_xmit 63 0.0497
c010a768 handle_IRQ_event 70 0.5833
c024d4f8 __strncpy_from_user 74 2.0556
c021e53c tcp_sendmsg 75 0.0302
c022d9ec tcp_v4_rcv 77 0.0482
c022383c tcp_clean_rtx_queue 90 0.1027
c0208434 __kfree_skb 104 0.4000
c021d6ec do_tcp_sendpages 122 0.0353
c01d3a08 boomerang_rx 150 0.1276
c01d32ac boomerang_interrupt 287 0.2781
c01d2c94 boomerang_start_xmit 324 0.4475
c024d1cc csum_partial_copy_generic 443 1.7863
c01255f0 file_read_actor 643 7.6548
00000000 total 3852 0.0028
Device driver cost is 15-20% of the kernel occupancy.
We can pull ZC's 12% cost down a little bit by using a slab
cache on the skbuffs, and by forcing better copy alignment (not
sure how though). Also perhaps by doing some colouring on the data
structures which are being used. These differences are all
quite small, which makes this stuff pretty tricky.
I now need to go off and hunt down a Pentium performance
counter patch which take less than a year to understand,
find out where these alleged misses are happening.
-
Andrew Morton writes:
> Changing the memory copy function did make some difference
> in my setup. But the performance drop on send(8k) is only approx 10%,
> partly because I changed the way I'm testing it - `cyclesoak' is
> now penalised more heavily by cache misses, and amount of cache
> missing which networking causes cyclesoak is basically the same,
> whether or not the ZC patch is applied.
Ok ok ok, but are we at the point where there are no sizable "over the
wire" performance anomalies anymore? That is what is important, what
are the localhost bandwidth measurements looking like for you now
with/without the patch applied?
I want to reach a known state where we can conclude "over the wire is
about as good or better than before, but there is a cpu/cache usage
penalty from the zerocopy stuff".
This is important. It lets us get to the next stage which is to
use your tools, numbers, and some profiling to see if we can get
some of that cpu overhead back.
Later,
David S. Miller
[email protected]
"David S. Miller" wrote:
>
> Andrew Morton writes:
> > Changing the memory copy function did make some difference
> > in my setup. But the performance drop on send(8k) is only approx 10%,
> > partly because I changed the way I'm testing it - `cyclesoak' is
> > now penalised more heavily by cache misses, and amount of cache
> > missing which networking causes cyclesoak is basically the same,
> > whether or not the ZC patch is applied.
>
> Ok ok ok, but are we at the point where there are no sizable "over the
> wire" performance anomalies anymore? That is what is important, what
> are the localhost bandwidth measurements looking like for you now
> with/without the patch applied?
Using 2.4.2-pre3 + zerocopy-2.4.2p3-1.diff
All numbers in megs/sec
zcc/zcs is doing read(8k)/send(8k) to localhost.
On the dual 500MHz PII:
zcc/zcs bw_tcp
Unpatched: 70 66
Patched: 67 66
Single 500MHz PII:
Unpatched: 58 54
Patched: 49 52
Single 650MHz PIII Coppermine:
Unpatched: 140 180-250
Patched: 107 159
With or without ZC, there is Wierd Stuff happening with local
networking. Throughput is all over the place.
- With zcs reporting throughput once per second, the numbers were jumping
around by +/-10%. Had to bump the averaging period to 5 seconds to
make much sense of it. With a real network, they're rock solid.
- The difference between the PII and PIII is far beyond anything I see
with any other workload.
- The difference between zcc/zcs and bw_tcp on the PIII is interesting.
It's still apparent when zcc/zcs uses a 64k transfer buffer, like bw_tcp.
zcc/zcs is doing file system reads, whereas bw_tcp isn't. But the
discrepancy isn't there on the PII.
- On the unpatched kernel, I saw one bw_tcp run after a reboot report
410 Mbytes/sec. Thereafter it's around 210. err.. make that 180. No,
make that 254. WTF?
Amongst all the noise it seems there's a problem on the PIII but
not the PII.
It's getting very lonely testing this stuff. It would be useful if
someone else could help out - at least running the bw_tcp tests. It's
pretty simple:
bw_tcp -s ; bw_tcp 0
> I want to reach a known state where we can conclude "over the wire is
> about as good or better than before, but there is a cpu/cache usage
> penalty from the zerocopy stuff".
>
> This is important. It lets us get to the next stage which is to
> use your tools, numbers, and some profiling to see if we can get
> some of that cpu overhead back.
Seems, with the 100baseT NIC the performance drop on the Coppermine
is only half that of the Mendocino. I _think_ the Mendocino is
only 4-way associative, but reports vary on this. Coppermine is 8-way.
-
On Wed, Feb 14, 2001 at 12:27:10AM +1100, Andrew Morton wrote:
>
> It's getting very lonely testing this stuff. It would be useful if
> someone else could help out - at least running the bw_tcp tests. It's
> pretty simple:
>
> bw_tcp -s ; bw_tcp 0
OK, here's my bw_tcp results on a K6-2 450. I ran bw_tcp 10 times, then
averaged the results.
bw_tcp
2.4.2-pre3 57.0
2.4.2-pre3zc 52.6
-Dave