Return-path: Received: from mail-wi0-f179.google.com ([209.85.212.179]:56701 "EHLO mail-wi0-f179.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751202AbbBDLfY convert rfc822-to-8bit (ORCPT ); Wed, 4 Feb 2015 06:35:24 -0500 Received: by mail-wi0-f179.google.com with SMTP id l15so2959007wiw.0 for ; Wed, 04 Feb 2015 03:35:22 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <1422973660.907.10.camel@edumazet-glaptop2.roam.corp.google.com> References: <1422537297.21689.15.camel@edumazet-glaptop2.roam.corp.google.com> <1422628835.21689.95.camel@edumazet-glaptop2.roam.corp.google.com> <1422903136.21689.114.camel@edumazet-glaptop2.roam.corp.google.com> <1422926330.21689.138.camel@edumazet-glaptop2.roam.corp.google.com> <1422973660.907.10.camel@edumazet-glaptop2.roam.corp.google.com> Date: Wed, 4 Feb 2015 12:35:22 +0100 Message-ID: (sfid-20150204_123532_386077_F584936C) Subject: Re: Throughput regression with `tcp: refine TSO autosizing` From: Michal Kazior To: Eric Dumazet Cc: linux-wireless , Network Development , eyalpe@dev.mellanox.co.il Content-Type: text/plain; charset=UTF-8 Sender: linux-wireless-owner@vger.kernel.org List-ID: On 3 February 2015 at 15:27, Eric Dumazet wrote: > On Tue, 2015-02-03 at 12:50 +0100, Michal Kazior wrote: [...] >> IOW: >> - stretch acks / TSO defer don't seem to help much (when compared to >> throughput results from yesterday) >> - GRO helps >> - disabling A-MSDU on sender helps >> - net/master+GRO still doesn't reach the performance from before the >> regression (~600mbps w/ GRO) >> >> You can grab logs and dumps here: http://www.filedropper.com/test2tar >> > > Thanks for these traces. > > There is absolutely a problem at the sender, as we can see a big 2ms > delay between reception of ACK and send of following packets. > TCP stack should generate them immediately. > Are you using some kind of netem qdisc ? Both systems have identical setup: ; tc qdisc qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc mq 0: dev wlan1 root qdisc pfifo_fast 0: dev wlan1 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev wlan1 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev wlan1 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev wlan1 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 > These 2ms delays, in a flow with a 5ms RTT are terrible. > > 06:54:57.408391 IP 192.168.1.2.5001 > 192.168.1.3.51645: Flags [.], ack 4294899240, win 11268, options [nop,nop,TS val 1053302 ecr 1052250], length 0 > 06:54:57.408418 IP 192.168.1.2.5001 > 192.168.1.3.51645: Flags [.], ack 4294910824, win 11268, options [nop,nop,TS val 1053303 ecr 1052251], length 0 > 06:54:57.408431 IP 192.168.1.2.5001 > 192.168.1.3.51645: Flags [.], ack 4294936888, win 11268, options [nop,nop,TS val 1053303 ecr 1052251], length 0 > 06:54:57.408453 IP 192.168.1.2.5001 > 192.168.1.3.51645: Flags [.], ack 4294962952, win 11268, options [nop,nop,TS val 1053303 ecr 1052251], length 0 > 06:54:57.408474 IP 192.168.1.2.5001 > 192.168.1.3.51645: Flags [.], ack 0, win 11268, options [nop,nop,TS val 1053303 ecr 1052251], length 0 > > 06:54:57.410243 IP 192.168.1.3.51645 > 192.168.1.2.5001: Flags [.], seq 82536:83984, ack 1, win 457, options [nop,nop,TS val 1052256 ecr 1053303], length 1448 [...] > > Are packets TX completed after a timer or something ? As far as ath10k is concerned - no timers here. Not sure about firmware itself though. > Some very heavy stuff might run from tasklet (or other softirq triggered) event. > > BTW, traces tend to show that you 'receive' multiple ACK in the same burst, > its not clear if they are delayed at one side or the other. > > GRO should delay only GRO candidates. ACK packets are not GRO candidates. > > Have you tried to disable GSO on sender ? I assume I do that via ethtool? This is my current setup on both systems: ; ethtool -k wlan1 Features for wlan1: rx-checksumming: off [fixed] tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on [fixed] tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off [fixed] tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: off tx-tcp-segmentation: off [fixed] tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off [fixed] udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off [requested on] generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: on [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] ; ethtool -K wlan1 generic-segmentation-offload off ethtool: bad command line argument(s) For more information run ethtool -h > (Or maybe wifi drivers should start to use skb->xmit_more as a signal to end aggregation) This could work if your firmware/device supports this kind of thing. To my understanding ath10k firmware doesn't. Michał