Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757116AbbDOR3U (ORCPT ); Wed, 15 Apr 2015 13:29:20 -0400 Received: from mail-oi0-f50.google.com ([209.85.218.50]:33188 "EHLO mail-oi0-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756692AbbDOR3M (ORCPT ); Wed, 15 Apr 2015 13:29:12 -0400 Message-ID: <1429118948.7346.114.camel@edumazet-glaptop2.roam.corp.google.com> Subject: Re: [Xen-devel] "tcp: refine TSO autosizing" causes performance regression on Xen From: Eric Dumazet To: George Dunlap Cc: Jonathan Davies , "xen-devel@lists.xensource.com" , Wei Liu , Ian Campbell , Stefano Stabellini , netdev , Linux Kernel Mailing List , Eric Dumazet , Paul Durrant , Christoffer Dall , Felipe Franciosi , linux-arm-kernel@lists.infradead.org, David Vrabel Date: Wed, 15 Apr 2015 10:29:08 -0700 In-Reply-To: <552E9E8D.1080000@eu.citrix.com> References: <1428596218.25985.263.camel@edumazet-glaptop2.roam.corp.google.com> <1428932970.3834.4.camel@edumazet-glaptop2.roam.corp.google.com> <1429115934.7346.107.camel@edumazet-glaptop2.roam.corp.google.com> <552E9E8D.1080000@eu.citrix.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.10.4-0ubuntu2 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3021 Lines: 65 On Wed, 2015-04-15 at 18:23 +0100, George Dunlap wrote: > On 04/15/2015 05:38 PM, Eric Dumazet wrote: > > My thoughts that instead of these long talks you should guys read the > > code : > > > > /* TCP Small Queues : > > * Control number of packets in qdisc/devices to two packets / or ~1 ms. > > * This allows for : > > * - better RTT estimation and ACK scheduling > > * - faster recovery > > * - high rates > > * Alas, some drivers / subsystems require a fair amount > > * of queued bytes to ensure line rate. > > * One example is wifi aggregation (802.11 AMPDU) > > */ > > limit = max(2 * skb->truesize, sk->sk_pacing_rate >> 10); > > limit = min_t(u32, limit, sysctl_tcp_limit_output_bytes); > > > > > > Then you'll see that most of your questions are already answered. > > > > Feel free to try to improve the behavior, if it does not hurt critical workloads > > like TCP_RR, where we we send very small messages, millions times per second. > > First of all, with regard to critical workloads, once this patch gets > into distros, *normal TCP streams* on every VM running on Amazon, > Rackspace, Linode, &c will get a 30% hit in performance *by default*. > Normal TCP streams on xennet *are* a critical workload, and deserve the > same kind of accommodation as TCP_RR (if not more). The same goes for > virtio_net. > > Secondly, according to Stefano's and Jonathan's tests, > tcp_limit_output_bytes completely fixes the problem for Xen. > > Which means that max(2*skb->truesize, sk->sk_pacing_rate >>10) is > *already* larger for Xen; that calculation mentioned in the comment is > *already* doing the right thing. > > As Jonathan pointed out, sysctl_tcp_limit_output_bytes is overriding an > automatic TSQ calculation which is actually choosing an effective value > for xennet. > > It certainly makes sense for sysctl_tcp_limit_output_bytes to be an > actual maximum limit. I went back and looked at the original patch > which introduced it (46d3ceabd), and it looks to me like it was designed > to be a rough, quick estimate of "two packets outstanding" (by choosing > the maximum size of the packet, 64k, and multiplying it by two). > > Now that you have a better algorithm -- the size of 2 actual packets or > the amount transmitted in 1ms -- it seems like the default > sysctl_tcp_limit_output_bytes should be higher, and let the automatic > TSQ you have on the first line throttle things down when necessary. I asked you guys to make a test by increasing sysctl_tcp_limit_output_bytes You have no need to explain me the code I wrote, thank you. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/