Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751434AbaBHJSi (ORCPT ); Sat, 8 Feb 2014 04:18:38 -0500 Received: from infra.glanzmann.de ([88.198.249.254]:60531 "EHLO infra.glanzmann.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751063AbaBHJSd (ORCPT ); Sat, 8 Feb 2014 04:18:33 -0500 Date: Sat, 8 Feb 2014 10:18:28 +0100 From: Thomas Glanzmann To: Eric Dumazet , "David S. Miller" Cc: "Nicholas A. Bellinger" , target-devel , Linux Network Development , LKML Subject: REGRESSION f54b311142a92ea2e42598e347b84e1655caf8e3 tcp auto corking slows down iSCSI file system creation by factor of 70 [WAS: 4 TB VMFS creation takes 15 minutes vs 26 seconds] Message-ID: <20140208091828.GA16336@glanzmann.de> Mail-Followup-To: Eric Dumazet , "David S. Miller" , "Nicholas A. Bellinger" , target-devel , Linux Network Development , LKML References: <20140206153640.GB4103@glanzmann.de> <1391727771.14985.41.camel@haakon3.risingtidesystems.com> <20140207051500.GB10916@glanzmann.de> <20140207075536.GB17815@glanzmann.de> <1391801597.1155.28.camel@haakon3.risingtidesystems.com> <20140207205142.GA8609@glanzmann.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140207205142.GA8609@glanzmann.de> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello Eric, > * Thomas Glanzmann [2014-02-07 08:55]: > > Creating a 4 TB VMFS filesystem over iSCSI takes 24 seconds on 3.12 > > and 15 minutes on 3.14.0-rc2+. * Nicholas A. Bellinger [2014-02-07 20:30]: > Would it be possible to try a couple of different stable kernel > versions to help track this down? I bisected[1] it and found the offending commit f54b311 tcp auto corking [2] 'if we have a small send and a previous packet is already in the qdisc or device queue, defer until TX completion or we get more data.' - Description by David S. Miller I gathered a pcap with tcp_autocorking on and off. On: - took 4 seconds to create a 500 GB VMFS file system https://thomas.glanzmann.de/tmp/tcp_auto_corking_on.pcap.bz2 https://thomas.glanzmann.de/tmp/screenshot-mini-2014-02-08-09:45:43.png https://thomas.glanzmann.de/tmp/screenshot-mini-2014-02-08-09:52:28.png Off: - took 2 minutes 24 seconds to create a 500 GB VMFS file system sysctl net.ipv4.tcp_autocorking=0 https://thomas.glanzmann.de/tmp/tcp_auto_corking_off.pcap.bz2 https://thomas.glanzmann.de/tmp/screenshot-mini-2014-02-08-09:46:34.png https://thomas.glanzmann.de/tmp/screenshot-mini-2014-02-08-09:53:17.png First graph can be generated by opening bunziping the file, opening it in wireshark and select Statistics > IO Grap and change the unit to Bytes/Tick. The second graph can be generated by selecting Statistics > TCP Stream Graph > Round Trip Time. You can also see that the round trip time increases by factor 25 at least. I once saw a similar problem with dealyed ACK packets of the paravirtulized network driver in xen it caused that the tcp window filled up and slowed down the throughput from 30 MB/s to less than 100 KB/s the symptom was that the login to a Windows desktop took more than 10 minutes while it used to be below 30 seconds because the profile of the user was loaded slowly from a CIFS server. At that time the culprit were also delayed small packets: ACK packets in the CIFS case. However I only proofed iSCSI regression so far for tcp auto corking but assume we will see many others if we leave it enabled. I found the problem by doing the following: - I compiled kernel by executing the following commands: yes '' | make oldconfig time make -j 24 / make modules_install / mkinitramfs -o /boot/initrd.img-bisect - I cleaned the iSCSI configuration after each test by issuing: /etc/init.d/target stop rm /iscsi?/* /etc/target/* - I configured iSCSI after each reboot cat > lio-v101.conf <