Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935607AbZDCRxR (ORCPT ); Fri, 3 Apr 2009 13:53:17 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1760580AbZDCRw6 (ORCPT ); Fri, 3 Apr 2009 13:52:58 -0400 Received: from mail-bw0-f169.google.com ([209.85.218.169]:45293 "EHLO mail-bw0-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755627AbZDCRw5 (ORCPT ); Fri, 3 Apr 2009 13:52:57 -0400 DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=iBpSvBY56S6Mt61HWhXteTDix3dJowakyhLM3Zw2S7uMTYU99LS1UrfDxaAqfWHv+9 0mI1l7hDxqfZwk9vJVGEx2MGf6ztlmiJN8CWNu/g0nahNWGj+NRn2ufdYEkCpVLdKVwt 0zWVJWMT7KmaxP5drRa9yHQe+5pr3o2dgdbmk= MIME-Version: 1.0 In-Reply-To: References: Date: Fri, 3 Apr 2009 19:52:53 +0200 Message-ID: Subject: Re: [Scst-devel] ISCSI-SCST performance (with also IET and STGTdata) From: Bart Van Assche To: "Sufficool, Stanley" Cc: Vladislav Bolkhovitin , scst-devel , linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2490 Lines: 57 On Fri, Apr 3, 2009 at 7:13 PM, Sufficool, Stanley wrote: >> On Thu, Apr 2, 2009 at 7:16 PM, Vladislav Bolkhovitin >> wrote: >> > Bart Van Assche, on 04/02/2009 12:14 AM wrote: >> >> I have repeated some of these performance tests for iSCSI >> >> over IPoIB >> >> (two DDR PCIe 1.0 ConnectX HCA's connected back to back). >> >> The results >> >> for the buffered I/O test with a block size of 512K (initiator) >> >> against a file of 1GB residing on a tmpfs filesystem on the target >> >> are as follows: >> >> >> >> write-test: iSCSI-SCST 243 MB/s; IET 192 MB/s. >> >> read-test: iSCSI-SCST 291 MB/s; IET 223 MB/s. >> >> >> >> And for a block size of 4 KB: >> >> >> >> write-test: iSCSI-SCST 43 MB/s; IET 42 MB/s. >> >> read-test: iSCSI-SCST 288 MB/s; IET 221 MB/s. >> > >> > Do you have any thoughts why writes are so bad? It shouldn't be so.. >> >> It's not impossible that with the 4 KB write test I hit the >> limits of the initiator system (Intel E6750 CPU, 2.66 GHz, >> two cores). Some statistics I gathered during the 4 KB write test: >> Target: CPU load 0.5, 16500 mlx4-comp-0 interrupts per >> second, same number of interrupts processed by each core (8250/s). >> Initiator: CPU load 1.0, 32850 mlx4-comp-0 interrupts per >> second, all interrupts occurred on the same core. > > Are you using connected mode IPoIB and setting the MTU to 4KB? Would > fragmentation of IPoIB drive up the interrupt rates? All tests have been run with default IPoIB settings: an MTU of 2044 bytes and datagram mode. The following data has been obtained from the target system after several 4 KB write tests: $ cat /sys/class/net/ib0/mode datagram $ /sbin/ifconfig ib0 ib0 Link encap:UNSPEC HWaddr 80-00-00-48-FE-80-00-00-00-00-00-00-00-00-00-00 inet addr:192.168.2.1 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::202:c903:2:d217/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 RX packets:88482013 errors:0 dropped:0 overruns:0 frame:0 TX packets:38444824 errors:0 dropped:11 overruns:0 carrier:0 collisions:0 txqueuelen:256 RX bytes:135770573672 (129480.9 Mb) TX bytes:5647702210 (5386.0 Mb) Bart. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/