Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757617AbaAIV1P (ORCPT ); Thu, 9 Jan 2014 16:27:15 -0500 Received: from mail-we0-f175.google.com ([74.125.82.175]:47822 "EHLO mail-we0-f175.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757570AbaAIV1B (ORCPT ); Thu, 9 Jan 2014 16:27:01 -0500 MIME-Version: 1.0 In-Reply-To: <52CEFE62.7070009@interlog.com> References: <20140106201032.GA13491@quack.suse.cz> <52CC6A53.9010508@emulex.com> <52CEFE62.7070009@interlog.com> From: Sergey Meirovich Date: Thu, 9 Jan 2014 23:26:40 +0200 Message-ID: Subject: Re: Terrible performance of sequential O_DIRECT 4k writes in SAN environment. ~3 times slower then Solars 10 with the same HBA/Storage. To: dgilbert@interlog.com Cc: James Smart , Jan Kara , linux-scsi , Linux Kernel Mailing List , Gluk , Christoph Hellwig Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Duglas, On 9 January 2014 21:54, Douglas Gilbert wrote: > On 14-01-08 08:57 AM, Sergey Meirovich wrote: ... >> >> The strangest thing to me that this is the problem with sequential >> write. For example the fnic one machine is zoned to EMC XtremIO and >> had results: 14.43Mb/sec 3693.65 Requests/sec for sequential 4k. The >> same fnic machine perfrormed rather impressive for random 4k >> 451.11Mb/sec 115485.02 Requests/sec > > > You could bypass O_DIRECT and use ddpt together with > a bsg pass-through (bsg is a little faster than sg > for these purposes). > > For example: > > # lsscsi -g > [0:0:0:0] disk ATA INTEL SSDSC2CW12 400i /dev/sda /dev/sg0 > [14:0:0:0] disk Linux scsi_debug 0004 - /dev/sg1 > > # ddpt if=/dev/bsg/14:0:0:0 bs=512 bpt=128 count=1m > Output file not specified so no copy, just reading input > 1048576+0 records in > 0+0 records out > time to read data: 0.283566 secs at 1893.28 MB/sec > > bs= should match the block size of the storage device and > the size of each SCSI READ is dictated by bpt= (so 64 KB > in this case). > > Such a test should show you if your performance problem > is in the block layer or below, or above the block layer > (at least the point where pass-through commands are > injected). > > Doug Gilbert Thanks for an excellent idea! Seems like this is not Direct IO issue.Just tried it against fnic/XtremIO. 4k over via bsg is still 17.278 Mb/s for write. . [root@dca-poc-gtsxdb3 bsg]# /tmp/ddpt-0.91/src/ddpt if=/dev/zero of=/dev/bsg/0:0:4:1 bs=512 bpt=8 count=1m 1048576+0 records in 1048576+0 records out time to transfer data: 31.076487 secs at 17.28 MB/sec [root@dca-poc-gtsxdb3 bsg]# /tmp/ddpt-0.91/src/ddpt if=/dev/zero of=/dev/bsg/0:0:4:1 bs=512 bpt=16 count=512k 524288+0 records in 524288+0 records out time to transfer data: 8.511421 secs at 31.54 MB/sec [root@dca-poc-gtsxdb3 bsg]# /tmp/ddpt-0.91/src/ddpt if=/dev/zero of=/dev/bsg/0:0:4:1 bs=512 bpt=32 count=256k 262144+0 records in 262144+0 records out time to transfer data: 2.426037 secs at 55.32 MB/sec [root@dca-poc-gtsxdb3 bsg]# -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/