Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754212AbbFDRBL (ORCPT ); Thu, 4 Jun 2015 13:01:11 -0400 Received: from mail-wi0-f173.google.com ([209.85.212.173]:35808 "EHLO mail-wi0-f173.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752937AbbFDRBI (ORCPT ); Thu, 4 Jun 2015 13:01:08 -0400 Message-ID: <55708455.2080500@dev.mellanox.co.il> Date: Thu, 04 Jun 2015 20:01:09 +0300 From: Sagi Grimberg User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: "Nicholas A. Bellinger" , Christoph Hellwig CC: "Nicholas A. Bellinger" , target-devel , linux-scsi , linux-kernel , Hannes Reinecke , Sagi Grimberg Subject: Re: [RFC 0/2] target: Add TFO->complete_irq queue_work bypass References: <1432281446-31080-1-git-send-email-nab@daterainc.com> <20150603125756.GA19696@lst.de> <1433401569.18125.112.camel@haakon3.risingtidesystems.com> In-Reply-To: <1433401569.18125.112.camel@haakon3.risingtidesystems.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1820 Lines: 38 On 6/4/2015 10:06 AM, Nicholas A. Bellinger wrote: > On Wed, 2015-06-03 at 14:57 +0200, Christoph Hellwig wrote: >> This makes lockdep very unhappy, rightly so. If you execute >> one end_io function inside another you basŃ–cally nest every possible >> lock taken in the I/O completion path. Also adding more work >> to the hardirq path generally isn't a smart idea. Can you explain >> what issues you were seeing and how much this helps? Note that >> the workqueue usage in the target core so far is fairly basic, so >> there should some low hanging fruit. > > So I've been using tcm_loop + RAMDISK backends for prototyping, but this > patch is intended for vhost-scsi so it can avoid the unnecessary > queue_work() context switch within target_complete_cmd() for all backend > driver types. > > This is because vhost_work_queue() is just updating vhost_dev->work_list > and immediately wake_up_process() into a different vhost_worker() > process context. For heavy small block workloads into fast IBLOCK > backends, avoiding this extra context switch should be a nice efficiency > win. I can see that, did you get a chance to measure the expected latency improvement? > > Also, AFAIK RDMA fabrics are allowed to do ib_post_send() response > callbacks directly from IRQ context as well. This is correct in general, ib_post_send is not allowed to schedule. isert/srpt might benefit here in latency, but it would require the the drivers to pre-allocate the sgls (ib_sge's) and use a worst-case approach (or use GFP_ATOMIC allocations - I'm not sure which is better...) -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/