Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759733AbbFCM6L (ORCPT ); Wed, 3 Jun 2015 08:58:11 -0400 Received: from verein.lst.de ([213.95.11.211]:52310 "EHLO newverein.lst.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757094AbbFCM57 (ORCPT ); Wed, 3 Jun 2015 08:57:59 -0400 Date: Wed, 3 Jun 2015 14:57:56 +0200 From: Christoph Hellwig To: "Nicholas A. Bellinger" Cc: target-devel , linux-scsi , linux-kernel , Christoph Hellwig , Hannes Reinecke , Sagi Grimberg , Nicholas Bellinger Subject: Re: [RFC 0/2] target: Add TFO->complete_irq queue_work bypass Message-ID: <20150603125756.GA19696@lst.de> References: <1432281446-31080-1-git-send-email-nab@daterainc.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1432281446-31080-1-git-send-email-nab@daterainc.com> User-Agent: Mutt/1.5.17 (2007-11-01) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 6580 Lines: 110 This makes lockdep very unhappy, rightly so. If you execute one end_io function inside another you basіcally nest every possible lock taken in the I/O completion path. Also adding more work to the hardirq path generally isn't a smart idea. Can you explain what issues you were seeing and how much this helps? Note that the workqueue usage in the target core so far is fairly basic, so there should some low hanging fruit. [ 21.119148] [ 21.119382] ============================================= [ 21.120012] [ INFO: possible recursive locking detected ] [ 21.120639] 4.1.0-rc1+ #489 Not tainted [ 21.121131] --------------------------------------------- [ 21.121754] swapper/0/0 is trying to acquire lock: [ 21.122324] (&(&fq->mq_flush_lock)->rlock){-.....}, at: [] flush_end_io+0x66/0x220 [ 21.122451] [ 21.122451] but task is already holding lock: [ 21.122451] (&(&fq->mq_flush_lock)->rlock){-.....}, at: [] flush_end_io+0x66/0x220 [ 21.122451] [ 21.122451] other info that might help us debug this: [ 21.122451] Possible unsafe locking scenario: [ 21.122451] [ 21.122451] CPU0 [ 21.122451] ---- [ 21.122451] lock(&(&fq->mq_flush_lock)->rlock); [ 21.122451] lock(&(&fq->mq_flush_lock)->rlock); [ 21.122451] [ 21.122451] *** DEADLOCK *** [ 21.122451] [ 21.122451] May be due to missing lock nesting notation [ 21.122451] [ 21.122451] 3 locks held by swapper/0/0: [ 21.122451] #0: (&(&vp_dev->lock)->rlock){-.-...}, at: [] vp_vring_interrupt+0x2c/0x90 [ 21.122451] #1: (&(&virtscsi_vq->vq_lock)->rlock){-.-...}, at: [] virtscsi_vq_done+0x26/0x90 [ 21.122451] #2: (&(&fq->mq_flush_lock)->rlock){-.....}, at: [] flush_end_io+0x66/0x220 [ 21.122451] [ 21.122451] stack backtrace: [ 21.122451] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.1.0-rc1+ #489 [ 21.122451] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007 [ 21.122451] ffffffff82c34820 ffff88007fc03618 ffffffff81e3396f 0000000000000000 [ 21.122451] ffffffff82445500 ffff88007fc036c8 ffffffff811209a9 ffff88007fc17d18 [ 21.122451] 0000000000000000 0000000000000000 ffffffff82c34820 ffff88007fc17d18 [ 21.122451] Call Trace: [ 21.122451] [] dump_stack+0x45/0x57 [ 21.122451] [] validate_chain.isra.37+0xd39/0x1160 [ 21.122451] [] __lock_acquire+0x488/0xd40 [ 21.122451] [] ? __kernel_text_address+0x58/0x80 [ 21.122451] [] lock_acquire+0xaf/0x130 [ 21.122451] [] ? flush_end_io+0x66/0x220 [ 21.122451] [] _raw_spin_lock_irqsave+0x49/0x60 [ 21.122451] [] ? flush_end_io+0x66/0x220 [ 21.122451] [] flush_end_io+0x66/0x220 [ 21.122451] [] __blk_mq_end_request+0x2f/0x70 [ 21.122451] [] scsi_end_request+0x7d/0x1e0 [ 21.122451] [] scsi_io_completion+0x110/0x610 [ 21.122451] [] ? __lock_acquire+0x488/0xd40 [ 21.122451] [] scsi_finish_command+0xdb/0x140 [ 21.122451] [] scsi_softirq_done+0x136/0x150 [ 21.122451] [] __blk_mq_complete_request+0x8e/0x130 [ 21.122451] [] blk_mq_complete_request+0x29/0x30 [ 21.122451] [] scsi_mq_done+0x28/0x60 [ 21.122451] [] tcm_loop_queue_status+0x3b/0xb0 [ 21.122451] [] target_complete_irq+0x8a/0x250 [ 21.122451] [] target_complete_cmd+0x1fa/0x2a0 [ 21.122451] [] iblock_end_io_flush+0x2a/0x60 [ 21.122451] [] bio_endio+0x53/0x90 [ 21.122451] [] blk_update_request+0x98/0x360 [ 21.122451] [] blk_mq_end_request+0x1e/0x80 [ 21.122451] [] blk_flush_complete_seq+0xe4/0x320 [ 21.122451] [] flush_end_io+0x138/0x220 [ 21.122451] [] __blk_mq_end_request+0x2f/0x70 [ 21.122451] [] scsi_end_request+0x7d/0x1e0 [ 21.122451] [] scsi_io_completion+0x110/0x610 [ 21.122451] [] ? trace_hardirqs_off+0xd/0x10 [ 21.122451] [] scsi_finish_command+0xdb/0x140 [ 21.122451] [] scsi_softirq_done+0x136/0x150 [ 21.122451] [] __blk_mq_complete_request+0x8e/0x130 [ 21.122451] [] blk_mq_complete_request+0x29/0x30 [ 21.122451] [] scsi_mq_done+0x28/0x60 [ 21.122451] [] virtscsi_complete_cmd+0x10f/0x1e0 [ 21.122451] [] ? virtscsi_ctrl_done+0x30/0x30 [ 21.122451] [] virtscsi_vq_done+0x49/0x90 [ 21.122451] [] virtscsi_req_done+0x39/0x40 [ 21.122451] [] vring_interrupt+0x30/0x60 [ 21.122451] [] vp_vring_interrupt+0x5b/0x90 [ 21.122451] [] handle_irq_event_percpu+0x60/0x1d0 [ 21.122451] [] handle_irq_event+0x43/0x70 [ 21.122451] [] handle_edge_irq+0x96/0x110 [ 21.122451] [] handle_irq+0x58/0x130 [ 21.122451] [] ? atomic_notifier_call_chain+0x11/0x20 [ 21.122451] [] do_IRQ+0x57/0x100 [ 21.122451] [] common_interrupt+0x73/0x73 [ 21.122451] [] ? native_safe_halt+0x6/0x10 [ 21.122451] [] ? trace_hardirqs_on+0xd/0x10 [ 21.122451] [] default_idle+0x1e/0xc0 [ 21.122451] [] arch_cpu_idle+0xa/0x10 [ 21.122451] [] cpu_startup_entry+0x383/0x430 [ 21.122451] [] rest_init+0x128/0x130 [ 21.122451] [] ? csum_partial_copy_generic+0x170/0x170 [ 21.122451] [] start_kernel+0x54a/0x557 [ 21.122451] [] ? set_init_arg+0x58/0x58 [ 21.122451] [] ? early_idt_handlers+0x117/0x120 [ 21.122451] [] x86_64_start_reservations+0x2a/0x2c [ 21.122451] [] x86_64_start_kernel+0x13e/0x14d -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/